How does smallpond handle the checkpoint of a ray task?

[javencuhk]

Hi,

I am recently reading the source code of smallpond and curious about the checkpoint mechanism.

It seems that smallepond uses the file under the the folder "data/stating/started_tasks/" for recording checkpoint, however, I have not found any place that a ray task writes its state to this file.

Could anyone help to illuistrate this? Much appreciate！

[wangrunji0408]

The task only creates an empty file under this directory to indicate the task has started. It also dumps the output to a file under the "completed_tasks" directory, indicating the task has completed. Checkpointing happens at the task level. The entire job can be resumed from completed tasks.