Jobs Do Not Always Start Or Complete

[Zach-Attach]

Describe the bug
Jobs Do not always complete OR do not even record anything in the first place.

To Reproduce
Steps to reproduce the behavior:

1. Run code
2. Some of the brain/env combos will not have any results in them (empty csv) OR will not complete (no models/latest_model.zip).

[Zach-Attach]

It appears that this does not happen when running a single brain/env combo in a single run of the script

[desaibhargav]

I'm interested in taking a deeper look at this as well, I'll keep this thread posted with what I find

[Zach-Attach]

Looks like the issue can be avoided when turning off checkpoints while running the script with a single job, waiting 120 seconds between the start of each run. It appears that part of the issue is related to saving checkpoints, so I have made saving checkpoints now optional.

[desaibhargav]

I just wanted to drop an update here. I've taken a look at this and the issue seems to be in the workers. It's also tricky to get the workers to return the correct errors, they just exit. I'll post another update by the end of the week.