As discussed in 23 Friday Coffea meeting:
- A spark partition per basket is probably not the best way forward
- Similarly a spark partition per file probably has too much skew.
It would be reasonable to implement a partitioning that tries to fill a set number of events (like 200k) per partition, to avoid later skew and start off with a reasonable parallelization of the processing task.
As discussed in 23 Friday Coffea meeting:
It would be reasonable to implement a partitioning that tries to fill a set number of events (like 200k) per partition, to avoid later skew and start off with a reasonable parallelization of the processing task.