Skip to content

Make partitions according to a specified number of events. #51

Description

@lgray

As discussed in 23 Friday Coffea meeting:

  • A spark partition per basket is probably not the best way forward
  • Similarly a spark partition per file probably has too much skew.

It would be reasonable to implement a partitioning that tries to fill a set number of events (like 200k) per partition, to avoid later skew and start off with a reasonable parallelization of the processing task.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions