Add bundle-local deduplication for time series and fix metric counter… - #693
Conversation
vish-cs
commented
Jul 28, 2026
vish-cs
commented
- Implement bundle-local deduplication in ExtractTimeSeriesFn using @StartBundle and a bundle-scoped HashSet to reduce 95%+ of duplicate time series keys before global Dataflow shuffle.
- Refactor anonymous inner DoFn classes in GraphReader (CountTimeSeriesFn, ExtractSeriesFromOptimizedFn, ExtractObservationsFromOptimizedFn, GraphToNodesFn, GraphToEdgesFn) into static named DoFn classes with private final Counter fields to ensure reliable Beam metric reporting in Dataflow.
- Add unit test testExtractTimeSeriesFn_BundleDeduplication in GraphReaderTest.
Not up to standards ⛔🔴 Issues
|
| Category | Results |
|---|---|
| CodeStyle | 1 minor |
| Complexity | 1 medium |
🟢 Metrics 5 complexity
Metric Results Complexity 5
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request refactors GraphReader by replacing anonymous inner DoFn classes with named static inner classes to improve code organization. It also introduces bundle-scoped deduplication of TimeSeries in ExtractTimeSeriesFn using a transient Set initialized in @StartBundle, and adds a corresponding unit test. The feedback suggests clearing this set in @FinishBundle to prevent potential memory retention on long-running Beam workers.
… serialization - Implement bundle-local deduplication in ExtractTimeSeriesFn using @StartBundle and a bundle-scoped HashSet to reduce 95%+ of duplicate time series keys before global Dataflow shuffle. - Refactor anonymous inner DoFn classes in GraphReader (CountTimeSeriesFn, ExtractSeriesFromOptimizedFn, ExtractObservationsFromOptimizedFn, GraphToNodesFn, GraphToEdgesFn) into static named DoFn classes with private final Counter fields to ensure reliable Beam metric reporting in Dataflow. - Add unit test testExtractTimeSeriesFn_BundleDeduplication in GraphReaderTest.
2 participants
