Short Description
In the world of digital advertising, calculating the Click-Through Rate (CTR) in real-time is crucial for campaign optimization. This talk provides a practical, end-to-end demonstration of how to build a robust stream processing pipeline to tackle this challenge using familiar Python tools.
We will walk through a complete project that ingests two separate data streams (impressions and clicks) from a Go-based producer into Kafka. Then, we'll dive deep into a PyFlink job that performs a time-bound Interval Join to correlate clicks with their corresponding impressions, aggregates the results over 30-second Tumbling Windows, and calculates the real-time CTR.
Attendees will leave with a clear understanding of stream processing fundamentals and practical patterns for using PyFlink. We'll cover key concepts like watermarking, checkpointing for exactly-once guarantees. The session will conclude with a live demo showcasing the entire pipeline in action.
Agenda
1. The "Why": The Business Case for Real-Time CTR
- What is CTR and why is it a critical metric?
- The limitations of batch processing for this use case.
- High-level overview of our streaming architecture.
2. The Stack: Our Cast of Characters
- Go Data Producer: Simulating realistic user impression and click data.
- Apache Kafka: The resilient message bus decoupling our services.
- Apache Flink (via PyFlink): The core engine for stateful stream processing, accessible to Python developers.
- Docker: Containerizing our entire environment for easy setup and teardown.
3. The "How": Deep Dive into the PyFlink Job
- Defining Sources and Sinks: Reading from Kafka and writing partitioned CSVs to the filesystem.
- Time is Everything: Understanding Event Time, Timestamps (
TIMESTAMP_LTZ), and Watermarks.
- The Core Logic: Interval Joins: Correlating an impression with a click only if it occurs within a 15-second window.
- Windowed Aggregations: Using 30-second Tumbling Windows to group data and calculate impressions, clicks, and the final CTR per campaign.
- Exactly-Once Guarantees: How Flink's checkpointing mechanism works with the File Sink to prevent data loss or duplication.
4. Live Demo!
- Execute a single script (
run_demo.sh) to launch the entire stack.
- Show the Flink UI with the running job graph.
- Briefly view the raw data flowing through Kafka topics.
- Run a Python script (
read_results.py) to parse the output files and display the final, aggregated CTR data in a Pandas DataFrame.
5. Q&A
Short Description
In the world of digital advertising, calculating the Click-Through Rate (CTR) in real-time is crucial for campaign optimization. This talk provides a practical, end-to-end demonstration of how to build a robust stream processing pipeline to tackle this challenge using familiar Python tools.
We will walk through a complete project that ingests two separate data streams (impressions and clicks) from a Go-based producer into Kafka. Then, we'll dive deep into a PyFlink job that performs a time-bound
Interval Jointo correlate clicks with their corresponding impressions, aggregates the results over 30-secondTumbling Windows, and calculates the real-time CTR.Attendees will leave with a clear understanding of stream processing fundamentals and practical patterns for using PyFlink. We'll cover key concepts like watermarking, checkpointing for exactly-once guarantees. The session will conclude with a live demo showcasing the entire pipeline in action.
Agenda
1. The "Why": The Business Case for Real-Time CTR
2. The Stack: Our Cast of Characters
3. The "How": Deep Dive into the PyFlink Job
TIMESTAMP_LTZ), and Watermarks.4. Live Demo!
run_demo.sh) to launch the entire stack.read_results.py) to parse the output files and display the final, aggregated CTR data in a Pandas DataFrame.5. Q&A