Data Engineering / Project
Real-time stock market data pipeline
A Kafka pipeline that replays historical market data, computes indicators and serves Parquet-backed analytics through Streamlit and FastAPI.
- My role
- Data pipeline & API development
- Period
- Sep — Dec 2025
- Technical indicators
- 6
- Messages / batch
- 100
Technical indicators and processing batch size.
Inside the system
Follow a market event
Historical records are replayed as a stream; this is a data engineering project, not a live trading service.
- 01
Historical CSV
Replay market records
- 02
Producer + Kafka
Indicators and message stream
- 03
Consumer
Batches of 100 messages
- 04
Parquet
Local storage or S3
Streamlit
Explore the processed data
FastAPI
Query through REST endpoints
01 / The context
What the project addresses
A data engineering project that simulates a market stream from historical CSV data. Kafka connects ingestion to analytics, batch storage, a dashboard and an API.
02 / My contribution
What I worked on
Developed ingestion, transformation and orchestration, technical indicators, storage, dashboard and API functionality.
The challenge
Make streaming market data usable across storage, analytics and application layers.
03 / The approach
How the pieces fit together
- Replay historical CSV records as Kafka messages and normalize nested fields and dates.
- Collect batches of 100 messages and write Parquet files to local storage or S3.
- Compute six indicators, including SMA, EMA, RSI and volatility, with price and volume alerts.
- Expose the processed data through Streamlit and FastAPI; use Docker Compose for the Kafka environment.
04 / The outcome
What the work delivered
- Connected simulated streaming, analytics, storage and serving in a single project.
- Made the ingestion-to-application workflow inspectable through public producer, consumer, analytics and API code.