All projects

Data Engineering / Project

Real-time stock market data pipeline

A Kafka pipeline that replays historical market data, computes indicators and serves Parquet-backed analytics through Streamlit and FastAPI.

My role
Data pipeline & API development
Period
Sep — Dec 2025
Technical indicators
6
Messages / batch
100

Technical indicators and processing batch size.

Inside the system

Follow a market event

Historical records are replayed as a stream; this is a data engineering project, not a live trading service.

  1. 01

    Historical CSV

    Replay market records

  2. 02

    Producer + Kafka

    Indicators and message stream

  3. 03

    Consumer

    Batches of 100 messages

  4. 04

    Parquet

    Local storage or S3

  • Streamlit

    Explore the processed data

  • FastAPI

    Query through REST endpoints

01 / The context

What the project addresses

A data engineering project that simulates a market stream from historical CSV data. Kafka connects ingestion to analytics, batch storage, a dashboard and an API.

02 / My contribution

What I worked on

Developed ingestion, transformation and orchestration, technical indicators, storage, dashboard and API functionality.

The challenge

Make streaming market data usable across storage, analytics and application layers.

03 / The approach

How the pieces fit together

  1. Replay historical CSV records as Kafka messages and normalize nested fields and dates.
  2. Collect batches of 100 messages and write Parquet files to local storage or S3.
  3. Compute six indicators, including SMA, EMA, RSI and volatility, with price and volume alerts.
  4. Expose the processed data through Streamlit and FastAPI; use Docker Compose for the Kafka environment.

04 / The outcome

What the work delivered

  • Connected simulated streaming, analytics, storage and serving in a single project.
  • Made the ingestion-to-application workflow inspectable through public producer, consumer, analytics and API code.
Next case study

Finding the signal in sales