Real-time crypto analytics pipeline (streaming + ML)
My MSc Big Data Analytics capstone: an end-to-end streaming pipeline over live Binance data for 5 cryptocurrencies — Kafka → Spark → MongoDB → a live Streamlit dashboard — with a Random Forest model predicting short-term price direction.
The goal
Take live market data for five cryptocurrencies and turn it into a system that ingests, processes, stores, and visualizes it in real time; and layer a machine- learning model on top to predict short-term price direction. The point wasn't to get rich; it was to build a production-shaped data pipeline, not a notebook.
What I built
An end-to-end streaming pipeline:
- Ingestion: live Binance WebSocket feed for 5 cryptocurrencies, into Apache Kafka (dual-listener setup).
- Processing: a Spark consumer reading off Kafka.
- Storage: MongoDB.
- Monitoring: Kafka UI and Mongo Express for pipeline observability.
- Dashboard: a Streamlit + Plotly front end refreshing every 5 seconds.
It sustained 18,000+ records/hour with zero consumer lag.
What it looks like
The machine-learning part
I led the modelling: a Random Forest classifier trained on 103k+ real 1-minute candles, with engineered features (RSI, MA7/MA25, volatility). It reached 52.09% directional accuracy.
I'm deliberately not dressing that number up. On noisy 1-minute crypto data, just beating a coin-flip with a disciplined, leak-free feature pipeline is the honest, defensible result , also knowing the difference between that and an over-fit "90% accuracy" is itself the point of the exercise.
What I learned
The modelling was the small part; the engineering was the real lesson. Keeping a streaming system healthy; no consumer lag, clean back-pressure, observable at every stage, and that was where most of the difficulty (and most of the value) actually lived.