Skip to content
Back to Projects
Archived
Data Science
Open Source
Aug 2025

FraudStream

A production-grade streaming pipeline that scores financial transactions for fraud in milliseconds rather than hours, built on Kafka, Spark Structured Streaming, and a Snowflake medallion warehouse.

Private repo
8
Fraud patterns
Sub-second
Alert latency
Medallion
Warehouse
HMAC-SHA256
PII
FraudStream project visual

The constraint

Financial institutions process thousands of card transactions a second, and hidden among them are account takeovers, card-testing attacks, and synthetic-identity fraud. Traditional batch processing surfaces these hours or days later — by which point the money is gone and the job is recovery, not prevention.

The approach

A streaming pipeline where transactions arrive through Kafka under Avro schema validation, pass multi-level data-quality checks, get enriched with velocity and anomaly features in Spark Structured Streaming, and land in Snowflake's Bronze/Silver/Gold layers. PII is masked with HMAC-SHA256 before anything is stored.

Architecture
Transactions scored in-stream, masked before they are ever stored.
High level
SourcesMobile · POS · OnlineKafkaAvro + Schema RegistrySpark StreamingFeatures + qualityDetection8 patterns, sub-secondSnowflakeBronze / Silver / Goldingestvalidatescoremasked
Process
  1. 1
    Contract at the edge

    Put Avro and a Schema Registry at ingestion so malformed producers fail at the boundary rather than corrupting downstream layers.

  2. 2
    Quality as a measured signal

    Layered required-field, type, range, and business-rule checks, then scored and trended quality so degradation triggers an alert instead of a silent drift.

  3. 3
    Feature engineering in-stream

    Calculated velocity metrics, risk scores, and geographic anomalies inside the streaming job so detection does not wait on a batch window.

  4. 4
    Mask before you store

    Applied HMAC-SHA256 to PII before the Bronze layer, so no raw identifier is ever written to the warehouse at rest.

Outcomes

  • Eight distinct fraud patterns detected, including card testing, geographic impossibility, merchant collusion, and bust-out fraud
  • Sub-second latency from transaction to alert
  • Schema validation via Avro and a Schema Registry, with quality scoring and trend monitoring
  • ML-ready feature set so a model can be dropped in without re-plumbing the pipeline
What I'd do next
  • Replace the rules layer with a trained model and hold the rules as a fallback and a baseline.
  • Add drift detection on the feature distributions, not just the data-quality checks.
  • Run a replay harness against labelled historical fraud to get real precision/recall numbers rather than pattern coverage.
Details

Engineered a real-time fraud detection pipeline processing financial transactions through Kafka, Spark Structured Streaming, and Snowflake, detecting eight fraud patterns with sub-second transaction-to-alert latency.

Implemented Avro schema validation, multi-level data quality checks with scoring and trend monitoring, and HMAC-SHA256 PII masking ahead of the Bronze layer.

Related projects