All Projects
Data EngineeringProduction
Data Engine
Spreadsheet Ingestion & Validation Pipeline
PythonPostgreSQLDockerWorkers
Overview
A staged pipeline that validates, transforms, and loads structured spreadsheet data.
5
Application Layers (stages)
The Problem
Onboarding spreadsheets were large and messy. Before anything reached production, the raw input had to pass several checks: correctness, aggregate consistency, and business rules. Only then could it be transformed and loaded.
What I Built
A pipeline split into upload validation, aggregate validation, review validation, transformation, and production loading. The work runs in background workers so requests are never blocked, and errors are tracked per batch so we can fix and reprocess just the bad rows.
Architecture
Workers consume uploaded files, run each validation stage, and write clean data into PostgreSQL. Every stage is isolated, so a failure stays contained and retryable. Docker keeps development and production consistent.
Technical Decisions
- Kept stages isolated so a failure in one can't corrupt the rest.
- Moved file processing to background workers so requests never block.
- Kept PostgreSQL as the single source of truth for validated tenant data.
- Tracked errors per batch so we could reprocess exactly what failed.
Validation depth
Three validation levels had to run in order, and nothing partial could leak into production between them.
Error recovery
Bad rows needed clear reporting and a way to be fixed and reprocessed without losing the good data around them.
My Contribution
- Designed and built the pipeline.
- Implemented the validation, transformation, and loading stages.
- Set up the background workers and Docker deployment.
- Added error handling and state tracking so recovery is reliable.