Loading Portfolio
Preparing your data engineering showcase...
System Initialization
Data Forge
Schema-aware synthetic data platform for databases, APIs, and pipelines. Generates realistic, relational, and privacy-safe data.
May 2024
2 months
Solo project
High
Project Overview
Data Forge is a schema-aware synthetic data generation platform designed for databases, APIs, and data pipelines. It creates highly realistic, relational, and privacy-safe data automatically, allowing developers and data engineers to mock out entire schemas without exposing PII.
System Architecture
Key Features
- Schema-aware relational data generation
- Time-series and realistic historical data mocking
- Domain-specific data packs
- Export to DuckDB, CSV, Parquet
Technical Implementation
architecture
FastAPI backend driving DuckDB storage with a React/Next.js frontend
database
DuckDB
deployment
Local-first and containerized
Challenges & Solutions
[ Diagnostic: Roadblocks ]
- Generating perfectly relational mock data across deep foreign key hierarchies
- Achieving high-throughput generation locally
[ Execution: Resolutions ]
- Implemented intelligent dependency graphs for foreign keys
- Leveraged DuckDB for lightning-fast vectorized mock insertions
Impact & Results
Drastically reduces the time required to securely mock databases for testing while ensuring strict data privacy.
Project Access
Tech Stack
Next.js
FastAPI
DuckDB
Python
React
Tags
Synthetic Data
Pipelines
Privacy
DuckDB
FastAPI
Key Outcomes
- Time-aware data mocking with domain packs
- Export privacy-conscious relational data