Copia runs e-commerce for customers that most platforms ignore — including rural households ordering through a network of local agents. Behind that sits an unforgiving data problem: 2.5M+ rows a day moving through the pipelines across two countries, operational dashboards the business runs on, and ML systems that have to answer in under a second.
The problem
When I took over the platform, analytics and operations were competing with production traffic for the same databases, reporting freshness was unpredictable, and there was no dedicated data engineering function. The business needed three things at once: transactional infrastructure that would not go down, analytics that reflected reality within minutes rather than days, and a path to production ML — all on a startup budget.
Constraints
- Mission-critical OLTP. Transaction systems needed high availability with fast, tested failover — an outage stops ordering across two countries.
- Freshness people could rely on. Operations teams made same-day decisions from dashboards; "the data warehouse loads overnight" was not acceptable.
- Cost discipline. Every architectural choice had to earn its bill — which ultimately shaped both the CDC design and the MLOps work.
- Team building. The data engineering function had to be hired, mentored, and organised while the platform was being built, with sprints running across Asia and Africa time zones.
Architecture
The platform separates three concerns cleanly. Transactional: PostgreSQL on EC2 with both logical and physical replication across multiple availability zones — 99.95% uptime and <10-second failover for the systems that take orders. Analytical: a Lambda architecture feeding Amazon Redshift, using CDC patterns with incremental loads so the warehouse tracked the OLTP systems within a 15-minute freshness SLO, with Spark handling the heavy transformations. ML: SageMaker-based MLOps serving a recommendation engine at sub-second API latency, whose targeted discounts cut the retargeting programme’s cost by 60%.
Order traffic lands in replicated Postgres; CDC keeps Redshift fresh within 15 minutes; the same warehouse feeds BI and the recommendation models.
Design decision
CDC with incremental loads, rather than nightly full extracts, is what made 15-minute freshness affordable: the pipelines move only what changed, so warehouse cost scales with activity rather than with table size. The ML win was a business one — the recommendation engine’s targeted discounts are what cut the retargeting programme’s cost by 60% — delivered with response times protected by an explicit sub-second budget.
Beyond the pipelines
Automation through AWS Step Functions took backup, monitoring, and maintenance out of human hands, reducing manual operations time by 70%. I directed weekly engineering sprints for distributed teams across Asia and Africa, and established the data governance council and enterprise data strategy that gave data quality, security compliance, and reliability standards an owner across the organisation.