Industry 4.0 · Backend
Reduced processing latency by approximately 50% through architectural redesign
How I re-architected the backend that processes large-scale inspection data — queues, schedulers, and multithreading — to run 50% faster and unify results across machine types.
Problem
Inspection machines on the factory floor each emit high volumes of data in their own formats, and the line needed a single analytics view fast enough to keep up. That meant moving and transforming large volumes of data reliably and quickly — and correlating results that arrived in different shapes.
Constraints
- High data volume, produced continuously by multiple machine types.
- Results had to be correlated across different inspection machines.
- Processing latency directly affects how usable the analytics are.
- Relational data model with heavy read/write throughput.
Architecture
Key decisions & tradeoffs
Queues + scheduled multithreaded workers
Processing inline with ingestion couldn't keep up. I decoupled it with a job queue and ran scheduled, multithreaded workers, so throughput scaled with the workload and smoothed out bursts from the line — the core of the 50% speedup.
Push heavy transforms into the database
For large relational datasets, moving row-by-row through the app was the bottleneck. I moved set-based work into stored procedures close to the data and let the ORM handle object mapping — cutting round trips and processing time.
A unifying data model across machine types
Different machines speak different formats. I designed a common schema and a normalisation step that turned several machine dialects into one queryable dataset, which is what made cross-machine analytics possible at all.
An interesting failure
The naïve pipeline couldn't keep pace
My first version processed records synchronously as they arrived. It worked in testing and fell behind under real line volume. Re-architecting around queues, scheduling, and multithreading — plus moving transforms into the DB — is where the throughput actually came from.
What I'd carry forward
Throughput problems are architecture problems. The win came from decoupling, scheduling, and doing set-based work near the data — not from micro-optimising the original loop.