Turning location data into
information you can act on.
The customer challenge: raw GPS data is not usable for business decisions as it is. The design principle: work backwards from business impact.
The data was there.
Certainty wasn't.
Vehicles were sending in location data, and an enormous volume of it accumulated every day. But the raw data was not in a state that could support business decisions.
GPS coordinates carry error. Try to determine mechanically which road a vehicle is on right now, and you mistake a highway for the surface road running alongside it, or the travel path jumps around near intersections. The data existed — but nothing certain could be said from it.
At the same time, the volume kept growing with the number of vehicles. Push for accuracy and the processing gets heavier and infrastructure costs swell. Push for real-time and the architecture gets more complex and operations depend on a few specialists. Accuracy, speed, cost, and operability all pulled against each other, and there was no settled answer for where to land. That was the question at the start of the project.
Decide from business impact,
not from technology.
This domain has no single correct answer. The infrastructure works on-premises or in the cloud. There are multiple stream-processing engines to choose from. For the road-matching inference itself, the options range from geometric methods to probabilistic models to machine-learning approaches. Any of them is technically viable.
That is exactly why we did not start from comparing technologies. We defined first how much impact the system would have on the business, and worked backwards from there to the architecture.
Concretely, we put these questions before any technology selection:
- Which business decisions will this output feed into?
- What level of accuracy and freshness do those decisions actually need?
- Is accuracy beyond that level worth additional investment?
- After the build, who keeps operating this?
By the time these questions were answered, the outline of the required architecture was largely decided.
Low-latency work and large-scale analytics
do not belong on the same engine.
Sorting out the business impact made one thing clear: the required processing characteristics were not one thing.
Determining which road each vehicle is on, continuously and statefully per vehicle, is work where real-time performance translates directly into business value. That went onto a stream-processing engine that manages state per vehicle: Flink.
Estimating population-level people flow from the same GPS data is analytical work — extrapolating from a sample to a population. What it needs is not freshness but the ability to digest data at scale. That went onto Spark Streaming.
Fluentd handles collection; Kafka handles messaging. Instead of loading everything onto one engine, the architecture places the right technology against each kind of business value.
Not the highest accuracy —
the least complexity that meets the requirement.
Higher-accuracy inference methods existed. But if one extra step of accuracy sends architectural complexity soaring, makes operations depend on specialists, and inflates infrastructure cost, the impact never repays the spend. Putting “the least complexity that satisfies the business requirement” at the center of the evaluation criteria was the technical core of this engagement.
The physical limits that decide
high-volume streaming.
In streaming-platform design, physical-layer bottlenecks often cap performance before algorithms or engine choice ever come into play. This engagement was architected with the following constraints on the table from the start.
Storage I/O: state management is disk access
In stateful processing that holds per-vehicle driving state, the moment state outgrows memory it moves to disk-based state management (LSM-tree storage such as RocksDB). Random-read performance is what matters there.
- LSM-trees are strong on writes, but reads search across multiple SSTables, producing read amplification
- Background compaction adds write amplification and eats I/O bandwidth, surfacing as latency spikes
- Local NVMe SSD and network storage differ by an order of magnitude in both latency and IOPS. Put large state on network storage and no amount of algorithmic tuning will break the ceiling
The standard countermeasures: TTL design for state (reliably discarding state that is no longer needed), key-design reviews that shrink the state itself, and placing state-heavy work on nodes with local NVMe.
Network bandwidth: shuffle and replication consume it
When relaying high-volume data, network bandwidth becomes the effective throughput ceiling.
- Replication in the messaging layer — copying data for availability — directly multiplies network traffic. With a replica count of three, write bandwidth needs are three times higher by simple arithmetic
- When stream processing redistributes by key (shuffle), data crosses between nodes. Exceed the bandwidth and backpressure propagates upstream until processing stalls
- In the cloud, traffic across availability zones carries both latency and data-transfer cost. Spreading across zones for availability and watching bandwidth cost jump is a frequent pattern
The countermeasures: serialization format (compress payloads with a schema-bearing binary format instead of row-oriented JSON), partition-key design that avoids unnecessary shuffles, and early filtering upstream — dropping clearly unneeded data before it consumes bandwidth.
Find the narrowest point across the whole data path
Across the chain of collection agent, messaging, processing engine, and sink, the narrowest point sets the throughput of the whole. Spend time tuning the processing engine, and if the bottleneck is disk writes in the messaging layer, nothing improves.
In this engagement, we worked backwards from throughput requirements to data volume before implementation, estimated the bandwidth and I/O each layer needed, and only then fixed the architecture. Skip that step and you rebuild the architecture at load-testing time.
Early aggregation lowers cost everywhere downstream
Given the physical constraints, the single most effective move is reducing the volume sent downstream in the first place.
- Filtering and compression at the collection agent
- Thinning out coordinates with no business meaning — duplicate positions from parked vehicles, for example
- Pre-aggregation per window
Cut ten percent upstream and you free ten percent of cost and bandwidth in messaging, processing, and storage alike. The effect compounds the further downstream you go.
What changed.
Raw data became information that supports decisions
What had been a list of coordinates became something the business can use directly: which vehicle is on which road. The value is not that the data exists — it is that decisions can be made from it.
Choosing by cost-effectiveness kept the investment proportionate
Not the state-of-the-art, highest-accuracy build, but the one derived from business requirements. Both infrastructure and implementation costs stayed contained while the needed business value was delivered.
Operations did not end up depending on specialists
The people who run the system are the field team. By containing the parts that tend to become specialist-dependent and shaping the system so operations can sustain it, the platform keeps producing value after the build.
One dataset now feeds multiple kinds of value
The same GPS data serves both road matching and people-flow estimation. Collect the data once, and new processing can be added later as requirements emerge — the architecture itself remains as room to grow.
Working backwards from physical limits avoided a rebuild
Estimating the hard ceilings of storage I/O and network bandwidth before implementation avoided the most expensive kind of rework: rebuilding the architecture at load-testing time.
Technologies change generations.
The order of decisions repeats.
Technologies turn over in cycles of a few years. But the order itself — define the business impact first, then derive the architecture from it — is a method that remains reproducible even as the technology underneath is replaced.
Choosing high-performance technology and building an architecture that works for the business are not always the same thing.
What ENGORGIO provides is precision of impact and speed of realization: the precision to see where the effect will land, and the speed to turn it into something that actually runs. Only when both are present does data turn into business results.
Share your intent for the future.
The first conversation: find the starting point of transformation together. What you want to change. What hasn't changed yet. Start there.