Accuracy and scale —
one foundation clears both.
The customer challenge: high-volume GPS data from vehicles carries built-in error, and isn't usable for business decisions as it is. The design principle: work backwards from business impact.
How do we draw meaning
out of the data?
Vehicles were sending in location data, and an enormous volume of it accumulated every day. But the raw data was not in a state that could support business decisions.
GPS coordinates carry error. Try to determine mechanically which road a vehicle is on right now, and you mistake a highway for the surface road running alongside it, or the travel path jumps around near intersections. The data existed — but nothing certain could be said from it.
At the same time, the volume kept growing with the number of vehicles. Push for accuracy and the processing gets heavier and infrastructure costs swell. Push for real-time and the architecture gets more complex and operations depend on a few specialists. Accuracy, speed, cost, and operability all pulled against each other, and there was no settled answer for where to land. That was the question at the start of the project.
Decide from business impact,
not from technology.
This domain has no single correct answer. The infrastructure works on-premises or in the cloud. There are multiple stream-processing engines to choose from. For the road-matching inference itself, the options range from geometric methods to probabilistic models to machine-learning approaches. Any of them is technically viable.
That is exactly why we did not start from comparing technologies. We defined first how much impact the system would have on the business, and worked backwards from there to the architecture.
Concretely, we put these questions before any technology selection:
- Which business decisions will this output feed into?
- What level of accuracy and freshness do those decisions actually need?
- Is accuracy beyond that level worth additional investment?
- After the build, who keeps operating this?
By the time these questions were answered, the outline of the required architecture was largely decided.
Low-latency work and large-scale analytics
do not belong on the same engine.
Sorting out the business impact made one thing clear: the required processing characteristics were not one thing.
Determining which road each vehicle is on, continuously and statefully per vehicle, is work where real-time performance translates directly into business value. That went onto a stream-processing engine that manages state per vehicle: Flink.
Estimating population-level people flow from mobile-phone GPS data is analytical work — extrapolating from a sample to a population. What it needs is not freshness but the ability to digest data at scale. That went onto Spark Streaming.
Fluentd handles collection; Kafka handles messaging. Instead of loading everything onto one engine, the architecture places the right technology against each kind of business value.
Not the highest accuracy —
the least complexity that meets the requirement.
Higher-accuracy inference methods existed. But if one extra step of accuracy sends architectural complexity soaring, makes operations depend on specialists, and inflates infrastructure cost, the impact never repays the spend. Putting “the least complexity that satisfies the business requirement” at the center of the evaluation criteria was the technical core of this engagement.
The physical limits that decide
high-volume streaming.
In high-volume streaming, physical-layer bottlenecks cap performance before algorithms or engine choice come into play. Hold per-vehicle state on disk and its read speed becomes the ceiling; relay high-volume data and the network bandwidth spent on replication and moving data between nodes becomes it. So in this engagement we estimated, before implementation, which point in the path — collection, messaging, processing, sink — was narrowest, and cut unneeded data upstream to shrink the volume before fixing the architecture. Skip that and you rebuild the architecture itself at load-testing time.
What changed.
Raw data became information that supports decisions
What had been a list of coordinates became something the business can use directly: which vehicle is on which road. The value is not that the data exists — it is that decisions can be made from it.
Choosing by cost-effectiveness kept the investment proportionate
Not the state-of-the-art, highest-accuracy build, but the one derived from business requirements. Both infrastructure and implementation costs stayed contained while the needed business value was delivered.
Operations did not end up depending on specialists
The people who run the system are the field team. By containing the parts that tend to become specialist-dependent and shaping the system so operations can sustain it, the platform keeps producing value after the build.
Multiple data sources, handled on one platform
Vehicle data and mobile-phone GPS are taken in on one platform and serve two separate uses: road matching and people-flow estimation. As sources and uses grow, new processing can be added later as requirements emerge — the architecture itself remains as room to grow.
Working backwards from physical limits avoided a rebuild
Estimating the hard ceilings of storage I/O and network bandwidth before implementation avoided the most expensive kind of rework: rebuilding the architecture at load-testing time.
Technologies change generations.
The order of decisions repeats.
Technologies turn over in cycles of a few years. But the order itself — define the business impact first, then derive the architecture from it — is a method that remains reproducible even as the technology underneath is replaced.
Choosing high-performance technology and building an architecture that works for the business are not always the same thing.
What ENGORGIO provides is precision of impact and speed of realization: the precision to see where the effect will land, and the speed to turn it into something that actually runs. Only when both are present does data turn into business results.
End the silos.
Rebuild as one platform.
The customer challenge: with no shared foundation, platforms kept being built project by project, and data fractured across departments. The design principle: start small from the point of highest business impact and grow it into the standard.
The platforms were there.
The company couldn't use them.
Each business unit had its own data platform. Every time a project started, a platform was built in whatever shape suited that project best.
Each platform worked well enough for its own purpose. The problem was that no company-wide architecture connected them. Design philosophy differed by project; so did how permissions were held and where data lived. Inside the same company, the moment you tried to use data across departments, everything stalled. That was the state that had built up.
The silos did not form because departments were uncooperative — nor because the work started small. The root cause was accumulating project-by-project optimization with no shared foundation and no intent to converge, later, into one company-wide platform.
Not a company-wide standard up front —
start small and grow it.
Try to define the perfect company-wide standard up front and you run out of time before anything moves; impose a standard that does not fit the real work and no one uses it. So we treated the company-wide standard not as something to lay down first, but as something to grow into.
How we start is worked backwards from business impact. Which business, and how much value? We start small from the single point where it matters most. But not a throwaway one-off: we start small on a foundation deliberately built to grow into the company-wide standard. That is the dividing line from the silos of before.
Concretely, we put these questions before any technology selection:
- Which business decisions, in which units, will this platform support?
- Where do the moments of using data across departments arise?
- Where is the boundary between what must be uniform company-wide and what can be left to departments?
- In what order do we grow the small start out to the whole company?
By the time these were answered, the first point to build and the path to grow it company-wide were decided. What starts small becomes, once proven, the core of the company-wide standard.
Let business units
use it themselves.
Once the small start was proven and the core of the standard existed, the next stage opened the platform up to the business units. Instead of the center building and handing out forever, each department gets a state it can use on its own. That is self-service.
But opening up brings new walls. In this engagement, we cleared two in particular.
Provisioning becomes person-dependent again
Open the platform, but if provisioning new environments and permissions still runs through requests to one specific person, that person becomes the new bottleneck. The project-by-project dependency you thought you had cleared reappears in a different place: provisioning. We standardized the flow from request to provisioning, so departments can start using the platform without waiting.
Skill gaps among users
Business-unit users are not all data specialists. Hand the same platform to everyone and a gap opens between the departments that can wield it and those that cannot. We provided templates and a minimum of guardrails, so the standard way of working is reachable without deep expertise.
Clear the walls, let usage spread, and something else happens. The more people use it, the more usage patterns appear that stray from the original design intent. Not out of bad intent — each department optimizes for its own convenience. That is where the next governance became necessary.
What the company enforces,
and what departments decide.
Bind everything uniformly company-wide to stop the drift, and you lose the speed self-service bought. Leave it all to each department, and you slide back into silos. So we split the rules into two layers.
Company-wide rules are set centrally, by the strategy subsidiary. The minimum that must hold across departments is decided in one place. Business-unit rules are set by each department itself, because how to use data inside a unit is something that unit knows best.
Catalog structure / naming / environment separation / audit & access control
Schema design / in-unit grants / data quality / usage rules
Company-wide catalog structure and permission model
The company-layer rules were implemented as Unity Catalog structure. Catalogs are separated per department, with naming and environment separation unified company-wide. Audit and access control are enforced centrally, in one place.
Management rights delegated to departments
The business-unit layer was implemented by delegating management rights to each department. Inside its own catalog, a department handles its own schema design and grants. The company-wide frame holds, while freedom inside the unit remains. We rendered the boundary between the two layers as permission design.
What changed.
A small start grew into the company-wide standard
The platform that started from the point of highest business impact became, once proven, the core of the company-wide standard. Using data across departments is no longer a special task.
Opening up without making the center a bottleneck
Self-service let each department use the platform at its own speed. Clearing the person-dependency in provisioning avoided the pattern where the central team becomes the counter for every department.
Speed and control, together
Two-layer governance kept the company-wide minimum while leaving freedom inside each unit. Uniform control never blunted the speed of self-service, and the slide back into silos was prevented.
Governance that stays as a mechanism
Rules were implemented as Unity Catalog structure and a permission model, not left to human operation, so the same standard keeps holding after the build.
Two-layer governance,
running from day one.
LakeShepherd is what turned the two-layer governance designed in this engagement into a model any organization can reproduce, on Unity Catalog.
It implements organization, permissions, environment separation, and audit as one reproducible operating model. The company-layer catalog structure and permission model are ready from day one, and delegation to departments happens within that frame. Governance is not added later — it runs the moment the platform launches. That is LakeShepherd’s role.
Share your intent for the future.
The first conversation: find the starting point of transformation together. What you want to change. What hasn't changed yet. Start there.