Real-time decision system
Some decisions depended on a live lookup at processing time. When the lookup failed, the case fell to manual review.
Automation bots and an AI assistant now get that answer from a decision service built on data rebuilt every night, so a case no longer waits on a live lookup. The requests it cannot answer are counted every day, and nearly all trace to gaps in the model's coverage.
- Role
- End-to-end owner
- Context
- Decision automation
- Stack
- PythonFastAPIPydanticKubernetesBigQueryAirflowCloud Monitoringpytest
- decisions, served to automation bots and an AI assistant
- Real-time
- rebuild of the serving tables the service reads
- Nightly
- rollout of a breaking contract change, until every caller had moved
- Staged
- daily metrics, checked against source data
- Reconciled
This is a public summary. Figures and internal specifics from CVS Health/Aetna are left out; I am glad to go into more depth in conversation.
The system
- ServicesBuilt and owned by me
Decision API
A privacy-safe real-time service on Kubernetes. Built for correctness under load: async handling with bounded concurrency, input validation, time limits on upstream calls and health probes.
- Released behind automated tests and a load test
- A breaking contract change, staged until every caller had moved
- Sensitive data kept out of the logs
What I owned
- Nightly pipelines
- Serving tables
- Decision API
- Reference data product
- Lookup service
- Telemetry
- Daily metrics
- Findings
The whole system, from design to production support.
- Nightly pipelines that rebuild the serving tables behind the decisions.
- A privacy-safe decision API on Kubernetes, released behind automated tests and a load test.
- A lookup service, and the reference data product it reads, refreshed monthly behind a guarded promotion.
- The telemetry and monitoring that show whether any of it works.
Built for correctness under load
- Serving tables
- Decision API
The service handles requests asynchronously with bounded concurrency, validates every input, puts a time limit on its upstream calls and exposes health probes that check the data behind it. Logging is structured, with identifiers hashed, to keep sensitive data out of the logs.
Lookups are effective-dated, so an answer reflects the data in force on the date in question.
Rolling it out without breaking callers
- Decision API
- Automation bots
- AI assistant
A decision service is only useful if the people around it agree on what it does. I designed the API contract and ran the working sessions with business operations, Data Science and the automation team.
One change to the contract was breaking. I staged its enforcement across dev, test and prod and monitored adoption until every caller had switched, so nothing broke on the day it became mandatory.
Measuring whether it works
- Telemetry
- Daily metrics
- Findings
- Decision model
Every request writes a structured event. A daily aggregation turns those events into metrics on adoption, decision rate and latency, and it stays correct when a run is repeated. Before anyone relied on the dashboard, I recomputed it from source data and reconciled the two.
The monitoring also produced findings. Nearly all missed decisions traced to specific gaps in the model's coverage, which went back to Data Science. And a healthy overall latency was hiding a slow path, so I split latency by code path.