Declarative Recovery-Correctness Analysis for Distributed Microservices
Planning Presentation v1
ChaosLab is a declarative recovery-correctness analysis platform for distributed microservices. It injects controlled failure into a running application and verifies whether the system's business state — not just its liveness — is correct after recovery.
Distributed systems are engineered to survive failure, but "recovered" and "correct" are not the same thing. Duplicate payments, missed inventory deductions, and contradictory service state can all occur silently behind green health checks.
This deck covers our approved project scope, proposed functions, architecture, non-functional requirements, and initial planning for the semester.
Note: fault injection (Chaos Mesh) is supporting infrastructure. ChaosLab's actual contribution is verifying recovery correctness after that failure.
| Function | Description |
|---|---|
| Define Oracle | Author a declarative recovery-correctness oracle for a business invariant. |
| Run Experiment | Trigger a controlled fault via Chaos Mesh through the Experiment Manager. |
| Collect Evidence | Gather logs, DB snapshots, and OpenTelemetry traces after recovery. |
| Correlate Evidence | Merge evidence into one cross-service timeline of what happened. |
| Evaluate Oracle | Check correlated evidence against the declarative oracle. |
| Generate Verdict | Produce an auditable PASS/FAIL recovery-correctness report. |
Already run chaos experiments; need to know if recovery preserved correct business state, not just uptime.
Own services with cross-service invariants — payments, inventory, order state — and want recovery-path regression coverage.
Validate a release candidate under failure conditions before it reaches production.
The ChaosLab website: project pages, presentation archive, and an admin dashboard for publishing (UI built, backend pending).
Kubernetes cluster running Chaos Mesh; Online Boutique deployed as the target application under test.
Database queries for persisted state, log ingestion, and OpenTelemetry traces for cross-service request flow.
Experiments are reproducible under the same fault configuration; verdict generation is deterministic given the same evidence.
Oracles are readable without deep ChaosLab internals knowledge; verdicts clearly explain why a result was PASS or FAIL.
Oracle Engine, Evidence Collector, and Experiment Manager stay independent, testable modules with a versioned oracle schema.
| Task | Semester Timeline (12 weeks) | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Planning & Feasibility | ||||||||||||
| Oracle Engine Design | ||||||||||||
| Experiment Manager | ||||||||||||
| Evidence Collector | ||||||||||||
| Correlation Engine | ||||||||||||
| Verdict Engine | ||||||||||||
| Mutation Testing | ||||||||||||
| Integration & Demo | ||||||||||||
Online Boutique, Chaos Mesh, and OpenTelemetry are all open-source and self-hostable on a single Kubernetes cluster — validated as technically feasible within the semester timeline.
ChaosLab verifies recovery correctness, not just recovery liveness, for distributed microservices — using declarative oracles, evidence correlation, and mutation-tested verdicts.
Begin Oracle Engine design and Experiment Manager integration with Chaos Mesh, per the timeline in Initial Planning.
We welcome questions and feedback on the proposed scope and architecture.