Project — ChaosLab

01 Problem Statement

Modern distributed applications are built to survive failure: pods restart, requests retry, circuit breakers trip and reset. Chaos-engineering tools like Chaos Mesh are excellent at injecting that failure and confirming a system becomes live again — services respond, health checks pass, pods run.

What they do not check is whether the system's business state is still correct after that recovery. A payment can be double-charged. An order can be confirmed without inventory ever being deducted. Two services can each report success while disagreeing about what actually happened. None of this shows up in a liveness probe.

ChaosLab's problem, precisely stated: given a distributed application, a controlled failure, and its subsequent recovery, was the resulting business state correct — and can that be proven with evidence, not assumed from green health checks?

02 Objectives

Design a declarative language/format for expressing recovery-correctness oracles (business invariants) independent of any single application.
Build an Experiment Manager that drives controlled fault injection via Chaos Mesh against a target distributed application.
Build an Evidence Collector that gathers logs, database state, and OpenTelemetry traces after each experiment.
Build a Correlation Engine that reconstructs a coherent timeline of what happened across services from that evidence.
Build a Verdict Generator that evaluates collected evidence against declarative oracles and produces a PASS/FAIL recovery-correctness verdict.
Validate the engine using ground-truth mutation testing — deliberately broken recovery paths that oracles must catch.

03 Scope

In Scope

Recovery-correctness verification for a target distributed application (Online Boutique)
Declarative oracle definitions for business invariants
Evidence correlation across logs, DB state, and traces
Chaos Mesh–driven controlled fault experiments
A project website with presentations, documentation, and (later) an admin publishing workflow

Out of Scope

Building a new chaos-engineering tool (Chaos Mesh is used as-is)
General-purpose APM or observability tooling
Automatic remediation or self-healing of detected failures
Supporting arbitrary applications without an oracle definition

04 Intended Users

Platform / SRE Teams

Teams that already run chaos experiments and need to know if recovery preserved correct business state, not just uptime.

Backend Engineers

Engineers who own services with cross-service invariants (payments, inventory, order state) and want regression coverage for recovery paths.

QA / Release Engineers

Teams validating a release candidate under failure conditions before it reaches production.

05 Proposed Functions

Function	Description
Define Oracle	Author a declarative recovery-correctness oracle describing expected business invariants.
Run Experiment	Trigger a controlled fault (via Chaos Mesh) against the target application through the Experiment Manager.
Collect Evidence	Gather logs, database snapshots, and OpenTelemetry traces during and after recovery.
Correlate Evidence	Merge evidence into a single cross-service timeline of what actually happened.
Evaluate Oracle	Check the correlated evidence against the declarative oracle.
Generate Verdict	Produce a PASS/FAIL recovery-correctness report backed by the supporting evidence.
Publish Artifacts	(Website) Publish presentations and project documentation with version history.

06 Key Features

Declarative recovery oracles, reusable across experiments
Automatic evidence correlation across heterogeneous sources
Business-invariant verification, not just service-liveness checks
Reproducible, controlled fault experiments
Ground-truth mutation testing to validate oracle sensitivity
Auditable recovery-correctness reports

07 External Interfaces

System	Interface
Chaos Mesh	Kubernetes CRDs / API to schedule and trigger fault experiments.
Online Boutique	Target distributed application under test (microservices demo app).
Database	Direct queries to verify persisted business state after recovery.
Logs	Structured application/service logs collected post-experiment.
OpenTelemetry	Distributed traces used to reconstruct cross-service request flow.

08 Technical Challenges

Designing an oracle language expressive enough for real business invariants, but simple enough to author declaratively.
Correlating asynchronous, distributed evidence (logs, traces, DB state) into one consistent timeline despite clock skew and delivery ordering.
Distinguishing "recovery still in progress" from "recovery finished incorrectly" without flaky false positives.
Building mutation scenarios that are genuinely representative of real recovery bugs, to trust the oracle's PASS verdicts.
Keeping experiments reproducible against a live Kubernetes cluster running Chaos Mesh.

Important framing: fault injection (via Chaos Mesh) is supporting infrastructure — it produces the failure scenario. ChaosLab's actual contribution, per our professor's guidance, is verifying recovery correctness after that failure: the oracle definition, evidence correlation, and verdict generation. ChaosLab should not be described as merely a fault-injection platform.