Architecture

DASH is a Rust workspace organized into library crates and four service binaries. Ingested claims are durably written to a write-ahead log, replayed into an in-memory Claim + Evidence + Edge store, and indexed for HNSW ANN candidate generation.

Workspace layout

Library crates (pkg/):

• pkg/schema - data model types
• pkg/store - in-memory store, durable WAL with replay, checkpoints, compaction, redb persistence
• pkg/ranking - retrieval planner and scoring
• pkg/graph - graph-backed recall layer
• pkg/auth - JWT + scoped API keys
• pkg/embeddings - embedding providers (HashEmbeddingProvider default, Ollama, OpenAI)

Service binaries (services/):

• services/ingestion - binds 127.0.0.1:8081, accepts claims
• services/retrieval - binds 127.0.0.1:8080, serves retrieval and embeddings
• services/indexer - builds and maintains indices
• services/control-plane - fleet coordination

The retrieval path

1. The planner takes the query (string or precomputed query_embedding).
2. The built-in multi-level HNSW graph generates vector candidates, with DASH_*_ANN_* tuning knobs.
3. Metadata filters, time-range filters, and stance demotion/filtering narrow the set.
4. Optional graph expansion follows claim edges.
5. Results are projected into citation-bearing responses.

Semantic-first retrieval (InMemoryStore::retrieve_semantic) treats the dense-similarity score (cosine in [-1, 1], mapped to [0, 1]) as the primary ranking signal when the caller passes a pre-computed query vector; the lexical score becomes a small tie-breaker.

Persistence

• Write-ahead log - durable WAL with replay, checkpoints, and compaction in pkg/store. WAL durability guardrails reject unsafe flush policies by default; stress testing requires the explicit DASH_INGEST_ALLOW_UNSAFE_WAL_DURABILITY=true override.
• redb persistence - additive on-disk durability for claims, evidence, edges, vectors, and the tenant-to-claim set. Enabled via DASH_INGEST_PERSISTENCE_PATH / DASH_RETRIEVAL_PERSISTENCE_PATH. Unset (the default), DASH runs in WAL-only mode.
• Embedding providers - env-driven selection with DASH_EMBEDDING_PROVIDER=hash|ollama|openai.

Auth and multi-tenancy

• JWT auth (HS256) with key rotation by kid, optional iss/aud checks, a fallback secrets list, and per-tenant claim enforcement.
• Scoped API keys plus revocation (DASH_*_SCOPED_API_KEYS, DASH_*_REVOKED_API_KEYS).
• Multi-tenant allowlist enforced in the authz layer.
• Per-tenant rate limits (DASH_*_RATE_LIMIT_*).

Audit log

Every authenticated state change is recorded as a SHA-256-chained JSON line. Verify the chain with scripts/verify_audit_chain.sh.

Observability

• Health endpoints at /health on both services.
• Prometheus metrics at /metrics.

Operational tooling

The scripts/ directory ships backup, restore, recovery drill, failover drill, an SLO guard, a release-candidate gate, and the audit chain verifier.

Testing and quality gates

• 415 Rust unit/integration tests passing (verified with cargo test --workspace --all-features), plus Go, TypeScript, and Python SDK suites across the workspace.
• cargo clippy --workspace --all-targets and cargo build --workspace are clean.
• cargo-fuzz harnesses cover the JWT verifier, OpenAI request parser, ranking function, and WAL record parser.
• A performance benchmark suite (perf_bench) covers ingest, retrieve-lexical, retrieve-semantic, ANN-at-scale, and WAL-replay, with smoke, hybrid, and large profiles and a CI-enforced regression guard against prior scorecards.

SDKs

SDK	Package	Notes
Python	dash-py	sync + async, typed dataclasses, OpenAI drop-in examples
Go	dash-go	Go 1.21+, functional options, errors.Is/As support
TypeScript	dash-ts	ESM-first, Node 18+, zero runtime deps, full type safety
Java, Kotlin, C#	in sdks/	see the repository

All SDKs are OpenAI drop-in compatible.

Deployment

Docker + docker-compose (multi-stage, multi-arch linux/amd64 + linux/arm64, non-root runtime, healthcheck, dependency ordering, dev overlay with hot-reload). docker compose up -d brings the ingestion + retrieval services online.