Troubleshooting

Services won't start

# Confirm both health endpoints
curl http://localhost:8080/health
curl http://localhost:8081/health

If either fails: check Docker (docker compose ps), then the service logs. Without DASH_INGEST_WAL_PATH / DASH_RETRIEVAL_WAL_PATH the services run in-memory and seed a sample claim - useful for smoke-testing config vs data issues.

Retrieval returns nothing

1. Wrong service or port? Retrieval is :8080, ingestion is :8081. Ingesting to the retrieval port (or vice versa) silently misses.
2. stance_mode: support_only filtering everything? Any claim with a contradicting evidence record is dropped. Re-run with balanced (default) to see demoted results and confirm the claim exists.
3. Temporal window too narrow? A time_range outside every claim's [valid_from, valid_to] returns empty. Widen the window first, then narrow.
4. Tenant mismatch? Queries are tenant-scoped. Ingesting under t1 and querying under t2 returns nothing by design.

Scores look wrong

• Contradicted claims rank low on purpose. supports: 1, contradicts: 1 demotes the score in balanced mode - that is the differentiator working, not a bug.
• Semantic vs lexical. Passing a precomputed query_embedding makes dense similarity the primary signal; a raw string leans lexical with semantic as tie-break. Mismatched expectations here look like "wrong" ranking.
• Source quality weights. A supports from a 0.4-quality source counts less than a contradicts from a 0.95 source. Check the citations, not just the score.

Ingest rejected (400)

Server-side validation fails the whole write: confidence and source_quality must be in [0.0, 1.0], valid_from <= valid_to, and all IDs non-empty. The error names the offending field.

Embeddings endpoint errors

• Default provider is deterministic hash (no network, no key) - it always works offline.
• DASH_EMBEDDING_PROVIDER=ollama|openai failures are provider-connectivity issues, not DASH issues: verify the provider is reachable with its own CLI first.
• Any OpenAI client works via OPENAI_API_BASE=http://localhost:8080/v1 - if one client fails and curl succeeds, check the client's base-URL handling (trailing slashes, /v1 duplication).

WAL and durability

• WAL-only mode (no persistence paths set) loses in-memory state on restart - by design. Set DASH_INGEST_WAL_PATH / DASH_RETRIEVAL_WAL_PATH for replay across restarts.
• Never set DASH_INGEST_ALLOW_UNSAFE_WAL_DURABILITY=true outside stress testing.
• Verify audit integrity any time: scripts/verify_audit_chain.sh.

Auth failures (401/403)

• Services need a shared key (DASH_INGEST_API_KEY / DASH_RETRIEVAL_API_KEY) when auth is on.
• JWT kid must match a configured key; check iss/aud if enforced.
• Revoked keys fail closed - check DASH_*_REVOKED_API_KEYS.

Still stuck?

• GET /metrics exposes service metrics - compare throughput against the perf_bench CI scorecards (ingest_throughput_sequential, retrieve_throughput_lexical, retrieve_throughput_semantic, ann_search_throughput_at_scale, wal_replay_throughput).
• The fuzz/ harnesses cover the JWT verifier, OpenAI parser, ranking, and WAL parser - crash inputs there are known-fixed; update before reporting.