Draft

ADR-0003: Central Observability Service — Query-in-Place, Ingestion as Opt-in

Status: TBDDate: 2026-07-20Deciders: Propel CloudOpsPattern: observability-control-plane
Why this matters

one pane monitors every customer and our own platform — uptime, metrics, logs — for ~$130–370/mo at 10 customers, with customer telemetry staying in their accounts by default and log ingestion available as a per-customer opt-in, not an architecture change.

Context

We operate infrastructure for multiple customers plus our own platform. Each estate needs uptime monitoring, dashboards, alerting, and log access — but customers differ on one axis: some are fine with us hosting their logs; others require logs never leave their account. One central service must serve both without forking the design.

flowchart LR
    subgraph P["Propel control plane"]
        G["Grafana ·
workspace per customer"]
        UK["Uptime Kuma
status pages"]
        ING[("opt-in log store
(per customer)")]
    end
    subgraph C1["Customer A (default)"]
        T1[(telemetry stays here)]
    end
    subgraph C2["Customer B (opted in)"]
        T2[(telemetry)]
    end
    G -.query-in-place.-> T1
    T2 -->|ships logs| ING --> G
    style G fill:#EDEDFF,stroke:#3B54F7
    style ING fill:#F3EDFF,stroke:#8A54E8

Decision TBD

Suggested decision — not yet agreed, expand to review

Build one central control plane with two log modes selected per customer:

  1. Pane: Amazon Managed Grafana, one workspace per customer (hard isolation, AWS-managed SSO, seats billable to the customer) + Uptime Kuma for uptime/status pages + our own platform's workspace.
  2. Access: cross-account read-only roles (ExternalId, no static keys), provisioned via each customer's landing zone; OTel-first instrumentation so backends stay swappable.
  3. Logs — Mode A (default): query-in-place. Logs stay in the customer account (their retention, their KMS); Grafana runs Logs Insights live. Nothing stored on our side.
  4. Logs — Mode B (opt-in): ingestion. Customer ships logs to a per-customer store on our side (OpenObserve on S3 — cheap, isolated per customer) when they want richer search/longer retention and consent to us hosting. A frontmatter-level toggle per customer, not a redesign.

Mode is a per-customer contract choice (maps to the log-privacy tiers in the Observability guide); both modes feed the same pane, alerts, and status pages.

Consequences

Positive — one service for platform + all customers; residency-strict customers and convenience-first customers coexist; seat + ingestion costs are per-customer pass-throughs; zero servers for the pane (AMG). Negative — two log paths to document and test; AMG staff seats multiply per workspace (keep staff oversight central); Mode B adds an OpenObserve instance per opted-in customer (~S3 pennies + small compute, but it's ops surface). Risks — mode creep (customers flip-flopping) → make the mode a contract term reviewed annually, not a dial.

Alternatives considered (10 customers, with cost)

Option $/mo Why not
This design (AMG + query-in-place, opt-in ingest) ~$130–370 net of billable seats — chosen
Self-hosted Grafana OSS (HA ECS + RDS + ingress) ~$425–845 + maintenance Cheaper only past ~20 customers; we build SSO/HA/patching
Full stack per customer (Grafana each) ~$150–300 × N No single pane; N× ops
Ingest-everything central store ~$45+ + per-GB, grows with volume Breaks residency for strict customers — the exact constraint that shaped this
Datadog / SaaS APM ~$15–31/host/mo → $1k+ Cost scales with hosts; customer data on a third party by default

Open questions

  • Mode B store: OpenObserve vs CloudWatch cross-account destinations — decide at first opt-in (follow-up ADR-0003.1).
  • Alert routing: per-customer Slack channels vs PagerDuty services — decide at first managed SLA.

Discussion

#cloudops. Graduates the pattern to golden-path when the first customer deployment proves workspace-per-customer end to end.