Observability & Logging

How Propel monitors infrastructure — uptime, metrics, logs, traces — and the data-privacy guarantees that make it safe across many client accounts. Serves two audiences: internal (how we build and cost it) and client-facing (where your data lives and how it's protected).

How it works

Separate the control plane from the data plane. Your telemetry stays in your account (native services or cheap OSS on S3); Propel runs only a thin pane that queries in place and stores nothing. That keeps our cost flat as clients are added and means client data never has to live with us.

flowchart LR
    subgraph C["Your AWS account — data + cost live here"]
        APP[Your apps] --> OTEL[OpenTelemetry
Collector]
        OTEL --> MET[Metrics
CloudWatch / AMP]
        OTEL --> LOG[Logs
CloudWatch Logs]
        OTEL --> TRC[Traces
X-Ray / Tempo]
    end
    subgraph P["Propel control plane — stores nothing"]
        GRAF[Grafana
one org / workspace per client]
        UPTIME[Uptime Kuma
+ status pages]
    end
    MET -.query in place.-> GRAF
    LOG -.query in place.-> GRAF
    TRC -.query in place.-> GRAF
    GRAF --> ALERT[Alerts → Slack / PagerDuty]
    UPTIME --> ALERT

• Instrument on OpenTelemetry, not a vendor SDK — so the backend can change (CloudWatch today, self-hosted or a client's Datadog tomorrow) without touching app code.
• "Query-in-place, stores nothing" is real. Grafana assumes a cross-account read role and calls CloudWatch APIs live (GetMetricData; Logs Insights). A "last 30 days" panel queries the client account on demand — the window is bounded by the client's log retention. Grafana's DB holds only dashboards/config/alert rules, never the data. Caveats: keep query caching off; alert history may hold small value snippets; logs must live in CloudWatch (S3-only → query via Athena); Logs Insights is slower/pricier at volume (see OpenObserve below).

What we collect

No central log store by default — centralizing telemetry is where residency breaks and ingestion + storage costs appear. OpenObserve enters only if CloudWatch Logs gets too slow/expensive: it's a cheap, S3-backed log/trace store you run in the client's account and point Grafana at — a backend under the same pane, not a second UI.

Platform & cost

The pane (dashboards/alerting) is the only thing Propel pays for; telemetry storage is client-side in every option.

Why AMG's realistic total beats self-hosted: a client-facing self-hosted pane isn't a $15 box — it needs HA (2–3 Fargate tasks), a real RDS Postgres, a load balancer, and private networking (NAT/VPC endpoints) ≈ $175–295/mo infra, plus a maintenance tax (~$200–500/mo) and one-time build of ingress + SSO. AMG has none of that.

TCO at 10 customers × 3 viewers + Propel staff (query costs ~$50/mo either way; maintenance at $100/loaded hr):

✅ Recommendation: Amazon Managed Grafana, one workspace per customer. Cheaper and far less work at our scale; AWS handles HA, private login, and patching; the customer seat is billable. Revisit self-hosted only past ~20 customers or for multi-cloud beyond AMG's data sources. Keep staff oversight in a central cross-account view (avoids the per-workspace staff-seat multiplier); run Uptime Kuma for status pages regardless (free, no seats).

Customer login & data isolation

Customers can log in, see only their own data, and the seat is billable — this is the clean answer to private access + multi-tenancy:

• Private SSO, built in. AMG authenticates via IAM Identity Center / SAML — AWS hosts the MFA-capable login. No Cloudflare/ALB/VPN to build.
• Hard isolation: one workspace per customer — each is a separate Grafana with only that customer's users and a data source scoped to only their account. No path to another tenant's data.
• Billable seat: treat the $5/viewer as a pass-through and mark it up in the customer's fee.
• Staff-seat gotcha: a user active in N workspaces is billed N times → keep staff oversight central, use per-customer workspaces mainly for customer self-service.
• Hooking into accounts: a read-only cross-account IAM role per client (sts:AssumeRole + unique ExternalId, no static keys), provisioned via the client's landing zone, separate from the deploy role.

(Self-hosted alternative if you outgrow AMG: Grafana one-org-per-client behind Cloudflare Access / oauth2-proxy, OIDC role-mapping to scope each client to their org — but you build and run that ingress yourself.)

Log privacy & PII

Effective operations needs logs — but that never requires hosting your logs with us. Retention lives where the logs are; access is federated and controlled.

Pick a log-privacy tier:

PII scrubbing — layered, earliest-possible: ① app discipline (never log PII); ② scrub at the OTel Collector (redaction/transform masks/hashes before storage — free, primary control); ③ CloudWatch Logs data-protection masks PII at ingestion, unmask needs an IAM permission our read role lacks; ④ logs encrypted with your KMS key, deep dives are break-glass.

The guarantee: Your telemetry stays in your account under your KMS key; we log in through your MFA-gated SSO, reach your data only via a scoped role you granted with PII already scrubbed, and you can see and revoke everything we do.

Underpins the monitoring in every engagement scenario. Internal delivery standards: Engineering Operating Model.