Draft

scenarios — Holdout Scenario Store

Internalactive

Isolated test scenario repository for Propel's AI-Native Software Factory. Coding agents never have read access here — only the orchestration/validation layer reads scenarios; agents receive only pass/fail messages. Prevents agents from gaming or overfitting to test definitions.

flowchart LR
    AUTHOR[Human authors
scenario YAML] --> STORE[propel.scenarios
(isolated repo)]
    STORE --> ORCH[Orchestration layer
validation runner]
    ORCH -->|pass/fail only| AGENT[Coding agents]
    AGENT -->|code output| ORCH
    style STORE fill:#EDEDFF,stroke:#3B54F7
    style AGENT fill:#E4E4FF,stroke:#23233A

Scenario taxonomy (6 types):

  1. user-journeys — functional flows end-to-end
  2. edge-cases — failure and error scenarios
  3. performance — constraint / non-functional
  4. security — auth, authz, data safety
  5. ai-judged — subjective quality (AI-as-judge with calibration examples)
  6. architecture — fitness functions & structural integrity

Structure:

customers/<customer>/<product>/user-journeys/
                               edge-cases/
                               ai-judged/
                               calibration/   ← known-good/bad for judge calibration
                               adversarial/   ← red-team generated
shared/<standard>/             ← reusable conformance sets (release-process, etc.)
  • Current customers: Novamera
  • Shared standards: Release Process (branch protection, CI/CD, Husky hooks, Slack notifications, semantic versioning)
  • Authoring rules: see CONVENTIONS.md in repo; architecture in ARCHITECTURE.md.