OpenProgram test system

This page defines test categories, directories, dependency boundaries, commands and CI responsibilities. Runtime dependencies determine test levels; product domains follow module names in openprogram/ . Structural migration does not change product behavior; genuine product defects found by tests are fixed and verified in separate batches.

1. Current implementation

There are currently 394 Git-tracked Python test files: 226 unit, 138 component, 14 contracts, 9 integration, 4 e2e and 3 live. Execution-level cases all reside in tests/<layer>/<product-domain>/, with no tracked test files directly at a level root.

Repository contracts enumerate only Git-tracked files. Structural contracts and autouse runtime guards jointly prohibit TestClient, subprocess, sockets, real threads/thread pools, fixed waits and global thread-class replacement in unit tests. CI separately runs contracts, unit on three Python versions, component, integration, non-browser e2e, built-Web browser, Web, CLI, Desktop and coverage.

The remaining structural gap concerns top-level repository constraints, not test levels: prevent new root development scripts, undeclared top-level directories and deleted source paths from returning to active guides.

2. Reference frameworks and responsibility boundaries

References come from official project documentation. The repository retains existing tools; this design adds no general-purpose test runner.

3. Target structure

tests/
  contracts/{repository,security}/
  unit/<product-domain>/
  component/<product-domain>/
  integration/<product-domain>/
  e2e/{agent,context,web}/
  live/{providers,channels}/
  support/
  conftest.py

When a test meets multiple levels, choose the one with the strongest dependencies. Directories have at most level/product-domain depth. Markers describe only execution capabilities: slow, browser, sandbox, live; do not repeat levels already expressed by directories.

4. Execution and state isolation

• Repository contracts read only Git-tracked files. Callers supply roots to generic scanners, excluding nested repositories and build directories.
• Top-level conftest.py retains only isolation required to prevent access to real user state; domain fixtures stay local.
• Temporary HOME has deterministic cleanup. Tests must not depend on real ~/.openprogram.
• Wait for concurrent results with Event, Condition or deadline-based conditional waits; fixed sleeps are prohibited in unit tests.
• Tests must not replace process-global standard-library thread classes or directly create real threads/thread pools. Background resources created inside production modules must be joined or closed during fixture teardown.
• Each production defect gets one regression test through the lowest public boundary.

5. Web, CLI and desktop

6. CI data flow

Python commands use uv run --locked. Matrix fail-fast is disabled so one version’s failure does not cancel other results. Ordinary PRs do not run live tests. All CI commands also appear in contribution documentation.

7. Adopted, adjusted and rejected

8. Acceptance criteria

• Clean checkouts and checkouts containing ignored/nested-repository files collect identical tracked tests and produce identical contract results.
• Unit test code does not directly use TestClient, subprocesses, sockets, real background threads/thread pools or fixed sleeps; internal production threads are reclaimed at teardown.
• Required Python suites emit no unhandled-thread warnings; three consecutive runs with fixed xdist workers have no worker crashes.
• Separately verify component behavior without frontend artifacts and e2e behavior with built CSP.
• CI runs all CLI, Web and desktop package scripts.
• Explicitly record test-count, skip and xfail changes across migration.
• Required Python suites have zero failures; known product defects cannot remain in a completed state.
• Establish reproducible coverage first, then prevent regression with a threshold no higher than the measured baseline; raising it requires additional meaningful tests.
• Unit structural contracts detect direct/aliased resource imports, direct sleep imports and replacement of standard-library thread classes. Autouse runtime guards intercept real-thread creation, thread-pool construction and fixed waits directly in test code. Unit tests permit only module-qualified asyncio.sleep(0), with positional or keyword forms allowed as scheduling yields.
• Audit npm dependency risks per workspace; use only upgrades compatible with current builds/tests, without forced breaking upgrades.

Merge-turn tests own their temporary SessionStore. The fixture registers close immediately after construction, so successful tests, assertion failures, and setup failures drain pending index writes and join store timers before the unit resource guard runs. The guard remains strict; cleanup does not depend on machine speed or a longer timeout.

History-page objects use the local name historyWindow, keeping source checks for browser window access unambiguous. Local-refresh test repositories include the real ASAR permission-restoration helper so version-change rejection is exercised after staging, without weakening the no-installation assertions.

9. Implementation status

10. Improvement plan and implementation records

Complete batches sequentially. Each first reproduces failure through a public/shared boundary, fixes the minimum shared cause, runs focused/affected tests, passes independent specification and quality review, then commits. Do not start the next batch before the prior review passes.