diff --git a/docs/help/testing.md b/docs/help/testing.md index 14322039ed4..f1d926fe37c 100644 --- a/docs/help/testing.md +++ b/docs/help/testing.md @@ -58,24 +58,6 @@ Think of the suites as “increasing realism” (and increasing flakiness/cost): - Projects note: - `pnpm test`, `pnpm test:projects`, and `pnpm test:watch` all use the same native Vitest `projects` config now. - The wrapper CLI shape is preserved, so `pnpm test -- src/foo.test.ts -t bar` still works. -- Planner note: - - Planner-backed lanes are explicit now: `pnpm test:planner`, `pnpm test:max`, `pnpm test:serial`, `pnpm test:bundled`, `pnpm test:extensions`, and `pnpm test:channels`. - - CI shards, pre-push mirrors, and other lane-tuned flows keep using the planner-backed scripts. - - The planner still keeps a small checked-in behavioral manifest for true pool/isolation overrides and a separate timing snapshot for the slowest unit files. - - Extension-only local runs now also use a checked-in extensions timing snapshot plus a slightly coarser shared batch target on high-memory hosts, so the shared extensions lane avoids spawning an extra batch when two measured shared runs are enough. - - High-memory local extension shared batches also run with a slightly higher worker cap than before, which shortened the two remaining shared extension batches without changing the isolated extension lanes. - - High-memory local channel runs now reuse the checked-in channel timing snapshot to split the shared channels lane into a few measured batches instead of one long shared worker. - - High-memory local channel shared batches also run with a slightly lower worker cap than shared unit batches, which helped targeted channel reruns avoid CPU oversubscription once isolated channel lanes are already in flight. - - Targeted local channel reruns now start splitting shared channel work a bit earlier, which keeps medium-sized targeted reruns from leaving one oversized shared channel batch on the critical path. - - Targeted local unit reruns also split medium-sized shared unit selections into measured batches, which helps large focused reruns overlap instead of waiting behind one long shared unit lane. - - High-memory local multi-surface runs also use slightly coarser shared `unit-fast` batches so the mixed planner spends less time spinning up extra shared unit workers before the later surfaces can overlap. - - Shared unit, extension, channel, and gateway runs all stay on Vitest `forks`. - - The wrapper keeps measured fork-isolated exceptions and heavy singleton lanes explicit in `test/fixtures/test-parallel.behavior.json`. - - The wrapper peels the heaviest measured files into dedicated lanes instead of relying on a growing hand-maintained exclusion list. - - CLI startup benchmarking now has distinct saved outputs: `pnpm test:startup:bench:smoke` writes the targeted smoke artifact at `.artifacts/cli-startup-bench-smoke.json`, `pnpm test:startup:bench:save` writes the full-suite artifact at `.artifacts/cli-startup-bench-all.json` with `runs=5` and `warmup=1`, and `pnpm test:startup:bench:update` refreshes the checked-in fixture at `test/fixtures/cli-startup-bench.json` with `runs=5` and `warmup=1`. - - For surface-only local runs, unit, extension, and channel shared lanes can overlap their isolated hotspots instead of waiting behind one serial prefix. - - For multi-surface local runs, the wrapper keeps the shared surface phases ordered, but batches inside the same shared phase now fan out together, deferred isolated work can overlap the next shared phase, and spare `unit-fast` headroom now starts that deferred work earlier instead of leaving those slots idle. - - Refresh the timing snapshots with `pnpm test:perf:update-timings` and `pnpm test:perf:update-timings:extensions` after major suite shape changes. - Embedded runner note: - When you change message-tool discovery inputs or compaction runtime context, keep both levels of coverage. @@ -90,20 +72,16 @@ Think of the suites as “increasing realism” (and increasing flakiness/cost): - Pool note: - Base Vitest config still defaults to `forks`. - Unit and boundary projects stay on `forks`. - - Channel, extension, and gateway planner lanes also stay on `forks`. + - Channel, extension, and gateway configs also stay on `forks`. - Unit, channel, and extension configs default to `isolate: false` for faster file startup. - `pnpm test` inherits the isolation defaults from `vitest.projects.config.ts`. - Opt back into unit-file isolation with `OPENCLAW_TEST_ISOLATE=1 pnpm test`. - `OPENCLAW_TEST_NO_ISOLATE=0` or `OPENCLAW_TEST_NO_ISOLATE=false` also force isolated runs. - Fast-local iteration note: - `pnpm test:changed` runs the native projects config with `--changed origin/main`. - - `pnpm test:changed:max` keeps the same changed-file filter but uses the planner's aggressive local profile. - - `pnpm test:max` exposes that same planner profile for a full local run. - - `pnpm test:planner` remains available when you want the old planner-backed local run shape. - - On supported local Node versions, including Node 25, the normal planner profile can use top-level lane parallelism. `pnpm test:max` still pushes the planner harder when you want a more aggressive local run. - - The base Vitest config marks the wrapper manifests/config files as `forceRerunTriggers` so changed-mode reruns stay correct when scheduler inputs change. - - The wrapper keeps `OPENCLAW_VITEST_FS_MODULE_CACHE` enabled on supported hosts, but assigns a lane-local `OPENCLAW_VITEST_FS_MODULE_CACHE_PATH` so concurrent Vitest processes do not race on one shared experimental cache directory. - - Set `OPENCLAW_VITEST_FS_MODULE_CACHE_PATH=/abs/path` if you want one explicit cache location for direct single-run profiling. + - `pnpm test:max` and `pnpm test:changed:max` keep the same native projects config, just with a higher worker cap. + - The base Vitest config marks the projects/config files as `forceRerunTriggers` so changed-mode reruns stay correct when test wiring changes. + - The config keeps `OPENCLAW_VITEST_FS_MODULE_CACHE` enabled on supported hosts; set `OPENCLAW_VITEST_FS_MODULE_CACHE_PATH=/abs/path` if you want one explicit cache location for direct profiling. - Perf-debug note: - `pnpm test:perf:imports` enables Vitest import-duration reporting plus import-breakdown output. - `pnpm test:perf:imports:changed` scopes the same profiling view to files changed since `origin/main`. diff --git a/docs/plugins/sdk-testing.md b/docs/plugins/sdk-testing.md index 30f90ea9576..82ddec9d410 100644 --- a/docs/plugins/sdk-testing.md +++ b/docs/plugins/sdk-testing.md @@ -252,7 +252,7 @@ pnpm test:coverage If local runs cause memory pressure: ```bash -OPENCLAW_TEST_PROFILE=low OPENCLAW_TEST_SERIAL_GATEWAY=1 pnpm test +OPENCLAW_VITEST_MAX_WORKERS=1 pnpm test ``` ## Related diff --git a/docs/reference/test.md b/docs/reference/test.md index cc6ee565985..deb64fdcfc0 100644 --- a/docs/reference/test.md +++ b/docs/reference/test.md @@ -12,17 +12,16 @@ title: "Tests" - `pnpm test:force`: Kills any lingering gateway process holding the default control port, then runs the full Vitest suite with an isolated gateway port so server tests don’t collide with a running instance. Use this when a prior gateway run left port 18789 occupied. - `pnpm test:coverage`: Runs the unit suite with V8 coverage (via `vitest.unit.config.ts`). Global thresholds are 70% lines/branches/functions/statements. Coverage excludes integration-heavy entrypoints (CLI wiring, gateway/telegram bridges, webchat static server) to keep the target focused on unit-testable logic. - `pnpm test:coverage:changed`: Runs unit coverage only for files changed since `origin/main`. -- `pnpm test:changed`: runs the wrapper with `--changed origin/main`. The base Vitest config treats the wrapper manifests/config files as `forceRerunTriggers` so scheduler changes still rerun broadly when needed. -- `pnpm test`: runs the full wrapper. It keeps only a small behavioral override manifest in git, then uses a checked-in timing snapshot to peel the heaviest measured unit files into dedicated lanes. -- Unit files default to `threads` in the wrapper; keep fork-only exceptions documented in `test/fixtures/test-parallel.behavior.json`. -- `pnpm test:channels` now defaults to `threads` via `vitest.channels.config.ts`; the March 22, 2026 direct full-suite control run passed clean without channel-specific fork exceptions. -- `pnpm test:extensions` runs through the wrapper and keeps documented extension fork-only exceptions in `test/fixtures/test-parallel.behavior.json`; the shared extension lane still defaults to `threads`. +- `pnpm test:changed`: runs the native Vitest projects config with `--changed origin/main`. The base config treats the projects/config files as `forceRerunTriggers` so wiring changes still rerun broadly when needed. +- `pnpm test`: runs the native Vitest projects config (`unit` + `boundary`) through the lightweight wrapper in `scripts/test-projects.mjs`. +- Unit, channel, and extension configs default to `pool: "forks"`. +- `pnpm test:channels` runs `vitest.channels.config.ts`. +- `pnpm test:extensions` runs `vitest.extensions.config.ts`. - `pnpm test:extensions`: runs extension/plugin suites. - `pnpm test:perf:imports`: enables Vitest import-duration + import-breakdown reporting for the wrapper. - `pnpm test:perf:imports:changed`: same import profiling, but only for files changed since `origin/main`. - `pnpm test:perf:profile:main`: writes a CPU profile for the Vitest main thread (`.artifacts/vitest-main-profile`). - `pnpm test:perf:profile:runner`: writes CPU + heap profiles for the unit runner (`.artifacts/vitest-runner-profile`). -- `pnpm test:perf:update-timings`: refreshes the checked-in slow-file timing snapshot used by `scripts/test-parallel.mjs`. - Gateway integration: opt-in via `OPENCLAW_TEST_INCLUDE_GATEWAY=1 pnpm test` or `pnpm test:gateway`. - `pnpm test:e2e`: Runs gateway end-to-end smoke tests (multi-instance WS/HTTP/node pairing). Defaults to `forks` + adaptive workers in `vitest.e2e.config.ts`; tune with `OPENCLAW_E2E_WORKERS=` and set `OPENCLAW_E2E_VERBOSE=1` for verbose logs. - `pnpm test:live`: Runs provider live tests (minimax/zai). Requires API keys and `LIVE=1` (or provider-specific `*_LIVE_TEST=1`) to unskip. @@ -38,9 +37,9 @@ For local PR land/gate checks, run: - `pnpm test` - `pnpm check:docs` -If `pnpm test` flakes on a loaded host, rerun once before treating it as a regression, then isolate with `pnpm vitest run `. For memory-constrained hosts, use: +If `pnpm test` flakes on a loaded host, rerun once before treating it as a regression, then isolate with `pnpm test -- `. For memory-constrained hosts, use: -- `OPENCLAW_TEST_PROFILE=low OPENCLAW_TEST_SERIAL_GATEWAY=1 pnpm test` +- `OPENCLAW_VITEST_MAX_WORKERS=1 pnpm test` - `OPENCLAW_VITEST_FS_MODULE_CACHE_PATH=/tmp/openclaw-vitest-cache pnpm test:changed` ## Model latency bench (local keys)