clawdie/hermes-bsd
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2
hermes-bsd/scripts/run_tests.sh

79 lines
3.3 KiB
Bash
Raw Normal View History

test: make test env hermetic; enforce CI parity via scripts/run_tests.sh (#11577) * test: make test env hermetic; enforce CI parity via scripts/run_tests.sh Fixes the recurring 'works locally, fails in CI' (and vice versa) class of flakes by making tests hermetic and providing a canonical local runner that matches CI's environment. ## Layer 1 — hermetic conftest.py (tests/conftest.py) Autouse fixture now unsets every credential-shaped env var before every test, so developer-local API keys can't leak into tests that assert 'auto-detect provider when key present'. Pattern: unset any var ending in _API_KEY, _TOKEN, _SECRET, _PASSWORD, _CREDENTIALS, _ACCESS_KEY, _PRIVATE_KEY, etc. Plus an explicit list of credential names that don't fit the suffix pattern (AWS_ACCESS_KEY_ID, FAL_KEY, GH_TOKEN, etc.) and all the provider BASE_URL overrides that change auto-detect behavior. Also unsets HERMES_* behavioral vars (HERMES_YOLO_MODE, HERMES_QUIET, HERMES_SESSION_*, etc.) that mutate agent behavior. Also: - Redirects HOME to a per-test tempdir (not just HERMES_HOME), so code reading ~/.hermes/* directly can't touch the real dir. - Pins TZ=UTC, LANG=C.UTF-8, LC_ALL=C.UTF-8, PYTHONHASHSEED=0 to match CI's deterministic runtime. The old _isolate_hermes_home fixture name is preserved as an alias so any test that yields it explicitly still works. ## Layer 2 — scripts/run_tests.sh canonical runner 'Always use scripts/run_tests.sh, never call pytest directly' is the new rule (documented in AGENTS.md). The script: - Unsets all credential env vars (belt-and-suspenders for callers who bypass conftest — e.g. IDE integrations) - Pins TZ/LANG/PYTHONHASHSEED - Uses -n 4 xdist workers (matches GHA ubuntu-latest; -n auto on a 20-core workstation surfaces test-ordering flakes CI will never see, causing the infamous 'passes in CI, fails locally' drift) - Finds the venv in .venv, venv, or main checkout's venv - Passes through arbitrary pytest args Installs pytest-split on demand so the script can also be used to run matrix-split subsets locally for debugging. ## Remove 3 module-level dotenv stubs that broke test isolation tests/hermes_cli/test_{arcee,xiaomi,api_key}_provider.py each had a module-level: if 'dotenv' not in sys.modules: fake_dotenv = types.ModuleType('dotenv') fake_dotenv.load_dotenv = lambda *a, **kw: None sys.modules['dotenv'] = fake_dotenv This patches sys.modules['dotenv'] to a fake at import time with no teardown. Under pytest-xdist LoadScheduling, whichever worker collected one of these files first poisoned its sys.modules; subsequent tests in the same worker that imported load_dotenv transitively (e.g. test_env_loader.py via hermes_cli.env_loader) got the no-op lambda and saw their assertions fail. dotenv is a required dependency (python-dotenv>=1.2.1 in pyproject.toml), so the defensive stub was never needed. Removed. ## Validation - tests/hermes_cli/ alone: 2178 passed, 1 skipped, 0 failed (was 4 failures in test_env_loader.py before this fix) - tests/test_plugin_skills.py, tests/hermes_cli/test_plugins.py, tests/test_hermes_logging.py combined: 123 passed (the caplog regression tests from PR #11453 still pass) - Local full run shows no F/E clusters in the 0-55% range that were previously present before the conftest hardening ## Background See AGENTS.md 'Testing' section for the full list of drift sources this closes. Matrix split (closed as #11566) will be re-attempted once this foundation lands — cross-test pollution was the root cause of the shard-3 hang in that PR. * fix(conftest): don't redirect HOME — it broke CI subprocesses PR #11577's autouse fixture was setting HOME to a per-test tempdir. CI started timing out at 97% complete with dozens of E/F markers and orphan python processes at cleanup — tests (or transitive deps) spawn subprocesses that expect a stable HOME, and the redirect broke them in non-obvious ways. Env-var unsetting and TZ/LANG/hashseed pinning (the actual CI-drift fixes) are unchanged and still in place. HERMES_HOME redirection is also unchanged — that's the canonical way to isolate tests from ~/.hermes/, not HOME. Any code in the codebase reading ~/.hermes/* via `Path.home() / ".hermes"` instead of `get_hermes_home()` is a bug to fix at the callsite, not something to paper over in conftest.
2026-04-17 06:09:09 -07:00
#!/usr/bin/env bash
# Canonical test runner for hermes-agent. Run this instead of calling
# `pytest` directly to guarantee your local run matches CI behavior.
#
# What this script enforces:
test: use subprocesses for each test file (#29016) * ci(tests): install ripgrep from prebuilt tarball instead of apt apt-get update + install of ripgrep takes ~4 min on the GHA Ubuntu runners (the apt-get update against archive.ubuntu.com is the slow part; ripgrep itself is small). Switching to the upstream musl binary tarball cuts the step to a few seconds. - Pinned to ripgrep 15.1.0 with sha256 verification (same hash as published in the releases sha256 sidecar file). - Drops the `rg` binary into /usr/local/bin so it is on PATH for every subsequent step without GITHUB_PATH manipulation. - Applied to both the test and e2e jobs in tests.yml. * fix(cli): compile syntax check to tempdir, not source __pycache__ `_validate_critical_files_syntax` runs `py_compile.compile()` on each critical bootstrap file after a successful `git pull`. The default `py_compile` writes the resulting `.pyc` next to the source under `__pycache__/`, which causes two real problems: 1. Parallel test workers walking the same source tree (e.g. running the suite under per-file process isolation) can race against each other on the `__pycache__` write — manifests as flaky 'directory not empty' errors during teardown. 2. In production, the post-pull syntax check leaves a `.pyc` behind that the next interpreter run might pick up — fine when the interpreter version matches, sketchy if it doesn't. Fix: write the compiled output to a `tempfile.TemporaryDirectory()` that's discarded on function exit. We only care about the compile-or-not signal, not the artifact. * test(runner): per-file process isolation, drop manual state reset + xdist Replace fragile manual _reset_module_state test fixtures with robust per-file subprocess isolation. Each test file runs in a fresh `python -m pytest <file>` subprocess via ThreadPoolExecutor. No xdist, no custom pytest plugin, no shared worker state. Key changes: * scripts/run_tests_parallel.py — new runner: discovers test files, runs N in parallel via ThreadPoolExecutor, captures stdout per file, treats exit code 5 (no tests collected) as pass, kills all children on exit. Change from cpu_count to cpu_count*2. The runner is I/O-bound (waiting on subprocess.communicate() from pytest children) The parent process does almost no CPU work, so 2x oversubscription keeps more pipes full. When a file fails, immediately show the last 30 lines of pytest output (stack traces + FAILED summary) plus a ready-to-copy repro command: python -m pytest tests/agent/test_auxiliary_client.py * scripts/run_tests.sh — delegates to run_tests_parallel.py * .github/workflows/tests.yml — test step: python scripts/run_tests_parallel.py * pyproject.toml — drop pytest-xdist, pytest-split; simplify addopts * tests/conftest.py — remove ~200 lines of manual state-reset fixtures * AGENTS.md — update Testing section for per-file design * test(runner): speed gateway test antipattern scan up * fix(test): web search provider plugin test missing xai * fix(tests): make 14 test files pass under per-file subprocess isolation Tests that relied on cross-file state pollution from xdist workers fail when run in isolation (per-file subprocess model). Root causes and fixes: Tool registry not populated: - test_video_generation_tool_surface_matrix: add discover_builtin_tools() - test_web_providers_brave_free/ddgs/searxng/general: autouse fixtures registering all 8 bundled web providers, reset after each test - test_website_policy: same provider registration pattern - test_web_tools_tavily: same pattern across 3 dispatch test classes - Also add is_safe_url/check_website_access mocks where SSRF check blocks example.com (DNS resolution fails in isolated envs) Stale check_fn cache: - test_kanban_tools: invalidate_check_fn_cache() + _clear_tool_defs_cache() in both kanban guidance tests (prior test cached False for kanban_show) - test_discord_tool: cache invalidation in setup/teardown - test_homeassistant_tool: invalidate_check_fn_cache() before registry queries Module-level state pollution: - test_auxiliary_client: autouse fixture clearing _aux_unhealthy_until cache - test_skill_commands: set_session_vars() instead of patch.dict(os.environ) (ContextVar takes precedence over os.environ) - test_dm_topics: overwrite sys.modules + separate telegram.constants mock + force-reimport of gateway.platforms.telegram - test_terminal_tool_requirements: removed duplicate class declaration, autouse _clear_caches fixture * change(tests): run_tests.sh explicitly includes env vars instead of manually dropping some vars, now we just only include some * fix(tests): 5 more isolation/NixOS fixes - test_approval_plugin_hooks: isolate HERMES_HOME so real user's command_allowlist doesn't short-circuit the approval path - test_google_chat: skipif when Platform.GOOGLE_CHAT not in enum (feature not merged on this branch) - test_write_deny: test systemd prefix against tmp_path instead of /etc/systemd which resolves to /nix/store on NixOS - test_pty_bridge: use shutil.which('cat') instead of /bin/cat (doesn't exist on NixOS) - profiles.py: rmtree onexc handler chmod's parent dirs too, fixing profile deletion when copytree preserved read-only modes from nix store * fix(tests): clear unhealthy cache in autouse fixture for auxiliary_client * fix(tests): skip send_message when telegram not installed; handle missing worker_id in browser_supervisor * fix: py3.11 rmtree onexc compat + belt-and-suspenders unhealthy cache clear for expired codex test * fix: address PR #29016 review feedback - Remove tracked .pytest-cache/ artifact and add to .gitignore - Fix stale 'xdist worker' comment in conftest.py - Deduplicate web provider registration into tests/tools/conftest.py shared helper (register_all_web_providers), replacing 8 copy-pasted blocks across 6 test files - Update PR description: remove stale recovered-test-files claim, fix worker count to match code (cpu_count*2) * fix: eliminate race in stale-cache achievements test The background scan thread could complete and overwrite _SNAPSHOT_CACHE before evaluate_all() returned the stale data — only 10 fake sessions made the scan finish instantly. Added scan_delay param to _FakeSessionDB and set it to 2s in the stale-cache test so the background thread can't win the race.
2026-05-21 07:10:04 -04:00
# * Per-file isolation via scripts/run_tests_parallel.py — each test
# file runs in its own freshly-spawned `python -m pytest <file>`
# subprocess. No xdist, no shared workers, no module-level leakage
# between files.
test: make test env hermetic; enforce CI parity via scripts/run_tests.sh (#11577) * test: make test env hermetic; enforce CI parity via scripts/run_tests.sh Fixes the recurring 'works locally, fails in CI' (and vice versa) class of flakes by making tests hermetic and providing a canonical local runner that matches CI's environment. ## Layer 1 — hermetic conftest.py (tests/conftest.py) Autouse fixture now unsets every credential-shaped env var before every test, so developer-local API keys can't leak into tests that assert 'auto-detect provider when key present'. Pattern: unset any var ending in _API_KEY, _TOKEN, _SECRET, _PASSWORD, _CREDENTIALS, _ACCESS_KEY, _PRIVATE_KEY, etc. Plus an explicit list of credential names that don't fit the suffix pattern (AWS_ACCESS_KEY_ID, FAL_KEY, GH_TOKEN, etc.) and all the provider BASE_URL overrides that change auto-detect behavior. Also unsets HERMES_* behavioral vars (HERMES_YOLO_MODE, HERMES_QUIET, HERMES_SESSION_*, etc.) that mutate agent behavior. Also: - Redirects HOME to a per-test tempdir (not just HERMES_HOME), so code reading ~/.hermes/* directly can't touch the real dir. - Pins TZ=UTC, LANG=C.UTF-8, LC_ALL=C.UTF-8, PYTHONHASHSEED=0 to match CI's deterministic runtime. The old _isolate_hermes_home fixture name is preserved as an alias so any test that yields it explicitly still works. ## Layer 2 — scripts/run_tests.sh canonical runner 'Always use scripts/run_tests.sh, never call pytest directly' is the new rule (documented in AGENTS.md). The script: - Unsets all credential env vars (belt-and-suspenders for callers who bypass conftest — e.g. IDE integrations) - Pins TZ/LANG/PYTHONHASHSEED - Uses -n 4 xdist workers (matches GHA ubuntu-latest; -n auto on a 20-core workstation surfaces test-ordering flakes CI will never see, causing the infamous 'passes in CI, fails locally' drift) - Finds the venv in .venv, venv, or main checkout's venv - Passes through arbitrary pytest args Installs pytest-split on demand so the script can also be used to run matrix-split subsets locally for debugging. ## Remove 3 module-level dotenv stubs that broke test isolation tests/hermes_cli/test_{arcee,xiaomi,api_key}_provider.py each had a module-level: if 'dotenv' not in sys.modules: fake_dotenv = types.ModuleType('dotenv') fake_dotenv.load_dotenv = lambda *a, **kw: None sys.modules['dotenv'] = fake_dotenv This patches sys.modules['dotenv'] to a fake at import time with no teardown. Under pytest-xdist LoadScheduling, whichever worker collected one of these files first poisoned its sys.modules; subsequent tests in the same worker that imported load_dotenv transitively (e.g. test_env_loader.py via hermes_cli.env_loader) got the no-op lambda and saw their assertions fail. dotenv is a required dependency (python-dotenv>=1.2.1 in pyproject.toml), so the defensive stub was never needed. Removed. ## Validation - tests/hermes_cli/ alone: 2178 passed, 1 skipped, 0 failed (was 4 failures in test_env_loader.py before this fix) - tests/test_plugin_skills.py, tests/hermes_cli/test_plugins.py, tests/test_hermes_logging.py combined: 123 passed (the caplog regression tests from PR #11453 still pass) - Local full run shows no F/E clusters in the 0-55% range that were previously present before the conftest hardening ## Background See AGENTS.md 'Testing' section for the full list of drift sources this closes. Matrix split (closed as #11566) will be re-attempted once this foundation lands — cross-test pollution was the root cause of the shard-3 hang in that PR. * fix(conftest): don't redirect HOME — it broke CI subprocesses PR #11577's autouse fixture was setting HOME to a per-test tempdir. CI started timing out at 97% complete with dozens of E/F markers and orphan python processes at cleanup — tests (or transitive deps) spawn subprocesses that expect a stable HOME, and the redirect broke them in non-obvious ways. Env-var unsetting and TZ/LANG/hashseed pinning (the actual CI-drift fixes) are unchanged and still in place. HERMES_HOME redirection is also unchanged — that's the canonical way to isolate tests from ~/.hermes/, not HOME. Any code in the codebase reading ~/.hermes/* via `Path.home() / ".hermes"` instead of `get_hermes_home()` is a bug to fix at the callsite, not something to paper over in conftest.
2026-04-17 06:09:09 -07:00
# * TZ=UTC, LANG=C.UTF-8, PYTHONHASHSEED=0 (deterministic)
test: use subprocesses for each test file (#29016) * ci(tests): install ripgrep from prebuilt tarball instead of apt apt-get update + install of ripgrep takes ~4 min on the GHA Ubuntu runners (the apt-get update against archive.ubuntu.com is the slow part; ripgrep itself is small). Switching to the upstream musl binary tarball cuts the step to a few seconds. - Pinned to ripgrep 15.1.0 with sha256 verification (same hash as published in the releases sha256 sidecar file). - Drops the `rg` binary into /usr/local/bin so it is on PATH for every subsequent step without GITHUB_PATH manipulation. - Applied to both the test and e2e jobs in tests.yml. * fix(cli): compile syntax check to tempdir, not source __pycache__ `_validate_critical_files_syntax` runs `py_compile.compile()` on each critical bootstrap file after a successful `git pull`. The default `py_compile` writes the resulting `.pyc` next to the source under `__pycache__/`, which causes two real problems: 1. Parallel test workers walking the same source tree (e.g. running the suite under per-file process isolation) can race against each other on the `__pycache__` write — manifests as flaky 'directory not empty' errors during teardown. 2. In production, the post-pull syntax check leaves a `.pyc` behind that the next interpreter run might pick up — fine when the interpreter version matches, sketchy if it doesn't. Fix: write the compiled output to a `tempfile.TemporaryDirectory()` that's discarded on function exit. We only care about the compile-or-not signal, not the artifact. * test(runner): per-file process isolation, drop manual state reset + xdist Replace fragile manual _reset_module_state test fixtures with robust per-file subprocess isolation. Each test file runs in a fresh `python -m pytest <file>` subprocess via ThreadPoolExecutor. No xdist, no custom pytest plugin, no shared worker state. Key changes: * scripts/run_tests_parallel.py — new runner: discovers test files, runs N in parallel via ThreadPoolExecutor, captures stdout per file, treats exit code 5 (no tests collected) as pass, kills all children on exit. Change from cpu_count to cpu_count*2. The runner is I/O-bound (waiting on subprocess.communicate() from pytest children) The parent process does almost no CPU work, so 2x oversubscription keeps more pipes full. When a file fails, immediately show the last 30 lines of pytest output (stack traces + FAILED summary) plus a ready-to-copy repro command: python -m pytest tests/agent/test_auxiliary_client.py * scripts/run_tests.sh — delegates to run_tests_parallel.py * .github/workflows/tests.yml — test step: python scripts/run_tests_parallel.py * pyproject.toml — drop pytest-xdist, pytest-split; simplify addopts * tests/conftest.py — remove ~200 lines of manual state-reset fixtures * AGENTS.md — update Testing section for per-file design * test(runner): speed gateway test antipattern scan up * fix(test): web search provider plugin test missing xai * fix(tests): make 14 test files pass under per-file subprocess isolation Tests that relied on cross-file state pollution from xdist workers fail when run in isolation (per-file subprocess model). Root causes and fixes: Tool registry not populated: - test_video_generation_tool_surface_matrix: add discover_builtin_tools() - test_web_providers_brave_free/ddgs/searxng/general: autouse fixtures registering all 8 bundled web providers, reset after each test - test_website_policy: same provider registration pattern - test_web_tools_tavily: same pattern across 3 dispatch test classes - Also add is_safe_url/check_website_access mocks where SSRF check blocks example.com (DNS resolution fails in isolated envs) Stale check_fn cache: - test_kanban_tools: invalidate_check_fn_cache() + _clear_tool_defs_cache() in both kanban guidance tests (prior test cached False for kanban_show) - test_discord_tool: cache invalidation in setup/teardown - test_homeassistant_tool: invalidate_check_fn_cache() before registry queries Module-level state pollution: - test_auxiliary_client: autouse fixture clearing _aux_unhealthy_until cache - test_skill_commands: set_session_vars() instead of patch.dict(os.environ) (ContextVar takes precedence over os.environ) - test_dm_topics: overwrite sys.modules + separate telegram.constants mock + force-reimport of gateway.platforms.telegram - test_terminal_tool_requirements: removed duplicate class declaration, autouse _clear_caches fixture * change(tests): run_tests.sh explicitly includes env vars instead of manually dropping some vars, now we just only include some * fix(tests): 5 more isolation/NixOS fixes - test_approval_plugin_hooks: isolate HERMES_HOME so real user's command_allowlist doesn't short-circuit the approval path - test_google_chat: skipif when Platform.GOOGLE_CHAT not in enum (feature not merged on this branch) - test_write_deny: test systemd prefix against tmp_path instead of /etc/systemd which resolves to /nix/store on NixOS - test_pty_bridge: use shutil.which('cat') instead of /bin/cat (doesn't exist on NixOS) - profiles.py: rmtree onexc handler chmod's parent dirs too, fixing profile deletion when copytree preserved read-only modes from nix store * fix(tests): clear unhealthy cache in autouse fixture for auxiliary_client * fix(tests): skip send_message when telegram not installed; handle missing worker_id in browser_supervisor * fix: py3.11 rmtree onexc compat + belt-and-suspenders unhealthy cache clear for expired codex test * fix: address PR #29016 review feedback - Remove tracked .pytest-cache/ artifact and add to .gitignore - Fix stale 'xdist worker' comment in conftest.py - Deduplicate web provider registration into tests/tools/conftest.py shared helper (register_all_web_providers), replacing 8 copy-pasted blocks across 6 test files - Update PR description: remove stale recovered-test-files claim, fix worker count to match code (cpu_count*2) * fix: eliminate race in stale-cache achievements test The background scan thread could complete and overwrite _SNAPSHOT_CACHE before evaluate_all() returned the stale data — only 10 fake sessions made the scan finish instantly. Added scan_delay param to _FakeSessionDB and set it to 2s in the stale-cache test so the background thread can't win the race.
2026-05-21 07:10:04 -04:00
# * Env vars blanked (conftest.py also does this, but this
# is belt-and-suspenders for anyone running pytest outside our
# conftest path — e.g. on a single file)
# * Proper venv activation (probes .venv, venv, then ~/.hermes/...)
test: make test env hermetic; enforce CI parity via scripts/run_tests.sh (#11577) * test: make test env hermetic; enforce CI parity via scripts/run_tests.sh Fixes the recurring 'works locally, fails in CI' (and vice versa) class of flakes by making tests hermetic and providing a canonical local runner that matches CI's environment. ## Layer 1 — hermetic conftest.py (tests/conftest.py) Autouse fixture now unsets every credential-shaped env var before every test, so developer-local API keys can't leak into tests that assert 'auto-detect provider when key present'. Pattern: unset any var ending in _API_KEY, _TOKEN, _SECRET, _PASSWORD, _CREDENTIALS, _ACCESS_KEY, _PRIVATE_KEY, etc. Plus an explicit list of credential names that don't fit the suffix pattern (AWS_ACCESS_KEY_ID, FAL_KEY, GH_TOKEN, etc.) and all the provider BASE_URL overrides that change auto-detect behavior. Also unsets HERMES_* behavioral vars (HERMES_YOLO_MODE, HERMES_QUIET, HERMES_SESSION_*, etc.) that mutate agent behavior. Also: - Redirects HOME to a per-test tempdir (not just HERMES_HOME), so code reading ~/.hermes/* directly can't touch the real dir. - Pins TZ=UTC, LANG=C.UTF-8, LC_ALL=C.UTF-8, PYTHONHASHSEED=0 to match CI's deterministic runtime. The old _isolate_hermes_home fixture name is preserved as an alias so any test that yields it explicitly still works. ## Layer 2 — scripts/run_tests.sh canonical runner 'Always use scripts/run_tests.sh, never call pytest directly' is the new rule (documented in AGENTS.md). The script: - Unsets all credential env vars (belt-and-suspenders for callers who bypass conftest — e.g. IDE integrations) - Pins TZ/LANG/PYTHONHASHSEED - Uses -n 4 xdist workers (matches GHA ubuntu-latest; -n auto on a 20-core workstation surfaces test-ordering flakes CI will never see, causing the infamous 'passes in CI, fails locally' drift) - Finds the venv in .venv, venv, or main checkout's venv - Passes through arbitrary pytest args Installs pytest-split on demand so the script can also be used to run matrix-split subsets locally for debugging. ## Remove 3 module-level dotenv stubs that broke test isolation tests/hermes_cli/test_{arcee,xiaomi,api_key}_provider.py each had a module-level: if 'dotenv' not in sys.modules: fake_dotenv = types.ModuleType('dotenv') fake_dotenv.load_dotenv = lambda *a, **kw: None sys.modules['dotenv'] = fake_dotenv This patches sys.modules['dotenv'] to a fake at import time with no teardown. Under pytest-xdist LoadScheduling, whichever worker collected one of these files first poisoned its sys.modules; subsequent tests in the same worker that imported load_dotenv transitively (e.g. test_env_loader.py via hermes_cli.env_loader) got the no-op lambda and saw their assertions fail. dotenv is a required dependency (python-dotenv>=1.2.1 in pyproject.toml), so the defensive stub was never needed. Removed. ## Validation - tests/hermes_cli/ alone: 2178 passed, 1 skipped, 0 failed (was 4 failures in test_env_loader.py before this fix) - tests/test_plugin_skills.py, tests/hermes_cli/test_plugins.py, tests/test_hermes_logging.py combined: 123 passed (the caplog regression tests from PR #11453 still pass) - Local full run shows no F/E clusters in the 0-55% range that were previously present before the conftest hardening ## Background See AGENTS.md 'Testing' section for the full list of drift sources this closes. Matrix split (closed as #11566) will be re-attempted once this foundation lands — cross-test pollution was the root cause of the shard-3 hang in that PR. * fix(conftest): don't redirect HOME — it broke CI subprocesses PR #11577's autouse fixture was setting HOME to a per-test tempdir. CI started timing out at 97% complete with dozens of E/F markers and orphan python processes at cleanup — tests (or transitive deps) spawn subprocesses that expect a stable HOME, and the redirect broke them in non-obvious ways. Env-var unsetting and TZ/LANG/hashseed pinning (the actual CI-drift fixes) are unchanged and still in place. HERMES_HOME redirection is also unchanged — that's the canonical way to isolate tests from ~/.hermes/, not HOME. Any code in the codebase reading ~/.hermes/* via `Path.home() / ".hermes"` instead of `get_hermes_home()` is a bug to fix at the callsite, not something to paper over in conftest.
2026-04-17 06:09:09 -07:00
#
# Usage:
test: use subprocesses for each test file (#29016) * ci(tests): install ripgrep from prebuilt tarball instead of apt apt-get update + install of ripgrep takes ~4 min on the GHA Ubuntu runners (the apt-get update against archive.ubuntu.com is the slow part; ripgrep itself is small). Switching to the upstream musl binary tarball cuts the step to a few seconds. - Pinned to ripgrep 15.1.0 with sha256 verification (same hash as published in the releases sha256 sidecar file). - Drops the `rg` binary into /usr/local/bin so it is on PATH for every subsequent step without GITHUB_PATH manipulation. - Applied to both the test and e2e jobs in tests.yml. * fix(cli): compile syntax check to tempdir, not source __pycache__ `_validate_critical_files_syntax` runs `py_compile.compile()` on each critical bootstrap file after a successful `git pull`. The default `py_compile` writes the resulting `.pyc` next to the source under `__pycache__/`, which causes two real problems: 1. Parallel test workers walking the same source tree (e.g. running the suite under per-file process isolation) can race against each other on the `__pycache__` write — manifests as flaky 'directory not empty' errors during teardown. 2. In production, the post-pull syntax check leaves a `.pyc` behind that the next interpreter run might pick up — fine when the interpreter version matches, sketchy if it doesn't. Fix: write the compiled output to a `tempfile.TemporaryDirectory()` that's discarded on function exit. We only care about the compile-or-not signal, not the artifact. * test(runner): per-file process isolation, drop manual state reset + xdist Replace fragile manual _reset_module_state test fixtures with robust per-file subprocess isolation. Each test file runs in a fresh `python -m pytest <file>` subprocess via ThreadPoolExecutor. No xdist, no custom pytest plugin, no shared worker state. Key changes: * scripts/run_tests_parallel.py — new runner: discovers test files, runs N in parallel via ThreadPoolExecutor, captures stdout per file, treats exit code 5 (no tests collected) as pass, kills all children on exit. Change from cpu_count to cpu_count*2. The runner is I/O-bound (waiting on subprocess.communicate() from pytest children) The parent process does almost no CPU work, so 2x oversubscription keeps more pipes full. When a file fails, immediately show the last 30 lines of pytest output (stack traces + FAILED summary) plus a ready-to-copy repro command: python -m pytest tests/agent/test_auxiliary_client.py * scripts/run_tests.sh — delegates to run_tests_parallel.py * .github/workflows/tests.yml — test step: python scripts/run_tests_parallel.py * pyproject.toml — drop pytest-xdist, pytest-split; simplify addopts * tests/conftest.py — remove ~200 lines of manual state-reset fixtures * AGENTS.md — update Testing section for per-file design * test(runner): speed gateway test antipattern scan up * fix(test): web search provider plugin test missing xai * fix(tests): make 14 test files pass under per-file subprocess isolation Tests that relied on cross-file state pollution from xdist workers fail when run in isolation (per-file subprocess model). Root causes and fixes: Tool registry not populated: - test_video_generation_tool_surface_matrix: add discover_builtin_tools() - test_web_providers_brave_free/ddgs/searxng/general: autouse fixtures registering all 8 bundled web providers, reset after each test - test_website_policy: same provider registration pattern - test_web_tools_tavily: same pattern across 3 dispatch test classes - Also add is_safe_url/check_website_access mocks where SSRF check blocks example.com (DNS resolution fails in isolated envs) Stale check_fn cache: - test_kanban_tools: invalidate_check_fn_cache() + _clear_tool_defs_cache() in both kanban guidance tests (prior test cached False for kanban_show) - test_discord_tool: cache invalidation in setup/teardown - test_homeassistant_tool: invalidate_check_fn_cache() before registry queries Module-level state pollution: - test_auxiliary_client: autouse fixture clearing _aux_unhealthy_until cache - test_skill_commands: set_session_vars() instead of patch.dict(os.environ) (ContextVar takes precedence over os.environ) - test_dm_topics: overwrite sys.modules + separate telegram.constants mock + force-reimport of gateway.platforms.telegram - test_terminal_tool_requirements: removed duplicate class declaration, autouse _clear_caches fixture * change(tests): run_tests.sh explicitly includes env vars instead of manually dropping some vars, now we just only include some * fix(tests): 5 more isolation/NixOS fixes - test_approval_plugin_hooks: isolate HERMES_HOME so real user's command_allowlist doesn't short-circuit the approval path - test_google_chat: skipif when Platform.GOOGLE_CHAT not in enum (feature not merged on this branch) - test_write_deny: test systemd prefix against tmp_path instead of /etc/systemd which resolves to /nix/store on NixOS - test_pty_bridge: use shutil.which('cat') instead of /bin/cat (doesn't exist on NixOS) - profiles.py: rmtree onexc handler chmod's parent dirs too, fixing profile deletion when copytree preserved read-only modes from nix store * fix(tests): clear unhealthy cache in autouse fixture for auxiliary_client * fix(tests): skip send_message when telegram not installed; handle missing worker_id in browser_supervisor * fix: py3.11 rmtree onexc compat + belt-and-suspenders unhealthy cache clear for expired codex test * fix: address PR #29016 review feedback - Remove tracked .pytest-cache/ artifact and add to .gitignore - Fix stale 'xdist worker' comment in conftest.py - Deduplicate web provider registration into tests/tools/conftest.py shared helper (register_all_web_providers), replacing 8 copy-pasted blocks across 6 test files - Update PR description: remove stale recovered-test-files claim, fix worker count to match code (cpu_count*2) * fix: eliminate race in stale-cache achievements test The background scan thread could complete and overwrite _SNAPSHOT_CACHE before evaluate_all() returned the stale data — only 10 fake sessions made the scan finish instantly. Added scan_delay param to _FakeSessionDB and set it to 2s in the stale-cache test so the background thread can't win the race.
2026-05-21 07:10:04 -04:00
# scripts/run_tests.sh # full suite
# scripts/run_tests.sh -j 4 # cap parallelism
# scripts/run_tests.sh tests/agent/ # discover only here
# scripts/run_tests.sh tests/agent/ tests/acp/ # multiple roots
# scripts/run_tests.sh tests/foo.py # single file
# scripts/run_tests.sh tests/foo.py -- --tb=long # path + pytest args
# scripts/run_tests.sh -- -v --tb=long # pytest args only
#
# Everything after a literal '--' is passed through to each per-file
# pytest invocation. Positional path arguments before '--' override
# the default discovery root (tests/).
test: make test env hermetic; enforce CI parity via scripts/run_tests.sh (#11577) * test: make test env hermetic; enforce CI parity via scripts/run_tests.sh Fixes the recurring 'works locally, fails in CI' (and vice versa) class of flakes by making tests hermetic and providing a canonical local runner that matches CI's environment. ## Layer 1 — hermetic conftest.py (tests/conftest.py) Autouse fixture now unsets every credential-shaped env var before every test, so developer-local API keys can't leak into tests that assert 'auto-detect provider when key present'. Pattern: unset any var ending in _API_KEY, _TOKEN, _SECRET, _PASSWORD, _CREDENTIALS, _ACCESS_KEY, _PRIVATE_KEY, etc. Plus an explicit list of credential names that don't fit the suffix pattern (AWS_ACCESS_KEY_ID, FAL_KEY, GH_TOKEN, etc.) and all the provider BASE_URL overrides that change auto-detect behavior. Also unsets HERMES_* behavioral vars (HERMES_YOLO_MODE, HERMES_QUIET, HERMES_SESSION_*, etc.) that mutate agent behavior. Also: - Redirects HOME to a per-test tempdir (not just HERMES_HOME), so code reading ~/.hermes/* directly can't touch the real dir. - Pins TZ=UTC, LANG=C.UTF-8, LC_ALL=C.UTF-8, PYTHONHASHSEED=0 to match CI's deterministic runtime. The old _isolate_hermes_home fixture name is preserved as an alias so any test that yields it explicitly still works. ## Layer 2 — scripts/run_tests.sh canonical runner 'Always use scripts/run_tests.sh, never call pytest directly' is the new rule (documented in AGENTS.md). The script: - Unsets all credential env vars (belt-and-suspenders for callers who bypass conftest — e.g. IDE integrations) - Pins TZ/LANG/PYTHONHASHSEED - Uses -n 4 xdist workers (matches GHA ubuntu-latest; -n auto on a 20-core workstation surfaces test-ordering flakes CI will never see, causing the infamous 'passes in CI, fails locally' drift) - Finds the venv in .venv, venv, or main checkout's venv - Passes through arbitrary pytest args Installs pytest-split on demand so the script can also be used to run matrix-split subsets locally for debugging. ## Remove 3 module-level dotenv stubs that broke test isolation tests/hermes_cli/test_{arcee,xiaomi,api_key}_provider.py each had a module-level: if 'dotenv' not in sys.modules: fake_dotenv = types.ModuleType('dotenv') fake_dotenv.load_dotenv = lambda *a, **kw: None sys.modules['dotenv'] = fake_dotenv This patches sys.modules['dotenv'] to a fake at import time with no teardown. Under pytest-xdist LoadScheduling, whichever worker collected one of these files first poisoned its sys.modules; subsequent tests in the same worker that imported load_dotenv transitively (e.g. test_env_loader.py via hermes_cli.env_loader) got the no-op lambda and saw their assertions fail. dotenv is a required dependency (python-dotenv>=1.2.1 in pyproject.toml), so the defensive stub was never needed. Removed. ## Validation - tests/hermes_cli/ alone: 2178 passed, 1 skipped, 0 failed (was 4 failures in test_env_loader.py before this fix) - tests/test_plugin_skills.py, tests/hermes_cli/test_plugins.py, tests/test_hermes_logging.py combined: 123 passed (the caplog regression tests from PR #11453 still pass) - Local full run shows no F/E clusters in the 0-55% range that were previously present before the conftest hardening ## Background See AGENTS.md 'Testing' section for the full list of drift sources this closes. Matrix split (closed as #11566) will be re-attempted once this foundation lands — cross-test pollution was the root cause of the shard-3 hang in that PR. * fix(conftest): don't redirect HOME — it broke CI subprocesses PR #11577's autouse fixture was setting HOME to a per-test tempdir. CI started timing out at 97% complete with dozens of E/F markers and orphan python processes at cleanup — tests (or transitive deps) spawn subprocesses that expect a stable HOME, and the redirect broke them in non-obvious ways. Env-var unsetting and TZ/LANG/hashseed pinning (the actual CI-drift fixes) are unchanged and still in place. HERMES_HOME redirection is also unchanged — that's the canonical way to isolate tests from ~/.hermes/, not HOME. Any code in the codebase reading ~/.hermes/* via `Path.home() / ".hermes"` instead of `get_hermes_home()` is a bug to fix at the callsite, not something to paper over in conftest.
2026-04-17 06:09:09 -07:00
set -euo pipefail
# ── Locate repo root ────────────────────────────────────────────────────────
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
# ── Activate venv ───────────────────────────────────────────────────────────
VENV=""
for candidate in "$REPO_ROOT/.venv" "$REPO_ROOT/venv" "$HOME/.hermes/hermes-agent/venv"; do
if [ -f "$candidate/bin/activate" ]; then
VENV="$candidate"
break
fi
done
if [ -z "$VENV" ]; then
echo "error: no virtualenv found in $REPO_ROOT/.venv or $REPO_ROOT/venv" >&2
exit 1
fi
PYTHON="$VENV/bin/python"
test: use subprocesses for each test file (#29016) * ci(tests): install ripgrep from prebuilt tarball instead of apt apt-get update + install of ripgrep takes ~4 min on the GHA Ubuntu runners (the apt-get update against archive.ubuntu.com is the slow part; ripgrep itself is small). Switching to the upstream musl binary tarball cuts the step to a few seconds. - Pinned to ripgrep 15.1.0 with sha256 verification (same hash as published in the releases sha256 sidecar file). - Drops the `rg` binary into /usr/local/bin so it is on PATH for every subsequent step without GITHUB_PATH manipulation. - Applied to both the test and e2e jobs in tests.yml. * fix(cli): compile syntax check to tempdir, not source __pycache__ `_validate_critical_files_syntax` runs `py_compile.compile()` on each critical bootstrap file after a successful `git pull`. The default `py_compile` writes the resulting `.pyc` next to the source under `__pycache__/`, which causes two real problems: 1. Parallel test workers walking the same source tree (e.g. running the suite under per-file process isolation) can race against each other on the `__pycache__` write — manifests as flaky 'directory not empty' errors during teardown. 2. In production, the post-pull syntax check leaves a `.pyc` behind that the next interpreter run might pick up — fine when the interpreter version matches, sketchy if it doesn't. Fix: write the compiled output to a `tempfile.TemporaryDirectory()` that's discarded on function exit. We only care about the compile-or-not signal, not the artifact. * test(runner): per-file process isolation, drop manual state reset + xdist Replace fragile manual _reset_module_state test fixtures with robust per-file subprocess isolation. Each test file runs in a fresh `python -m pytest <file>` subprocess via ThreadPoolExecutor. No xdist, no custom pytest plugin, no shared worker state. Key changes: * scripts/run_tests_parallel.py — new runner: discovers test files, runs N in parallel via ThreadPoolExecutor, captures stdout per file, treats exit code 5 (no tests collected) as pass, kills all children on exit. Change from cpu_count to cpu_count*2. The runner is I/O-bound (waiting on subprocess.communicate() from pytest children) The parent process does almost no CPU work, so 2x oversubscription keeps more pipes full. When a file fails, immediately show the last 30 lines of pytest output (stack traces + FAILED summary) plus a ready-to-copy repro command: python -m pytest tests/agent/test_auxiliary_client.py * scripts/run_tests.sh — delegates to run_tests_parallel.py * .github/workflows/tests.yml — test step: python scripts/run_tests_parallel.py * pyproject.toml — drop pytest-xdist, pytest-split; simplify addopts * tests/conftest.py — remove ~200 lines of manual state-reset fixtures * AGENTS.md — update Testing section for per-file design * test(runner): speed gateway test antipattern scan up * fix(test): web search provider plugin test missing xai * fix(tests): make 14 test files pass under per-file subprocess isolation Tests that relied on cross-file state pollution from xdist workers fail when run in isolation (per-file subprocess model). Root causes and fixes: Tool registry not populated: - test_video_generation_tool_surface_matrix: add discover_builtin_tools() - test_web_providers_brave_free/ddgs/searxng/general: autouse fixtures registering all 8 bundled web providers, reset after each test - test_website_policy: same provider registration pattern - test_web_tools_tavily: same pattern across 3 dispatch test classes - Also add is_safe_url/check_website_access mocks where SSRF check blocks example.com (DNS resolution fails in isolated envs) Stale check_fn cache: - test_kanban_tools: invalidate_check_fn_cache() + _clear_tool_defs_cache() in both kanban guidance tests (prior test cached False for kanban_show) - test_discord_tool: cache invalidation in setup/teardown - test_homeassistant_tool: invalidate_check_fn_cache() before registry queries Module-level state pollution: - test_auxiliary_client: autouse fixture clearing _aux_unhealthy_until cache - test_skill_commands: set_session_vars() instead of patch.dict(os.environ) (ContextVar takes precedence over os.environ) - test_dm_topics: overwrite sys.modules + separate telegram.constants mock + force-reimport of gateway.platforms.telegram - test_terminal_tool_requirements: removed duplicate class declaration, autouse _clear_caches fixture * change(tests): run_tests.sh explicitly includes env vars instead of manually dropping some vars, now we just only include some * fix(tests): 5 more isolation/NixOS fixes - test_approval_plugin_hooks: isolate HERMES_HOME so real user's command_allowlist doesn't short-circuit the approval path - test_google_chat: skipif when Platform.GOOGLE_CHAT not in enum (feature not merged on this branch) - test_write_deny: test systemd prefix against tmp_path instead of /etc/systemd which resolves to /nix/store on NixOS - test_pty_bridge: use shutil.which('cat') instead of /bin/cat (doesn't exist on NixOS) - profiles.py: rmtree onexc handler chmod's parent dirs too, fixing profile deletion when copytree preserved read-only modes from nix store * fix(tests): clear unhealthy cache in autouse fixture for auxiliary_client * fix(tests): skip send_message when telegram not installed; handle missing worker_id in browser_supervisor * fix: py3.11 rmtree onexc compat + belt-and-suspenders unhealthy cache clear for expired codex test * fix: address PR #29016 review feedback - Remove tracked .pytest-cache/ artifact and add to .gitignore - Fix stale 'xdist worker' comment in conftest.py - Deduplicate web provider registration into tests/tools/conftest.py shared helper (register_all_web_providers), replacing 8 copy-pasted blocks across 6 test files - Update PR description: remove stale recovered-test-files claim, fix worker count to match code (cpu_count*2) * fix: eliminate race in stale-cache achievements test The background scan thread could complete and overwrite _SNAPSHOT_CACHE before evaluate_all() returned the stale data — only 10 fake sessions made the scan finish instantly. Added scan_delay param to _FakeSessionDB and set it to 2s in the stale-cache test so the background thread can't win the race.
2026-05-21 07:10:04 -04:00
# ── Live-gateway plugin (computed before we drop env) ───────────────────────
EXTRA_PYTHONPATH=""
EXTRA_PYTEST_PLUGINS=""
test(conftest): plug every gateway-kill leak path (#23486) The existing _live_system_guard (PR #23397) blocked os.kill / os.killpg and a narrow subset of subprocess invocations. Tests still SIGTERMed the live gateway today (May 10) because the guard had structural holes. Plug them all: - subprocess: also wrap getoutput, getstatusoutput - os.system, os.popen - completely unwrapped before - pty.spawn - completely unwrapped before - asyncio.create_subprocess_exec / create_subprocess_shell - bypassed the subprocess module entirely; now wrapped - Subprocess command inspection now looks at the WHOLE command string, not just tokens[0]. Catches sudo systemctl, env systemctl, bash -c 'systemctl', setsid systemctl, /usr/bin/systemctl, etc. - New process-killer block: pkill / killall / taskkill / fuser targeting hermes/python patterns is now refused - os.kill PID 0 (own group) allowed; PID -1 (every process we can signal) refused - subprocess.Popen wrapper preserves __class_getitem__ so third-party packages that use Popen[bytes] as a type annotation still import Coverage is locked in by tests/test_live_system_guard_self_test.py - exercises every primitive against a guaranteed-foreign PID and asserts the guard fires. Adding a new kill primitive without updating the guard breaks CI. scripts/run_tests.sh now also force-loads ~/.hermes/pytest_live_guard.py when present (developer-machine convenience), so even worktrees that predate this commit get the protection on subsequent test runs through the canonical wrapper.
2026-05-10 18:55:28 -07:00
if [ -f "$HOME/.hermes/pytest_live_guard.py" ]; then
test: use subprocesses for each test file (#29016) * ci(tests): install ripgrep from prebuilt tarball instead of apt apt-get update + install of ripgrep takes ~4 min on the GHA Ubuntu runners (the apt-get update against archive.ubuntu.com is the slow part; ripgrep itself is small). Switching to the upstream musl binary tarball cuts the step to a few seconds. - Pinned to ripgrep 15.1.0 with sha256 verification (same hash as published in the releases sha256 sidecar file). - Drops the `rg` binary into /usr/local/bin so it is on PATH for every subsequent step without GITHUB_PATH manipulation. - Applied to both the test and e2e jobs in tests.yml. * fix(cli): compile syntax check to tempdir, not source __pycache__ `_validate_critical_files_syntax` runs `py_compile.compile()` on each critical bootstrap file after a successful `git pull`. The default `py_compile` writes the resulting `.pyc` next to the source under `__pycache__/`, which causes two real problems: 1. Parallel test workers walking the same source tree (e.g. running the suite under per-file process isolation) can race against each other on the `__pycache__` write — manifests as flaky 'directory not empty' errors during teardown. 2. In production, the post-pull syntax check leaves a `.pyc` behind that the next interpreter run might pick up — fine when the interpreter version matches, sketchy if it doesn't. Fix: write the compiled output to a `tempfile.TemporaryDirectory()` that's discarded on function exit. We only care about the compile-or-not signal, not the artifact. * test(runner): per-file process isolation, drop manual state reset + xdist Replace fragile manual _reset_module_state test fixtures with robust per-file subprocess isolation. Each test file runs in a fresh `python -m pytest <file>` subprocess via ThreadPoolExecutor. No xdist, no custom pytest plugin, no shared worker state. Key changes: * scripts/run_tests_parallel.py — new runner: discovers test files, runs N in parallel via ThreadPoolExecutor, captures stdout per file, treats exit code 5 (no tests collected) as pass, kills all children on exit. Change from cpu_count to cpu_count*2. The runner is I/O-bound (waiting on subprocess.communicate() from pytest children) The parent process does almost no CPU work, so 2x oversubscription keeps more pipes full. When a file fails, immediately show the last 30 lines of pytest output (stack traces + FAILED summary) plus a ready-to-copy repro command: python -m pytest tests/agent/test_auxiliary_client.py * scripts/run_tests.sh — delegates to run_tests_parallel.py * .github/workflows/tests.yml — test step: python scripts/run_tests_parallel.py * pyproject.toml — drop pytest-xdist, pytest-split; simplify addopts * tests/conftest.py — remove ~200 lines of manual state-reset fixtures * AGENTS.md — update Testing section for per-file design * test(runner): speed gateway test antipattern scan up * fix(test): web search provider plugin test missing xai * fix(tests): make 14 test files pass under per-file subprocess isolation Tests that relied on cross-file state pollution from xdist workers fail when run in isolation (per-file subprocess model). Root causes and fixes: Tool registry not populated: - test_video_generation_tool_surface_matrix: add discover_builtin_tools() - test_web_providers_brave_free/ddgs/searxng/general: autouse fixtures registering all 8 bundled web providers, reset after each test - test_website_policy: same provider registration pattern - test_web_tools_tavily: same pattern across 3 dispatch test classes - Also add is_safe_url/check_website_access mocks where SSRF check blocks example.com (DNS resolution fails in isolated envs) Stale check_fn cache: - test_kanban_tools: invalidate_check_fn_cache() + _clear_tool_defs_cache() in both kanban guidance tests (prior test cached False for kanban_show) - test_discord_tool: cache invalidation in setup/teardown - test_homeassistant_tool: invalidate_check_fn_cache() before registry queries Module-level state pollution: - test_auxiliary_client: autouse fixture clearing _aux_unhealthy_until cache - test_skill_commands: set_session_vars() instead of patch.dict(os.environ) (ContextVar takes precedence over os.environ) - test_dm_topics: overwrite sys.modules + separate telegram.constants mock + force-reimport of gateway.platforms.telegram - test_terminal_tool_requirements: removed duplicate class declaration, autouse _clear_caches fixture * change(tests): run_tests.sh explicitly includes env vars instead of manually dropping some vars, now we just only include some * fix(tests): 5 more isolation/NixOS fixes - test_approval_plugin_hooks: isolate HERMES_HOME so real user's command_allowlist doesn't short-circuit the approval path - test_google_chat: skipif when Platform.GOOGLE_CHAT not in enum (feature not merged on this branch) - test_write_deny: test systemd prefix against tmp_path instead of /etc/systemd which resolves to /nix/store on NixOS - test_pty_bridge: use shutil.which('cat') instead of /bin/cat (doesn't exist on NixOS) - profiles.py: rmtree onexc handler chmod's parent dirs too, fixing profile deletion when copytree preserved read-only modes from nix store * fix(tests): clear unhealthy cache in autouse fixture for auxiliary_client * fix(tests): skip send_message when telegram not installed; handle missing worker_id in browser_supervisor * fix: py3.11 rmtree onexc compat + belt-and-suspenders unhealthy cache clear for expired codex test * fix: address PR #29016 review feedback - Remove tracked .pytest-cache/ artifact and add to .gitignore - Fix stale 'xdist worker' comment in conftest.py - Deduplicate web provider registration into tests/tools/conftest.py shared helper (register_all_web_providers), replacing 8 copy-pasted blocks across 6 test files - Update PR description: remove stale recovered-test-files claim, fix worker count to match code (cpu_count*2) * fix: eliminate race in stale-cache achievements test The background scan thread could complete and overwrite _SNAPSHOT_CACHE before evaluate_all() returned the stale data — only 10 fake sessions made the scan finish instantly. Added scan_delay param to _FakeSessionDB and set it to 2s in the stale-cache test so the background thread can't win the race.
2026-05-21 07:10:04 -04:00
EXTRA_PYTHONPATH="$HOME/.hermes"
EXTRA_PYTEST_PLUGINS="pytest_live_guard"
test(conftest): plug every gateway-kill leak path (#23486) The existing _live_system_guard (PR #23397) blocked os.kill / os.killpg and a narrow subset of subprocess invocations. Tests still SIGTERMed the live gateway today (May 10) because the guard had structural holes. Plug them all: - subprocess: also wrap getoutput, getstatusoutput - os.system, os.popen - completely unwrapped before - pty.spawn - completely unwrapped before - asyncio.create_subprocess_exec / create_subprocess_shell - bypassed the subprocess module entirely; now wrapped - Subprocess command inspection now looks at the WHOLE command string, not just tokens[0]. Catches sudo systemctl, env systemctl, bash -c 'systemctl', setsid systemctl, /usr/bin/systemctl, etc. - New process-killer block: pkill / killall / taskkill / fuser targeting hermes/python patterns is now refused - os.kill PID 0 (own group) allowed; PID -1 (every process we can signal) refused - subprocess.Popen wrapper preserves __class_getitem__ so third-party packages that use Popen[bytes] as a type annotation still import Coverage is locked in by tests/test_live_system_guard_self_test.py - exercises every primitive against a guaranteed-foreign PID and asserts the guard fires. Adding a new kill primitive without updating the guard breaks CI. scripts/run_tests.sh now also force-loads ~/.hermes/pytest_live_guard.py when present (developer-machine convenience), so even worktrees that predate this commit get the protection on subsequent test runs through the canonical wrapper.
2026-05-10 18:55:28 -07:00
fi
test: make test env hermetic; enforce CI parity via scripts/run_tests.sh (#11577) * test: make test env hermetic; enforce CI parity via scripts/run_tests.sh Fixes the recurring 'works locally, fails in CI' (and vice versa) class of flakes by making tests hermetic and providing a canonical local runner that matches CI's environment. ## Layer 1 — hermetic conftest.py (tests/conftest.py) Autouse fixture now unsets every credential-shaped env var before every test, so developer-local API keys can't leak into tests that assert 'auto-detect provider when key present'. Pattern: unset any var ending in _API_KEY, _TOKEN, _SECRET, _PASSWORD, _CREDENTIALS, _ACCESS_KEY, _PRIVATE_KEY, etc. Plus an explicit list of credential names that don't fit the suffix pattern (AWS_ACCESS_KEY_ID, FAL_KEY, GH_TOKEN, etc.) and all the provider BASE_URL overrides that change auto-detect behavior. Also unsets HERMES_* behavioral vars (HERMES_YOLO_MODE, HERMES_QUIET, HERMES_SESSION_*, etc.) that mutate agent behavior. Also: - Redirects HOME to a per-test tempdir (not just HERMES_HOME), so code reading ~/.hermes/* directly can't touch the real dir. - Pins TZ=UTC, LANG=C.UTF-8, LC_ALL=C.UTF-8, PYTHONHASHSEED=0 to match CI's deterministic runtime. The old _isolate_hermes_home fixture name is preserved as an alias so any test that yields it explicitly still works. ## Layer 2 — scripts/run_tests.sh canonical runner 'Always use scripts/run_tests.sh, never call pytest directly' is the new rule (documented in AGENTS.md). The script: - Unsets all credential env vars (belt-and-suspenders for callers who bypass conftest — e.g. IDE integrations) - Pins TZ/LANG/PYTHONHASHSEED - Uses -n 4 xdist workers (matches GHA ubuntu-latest; -n auto on a 20-core workstation surfaces test-ordering flakes CI will never see, causing the infamous 'passes in CI, fails locally' drift) - Finds the venv in .venv, venv, or main checkout's venv - Passes through arbitrary pytest args Installs pytest-split on demand so the script can also be used to run matrix-split subsets locally for debugging. ## Remove 3 module-level dotenv stubs that broke test isolation tests/hermes_cli/test_{arcee,xiaomi,api_key}_provider.py each had a module-level: if 'dotenv' not in sys.modules: fake_dotenv = types.ModuleType('dotenv') fake_dotenv.load_dotenv = lambda *a, **kw: None sys.modules['dotenv'] = fake_dotenv This patches sys.modules['dotenv'] to a fake at import time with no teardown. Under pytest-xdist LoadScheduling, whichever worker collected one of these files first poisoned its sys.modules; subsequent tests in the same worker that imported load_dotenv transitively (e.g. test_env_loader.py via hermes_cli.env_loader) got the no-op lambda and saw their assertions fail. dotenv is a required dependency (python-dotenv>=1.2.1 in pyproject.toml), so the defensive stub was never needed. Removed. ## Validation - tests/hermes_cli/ alone: 2178 passed, 1 skipped, 0 failed (was 4 failures in test_env_loader.py before this fix) - tests/test_plugin_skills.py, tests/hermes_cli/test_plugins.py, tests/test_hermes_logging.py combined: 123 passed (the caplog regression tests from PR #11453 still pass) - Local full run shows no F/E clusters in the 0-55% range that were previously present before the conftest hardening ## Background See AGENTS.md 'Testing' section for the full list of drift sources this closes. Matrix split (closed as #11566) will be re-attempted once this foundation lands — cross-test pollution was the root cause of the shard-3 hang in that PR. * fix(conftest): don't redirect HOME — it broke CI subprocesses PR #11577's autouse fixture was setting HOME to a per-test tempdir. CI started timing out at 97% complete with dozens of E/F markers and orphan python processes at cleanup — tests (or transitive deps) spawn subprocesses that expect a stable HOME, and the redirect broke them in non-obvious ways. Env-var unsetting and TZ/LANG/hashseed pinning (the actual CI-drift fixes) are unchanged and still in place. HERMES_HOME redirection is also unchanged — that's the canonical way to isolate tests from ~/.hermes/, not HOME. Any code in the codebase reading ~/.hermes/* via `Path.home() / ".hermes"` instead of `get_hermes_home()` is a bug to fix at the callsite, not something to paper over in conftest.
2026-04-17 06:09:09 -07:00
test: use subprocesses for each test file (#29016) * ci(tests): install ripgrep from prebuilt tarball instead of apt apt-get update + install of ripgrep takes ~4 min on the GHA Ubuntu runners (the apt-get update against archive.ubuntu.com is the slow part; ripgrep itself is small). Switching to the upstream musl binary tarball cuts the step to a few seconds. - Pinned to ripgrep 15.1.0 with sha256 verification (same hash as published in the releases sha256 sidecar file). - Drops the `rg` binary into /usr/local/bin so it is on PATH for every subsequent step without GITHUB_PATH manipulation. - Applied to both the test and e2e jobs in tests.yml. * fix(cli): compile syntax check to tempdir, not source __pycache__ `_validate_critical_files_syntax` runs `py_compile.compile()` on each critical bootstrap file after a successful `git pull`. The default `py_compile` writes the resulting `.pyc` next to the source under `__pycache__/`, which causes two real problems: 1. Parallel test workers walking the same source tree (e.g. running the suite under per-file process isolation) can race against each other on the `__pycache__` write — manifests as flaky 'directory not empty' errors during teardown. 2. In production, the post-pull syntax check leaves a `.pyc` behind that the next interpreter run might pick up — fine when the interpreter version matches, sketchy if it doesn't. Fix: write the compiled output to a `tempfile.TemporaryDirectory()` that's discarded on function exit. We only care about the compile-or-not signal, not the artifact. * test(runner): per-file process isolation, drop manual state reset + xdist Replace fragile manual _reset_module_state test fixtures with robust per-file subprocess isolation. Each test file runs in a fresh `python -m pytest <file>` subprocess via ThreadPoolExecutor. No xdist, no custom pytest plugin, no shared worker state. Key changes: * scripts/run_tests_parallel.py — new runner: discovers test files, runs N in parallel via ThreadPoolExecutor, captures stdout per file, treats exit code 5 (no tests collected) as pass, kills all children on exit. Change from cpu_count to cpu_count*2. The runner is I/O-bound (waiting on subprocess.communicate() from pytest children) The parent process does almost no CPU work, so 2x oversubscription keeps more pipes full. When a file fails, immediately show the last 30 lines of pytest output (stack traces + FAILED summary) plus a ready-to-copy repro command: python -m pytest tests/agent/test_auxiliary_client.py * scripts/run_tests.sh — delegates to run_tests_parallel.py * .github/workflows/tests.yml — test step: python scripts/run_tests_parallel.py * pyproject.toml — drop pytest-xdist, pytest-split; simplify addopts * tests/conftest.py — remove ~200 lines of manual state-reset fixtures * AGENTS.md — update Testing section for per-file design * test(runner): speed gateway test antipattern scan up * fix(test): web search provider plugin test missing xai * fix(tests): make 14 test files pass under per-file subprocess isolation Tests that relied on cross-file state pollution from xdist workers fail when run in isolation (per-file subprocess model). Root causes and fixes: Tool registry not populated: - test_video_generation_tool_surface_matrix: add discover_builtin_tools() - test_web_providers_brave_free/ddgs/searxng/general: autouse fixtures registering all 8 bundled web providers, reset after each test - test_website_policy: same provider registration pattern - test_web_tools_tavily: same pattern across 3 dispatch test classes - Also add is_safe_url/check_website_access mocks where SSRF check blocks example.com (DNS resolution fails in isolated envs) Stale check_fn cache: - test_kanban_tools: invalidate_check_fn_cache() + _clear_tool_defs_cache() in both kanban guidance tests (prior test cached False for kanban_show) - test_discord_tool: cache invalidation in setup/teardown - test_homeassistant_tool: invalidate_check_fn_cache() before registry queries Module-level state pollution: - test_auxiliary_client: autouse fixture clearing _aux_unhealthy_until cache - test_skill_commands: set_session_vars() instead of patch.dict(os.environ) (ContextVar takes precedence over os.environ) - test_dm_topics: overwrite sys.modules + separate telegram.constants mock + force-reimport of gateway.platforms.telegram - test_terminal_tool_requirements: removed duplicate class declaration, autouse _clear_caches fixture * change(tests): run_tests.sh explicitly includes env vars instead of manually dropping some vars, now we just only include some * fix(tests): 5 more isolation/NixOS fixes - test_approval_plugin_hooks: isolate HERMES_HOME so real user's command_allowlist doesn't short-circuit the approval path - test_google_chat: skipif when Platform.GOOGLE_CHAT not in enum (feature not merged on this branch) - test_write_deny: test systemd prefix against tmp_path instead of /etc/systemd which resolves to /nix/store on NixOS - test_pty_bridge: use shutil.which('cat') instead of /bin/cat (doesn't exist on NixOS) - profiles.py: rmtree onexc handler chmod's parent dirs too, fixing profile deletion when copytree preserved read-only modes from nix store * fix(tests): clear unhealthy cache in autouse fixture for auxiliary_client * fix(tests): skip send_message when telegram not installed; handle missing worker_id in browser_supervisor * fix: py3.11 rmtree onexc compat + belt-and-suspenders unhealthy cache clear for expired codex test * fix: address PR #29016 review feedback - Remove tracked .pytest-cache/ artifact and add to .gitignore - Fix stale 'xdist worker' comment in conftest.py - Deduplicate web provider registration into tests/tools/conftest.py shared helper (register_all_web_providers), replacing 8 copy-pasted blocks across 6 test files - Update PR description: remove stale recovered-test-files claim, fix worker count to match code (cpu_count*2) * fix: eliminate race in stale-cache achievements test The background scan thread could complete and overwrite _SNAPSHOT_CACHE before evaluate_all() returned the stale data — only 10 fake sessions made the scan finish instantly. Added scan_delay param to _FakeSessionDB and set it to 2s in the stale-cache test so the background thread can't win the race.
2026-05-21 07:10:04 -04:00
# ── Run in hermetic env ──────────────────────────────────────────────────────
# env -i: start with empty environment, opt-in only what we need.
# No credential var can leak — you'd have to explicitly add it here.
echo "▶ running per-file parallel test suite via run_tests_parallel.py"
echo " (TZ=UTC LANG=C.UTF-8 PYTHONHASHSEED=0; clean env)"
test: make test env hermetic; enforce CI parity via scripts/run_tests.sh (#11577) * test: make test env hermetic; enforce CI parity via scripts/run_tests.sh Fixes the recurring 'works locally, fails in CI' (and vice versa) class of flakes by making tests hermetic and providing a canonical local runner that matches CI's environment. ## Layer 1 — hermetic conftest.py (tests/conftest.py) Autouse fixture now unsets every credential-shaped env var before every test, so developer-local API keys can't leak into tests that assert 'auto-detect provider when key present'. Pattern: unset any var ending in _API_KEY, _TOKEN, _SECRET, _PASSWORD, _CREDENTIALS, _ACCESS_KEY, _PRIVATE_KEY, etc. Plus an explicit list of credential names that don't fit the suffix pattern (AWS_ACCESS_KEY_ID, FAL_KEY, GH_TOKEN, etc.) and all the provider BASE_URL overrides that change auto-detect behavior. Also unsets HERMES_* behavioral vars (HERMES_YOLO_MODE, HERMES_QUIET, HERMES_SESSION_*, etc.) that mutate agent behavior. Also: - Redirects HOME to a per-test tempdir (not just HERMES_HOME), so code reading ~/.hermes/* directly can't touch the real dir. - Pins TZ=UTC, LANG=C.UTF-8, LC_ALL=C.UTF-8, PYTHONHASHSEED=0 to match CI's deterministic runtime. The old _isolate_hermes_home fixture name is preserved as an alias so any test that yields it explicitly still works. ## Layer 2 — scripts/run_tests.sh canonical runner 'Always use scripts/run_tests.sh, never call pytest directly' is the new rule (documented in AGENTS.md). The script: - Unsets all credential env vars (belt-and-suspenders for callers who bypass conftest — e.g. IDE integrations) - Pins TZ/LANG/PYTHONHASHSEED - Uses -n 4 xdist workers (matches GHA ubuntu-latest; -n auto on a 20-core workstation surfaces test-ordering flakes CI will never see, causing the infamous 'passes in CI, fails locally' drift) - Finds the venv in .venv, venv, or main checkout's venv - Passes through arbitrary pytest args Installs pytest-split on demand so the script can also be used to run matrix-split subsets locally for debugging. ## Remove 3 module-level dotenv stubs that broke test isolation tests/hermes_cli/test_{arcee,xiaomi,api_key}_provider.py each had a module-level: if 'dotenv' not in sys.modules: fake_dotenv = types.ModuleType('dotenv') fake_dotenv.load_dotenv = lambda *a, **kw: None sys.modules['dotenv'] = fake_dotenv This patches sys.modules['dotenv'] to a fake at import time with no teardown. Under pytest-xdist LoadScheduling, whichever worker collected one of these files first poisoned its sys.modules; subsequent tests in the same worker that imported load_dotenv transitively (e.g. test_env_loader.py via hermes_cli.env_loader) got the no-op lambda and saw their assertions fail. dotenv is a required dependency (python-dotenv>=1.2.1 in pyproject.toml), so the defensive stub was never needed. Removed. ## Validation - tests/hermes_cli/ alone: 2178 passed, 1 skipped, 0 failed (was 4 failures in test_env_loader.py before this fix) - tests/test_plugin_skills.py, tests/hermes_cli/test_plugins.py, tests/test_hermes_logging.py combined: 123 passed (the caplog regression tests from PR #11453 still pass) - Local full run shows no F/E clusters in the 0-55% range that were previously present before the conftest hardening ## Background See AGENTS.md 'Testing' section for the full list of drift sources this closes. Matrix split (closed as #11566) will be re-attempted once this foundation lands — cross-test pollution was the root cause of the shard-3 hang in that PR. * fix(conftest): don't redirect HOME — it broke CI subprocesses PR #11577's autouse fixture was setting HOME to a per-test tempdir. CI started timing out at 97% complete with dozens of E/F markers and orphan python processes at cleanup — tests (or transitive deps) spawn subprocesses that expect a stable HOME, and the redirect broke them in non-obvious ways. Env-var unsetting and TZ/LANG/hashseed pinning (the actual CI-drift fixes) are unchanged and still in place. HERMES_HOME redirection is also unchanged — that's the canonical way to isolate tests from ~/.hermes/, not HOME. Any code in the codebase reading ~/.hermes/* via `Path.home() / ".hermes"` instead of `get_hermes_home()` is a bug to fix at the callsite, not something to paper over in conftest.
2026-04-17 06:09:09 -07:00
test: use subprocesses for each test file (#29016) * ci(tests): install ripgrep from prebuilt tarball instead of apt apt-get update + install of ripgrep takes ~4 min on the GHA Ubuntu runners (the apt-get update against archive.ubuntu.com is the slow part; ripgrep itself is small). Switching to the upstream musl binary tarball cuts the step to a few seconds. - Pinned to ripgrep 15.1.0 with sha256 verification (same hash as published in the releases sha256 sidecar file). - Drops the `rg` binary into /usr/local/bin so it is on PATH for every subsequent step without GITHUB_PATH manipulation. - Applied to both the test and e2e jobs in tests.yml. * fix(cli): compile syntax check to tempdir, not source __pycache__ `_validate_critical_files_syntax` runs `py_compile.compile()` on each critical bootstrap file after a successful `git pull`. The default `py_compile` writes the resulting `.pyc` next to the source under `__pycache__/`, which causes two real problems: 1. Parallel test workers walking the same source tree (e.g. running the suite under per-file process isolation) can race against each other on the `__pycache__` write — manifests as flaky 'directory not empty' errors during teardown. 2. In production, the post-pull syntax check leaves a `.pyc` behind that the next interpreter run might pick up — fine when the interpreter version matches, sketchy if it doesn't. Fix: write the compiled output to a `tempfile.TemporaryDirectory()` that's discarded on function exit. We only care about the compile-or-not signal, not the artifact. * test(runner): per-file process isolation, drop manual state reset + xdist Replace fragile manual _reset_module_state test fixtures with robust per-file subprocess isolation. Each test file runs in a fresh `python -m pytest <file>` subprocess via ThreadPoolExecutor. No xdist, no custom pytest plugin, no shared worker state. Key changes: * scripts/run_tests_parallel.py — new runner: discovers test files, runs N in parallel via ThreadPoolExecutor, captures stdout per file, treats exit code 5 (no tests collected) as pass, kills all children on exit. Change from cpu_count to cpu_count*2. The runner is I/O-bound (waiting on subprocess.communicate() from pytest children) The parent process does almost no CPU work, so 2x oversubscription keeps more pipes full. When a file fails, immediately show the last 30 lines of pytest output (stack traces + FAILED summary) plus a ready-to-copy repro command: python -m pytest tests/agent/test_auxiliary_client.py * scripts/run_tests.sh — delegates to run_tests_parallel.py * .github/workflows/tests.yml — test step: python scripts/run_tests_parallel.py * pyproject.toml — drop pytest-xdist, pytest-split; simplify addopts * tests/conftest.py — remove ~200 lines of manual state-reset fixtures * AGENTS.md — update Testing section for per-file design * test(runner): speed gateway test antipattern scan up * fix(test): web search provider plugin test missing xai * fix(tests): make 14 test files pass under per-file subprocess isolation Tests that relied on cross-file state pollution from xdist workers fail when run in isolation (per-file subprocess model). Root causes and fixes: Tool registry not populated: - test_video_generation_tool_surface_matrix: add discover_builtin_tools() - test_web_providers_brave_free/ddgs/searxng/general: autouse fixtures registering all 8 bundled web providers, reset after each test - test_website_policy: same provider registration pattern - test_web_tools_tavily: same pattern across 3 dispatch test classes - Also add is_safe_url/check_website_access mocks where SSRF check blocks example.com (DNS resolution fails in isolated envs) Stale check_fn cache: - test_kanban_tools: invalidate_check_fn_cache() + _clear_tool_defs_cache() in both kanban guidance tests (prior test cached False for kanban_show) - test_discord_tool: cache invalidation in setup/teardown - test_homeassistant_tool: invalidate_check_fn_cache() before registry queries Module-level state pollution: - test_auxiliary_client: autouse fixture clearing _aux_unhealthy_until cache - test_skill_commands: set_session_vars() instead of patch.dict(os.environ) (ContextVar takes precedence over os.environ) - test_dm_topics: overwrite sys.modules + separate telegram.constants mock + force-reimport of gateway.platforms.telegram - test_terminal_tool_requirements: removed duplicate class declaration, autouse _clear_caches fixture * change(tests): run_tests.sh explicitly includes env vars instead of manually dropping some vars, now we just only include some * fix(tests): 5 more isolation/NixOS fixes - test_approval_plugin_hooks: isolate HERMES_HOME so real user's command_allowlist doesn't short-circuit the approval path - test_google_chat: skipif when Platform.GOOGLE_CHAT not in enum (feature not merged on this branch) - test_write_deny: test systemd prefix against tmp_path instead of /etc/systemd which resolves to /nix/store on NixOS - test_pty_bridge: use shutil.which('cat') instead of /bin/cat (doesn't exist on NixOS) - profiles.py: rmtree onexc handler chmod's parent dirs too, fixing profile deletion when copytree preserved read-only modes from nix store * fix(tests): clear unhealthy cache in autouse fixture for auxiliary_client * fix(tests): skip send_message when telegram not installed; handle missing worker_id in browser_supervisor * fix: py3.11 rmtree onexc compat + belt-and-suspenders unhealthy cache clear for expired codex test * fix: address PR #29016 review feedback - Remove tracked .pytest-cache/ artifact and add to .gitignore - Fix stale 'xdist worker' comment in conftest.py - Deduplicate web provider registration into tests/tools/conftest.py shared helper (register_all_web_providers), replacing 8 copy-pasted blocks across 6 test files - Update PR description: remove stale recovered-test-files claim, fix worker count to match code (cpu_count*2) * fix: eliminate race in stale-cache achievements test The background scan thread could complete and overwrite _SNAPSHOT_CACHE before evaluate_all() returned the stale data — only 10 fake sessions made the scan finish instantly. Added scan_delay param to _FakeSessionDB and set it to 2s in the stale-cache test so the background thread can't win the race.
2026-05-21 07:10:04 -04:00
cd "$REPO_ROOT"
test: make test env hermetic; enforce CI parity via scripts/run_tests.sh (#11577) * test: make test env hermetic; enforce CI parity via scripts/run_tests.sh Fixes the recurring 'works locally, fails in CI' (and vice versa) class of flakes by making tests hermetic and providing a canonical local runner that matches CI's environment. ## Layer 1 — hermetic conftest.py (tests/conftest.py) Autouse fixture now unsets every credential-shaped env var before every test, so developer-local API keys can't leak into tests that assert 'auto-detect provider when key present'. Pattern: unset any var ending in _API_KEY, _TOKEN, _SECRET, _PASSWORD, _CREDENTIALS, _ACCESS_KEY, _PRIVATE_KEY, etc. Plus an explicit list of credential names that don't fit the suffix pattern (AWS_ACCESS_KEY_ID, FAL_KEY, GH_TOKEN, etc.) and all the provider BASE_URL overrides that change auto-detect behavior. Also unsets HERMES_* behavioral vars (HERMES_YOLO_MODE, HERMES_QUIET, HERMES_SESSION_*, etc.) that mutate agent behavior. Also: - Redirects HOME to a per-test tempdir (not just HERMES_HOME), so code reading ~/.hermes/* directly can't touch the real dir. - Pins TZ=UTC, LANG=C.UTF-8, LC_ALL=C.UTF-8, PYTHONHASHSEED=0 to match CI's deterministic runtime. The old _isolate_hermes_home fixture name is preserved as an alias so any test that yields it explicitly still works. ## Layer 2 — scripts/run_tests.sh canonical runner 'Always use scripts/run_tests.sh, never call pytest directly' is the new rule (documented in AGENTS.md). The script: - Unsets all credential env vars (belt-and-suspenders for callers who bypass conftest — e.g. IDE integrations) - Pins TZ/LANG/PYTHONHASHSEED - Uses -n 4 xdist workers (matches GHA ubuntu-latest; -n auto on a 20-core workstation surfaces test-ordering flakes CI will never see, causing the infamous 'passes in CI, fails locally' drift) - Finds the venv in .venv, venv, or main checkout's venv - Passes through arbitrary pytest args Installs pytest-split on demand so the script can also be used to run matrix-split subsets locally for debugging. ## Remove 3 module-level dotenv stubs that broke test isolation tests/hermes_cli/test_{arcee,xiaomi,api_key}_provider.py each had a module-level: if 'dotenv' not in sys.modules: fake_dotenv = types.ModuleType('dotenv') fake_dotenv.load_dotenv = lambda *a, **kw: None sys.modules['dotenv'] = fake_dotenv This patches sys.modules['dotenv'] to a fake at import time with no teardown. Under pytest-xdist LoadScheduling, whichever worker collected one of these files first poisoned its sys.modules; subsequent tests in the same worker that imported load_dotenv transitively (e.g. test_env_loader.py via hermes_cli.env_loader) got the no-op lambda and saw their assertions fail. dotenv is a required dependency (python-dotenv>=1.2.1 in pyproject.toml), so the defensive stub was never needed. Removed. ## Validation - tests/hermes_cli/ alone: 2178 passed, 1 skipped, 0 failed (was 4 failures in test_env_loader.py before this fix) - tests/test_plugin_skills.py, tests/hermes_cli/test_plugins.py, tests/test_hermes_logging.py combined: 123 passed (the caplog regression tests from PR #11453 still pass) - Local full run shows no F/E clusters in the 0-55% range that were previously present before the conftest hardening ## Background See AGENTS.md 'Testing' section for the full list of drift sources this closes. Matrix split (closed as #11566) will be re-attempted once this foundation lands — cross-test pollution was the root cause of the shard-3 hang in that PR. * fix(conftest): don't redirect HOME — it broke CI subprocesses PR #11577's autouse fixture was setting HOME to a per-test tempdir. CI started timing out at 97% complete with dozens of E/F markers and orphan python processes at cleanup — tests (or transitive deps) spawn subprocesses that expect a stable HOME, and the redirect broke them in non-obvious ways. Env-var unsetting and TZ/LANG/hashseed pinning (the actual CI-drift fixes) are unchanged and still in place. HERMES_HOME redirection is also unchanged — that's the canonical way to isolate tests from ~/.hermes/, not HOME. Any code in the codebase reading ~/.hermes/* via `Path.home() / ".hermes"` instead of `get_hermes_home()` is a bug to fix at the callsite, not something to paper over in conftest.
2026-04-17 06:09:09 -07:00
test: use subprocesses for each test file (#29016) * ci(tests): install ripgrep from prebuilt tarball instead of apt apt-get update + install of ripgrep takes ~4 min on the GHA Ubuntu runners (the apt-get update against archive.ubuntu.com is the slow part; ripgrep itself is small). Switching to the upstream musl binary tarball cuts the step to a few seconds. - Pinned to ripgrep 15.1.0 with sha256 verification (same hash as published in the releases sha256 sidecar file). - Drops the `rg` binary into /usr/local/bin so it is on PATH for every subsequent step without GITHUB_PATH manipulation. - Applied to both the test and e2e jobs in tests.yml. * fix(cli): compile syntax check to tempdir, not source __pycache__ `_validate_critical_files_syntax` runs `py_compile.compile()` on each critical bootstrap file after a successful `git pull`. The default `py_compile` writes the resulting `.pyc` next to the source under `__pycache__/`, which causes two real problems: 1. Parallel test workers walking the same source tree (e.g. running the suite under per-file process isolation) can race against each other on the `__pycache__` write — manifests as flaky 'directory not empty' errors during teardown. 2. In production, the post-pull syntax check leaves a `.pyc` behind that the next interpreter run might pick up — fine when the interpreter version matches, sketchy if it doesn't. Fix: write the compiled output to a `tempfile.TemporaryDirectory()` that's discarded on function exit. We only care about the compile-or-not signal, not the artifact. * test(runner): per-file process isolation, drop manual state reset + xdist Replace fragile manual _reset_module_state test fixtures with robust per-file subprocess isolation. Each test file runs in a fresh `python -m pytest <file>` subprocess via ThreadPoolExecutor. No xdist, no custom pytest plugin, no shared worker state. Key changes: * scripts/run_tests_parallel.py — new runner: discovers test files, runs N in parallel via ThreadPoolExecutor, captures stdout per file, treats exit code 5 (no tests collected) as pass, kills all children on exit. Change from cpu_count to cpu_count*2. The runner is I/O-bound (waiting on subprocess.communicate() from pytest children) The parent process does almost no CPU work, so 2x oversubscription keeps more pipes full. When a file fails, immediately show the last 30 lines of pytest output (stack traces + FAILED summary) plus a ready-to-copy repro command: python -m pytest tests/agent/test_auxiliary_client.py * scripts/run_tests.sh — delegates to run_tests_parallel.py * .github/workflows/tests.yml — test step: python scripts/run_tests_parallel.py * pyproject.toml — drop pytest-xdist, pytest-split; simplify addopts * tests/conftest.py — remove ~200 lines of manual state-reset fixtures * AGENTS.md — update Testing section for per-file design * test(runner): speed gateway test antipattern scan up * fix(test): web search provider plugin test missing xai * fix(tests): make 14 test files pass under per-file subprocess isolation Tests that relied on cross-file state pollution from xdist workers fail when run in isolation (per-file subprocess model). Root causes and fixes: Tool registry not populated: - test_video_generation_tool_surface_matrix: add discover_builtin_tools() - test_web_providers_brave_free/ddgs/searxng/general: autouse fixtures registering all 8 bundled web providers, reset after each test - test_website_policy: same provider registration pattern - test_web_tools_tavily: same pattern across 3 dispatch test classes - Also add is_safe_url/check_website_access mocks where SSRF check blocks example.com (DNS resolution fails in isolated envs) Stale check_fn cache: - test_kanban_tools: invalidate_check_fn_cache() + _clear_tool_defs_cache() in both kanban guidance tests (prior test cached False for kanban_show) - test_discord_tool: cache invalidation in setup/teardown - test_homeassistant_tool: invalidate_check_fn_cache() before registry queries Module-level state pollution: - test_auxiliary_client: autouse fixture clearing _aux_unhealthy_until cache - test_skill_commands: set_session_vars() instead of patch.dict(os.environ) (ContextVar takes precedence over os.environ) - test_dm_topics: overwrite sys.modules + separate telegram.constants mock + force-reimport of gateway.platforms.telegram - test_terminal_tool_requirements: removed duplicate class declaration, autouse _clear_caches fixture * change(tests): run_tests.sh explicitly includes env vars instead of manually dropping some vars, now we just only include some * fix(tests): 5 more isolation/NixOS fixes - test_approval_plugin_hooks: isolate HERMES_HOME so real user's command_allowlist doesn't short-circuit the approval path - test_google_chat: skipif when Platform.GOOGLE_CHAT not in enum (feature not merged on this branch) - test_write_deny: test systemd prefix against tmp_path instead of /etc/systemd which resolves to /nix/store on NixOS - test_pty_bridge: use shutil.which('cat') instead of /bin/cat (doesn't exist on NixOS) - profiles.py: rmtree onexc handler chmod's parent dirs too, fixing profile deletion when copytree preserved read-only modes from nix store * fix(tests): clear unhealthy cache in autouse fixture for auxiliary_client * fix(tests): skip send_message when telegram not installed; handle missing worker_id in browser_supervisor * fix: py3.11 rmtree onexc compat + belt-and-suspenders unhealthy cache clear for expired codex test * fix: address PR #29016 review feedback - Remove tracked .pytest-cache/ artifact and add to .gitignore - Fix stale 'xdist worker' comment in conftest.py - Deduplicate web provider registration into tests/tools/conftest.py shared helper (register_all_web_providers), replacing 8 copy-pasted blocks across 6 test files - Update PR description: remove stale recovered-test-files claim, fix worker count to match code (cpu_count*2) * fix: eliminate race in stale-cache achievements test The background scan thread could complete and overwrite _SNAPSHOT_CACHE before evaluate_all() returned the stale data — only 10 fake sessions made the scan finish instantly. Added scan_delay param to _FakeSessionDB and set it to 2s in the stale-cache test so the background thread can't win the race.
2026-05-21 07:10:04 -04:00
exec env -i \
PATH="$PATH" \
HOME="$HOME" \
TZ=UTC \
LANG=C.UTF-8 \
LC_ALL=C.UTF-8 \
PYTHONHASHSEED=0 \
${EXTRA_PYTHONPATH:+PYTHONPATH="$EXTRA_PYTHONPATH"} \
${EXTRA_PYTEST_PLUGINS:+PYTEST_PLUGINS="$EXTRA_PYTEST_PLUGINS"} \
"$PYTHON" "$SCRIPT_DIR/run_tests_parallel.py" "$@"
Reference in a new issue Copy permalink