hermes-bsd/tests
Teknium ccd899318e
fix(cron): split scanner into two tiers so skill prose stops false-positiving (#32339)
The runtime cron prompt scanner (added in #3968 to plug the
"malicious skill carrying an injection payload" gap) reuses the same
critical-severity patterns as the create-time user-prompt scan against
the *assembled* prompt — which includes loaded skill markdown.

That works fine for narrow patterns like "ignore previous instructions"
which never legitimately appear in prose. It catastrophically false-
positives on command-shape patterns like `cat ~/.hermes/.env`,
`authorized_keys`, `/etc/sudoers`, and `rm -rf /`, which routinely
appear in security postmortems and runbooks as **descriptive prose**
about attacks, not as actual commands.

Concrete failure: the bundled `hermes-agent-dev` skill contains a
security postmortem section saying "the attacker could just
`cat ~/.hermes/.env`". Every PR-scout cron job that loaded this skill
was silently blocked with `Blocked: prompt matches threat pattern
'read_secrets'`. All 11 scout jobs failed for weeks.

Fix: split the scanner into two tiers and route by context:

  - `_scan_cron_prompt` (strict, unchanged behavior) runs against
    the small user-authored cron prompt at create/update and as a
    runtime defense-in-depth when no skills are attached. A legit
    user prompt has no business saying `cat .env`, so the strict
    patterns still apply there.

  - `_scan_cron_skill_assembled` (new, looser) runs against the
    assembled prompt when skills are attached. It only catches
    unambiguous prompt-injection directives ("ignore previous
    instructions", "disregard your rules", "system prompt override",
    "do not tell the user") plus invisible-unicode markers. Command-
    shape patterns are dropped because they false-positive on prose.

This is defense-in-depth, not the only line of defense. Skill bodies
are already scanned at install time by `skills_guard.py`; the runtime
cron scan exists purely as a tripwire for an obvious injection
directive surviving a malicious install. Catching prose mentions of
commands was never the goal of #3968 — the test that planted a skill
containing `cat ~/.hermes/.env` was the wrong shape of test for the
threat model.

Tests:
- `_scan_cron_prompt` strict behavior preserved (56 existing tests
  unchanged: bare `cat .env`, `rm -rf /`, etc. still block).
- New `TestScanCronSkillAssembled` class verifies the looser scanner:
  injection / disregard / system-override / do-not-tell-the-user /
  invisible-unicode still block; descriptive prose about attack
  commands is allowed; GitHub auth-header allowlist still works.
- `test_skill_with_env_exfil_payload_raises` (planted `cat .env`
  in skill body) replaced with `test_skill_with_env_exfil_command
  _in_prose_is_allowed` documenting the new correct behavior with
  the real-world postmortem-style example that triggered the bug.
- All 11 originally-failing PR-scout jobs validated end-to-end via
  `_build_job_prompt` — assembled prompts now build successfully
  with the `hermes-agent-dev` skill attached.

Total: 75/75 tests in cron + cronjob_tools + threat scanner pass;
544/544 across the wider cron / memory / threat-pattern surface.
2026-05-25 18:20:45 -07:00
..
acp test(acp): drop flaky runtime_calls[-1] tail-position assertion 2026-05-24 23:23:12 -07:00
acp_adapter
agent fix(anthropic): API-key path skips OAuth autodiscovery + prunes stale entries 2026-05-25 17:41:40 -07:00
cli feat(cli): show live background terminal-process count in status bar (#32061) 2026-05-25 05:35:02 -07:00
cron fix(cron): split scanner into two tiers so skill prose stops false-positiving (#32339) 2026-05-25 18:20:45 -07:00
docker test(docker): fix svstat 'want up' assertion in profile-gateway lifecycle test 2026-05-25 12:25:06 +10:00
e2e refactor(gateway): migrate Discord adapter to bundled plugin (full Teams parity) 2026-05-22 14:21:41 -07:00
fakes
gateway fix(gateway): coerce scalar model: to dict before /model --global persist (#32272) 2026-05-25 15:22:23 -07:00
hermes_cli fix(dashboard): suffix-allowlist plugin assets + denylist subprocess-influencing env vars (#32277) 2026-05-25 15:07:19 -07:00
hermes_state
honcho_plugin
integration refactor(gateway): migrate Discord adapter to bundled plugin (full Teams parity) 2026-05-22 14:21:41 -07:00
openviking_plugin
plugins feat(stt): add stt.providers.<name> command-provider registry 2026-05-25 01:41:19 -07:00
providers fix(custom): pass custom provider extra body 2026-05-21 07:48:53 -07:00
run_agent fix(credential-pool): correct pool rotation when weekly usage limit is reached 2026-05-25 06:32:30 -07:00
scripts
skills
stress
tools fix(cron): split scanner into two tiers so skill prose stops false-positiving (#32339) 2026-05-25 18:20:45 -07:00
tui_gateway
website
__init__.py
conftest.py test: isolate API server env in gateway tests 2026-05-25 14:54:02 -07:00
run_interrupt_test.py
test_account_usage.py
test_atomic_replace_symlinks.py
test_base_url_hostname.py
test_batch_runner_checkpoint.py
test_bitwarden_secrets.py perf(cli): cut hermes startup 63% — flip head-to-head vs codex (#31968) 2026-05-25 03:06:39 -07:00
test_cli_file_drop.py
test_cli_manual_compress.py
test_cli_skin_integration.py
test_ctx_halving_fix.py
test_empty_model_fallback.py
test_env_loader_secret_sources.py fix(secrets): only apply external secrets once per HERMES_HOME per process (#32271) 2026-05-25 15:18:55 -07:00
test_evidence_store.py
test_gateway_streaming_nested_config.py
test_get_tool_definitions_cache_isolation.py
test_hermes_bootstrap.py
test_hermes_constants.py fix(security): guard os.chmod(parent) against / and top-level dirs 2026-05-20 22:56:55 -07:00
test_hermes_home_profile_warning.py
test_hermes_logging.py
test_hermes_state.py fix(gateway): separate observed Telegram group context 2026-05-23 01:33:42 -07:00
test_hermes_state_wal_fallback.py
test_honcho_client_config.py
test_install_sh_browser_install.py
test_install_sh_pythonpath_sanitization.py
test_install_sh_setup_wizard_tty_probe.py
test_install_sh_symlink_stomp.py
test_install_sh_termux_network_prereqs.py
test_ipv4_preference.py
test_lazy_session_regressions.py
test_lint_config.py
test_live_system_guard_self_test.py
test_mcp_serve.py
test_mini_swe_runner.py
test_minimax_model_validation.py
test_minimax_oauth.py fix(minimax-oauth): refresh short-lived access tokens per request (#30619) 2026-05-22 15:16:15 -07:00
test_minisweagent_path.py
test_model_picker_scroll.py
test_model_tools.py
test_model_tools_async_bridge.py
test_ollama_num_ctx.py
test_package_json_lazy_deps.py
test_packaging_metadata.py
test_plugin_skills.py
test_process_loop_event_loop_warning.py
test_project_metadata.py
test_retry_utils.py
test_run_tests_parallel.py test: use subprocesses for each test file (#29016) 2026-05-21 16:40:04 +05:30
test_sanitize_tool_error.py
test_sql_injection.py
test_subprocess_home_isolation.py
test_termux_all_extra_compat.py
test_timezone.py
test_toolset_distributions.py
test_toolsets.py
test_trajectory_compressor.py
test_trajectory_compressor_async.py
test_transform_llm_output_hook.py
test_transform_tool_result_hook.py
test_tui_gateway_server.py fix(tui): stop slash dropdown from chopping last char of /goal (#31311) 2026-05-23 22:12:55 -07:00
test_utils_truthy_values.py
test_yuanbao_integration.py
test_yuanbao_markdown.py
test_yuanbao_pipeline.py
test_yuanbao_proto.py