hermes-bsd/hermes_cli
annguyenNous 07424da76f fix(cron): keep ticker alive on BaseException + heartbeat-aware status
The in-process cron ticker (cron/scheduler_provider.py) caught only
`Exception` and logged at DEBUG, so a `SystemExit`/`KeyboardInterrupt`
raised from a misbehaving provider SDK or agent retry path killed the
ticker thread silently. The gateway PROCESS stayed up, so `hermes cron
status` — which only checks `find_gateway_pids()` — kept reporting
"✓ jobs will fire automatically" while no jobs ever fired (#32612,
#32895).

This makes ticker death survivable and detectable:

- The ticker loop now catches `BaseException` and logs at ERROR with a
  traceback, so a single bad tick no longer tears the thread down and
  the failure is visible in the gateway log.
- The loop records a heartbeat (`cron/ticker_heartbeat`, epoch seconds)
  on startup and after every tick — best-effort, never raised into the
  loop. Both ticker entry points (the gateway and the desktop fallback
  in web_server.py) funnel through `InProcessCronScheduler.start`, so one
  heartbeat site covers both.
- `hermes cron status` now reads the heartbeat age: if the gateway is
  running but the heartbeat is stale (> 200s, i.e. several missed ~60s
  ticks), it reports the ticker as STALLED and suggests a restart instead
  of falsely claiming jobs will fire. A missing heartbeat (older build /
  never ran) is treated as "unknown", not "dead".

Adds tests for BaseException survival, per-iteration heartbeat recording,
heartbeat round-trip/age, staleness detection, and silent-write-failure.

Salvaged from #49660 (BaseException survival on current structure),
extended with the heartbeat + honest-status reporting that the earlier
(pre-refactor) watchdog PRs #35616 and #33849 proposed.

Fixes #32612
Fixes #32895

Co-authored-by: banditburai <promptsiren@gmail.com>
Co-authored-by: sweetcornna <96944678+sweetcornna@users.noreply.github.com>
2026-06-21 13:00:50 +05:30
..
dashboard_auth fix(cron): serve /api/cron/fire on the dashboard app (hosted-agent surface) 2026-06-19 12:43:30 +10:00
proxy
subcommands feat(skills): find & diff user-modified bundled skills 2026-06-18 12:26:20 +05:30
__init__.py chore: release v0.17.0 (2026.6.19) 2026-06-19 12:38:31 -07:00
_parser.py fix(desktop): keep composer usable during reconnect (#45488) 2026-06-13 02:36:09 -07:00
_subprocess_compat.py
active_sessions.py feat(sessions): add optional max session cap 2026-06-08 15:12:12 -07:00
auth.py fix(spotify): quarantine dead tokens on terminal refresh failure 2026-06-20 23:23:47 -07:00
auth_commands.py fix(auth): add Codex OAuth accounts as distinct pool entries 2026-06-08 11:57:03 -07:00
azure_detect.py
backup.py fix(backup): exclude regeneratable dependency and cache dirs 2026-06-19 14:37:41 +05:30
banner.py fix(mcp): avoid false failed startup status 2026-06-11 09:01:52 -07:00
blueprint_cmd.py refactor(cron): rebrand Cron Recipes -> Automation Blueprints 2026-06-11 10:49:47 -07:00
browser_connect.py
build_info.py
bundles.py
callbacks.py
checkpoints.py
claw.py
cli_agent_setup_mixin.py fix(cli): publish agent ref to cli module so memory on_session_end fires on exit 2026-06-19 16:59:43 -07:00
cli_commands_mixin.py feat(delegation): async background subagents via delegate_task(background=true) (#40946) 2026-06-15 13:33:12 -07:00
cli_output.py
clipboard.py
codex_models.py
codex_runtime_plugin_migration.py
codex_runtime_switch.py
colors.py
commands.py fix(docs): align slash-command and docker docs 2026-06-20 23:23:47 -07:00
completion.py
config.py docs(compression): fix stale 'discarded' wording on in_place config flag 2026-06-20 10:57:07 -07:00
container_boot.py fix(docker): skip gateway reconciliation in dashboard container (autodetect) (#46293) 2026-06-15 20:51:48 +10:00
copilot_auth.py
cron.py fix(cron): keep ticker alive on BaseException + heartbeat-aware status 2026-06-21 13:00:50 +05:30
curator.py feat(curator): make skill consolidation opt-in (prune stays default-on) (#47840) 2026-06-17 05:20:32 -07:00
curses_ui.py
dashboard_register.py fix(cli): persist custom --portal-url to .env on dashboard register (#42435) 2026-06-09 13:56:33 +10:00
debug.py fix(debug): include gui.log (dashboard/TUI/pty/websocket) in hermes debug share 2026-06-19 07:05:42 -07:00
default_soul.py
dep_ensure.py
dingtalk_auth.py
doctor.py fix(doctor): audit WhatsApp bridge at its resolved (HERMES_HOME) dir (#49890) 2026-06-20 19:55:12 -07:00
dump.py fix(dump): show commit date instead of release date in hermes debug (#48104) 2026-06-17 16:53:42 -07:00
env_loader.py feat(managed-scope): apply managed .env last with override 2026-06-19 07:46:33 -07:00
fallback_cmd.py
fallback_config.py
gateway.py refactor(gateway): migrate slack/dingtalk/whatsapp/matrix/feishu/telegram/wecom/email/sms adapters to bundled plugins 2026-06-20 10:26:45 -07:00
gateway_enroll.py feat(relay): connector⇄gateway channel auth + signed-HTTP inbound receiver + enroll CLI (#48147) 2026-06-18 12:01:54 +10:00
gateway_windows.py fix(gateway): Windows restart no longer causes a silent outage 2026-06-19 06:31:56 -07:00
goals.py fix(compression): preserve goal, platform, and session indexing across rotation 2026-06-20 20:06:24 -07:00
gui_uninstall.py
hooks.py
inventory.py feat(model-picker): add Refresh Models control to bust stale model cache (#48691) 2026-06-18 21:37:41 -07:00
kanban.py
kanban_db.py fix(kanban): anchor no-path worktree tasks on board default_workdir 2026-06-20 19:12:23 -07:00
kanban_decompose.py
kanban_diagnostics.py
kanban_specify.py
kanban_swarm.py
logs.py
main.py fix(whatsapp): resolve bridge dir with HERMES_HOME mirror in Docker 2026-06-20 17:05:27 -07:00
managed_scope.py fix(managed-scope): honor managed scope in all standalone config loaders 2026-06-19 07:46:33 -07:00
managed_uv.py
mcp_catalog.py fix(mcp): block exfil-shaped stdio server configs (#46083) 2026-06-14 04:24:14 -07:00
mcp_config.py fix(mcp): block suspicious stdio configs before probe (#46112) 2026-06-14 04:46:54 -07:00
mcp_picker.py
mcp_security.py fix(mcp): block exfil-shaped stdio server configs (#46083) 2026-06-14 04:24:14 -07:00
mcp_startup.py fix(mcp): address adversarial review round 2 (stale-publish race, parity holes) 2026-06-19 11:57:43 -07:00
memory_providers.py fix(desktop): show Hindsight memory provider (#37546) 2026-06-18 16:48:47 -05:00
memory_setup.py feat(memory): improve OpenViking setup UX 2026-06-17 01:04:26 +08:00
middleware.py
migrate.py
model_catalog.py feat(models): seed model-catalog disk cache from checkout on update (#42614) 2026-06-08 22:31:06 -07:00
model_cost_guard.py fix(model): require confirmation for expensive model selections 2026-06-10 00:24:06 -07:00
model_normalize.py fix(gemini): strip native self prefixes before generateContent (#36141) 2026-06-13 13:47:08 -07:00
model_setup_flows.py fix(model): clear stale endpoint credentials across switches 2026-06-19 19:58:26 -07:00
model_switch.py fix(model): persist /model switch by default across sessions 2026-06-19 07:07:06 -07:00
models.py feat(xai): add grok-composer-2.5-fast to xAI OAuth model picker 2026-06-17 09:49:46 -07:00
nous_account.py feat(billing): /credits command — balance + portal top-up handoff (#44776) 2026-06-12 08:51:10 +00:00
nous_billing.py feat(billing): /billing terminal billing — interactive TUI + CLI client (#45449) 2026-06-19 01:53:32 +05:30
nous_subscription.py fix(whatsapp-cloud): review follow-ups for #43921 2026-06-11 07:51:01 -07:00
oneshot.py
pairing.py
partial_compress.py
platforms.py
plugins.py fix(desktop): keep composer usable during reconnect (#45488) 2026-06-13 02:36:09 -07:00
plugins_cmd.py fix(plugins): normalize browser-pasted GitHub repo URLs (#33539) 2026-06-13 13:23:59 -07:00
portal_cli.py
profile_describer.py
profile_distribution.py
profiles.py feat(gateway): multiplex phase 0 — config flag, profile enumeration, profile-stamped session keys 2026-06-19 07:34:15 -07:00
prompt_size.py
provider_catalog.py feat: unified provider_catalog() — one source for CLI picker and desktop tabs 2026-06-19 07:26:46 -07:00
providers.py fix(doctor): allow vendor slugs for named custom providers 2026-06-08 15:53:09 -07:00
psutil_android.py
pt_input_extras.py
pty_bridge.py fix(pty-bridge): mark os.killpg/getpgid windows-footgun-ok (POSIX-only module) 2026-06-08 07:03:12 -07:00
relaunch.py
runtime_provider.py feat(gateway): multiplex phase 2 — fail-closed profile credential isolation (Workstream A) 2026-06-19 07:34:15 -07:00
secret_prompt.py feat(memory): improve OpenViking setup UX 2026-06-17 01:04:26 +08:00
secrets_cli.py
security_advisories.py
security_audit.py
send_cmd.py fix(managed-scope): honor managed scope in config→env bridges too 2026-06-19 07:46:33 -07:00
service_manager.py fix(docker): supervised gateway uses --replace to take over stale holder (NS-505) (#47555) 2026-06-18 10:49:02 +10:00
session_listing.py fix: harden salvaged session and browser improvements 2026-06-15 07:46:34 -07:00
session_recap.py
setup.py docs(cli): fix broken terminal-backend guide link in setup wizard 2026-06-20 23:23:47 -07:00
setup_whatsapp_cloud.py fix(whatsapp-cloud): review follow-ups for #43921 2026-06-11 07:51:01 -07:00
skills_config.py fix(skills): apply global|platform disabled union to all resolution sites 2026-06-14 22:54:54 +05:30
skills_hub.py feat(skills): find & diff user-modified bundled skills 2026-06-18 12:26:20 +05:30
skin_engine.py
slack_cli.py
status.py Merge commit '6110aed9b' into feat/whatsapp-cloud-api 2026-06-10 21:39:22 -04:00
stdio.py
suggestions_cmd.py refactor(cron): rebrand Cron Recipes -> Automation Blueprints 2026-06-11 10:49:47 -07:00
telegram_managed_bot.py
timeouts.py
tips.py fix: make profile subprocess HOME policy explicit 2026-06-14 03:20:21 -07:00
tools_config.py refactor: remove agent-callable send_message tool (#47856) 2026-06-17 07:11:23 -07:00
uninstall.py
voice.py
web_server.py fix(model): clear stale endpoint credentials across switches 2026-06-19 19:58:26 -07:00
webhook.py
win_pty_bridge.py feat(windows): enable dashboard /chat tab via ConPTY (win_pty_bridge) + tests (#42251) 2026-06-08 11:32:43 -07:00
write_approval_commands.py refactor(memory,skills): replace tri-state write_mode with boolean write_approval (default off) (#43354) 2026-06-09 23:21:14 -07:00
xai_retirement.py