diff --git a/.agent/skills/add-discord/SKILL.md b/.agent/skills/add-discord/SKILL.md new file mode 100644 index 0000000..96d6dd8 --- /dev/null +++ b/.agent/skills/add-discord/SKILL.md @@ -0,0 +1,230 @@ +--- +name: add-discord +description: Add Discord channel support to Clawdie. Use when the user wants to connect the agent to a Discord server. Covers bot setup, permissions, skill application, and verification. +--- + +# Add Discord Channel + +This skill adds Discord support to NanoClaw using the skills engine for deterministic code changes, then walks through interactive setup. + +## Phase 1: Pre-flight + +### Check if already applied + +Read `.nanoclaw/state.yaml`. If `discord` is in `applied_skills`, skip to Phase 3 (Setup). The code changes are already in place. + +### Ask the user + +Use `AskUserQuestion` to collect configuration: + +AskUserQuestion: Should Discord replace WhatsApp or run alongside it? + +- **Replace WhatsApp** - Discord will be the only channel (sets DISCORD_ONLY=true) +- **Alongside** - Both Discord and WhatsApp channels active + +AskUserQuestion: Do you have a Discord bot token, or do you need to create one? + +If they have one, collect it now. If not, we'll create one in Phase 3. + +## Phase 2: Apply Code Changes + +Run the skills engine to apply this skill's code package. The package files are in this directory alongside this SKILL.md. + +### Initialize skills system (if needed) + +If `.nanoclaw/` directory doesn't exist yet: + +```bash +npx tsx scripts/apply-skill.ts --init +``` + +Or call `initSkillsSystem()` from `skills-engine/migrate.ts`. + +### Apply the skill + +```bash +npx tsx scripts/apply-skill.ts .agent/skills/add-discord +``` + +This deterministically: + +- Adds `src/channels/discord.ts` (DiscordChannel class implementing Channel interface) +- Adds `src/channels/discord.test.ts` (unit tests with discord.js mock) +- Three-way merges Discord support into `src/index.ts` (multi-channel support, findChannel routing) +- Three-way merges Discord config into `src/config.ts` (DISCORD_BOT_TOKEN, DISCORD_ONLY exports) +- Three-way merges updated routing tests into `src/routing.test.ts` +- Installs the `discord.js` npm dependency +- Updates `.env.example` with `DISCORD_BOT_TOKEN` and `DISCORD_ONLY` +- Records the application in `.nanoclaw/state.yaml` + +If the apply reports merge conflicts, read the intent files: + +- `modify/src/index.ts.intent.md` — what changed and invariants for index.ts +- `modify/src/config.ts.intent.md` — what changed for config.ts + +### Validate code changes + +```bash +npm test +npm run build +``` + +All tests must pass (including the new Discord tests) and build must be clean before proceeding. + +## Phase 3: Setup + +### Create Discord Bot (if needed) + +If the user doesn't have a bot token, tell them: + +> I need you to create a Discord bot: +> +> 1. Go to the [Discord Developer Portal](https://discord.com/developers/applications) +> 2. Click **New Application** and give it a name (e.g., "Andy Assistant") +> 3. Go to the **Bot** tab on the left sidebar +> 4. Click **Reset Token** to generate a new bot token — copy it immediately (you can only see it once) +> 5. Under **Privileged Gateway Intents**, enable: +> - **Message Content Intent** (required to read message text) +> - **Server Members Intent** (optional, for member display names) +> 6. Go to **OAuth2** > **URL Generator**: +> - Scopes: select `bot` +> - Bot Permissions: select `Send Messages`, `Read Message History`, `View Channels` +> - Copy the generated URL and open it in your browser to invite the bot to your server + +Wait for the user to provide the token. + +### Configure environment + +Add to `.env`: + +```bash +DISCORD_BOT_TOKEN= +``` + +If they chose to replace WhatsApp: + +```bash +DISCORD_ONLY=true +``` + +Sync to container environment: + +```bash +cp .env data/env/env +``` + +The container reads environment from `data/env/env`, not `.env` directly. + +### Build and restart + +```bash +npm run build +launchctl kickstart -k gui/$(id -u)/com.nanoclaw +``` + +## Phase 4: Registration + +### Get Channel ID + +Tell the user: + +> To get the channel ID for registration: +> +> 1. In Discord, go to **User Settings** > **Advanced** > Enable **Developer Mode** +> 2. Right-click the text channel you want the bot to respond in +> 3. Click **Copy Channel ID** +> +> The channel ID will be a long number like `1234567890123456`. + +Wait for the user to provide the channel ID (format: `dc:1234567890123456`). + +### Register the channel + +Use the IPC register flow or register directly. The channel ID, name, and folder name are needed. + +For a main channel (responds to all messages, uses the `main` folder): + +```typescript +registerGroup('dc:', { + name: ' #', + folder: 'main', + trigger: `@${ASSISTANT_NAME}`, + added_at: new Date().toISOString(), + requiresTrigger: false, +}); +``` + +For additional channels (trigger-only): + +```typescript +registerGroup('dc:', { + name: ' #', + folder: '', + trigger: `@${ASSISTANT_NAME}`, + added_at: new Date().toISOString(), + requiresTrigger: true, +}); +``` + +## Phase 5: Verify + +### Test the connection + +Tell the user: + +> Send a message in your registered Discord channel: +> +> - For main channel: Any message works +> - For non-main: @mention the bot in Discord +> +> The bot should respond within a few seconds. + +### Check logs if needed + +```bash +tail -f logs/nanoclaw.log +``` + +## Troubleshooting + +### Bot not responding + +1. Check `DISCORD_BOT_TOKEN` is set in `.env` AND synced to `data/env/env` +2. Check channel is registered: `psql "$OPS_DB_URL" -c "SELECT * FROM registered_groups WHERE jid LIKE 'dc:%'"` +3. For non-main channels: message must include trigger pattern (@mention the bot) +4. Service is running: `launchctl list | grep nanoclaw` +5. Verify the bot has been invited to the server (check OAuth2 URL was used) + +### Bot only responds to @mentions + +This is the default behavior for non-main channels (`requiresTrigger: true`). To change: + +- Update the registered group's `requiresTrigger` to `false` +- Or register the channel as the main channel + +### Message Content Intent not enabled + +If the bot connects but can't read messages, ensure: + +1. Go to [Discord Developer Portal](https://discord.com/developers/applications) +2. Select your application > **Bot** tab +3. Under **Privileged Gateway Intents**, enable **Message Content Intent** +4. Restart NanoClaw + +### Getting Channel ID + +If you can't copy the channel ID: + +- Ensure **Developer Mode** is enabled: User Settings > Advanced > Developer Mode +- Right-click the channel name in the server sidebar > Copy Channel ID + +## After Setup + +The Discord bot supports: + +- Text messages in registered channels +- Attachment descriptions (images, videos, files shown as placeholders) +- Reply context (shows who the user is replying to) +- @mention translation (Discord `<@botId>` → NanoClaw trigger format) +- Message splitting for responses over 2000 characters +- Typing indicators while the agent processes diff --git a/.agent/skills/add-discord/add/src/channels/discord.test.ts b/.agent/skills/add-discord/add/src/channels/discord.test.ts new file mode 100644 index 0000000..eff0b77 --- /dev/null +++ b/.agent/skills/add-discord/add/src/channels/discord.test.ts @@ -0,0 +1,762 @@ +import { describe, it, expect, beforeEach, vi, afterEach } from 'vitest'; + +// --- Mocks --- + +// Mock config +vi.mock('../config.js', () => ({ + ASSISTANT_NAME: 'Andy', + TRIGGER_PATTERN: /^@Andy\b/i, +})); + +// Mock logger +vi.mock('../logger.js', () => ({ + logger: { + debug: vi.fn(), + info: vi.fn(), + warn: vi.fn(), + error: vi.fn(), + }, +})); + +// --- discord.js mock --- + +type Handler = (...args: any[]) => any; + +const clientRef = vi.hoisted(() => ({ current: null as any })); + +vi.mock('discord.js', () => { + const Events = { + MessageCreate: 'messageCreate', + ClientReady: 'ready', + Error: 'error', + }; + + const GatewayIntentBits = { + Guilds: 1, + GuildMessages: 2, + MessageContent: 4, + DirectMessages: 8, + }; + + class MockClient { + eventHandlers = new Map(); + user: any = { id: '999888777', tag: 'Andy#1234' }; + private _ready = false; + + constructor(_opts: any) { + clientRef.current = this; + } + + on(event: string, handler: Handler) { + const existing = this.eventHandlers.get(event) || []; + existing.push(handler); + this.eventHandlers.set(event, existing); + return this; + } + + once(event: string, handler: Handler) { + return this.on(event, handler); + } + + async login(_token: string) { + this._ready = true; + // Fire the ready event + const readyHandlers = this.eventHandlers.get('ready') || []; + for (const h of readyHandlers) { + h({ user: this.user }); + } + } + + isReady() { + return this._ready; + } + + channels = { + fetch: vi.fn().mockResolvedValue({ + send: vi.fn().mockResolvedValue(undefined), + sendTyping: vi.fn().mockResolvedValue(undefined), + }), + }; + + destroy() { + this._ready = false; + } + } + + // Mock TextChannel type + class TextChannel {} + + return { + Client: MockClient, + Events, + GatewayIntentBits, + TextChannel, + }; +}); + +import { DiscordChannel, DiscordChannelOpts } from './discord.js'; + +// --- Test helpers --- + +function createTestOpts( + overrides?: Partial, +): DiscordChannelOpts { + return { + onMessage: vi.fn(), + onChatMetadata: vi.fn(), + registeredGroups: vi.fn(() => ({ + 'dc:1234567890123456': { + name: 'Test Server #general', + folder: 'test-server', + trigger: '@Andy', + added_at: '2024-01-01T00:00:00.000Z', + }, + })), + ...overrides, + }; +} + +function createMessage(overrides: { + channelId?: string; + content?: string; + authorId?: string; + authorUsername?: string; + authorDisplayName?: string; + memberDisplayName?: string; + isBot?: boolean; + guildName?: string; + channelName?: string; + messageId?: string; + createdAt?: Date; + attachments?: Map; + reference?: { messageId?: string }; + mentionsBotId?: boolean; +}) { + const channelId = overrides.channelId ?? '1234567890123456'; + const authorId = overrides.authorId ?? '55512345'; + const botId = '999888777'; // matches mock client user id + + const mentionsMap = new Map(); + if (overrides.mentionsBotId) { + mentionsMap.set(botId, { id: botId }); + } + + return { + channelId, + id: overrides.messageId ?? 'msg_001', + content: overrides.content ?? 'Hello everyone', + createdAt: overrides.createdAt ?? new Date('2024-01-01T00:00:00.000Z'), + author: { + id: authorId, + username: overrides.authorUsername ?? 'alice', + displayName: overrides.authorDisplayName ?? 'Alice', + bot: overrides.isBot ?? false, + }, + member: overrides.memberDisplayName + ? { displayName: overrides.memberDisplayName } + : null, + guild: overrides.guildName + ? { name: overrides.guildName } + : null, + channel: { + name: overrides.channelName ?? 'general', + messages: { + fetch: vi.fn().mockResolvedValue({ + author: { username: 'Bob', displayName: 'Bob' }, + member: { displayName: 'Bob' }, + }), + }, + }, + mentions: { + users: mentionsMap, + }, + attachments: overrides.attachments ?? new Map(), + reference: overrides.reference ?? null, + }; +} + +function currentClient() { + return clientRef.current; +} + +async function triggerMessage(message: any) { + const handlers = currentClient().eventHandlers.get('messageCreate') || []; + for (const h of handlers) await h(message); +} + +// --- Tests --- + +describe('DiscordChannel', () => { + beforeEach(() => { + vi.clearAllMocks(); + }); + + afterEach(() => { + vi.restoreAllMocks(); + }); + + // --- Connection lifecycle --- + + describe('connection lifecycle', () => { + it('resolves connect() when client is ready', async () => { + const opts = createTestOpts(); + const channel = new DiscordChannel('test-token', opts); + + await channel.connect(); + + expect(channel.isConnected()).toBe(true); + }); + + it('registers message handlers on connect', async () => { + const opts = createTestOpts(); + const channel = new DiscordChannel('test-token', opts); + + await channel.connect(); + + expect(currentClient().eventHandlers.has('messageCreate')).toBe(true); + expect(currentClient().eventHandlers.has('error')).toBe(true); + expect(currentClient().eventHandlers.has('ready')).toBe(true); + }); + + it('disconnects cleanly', async () => { + const opts = createTestOpts(); + const channel = new DiscordChannel('test-token', opts); + + await channel.connect(); + expect(channel.isConnected()).toBe(true); + + await channel.disconnect(); + expect(channel.isConnected()).toBe(false); + }); + + it('isConnected() returns false before connect', () => { + const opts = createTestOpts(); + const channel = new DiscordChannel('test-token', opts); + + expect(channel.isConnected()).toBe(false); + }); + }); + + // --- Text message handling --- + + describe('text message handling', () => { + it('delivers message for registered channel', async () => { + const opts = createTestOpts(); + const channel = new DiscordChannel('test-token', opts); + await channel.connect(); + + const msg = createMessage({ + content: 'Hello everyone', + guildName: 'Test Server', + channelName: 'general', + }); + await triggerMessage(msg); + + expect(opts.onChatMetadata).toHaveBeenCalledWith( + 'dc:1234567890123456', + expect.any(String), + 'Test Server #general', + ); + expect(opts.onMessage).toHaveBeenCalledWith( + 'dc:1234567890123456', + expect.objectContaining({ + id: 'msg_001', + chat_jid: 'dc:1234567890123456', + sender: '55512345', + sender_name: 'Alice', + content: 'Hello everyone', + is_from_me: false, + }), + ); + }); + + it('only emits metadata for unregistered channels', async () => { + const opts = createTestOpts(); + const channel = new DiscordChannel('test-token', opts); + await channel.connect(); + + const msg = createMessage({ + channelId: '9999999999999999', + content: 'Unknown channel', + guildName: 'Other Server', + }); + await triggerMessage(msg); + + expect(opts.onChatMetadata).toHaveBeenCalledWith( + 'dc:9999999999999999', + expect.any(String), + expect.any(String), + ); + expect(opts.onMessage).not.toHaveBeenCalled(); + }); + + it('ignores bot messages', async () => { + const opts = createTestOpts(); + const channel = new DiscordChannel('test-token', opts); + await channel.connect(); + + const msg = createMessage({ isBot: true, content: 'I am a bot' }); + await triggerMessage(msg); + + expect(opts.onMessage).not.toHaveBeenCalled(); + expect(opts.onChatMetadata).not.toHaveBeenCalled(); + }); + + it('uses member displayName when available (server nickname)', async () => { + const opts = createTestOpts(); + const channel = new DiscordChannel('test-token', opts); + await channel.connect(); + + const msg = createMessage({ + content: 'Hi', + memberDisplayName: 'Alice Nickname', + authorDisplayName: 'Alice Global', + guildName: 'Server', + }); + await triggerMessage(msg); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'dc:1234567890123456', + expect.objectContaining({ sender_name: 'Alice Nickname' }), + ); + }); + + it('falls back to author displayName when no member', async () => { + const opts = createTestOpts(); + const channel = new DiscordChannel('test-token', opts); + await channel.connect(); + + const msg = createMessage({ + content: 'Hi', + memberDisplayName: undefined, + authorDisplayName: 'Alice Global', + guildName: 'Server', + }); + await triggerMessage(msg); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'dc:1234567890123456', + expect.objectContaining({ sender_name: 'Alice Global' }), + ); + }); + + it('uses sender name for DM chats (no guild)', async () => { + const opts = createTestOpts({ + registeredGroups: vi.fn(() => ({ + 'dc:1234567890123456': { + name: 'DM', + folder: 'dm', + trigger: '@Andy', + added_at: '2024-01-01T00:00:00.000Z', + }, + })), + }); + const channel = new DiscordChannel('test-token', opts); + await channel.connect(); + + const msg = createMessage({ + content: 'Hello', + guildName: undefined, + authorDisplayName: 'Alice', + }); + await triggerMessage(msg); + + expect(opts.onChatMetadata).toHaveBeenCalledWith( + 'dc:1234567890123456', + expect.any(String), + 'Alice', + ); + }); + + it('uses guild name + channel name for server messages', async () => { + const opts = createTestOpts(); + const channel = new DiscordChannel('test-token', opts); + await channel.connect(); + + const msg = createMessage({ + content: 'Hello', + guildName: 'My Server', + channelName: 'bot-chat', + }); + await triggerMessage(msg); + + expect(opts.onChatMetadata).toHaveBeenCalledWith( + 'dc:1234567890123456', + expect.any(String), + 'My Server #bot-chat', + ); + }); + }); + + // --- @mention translation --- + + describe('@mention translation', () => { + it('translates <@botId> mention to trigger format', async () => { + const opts = createTestOpts(); + const channel = new DiscordChannel('test-token', opts); + await channel.connect(); + + const msg = createMessage({ + content: '<@999888777> what time is it?', + mentionsBotId: true, + guildName: 'Server', + }); + await triggerMessage(msg); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'dc:1234567890123456', + expect.objectContaining({ + content: '@Andy what time is it?', + }), + ); + }); + + it('does not translate if message already matches trigger', async () => { + const opts = createTestOpts(); + const channel = new DiscordChannel('test-token', opts); + await channel.connect(); + + const msg = createMessage({ + content: '@Andy hello <@999888777>', + mentionsBotId: true, + guildName: 'Server', + }); + await triggerMessage(msg); + + // Should NOT prepend @Andy — already starts with trigger + // But the <@botId> should still be stripped + expect(opts.onMessage).toHaveBeenCalledWith( + 'dc:1234567890123456', + expect.objectContaining({ + content: '@Andy hello', + }), + ); + }); + + it('does not translate when bot is not mentioned', async () => { + const opts = createTestOpts(); + const channel = new DiscordChannel('test-token', opts); + await channel.connect(); + + const msg = createMessage({ + content: 'hello everyone', + guildName: 'Server', + }); + await triggerMessage(msg); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'dc:1234567890123456', + expect.objectContaining({ + content: 'hello everyone', + }), + ); + }); + + it('handles <@!botId> (nickname mention format)', async () => { + const opts = createTestOpts(); + const channel = new DiscordChannel('test-token', opts); + await channel.connect(); + + const msg = createMessage({ + content: '<@!999888777> check this', + mentionsBotId: true, + guildName: 'Server', + }); + await triggerMessage(msg); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'dc:1234567890123456', + expect.objectContaining({ + content: '@Andy check this', + }), + ); + }); + }); + + // --- Attachments --- + + describe('attachments', () => { + it('stores image attachment with placeholder', async () => { + const opts = createTestOpts(); + const channel = new DiscordChannel('test-token', opts); + await channel.connect(); + + const attachments = new Map([ + ['att1', { name: 'photo.png', contentType: 'image/png' }], + ]); + const msg = createMessage({ + content: '', + attachments, + guildName: 'Server', + }); + await triggerMessage(msg); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'dc:1234567890123456', + expect.objectContaining({ + content: '[Image: photo.png]', + }), + ); + }); + + it('stores video attachment with placeholder', async () => { + const opts = createTestOpts(); + const channel = new DiscordChannel('test-token', opts); + await channel.connect(); + + const attachments = new Map([ + ['att1', { name: 'clip.mp4', contentType: 'video/mp4' }], + ]); + const msg = createMessage({ + content: '', + attachments, + guildName: 'Server', + }); + await triggerMessage(msg); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'dc:1234567890123456', + expect.objectContaining({ + content: '[Video: clip.mp4]', + }), + ); + }); + + it('stores file attachment with placeholder', async () => { + const opts = createTestOpts(); + const channel = new DiscordChannel('test-token', opts); + await channel.connect(); + + const attachments = new Map([ + ['att1', { name: 'report.pdf', contentType: 'application/pdf' }], + ]); + const msg = createMessage({ + content: '', + attachments, + guildName: 'Server', + }); + await triggerMessage(msg); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'dc:1234567890123456', + expect.objectContaining({ + content: '[File: report.pdf]', + }), + ); + }); + + it('includes text content with attachments', async () => { + const opts = createTestOpts(); + const channel = new DiscordChannel('test-token', opts); + await channel.connect(); + + const attachments = new Map([ + ['att1', { name: 'photo.jpg', contentType: 'image/jpeg' }], + ]); + const msg = createMessage({ + content: 'Check this out', + attachments, + guildName: 'Server', + }); + await triggerMessage(msg); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'dc:1234567890123456', + expect.objectContaining({ + content: 'Check this out\n[Image: photo.jpg]', + }), + ); + }); + + it('handles multiple attachments', async () => { + const opts = createTestOpts(); + const channel = new DiscordChannel('test-token', opts); + await channel.connect(); + + const attachments = new Map([ + ['att1', { name: 'a.png', contentType: 'image/png' }], + ['att2', { name: 'b.txt', contentType: 'text/plain' }], + ]); + const msg = createMessage({ + content: '', + attachments, + guildName: 'Server', + }); + await triggerMessage(msg); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'dc:1234567890123456', + expect.objectContaining({ + content: '[Image: a.png]\n[File: b.txt]', + }), + ); + }); + }); + + // --- Reply context --- + + describe('reply context', () => { + it('includes reply author in content', async () => { + const opts = createTestOpts(); + const channel = new DiscordChannel('test-token', opts); + await channel.connect(); + + const msg = createMessage({ + content: 'I agree with that', + reference: { messageId: 'original_msg_id' }, + guildName: 'Server', + }); + await triggerMessage(msg); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'dc:1234567890123456', + expect.objectContaining({ + content: '[Reply to Bob] I agree with that', + }), + ); + }); + }); + + // --- sendMessage --- + + describe('sendMessage', () => { + it('sends message via channel', async () => { + const opts = createTestOpts(); + const channel = new DiscordChannel('test-token', opts); + await channel.connect(); + + await channel.sendMessage('dc:1234567890123456', 'Hello'); + + const fetchedChannel = await currentClient().channels.fetch('1234567890123456'); + expect(currentClient().channels.fetch).toHaveBeenCalledWith('1234567890123456'); + }); + + it('strips dc: prefix from JID', async () => { + const opts = createTestOpts(); + const channel = new DiscordChannel('test-token', opts); + await channel.connect(); + + await channel.sendMessage('dc:9876543210', 'Test'); + + expect(currentClient().channels.fetch).toHaveBeenCalledWith('9876543210'); + }); + + it('handles send failure gracefully', async () => { + const opts = createTestOpts(); + const channel = new DiscordChannel('test-token', opts); + await channel.connect(); + + currentClient().channels.fetch.mockRejectedValueOnce( + new Error('Channel not found'), + ); + + // Should not throw + await expect( + channel.sendMessage('dc:1234567890123456', 'Will fail'), + ).resolves.toBeUndefined(); + }); + + it('does nothing when client is not initialized', async () => { + const opts = createTestOpts(); + const channel = new DiscordChannel('test-token', opts); + + // Don't connect — client is null + await channel.sendMessage('dc:1234567890123456', 'No client'); + + // No error, no API call + }); + + it('splits messages exceeding 2000 characters', async () => { + const opts = createTestOpts(); + const channel = new DiscordChannel('test-token', opts); + await channel.connect(); + + const mockChannel = { + send: vi.fn().mockResolvedValue(undefined), + sendTyping: vi.fn(), + }; + currentClient().channels.fetch.mockResolvedValue(mockChannel); + + const longText = 'x'.repeat(3000); + await channel.sendMessage('dc:1234567890123456', longText); + + expect(mockChannel.send).toHaveBeenCalledTimes(2); + expect(mockChannel.send).toHaveBeenNthCalledWith(1, 'x'.repeat(2000)); + expect(mockChannel.send).toHaveBeenNthCalledWith(2, 'x'.repeat(1000)); + }); + }); + + // --- ownsJid --- + + describe('ownsJid', () => { + it('owns dc: JIDs', () => { + const channel = new DiscordChannel('test-token', createTestOpts()); + expect(channel.ownsJid('dc:1234567890123456')).toBe(true); + }); + + it('does not own WhatsApp group JIDs', () => { + const channel = new DiscordChannel('test-token', createTestOpts()); + expect(channel.ownsJid('12345@g.us')).toBe(false); + }); + + it('does not own Telegram JIDs', () => { + const channel = new DiscordChannel('test-token', createTestOpts()); + expect(channel.ownsJid('tg:123456789')).toBe(false); + }); + + it('does not own unknown JID formats', () => { + const channel = new DiscordChannel('test-token', createTestOpts()); + expect(channel.ownsJid('random-string')).toBe(false); + }); + }); + + // --- setTyping --- + + describe('setTyping', () => { + it('sends typing indicator when isTyping is true', async () => { + const opts = createTestOpts(); + const channel = new DiscordChannel('test-token', opts); + await channel.connect(); + + const mockChannel = { + send: vi.fn(), + sendTyping: vi.fn().mockResolvedValue(undefined), + }; + currentClient().channels.fetch.mockResolvedValue(mockChannel); + + await channel.setTyping('dc:1234567890123456', true); + + expect(mockChannel.sendTyping).toHaveBeenCalled(); + }); + + it('does nothing when isTyping is false', async () => { + const opts = createTestOpts(); + const channel = new DiscordChannel('test-token', opts); + await channel.connect(); + + await channel.setTyping('dc:1234567890123456', false); + + // channels.fetch should NOT be called + expect(currentClient().channels.fetch).not.toHaveBeenCalled(); + }); + + it('does nothing when client is not initialized', async () => { + const opts = createTestOpts(); + const channel = new DiscordChannel('test-token', opts); + + // Don't connect + await channel.setTyping('dc:1234567890123456', true); + + // No error + }); + }); + + // --- Channel properties --- + + describe('channel properties', () => { + it('has name "discord"', () => { + const channel = new DiscordChannel('test-token', createTestOpts()); + expect(channel.name).toBe('discord'); + }); + }); +}); diff --git a/.agent/skills/add-discord/add/src/channels/discord.ts b/.agent/skills/add-discord/add/src/channels/discord.ts new file mode 100644 index 0000000..997d489 --- /dev/null +++ b/.agent/skills/add-discord/add/src/channels/discord.ts @@ -0,0 +1,236 @@ +import { Client, Events, GatewayIntentBits, Message, TextChannel } from 'discord.js'; + +import { ASSISTANT_NAME, TRIGGER_PATTERN } from '../config.js'; +import { logger } from '../logger.js'; +import { + Channel, + OnChatMetadata, + OnInboundMessage, + RegisteredGroup, +} from '../types.js'; + +export interface DiscordChannelOpts { + onMessage: OnInboundMessage; + onChatMetadata: OnChatMetadata; + registeredGroups: () => Record; +} + +export class DiscordChannel implements Channel { + name = 'discord'; + + private client: Client | null = null; + private opts: DiscordChannelOpts; + private botToken: string; + + constructor(botToken: string, opts: DiscordChannelOpts) { + this.botToken = botToken; + this.opts = opts; + } + + async connect(): Promise { + this.client = new Client({ + intents: [ + GatewayIntentBits.Guilds, + GatewayIntentBits.GuildMessages, + GatewayIntentBits.MessageContent, + GatewayIntentBits.DirectMessages, + ], + }); + + this.client.on(Events.MessageCreate, async (message: Message) => { + // Ignore bot messages (including own) + if (message.author.bot) return; + + const channelId = message.channelId; + const chatJid = `dc:${channelId}`; + let content = message.content; + const timestamp = message.createdAt.toISOString(); + const senderName = + message.member?.displayName || + message.author.displayName || + message.author.username; + const sender = message.author.id; + const msgId = message.id; + + // Determine chat name + let chatName: string; + if (message.guild) { + const textChannel = message.channel as TextChannel; + chatName = `${message.guild.name} #${textChannel.name}`; + } else { + chatName = senderName; + } + + // Translate Discord @bot mentions into TRIGGER_PATTERN format. + // Discord mentions look like <@botUserId> — these won't match + // TRIGGER_PATTERN (e.g., ^@Andy\b), so we prepend the trigger + // when the bot is @mentioned. + if (this.client?.user) { + const botId = this.client.user.id; + const isBotMentioned = + message.mentions.users.has(botId) || + content.includes(`<@${botId}>`) || + content.includes(`<@!${botId}>`); + + if (isBotMentioned) { + // Strip the <@botId> mention to avoid visual clutter + content = content + .replace(new RegExp(`<@!?${botId}>`, 'g'), '') + .trim(); + // Prepend trigger if not already present + if (!TRIGGER_PATTERN.test(content)) { + content = `@${ASSISTANT_NAME} ${content}`; + } + } + } + + // Handle attachments — store placeholders so the agent knows something was sent + if (message.attachments.size > 0) { + const attachmentDescriptions = [...message.attachments.values()].map((att) => { + const contentType = att.contentType || ''; + if (contentType.startsWith('image/')) { + return `[Image: ${att.name || 'image'}]`; + } else if (contentType.startsWith('video/')) { + return `[Video: ${att.name || 'video'}]`; + } else if (contentType.startsWith('audio/')) { + return `[Audio: ${att.name || 'audio'}]`; + } else { + return `[File: ${att.name || 'file'}]`; + } + }); + if (content) { + content = `${content}\n${attachmentDescriptions.join('\n')}`; + } else { + content = attachmentDescriptions.join('\n'); + } + } + + // Handle reply context — include who the user is replying to + if (message.reference?.messageId) { + try { + const repliedTo = await message.channel.messages.fetch( + message.reference.messageId, + ); + const replyAuthor = + repliedTo.member?.displayName || + repliedTo.author.displayName || + repliedTo.author.username; + content = `[Reply to ${replyAuthor}] ${content}`; + } catch { + // Referenced message may have been deleted + } + } + + // Store chat metadata for discovery + this.opts.onChatMetadata(chatJid, timestamp, chatName); + + // Only deliver full message for registered groups + const group = this.opts.registeredGroups()[chatJid]; + if (!group) { + logger.debug( + { chatJid, chatName }, + 'Message from unregistered Discord channel', + ); + return; + } + + // Deliver message — startMessageLoop() will pick it up + this.opts.onMessage(chatJid, { + id: msgId, + chat_jid: chatJid, + sender, + sender_name: senderName, + content, + timestamp, + is_from_me: false, + }); + + logger.info( + { chatJid, chatName, sender: senderName }, + 'Discord message stored', + ); + }); + + // Handle errors gracefully + this.client.on(Events.Error, (err) => { + logger.error({ err: err.message }, 'Discord client error'); + }); + + return new Promise((resolve) => { + this.client!.once(Events.ClientReady, (readyClient) => { + logger.info( + { username: readyClient.user.tag, id: readyClient.user.id }, + 'Discord bot connected', + ); + console.log(`\n Discord bot: ${readyClient.user.tag}`); + console.log( + ` Use /chatid command or check channel IDs in Discord settings\n`, + ); + resolve(); + }); + + this.client!.login(this.botToken); + }); + } + + async sendMessage(jid: string, text: string): Promise { + if (!this.client) { + logger.warn('Discord client not initialized'); + return; + } + + try { + const channelId = jid.replace(/^dc:/, ''); + const channel = await this.client.channels.fetch(channelId); + + if (!channel || !('send' in channel)) { + logger.warn({ jid }, 'Discord channel not found or not text-based'); + return; + } + + const textChannel = channel as TextChannel; + + // Discord has a 2000 character limit per message — split if needed + const MAX_LENGTH = 2000; + if (text.length <= MAX_LENGTH) { + await textChannel.send(text); + } else { + for (let i = 0; i < text.length; i += MAX_LENGTH) { + await textChannel.send(text.slice(i, i + MAX_LENGTH)); + } + } + logger.info({ jid, length: text.length }, 'Discord message sent'); + } catch (err) { + logger.error({ jid, err }, 'Failed to send Discord message'); + } + } + + isConnected(): boolean { + return this.client !== null && this.client.isReady(); + } + + ownsJid(jid: string): boolean { + return jid.startsWith('dc:'); + } + + async disconnect(): Promise { + if (this.client) { + this.client.destroy(); + this.client = null; + logger.info('Discord bot stopped'); + } + } + + async setTyping(jid: string, isTyping: boolean): Promise { + if (!this.client || !isTyping) return; + try { + const channelId = jid.replace(/^dc:/, ''); + const channel = await this.client.channels.fetch(channelId); + if (channel && 'sendTyping' in channel) { + await (channel as TextChannel).sendTyping(); + } + } catch (err) { + logger.debug({ jid, err }, 'Failed to send Discord typing indicator'); + } + } +} diff --git a/.agent/skills/add-discord/manifest.yaml b/.agent/skills/add-discord/manifest.yaml new file mode 100644 index 0000000..f2cf2c8 --- /dev/null +++ b/.agent/skills/add-discord/manifest.yaml @@ -0,0 +1,20 @@ +skill: discord +version: 1.0.0 +description: "Discord Bot integration via discord.js" +core_version: 0.1.0 +adds: + - src/channels/discord.ts + - src/channels/discord.test.ts +modifies: + - src/index.ts + - src/config.ts + - src/routing.test.ts +structured: + npm_dependencies: + discord.js: "^14.18.0" + env_additions: + - DISCORD_BOT_TOKEN + - DISCORD_ONLY +conflicts: [] +depends: [] +test: "npx vitest run src/channels/discord.test.ts" diff --git a/.agent/skills/add-discord/modify/src/config.ts b/.agent/skills/add-discord/modify/src/config.ts new file mode 100644 index 0000000..41cc2ce --- /dev/null +++ b/.agent/skills/add-discord/modify/src/config.ts @@ -0,0 +1,77 @@ +import os from 'os'; +import path from 'path'; + +import { readEnvFile } from './env.js'; + +// Read config values from .env (falls back to process.env). +// Secrets are NOT read here — they stay on disk and are loaded only +// where needed (container-runner.ts) to avoid leaking to child processes. +const envConfig = readEnvFile([ + 'ASSISTANT_NAME', + 'ASSISTANT_HAS_OWN_NUMBER', + 'DISCORD_BOT_TOKEN', + 'DISCORD_ONLY', +]); + +export const ASSISTANT_NAME = + process.env.ASSISTANT_NAME || envConfig.ASSISTANT_NAME || 'Andy'; +export const ASSISTANT_HAS_OWN_NUMBER = + (process.env.ASSISTANT_HAS_OWN_NUMBER || envConfig.ASSISTANT_HAS_OWN_NUMBER) === 'true'; +export const POLL_INTERVAL = 2000; +export const SCHEDULER_POLL_INTERVAL = 60000; + +// Absolute paths needed for container mounts +const PROJECT_ROOT = process.cwd(); +const HOME_DIR = process.env.HOME || os.homedir(); + +// Mount security: allowlist stored OUTSIDE project root, never mounted into containers +export const MOUNT_ALLOWLIST_PATH = path.join( + HOME_DIR, + '.config', + (process.env.AGENT_NAME || 'clawdie') + '-cp', + 'mount-allowlist.json', +); +export const STORE_DIR = path.resolve(PROJECT_ROOT, 'store'); +export const GROUPS_DIR = path.resolve(PROJECT_ROOT, 'groups'); +export const DATA_DIR = path.resolve(PROJECT_ROOT, 'data'); +export const MAIN_GROUP_FOLDER = 'main'; + +export const CONTAINER_IMAGE = + process.env.CONTAINER_IMAGE || (process.env.AGENT_NAME || 'clawdie') + '-cp-agent:latest'; +export const CONTAINER_TIMEOUT = parseInt( + process.env.CONTAINER_TIMEOUT || '1800000', + 10, +); +export const CONTAINER_MAX_OUTPUT_SIZE = parseInt( + process.env.CONTAINER_MAX_OUTPUT_SIZE || '10485760', + 10, +); // 10MB default +export const IPC_POLL_INTERVAL = 1000; +export const IDLE_TIMEOUT = parseInt( + process.env.IDLE_TIMEOUT || '1800000', + 10, +); // 30min default — how long to keep container alive after last result +export const MAX_CONCURRENT_CONTAINERS = Math.max( + 1, + parseInt(process.env.MAX_CONCURRENT_CONTAINERS || '5', 10) || 5, +); + +function escapeRegex(str: string): string { + return str.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); +} + +export const TRIGGER_PATTERN = new RegExp( + `^@${escapeRegex(ASSISTANT_NAME)}\\b`, + 'i', +); + +// Timezone for scheduled tasks (cron expressions, etc.) +// Uses system timezone by default +export const TIMEZONE = + process.env.TZ || Intl.DateTimeFormat().resolvedOptions().timeZone; + +// Discord configuration +export const DISCORD_BOT_TOKEN = + process.env.DISCORD_BOT_TOKEN || envConfig.DISCORD_BOT_TOKEN || ''; +export const DISCORD_ONLY = + (process.env.DISCORD_ONLY || envConfig.DISCORD_ONLY) === 'true'; diff --git a/.agent/skills/add-discord/modify/src/config.ts.intent.md b/.agent/skills/add-discord/modify/src/config.ts.intent.md new file mode 100644 index 0000000..624d4c7 --- /dev/null +++ b/.agent/skills/add-discord/modify/src/config.ts.intent.md @@ -0,0 +1,25 @@ +# Intent: src/config.ts modifications + +## What changed + +Added two new configuration exports for Discord channel support. + +## Key sections + +- **readEnvFile call**: Must include `DISCORD_BOT_TOKEN` and `DISCORD_ONLY` in the keys array. NanoClaw does NOT load `.env` into `process.env` — all `.env` values must be explicitly requested via `readEnvFile()`. +- **DISCORD_BOT_TOKEN**: Read from `process.env` first, then `envConfig` fallback, defaults to empty string (channel disabled when empty) +- **DISCORD_ONLY**: Boolean flag from `process.env` or `envConfig`, when `true` disables WhatsApp channel creation + +## Invariants + +- All existing config exports remain unchanged +- New Discord keys are added to the `readEnvFile` call alongside existing keys +- New exports are appended at the end of the file +- No existing behavior is modified — Discord config is additive only +- Both `process.env` and `envConfig` are checked (same pattern as `ASSISTANT_NAME`) + +## Must-keep + +- All existing exports (`ASSISTANT_NAME`, `POLL_INTERVAL`, `TRIGGER_PATTERN`, etc.) +- The `readEnvFile` pattern — ALL config read from `.env` must go through this function +- The `escapeRegex` helper and `TRIGGER_PATTERN` construction diff --git a/.agent/skills/add-discord/modify/src/index.ts b/.agent/skills/add-discord/modify/src/index.ts new file mode 100644 index 0000000..4b6f30e --- /dev/null +++ b/.agent/skills/add-discord/modify/src/index.ts @@ -0,0 +1,509 @@ +import fs from 'fs'; +import path from 'path'; + +import { + ASSISTANT_NAME, + DISCORD_BOT_TOKEN, + DISCORD_ONLY, + IDLE_TIMEOUT, + MAIN_GROUP_FOLDER, + POLL_INTERVAL, + TRIGGER_PATTERN, +} from './config.js'; +import { DiscordChannel } from './channels/discord.js'; +import { WhatsAppChannel } from './channels/whatsapp.js'; +import { + ContainerOutput, + runContainerAgent, + writeGroupsSnapshot, + writeTasksSnapshot, +} from './container-runner.js'; +import { cleanupOrphans, ensureContainerRuntimeRunning } from './container-runtime.js'; +import { + getAllChats, + getAllRegisteredGroups, + getAllSessions, + getAllTasks, + getMessagesSince, + getNewMessages, + getRouterState, + initDatabase, + setRegisteredGroup, + setRouterState, + setSession, + storeChatMetadata, + storeMessage, +} from './db.js'; +import { GroupQueue } from './group-queue.js'; +import { resolveGroupFolderPath } from './group-folder.js'; +import { startIpcWatcher } from './ipc.js'; +import { findChannel, formatMessages, formatOutbound } from './router.js'; +import { startSchedulerLoop } from './task-scheduler.js'; +import { Channel, NewMessage, RegisteredGroup } from './types.js'; +import { logger } from './logger.js'; + +// Re-export for backwards compatibility during refactor +export { escapeXml, formatMessages } from './router.js'; + +let lastTimestamp = ''; +let sessions: Record = {}; +let registeredGroups: Record = {}; +let lastAgentTimestamp: Record = {}; +let messageLoopRunning = false; + +let whatsapp: WhatsAppChannel; +const channels: Channel[] = []; +const queue = new GroupQueue(); + +function loadState(): void { + lastTimestamp = getRouterState('last_timestamp') || ''; + const agentTs = getRouterState('last_agent_timestamp'); + try { + lastAgentTimestamp = agentTs ? JSON.parse(agentTs) : {}; + } catch { + logger.warn('Corrupted last_agent_timestamp in DB, resetting'); + lastAgentTimestamp = {}; + } + sessions = getAllSessions(); + registeredGroups = getAllRegisteredGroups(); + logger.info( + { groupCount: Object.keys(registeredGroups).length }, + 'State loaded', + ); +} + +function saveState(): void { + setRouterState('last_timestamp', lastTimestamp); + setRouterState( + 'last_agent_timestamp', + JSON.stringify(lastAgentTimestamp), + ); +} + +function registerGroup(jid: string, group: RegisteredGroup): void { + let groupDir: string; + try { + groupDir = resolveGroupFolderPath(group.folder); + } catch (err) { + logger.warn( + { jid, folder: group.folder, err }, + 'Rejecting group registration with invalid folder', + ); + return; + } + + registeredGroups[jid] = group; + setRegisteredGroup(jid, group); + + // Create group folder + fs.mkdirSync(path.join(groupDir, 'logs'), { recursive: true }); + + logger.info( + { jid, name: group.name, folder: group.folder }, + 'Group registered', + ); +} + +/** + * Get available groups list for the agent. + * Returns groups ordered by most recent activity. + */ +export function getAvailableGroups(): import('./container-runner.js').AvailableGroup[] { + const chats = getAllChats(); + const registeredJids = new Set(Object.keys(registeredGroups)); + + return chats + .filter((c) => c.jid !== '__group_sync__' && c.is_group) + .map((c) => ({ + jid: c.jid, + name: c.name, + lastActivity: c.last_message_time, + isRegistered: registeredJids.has(c.jid), + })); +} + +/** @internal - exported for testing */ +export function _setRegisteredGroups(groups: Record): void { + registeredGroups = groups; +} + +/** + * Process all pending messages for a group. + * Called by the GroupQueue when it's this group's turn. + */ +async function processGroupMessages(chatJid: string): Promise { + const group = registeredGroups[chatJid]; + if (!group) return true; + + const channel = findChannel(channels, chatJid); + if (!channel) { + console.log(`Warning: no channel owns JID ${chatJid}, skipping messages`); + return true; + } + + const isMainGroup = group.folder === MAIN_GROUP_FOLDER; + + const sinceTimestamp = lastAgentTimestamp[chatJid] || ''; + const missedMessages = getMessagesSince(chatJid, sinceTimestamp, ASSISTANT_NAME); + + if (missedMessages.length === 0) return true; + + // For non-main groups, check if trigger is required and present + if (!isMainGroup && group.requiresTrigger !== false) { + const hasTrigger = missedMessages.some((m) => + TRIGGER_PATTERN.test(m.content.trim()), + ); + if (!hasTrigger) return true; + } + + const prompt = formatMessages(missedMessages); + + // Advance cursor so the piping path in startMessageLoop won't re-fetch + // these messages. Save the old cursor so we can roll back on error. + const previousCursor = lastAgentTimestamp[chatJid] || ''; + lastAgentTimestamp[chatJid] = + missedMessages[missedMessages.length - 1].timestamp; + saveState(); + + logger.info( + { group: group.name, messageCount: missedMessages.length }, + 'Processing messages', + ); + + // Track idle timer for closing stdin when agent is idle + let idleTimer: ReturnType | null = null; + + const resetIdleTimer = () => { + if (idleTimer) clearTimeout(idleTimer); + idleTimer = setTimeout(() => { + logger.debug({ group: group.name }, 'Idle timeout, closing container stdin'); + queue.closeStdin(chatJid); + }, IDLE_TIMEOUT); + }; + + await channel.setTyping?.(chatJid, true); + let hadError = false; + let outputSentToUser = false; + + const output = await runAgent(group, prompt, chatJid, async (result) => { + // Streaming output callback — called for each agent result + if (result.result) { + const raw = typeof result.result === 'string' ? result.result : JSON.stringify(result.result); + // Strip ... blocks — agent uses these for internal reasoning + const text = raw.replace(/[\s\S]*?<\/internal>/g, '').trim(); + logger.info({ group: group.name }, `Agent output: ${raw.slice(0, 200)}`); + if (text) { + await channel.sendMessage(chatJid, text); + outputSentToUser = true; + } + // Only reset idle timer on actual results, not session-update markers (result: null) + resetIdleTimer(); + } + + if (result.status === 'success') { + queue.notifyIdle(chatJid); + } + + if (result.status === 'error') { + hadError = true; + } + }); + + await channel.setTyping?.(chatJid, false); + if (idleTimer) clearTimeout(idleTimer); + + if (output === 'error' || hadError) { + // If we already sent output to the user, don't roll back the cursor — + // the user got their response and re-processing would send duplicates. + if (outputSentToUser) { + logger.warn({ group: group.name }, 'Agent error after output was sent, skipping cursor rollback to prevent duplicates'); + return true; + } + // Roll back cursor so retries can re-process these messages + lastAgentTimestamp[chatJid] = previousCursor; + saveState(); + logger.warn({ group: group.name }, 'Agent error, rolled back message cursor for retry'); + return false; + } + + return true; +} + +async function runAgent( + group: RegisteredGroup, + prompt: string, + chatJid: string, + onOutput?: (output: ContainerOutput) => Promise, +): Promise<'success' | 'error'> { + const isMain = group.folder === MAIN_GROUP_FOLDER; + const sessionId = sessions[group.folder]; + + // Update tasks snapshot for container to read (filtered by group) + const tasks = getAllTasks(); + writeTasksSnapshot( + group.folder, + isMain, + tasks.map((t) => ({ + id: t.id, + groupFolder: t.group_folder, + prompt: t.prompt, + schedule_type: t.schedule_type, + schedule_value: t.schedule_value, + status: t.status, + next_run: t.next_run, + })), + ); + + // Update available groups snapshot (main group only can see all groups) + const availableGroups = getAvailableGroups(); + writeGroupsSnapshot( + group.folder, + isMain, + availableGroups, + new Set(Object.keys(registeredGroups)), + ); + + // Wrap onOutput to track session ID from streamed results + const wrappedOnOutput = onOutput + ? async (output: ContainerOutput) => { + if (output.newSessionId) { + sessions[group.folder] = output.newSessionId; + setSession(group.folder, output.newSessionId); + } + await onOutput(output); + } + : undefined; + + try { + const output = await runContainerAgent( + group, + { + prompt, + sessionId, + groupFolder: group.folder, + chatJid, + isMain, + assistantName: ASSISTANT_NAME, + }, + (proc, containerName) => queue.registerProcess(chatJid, proc, containerName, group.folder), + wrappedOnOutput, + ); + + if (output.newSessionId) { + sessions[group.folder] = output.newSessionId; + setSession(group.folder, output.newSessionId); + } + + if (output.status === 'error') { + logger.error( + { group: group.name, error: output.error }, + 'Container agent error', + ); + return 'error'; + } + + return 'success'; + } catch (err) { + logger.error({ group: group.name, err }, 'Agent error'); + return 'error'; + } +} + +async function startMessageLoop(): Promise { + if (messageLoopRunning) { + logger.debug('Message loop already running, skipping duplicate start'); + return; + } + messageLoopRunning = true; + + logger.info(`NanoClaw running (trigger: @${ASSISTANT_NAME})`); + + while (true) { + try { + const jids = Object.keys(registeredGroups); + const { messages, newTimestamp } = getNewMessages(jids, lastTimestamp, ASSISTANT_NAME); + + if (messages.length > 0) { + logger.info({ count: messages.length }, 'New messages'); + + // Advance the "seen" cursor for all messages immediately + lastTimestamp = newTimestamp; + saveState(); + + // Deduplicate by group + const messagesByGroup = new Map(); + for (const msg of messages) { + const existing = messagesByGroup.get(msg.chat_jid); + if (existing) { + existing.push(msg); + } else { + messagesByGroup.set(msg.chat_jid, [msg]); + } + } + + for (const [chatJid, groupMessages] of messagesByGroup) { + const group = registeredGroups[chatJid]; + if (!group) continue; + + const channel = findChannel(channels, chatJid); + if (!channel) { + console.log(`Warning: no channel owns JID ${chatJid}, skipping messages`); + continue; + } + + const isMainGroup = group.folder === MAIN_GROUP_FOLDER; + const needsTrigger = !isMainGroup && group.requiresTrigger !== false; + + // For non-main groups, only act on trigger messages. + // Non-trigger messages accumulate in DB and get pulled as + // context when a trigger eventually arrives. + if (needsTrigger) { + const hasTrigger = groupMessages.some((m) => + TRIGGER_PATTERN.test(m.content.trim()), + ); + if (!hasTrigger) continue; + } + + // Pull all messages since lastAgentTimestamp so non-trigger + // context that accumulated between triggers is included. + const allPending = getMessagesSince( + chatJid, + lastAgentTimestamp[chatJid] || '', + ASSISTANT_NAME, + ); + const messagesToSend = + allPending.length > 0 ? allPending : groupMessages; + const formatted = formatMessages(messagesToSend); + + if (queue.sendMessage(chatJid, formatted)) { + logger.debug( + { chatJid, count: messagesToSend.length }, + 'Piped messages to active container', + ); + lastAgentTimestamp[chatJid] = + messagesToSend[messagesToSend.length - 1].timestamp; + saveState(); + // Show typing indicator while the container processes the piped message + channel.setTyping?.(chatJid, true)?.catch((err) => + logger.warn({ chatJid, err }, 'Failed to set typing indicator'), + ); + } else { + // No active container — enqueue for a new one + queue.enqueueMessageCheck(chatJid); + } + } + } + } catch (err) { + logger.error({ err }, 'Error in message loop'); + } + await new Promise((resolve) => setTimeout(resolve, POLL_INTERVAL)); + } +} + +/** + * Startup recovery: check for unprocessed messages in registered groups. + * Handles crash between advancing lastTimestamp and processing messages. + */ +function recoverPendingMessages(): void { + for (const [chatJid, group] of Object.entries(registeredGroups)) { + const sinceTimestamp = lastAgentTimestamp[chatJid] || ''; + const pending = getMessagesSince(chatJid, sinceTimestamp, ASSISTANT_NAME); + if (pending.length > 0) { + logger.info( + { group: group.name, pendingCount: pending.length }, + 'Recovery: found unprocessed messages', + ); + queue.enqueueMessageCheck(chatJid); + } + } +} + +function ensureContainerSystemRunning(): void { + ensureContainerRuntimeRunning(); + cleanupOrphans(); +} + +async function main(): Promise { + ensureContainerSystemRunning(); + initDatabase(); + logger.info('Database initialized'); + loadState(); + + // Graceful shutdown handlers + const shutdown = async (signal: string) => { + logger.info({ signal }, 'Shutdown signal received'); + await queue.shutdown(10000); + for (const ch of channels) await ch.disconnect(); + process.exit(0); + }; + process.on('SIGTERM', () => shutdown('SIGTERM')); + process.on('SIGINT', () => shutdown('SIGINT')); + + // Channel callbacks (shared by all channels) + const channelOpts = { + onMessage: (_chatJid: string, msg: NewMessage) => storeMessage(msg), + onChatMetadata: (chatJid: string, timestamp: string, name?: string, channel?: string, isGroup?: boolean) => + storeChatMetadata(chatJid, timestamp, name, channel, isGroup), + registeredGroups: () => registeredGroups, + }; + + // Create and connect channels + if (DISCORD_BOT_TOKEN) { + const discord = new DiscordChannel(DISCORD_BOT_TOKEN, channelOpts); + channels.push(discord); + await discord.connect(); + } + + if (!DISCORD_ONLY) { + whatsapp = new WhatsAppChannel(channelOpts); + channels.push(whatsapp); + await whatsapp.connect(); + } + + // Start subsystems (independently of connection handler) + startSchedulerLoop({ + registeredGroups: () => registeredGroups, + getSessions: () => sessions, + queue, + onProcess: (groupJid, proc, containerName, groupFolder) => queue.registerProcess(groupJid, proc, containerName, groupFolder), + sendMessage: async (jid, rawText) => { + const channel = findChannel(channels, jid); + if (!channel) { + console.log(`Warning: no channel owns JID ${jid}, cannot send message`); + return; + } + const text = formatOutbound(rawText); + if (text) await channel.sendMessage(jid, text); + }, + }); + startIpcWatcher({ + sendMessage: (jid, text) => { + const channel = findChannel(channels, jid); + if (!channel) throw new Error(`No channel for JID: ${jid}`); + return channel.sendMessage(jid, text); + }, + registeredGroups: () => registeredGroups, + registerGroup, + syncGroupMetadata: (force) => whatsapp?.syncGroupMetadata(force) ?? Promise.resolve(), + getAvailableGroups, + writeGroupsSnapshot: (gf, im, ag, rj) => writeGroupsSnapshot(gf, im, ag, rj), + }); + queue.setProcessMessagesFn(processGroupMessages); + recoverPendingMessages(); + startMessageLoop().catch((err) => { + logger.fatal({ err }, 'Message loop crashed unexpectedly'); + process.exit(1); + }); +} + +// Guard: only run when executed directly, not when imported by tests +const isDirectRun = + process.argv[1] && + new URL(import.meta.url).pathname === new URL(`file://${process.argv[1]}`).pathname; + +if (isDirectRun) { + main().catch((err) => { + logger.error({ err }, 'Failed to start NanoClaw'); + process.exit(1); + }); +} diff --git a/.agent/skills/add-discord/modify/src/index.ts.intent.md b/.agent/skills/add-discord/modify/src/index.ts.intent.md new file mode 100644 index 0000000..29a8bf3 --- /dev/null +++ b/.agent/skills/add-discord/modify/src/index.ts.intent.md @@ -0,0 +1,50 @@ +# Intent: src/index.ts modifications + +## What changed + +Added Discord as a channel option alongside WhatsApp, introducing multi-channel infrastructure. + +## Key sections + +### Imports (top of file) + +- Added: `DiscordChannel` from `./channels/discord.js` +- Added: `DISCORD_BOT_TOKEN`, `DISCORD_ONLY` from `./config.js` +- Added: `findChannel` from `./router.js` +- Added: `Channel` from `./types.js` + +### Multi-channel infrastructure + +- Added: `const channels: Channel[] = []` array to hold all active channels +- Changed: `processGroupMessages` uses `findChannel(channels, chatJid)` instead of `whatsapp` directly +- Changed: `startMessageLoop` uses `findChannel(channels, chatJid)` instead of `whatsapp` directly +- Changed: `channel.setTyping?.()` instead of `whatsapp.setTyping()` +- Changed: `channel.sendMessage()` instead of `whatsapp.sendMessage()` + +### getAvailableGroups() + +- Unchanged: uses `c.is_group` filter from base (Discord channels pass `isGroup=true` via `onChatMetadata`) + +### main() + +- Added: `channelOpts` shared callback object for all channels +- Changed: WhatsApp conditional to `if (!DISCORD_ONLY)` +- Added: conditional Discord creation (`if (DISCORD_BOT_TOKEN)`) +- Changed: shutdown iterates `channels` array instead of just `whatsapp` +- Changed: subsystems use `findChannel(channels, jid)` for message routing + +## Invariants + +- All existing message processing logic (triggers, cursors, idle timers) is preserved +- The `runAgent` function is completely unchanged +- State management (loadState/saveState) is unchanged +- Recovery logic is unchanged +- Container runtime check is unchanged (ensureContainerSystemRunning) + +## Must-keep + +- The `escapeXml` and `formatMessages` re-exports +- The `_setRegisteredGroups` test helper +- The `isDirectRun` guard at bottom +- All error handling and cursor rollback logic in processGroupMessages +- The outgoing queue flush and reconnection logic (in WhatsAppChannel, not here) diff --git a/.agent/skills/add-discord/modify/src/routing.test.ts b/.agent/skills/add-discord/modify/src/routing.test.ts new file mode 100644 index 0000000..6144af0 --- /dev/null +++ b/.agent/skills/add-discord/modify/src/routing.test.ts @@ -0,0 +1,147 @@ +import { describe, it, expect, beforeEach } from 'vitest'; + +import { _initTestDatabase, getAllChats, storeChatMetadata } from './db.js'; +import { getAvailableGroups, _setRegisteredGroups } from './index.js'; + +beforeEach(() => { + _initTestDatabase(); + _setRegisteredGroups({}); +}); + +// --- JID ownership patterns --- + +describe('JID ownership patterns', () => { + // These test the patterns that will become ownsJid() on the Channel interface + + it('WhatsApp group JID: ends with @g.us', () => { + const jid = '12345678@g.us'; + expect(jid.endsWith('@g.us')).toBe(true); + }); + + it('Discord JID: starts with dc:', () => { + const jid = 'dc:1234567890123456'; + expect(jid.startsWith('dc:')).toBe(true); + }); + + it('WhatsApp DM JID: ends with @s.whatsapp.net', () => { + const jid = '12345678@s.whatsapp.net'; + expect(jid.endsWith('@s.whatsapp.net')).toBe(true); + }); +}); + +// --- getAvailableGroups --- + +describe('getAvailableGroups', () => { + it('returns only groups, excludes DMs', () => { + storeChatMetadata('group1@g.us', '2024-01-01T00:00:01.000Z', 'Group 1', 'whatsapp', true); + storeChatMetadata('user@s.whatsapp.net', '2024-01-01T00:00:02.000Z', 'User DM', 'whatsapp', false); + storeChatMetadata('group2@g.us', '2024-01-01T00:00:03.000Z', 'Group 2', 'whatsapp', true); + + const groups = getAvailableGroups(); + expect(groups).toHaveLength(2); + expect(groups.map((g) => g.jid)).toContain('group1@g.us'); + expect(groups.map((g) => g.jid)).toContain('group2@g.us'); + expect(groups.map((g) => g.jid)).not.toContain('user@s.whatsapp.net'); + }); + + it('includes Discord channel JIDs', () => { + storeChatMetadata('dc:1234567890123456', '2024-01-01T00:00:01.000Z', 'Discord Channel', 'discord', true); + storeChatMetadata('user@s.whatsapp.net', '2024-01-01T00:00:02.000Z', 'User DM', 'whatsapp', false); + + const groups = getAvailableGroups(); + expect(groups).toHaveLength(1); + expect(groups[0].jid).toBe('dc:1234567890123456'); + }); + + it('marks registered Discord channels correctly', () => { + storeChatMetadata('dc:1234567890123456', '2024-01-01T00:00:01.000Z', 'DC Registered', 'discord', true); + storeChatMetadata('dc:9999999999999999', '2024-01-01T00:00:02.000Z', 'DC Unregistered', 'discord', true); + + _setRegisteredGroups({ + 'dc:1234567890123456': { + name: 'DC Registered', + folder: 'dc-registered', + trigger: '@Andy', + added_at: '2024-01-01T00:00:00.000Z', + }, + }); + + const groups = getAvailableGroups(); + const dcReg = groups.find((g) => g.jid === 'dc:1234567890123456'); + const dcUnreg = groups.find((g) => g.jid === 'dc:9999999999999999'); + + expect(dcReg?.isRegistered).toBe(true); + expect(dcUnreg?.isRegistered).toBe(false); + }); + + it('excludes __group_sync__ sentinel', () => { + storeChatMetadata('__group_sync__', '2024-01-01T00:00:00.000Z'); + storeChatMetadata('group@g.us', '2024-01-01T00:00:01.000Z', 'Group', 'whatsapp', true); + + const groups = getAvailableGroups(); + expect(groups).toHaveLength(1); + expect(groups[0].jid).toBe('group@g.us'); + }); + + it('marks registered groups correctly', () => { + storeChatMetadata('reg@g.us', '2024-01-01T00:00:01.000Z', 'Registered', 'whatsapp', true); + storeChatMetadata('unreg@g.us', '2024-01-01T00:00:02.000Z', 'Unregistered', 'whatsapp', true); + + _setRegisteredGroups({ + 'reg@g.us': { + name: 'Registered', + folder: 'registered', + trigger: '@Andy', + added_at: '2024-01-01T00:00:00.000Z', + }, + }); + + const groups = getAvailableGroups(); + const reg = groups.find((g) => g.jid === 'reg@g.us'); + const unreg = groups.find((g) => g.jid === 'unreg@g.us'); + + expect(reg?.isRegistered).toBe(true); + expect(unreg?.isRegistered).toBe(false); + }); + + it('returns groups ordered by most recent activity', () => { + storeChatMetadata('old@g.us', '2024-01-01T00:00:01.000Z', 'Old', 'whatsapp', true); + storeChatMetadata('new@g.us', '2024-01-01T00:00:05.000Z', 'New', 'whatsapp', true); + storeChatMetadata('mid@g.us', '2024-01-01T00:00:03.000Z', 'Mid', 'whatsapp', true); + + const groups = getAvailableGroups(); + expect(groups[0].jid).toBe('new@g.us'); + expect(groups[1].jid).toBe('mid@g.us'); + expect(groups[2].jid).toBe('old@g.us'); + }); + + it('excludes non-group chats regardless of JID format', () => { + // Unknown JID format stored without is_group should not appear + storeChatMetadata('unknown-format-123', '2024-01-01T00:00:01.000Z', 'Unknown'); + // Explicitly non-group with unusual JID + storeChatMetadata('custom:abc', '2024-01-01T00:00:02.000Z', 'Custom DM', 'custom', false); + // A real group for contrast + storeChatMetadata('group@g.us', '2024-01-01T00:00:03.000Z', 'Group', 'whatsapp', true); + + const groups = getAvailableGroups(); + expect(groups).toHaveLength(1); + expect(groups[0].jid).toBe('group@g.us'); + }); + + it('returns empty array when no chats exist', () => { + const groups = getAvailableGroups(); + expect(groups).toHaveLength(0); + }); + + it('mixes WhatsApp and Discord chats ordered by activity', () => { + storeChatMetadata('wa@g.us', '2024-01-01T00:00:01.000Z', 'WhatsApp', 'whatsapp', true); + storeChatMetadata('dc:555', '2024-01-01T00:00:03.000Z', 'Discord', 'discord', true); + storeChatMetadata('wa2@g.us', '2024-01-01T00:00:02.000Z', 'WhatsApp 2', 'whatsapp', true); + + const groups = getAvailableGroups(); + expect(groups).toHaveLength(3); + expect(groups[0].jid).toBe('dc:555'); + expect(groups[1].jid).toBe('wa2@g.us'); + expect(groups[2].jid).toBe('wa@g.us'); + }); +}); diff --git a/.agent/skills/add-discord/tests/discord.test.ts b/.agent/skills/add-discord/tests/discord.test.ts new file mode 100644 index 0000000..a644aa7 --- /dev/null +++ b/.agent/skills/add-discord/tests/discord.test.ts @@ -0,0 +1,133 @@ +import { describe, expect, it } from 'vitest'; +import fs from 'fs'; +import path from 'path'; + +describe('discord skill package', () => { + const skillDir = path.resolve(__dirname, '..'); + + it('has a valid manifest', () => { + const manifestPath = path.join(skillDir, 'manifest.yaml'); + expect(fs.existsSync(manifestPath)).toBe(true); + + const content = fs.readFileSync(manifestPath, 'utf-8'); + expect(content).toContain('skill: discord'); + expect(content).toContain('version: 1.0.0'); + expect(content).toContain('discord.js'); + }); + + it('has all files declared in adds', () => { + const addFile = path.join(skillDir, 'add', 'src', 'channels', 'discord.ts'); + expect(fs.existsSync(addFile)).toBe(true); + + const content = fs.readFileSync(addFile, 'utf-8'); + expect(content).toContain('class DiscordChannel'); + expect(content).toContain('implements Channel'); + + // Test file for the channel + const testFile = path.join(skillDir, 'add', 'src', 'channels', 'discord.test.ts'); + expect(fs.existsSync(testFile)).toBe(true); + + const testContent = fs.readFileSync(testFile, 'utf-8'); + expect(testContent).toContain("describe('DiscordChannel'"); + }); + + it('has all files declared in modifies', () => { + const indexFile = path.join(skillDir, 'modify', 'src', 'index.ts'); + const configFile = path.join(skillDir, 'modify', 'src', 'config.ts'); + const routingTestFile = path.join(skillDir, 'modify', 'src', 'routing.test.ts'); + + expect(fs.existsSync(indexFile)).toBe(true); + expect(fs.existsSync(configFile)).toBe(true); + expect(fs.existsSync(routingTestFile)).toBe(true); + + const indexContent = fs.readFileSync(indexFile, 'utf-8'); + expect(indexContent).toContain('DiscordChannel'); + expect(indexContent).toContain('DISCORD_BOT_TOKEN'); + expect(indexContent).toContain('DISCORD_ONLY'); + expect(indexContent).toContain('findChannel'); + expect(indexContent).toContain('channels: Channel[]'); + + const configContent = fs.readFileSync(configFile, 'utf-8'); + expect(configContent).toContain('DISCORD_BOT_TOKEN'); + expect(configContent).toContain('DISCORD_ONLY'); + }); + + it('has intent files for modified files', () => { + expect(fs.existsSync(path.join(skillDir, 'modify', 'src', 'index.ts.intent.md'))).toBe(true); + expect(fs.existsSync(path.join(skillDir, 'modify', 'src', 'config.ts.intent.md'))).toBe(true); + }); + + it('modified index.ts preserves core structure', () => { + const content = fs.readFileSync( + path.join(skillDir, 'modify', 'src', 'index.ts'), + 'utf-8', + ); + + // Core functions still present + expect(content).toContain('function loadState()'); + expect(content).toContain('function saveState()'); + expect(content).toContain('function registerGroup('); + expect(content).toContain('function getAvailableGroups()'); + expect(content).toContain('function processGroupMessages('); + expect(content).toContain('function runAgent('); + expect(content).toContain('function startMessageLoop()'); + expect(content).toContain('function recoverPendingMessages()'); + expect(content).toContain('function ensureContainerSystemRunning()'); + expect(content).toContain('async function main()'); + + // Test helper preserved + expect(content).toContain('_setRegisteredGroups'); + + // Direct-run guard preserved + expect(content).toContain('isDirectRun'); + }); + + it('modified index.ts includes Discord channel creation', () => { + const content = fs.readFileSync( + path.join(skillDir, 'modify', 'src', 'index.ts'), + 'utf-8', + ); + + // Multi-channel architecture + expect(content).toContain('const channels: Channel[] = []'); + expect(content).toContain('channels.push(whatsapp)'); + expect(content).toContain('channels.push(discord)'); + + // Conditional channel creation + expect(content).toContain('if (!DISCORD_ONLY)'); + expect(content).toContain('if (DISCORD_BOT_TOKEN)'); + + // Shutdown disconnects all channels + expect(content).toContain('for (const ch of channels) await ch.disconnect()'); + }); + + it('modified config.ts preserves all existing exports', () => { + const content = fs.readFileSync( + path.join(skillDir, 'modify', 'src', 'config.ts'), + 'utf-8', + ); + + // All original exports preserved + expect(content).toContain('export const ASSISTANT_NAME'); + expect(content).toContain('export const POLL_INTERVAL'); + expect(content).toContain('export const TRIGGER_PATTERN'); + expect(content).toContain('export const CONTAINER_IMAGE'); + expect(content).toContain('export const DATA_DIR'); + expect(content).toContain('export const TIMEZONE'); + + // Discord exports added + expect(content).toContain('export const DISCORD_BOT_TOKEN'); + expect(content).toContain('export const DISCORD_ONLY'); + }); + + it('modified routing.test.ts includes Discord JID tests', () => { + const content = fs.readFileSync( + path.join(skillDir, 'modify', 'src', 'routing.test.ts'), + 'utf-8', + ); + + expect(content).toContain("Discord JID: starts with dc:"); + expect(content).toContain("dc:1234567890123456"); + expect(content).toContain("dc:"); + }); +}); diff --git a/.agent/skills/add-gmail/SKILL.md b/.agent/skills/add-gmail/SKILL.md new file mode 100644 index 0000000..cd2d25d --- /dev/null +++ b/.agent/skills/add-gmail/SKILL.md @@ -0,0 +1,244 @@ +--- +name: add-gmail +description: Add Gmail integration to NanoClaw. Can be configured as a tool (agent reads/sends emails when triggered from WhatsApp) or as a full channel (emails can trigger the agent, schedule tasks, and receive replies). Guides through GCP OAuth setup and implements the integration. +--- + +# Add Gmail Integration + +This skill adds Gmail support to NanoClaw — either as a tool (read, send, search, draft) or as a full channel that polls the inbox. + +## Phase 1: Pre-flight + +### Check if already applied + +Read `.nanoclaw/state.yaml`. If `gmail` is in `applied_skills`, skip to Phase 3 (Setup). The code changes are already in place. + +### Ask the user + +Use `AskUserQuestion`: + +AskUserQuestion: Should incoming emails be able to trigger the agent? + +- **Yes** — Full channel mode: the agent listens on Gmail and responds to incoming emails automatically +- **No** — Tool-only: the agent gets full Gmail tools (read, send, search, draft) but won't monitor the inbox. No channel code is added. + +## Phase 2: Apply Code Changes + +### Initialize skills system (if needed) + +If `.nanoclaw/` directory doesn't exist yet: + +```bash +npx tsx scripts/apply-skill.ts --init +``` + +### Path A: Tool-only (user chose "No") + +Do NOT run the full apply script. Only two source files need changes. This avoids adding dead code (`gmail.ts`, `gmail.test.ts`, index.ts channel logic, routing tests, `googleapis` dependency). + +#### 1. Mount Gmail credentials in container + +Apply the changes described in `modify/src/container-runner.ts.intent.md` to `src/container-runner.ts`: import `os`, add a conditional read-write mount of `~/.gmail-mcp` to `/home/node/.gmail-mcp` in `buildVolumeMounts()` after the session mounts. + +#### 2. Add Gmail MCP server to agent runner + +Apply the changes described in `modify/container/agent-runner/src/index.ts.intent.md` to `container/agent-runner/src/index.ts`: add `gmail` MCP server (`npx -y @gongrzhe/server-gmail-autoauth-mcp`) and `'mcp__gmail__*'` to `allowedTools`. + +#### 3. Record in state + +Add `gmail` to `.nanoclaw/state.yaml` under `applied_skills` with `mode: tool-only`. + +#### 4. Validate + +```bash +npm run build +``` + +Build must be clean before proceeding. Skip to Phase 3. + +### Path B: Channel mode (user chose "Yes") + +Run the full skills engine to apply all code changes: + +```bash +npx tsx scripts/apply-skill.ts .agent/skills/add-gmail +``` + +This deterministically: + +- Adds `src/channels/gmail.ts` (GmailChannel class implementing Channel interface) +- Adds `src/channels/gmail.test.ts` (unit tests) +- Three-way merges Gmail channel wiring into `src/index.ts` (GmailChannel creation) +- Three-way merges Gmail credentials mount into `src/container-runner.ts` (~/.gmail-mcp -> /home/node/.gmail-mcp) +- Three-way merges Gmail MCP server into `container/agent-runner/src/index.ts` (@gongrzhe/server-gmail-autoauth-mcp) +- Three-way merges Gmail JID tests into `src/routing.test.ts` +- Installs the `googleapis` npm dependency +- Records the application in `.nanoclaw/state.yaml` + +If the apply reports merge conflicts, read the intent files: + +- `modify/src/index.ts.intent.md` — what changed and invariants for index.ts +- `modify/src/container-runner.ts.intent.md` — what changed for container-runner.ts +- `modify/container/agent-runner/src/index.ts.intent.md` — what changed for agent-runner + +#### Add email handling instructions + +Append the following to `groups/main/AGENT.md` (before the formatting section): + +```markdown +## Email Notifications + +When you receive an email notification (messages starting with `[Email from ...`), inform the user about it but do NOT reply to the email unless specifically asked. You have Gmail tools available — use them only when the user explicitly asks you to reply, forward, or take action on an email. +``` + +#### Validate + +```bash +npm test +npm run build +``` + +All tests must pass (including the new gmail tests) and build must be clean before proceeding. + +## Phase 3: Setup + +### Check existing Gmail credentials + +```bash +ls -la ~/.gmail-mcp/ 2>/dev/null || echo "No Gmail config found" +``` + +If `credentials.json` already exists, skip to "Build and restart" below. + +### GCP Project Setup + +Tell the user: + +> I need you to set up Google Cloud OAuth credentials: +> +> 1. Open https://console.cloud.google.com — create a new project or select existing +> 2. Go to **APIs & Services > Library**, search "Gmail API", click **Enable** +> 3. Go to **APIs & Services > Credentials**, click **+ CREATE CREDENTIALS > OAuth client ID** +> - If prompted for consent screen: choose "External", fill in app name and email, save +> - Application type: **Desktop app**, name: anything (e.g., "NanoClaw Gmail") +> 4. Click **DOWNLOAD JSON** and save as `gcp-oauth.keys.json` +> +> Where did you save the file? (Give me the full path, or paste the file contents here) + +If user provides a path, copy it: + +```bash +mkdir -p ~/.gmail-mcp +cp "/path/user/provided/gcp-oauth.keys.json" ~/.gmail-mcp/gcp-oauth.keys.json +``` + +If user pastes JSON content, write it to `~/.gmail-mcp/gcp-oauth.keys.json`. + +### OAuth Authorization + +Tell the user: + +> I'm going to run Gmail authorization. A browser window will open — sign in and grant access. If you see an "app isn't verified" warning, click "Advanced" then "Go to [app name] (unsafe)" — this is normal for personal OAuth apps. + +Run the authorization: + +```bash +npx -y @gongrzhe/server-gmail-autoauth-mcp auth +``` + +If that fails (some versions don't have an auth subcommand), try `timeout 60 npx -y @gongrzhe/server-gmail-autoauth-mcp || true`. Verify with `ls ~/.gmail-mcp/credentials.json`. + +### Build and restart + +Clear stale per-group agent-runner copies (they only get re-created if missing, so existing copies won't pick up the new Gmail server): + +```bash +rm -r data/sessions/*/agent-runner-src 2>/dev/null || true +``` + +Rebuild the container (agent-runner changed): + +```bash +cd container && ./build.sh +``` + +Then compile and restart: + +```bash +npm run build +launchctl kickstart -k gui/$(id -u)/com.nanoclaw # macOS +# Linux: systemctl --user restart nanoclaw +``` + +## Phase 4: Verify + +### Test tool access (both modes) + +Tell the user: + +> Gmail is connected! Send this in your main channel: +> +> `@Andy check my recent emails` or `@Andy list my Gmail labels` + +### Test channel mode (Channel mode only) + +Tell the user to send themselves a test email. The agent should pick it up within a minute. Monitor: `tail -f logs/nanoclaw.log | grep -iE "(gmail|email)"`. + +Once verified, offer filter customization via `AskUserQuestion` — by default, only emails in the Primary inbox trigger the agent (Promotions, Social, Updates, and Forums are excluded). The user can keep this default or narrow further by sender, label, or keywords. No code changes needed for filters. + +### Check logs if needed + +```bash +tail -f logs/nanoclaw.log +``` + +## Troubleshooting + +### Gmail connection not responding + +Test directly: + +```bash +npx -y @gongrzhe/server-gmail-autoauth-mcp +``` + +### OAuth token expired + +Re-authorize: + +```bash +rm ~/.gmail-mcp/credentials.json +npx -y @gongrzhe/server-gmail-autoauth-mcp +``` + +### Container can't access Gmail + +- Verify `~/.gmail-mcp` is mounted: check `src/container-runner.ts` for the `.gmail-mcp` mount +- Check container logs: `cat groups/main/logs/container-*.log | tail -50` + +### Emails not being detected (Channel mode only) + +- By default, the channel polls unread Primary inbox emails (`is:unread category:primary`) +- Check logs for Gmail polling errors + +## Removal + +### Tool-only mode + +1. Remove `~/.gmail-mcp` mount from `src/container-runner.ts` +2. Remove `gmail` MCP server and `mcp__gmail__*` from `container/agent-runner/src/index.ts` +3. Remove `gmail` from `.nanoclaw/state.yaml` +4. Clear stale agent-runner copies: `rm -r data/sessions/*/agent-runner-src 2>/dev/null || true` +5. Rebuild: `cd container && ./build.sh && cd .. && npm run build && launchctl kickstart -k gui/$(id -u)/com.nanoclaw` (macOS) or `systemctl --user restart nanoclaw` (Linux) + +### Channel mode + +1. Delete `src/channels/gmail.ts` and `src/channels/gmail.test.ts` +2. Remove `GmailChannel` import and creation from `src/index.ts` +3. Remove `~/.gmail-mcp` mount from `src/container-runner.ts` +4. Remove `gmail` MCP server and `mcp__gmail__*` from `container/agent-runner/src/index.ts` +5. Remove Gmail JID tests from `src/routing.test.ts` +6. Uninstall: `npm uninstall googleapis` +7. Remove `gmail` from `.nanoclaw/state.yaml` +8. Clear stale agent-runner copies: `rm -r data/sessions/*/agent-runner-src 2>/dev/null || true` +9. Rebuild: `cd container && ./build.sh && cd .. && npm run build && launchctl kickstart -k gui/$(id -u)/com.nanoclaw` (macOS) or `systemctl --user restart nanoclaw` (Linux) diff --git a/.agent/skills/add-gmail/add/src/channels/gmail.test.ts b/.agent/skills/add-gmail/add/src/channels/gmail.test.ts new file mode 100644 index 0000000..52602dd --- /dev/null +++ b/.agent/skills/add-gmail/add/src/channels/gmail.test.ts @@ -0,0 +1,71 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest'; + +import { GmailChannel, GmailChannelOpts } from './gmail.js'; + +function makeOpts(overrides?: Partial): GmailChannelOpts { + return { + onMessage: vi.fn(), + onChatMetadata: vi.fn(), + registeredGroups: () => ({}), + ...overrides, + }; +} + +describe('GmailChannel', () => { + let channel: GmailChannel; + + beforeEach(() => { + channel = new GmailChannel(makeOpts()); + }); + + describe('ownsJid', () => { + it('returns true for gmail: prefixed JIDs', () => { + expect(channel.ownsJid('gmail:abc123')).toBe(true); + expect(channel.ownsJid('gmail:thread-id-456')).toBe(true); + }); + + it('returns false for non-gmail JIDs', () => { + expect(channel.ownsJid('12345@g.us')).toBe(false); + expect(channel.ownsJid('tg:123')).toBe(false); + expect(channel.ownsJid('dc:456')).toBe(false); + expect(channel.ownsJid('user@s.whatsapp.net')).toBe(false); + }); + }); + + describe('name', () => { + it('is gmail', () => { + expect(channel.name).toBe('gmail'); + }); + }); + + describe('isConnected', () => { + it('returns false before connect', () => { + expect(channel.isConnected()).toBe(false); + }); + }); + + describe('disconnect', () => { + it('sets connected to false', async () => { + await channel.disconnect(); + expect(channel.isConnected()).toBe(false); + }); + }); + + describe('constructor options', () => { + it('accepts custom poll interval', () => { + const ch = new GmailChannel(makeOpts(), 30000); + expect(ch.name).toBe('gmail'); + }); + + it('defaults to unread query when no filter configured', () => { + const ch = new GmailChannel(makeOpts()); + const query = (ch as unknown as { buildQuery: () => string }).buildQuery(); + expect(query).toBe('is:unread category:primary'); + }); + + it('defaults with no options provided', () => { + const ch = new GmailChannel(makeOpts()); + expect(ch.name).toBe('gmail'); + }); + }); +}); diff --git a/.agent/skills/add-gmail/add/src/channels/gmail.ts b/.agent/skills/add-gmail/add/src/channels/gmail.ts new file mode 100644 index 0000000..b9ade60 --- /dev/null +++ b/.agent/skills/add-gmail/add/src/channels/gmail.ts @@ -0,0 +1,339 @@ +import fs from 'fs'; +import os from 'os'; +import path from 'path'; + +import { google, gmail_v1 } from 'googleapis'; +import { OAuth2Client } from 'google-auth-library'; + +import { MAIN_GROUP_FOLDER } from '../config.js'; +import { logger } from '../logger.js'; +import { + Channel, + OnChatMetadata, + OnInboundMessage, + RegisteredGroup, +} from '../types.js'; + +export interface GmailChannelOpts { + onMessage: OnInboundMessage; + onChatMetadata: OnChatMetadata; + registeredGroups: () => Record; +} + +interface ThreadMeta { + sender: string; + senderName: string; + subject: string; + messageId: string; // RFC 2822 Message-ID for In-Reply-To +} + +export class GmailChannel implements Channel { + name = 'gmail'; + + private oauth2Client: OAuth2Client | null = null; + private gmail: gmail_v1.Gmail | null = null; + private opts: GmailChannelOpts; + private pollIntervalMs: number; + private pollTimer: ReturnType | null = null; + private processedIds = new Set(); + private threadMeta = new Map(); + private consecutiveErrors = 0; + private userEmail = ''; + + constructor(opts: GmailChannelOpts, pollIntervalMs = 60000) { + this.opts = opts; + this.pollIntervalMs = pollIntervalMs; + } + + async connect(): Promise { + const credDir = path.join(os.homedir(), '.gmail-mcp'); + const keysPath = path.join(credDir, 'gcp-oauth.keys.json'); + const tokensPath = path.join(credDir, 'credentials.json'); + + if (!fs.existsSync(keysPath) || !fs.existsSync(tokensPath)) { + logger.warn( + 'Gmail credentials not found in ~/.gmail-mcp/. Skipping Gmail channel. Run /add-gmail to set up.', + ); + return; + } + + const keys = JSON.parse(fs.readFileSync(keysPath, 'utf-8')); + const tokens = JSON.parse(fs.readFileSync(tokensPath, 'utf-8')); + + const clientConfig = keys.installed || keys.web || keys; + const { client_id, client_secret, redirect_uris } = clientConfig; + this.oauth2Client = new google.auth.OAuth2( + client_id, + client_secret, + redirect_uris?.[0], + ); + this.oauth2Client.setCredentials(tokens); + + // Persist refreshed tokens + this.oauth2Client.on('tokens', (newTokens) => { + try { + const current = JSON.parse(fs.readFileSync(tokensPath, 'utf-8')); + Object.assign(current, newTokens); + fs.writeFileSync(tokensPath, JSON.stringify(current, null, 2)); + logger.debug('Gmail OAuth tokens refreshed'); + } catch (err) { + logger.warn({ err }, 'Failed to persist refreshed Gmail tokens'); + } + }); + + this.gmail = google.gmail({ version: 'v1', auth: this.oauth2Client }); + + // Verify connection + const profile = await this.gmail.users.getProfile({ userId: 'me' }); + this.userEmail = profile.data.emailAddress || ''; + logger.info({ email: this.userEmail }, 'Gmail channel connected'); + + // Start polling with error backoff + const schedulePoll = () => { + const backoffMs = this.consecutiveErrors > 0 + ? Math.min(this.pollIntervalMs * Math.pow(2, this.consecutiveErrors), 30 * 60 * 1000) + : this.pollIntervalMs; + this.pollTimer = setTimeout(() => { + this.pollForMessages() + .catch((err) => logger.error({ err }, 'Gmail poll error')) + .finally(() => { + if (this.gmail) schedulePoll(); + }); + }, backoffMs); + }; + + // Initial poll + await this.pollForMessages(); + schedulePoll(); + } + + async sendMessage(jid: string, text: string): Promise { + if (!this.gmail) { + logger.warn('Gmail not initialized'); + return; + } + + const threadId = jid.replace(/^gmail:/, ''); + const meta = this.threadMeta.get(threadId); + + if (!meta) { + logger.warn({ jid }, 'No thread metadata for reply, cannot send'); + return; + } + + const subject = meta.subject.startsWith('Re:') + ? meta.subject + : `Re: ${meta.subject}`; + + const headers = [ + `To: ${meta.sender}`, + `From: ${this.userEmail}`, + `Subject: ${subject}`, + `In-Reply-To: ${meta.messageId}`, + `References: ${meta.messageId}`, + 'Content-Type: text/plain; charset=utf-8', + '', + text, + ].join('\r\n'); + + const encodedMessage = Buffer.from(headers) + .toString('base64') + .replace(/\+/g, '-') + .replace(/\//g, '_') + .replace(/=+$/, ''); + + try { + await this.gmail.users.messages.send({ + userId: 'me', + requestBody: { + raw: encodedMessage, + threadId, + }, + }); + logger.info({ to: meta.sender, threadId }, 'Gmail reply sent'); + } catch (err) { + logger.error({ jid, err }, 'Failed to send Gmail reply'); + } + } + + isConnected(): boolean { + return this.gmail !== null; + } + + ownsJid(jid: string): boolean { + return jid.startsWith('gmail:'); + } + + async disconnect(): Promise { + if (this.pollTimer) { + clearTimeout(this.pollTimer); + this.pollTimer = null; + } + this.gmail = null; + this.oauth2Client = null; + logger.info('Gmail channel stopped'); + } + + // --- Private --- + + private buildQuery(): string { + return 'is:unread category:primary'; + } + + private async pollForMessages(): Promise { + if (!this.gmail) return; + + try { + const query = this.buildQuery(); + const res = await this.gmail.users.messages.list({ + userId: 'me', + q: query, + maxResults: 10, + }); + + const messages = res.data.messages || []; + + for (const stub of messages) { + if (!stub.id || this.processedIds.has(stub.id)) continue; + this.processedIds.add(stub.id); + + await this.processMessage(stub.id); + } + + // Cap processed ID set to prevent unbounded growth + if (this.processedIds.size > 5000) { + const ids = [...this.processedIds]; + this.processedIds = new Set(ids.slice(ids.length - 2500)); + } + + this.consecutiveErrors = 0; + } catch (err) { + this.consecutiveErrors++; + const backoffMs = Math.min(this.pollIntervalMs * Math.pow(2, this.consecutiveErrors), 30 * 60 * 1000); + logger.error({ err, consecutiveErrors: this.consecutiveErrors, nextPollMs: backoffMs }, 'Gmail poll failed'); + } + } + + private async processMessage(messageId: string): Promise { + if (!this.gmail) return; + + const msg = await this.gmail.users.messages.get({ + userId: 'me', + id: messageId, + format: 'full', + }); + + const headers = msg.data.payload?.headers || []; + const getHeader = (name: string) => + headers.find((h) => h.name?.toLowerCase() === name.toLowerCase()) + ?.value || ''; + + const from = getHeader('From'); + const subject = getHeader('Subject'); + const rfc2822MessageId = getHeader('Message-ID'); + const threadId = msg.data.threadId || messageId; + const timestamp = new Date( + parseInt(msg.data.internalDate || '0', 10), + ).toISOString(); + + // Extract sender name and email + const senderMatch = from.match(/^(.+?)\s*<(.+?)>$/); + const senderName = senderMatch ? senderMatch[1].replace(/"/g, '') : from; + const senderEmail = senderMatch ? senderMatch[2] : from; + + // Skip emails from self (our own replies) + if (senderEmail === this.userEmail) return; + + // Extract body text + const body = this.extractTextBody(msg.data.payload); + + if (!body) { + logger.debug({ messageId, subject }, 'Skipping email with no text body'); + return; + } + + const chatJid = `gmail:${threadId}`; + + // Cache thread metadata for replies + this.threadMeta.set(threadId, { + sender: senderEmail, + senderName, + subject, + messageId: rfc2822MessageId, + }); + + // Store chat metadata for group discovery + this.opts.onChatMetadata(chatJid, timestamp, subject, 'gmail', false); + + // Find the main group to deliver the email notification + const groups = this.opts.registeredGroups(); + const mainEntry = Object.entries(groups).find( + ([, g]) => g.folder === MAIN_GROUP_FOLDER, + ); + + if (!mainEntry) { + logger.debug( + { chatJid, subject }, + 'No main group registered, skipping email', + ); + return; + } + + const mainJid = mainEntry[0]; + const content = `[Email from ${senderName} <${senderEmail}>]\nSubject: ${subject}\n\n${body}`; + + this.opts.onMessage(mainJid, { + id: messageId, + chat_jid: mainJid, + sender: senderEmail, + sender_name: senderName, + content, + timestamp, + is_from_me: false, + }); + + // Mark as read + try { + await this.gmail.users.messages.modify({ + userId: 'me', + id: messageId, + requestBody: { removeLabelIds: ['UNREAD'] }, + }); + } catch (err) { + logger.warn({ messageId, err }, 'Failed to mark email as read'); + } + + logger.info( + { mainJid, from: senderName, subject }, + 'Gmail email delivered to main group', + ); + } + + private extractTextBody( + payload: gmail_v1.Schema$MessagePart | undefined, + ): string { + if (!payload) return ''; + + // Direct text/plain body + if (payload.mimeType === 'text/plain' && payload.body?.data) { + return Buffer.from(payload.body.data, 'base64').toString('utf-8'); + } + + // Multipart: search parts recursively + if (payload.parts) { + // Prefer text/plain + for (const part of payload.parts) { + if (part.mimeType === 'text/plain' && part.body?.data) { + return Buffer.from(part.body.data, 'base64').toString('utf-8'); + } + } + // Recurse into nested multipart + for (const part of payload.parts) { + const text = this.extractTextBody(part); + if (text) return text; + } + } + + return ''; + } +} diff --git a/.agent/skills/add-gmail/manifest.yaml b/.agent/skills/add-gmail/manifest.yaml new file mode 100644 index 0000000..ea7c66a --- /dev/null +++ b/.agent/skills/add-gmail/manifest.yaml @@ -0,0 +1,18 @@ +skill: gmail +version: 1.0.0 +description: "Gmail integration via Google APIs" +core_version: 0.1.0 +adds: + - src/channels/gmail.ts + - src/channels/gmail.test.ts +modifies: + - src/index.ts + - src/container-runner.ts + - container/agent-runner/src/index.ts + - src/routing.test.ts +structured: + npm_dependencies: + googleapis: "^144.0.0" +conflicts: [] +depends: [] +test: "npx vitest run src/channels/gmail.test.ts" diff --git a/.agent/skills/add-gmail/modify/container/agent-runner/src/index.ts b/.agent/skills/add-gmail/modify/container/agent-runner/src/index.ts new file mode 100644 index 0000000..4ea26b0 --- /dev/null +++ b/.agent/skills/add-gmail/modify/container/agent-runner/src/index.ts @@ -0,0 +1,703 @@ +/** + * NanoClaw Agent Runner + * Runs inside a container, receives config via stdin, outputs result to stdout + * + * Input protocol: + * Stdin: Full ContainerInput JSON (read until EOF, like before) + * IPC: Follow-up messages written as JSON files to /workspace/ipc/input/ + * Files: {type:"message", text:"..."}.json — polled and consumed + * Sentinel: /workspace/ipc/input/_close — signals session end + * + * Stdout protocol: + * Each result is wrapped in OUTPUT_START_MARKER / OUTPUT_END_MARKER pairs. + * Multiple results may be emitted (one per agent teams result). + * Final marker after loop ends signals completion. + */ + +import fs from 'fs'; +import path from 'path'; +import { + query, + HookCallback, + PreCompactHookInput, + PreToolUseHookInput, +} from '@anthropic-ai/claude-agent-sdk'; +import { fileURLToPath } from 'url'; + +interface ContainerInput { + prompt: string; + sessionId?: string; + groupFolder: string; + chatJid: string; + isMain: boolean; + isScheduledTask?: boolean; + assistantName?: string; + secrets?: Record; +} + +interface ContainerOutput { + status: 'success' | 'error'; + result: string | null; + newSessionId?: string; + error?: string; +} + +interface SessionEntry { + sessionId: string; + fullPath: string; + summary: string; + firstPrompt: string; +} + +interface SessionsIndex { + entries: SessionEntry[]; +} + +interface SDKUserMessage { + type: 'user'; + message: { role: 'user'; content: string }; + parent_tool_use_id: null; + session_id: string; +} + +const IPC_INPUT_DIR = '/workspace/ipc/input'; +const IPC_INPUT_CLOSE_SENTINEL = path.join(IPC_INPUT_DIR, '_close'); +const IPC_POLL_MS = 500; + +/** + * Push-based async iterable for streaming user messages to the SDK. + * Keeps the iterable alive until end() is called, preventing isSingleUserTurn. + */ +class MessageStream { + private queue: SDKUserMessage[] = []; + private waiting: (() => void) | null = null; + private done = false; + + push(text: string): void { + this.queue.push({ + type: 'user', + message: { role: 'user', content: text }, + parent_tool_use_id: null, + session_id: '', + }); + this.waiting?.(); + } + + end(): void { + this.done = true; + this.waiting?.(); + } + + async *[Symbol.asyncIterator](): AsyncGenerator { + while (true) { + while (this.queue.length > 0) { + yield this.queue.shift()!; + } + if (this.done) return; + await new Promise((r) => { + this.waiting = r; + }); + this.waiting = null; + } + } +} + +async function readStdin(): Promise { + return new Promise((resolve, reject) => { + let data = ''; + process.stdin.setEncoding('utf8'); + process.stdin.on('data', (chunk) => { + data += chunk; + }); + process.stdin.on('end', () => resolve(data)); + process.stdin.on('error', reject); + }); +} + +const OUTPUT_START_MARKER = '---NANOCLAW_OUTPUT_START---'; +const OUTPUT_END_MARKER = '---NANOCLAW_OUTPUT_END---'; + +function writeOutput(output: ContainerOutput): void { + console.log(OUTPUT_START_MARKER); + console.log(JSON.stringify(output)); + console.log(OUTPUT_END_MARKER); +} + +function log(message: string): void { + console.error(`[agent-runner] ${message}`); +} + +function getSessionSummary( + sessionId: string, + transcriptPath: string, +): string | null { + const projectDir = path.dirname(transcriptPath); + const indexPath = path.join(projectDir, 'sessions-index.json'); + + if (!fs.existsSync(indexPath)) { + log(`Sessions index not found at ${indexPath}`); + return null; + } + + try { + const index: SessionsIndex = JSON.parse( + fs.readFileSync(indexPath, 'utf-8'), + ); + const entry = index.entries.find((e) => e.sessionId === sessionId); + if (entry?.summary) { + return entry.summary; + } + } catch (err) { + log( + `Failed to read sessions index: ${err instanceof Error ? err.message : String(err)}`, + ); + } + + return null; +} + +/** + * Archive the full transcript to conversations/ before compaction. + */ +function createPreCompactHook(assistantName?: string): HookCallback { + return async (input, _toolUseId, _context) => { + const preCompact = input as PreCompactHookInput; + const transcriptPath = preCompact.transcript_path; + const sessionId = preCompact.session_id; + + if (!transcriptPath || !fs.existsSync(transcriptPath)) { + log('No transcript found for archiving'); + return {}; + } + + try { + const content = fs.readFileSync(transcriptPath, 'utf-8'); + const messages = parseTranscript(content); + + if (messages.length === 0) { + log('No messages to archive'); + return {}; + } + + const summary = getSessionSummary(sessionId, transcriptPath); + const name = summary ? sanitizeFilename(summary) : generateFallbackName(); + + const conversationsDir = '/workspace/group/conversations'; + fs.mkdirSync(conversationsDir, { recursive: true }); + + const date = new Date().toISOString().split('T')[0]; + const filename = `${date}-${name}.md`; + const filePath = path.join(conversationsDir, filename); + + const markdown = formatTranscriptMarkdown( + messages, + summary, + assistantName, + ); + fs.writeFileSync(filePath, markdown); + + log(`Archived conversation to ${filePath}`); + } catch (err) { + log( + `Failed to archive transcript: ${err instanceof Error ? err.message : String(err)}`, + ); + } + + return {}; + }; +} + +// Secrets to strip from Bash tool subprocess environments. +// These are needed by the agent for API auth but should never +// be visible to commands Kit runs. +const SECRET_ENV_VARS = ['ANTHROPIC_API_KEY', 'ZAI_API_KEY']; + +function createSanitizeBashHook(): HookCallback { + return async (input, _toolUseId, _context) => { + const preInput = input as PreToolUseHookInput; + const command = (preInput.tool_input as { command?: string })?.command; + if (!command) return {}; + + const unsetPrefix = `unset ${SECRET_ENV_VARS.join(' ')} 2>/dev/null; `; + return { + hookSpecificOutput: { + hookEventName: 'PreToolUse', + updatedInput: { + ...(preInput.tool_input as Record), + command: unsetPrefix + command, + }, + }, + }; + }; +} + +function sanitizeFilename(summary: string): string { + return summary + .toLowerCase() + .replace(/[^a-z0-9]+/g, '-') + .replace(/^-+|-+$/g, '') + .slice(0, 50); +} + +function generateFallbackName(): string { + const time = new Date(); + return `conversation-${time.getHours().toString().padStart(2, '0')}${time.getMinutes().toString().padStart(2, '0')}`; +} + +interface ParsedMessage { + role: 'user' | 'assistant'; + content: string; +} + +function parseTranscript(content: string): ParsedMessage[] { + const messages: ParsedMessage[] = []; + + for (const line of content.split('\n')) { + if (!line.trim()) continue; + try { + const entry = JSON.parse(line); + if (entry.type === 'user' && entry.message?.content) { + const text = + typeof entry.message.content === 'string' + ? entry.message.content + : entry.message.content + .map((c: { text?: string }) => c.text || '') + .join(''); + if (text) messages.push({ role: 'user', content: text }); + } else if (entry.type === 'assistant' && entry.message?.content) { + const textParts = entry.message.content + .filter((c: { type: string }) => c.type === 'text') + .map((c: { text: string }) => c.text); + const text = textParts.join(''); + if (text) messages.push({ role: 'assistant', content: text }); + } + } catch {} + } + + return messages; +} + +function formatTranscriptMarkdown( + messages: ParsedMessage[], + title?: string | null, + assistantName?: string, +): string { + const now = new Date(); + const months = [ + 'jan', + 'feb', + 'mar', + 'apr', + 'maj', + 'jun', + 'jul', + 'avg', + 'sep', + 'okt', + 'nov', + 'dec', + ]; + const formatDateTime = (d: Date) => { + const day = String(d.getDate()).padStart(2, '0'); + const month = months[d.getMonth()]; + const year = d.getFullYear(); + const hour = String(d.getHours()).padStart(2, '0'); + const minute = String(d.getMinutes()).padStart(2, '0'); + return `${day}.${month}.${year} ${hour}:${minute}`; + }; + + const lines: string[] = []; + lines.push(`# ${title || 'Conversation'}`); + lines.push(''); + lines.push(`Archived: ${formatDateTime(now)}`); + lines.push(''); + lines.push('---'); + lines.push(''); + + for (const msg of messages) { + const sender = msg.role === 'user' ? 'User' : assistantName || 'Assistant'; + const content = + msg.content.length > 2000 + ? msg.content.slice(0, 2000) + '...' + : msg.content; + lines.push(`**${sender}**: ${content}`); + lines.push(''); + } + + return lines.join('\n'); +} + +/** + * Check for _close sentinel. + */ +function shouldClose(): boolean { + if (fs.existsSync(IPC_INPUT_CLOSE_SENTINEL)) { + try { + fs.unlinkSync(IPC_INPUT_CLOSE_SENTINEL); + } catch { + /* ignore */ + } + return true; + } + return false; +} + +/** + * Drain all pending IPC input messages. + * Returns messages found, or empty array. + */ +function drainIpcInput(): string[] { + try { + fs.mkdirSync(IPC_INPUT_DIR, { recursive: true }); + const files = fs + .readdirSync(IPC_INPUT_DIR) + .filter((f) => f.endsWith('.json')) + .sort(); + + const messages: string[] = []; + for (const file of files) { + const filePath = path.join(IPC_INPUT_DIR, file); + try { + const data = JSON.parse(fs.readFileSync(filePath, 'utf-8')); + fs.unlinkSync(filePath); + if (data.type === 'message' && data.text) { + messages.push(data.text); + } + } catch (err) { + log( + `Failed to process input file ${file}: ${err instanceof Error ? err.message : String(err)}`, + ); + try { + fs.unlinkSync(filePath); + } catch { + /* ignore */ + } + } + } + return messages; + } catch (err) { + log(`IPC drain error: ${err instanceof Error ? err.message : String(err)}`); + return []; + } +} + +/** + * Wait for a new IPC message or _close sentinel. + * Returns the messages as a single string, or null if _close. + */ +function waitForIpcMessage(): Promise { + return new Promise((resolve) => { + const poll = () => { + if (shouldClose()) { + resolve(null); + return; + } + const messages = drainIpcInput(); + if (messages.length > 0) { + resolve(messages.join('\n')); + return; + } + setTimeout(poll, IPC_POLL_MS); + }; + poll(); + }); +} + +/** + * Run a single query and stream results via writeOutput. + * Uses MessageStream (AsyncIterable) to keep isSingleUserTurn=false, + * allowing agent teams subagents to run to completion. + * Also pipes IPC messages into the stream during the query. + */ +async function runQuery( + prompt: string, + sessionId: string | undefined, + mcpServerPath: string, + containerInput: ContainerInput, + sdkEnv: Record, + resumeAt?: string, +): Promise<{ + newSessionId?: string; + lastAssistantUuid?: string; + closedDuringQuery: boolean; +}> { + const stream = new MessageStream(); + stream.push(prompt); + + // Poll IPC for follow-up messages and _close sentinel during the query + let ipcPolling = true; + let closedDuringQuery = false; + const pollIpcDuringQuery = () => { + if (!ipcPolling) return; + if (shouldClose()) { + log('Close sentinel detected during query, ending stream'); + closedDuringQuery = true; + stream.end(); + ipcPolling = false; + return; + } + const messages = drainIpcInput(); + for (const text of messages) { + log(`Piping IPC message into active query (${text.length} chars)`); + stream.push(text); + } + setTimeout(pollIpcDuringQuery, IPC_POLL_MS); + }; + setTimeout(pollIpcDuringQuery, IPC_POLL_MS); + + let newSessionId: string | undefined; + let lastAssistantUuid: string | undefined; + let messageCount = 0; + let resultCount = 0; + + // Load global AGENT.md as additional system context (shared across all groups) + const globalClaudeMdPath = '/workspace/global/AGENT.md'; + let globalClaudeMd: string | undefined; + if (!containerInput.isMain && fs.existsSync(globalClaudeMdPath)) { + globalClaudeMd = fs.readFileSync(globalClaudeMdPath, 'utf-8'); + } + + // Discover additional directories mounted at /workspace/extra/* + // These are passed to the SDK so their AGENT.md files are loaded automatically + const extraDirs: string[] = []; + const extraBase = '/workspace/extra'; + if (fs.existsSync(extraBase)) { + for (const entry of fs.readdirSync(extraBase)) { + const fullPath = path.join(extraBase, entry); + if (fs.statSync(fullPath).isDirectory()) { + extraDirs.push(fullPath); + } + } + } + if (extraDirs.length > 0) { + log(`Additional directories: ${extraDirs.join(', ')}`); + } + + for await (const message of query({ + prompt: stream, + options: { + cwd: '/workspace/group', + additionalDirectories: extraDirs.length > 0 ? extraDirs : undefined, + resume: sessionId, + resumeSessionAt: resumeAt, + systemPrompt: globalClaudeMd + ? { + type: 'preset' as const, + preset: 'claude_code' as const, + append: globalClaudeMd, + } + : undefined, + allowedTools: [ + 'Bash', + 'Read', + 'Write', + 'Edit', + 'Glob', + 'Grep', + 'WebSearch', + 'WebFetch', + 'Task', + 'TaskOutput', + 'TaskStop', + 'TeamCreate', + 'TeamDelete', + 'SendMessage', + 'TodoWrite', + 'ToolSearch', + 'Skill', + 'NotebookEdit', + 'mcp__nanoclaw__*', + 'mcp__gmail__*', + ], + env: sdkEnv, + permissionMode: 'bypassPermissions', + allowDangerouslySkipPermissions: true, + settingSources: ['project', 'user'], + mcpServers: { + nanoclaw: { + command: 'node', + args: [mcpServerPath], + env: { + NANOCLAW_CHAT_JID: containerInput.chatJid, + NANOCLAW_GROUP_FOLDER: containerInput.groupFolder, + NANOCLAW_IS_MAIN: containerInput.isMain ? '1' : '0', + }, + }, + gmail: { + command: 'npx', + args: ['-y', '@gongrzhe/server-gmail-autoauth-mcp'], + }, + }, + hooks: { + PreCompact: [ + { hooks: [createPreCompactHook(containerInput.assistantName)] }, + ], + PreToolUse: [{ matcher: 'Bash', hooks: [createSanitizeBashHook()] }], + }, + }, + })) { + messageCount++; + const msgType = + message.type === 'system' + ? `system/${(message as { subtype?: string }).subtype}` + : message.type; + log(`[msg #${messageCount}] type=${msgType}`); + + if (message.type === 'assistant' && 'uuid' in message) { + lastAssistantUuid = (message as { uuid: string }).uuid; + } + + if (message.type === 'system' && message.subtype === 'init') { + newSessionId = message.session_id; + log(`Session initialized: ${newSessionId}`); + } + + if ( + message.type === 'system' && + (message as { subtype?: string }).subtype === 'task_notification' + ) { + const tn = message as { + task_id: string; + status: string; + summary: string; + }; + log( + `Task notification: task=${tn.task_id} status=${tn.status} summary=${tn.summary}`, + ); + } + + if (message.type === 'result') { + resultCount++; + const textResult = + 'result' in message ? (message as { result?: string }).result : null; + log( + `Result #${resultCount}: subtype=${message.subtype}${textResult ? ` text=${textResult.slice(0, 200)}` : ''}`, + ); + writeOutput({ + status: 'success', + result: textResult || null, + newSessionId, + }); + } + } + + ipcPolling = false; + log( + `Query done. Messages: ${messageCount}, results: ${resultCount}, lastAssistantUuid: ${lastAssistantUuid || 'none'}, closedDuringQuery: ${closedDuringQuery}`, + ); + return { newSessionId, lastAssistantUuid, closedDuringQuery }; +} + +async function main(): Promise { + let containerInput: ContainerInput; + + try { + const stdinData = await readStdin(); + containerInput = JSON.parse(stdinData); + // Delete the temp file the entrypoint wrote — it contains secrets + try { + fs.unlinkSync('/workspace/ipc/input.json'); + } catch { + /* may not exist */ + } + log(`Received input for group: ${containerInput.groupFolder}`); + } catch (err) { + writeOutput({ + status: 'error', + result: null, + error: `Failed to parse input: ${err instanceof Error ? err.message : String(err)}`, + }); + process.exit(1); + } + + // Build SDK env: merge secrets into process.env for the SDK only. + // Secrets never touch process.env itself, so Bash subprocesses can't see them. + const sdkEnv: Record = { ...process.env }; + for (const [key, value] of Object.entries(containerInput.secrets || {})) { + sdkEnv[key] = value; + } + + const __dirname = path.dirname(fileURLToPath(import.meta.url)); + const mcpServerPath = path.join(__dirname, 'ipc-mcp-stdio.js'); + + let sessionId = containerInput.sessionId; + fs.mkdirSync(IPC_INPUT_DIR, { recursive: true }); + + // Clean up stale _close sentinel from previous container runs + try { + fs.unlinkSync(IPC_INPUT_CLOSE_SENTINEL); + } catch { + /* ignore */ + } + + // Build initial prompt (drain any pending IPC messages too) + let prompt = containerInput.prompt; + if (containerInput.isScheduledTask) { + prompt = `[SCHEDULED TASK - The following message was sent automatically and is not coming directly from the user or group.]\n\n${prompt}`; + } + const pending = drainIpcInput(); + if (pending.length > 0) { + log(`Draining ${pending.length} pending IPC messages into initial prompt`); + prompt += '\n' + pending.join('\n'); + } + + // Query loop: run query → wait for IPC message → run new query → repeat + let resumeAt: string | undefined; + try { + while (true) { + log( + `Starting query (session: ${sessionId || 'new'}, resumeAt: ${resumeAt || 'latest'})...`, + ); + + const queryResult = await runQuery( + prompt, + sessionId, + mcpServerPath, + containerInput, + sdkEnv, + resumeAt, + ); + if (queryResult.newSessionId) { + sessionId = queryResult.newSessionId; + } + if (queryResult.lastAssistantUuid) { + resumeAt = queryResult.lastAssistantUuid; + } + + // If _close was consumed during the query, exit immediately. + // Don't emit a session-update marker (it would reset the host's + // idle timer and cause a 30-min delay before the next _close). + if (queryResult.closedDuringQuery) { + log('Close sentinel consumed during query, exiting'); + break; + } + + // Emit session update so host can track it + writeOutput({ status: 'success', result: null, newSessionId: sessionId }); + + log('Query ended, waiting for next IPC message...'); + + // Wait for the next message or _close sentinel + const nextMessage = await waitForIpcMessage(); + if (nextMessage === null) { + log('Close sentinel received, exiting'); + break; + } + + log(`Got new message (${nextMessage.length} chars), starting new query`); + prompt = nextMessage; + } + } catch (err) { + const errorMessage = err instanceof Error ? err.message : String(err); + log(`Agent error: ${errorMessage}`); + writeOutput({ + status: 'error', + result: null, + newSessionId: sessionId, + error: errorMessage, + }); + process.exit(1); + } +} + +main(); diff --git a/.agent/skills/add-gmail/modify/container/agent-runner/src/index.ts.intent.md b/.agent/skills/add-gmail/modify/container/agent-runner/src/index.ts.intent.md new file mode 100644 index 0000000..dc5ff4a --- /dev/null +++ b/.agent/skills/add-gmail/modify/container/agent-runner/src/index.ts.intent.md @@ -0,0 +1,37 @@ +# Intent: container/agent-runner/src/index.ts modifications + +## What changed + +Added Gmail MCP server to the agent's available tools so it can read and send emails. + +## Key sections + +### mcpServers (inside runQuery → query() call) + +- Added: `gmail` MCP server alongside the existing `nanoclaw` server: + ``` + gmail: { + command: 'npx', + args: ['-y', '@gongrzhe/server-gmail-autoauth-mcp'], + }, + ``` + +### allowedTools (inside runQuery → query() call) + +- Added: `'mcp__gmail__*'` to allow all Gmail MCP tools + +## Invariants + +- The `nanoclaw` MCP server configuration is unchanged +- All existing allowed tools are preserved +- The query loop, IPC handling, MessageStream, and all other logic is untouched +- Hooks (PreCompact, sanitize Bash) are unchanged +- Output protocol (markers) is unchanged + +## Must-keep + +- The `nanoclaw` MCP server with its environment variables +- All existing allowedTools entries +- The hook system (PreCompact, PreToolUse sanitize) +- The IPC input/close sentinel handling +- The MessageStream class and query loop diff --git a/.agent/skills/add-gmail/modify/src/container-runner.ts b/.agent/skills/add-gmail/modify/src/container-runner.ts new file mode 100644 index 0000000..88573de --- /dev/null +++ b/.agent/skills/add-gmail/modify/src/container-runner.ts @@ -0,0 +1,698 @@ +/** + * Container Runner for NanoClaw + * Spawns agent execution in containers and handles IPC + */ +import { ChildProcess, exec, spawn } from 'child_process'; +import fs from 'fs'; +import os from 'os'; +import path from 'path'; + +import { + CONTAINER_IMAGE, + CONTAINER_MAX_OUTPUT_SIZE, + CONTAINER_TIMEOUT, + DATA_DIR, + GROUPS_DIR, + IDLE_TIMEOUT, + TIMEZONE, +} from './config.js'; +import { readEnvFile } from './env.js'; +import { resolveGroupFolderPath, resolveGroupIpcPath } from './group-folder.js'; +import { logger } from './logger.js'; +import { + CONTAINER_RUNTIME_BIN, + readonlyMountArgs, + stopContainer, +} from './container-runtime.js'; +import { validateAdditionalMounts } from './mount-security.js'; +import { RegisteredGroup } from './types.js'; + +// Sentinel markers for robust output parsing (must match agent-runner) +const OUTPUT_START_MARKER = '---NANOCLAW_OUTPUT_START---'; +const OUTPUT_END_MARKER = '---NANOCLAW_OUTPUT_END---'; + +export interface ContainerInput { + prompt: string; + sessionId?: string; + groupFolder: string; + chatJid: string; + isMain: boolean; + isScheduledTask?: boolean; + assistantName?: string; + secrets?: Record; +} + +export interface ContainerOutput { + status: 'success' | 'error'; + result: string | null; + newSessionId?: string; + error?: string; +} + +interface VolumeMount { + hostPath: string; + containerPath: string; + readonly: boolean; +} + +function buildVolumeMounts( + group: RegisteredGroup, + isMain: boolean, +): VolumeMount[] { + const mounts: VolumeMount[] = []; + const projectRoot = process.cwd(); + const homeDir = os.homedir(); + const groupDir = resolveGroupFolderPath(group.folder); + + if (isMain) { + // Main gets the project root read-only. Writable paths the agent needs + // (group folder, IPC, .agent/) are mounted separately below. + // Read-only prevents the agent from modifying host application code + // (src/, dist/, package.json, etc.) which would bypass the sandbox + // entirely on next restart. + mounts.push({ + hostPath: projectRoot, + containerPath: '/workspace/project', + readonly: true, + }); + + // Main also gets its group folder as the working directory + mounts.push({ + hostPath: groupDir, + containerPath: '/workspace/group', + readonly: false, + }); + } else { + // Other groups only get their own folder + mounts.push({ + hostPath: groupDir, + containerPath: '/workspace/group', + readonly: false, + }); + + // Global memory directory (read-only for non-main) + // Only directory mounts are supported, not file mounts + const globalDir = path.join(GROUPS_DIR, 'global'); + if (fs.existsSync(globalDir)) { + mounts.push({ + hostPath: globalDir, + containerPath: '/workspace/global', + readonly: true, + }); + } + } + + // Per-group Claude sessions directory (isolated from other groups) + // Each group gets their own .agent/ to prevent cross-group session access + const groupSessionsDir = path.join( + DATA_DIR, + 'sessions', + group.folder, + '.agent', + ); + fs.mkdirSync(groupSessionsDir, { recursive: true }); + const settingsFile = path.join(groupSessionsDir, 'settings.json'); + if (!fs.existsSync(settingsFile)) { + fs.writeFileSync( + settingsFile, + JSON.stringify( + { + env: { + // Enable agent swarms (subagent orchestration) + // https://code.agent.com/docs/en/agent-teams#orchestrate-teams-of-claude-code-sessions + CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS: '1', + // Load AGENT.md from additional mounted directories + // https://code.agent.com/docs/en/memory#load-memory-from-additional-directories + CLAUDE_CODE_ADDITIONAL_DIRECTORIES_CLAUDE_MD: '1', + // Enable Claude's memory feature (persists user preferences between sessions) + // https://code.agent.com/docs/en/memory#manage-auto-memory + CLAUDE_CODE_DISABLE_AUTO_MEMORY: '0', + }, + }, + null, + 2, + ) + '\n', + ); + } + + // Sync skills from container/skills/ into each group's .agent/skills/ + const skillsSrc = path.join(process.cwd(), 'container', 'skills'); + const skillsDst = path.join(groupSessionsDir, 'skills'); + if (fs.existsSync(skillsSrc)) { + for (const skillDir of fs.readdirSync(skillsSrc)) { + const srcDir = path.join(skillsSrc, skillDir); + if (!fs.statSync(srcDir).isDirectory()) continue; + const dstDir = path.join(skillsDst, skillDir); + fs.cpSync(srcDir, dstDir, { recursive: true }); + } + } + mounts.push({ + hostPath: groupSessionsDir, + containerPath: '/home/node/.agent', + readonly: false, + }); + + // Gmail credentials directory (for Gmail MCP inside the container) + const gmailDir = path.join(homeDir, '.gmail-mcp'); + if (fs.existsSync(gmailDir)) { + mounts.push({ + hostPath: gmailDir, + containerPath: '/home/node/.gmail-mcp', + readonly: false, // MCP may need to refresh OAuth tokens + }); + } + + // Per-group IPC namespace: each group gets its own IPC directory + // This prevents cross-group privilege escalation via IPC + const groupIpcDir = resolveGroupIpcPath(group.folder); + fs.mkdirSync(path.join(groupIpcDir, 'messages'), { recursive: true }); + fs.mkdirSync(path.join(groupIpcDir, 'tasks'), { recursive: true }); + fs.mkdirSync(path.join(groupIpcDir, 'input'), { recursive: true }); + mounts.push({ + hostPath: groupIpcDir, + containerPath: '/workspace/ipc', + readonly: false, + }); + + // Copy agent-runner source into a per-group writable location so agents + // can customize it (add tools, change behavior) without affecting other + // groups. Recompiled on container startup via entrypoint.sh. + const agentRunnerSrc = path.join( + projectRoot, + 'container', + 'agent-runner', + 'src', + ); + const groupAgentRunnerDir = path.join( + DATA_DIR, + 'sessions', + group.folder, + 'agent-runner-src', + ); + if (!fs.existsSync(groupAgentRunnerDir) && fs.existsSync(agentRunnerSrc)) { + fs.cpSync(agentRunnerSrc, groupAgentRunnerDir, { recursive: true }); + } + mounts.push({ + hostPath: groupAgentRunnerDir, + containerPath: '/app/src', + readonly: false, + }); + + // Additional mounts validated against external allowlist (tamper-proof from containers) + if (group.containerConfig?.additionalMounts) { + const validatedMounts = validateAdditionalMounts( + group.containerConfig.additionalMounts, + group.name, + isMain, + ); + mounts.push(...validatedMounts); + } + + return mounts; +} + +/** + * Read allowed secrets from .env for passing to the container via stdin. + * Secrets are never written to disk or mounted as files. + */ +function readSecrets(): Record { + return readEnvFile(['ZAI_API_KEY', 'ANTHROPIC_API_KEY']); +} + +function buildContainerArgs( + mounts: VolumeMount[], + containerName: string, +): string[] { + const args: string[] = ['run', '-i', '--rm', '--name', containerName]; + + // Pass host timezone so container's local time matches the user's + args.push('-e', `TZ=${TIMEZONE}`); + + // Run as host user so bind-mounted files are accessible. + // Skip when running as root (uid 0), as the container's node user (uid 1000), + // or when getuid is unavailable (native Windows without WSL). + const hostUid = process.getuid?.(); + const hostGid = process.getgid?.(); + if (hostUid != null && hostUid !== 0 && hostUid !== 1000) { + args.push('--user', `${hostUid}:${hostGid}`); + args.push('-e', 'HOME=/home/node'); + } + + for (const mount of mounts) { + if (mount.readonly) { + args.push(...readonlyMountArgs(mount.hostPath, mount.containerPath)); + } else { + args.push('-v', `${mount.hostPath}:${mount.containerPath}`); + } + } + + args.push(CONTAINER_IMAGE); + + return args; +} + +export async function runContainerAgent( + group: RegisteredGroup, + input: ContainerInput, + onProcess: (proc: ChildProcess, containerName: string) => void, + onOutput?: (output: ContainerOutput) => Promise, +): Promise { + const startTime = Date.now(); + + const groupDir = resolveGroupFolderPath(group.folder); + fs.mkdirSync(groupDir, { recursive: true }); + + const mounts = buildVolumeMounts(group, input.isMain); + const safeName = group.folder.replace(/[^a-zA-Z0-9-]/g, '-'); + const containerName = `clawdie-cp-${safeName}-${Date.now()}`; + const containerArgs = buildContainerArgs(mounts, containerName); + + logger.debug( + { + group: group.name, + containerName, + mounts: mounts.map( + (m) => + `${m.hostPath} -> ${m.containerPath}${m.readonly ? ' (ro)' : ''}`, + ), + containerArgs: containerArgs.join(' '), + }, + 'Container mount configuration', + ); + + logger.info( + { + group: group.name, + containerName, + mountCount: mounts.length, + isMain: input.isMain, + }, + 'Spawning container agent', + ); + + const logsDir = path.join(groupDir, 'logs'); + fs.mkdirSync(logsDir, { recursive: true }); + + return new Promise((resolve) => { + const container = spawn(CONTAINER_RUNTIME_BIN, containerArgs, { + stdio: ['pipe', 'pipe', 'pipe'], + }); + + onProcess(container, containerName); + + let stdout = ''; + let stderr = ''; + let stdoutTruncated = false; + let stderrTruncated = false; + + // Pass secrets via stdin (never written to disk or mounted as files) + input.secrets = readSecrets(); + container.stdin.write(JSON.stringify(input)); + container.stdin.end(); + // Remove secrets from input so they don't appear in logs + delete input.secrets; + + // Streaming output: parse OUTPUT_START/END marker pairs as they arrive + let parseBuffer = ''; + let newSessionId: string | undefined; + let outputChain = Promise.resolve(); + + container.stdout.on('data', (data) => { + const chunk = data.toString(); + + // Always accumulate for logging + if (!stdoutTruncated) { + const remaining = CONTAINER_MAX_OUTPUT_SIZE - stdout.length; + if (chunk.length > remaining) { + stdout += chunk.slice(0, remaining); + stdoutTruncated = true; + logger.warn( + { group: group.name, size: stdout.length }, + 'Container stdout truncated due to size limit', + ); + } else { + stdout += chunk; + } + } + + // Stream-parse for output markers + if (onOutput) { + parseBuffer += chunk; + let startIdx: number; + while ((startIdx = parseBuffer.indexOf(OUTPUT_START_MARKER)) !== -1) { + const endIdx = parseBuffer.indexOf(OUTPUT_END_MARKER, startIdx); + if (endIdx === -1) break; // Incomplete pair, wait for more data + + const jsonStr = parseBuffer + .slice(startIdx + OUTPUT_START_MARKER.length, endIdx) + .trim(); + parseBuffer = parseBuffer.slice(endIdx + OUTPUT_END_MARKER.length); + + try { + const parsed: ContainerOutput = JSON.parse(jsonStr); + if (parsed.newSessionId) { + newSessionId = parsed.newSessionId; + } + hadStreamingOutput = true; + // Activity detected — reset the hard timeout + resetTimeout(); + // Call onOutput for all markers (including null results) + // so idle timers start even for "silent" query completions. + outputChain = outputChain.then(() => onOutput(parsed)); + } catch (err) { + logger.warn( + { group: group.name, error: err }, + 'Failed to parse streamed output chunk', + ); + } + } + } + }); + + container.stderr.on('data', (data) => { + const chunk = data.toString(); + const lines = chunk.trim().split('\n'); + for (const line of lines) { + if (line) logger.debug({ container: group.folder }, line); + } + // Don't reset timeout on stderr — SDK writes debug logs continuously. + // Timeout only resets on actual output (OUTPUT_MARKER in stdout). + if (stderrTruncated) return; + const remaining = CONTAINER_MAX_OUTPUT_SIZE - stderr.length; + if (chunk.length > remaining) { + stderr += chunk.slice(0, remaining); + stderrTruncated = true; + logger.warn( + { group: group.name, size: stderr.length }, + 'Container stderr truncated due to size limit', + ); + } else { + stderr += chunk; + } + }); + + let timedOut = false; + let hadStreamingOutput = false; + const configTimeout = group.containerConfig?.timeout || CONTAINER_TIMEOUT; + // Grace period: hard timeout must be at least IDLE_TIMEOUT + 30s so the + // graceful _close sentinel has time to trigger before the hard kill fires. + const timeoutMs = Math.max(configTimeout, IDLE_TIMEOUT + 30_000); + + const killOnTimeout = () => { + timedOut = true; + logger.error( + { group: group.name, containerName }, + 'Container timeout, stopping gracefully', + ); + exec(stopContainer(containerName), { timeout: 15000 }, (err) => { + if (err) { + logger.warn( + { group: group.name, containerName, err }, + 'Graceful stop failed, force killing', + ); + container.kill('SIGKILL'); + } + }); + }; + + let timeout = setTimeout(killOnTimeout, timeoutMs); + + // Reset the timeout whenever there's activity (streaming output) + const resetTimeout = () => { + clearTimeout(timeout); + timeout = setTimeout(killOnTimeout, timeoutMs); + }; + + container.on('close', (code) => { + clearTimeout(timeout); + const duration = Date.now() - startTime; + + if (timedOut) { + const ts = new Date().toISOString().replace(/[:.]/g, '-'); + const timeoutLog = path.join(logsDir, `container-${ts}.log`); + fs.writeFileSync( + timeoutLog, + [ + `=== Container Run Log (TIMEOUT) ===`, + `Timestamp: ${new Date().toISOString()}`, + `Group: ${group.name}`, + `Container: ${containerName}`, + `Duration: ${duration}ms`, + `Exit Code: ${code}`, + `Had Streaming Output: ${hadStreamingOutput}`, + ].join('\n'), + ); + + // Timeout after output = idle cleanup, not failure. + // The agent already sent its response; this is just the + // container being reaped after the idle period expired. + if (hadStreamingOutput) { + logger.info( + { group: group.name, containerName, duration, code }, + 'Container timed out after output (idle cleanup)', + ); + outputChain.then(() => { + resolve({ + status: 'success', + result: null, + newSessionId, + }); + }); + return; + } + + logger.error( + { group: group.name, containerName, duration, code }, + 'Container timed out with no output', + ); + + resolve({ + status: 'error', + result: null, + error: `Container timed out after ${configTimeout}ms`, + }); + return; + } + + const timestamp = new Date().toISOString().replace(/[:.]/g, '-'); + const logFile = path.join(logsDir, `container-${timestamp}.log`); + const isVerbose = + process.env.LOG_LEVEL === 'debug' || process.env.LOG_LEVEL === 'trace'; + + const logLines = [ + `=== Container Run Log ===`, + `Timestamp: ${new Date().toISOString()}`, + `Group: ${group.name}`, + `IsMain: ${input.isMain}`, + `Duration: ${duration}ms`, + `Exit Code: ${code}`, + `Stdout Truncated: ${stdoutTruncated}`, + `Stderr Truncated: ${stderrTruncated}`, + ``, + ]; + + const isError = code !== 0; + + if (isVerbose || isError) { + logLines.push( + `=== Input ===`, + JSON.stringify(input, null, 2), + ``, + `=== Container Args ===`, + containerArgs.join(' '), + ``, + `=== Mounts ===`, + mounts + .map( + (m) => + `${m.hostPath} -> ${m.containerPath}${m.readonly ? ' (ro)' : ''}`, + ) + .join('\n'), + ``, + `=== Stderr${stderrTruncated ? ' (TRUNCATED)' : ''} ===`, + stderr, + ``, + `=== Stdout${stdoutTruncated ? ' (TRUNCATED)' : ''} ===`, + stdout, + ); + } else { + logLines.push( + `=== Input Summary ===`, + `Prompt length: ${input.prompt.length} chars`, + `Session ID: ${input.sessionId || 'new'}`, + ``, + `=== Mounts ===`, + mounts + .map((m) => `${m.containerPath}${m.readonly ? ' (ro)' : ''}`) + .join('\n'), + ``, + ); + } + + fs.writeFileSync(logFile, logLines.join('\n')); + logger.debug({ logFile, verbose: isVerbose }, 'Container log written'); + + if (code !== 0) { + logger.error( + { + group: group.name, + code, + duration, + stderr, + stdout, + logFile, + }, + 'Container exited with error', + ); + + resolve({ + status: 'error', + result: null, + error: `Container exited with code ${code}: ${stderr.slice(-200)}`, + }); + return; + } + + // Streaming mode: wait for output chain to settle, return completion marker + if (onOutput) { + outputChain.then(() => { + logger.info( + { group: group.name, duration, newSessionId }, + 'Container completed (streaming mode)', + ); + resolve({ + status: 'success', + result: null, + newSessionId, + }); + }); + return; + } + + // Legacy mode: parse the last output marker pair from accumulated stdout + try { + // Extract JSON between sentinel markers for robust parsing + const startIdx = stdout.indexOf(OUTPUT_START_MARKER); + const endIdx = stdout.indexOf(OUTPUT_END_MARKER); + + let jsonLine: string; + if (startIdx !== -1 && endIdx !== -1 && endIdx > startIdx) { + jsonLine = stdout + .slice(startIdx + OUTPUT_START_MARKER.length, endIdx) + .trim(); + } else { + // Fallback: last non-empty line (backwards compatibility) + const lines = stdout.trim().split('\n'); + jsonLine = lines[lines.length - 1]; + } + + const output: ContainerOutput = JSON.parse(jsonLine); + + logger.info( + { + group: group.name, + duration, + status: output.status, + hasResult: !!output.result, + }, + 'Container completed', + ); + + resolve(output); + } catch (err) { + logger.error( + { + group: group.name, + stdout, + stderr, + error: err, + }, + 'Failed to parse container output', + ); + + resolve({ + status: 'error', + result: null, + error: `Failed to parse container output: ${err instanceof Error ? err.message : String(err)}`, + }); + } + }); + + container.on('error', (err) => { + clearTimeout(timeout); + logger.error( + { group: group.name, containerName, error: err }, + 'Container spawn error', + ); + resolve({ + status: 'error', + result: null, + error: `Container spawn error: ${err.message}`, + }); + }); + }); +} + +export function writeTasksSnapshot( + groupFolder: string, + isMain: boolean, + tasks: Array<{ + id: string; + groupFolder: string; + prompt: string; + schedule_type: string; + schedule_value: string; + status: string; + next_run: string | null; + }>, +): void { + // Write filtered tasks to the group's IPC directory + const groupIpcDir = resolveGroupIpcPath(groupFolder); + fs.mkdirSync(groupIpcDir, { recursive: true }); + + // Main sees all tasks, others only see their own + const filteredTasks = isMain + ? tasks + : tasks.filter((t) => t.groupFolder === groupFolder); + + const tasksFile = path.join(groupIpcDir, 'current_tasks.json'); + fs.writeFileSync(tasksFile, JSON.stringify(filteredTasks, null, 2)); +} + +export interface AvailableGroup { + jid: string; + name: string; + lastActivity: string; + isRegistered: boolean; +} + +/** + * Write available groups snapshot for the container to read. + * Only main group can see all available groups (for activation). + * Non-main groups only see their own registration status. + */ +export function writeGroupsSnapshot( + groupFolder: string, + isMain: boolean, + groups: AvailableGroup[], + registeredJids: Set, +): void { + const groupIpcDir = resolveGroupIpcPath(groupFolder); + fs.mkdirSync(groupIpcDir, { recursive: true }); + + // Main sees all groups; others see nothing (they can't activate groups) + const visibleGroups = isMain ? groups : []; + + const groupsFile = path.join(groupIpcDir, 'available_groups.json'); + fs.writeFileSync( + groupsFile, + JSON.stringify( + { + groups: visibleGroups, + lastSync: new Date().toISOString(), + }, + null, + 2, + ), + ); +} diff --git a/.agent/skills/add-gmail/modify/src/container-runner.ts.intent.md b/.agent/skills/add-gmail/modify/src/container-runner.ts.intent.md new file mode 100644 index 0000000..69cf3b3 --- /dev/null +++ b/.agent/skills/add-gmail/modify/src/container-runner.ts.intent.md @@ -0,0 +1,42 @@ +# Intent: src/container-runner.ts modifications + +## What changed + +Added a volume mount for Gmail OAuth credentials (`~/.gmail-mcp/`) so the Gmail MCP server inside the container can authenticate with Google. + +## Key sections + +### buildVolumeMounts() + +- Added: Gmail credentials mount after the `.agent` sessions mount: + ``` + const gmailDir = path.join(homeDir, '.gmail-mcp'); + if (fs.existsSync(gmailDir)) { + mounts.push({ + hostPath: gmailDir, + containerPath: '/home/node/.gmail-mcp', + readonly: false, // MCP may need to refresh OAuth tokens + }); + } + ``` +- Uses `os.homedir()` to resolve the home directory +- Mount is read-write because the Gmail MCP server needs to refresh OAuth tokens +- Mount is conditional — only added if `~/.gmail-mcp/` exists on the host + +### Imports + +- Added: `os` import for `os.homedir()` + +## Invariants + +- All existing mounts are unchanged +- Mount ordering is preserved (Gmail added after session mounts, before additional mounts) +- The `buildContainerArgs`, `runContainerAgent`, and all other functions are untouched +- Additional mount validation via `validateAdditionalMounts` is unchanged + +## Must-keep + +- All existing volume mounts (project root, group dir, global, sessions, IPC, agent-runner, additional) +- The mount security model (allowlist validation for additional mounts) +- The `readSecrets` function and stdin-based secret passing +- Container lifecycle (spawn, timeout, output parsing) diff --git a/.agent/skills/add-gmail/modify/src/index.ts b/.agent/skills/add-gmail/modify/src/index.ts new file mode 100644 index 0000000..be26a17 --- /dev/null +++ b/.agent/skills/add-gmail/modify/src/index.ts @@ -0,0 +1,507 @@ +import fs from 'fs'; +import path from 'path'; + +import { + ASSISTANT_NAME, + IDLE_TIMEOUT, + MAIN_GROUP_FOLDER, + POLL_INTERVAL, + TRIGGER_PATTERN, +} from './config.js'; +import { GmailChannel } from './channels/gmail.js'; +import { WhatsAppChannel } from './channels/whatsapp.js'; +import { + ContainerOutput, + runContainerAgent, + writeGroupsSnapshot, + writeTasksSnapshot, +} from './container-runner.js'; +import { cleanupOrphans, ensureContainerRuntimeRunning } from './container-runtime.js'; +import { + getAllChats, + getAllRegisteredGroups, + getAllSessions, + getAllTasks, + getMessagesSince, + getNewMessages, + getRouterState, + initDatabase, + setRegisteredGroup, + setRouterState, + setSession, + storeChatMetadata, + storeMessage, +} from './db.js'; +import { GroupQueue } from './group-queue.js'; +import { resolveGroupFolderPath } from './group-folder.js'; +import { startIpcWatcher } from './ipc.js'; +import { findChannel, formatMessages, formatOutbound } from './router.js'; +import { startSchedulerLoop } from './task-scheduler.js'; +import { Channel, NewMessage, RegisteredGroup } from './types.js'; +import { logger } from './logger.js'; + +// Re-export for backwards compatibility during refactor +export { escapeXml, formatMessages } from './router.js'; + +let lastTimestamp = ''; +let sessions: Record = {}; +let registeredGroups: Record = {}; +let lastAgentTimestamp: Record = {}; +let messageLoopRunning = false; + +let whatsapp: WhatsAppChannel; +const channels: Channel[] = []; +const queue = new GroupQueue(); + +function loadState(): void { + lastTimestamp = getRouterState('last_timestamp') || ''; + const agentTs = getRouterState('last_agent_timestamp'); + try { + lastAgentTimestamp = agentTs ? JSON.parse(agentTs) : {}; + } catch { + logger.warn('Corrupted last_agent_timestamp in DB, resetting'); + lastAgentTimestamp = {}; + } + sessions = getAllSessions(); + registeredGroups = getAllRegisteredGroups(); + logger.info( + { groupCount: Object.keys(registeredGroups).length }, + 'State loaded', + ); +} + +function saveState(): void { + setRouterState('last_timestamp', lastTimestamp); + setRouterState( + 'last_agent_timestamp', + JSON.stringify(lastAgentTimestamp), + ); +} + +function registerGroup(jid: string, group: RegisteredGroup): void { + let groupDir: string; + try { + groupDir = resolveGroupFolderPath(group.folder); + } catch (err) { + logger.warn( + { jid, folder: group.folder, err }, + 'Rejecting group registration with invalid folder', + ); + return; + } + + registeredGroups[jid] = group; + setRegisteredGroup(jid, group); + + // Create group folder + fs.mkdirSync(path.join(groupDir, 'logs'), { recursive: true }); + + logger.info( + { jid, name: group.name, folder: group.folder }, + 'Group registered', + ); +} + +/** + * Get available groups list for the agent. + * Returns groups ordered by most recent activity. + */ +export function getAvailableGroups(): import('./container-runner.js').AvailableGroup[] { + const chats = getAllChats(); + const registeredJids = new Set(Object.keys(registeredGroups)); + + return chats + .filter((c) => c.jid !== '__group_sync__' && c.is_group) + .map((c) => ({ + jid: c.jid, + name: c.name, + lastActivity: c.last_message_time, + isRegistered: registeredJids.has(c.jid), + })); +} + +/** @internal - exported for testing */ +export function _setRegisteredGroups(groups: Record): void { + registeredGroups = groups; +} + +/** + * Process all pending messages for a group. + * Called by the GroupQueue when it's this group's turn. + */ +async function processGroupMessages(chatJid: string): Promise { + const group = registeredGroups[chatJid]; + if (!group) return true; + + const channel = findChannel(channels, chatJid); + if (!channel) { + console.log(`Warning: no channel owns JID ${chatJid}, skipping messages`); + return true; + } + + const isMainGroup = group.folder === MAIN_GROUP_FOLDER; + + const sinceTimestamp = lastAgentTimestamp[chatJid] || ''; + const missedMessages = getMessagesSince(chatJid, sinceTimestamp, ASSISTANT_NAME); + + if (missedMessages.length === 0) return true; + + // For non-main groups, check if trigger is required and present + if (!isMainGroup && group.requiresTrigger !== false) { + const hasTrigger = missedMessages.some((m) => + TRIGGER_PATTERN.test(m.content.trim()), + ); + if (!hasTrigger) return true; + } + + const prompt = formatMessages(missedMessages); + + // Advance cursor so the piping path in startMessageLoop won't re-fetch + // these messages. Save the old cursor so we can roll back on error. + const previousCursor = lastAgentTimestamp[chatJid] || ''; + lastAgentTimestamp[chatJid] = + missedMessages[missedMessages.length - 1].timestamp; + saveState(); + + logger.info( + { group: group.name, messageCount: missedMessages.length }, + 'Processing messages', + ); + + // Track idle timer for closing stdin when agent is idle + let idleTimer: ReturnType | null = null; + + const resetIdleTimer = () => { + if (idleTimer) clearTimeout(idleTimer); + idleTimer = setTimeout(() => { + logger.debug({ group: group.name }, 'Idle timeout, closing container stdin'); + queue.closeStdin(chatJid); + }, IDLE_TIMEOUT); + }; + + await channel.setTyping?.(chatJid, true); + let hadError = false; + let outputSentToUser = false; + + const output = await runAgent(group, prompt, chatJid, async (result) => { + // Streaming output callback — called for each agent result + if (result.result) { + const raw = typeof result.result === 'string' ? result.result : JSON.stringify(result.result); + // Strip ... blocks — agent uses these for internal reasoning + const text = raw.replace(/[\s\S]*?<\/internal>/g, '').trim(); + logger.info({ group: group.name }, `Agent output: ${raw.slice(0, 200)}`); + if (text) { + await channel.sendMessage(chatJid, text); + outputSentToUser = true; + } + // Only reset idle timer on actual results, not session-update markers (result: null) + resetIdleTimer(); + } + + if (result.status === 'success') { + queue.notifyIdle(chatJid); + } + + if (result.status === 'error') { + hadError = true; + } + }); + + await channel.setTyping?.(chatJid, false); + if (idleTimer) clearTimeout(idleTimer); + + if (output === 'error' || hadError) { + // If we already sent output to the user, don't roll back the cursor — + // the user got their response and re-processing would send duplicates. + if (outputSentToUser) { + logger.warn({ group: group.name }, 'Agent error after output was sent, skipping cursor rollback to prevent duplicates'); + return true; + } + // Roll back cursor so retries can re-process these messages + lastAgentTimestamp[chatJid] = previousCursor; + saveState(); + logger.warn({ group: group.name }, 'Agent error, rolled back message cursor for retry'); + return false; + } + + return true; +} + +async function runAgent( + group: RegisteredGroup, + prompt: string, + chatJid: string, + onOutput?: (output: ContainerOutput) => Promise, +): Promise<'success' | 'error'> { + const isMain = group.folder === MAIN_GROUP_FOLDER; + const sessionId = sessions[group.folder]; + + // Update tasks snapshot for container to read (filtered by group) + const tasks = getAllTasks(); + writeTasksSnapshot( + group.folder, + isMain, + tasks.map((t) => ({ + id: t.id, + groupFolder: t.group_folder, + prompt: t.prompt, + schedule_type: t.schedule_type, + schedule_value: t.schedule_value, + status: t.status, + next_run: t.next_run, + })), + ); + + // Update available groups snapshot (main group only can see all groups) + const availableGroups = getAvailableGroups(); + writeGroupsSnapshot( + group.folder, + isMain, + availableGroups, + new Set(Object.keys(registeredGroups)), + ); + + // Wrap onOutput to track session ID from streamed results + const wrappedOnOutput = onOutput + ? async (output: ContainerOutput) => { + if (output.newSessionId) { + sessions[group.folder] = output.newSessionId; + setSession(group.folder, output.newSessionId); + } + await onOutput(output); + } + : undefined; + + try { + const output = await runContainerAgent( + group, + { + prompt, + sessionId, + groupFolder: group.folder, + chatJid, + isMain, + assistantName: ASSISTANT_NAME, + }, + (proc, containerName) => queue.registerProcess(chatJid, proc, containerName, group.folder), + wrappedOnOutput, + ); + + if (output.newSessionId) { + sessions[group.folder] = output.newSessionId; + setSession(group.folder, output.newSessionId); + } + + if (output.status === 'error') { + logger.error( + { group: group.name, error: output.error }, + 'Container agent error', + ); + return 'error'; + } + + return 'success'; + } catch (err) { + logger.error({ group: group.name, err }, 'Agent error'); + return 'error'; + } +} + +async function startMessageLoop(): Promise { + if (messageLoopRunning) { + logger.debug('Message loop already running, skipping duplicate start'); + return; + } + messageLoopRunning = true; + + logger.info(`NanoClaw running (trigger: @${ASSISTANT_NAME})`); + + while (true) { + try { + const jids = Object.keys(registeredGroups); + const { messages, newTimestamp } = getNewMessages(jids, lastTimestamp, ASSISTANT_NAME); + + if (messages.length > 0) { + logger.info({ count: messages.length }, 'New messages'); + + // Advance the "seen" cursor for all messages immediately + lastTimestamp = newTimestamp; + saveState(); + + // Deduplicate by group + const messagesByGroup = new Map(); + for (const msg of messages) { + const existing = messagesByGroup.get(msg.chat_jid); + if (existing) { + existing.push(msg); + } else { + messagesByGroup.set(msg.chat_jid, [msg]); + } + } + + for (const [chatJid, groupMessages] of messagesByGroup) { + const group = registeredGroups[chatJid]; + if (!group) continue; + + const channel = findChannel(channels, chatJid); + if (!channel) { + console.log(`Warning: no channel owns JID ${chatJid}, skipping messages`); + continue; + } + + const isMainGroup = group.folder === MAIN_GROUP_FOLDER; + const needsTrigger = !isMainGroup && group.requiresTrigger !== false; + + // For non-main groups, only act on trigger messages. + // Non-trigger messages accumulate in DB and get pulled as + // context when a trigger eventually arrives. + if (needsTrigger) { + const hasTrigger = groupMessages.some((m) => + TRIGGER_PATTERN.test(m.content.trim()), + ); + if (!hasTrigger) continue; + } + + // Pull all messages since lastAgentTimestamp so non-trigger + // context that accumulated between triggers is included. + const allPending = getMessagesSince( + chatJid, + lastAgentTimestamp[chatJid] || '', + ASSISTANT_NAME, + ); + const messagesToSend = + allPending.length > 0 ? allPending : groupMessages; + const formatted = formatMessages(messagesToSend); + + if (queue.sendMessage(chatJid, formatted)) { + logger.debug( + { chatJid, count: messagesToSend.length }, + 'Piped messages to active container', + ); + lastAgentTimestamp[chatJid] = + messagesToSend[messagesToSend.length - 1].timestamp; + saveState(); + // Show typing indicator while the container processes the piped message + channel.setTyping?.(chatJid, true)?.catch((err) => + logger.warn({ chatJid, err }, 'Failed to set typing indicator'), + ); + } else { + // No active container — enqueue for a new one + queue.enqueueMessageCheck(chatJid); + } + } + } + } catch (err) { + logger.error({ err }, 'Error in message loop'); + } + await new Promise((resolve) => setTimeout(resolve, POLL_INTERVAL)); + } +} + +/** + * Startup recovery: check for unprocessed messages in registered groups. + * Handles crash between advancing lastTimestamp and processing messages. + */ +function recoverPendingMessages(): void { + for (const [chatJid, group] of Object.entries(registeredGroups)) { + const sinceTimestamp = lastAgentTimestamp[chatJid] || ''; + const pending = getMessagesSince(chatJid, sinceTimestamp, ASSISTANT_NAME); + if (pending.length > 0) { + logger.info( + { group: group.name, pendingCount: pending.length }, + 'Recovery: found unprocessed messages', + ); + queue.enqueueMessageCheck(chatJid); + } + } +} + +function ensureContainerSystemRunning(): void { + ensureContainerRuntimeRunning(); + cleanupOrphans(); +} + +async function main(): Promise { + ensureContainerSystemRunning(); + initDatabase(); + logger.info('Database initialized'); + loadState(); + + // Graceful shutdown handlers + const shutdown = async (signal: string) => { + logger.info({ signal }, 'Shutdown signal received'); + await queue.shutdown(10000); + for (const ch of channels) await ch.disconnect(); + process.exit(0); + }; + process.on('SIGTERM', () => shutdown('SIGTERM')); + process.on('SIGINT', () => shutdown('SIGINT')); + + // Channel callbacks (shared by all channels) + const channelOpts = { + onMessage: (_chatJid: string, msg: NewMessage) => storeMessage(msg), + onChatMetadata: (chatJid: string, timestamp: string, name?: string, channel?: string, isGroup?: boolean) => + storeChatMetadata(chatJid, timestamp, name, channel, isGroup), + registeredGroups: () => registeredGroups, + }; + + // Create and connect channels + whatsapp = new WhatsAppChannel(channelOpts); + channels.push(whatsapp); + await whatsapp.connect(); + + const gmail = new GmailChannel(channelOpts); + channels.push(gmail); + try { + await gmail.connect(); + } catch (err) { + logger.warn({ err }, 'Gmail channel failed to connect, continuing without it'); + } + + // Start subsystems (independently of connection handler) + startSchedulerLoop({ + registeredGroups: () => registeredGroups, + getSessions: () => sessions, + queue, + onProcess: (groupJid, proc, containerName, groupFolder) => queue.registerProcess(groupJid, proc, containerName, groupFolder), + sendMessage: async (jid, rawText) => { + const channel = findChannel(channels, jid); + if (!channel) { + console.log(`Warning: no channel owns JID ${jid}, cannot send message`); + return; + } + const text = formatOutbound(rawText); + if (text) await channel.sendMessage(jid, text); + }, + }); + startIpcWatcher({ + sendMessage: (jid, text) => { + const channel = findChannel(channels, jid); + if (!channel) throw new Error(`No channel for JID: ${jid}`); + return channel.sendMessage(jid, text); + }, + registeredGroups: () => registeredGroups, + registerGroup, + syncGroupMetadata: (force) => whatsapp?.syncGroupMetadata(force) ?? Promise.resolve(), + getAvailableGroups, + writeGroupsSnapshot: (gf, im, ag, rj) => writeGroupsSnapshot(gf, im, ag, rj), + }); + queue.setProcessMessagesFn(processGroupMessages); + recoverPendingMessages(); + startMessageLoop().catch((err) => { + logger.fatal({ err }, 'Message loop crashed unexpectedly'); + process.exit(1); + }); +} + +// Guard: only run when executed directly, not when imported by tests +const isDirectRun = + process.argv[1] && + new URL(import.meta.url).pathname === new URL(`file://${process.argv[1]}`).pathname; + +if (isDirectRun) { + main().catch((err) => { + logger.error({ err }, 'Failed to start NanoClaw'); + process.exit(1); + }); +} diff --git a/.agent/skills/add-gmail/modify/src/index.ts.intent.md b/.agent/skills/add-gmail/modify/src/index.ts.intent.md new file mode 100644 index 0000000..cd700f5 --- /dev/null +++ b/.agent/skills/add-gmail/modify/src/index.ts.intent.md @@ -0,0 +1,40 @@ +# Intent: src/index.ts modifications + +## What changed + +Added Gmail as a channel. + +## Key sections + +### Imports (top of file) + +- Added: `GmailChannel` from `./channels/gmail.js` + +### main() + +- Added Gmail channel creation: + ``` + const gmail = new GmailChannel(channelOpts); + channels.push(gmail); + await gmail.connect(); + ``` +- Gmail uses the same `channelOpts` callbacks as other channels +- Incoming emails are delivered to the main group (agent decides how to respond, user can configure) + +## Invariants + +- All existing message processing logic (triggers, cursors, idle timers) is preserved +- The `runAgent` function is completely unchanged +- State management (loadState/saveState) is unchanged +- Recovery logic is unchanged +- Container runtime check is unchanged +- Any other channel creation is untouched +- Shutdown iterates `channels` array (Gmail is included automatically) + +## Must-keep + +- The `escapeXml` and `formatMessages` re-exports +- The `_setRegisteredGroups` test helper +- The `isDirectRun` guard at bottom +- All error handling and cursor rollback logic in processGroupMessages +- The outgoing queue flush and reconnection logic diff --git a/.agent/skills/add-gmail/modify/src/routing.test.ts b/.agent/skills/add-gmail/modify/src/routing.test.ts new file mode 100644 index 0000000..837b1da --- /dev/null +++ b/.agent/skills/add-gmail/modify/src/routing.test.ts @@ -0,0 +1,119 @@ +import { describe, it, expect, beforeEach } from 'vitest'; + +import { _initTestDatabase, getAllChats, storeChatMetadata } from './db.js'; +import { getAvailableGroups, _setRegisteredGroups } from './index.js'; + +beforeEach(() => { + _initTestDatabase(); + _setRegisteredGroups({}); +}); + +// --- JID ownership patterns --- + +describe('JID ownership patterns', () => { + // These test the patterns that will become ownsJid() on the Channel interface + + it('WhatsApp group JID: ends with @g.us', () => { + const jid = '12345678@g.us'; + expect(jid.endsWith('@g.us')).toBe(true); + }); + + it('WhatsApp DM JID: ends with @s.whatsapp.net', () => { + const jid = '12345678@s.whatsapp.net'; + expect(jid.endsWith('@s.whatsapp.net')).toBe(true); + }); + + it('Gmail JID: starts with gmail:', () => { + const jid = 'gmail:abc123def'; + expect(jid.startsWith('gmail:')).toBe(true); + }); + + it('Gmail thread JID: starts with gmail: followed by thread ID', () => { + const jid = 'gmail:18d3f4a5b6c7d8e9'; + expect(jid.startsWith('gmail:')).toBe(true); + }); +}); + +// --- getAvailableGroups --- + +describe('getAvailableGroups', () => { + it('returns only groups, excludes DMs', () => { + storeChatMetadata('group1@g.us', '2024-01-01T00:00:01.000Z', 'Group 1', 'whatsapp', true); + storeChatMetadata('user@s.whatsapp.net', '2024-01-01T00:00:02.000Z', 'User DM', 'whatsapp', false); + storeChatMetadata('group2@g.us', '2024-01-01T00:00:03.000Z', 'Group 2', 'whatsapp', true); + + const groups = getAvailableGroups(); + expect(groups).toHaveLength(2); + expect(groups.map((g) => g.jid)).toContain('group1@g.us'); + expect(groups.map((g) => g.jid)).toContain('group2@g.us'); + expect(groups.map((g) => g.jid)).not.toContain('user@s.whatsapp.net'); + }); + + it('excludes __group_sync__ sentinel', () => { + storeChatMetadata('__group_sync__', '2024-01-01T00:00:00.000Z'); + storeChatMetadata('group@g.us', '2024-01-01T00:00:01.000Z', 'Group', 'whatsapp', true); + + const groups = getAvailableGroups(); + expect(groups).toHaveLength(1); + expect(groups[0].jid).toBe('group@g.us'); + }); + + it('marks registered groups correctly', () => { + storeChatMetadata('reg@g.us', '2024-01-01T00:00:01.000Z', 'Registered', 'whatsapp', true); + storeChatMetadata('unreg@g.us', '2024-01-01T00:00:02.000Z', 'Unregistered', 'whatsapp', true); + + _setRegisteredGroups({ + 'reg@g.us': { + name: 'Registered', + folder: 'registered', + trigger: '@Andy', + added_at: '2024-01-01T00:00:00.000Z', + }, + }); + + const groups = getAvailableGroups(); + const reg = groups.find((g) => g.jid === 'reg@g.us'); + const unreg = groups.find((g) => g.jid === 'unreg@g.us'); + + expect(reg?.isRegistered).toBe(true); + expect(unreg?.isRegistered).toBe(false); + }); + + it('returns groups ordered by most recent activity', () => { + storeChatMetadata('old@g.us', '2024-01-01T00:00:01.000Z', 'Old', 'whatsapp', true); + storeChatMetadata('new@g.us', '2024-01-01T00:00:05.000Z', 'New', 'whatsapp', true); + storeChatMetadata('mid@g.us', '2024-01-01T00:00:03.000Z', 'Mid', 'whatsapp', true); + + const groups = getAvailableGroups(); + expect(groups[0].jid).toBe('new@g.us'); + expect(groups[1].jid).toBe('mid@g.us'); + expect(groups[2].jid).toBe('old@g.us'); + }); + + it('excludes non-group chats regardless of JID format', () => { + // Unknown JID format stored without is_group should not appear + storeChatMetadata('unknown-format-123', '2024-01-01T00:00:01.000Z', 'Unknown'); + // Explicitly non-group with unusual JID + storeChatMetadata('custom:abc', '2024-01-01T00:00:02.000Z', 'Custom DM', 'custom', false); + // A real group for contrast + storeChatMetadata('group@g.us', '2024-01-01T00:00:03.000Z', 'Group', 'whatsapp', true); + + const groups = getAvailableGroups(); + expect(groups).toHaveLength(1); + expect(groups[0].jid).toBe('group@g.us'); + }); + + it('returns empty array when no chats exist', () => { + const groups = getAvailableGroups(); + expect(groups).toHaveLength(0); + }); + + it('excludes Gmail threads from group list (Gmail threads are not groups)', () => { + storeChatMetadata('gmail:abc123', '2024-01-01T00:00:01.000Z', 'Email thread', 'gmail', false); + storeChatMetadata('group@g.us', '2024-01-01T00:00:02.000Z', 'Group', 'whatsapp', true); + + const groups = getAvailableGroups(); + expect(groups).toHaveLength(1); + expect(groups[0].jid).toBe('group@g.us'); + }); +}); diff --git a/.agent/skills/add-gmail/tests/gmail.test.ts b/.agent/skills/add-gmail/tests/gmail.test.ts new file mode 100644 index 0000000..02d9721 --- /dev/null +++ b/.agent/skills/add-gmail/tests/gmail.test.ts @@ -0,0 +1,40 @@ +import { describe, it, expect } from 'vitest'; +import fs from 'fs'; +import path from 'path'; + +const root = process.cwd(); +const read = (f: string) => fs.readFileSync(path.join(root, f), 'utf-8'); + +function getGmailMode(): 'tool-only' | 'channel' { + const p = path.join(root, '.nanoclaw/state.yaml'); + if (!fs.existsSync(p)) return 'channel'; + return read('.nanoclaw/state.yaml').includes('mode: tool-only') ? 'tool-only' : 'channel'; +} + +const mode = getGmailMode(); +const channelOnly = mode === 'tool-only'; + +describe('add-gmail skill', () => { + it('container-runner mounts ~/.gmail-mcp', () => { + expect(read('src/container-runner.ts')).toContain('.gmail-mcp'); + }); + + it('agent-runner has gmail MCP server', () => { + const content = read('container/agent-runner/src/index.ts'); + expect(content).toContain('mcp__gmail__*'); + expect(content).toContain('@gongrzhe/server-gmail-autoauth-mcp'); + }); + + it.skipIf(channelOnly)('gmail channel file exists', () => { + expect(fs.existsSync(path.join(root, 'src/channels/gmail.ts'))).toBe(true); + }); + + it.skipIf(channelOnly)('index.ts wires up GmailChannel', () => { + expect(read('src/index.ts')).toContain('GmailChannel'); + }); + + it.skipIf(channelOnly)('googleapis dependency installed', () => { + const pkg = JSON.parse(read('package.json')); + expect(pkg.dependencies?.googleapis || pkg.devDependencies?.googleapis).toBeDefined(); + }); +}); diff --git a/.agent/skills/add-parallel/SKILL.md b/.agent/skills/add-parallel/SKILL.md new file mode 100644 index 0000000..36aade5 --- /dev/null +++ b/.agent/skills/add-parallel/SKILL.md @@ -0,0 +1,320 @@ +--- +name: add-parallel +description: Add Parallel AI MCP integration to Clawdie for advanced web research capabilities. Use when the user wants the agent to perform multi-step web research using Parallel AI's tools. +--- + +# Add Parallel AI Integration + +Adds Parallel AI MCP integration to NanoClaw for advanced web research capabilities. + +## What This Adds + +- **Quick Search** - Fast web lookups using Parallel Search API (free to use) +- **Deep Research** - Comprehensive analysis using Parallel Task API (asks permission) +- **Non-blocking Design** - Uses NanoClaw scheduler for result polling (no container blocking) + +## Prerequisites + +User must have: + +1. Parallel AI API key from https://platform.parallel.ai +2. NanoClaw already set up and running +3. Docker installed and running + +## Implementation Steps + +Run all steps automatically. Only pause for user input when explicitly needed. + +### 1. Get Parallel AI API Key + +Use `AskUserQuestion: Do you have a Parallel AI API key, or should I help you get one?` + +**If they have one:** +Collect it now. + +**If they need one:** +Tell them: + +> 1. Go to https://platform.parallel.ai +> 2. Sign up or log in +> 3. Navigate to API Keys section +> 4. Create a new API key +> 5. Copy the key and paste it here + +Wait for the API key. + +### 2. Add API Key to Environment + +Add `PARALLEL_API_KEY` to `.env`: + +```bash +# Check if .env exists, create if not +if [ ! -f .env ]; then + touch .env +fi + +# Add PARALLEL_API_KEY if not already present +if ! grep -q "PARALLEL_API_KEY=" .env; then + echo "PARALLEL_API_KEY=${API_KEY_FROM_USER}" >> .env + echo "✓ Added PARALLEL_API_KEY to .env" +else + # Update existing key + sed -i.bak "s/^PARALLEL_API_KEY=.*/PARALLEL_API_KEY=${API_KEY_FROM_USER}/" .env + echo "✓ Updated PARALLEL_API_KEY in .env" +fi +``` + +Verify: + +```bash +grep "PARALLEL_API_KEY" .env | head -c 50 +``` + +### 3. Update Container Runner + +Add `PARALLEL_API_KEY` to allowed environment variables in `src/container-runner.ts`: + +Find the line: + +```typescript +const allowedVars = ['ZAI_API_KEY', 'ANTHROPIC_API_KEY']; +``` + +Replace with: + +```typescript +const allowedVars = ['ZAI_API_KEY', 'ANTHROPIC_API_KEY', 'PARALLEL_API_KEY']; +``` + +### 4. Configure MCP Servers in Agent Runner + +Update `container/agent-runner/src/index.ts`: + +Find the section where `mcpServers` is configured (around line 237-252): + +```typescript +const mcpServers: Record = { + nanoclaw: ipcMcp, +}; +``` + +Add Parallel AI MCP servers after the nanoclaw server: + +```typescript +const mcpServers: Record = { + nanoclaw: ipcMcp, +}; + +// Add Parallel AI MCP servers if API key is available +const parallelApiKey = process.env.PARALLEL_API_KEY; +if (parallelApiKey) { + mcpServers['parallel-search'] = { + type: 'http', // REQUIRED: Must specify type for HTTP MCP servers + url: 'https://search-mcp.parallel.ai/mcp', + headers: { + Authorization: `Bearer ${parallelApiKey}`, + }, + }; + mcpServers['parallel-task'] = { + type: 'http', // REQUIRED: Must specify type for HTTP MCP servers + url: 'https://task-mcp.parallel.ai/mcp', + headers: { + Authorization: `Bearer ${parallelApiKey}`, + }, + }; + log('Parallel AI MCP servers configured'); +} else { + log('PARALLEL_API_KEY not set, skipping Parallel AI integration'); +} +``` + +Also update the `allowedTools` array to include Parallel MCP tools (around line 242-248): + +```typescript +allowedTools: [ + 'Bash', + 'Read', 'Write', 'Edit', 'Glob', 'Grep', + 'WebSearch', 'WebFetch', + 'mcp__nanoclaw__*', + 'mcp__parallel-search__*', + 'mcp__parallel-task__*' +], +``` + +### 5. Add Usage Instructions to AGENT.md + +Add Parallel AI usage instructions to `groups/main/AGENT.md`: + +Find the "## What You Can Do" section and add after the existing bullet points: + +```markdown +- Use Parallel AI for web research and deep learning tasks +``` + +Then add a new section after "## What You Can Do": + +```markdown +## Web Research Tools + +You have access to two Parallel AI research tools: + +### Quick Web Search (`mcp__parallel-search__search`) + +**When to use:** Freely use for factual lookups, current events, definitions, recent information, or verifying facts. + +**Examples:** + +- "Who invented the transistor?" +- "What's the latest news about quantum computing?" +- "When was the UN founded?" +- "What are the top programming languages in 2026?" + +**Speed:** Fast (2-5 seconds) +**Cost:** Low +**Permission:** Not needed - use whenever it helps answer the question + +### Deep Research (`mcp__parallel-task__create_task_run`) + +**When to use:** Comprehensive analysis, learning about complex topics, comparing concepts, historical overviews, or structured research. + +**Examples:** + +- "Explain the development of quantum mechanics from 1900-1930" +- "Compare the literary styles of Hemingway and Faulkner" +- "Research the evolution of jazz from bebop to fusion" +- "Analyze the causes of the French Revolution" + +**Speed:** Slower (1-20 minutes depending on depth) +**Cost:** Higher (varies by processor tier) +**Permission:** ALWAYS use `AskUserQuestion` before using this tool + +**How to ask permission:** +``` + +AskUserQuestion: I can do deep research on [topic] using Parallel's Task API. This will take 2-5 minutes and provide comprehensive analysis with citations. Should I proceed? + +``` + +**After permission - DO NOT BLOCK! Use scheduler instead:** + +1. Create the task using `mcp__parallel-task__create_task_run` +2. Get the `run_id` from the response +3. Create a polling scheduled task using `mcp__nanoclaw__schedule_task`: +``` + +Prompt: "Check Parallel AI task run [run_id] and send results when ready. + +1. Use the Parallel Task MCP to check the task status +2. If status is 'completed', extract the results +3. Send results to user with mcp**nanoclaw**send_message +4. Use mcp**nanoclaw**complete_scheduled_task to mark this task as done + +If status is still 'running' or 'pending', do nothing (task will run again in 30s). +If status is 'failed', send error message and complete the task." + +Schedule: interval every 30 seconds +Context mode: isolated + +``` +4. Send acknowledgment with tracking link +5. Exit immediately - scheduler handles the rest + +### Choosing Between Them + +**Use Search when:** +- Question needs a quick fact or recent information +- Simple definition or clarification +- Verifying specific details +- Current events or news + +**Use Deep Research (with permission) when:** +- User wants to learn about a complex topic +- Question requires analysis or comparison +- Historical context or evolution of concepts +- Structured, comprehensive understanding needed +- User explicitly asks to "research" or "explain in depth" + +**Default behavior:** Prefer search for most questions. Only suggest deep research when the topic genuinely requires comprehensive analysis. +``` + +### 6. Rebuild Container + +Build the container with updated agent runner: + +```bash +./container/build.sh +``` + +Verify the build: + +```bash +echo '{}' | docker run -i --entrypoint /bin/echo nanoclaw-agent:latest "Container OK" +``` + +### 7. Restart Service + +Rebuild the main app and restart: + +```bash +npm run build +launchctl kickstart -k gui/$(id -u)/com.nanoclaw # macOS +# Linux: systemctl --user restart nanoclaw +``` + +Wait 3 seconds for service to start, then verify: + +```bash +sleep 3 +launchctl list | grep nanoclaw # macOS +# Linux: systemctl --user status nanoclaw +``` + +### 8. Test Integration + +Tell the user to test: + +> Send a message to your assistant: `@[YourAssistantName] what's the latest news about AI?` +> +> The assistant should use Parallel Search API to find current information. +> +> Then try: `@[YourAssistantName] can you research the history of artificial intelligence?` +> +> The assistant should ask for permission before using the Task API. + +Check logs to verify MCP servers loaded: + +```bash +tail -20 logs/nanoclaw.log +``` + +Look for: `Parallel AI MCP servers configured` + +## Troubleshooting + +**Container hangs or times out:** + +- Check that `type: 'http'` is specified in MCP server config +- Verify API key is correct in .env +- Check container logs: `cat groups/main/logs/container-*.log | tail -50` + +**MCP servers not loading:** + +- Ensure PARALLEL_API_KEY is in .env +- Verify container-runner.ts includes PARALLEL_API_KEY in allowedVars +- Check agent-runner logs for "Parallel AI MCP servers configured" message + +**Task polling not working:** + +- Verify scheduled task was created: `psql "$OPS_DB_URL" -c "SELECT * FROM scheduled_tasks"` +- Check task runs: `tail -f logs/nanoclaw.log | grep "scheduled task"` +- Ensure task prompt includes proper Parallel MCP tool names + +## Uninstalling + +To remove Parallel AI integration: + +1. Remove from .env: `sed -i.bak '/PARALLEL_API_KEY/d' .env` +2. Revert changes to container-runner.ts and agent-runner/src/index.ts +3. Remove Web Research Tools section from groups/main/AGENT.md +4. Rebuild: `./container/build.sh && npm run build` +5. Restart: `launchctl kickstart -k gui/$(id -u)/com.nanoclaw` (macOS) or `systemctl --user restart nanoclaw` (Linux) diff --git a/.agent/skills/add-protonmail/SKILL.md b/.agent/skills/add-protonmail/SKILL.md new file mode 100644 index 0000000..28f6074 --- /dev/null +++ b/.agent/skills/add-protonmail/SKILL.md @@ -0,0 +1,85 @@ +--- +name: add-protonmail +description: Add a send_email tool so the agent can send email from a ProtonMail custom domain address using an SMTP submission token. Use when the user wants the agent to send emails. +--- + +# Skill: add-protonmail + +Adds a `send_email` tool to your agent so it can send emails from your ProtonMail +custom domain address (e.g. `hello@clawdie.si`) using an SMTP submission token. + +No daemon required. No Proton Bridge. Just an SMTP token and nodemailer. + +## Prerequisites + +- A ProtonMail account with a custom domain (e.g. `hello@clawdie.si`) +- SMTP submission enabled for your domain in Proton settings + +## Setup + +### 1. Generate an SMTP token + +1. Log in to Proton Mail → Settings → All Settings → Proton Mail → SMTP submission +2. Enable SMTP submission for your domain +3. Click **Generate token** for your address +4. Copy the token (looks like a long random string) + +> Note: SMTP tokens are different from your login password. They are +> address-specific and can be revoked without affecting your account. + +### 2. Add to .env + +``` +PROTONMAIL_SMTP_USER=hello@clawdie.si +PROTONMAIL_SMTP_TOKEN=your_smtp_token_here +``` + +### 3. Apply the skill + +```bash +npx tsx scripts/apply-skill.ts .agent/skills/add-protonmail +``` + +Then rebuild the jail agent runner: + +```bash +cd jail/agent-runner && npm install && npm run build +``` + +## Tools the agent gets + +| Tool | Description | +| ------------ | ------------------------------------------ | +| `send_email` | Send an email from your ProtonMail address | + +The tool accepts: `to`, `subject`, `body`, and optional `cc` and `reply_to`. + +## Example conversations + +> "Send John an email confirming his order" +> → Agent calls `send_email(to: "john@example.com", subject: "Order confirmed", body: "Hi John...")` + +> "Email the team about the outage" +> → Agent composes and sends via `send_email` + +> "Reply to Sarah's question about pricing" +> → Agent drafts reply and sends via `send_email` + +## Limitations + +This skill is **send-only**. The agent cannot read the ProtonMail inbox or +receive emails automatically. For full inbound + outbound support, see the +Proton Bridge option in the skill roadmap. + +## Upgrading to full inbound channel + +Once Proton Bridge headless mode on FreeBSD is confirmed, this skill will be +extended to add a `ProtonmailChannel` that polls via IMAP and delivers incoming +emails to the agent automatically — the same pattern as `add-gmail`. + +## Security + +- SMTP token is separate from your login password +- Tokens can be revoked individually without affecting your account +- `PROTONMAIL_SMTP_TOKEN` is passed to the jail via encrypted stdin, never written to disk +- Emails sent via SMTP are stored in Sent with zero-access encryption diff --git a/.agent/skills/add-protonmail/add/jail/agent-runner/src/protonmail-tools.ts b/.agent/skills/add-protonmail/add/jail/agent-runner/src/protonmail-tools.ts new file mode 100644 index 0000000..b7f64e7 --- /dev/null +++ b/.agent/skills/add-protonmail/add/jail/agent-runner/src/protonmail-tools.ts @@ -0,0 +1,72 @@ +/** + * ProtonMail SMTP tools for Clawdie agent. + * Registered into the existing ipc-mcp-stdio server when PROTONMAIL_SMTP credentials are set. + * Uses ProtonMail SMTP submission tokens (custom domain addresses only). + * Send-only — no inbound polling. + */ +import type { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js'; +import type { Transporter } from 'nodemailer'; +import { z } from 'zod'; + +export function registerProtonmailTools(server: McpServer): void { + const user = process.env.PROTONMAIL_SMTP_USER; + const token = process.env.PROTONMAIL_SMTP_TOKEN; + if (!user || !token) return; + + let transporterInstance: Transporter | null = null; + + const getTransporter = async (): Promise => { + if (!transporterInstance) { + const nodemailer = await import('nodemailer'); + transporterInstance = nodemailer.default.createTransport({ + host: 'smtp.protonmail.ch', + port: 587, + secure: false, // STARTTLS + auth: { user, pass: token }, + }); + } + return transporterInstance; + }; + + server.tool( + 'send_email', + `Send an email from ${user} via ProtonMail. Use for notifications, replies, and outbound communication.`, + { + to: z.string().describe('Recipient email address (e.g. "john@example.com" or "Name ")'), + subject: z.string().describe('Email subject line'), + body: z.string().describe('Email body as plain text'), + cc: z.string().optional().describe('CC address (optional)'), + reply_to: z + .string() + .optional() + .describe('Reply-To address if different from sender (optional)'), + }, + async (args) => { + try { + const transporter = await getTransporter(); + const info = await transporter.sendMail({ + from: user, + to: args.to, + subject: args.subject, + text: args.body, + ...(args.cc ? { cc: args.cc } : {}), + ...(args.reply_to ? { replyTo: args.reply_to } : {}), + }); + return { + content: [{ + type: 'text' as const, + text: `Email sent to ${args.to}\nSubject: ${args.subject}\nMessage ID: ${info.messageId}`, + }], + }; + } catch (err) { + return { + content: [{ + type: 'text' as const, + text: `Failed to send email: ${err instanceof Error ? err.message : String(err)}`, + }], + isError: true, + }; + } + }, + ); +} diff --git a/.agent/skills/add-protonmail/manifest.yaml b/.agent/skills/add-protonmail/manifest.yaml new file mode 100644 index 0000000..f8db084 --- /dev/null +++ b/.agent/skills/add-protonmail/manifest.yaml @@ -0,0 +1,17 @@ +skill: protonmail +version: 1.0.0 +description: "ProtonMail email sending via SMTP token — send emails from your custom domain" +core_version: 0.4.0 +adds: + - jail/agent-runner/src/protonmail-tools.ts +modifies: + - src/jail-runner.ts + - jail/agent-runner/src/ipc-mcp-stdio.ts + - jail/agent-runner/package.json +structured: + npm_dependencies: + nodemailer: "^6.9.0" + "@types/nodemailer": "^6.4.0" +conflicts: [] +depends: [] +test: "" diff --git a/.agent/skills/add-protonmail/modify/jail/agent-runner/package.json.intent.md b/.agent/skills/add-protonmail/modify/jail/agent-runner/package.json.intent.md new file mode 100644 index 0000000..aa172bd --- /dev/null +++ b/.agent/skills/add-protonmail/modify/jail/agent-runner/package.json.intent.md @@ -0,0 +1,37 @@ +# Intent: jail/agent-runner/package.json modifications + +## What changed + +Added `nodemailer` as a runtime dependency and `@types/nodemailer` as a dev +dependency. + +## Key sections + +### dependencies + +- Added: `"nodemailer": "^6.9.0"` + +### devDependencies + +- Added: `"@types/nodemailer": "^6.4.0"` + +```json +{ + "dependencies": { + "@modelcontextprotocol/sdk": "^1.12.1", + "cron-parser": "^5.0.0", + "nodemailer": "^6.9.0", + "zod": "^4.0.0" + }, + "devDependencies": { + "@types/node": "^22.10.7", + "@types/nodemailer": "^6.4.0", + "typescript": "^5.7.3" + } +} +``` + +## Invariants + +- All existing dependencies remain unchanged +- package name, version, scripts are unchanged diff --git a/.agent/skills/add-protonmail/modify/jail/agent-runner/src/ipc-mcp-stdio.ts.intent.md b/.agent/skills/add-protonmail/modify/jail/agent-runner/src/ipc-mcp-stdio.ts.intent.md new file mode 100644 index 0000000..25b8596 --- /dev/null +++ b/.agent/skills/add-protonmail/modify/jail/agent-runner/src/ipc-mcp-stdio.ts.intent.md @@ -0,0 +1,36 @@ +# Intent: jail/agent-runner/src/ipc-mcp-stdio.ts modifications + +## What changed + +Imported and registered the ProtonMail MCP tools so the agent can send emails +from the configured ProtonMail address. Tools are only registered when both +`PROTONMAIL_SMTP_USER` and `PROTONMAIL_SMTP_TOKEN` are present. + +## Key sections + +### Imports (top of file) + +- Added: `import { registerProtonmailTools } from './protonmail-tools.js';` + +### After all existing server.tool() calls, before the transport connection + +- Added: `registerProtonmailTools(server);` + +```typescript +// Add near the bottom, before: +// const transport = new StdioServerTransport(); + +registerProtonmailTools(server); + +// Existing: +const transport = new StdioServerTransport(); +await server.connect(transport); +``` + +## Invariants + +- All existing tools (send_message, schedule_task, list_tasks, etc.) are unchanged +- Server name and version are unchanged +- IPC directory constants are unchanged +- Transport connection is unchanged +- If either ProtonMail env var is absent, `registerProtonmailTools` is a no-op diff --git a/.agent/skills/add-protonmail/modify/src/jail-runner.ts.intent.md b/.agent/skills/add-protonmail/modify/src/jail-runner.ts.intent.md new file mode 100644 index 0000000..cac75dd --- /dev/null +++ b/.agent/skills/add-protonmail/modify/src/jail-runner.ts.intent.md @@ -0,0 +1,38 @@ +# Intent: src/jail-runner.ts modifications + +## What changed + +Added `PROTONMAIL_SMTP_USER` and `PROTONMAIL_SMTP_TOKEN` to the secrets +allowlist so ProtonMail credentials are passed to the jail agent securely +via stdin — never via environment variables or files. + +## Key sections + +### readSecrets() function (around line 216) + +- Added: `'PROTONMAIL_SMTP_USER'` and `'PROTONMAIL_SMTP_TOKEN'` to the `readEnvFile` array + +```typescript +// Before: +return readEnvFile([ + 'ANTHROPIC_API_KEY', + ... + 'KIMI_API_KEY', +]); + +// After: +return readEnvFile([ + 'ANTHROPIC_API_KEY', + ... + 'KIMI_API_KEY', + 'PROTONMAIL_SMTP_USER', + 'PROTONMAIL_SMTP_TOKEN', +]); +``` + +## Invariants + +- All existing API keys remain in the list unchanged +- `readEnvFile()` signature and behavior are unchanged +- Secrets are still passed via stdin JSON, never written to disk or env +- If either ProtonMail env var is absent from `.env`, it is silently skipped diff --git a/.agent/skills/add-slack/SKILL.md b/.agent/skills/add-slack/SKILL.md new file mode 100644 index 0000000..f3706ae --- /dev/null +++ b/.agent/skills/add-slack/SKILL.md @@ -0,0 +1,235 @@ +--- +name: add-slack +description: Add Slack as a channel. Can replace WhatsApp entirely or run alongside it. Uses Socket Mode (no public URL needed). +--- + +# Add Slack Channel + +This skill adds Slack support to NanoClaw using the skills engine for deterministic code changes, then walks through interactive setup. + +## Phase 1: Pre-flight + +### Check if already applied + +Read `.nanoclaw/state.yaml`. If `slack` is in `applied_skills`, skip to Phase 3 (Setup). The code changes are already in place. + +### Ask the user + +1. **Mode**: Replace WhatsApp or add alongside it? + - Replace → will set `SLACK_ONLY=true` + - Alongside → both channels active (default) + +2. **Do they already have a Slack app configured?** If yes, collect the Bot Token and App Token now. If no, we'll create one in Phase 3. + +## Phase 2: Apply Code Changes + +Run the skills engine to apply this skill's code package. The package files are in this directory alongside this SKILL.md. + +### Initialize skills system (if needed) + +If `.nanoclaw/` directory doesn't exist yet: + +```bash +npx tsx scripts/apply-skill.ts --init +``` + +Or call `initSkillsSystem()` from `skills-engine/migrate.ts`. + +### Apply the skill + +```bash +npx tsx scripts/apply-skill.ts .agent/skills/add-slack +``` + +This deterministically: + +- Adds `src/channels/slack.ts` (SlackChannel class implementing Channel interface) +- Adds `src/channels/slack.test.ts` (46 unit tests) +- Three-way merges Slack support into `src/index.ts` (multi-channel support, conditional channel creation) +- Three-way merges Slack config into `src/config.ts` (SLACK_ONLY export) +- Three-way merges updated routing tests into `src/routing.test.ts` +- Installs the `@slack/bolt` npm dependency +- Updates `.env.example` with `SLACK_BOT_TOKEN`, `SLACK_APP_TOKEN`, and `SLACK_ONLY` +- Records the application in `.nanoclaw/state.yaml` + +If the apply reports merge conflicts, read the intent files: + +- `modify/src/index.ts.intent.md` — what changed and invariants for index.ts +- `modify/src/config.ts.intent.md` — what changed for config.ts +- `modify/src/routing.test.ts.intent.md` — what changed for routing tests + +### Validate code changes + +```bash +npm test +npm run build +``` + +All tests must pass (including the new slack tests) and build must be clean before proceeding. + +## Phase 3: Setup + +### Create Slack App (if needed) + +If the user doesn't have a Slack app, share [SLACK_SETUP.md](SLACK_SETUP.md) which has step-by-step instructions with screenshots guidance, troubleshooting, and a token reference table. + +Quick summary of what's needed: + +1. Create a Slack app at [api.slack.com/apps](https://api.slack.com/apps) +2. Enable Socket Mode and generate an App-Level Token (`xapp-...`) +3. Subscribe to bot events: `message.channels`, `message.groups`, `message.im` +4. Add OAuth scopes: `chat:write`, `channels:history`, `groups:history`, `im:history`, `channels:read`, `groups:read`, `users:read` +5. Install to workspace and copy the Bot Token (`xoxb-...`) + +Wait for the user to provide both tokens. + +### Configure environment + +Add to `.env`: + +```bash +SLACK_BOT_TOKEN=xoxb-your-bot-token +SLACK_APP_TOKEN=xapp-your-app-token +``` + +If they chose to replace WhatsApp: + +```bash +SLACK_ONLY=true +``` + +Sync to container environment: + +```bash +mkdir -p data/env && cp .env data/env/env +``` + +The container reads environment from `data/env/env`, not `.env` directly. + +### Build and restart + +```bash +npm run build +launchctl kickstart -k gui/$(id -u)/com.nanoclaw +``` + +## Phase 4: Registration + +### Get Channel ID + +Tell the user: + +> 1. Add the bot to a Slack channel (right-click channel → **View channel details** → **Integrations** → **Add apps**) +> 2. In that channel, the channel ID is in the URL when you open it in a browser: `https://app.slack.com/client/T.../C0123456789` — the `C...` part is the channel ID +> 3. Alternatively, right-click the channel name → **Copy link** — the channel ID is the last path segment +> +> The JID format for NanoClaw is: `slack:C0123456789` + +Wait for the user to provide the channel ID. + +### Register the channel + +Use the IPC register flow or register directly. The channel ID, name, and folder name are needed. + +For a main channel (responds to all messages, uses the `main` folder): + +```typescript +registerGroup('slack:', { + name: '', + folder: 'main', + trigger: `@${ASSISTANT_NAME}`, + added_at: new Date().toISOString(), + requiresTrigger: false, +}); +``` + +For additional channels (trigger-only): + +```typescript +registerGroup('slack:', { + name: '', + folder: '', + trigger: `@${ASSISTANT_NAME}`, + added_at: new Date().toISOString(), + requiresTrigger: true, +}); +``` + +## Phase 5: Verify + +### Test the connection + +Tell the user: + +> Send a message in your registered Slack channel: +> +> - For main channel: Any message works +> - For non-main: `@ hello` (using the configured trigger word) +> +> The bot should respond within a few seconds. + +### Check logs if needed + +```bash +tail -f logs/nanoclaw.log +``` + +## Troubleshooting + +### Bot not responding + +1. Check `SLACK_BOT_TOKEN` and `SLACK_APP_TOKEN` are set in `.env` AND synced to `data/env/env` +2. Check channel is registered: `psql "$OPS_DB_URL" -c "SELECT * FROM registered_groups WHERE jid LIKE 'slack:%'"` +3. For non-main channels: message must include trigger pattern +4. Service is running: `launchctl list | grep nanoclaw` + +### Bot connected but not receiving messages + +1. Verify Socket Mode is enabled in the Slack app settings +2. Verify the bot is subscribed to the correct events (`message.channels`, `message.groups`, `message.im`) +3. Verify the bot has been added to the channel +4. Check that the bot has the required OAuth scopes + +### Bot not seeing messages in channels + +By default, bots only see messages in channels they've been explicitly added to. Make sure to: + +1. Add the bot to each channel you want it to monitor +2. Check the bot has `channels:history` and/or `groups:history` scopes + +### "missing_scope" errors + +If the bot logs `missing_scope` errors: + +1. Go to **OAuth & Permissions** in your Slack app settings +2. Add the missing scope listed in the error message +3. **Reinstall the app** to your workspace — scope changes require reinstallation +4. Copy the new Bot Token (it changes on reinstall) and update `.env` +5. Sync: `mkdir -p data/env && cp .env data/env/env` +6. Restart: `launchctl kickstart -k gui/$(id -u)/com.nanoclaw` + +### Getting channel ID + +If the channel ID is hard to find: + +- In Slack desktop: right-click channel → **Copy link** → extract the `C...` ID from the URL +- In Slack web: the URL shows `https://app.slack.com/client/TXXXXXXX/C0123456789` +- Via API: `curl -s -H "Authorization: Bearer $SLACK_BOT_TOKEN" "https://slack.com/api/conversations.list" | jq '.channels[] | {id, name}'` + +## After Setup + +The Slack channel supports: + +- **Public channels** — Bot must be added to the channel +- **Private channels** — Bot must be invited to the channel +- **Direct messages** — Users can DM the bot directly +- **Multi-channel** — Can run alongside WhatsApp (default) or replace it (`SLACK_ONLY=true`) + +## Known Limitations + +- **Threads are flattened** — Threaded replies are delivered to the agent as regular channel messages. The agent sees them but has no awareness they originated in a thread. Responses always go to the channel, not back into the thread. Users in a thread will need to check the main channel for the bot's reply. Full thread-aware routing (respond in-thread) requires pipeline-wide changes: database schema, `NewMessage` type, `Channel.sendMessage` interface, and routing logic. +- **No typing indicator** — Slack's Bot API does not expose a typing indicator endpoint. The `setTyping()` method is a no-op. Users won't see "bot is typing..." while the agent works. +- **Message splitting is naive** — Long messages are split at a fixed 4000-character boundary, which may break mid-word or mid-sentence. A smarter split (on paragraph or sentence boundaries) would improve readability. +- **No file/image handling** — The bot only processes text content. File uploads, images, and rich message blocks are not forwarded to the agent. +- **Channel metadata sync is unbounded** — `syncChannelMetadata()` paginates through all channels the bot is a member of, but has no upper bound or timeout. Workspaces with thousands of channels may experience slow startup. +- **Workspace admin policies not detected** — If the Slack workspace restricts bot app installation, the setup will fail at the "Install to Workspace" step with no programmatic detection or guidance. See SLACK_SETUP.md troubleshooting section. diff --git a/.agent/skills/add-slack/SLACK_SETUP.md b/.agent/skills/add-slack/SLACK_SETUP.md new file mode 100644 index 0000000..a410f92 --- /dev/null +++ b/.agent/skills/add-slack/SLACK_SETUP.md @@ -0,0 +1,156 @@ +# Slack App Setup for NanoClaw + +Step-by-step guide to creating and configuring a Slack app for use with NanoClaw. + +## Prerequisites + +- A Slack workspace where you have admin permissions (or permission to install apps) +- Your NanoClaw instance with the `/add-slack` skill applied + +## Step 1: Create the Slack App + +1. Go to [api.slack.com/apps](https://api.slack.com/apps) +2. Click **Create New App** +3. Choose **From scratch** +4. Enter an app name (e.g., your `ASSISTANT_NAME` value, or any name you like) +5. Select the workspace you want to install it in +6. Click **Create App** + +## Step 2: Enable Socket Mode + +Socket Mode lets the bot connect to Slack without needing a public URL. This is what makes it work from your local machine. + +1. In the sidebar, click **Socket Mode** +2. Toggle **Enable Socket Mode** to **On** +3. When prompted for a token name, enter something like `nanoclaw` +4. Click **Generate** +5. **Copy the App-Level Token** — it starts with `xapp-`. Save this somewhere safe; you'll need it later. + +## Step 3: Subscribe to Events + +This tells Slack which messages to forward to your bot. + +1. In the sidebar, click **Event Subscriptions** +2. Toggle **Enable Events** to **On** +3. Under **Subscribe to bot events**, click **Add Bot User Event** and add these three events: + +| Event | What it does | +| ------------------ | -------------------------------------------------- | +| `message.channels` | Receive messages in public channels the bot is in | +| `message.groups` | Receive messages in private channels the bot is in | +| `message.im` | Receive direct messages to the bot | + +4. Click **Save Changes** at the bottom of the page + +## Step 4: Set Bot Permissions (OAuth Scopes) + +These scopes control what the bot is allowed to do. + +1. In the sidebar, click **OAuth & Permissions** +2. Scroll down to **Scopes** > **Bot Token Scopes** +3. Click **Add an OAuth Scope** and add each of these: + +| Scope | Why it's needed | +| ------------------ | ----------------------------------------- | +| `chat:write` | Send messages to channels and DMs | +| `channels:history` | Read messages in public channels | +| `groups:history` | Read messages in private channels | +| `im:history` | Read direct messages | +| `channels:read` | List channels (for metadata sync) | +| `groups:read` | List private channels (for metadata sync) | +| `users:read` | Look up user display names | + +## Step 5: Install to Workspace + +1. In the sidebar, click **Install App** +2. Click **Install to Workspace** +3. Review the permissions and click **Allow** +4. **Copy the Bot User OAuth Token** — it starts with `xoxb-`. Save this somewhere safe. + +## Step 6: Configure NanoClaw + +Add both tokens to your `.env` file: + +``` +SLACK_BOT_TOKEN=xoxb-your-bot-token-here +SLACK_APP_TOKEN=xapp-your-app-token-here +``` + +If you want Slack to replace WhatsApp entirely (no WhatsApp channel), also add: + +``` +SLACK_ONLY=true +``` + +Then sync the environment to the container: + +```bash +mkdir -p data/env && cp .env data/env/env +``` + +## Step 7: Add the Bot to Channels + +The bot only receives messages from channels it has been explicitly added to. + +1. Open the Slack channel you want the bot to monitor +2. Click the channel name at the top to open channel details +3. Go to **Integrations** > **Add apps** +4. Search for your bot name and add it + +Repeat for each channel you want the bot in. + +## Step 8: Get Channel IDs for Registration + +You need the Slack channel ID to register it with NanoClaw. + +**Option A — From the URL:** +Open the channel in Slack on the web. The URL looks like: + +``` +https://app.slack.com/client/TXXXXXXX/C0123456789 +``` + +The `C0123456789` part is the channel ID. + +**Option B — Right-click:** +Right-click the channel name in Slack > **Copy link** > the channel ID is the last path segment. + +**Option C — Via API:** + +```bash +curl -s -H "Authorization: Bearer $SLACK_BOT_TOKEN" \ + "https://slack.com/api/conversations.list" | jq '.channels[] | {id, name}' +``` + +The NanoClaw JID format is `slack:` followed by the channel ID, e.g., `slack:C0123456789`. + +## Token Reference + +| Token | Prefix | Where to find it | +| -------------------- | ------- | -------------------------------------------------------------------------- | +| Bot User OAuth Token | `xoxb-` | **OAuth & Permissions** > **Bot User OAuth Token** | +| App-Level Token | `xapp-` | **Basic Information** > **App-Level Tokens** (or during Socket Mode setup) | + +## Troubleshooting + +**Bot not receiving messages:** + +- Verify Socket Mode is enabled (Step 2) +- Verify all three events are subscribed (Step 3) +- Verify the bot has been added to the channel (Step 7) + +**"missing_scope" errors:** + +- Go back to **OAuth & Permissions** and add the missing scope +- After adding scopes, you must **reinstall the app** to your workspace (Slack will show a banner prompting you to do this) + +**Bot can't send messages:** + +- Verify the `chat:write` scope is added +- Verify the bot has been added to the target channel + +**Token not working:** + +- Bot tokens start with `xoxb-` — if yours doesn't, you may have copied the wrong token +- App tokens start with `xapp-` — these are generated in the Socket Mode or Basic Information pages +- If you regenerated a token, update `.env` and re-sync: `cp .env data/env/env` diff --git a/.agent/skills/add-slack/add/src/channels/slack.test.ts b/.agent/skills/add-slack/add/src/channels/slack.test.ts new file mode 100644 index 0000000..4c841d1 --- /dev/null +++ b/.agent/skills/add-slack/add/src/channels/slack.test.ts @@ -0,0 +1,848 @@ +import { describe, it, expect, beforeEach, vi, afterEach } from 'vitest'; + +// --- Mocks --- + +// Mock config +vi.mock('../config.js', () => ({ + ASSISTANT_NAME: 'Jonesy', + TRIGGER_PATTERN: /^@Jonesy\b/i, +})); + +// Mock logger +vi.mock('../logger.js', () => ({ + logger: { + debug: vi.fn(), + info: vi.fn(), + warn: vi.fn(), + error: vi.fn(), + }, +})); + +// Mock db +vi.mock('../db.js', () => ({ + updateChatName: vi.fn(), +})); + +// --- @slack/bolt mock --- + +type Handler = (...args: any[]) => any; + +const appRef = vi.hoisted(() => ({ current: null as any })); + +vi.mock('@slack/bolt', () => ({ + App: class MockApp { + eventHandlers = new Map(); + token: string; + appToken: string; + + client = { + auth: { + test: vi.fn().mockResolvedValue({ user_id: 'U_BOT_123' }), + }, + chat: { + postMessage: vi.fn().mockResolvedValue(undefined), + }, + conversations: { + list: vi.fn().mockResolvedValue({ + channels: [], + response_metadata: {}, + }), + }, + users: { + info: vi.fn().mockResolvedValue({ + user: { real_name: 'Alice Smith', name: 'alice' }, + }), + }, + }; + + constructor(opts: any) { + this.token = opts.token; + this.appToken = opts.appToken; + appRef.current = this; + } + + event(name: string, handler: Handler) { + this.eventHandlers.set(name, handler); + } + + async start() {} + async stop() {} + }, + LogLevel: { ERROR: 'error' }, +})); + +// Mock env +vi.mock('../env.js', () => ({ + readEnvFile: vi.fn().mockReturnValue({ + SLACK_BOT_TOKEN: 'xoxb-test-token', + SLACK_APP_TOKEN: 'xapp-test-token', + }), +})); + +import { SlackChannel, SlackChannelOpts } from './slack.js'; +import { updateChatName } from '../db.js'; +import { readEnvFile } from '../env.js'; + +// --- Test helpers --- + +function createTestOpts( + overrides?: Partial, +): SlackChannelOpts { + return { + onMessage: vi.fn(), + onChatMetadata: vi.fn(), + registeredGroups: vi.fn(() => ({ + 'slack:C0123456789': { + name: 'Test Channel', + folder: 'test-channel', + trigger: '@Jonesy', + added_at: '2024-01-01T00:00:00.000Z', + }, + })), + ...overrides, + }; +} + +function createMessageEvent(overrides: { + channel?: string; + channelType?: string; + user?: string; + text?: string; + ts?: string; + threadTs?: string; + subtype?: string; + botId?: string; +}) { + return { + channel: overrides.channel ?? 'C0123456789', + channel_type: overrides.channelType ?? 'channel', + user: overrides.user ?? 'U_USER_456', + text: 'text' in overrides ? overrides.text : 'Hello everyone', + ts: overrides.ts ?? '1704067200.000000', + thread_ts: overrides.threadTs, + subtype: overrides.subtype, + bot_id: overrides.botId, + }; +} + +function currentApp() { + return appRef.current; +} + +async function triggerMessageEvent(event: ReturnType) { + const handler = currentApp().eventHandlers.get('message'); + if (handler) await handler({ event }); +} + +// --- Tests --- + +describe('SlackChannel', () => { + beforeEach(() => { + vi.clearAllMocks(); + }); + + afterEach(() => { + vi.restoreAllMocks(); + }); + + // --- Connection lifecycle --- + + describe('connection lifecycle', () => { + it('resolves connect() when app starts', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + + await channel.connect(); + + expect(channel.isConnected()).toBe(true); + }); + + it('registers message event handler on construction', () => { + const opts = createTestOpts(); + new SlackChannel(opts); + + expect(currentApp().eventHandlers.has('message')).toBe(true); + }); + + it('gets bot user ID on connect', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + + await channel.connect(); + + expect(currentApp().client.auth.test).toHaveBeenCalled(); + }); + + it('disconnects cleanly', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + + await channel.connect(); + expect(channel.isConnected()).toBe(true); + + await channel.disconnect(); + expect(channel.isConnected()).toBe(false); + }); + + it('isConnected() returns false before connect', () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + + expect(channel.isConnected()).toBe(false); + }); + }); + + // --- Message handling --- + + describe('message handling', () => { + it('delivers message for registered channel', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + await channel.connect(); + + const event = createMessageEvent({ text: 'Hello everyone' }); + await triggerMessageEvent(event); + + expect(opts.onChatMetadata).toHaveBeenCalledWith( + 'slack:C0123456789', + expect.any(String), + undefined, + 'slack', + true, + ); + expect(opts.onMessage).toHaveBeenCalledWith( + 'slack:C0123456789', + expect.objectContaining({ + id: '1704067200.000000', + chat_jid: 'slack:C0123456789', + sender: 'U_USER_456', + content: 'Hello everyone', + is_from_me: false, + }), + ); + }); + + it('only emits metadata for unregistered channels', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + await channel.connect(); + + const event = createMessageEvent({ channel: 'C9999999999' }); + await triggerMessageEvent(event); + + expect(opts.onChatMetadata).toHaveBeenCalledWith( + 'slack:C9999999999', + expect.any(String), + undefined, + 'slack', + true, + ); + expect(opts.onMessage).not.toHaveBeenCalled(); + }); + + it('skips non-text subtypes (channel_join, etc.)', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + await channel.connect(); + + const event = createMessageEvent({ subtype: 'channel_join' }); + await triggerMessageEvent(event); + + expect(opts.onMessage).not.toHaveBeenCalled(); + expect(opts.onChatMetadata).not.toHaveBeenCalled(); + }); + + it('allows bot_message subtype through', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + await channel.connect(); + + const event = createMessageEvent({ + subtype: 'bot_message', + botId: 'B_OTHER_BOT', + text: 'Bot message', + }); + await triggerMessageEvent(event); + + expect(opts.onChatMetadata).toHaveBeenCalled(); + }); + + it('skips messages with no text', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + await channel.connect(); + + const event = createMessageEvent({ text: undefined as any }); + await triggerMessageEvent(event); + + expect(opts.onMessage).not.toHaveBeenCalled(); + }); + + it('detects bot messages by bot_id', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + await channel.connect(); + + const event = createMessageEvent({ + subtype: 'bot_message', + botId: 'B_MY_BOT', + text: 'Bot response', + }); + await triggerMessageEvent(event); + + // Has bot_id so should be marked as bot message + expect(opts.onMessage).toHaveBeenCalledWith( + 'slack:C0123456789', + expect.objectContaining({ + is_from_me: true, + is_bot_message: true, + sender_name: 'Jonesy', + }), + ); + }); + + it('detects bot messages by matching bot user ID', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + await channel.connect(); + + const event = createMessageEvent({ user: 'U_BOT_123', text: 'Self message' }); + await triggerMessageEvent(event); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'slack:C0123456789', + expect.objectContaining({ + is_from_me: true, + is_bot_message: true, + }), + ); + }); + + it('identifies IM channel type as non-group', async () => { + const opts = createTestOpts({ + registeredGroups: vi.fn(() => ({ + 'slack:D0123456789': { + name: 'DM', + folder: 'dm', + trigger: '@Jonesy', + added_at: '2024-01-01T00:00:00.000Z', + }, + })), + }); + const channel = new SlackChannel(opts); + await channel.connect(); + + const event = createMessageEvent({ + channel: 'D0123456789', + channelType: 'im', + }); + await triggerMessageEvent(event); + + expect(opts.onChatMetadata).toHaveBeenCalledWith( + 'slack:D0123456789', + expect.any(String), + undefined, + 'slack', + false, // IM is not a group + ); + }); + + it('converts ts to ISO timestamp', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + await channel.connect(); + + const event = createMessageEvent({ ts: '1704067200.000000' }); + await triggerMessageEvent(event); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'slack:C0123456789', + expect.objectContaining({ + timestamp: '2024-01-01T00:00:00.000Z', + }), + ); + }); + + it('resolves user name from Slack API', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + await channel.connect(); + + const event = createMessageEvent({ user: 'U_USER_456', text: 'Hello' }); + await triggerMessageEvent(event); + + expect(currentApp().client.users.info).toHaveBeenCalledWith({ + user: 'U_USER_456', + }); + expect(opts.onMessage).toHaveBeenCalledWith( + 'slack:C0123456789', + expect.objectContaining({ + sender_name: 'Alice Smith', + }), + ); + }); + + it('caches user names to avoid repeated API calls', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + await channel.connect(); + + // First message — API call + await triggerMessageEvent(createMessageEvent({ user: 'U_USER_456', text: 'First' })); + // Second message — should use cache + await triggerMessageEvent(createMessageEvent({ + user: 'U_USER_456', + text: 'Second', + ts: '1704067201.000000', + })); + + expect(currentApp().client.users.info).toHaveBeenCalledTimes(1); + }); + + it('falls back to user ID when API fails', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + await channel.connect(); + + currentApp().client.users.info.mockRejectedValueOnce(new Error('API error')); + + const event = createMessageEvent({ user: 'U_UNKNOWN', text: 'Hi' }); + await triggerMessageEvent(event); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'slack:C0123456789', + expect.objectContaining({ + sender_name: 'U_UNKNOWN', + }), + ); + }); + + it('flattens threaded replies into channel messages', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + await channel.connect(); + + const event = createMessageEvent({ + ts: '1704067201.000000', + threadTs: '1704067200.000000', // parent message ts — this is a reply + text: 'Thread reply', + }); + await triggerMessageEvent(event); + + // Threaded replies are delivered as regular channel messages + expect(opts.onMessage).toHaveBeenCalledWith( + 'slack:C0123456789', + expect.objectContaining({ + content: 'Thread reply', + }), + ); + }); + + it('delivers thread parent messages normally', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + await channel.connect(); + + const event = createMessageEvent({ + ts: '1704067200.000000', + threadTs: '1704067200.000000', // same as ts — this IS the parent + text: 'Thread parent', + }); + await triggerMessageEvent(event); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'slack:C0123456789', + expect.objectContaining({ + content: 'Thread parent', + }), + ); + }); + + it('delivers messages without thread_ts normally', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + await channel.connect(); + + const event = createMessageEvent({ text: 'Normal message' }); + await triggerMessageEvent(event); + + expect(opts.onMessage).toHaveBeenCalled(); + }); + }); + + // --- @mention translation --- + + describe('@mention translation', () => { + it('prepends trigger when bot is @mentioned via Slack format', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + await channel.connect(); // sets botUserId to 'U_BOT_123' + + const event = createMessageEvent({ + text: 'Hey <@U_BOT_123> what do you think?', + user: 'U_USER_456', + }); + await triggerMessageEvent(event); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'slack:C0123456789', + expect.objectContaining({ + content: '@Jonesy Hey <@U_BOT_123> what do you think?', + }), + ); + }); + + it('does not prepend trigger when trigger pattern already matches', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + await channel.connect(); + + const event = createMessageEvent({ + text: '@Jonesy <@U_BOT_123> hello', + user: 'U_USER_456', + }); + await triggerMessageEvent(event); + + // Content should be unchanged since it already matches TRIGGER_PATTERN + expect(opts.onMessage).toHaveBeenCalledWith( + 'slack:C0123456789', + expect.objectContaining({ + content: '@Jonesy <@U_BOT_123> hello', + }), + ); + }); + + it('does not translate mentions in bot messages', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + await channel.connect(); + + const event = createMessageEvent({ + text: 'Echo: <@U_BOT_123>', + subtype: 'bot_message', + botId: 'B_MY_BOT', + }); + await triggerMessageEvent(event); + + // Bot messages skip mention translation + expect(opts.onMessage).toHaveBeenCalledWith( + 'slack:C0123456789', + expect.objectContaining({ + content: 'Echo: <@U_BOT_123>', + }), + ); + }); + + it('does not translate mentions for other users', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + await channel.connect(); + + const event = createMessageEvent({ + text: 'Hey <@U_OTHER_USER> look at this', + user: 'U_USER_456', + }); + await triggerMessageEvent(event); + + // Mention is for a different user, not the bot + expect(opts.onMessage).toHaveBeenCalledWith( + 'slack:C0123456789', + expect.objectContaining({ + content: 'Hey <@U_OTHER_USER> look at this', + }), + ); + }); + }); + + // --- sendMessage --- + + describe('sendMessage', () => { + it('sends message via Slack client', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + await channel.connect(); + + await channel.sendMessage('slack:C0123456789', 'Hello'); + + expect(currentApp().client.chat.postMessage).toHaveBeenCalledWith({ + channel: 'C0123456789', + text: 'Hello', + }); + }); + + it('strips slack: prefix from JID', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + await channel.connect(); + + await channel.sendMessage('slack:D9876543210', 'DM message'); + + expect(currentApp().client.chat.postMessage).toHaveBeenCalledWith({ + channel: 'D9876543210', + text: 'DM message', + }); + }); + + it('queues message when disconnected', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + + // Don't connect — should queue + await channel.sendMessage('slack:C0123456789', 'Queued message'); + + expect(currentApp().client.chat.postMessage).not.toHaveBeenCalled(); + }); + + it('queues message on send failure', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + await channel.connect(); + + currentApp().client.chat.postMessage.mockRejectedValueOnce( + new Error('Network error'), + ); + + // Should not throw + await expect( + channel.sendMessage('slack:C0123456789', 'Will fail'), + ).resolves.toBeUndefined(); + }); + + it('splits long messages at 4000 character boundary', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + await channel.connect(); + + // Create a message longer than 4000 chars + const longText = 'A'.repeat(4500); + await channel.sendMessage('slack:C0123456789', longText); + + // Should be split into 2 messages: 4000 + 500 + expect(currentApp().client.chat.postMessage).toHaveBeenCalledTimes(2); + expect(currentApp().client.chat.postMessage).toHaveBeenNthCalledWith(1, { + channel: 'C0123456789', + text: 'A'.repeat(4000), + }); + expect(currentApp().client.chat.postMessage).toHaveBeenNthCalledWith(2, { + channel: 'C0123456789', + text: 'A'.repeat(500), + }); + }); + + it('sends exactly-4000-char messages as a single message', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + await channel.connect(); + + const text = 'B'.repeat(4000); + await channel.sendMessage('slack:C0123456789', text); + + expect(currentApp().client.chat.postMessage).toHaveBeenCalledTimes(1); + expect(currentApp().client.chat.postMessage).toHaveBeenCalledWith({ + channel: 'C0123456789', + text, + }); + }); + + it('splits messages into 3 parts when over 8000 chars', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + await channel.connect(); + + const longText = 'C'.repeat(8500); + await channel.sendMessage('slack:C0123456789', longText); + + // 4000 + 4000 + 500 = 3 messages + expect(currentApp().client.chat.postMessage).toHaveBeenCalledTimes(3); + }); + + it('flushes queued messages on connect', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + + // Queue messages while disconnected + await channel.sendMessage('slack:C0123456789', 'First queued'); + await channel.sendMessage('slack:C0123456789', 'Second queued'); + + expect(currentApp().client.chat.postMessage).not.toHaveBeenCalled(); + + // Connect triggers flush + await channel.connect(); + + expect(currentApp().client.chat.postMessage).toHaveBeenCalledWith({ + channel: 'C0123456789', + text: 'First queued', + }); + expect(currentApp().client.chat.postMessage).toHaveBeenCalledWith({ + channel: 'C0123456789', + text: 'Second queued', + }); + }); + }); + + // --- ownsJid --- + + describe('ownsJid', () => { + it('owns slack: JIDs', () => { + const channel = new SlackChannel(createTestOpts()); + expect(channel.ownsJid('slack:C0123456789')).toBe(true); + }); + + it('owns slack: DM JIDs', () => { + const channel = new SlackChannel(createTestOpts()); + expect(channel.ownsJid('slack:D0123456789')).toBe(true); + }); + + it('does not own WhatsApp group JIDs', () => { + const channel = new SlackChannel(createTestOpts()); + expect(channel.ownsJid('12345@g.us')).toBe(false); + }); + + it('does not own WhatsApp DM JIDs', () => { + const channel = new SlackChannel(createTestOpts()); + expect(channel.ownsJid('12345@s.whatsapp.net')).toBe(false); + }); + + it('does not own Telegram JIDs', () => { + const channel = new SlackChannel(createTestOpts()); + expect(channel.ownsJid('tg:123456')).toBe(false); + }); + + it('does not own unknown JID formats', () => { + const channel = new SlackChannel(createTestOpts()); + expect(channel.ownsJid('random-string')).toBe(false); + }); + }); + + // --- syncChannelMetadata --- + + describe('syncChannelMetadata', () => { + it('calls conversations.list and updates chat names', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + + currentApp().client.conversations.list.mockResolvedValue({ + channels: [ + { id: 'C001', name: 'general', is_member: true }, + { id: 'C002', name: 'random', is_member: true }, + { id: 'C003', name: 'external', is_member: false }, + ], + response_metadata: {}, + }); + + await channel.connect(); + + // connect() calls syncChannelMetadata internally + expect(updateChatName).toHaveBeenCalledWith('slack:C001', 'general'); + expect(updateChatName).toHaveBeenCalledWith('slack:C002', 'random'); + // Non-member channels are skipped + expect(updateChatName).not.toHaveBeenCalledWith('slack:C003', 'external'); + }); + + it('handles API errors gracefully', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + + currentApp().client.conversations.list.mockRejectedValue( + new Error('API error'), + ); + + // Should not throw + await expect(channel.connect()).resolves.toBeUndefined(); + }); + }); + + // --- setTyping --- + + describe('setTyping', () => { + it('resolves without error (no-op)', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + + // Should not throw — Slack has no bot typing indicator API + await expect( + channel.setTyping('slack:C0123456789', true), + ).resolves.toBeUndefined(); + }); + + it('accepts false without error', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + + await expect( + channel.setTyping('slack:C0123456789', false), + ).resolves.toBeUndefined(); + }); + }); + + // --- Constructor error handling --- + + describe('constructor', () => { + it('throws when SLACK_BOT_TOKEN is missing', () => { + vi.mocked(readEnvFile).mockReturnValueOnce({ + SLACK_BOT_TOKEN: '', + SLACK_APP_TOKEN: 'xapp-test-token', + }); + + expect(() => new SlackChannel(createTestOpts())).toThrow( + 'SLACK_BOT_TOKEN and SLACK_APP_TOKEN must be set in .env', + ); + }); + + it('throws when SLACK_APP_TOKEN is missing', () => { + vi.mocked(readEnvFile).mockReturnValueOnce({ + SLACK_BOT_TOKEN: 'xoxb-test-token', + SLACK_APP_TOKEN: '', + }); + + expect(() => new SlackChannel(createTestOpts())).toThrow( + 'SLACK_BOT_TOKEN and SLACK_APP_TOKEN must be set in .env', + ); + }); + }); + + // --- syncChannelMetadata pagination --- + + describe('syncChannelMetadata pagination', () => { + it('paginates through multiple pages of channels', async () => { + const opts = createTestOpts(); + const channel = new SlackChannel(opts); + + // First page returns a cursor; second page returns no cursor + currentApp().client.conversations.list + .mockResolvedValueOnce({ + channels: [ + { id: 'C001', name: 'general', is_member: true }, + ], + response_metadata: { next_cursor: 'cursor_page2' }, + }) + .mockResolvedValueOnce({ + channels: [ + { id: 'C002', name: 'random', is_member: true }, + ], + response_metadata: {}, + }); + + await channel.connect(); + + // Should have called conversations.list twice (once per page) + expect(currentApp().client.conversations.list).toHaveBeenCalledTimes(2); + expect(currentApp().client.conversations.list).toHaveBeenNthCalledWith(2, + expect.objectContaining({ cursor: 'cursor_page2' }), + ); + + // Both channels from both pages stored + expect(updateChatName).toHaveBeenCalledWith('slack:C001', 'general'); + expect(updateChatName).toHaveBeenCalledWith('slack:C002', 'random'); + }); + }); + + // --- Channel properties --- + + describe('channel properties', () => { + it('has name "slack"', () => { + const channel = new SlackChannel(createTestOpts()); + expect(channel.name).toBe('slack'); + }); + }); +}); diff --git a/.agent/skills/add-slack/add/src/channels/slack.ts b/.agent/skills/add-slack/add/src/channels/slack.ts new file mode 100644 index 0000000..81cc1ac --- /dev/null +++ b/.agent/skills/add-slack/add/src/channels/slack.ts @@ -0,0 +1,290 @@ +import { App, LogLevel } from '@slack/bolt'; +import type { GenericMessageEvent, BotMessageEvent } from '@slack/types'; + +import { ASSISTANT_NAME, TRIGGER_PATTERN } from '../config.js'; +import { updateChatName } from '../db.js'; +import { readEnvFile } from '../env.js'; +import { logger } from '../logger.js'; +import { + Channel, + OnInboundMessage, + OnChatMetadata, + RegisteredGroup, +} from '../types.js'; + +// Slack's chat.postMessage API limits text to ~4000 characters per call. +// Messages exceeding this are split into sequential chunks. +const MAX_MESSAGE_LENGTH = 4000; + +// The message subtypes we process. Bolt delivers all subtypes via app.event('message'); +// we filter to regular messages (GenericMessageEvent, subtype undefined) and bot messages +// (BotMessageEvent, subtype 'bot_message') so we can track our own output. +type HandledMessageEvent = GenericMessageEvent | BotMessageEvent; + +export interface SlackChannelOpts { + onMessage: OnInboundMessage; + onChatMetadata: OnChatMetadata; + registeredGroups: () => Record; +} + +export class SlackChannel implements Channel { + name = 'slack'; + + private app: App; + private botUserId: string | undefined; + private connected = false; + private outgoingQueue: Array<{ jid: string; text: string }> = []; + private flushing = false; + private userNameCache = new Map(); + + private opts: SlackChannelOpts; + + constructor(opts: SlackChannelOpts) { + this.opts = opts; + + // Read tokens from .env (not process.env — keeps secrets off the environment + // so they don't leak to child processes, matching NanoClaw's security pattern) + const env = readEnvFile(['SLACK_BOT_TOKEN', 'SLACK_APP_TOKEN']); + const botToken = env.SLACK_BOT_TOKEN; + const appToken = env.SLACK_APP_TOKEN; + + if (!botToken || !appToken) { + throw new Error( + 'SLACK_BOT_TOKEN and SLACK_APP_TOKEN must be set in .env', + ); + } + + this.app = new App({ + token: botToken, + appToken, + socketMode: true, + logLevel: LogLevel.ERROR, + }); + + this.setupEventHandlers(); + } + + private setupEventHandlers(): void { + // Use app.event('message') instead of app.message() to capture all + // message subtypes including bot_message (needed to track our own output) + this.app.event('message', async ({ event }) => { + // Bolt's event type is the full MessageEvent union (17+ subtypes). + // We filter on subtype first, then narrow to the two types we handle. + const subtype = (event as { subtype?: string }).subtype; + if (subtype && subtype !== 'bot_message') return; + + // After filtering, event is either GenericMessageEvent or BotMessageEvent + const msg = event as HandledMessageEvent; + + if (!msg.text) return; + + // Threaded replies are flattened into the channel conversation. + // The agent sees them alongside channel-level messages; responses + // always go to the channel, not back into the thread. + + const jid = `slack:${msg.channel}`; + const timestamp = new Date(parseFloat(msg.ts) * 1000).toISOString(); + const isGroup = msg.channel_type !== 'im'; + + // Always report metadata for group discovery + this.opts.onChatMetadata(jid, timestamp, undefined, 'slack', isGroup); + + // Only deliver full messages for registered groups + const groups = this.opts.registeredGroups(); + if (!groups[jid]) return; + + const isBotMessage = + !!msg.bot_id || msg.user === this.botUserId; + + let senderName: string; + if (isBotMessage) { + senderName = ASSISTANT_NAME; + } else { + senderName = + (await this.resolveUserName(msg.user)) || + msg.user || + 'unknown'; + } + + // Translate Slack <@UBOTID> mentions into TRIGGER_PATTERN format. + // Slack encodes @mentions as <@U12345>, which won't match TRIGGER_PATTERN + // (e.g., ^@\b), so we prepend the trigger when the bot is @mentioned. + let content = msg.text; + if (this.botUserId && !isBotMessage) { + const mentionPattern = `<@${this.botUserId}>`; + if (content.includes(mentionPattern) && !TRIGGER_PATTERN.test(content)) { + content = `@${ASSISTANT_NAME} ${content}`; + } + } + + this.opts.onMessage(jid, { + id: msg.ts, + chat_jid: jid, + sender: msg.user || msg.bot_id || '', + sender_name: senderName, + content, + timestamp, + is_from_me: isBotMessage, + is_bot_message: isBotMessage, + }); + }); + } + + async connect(): Promise { + await this.app.start(); + + // Get bot's own user ID for self-message detection. + // Resolve this BEFORE setting connected=true so that messages arriving + // during startup can correctly detect bot-sent messages. + try { + const auth = await this.app.client.auth.test(); + this.botUserId = auth.user_id as string; + logger.info({ botUserId: this.botUserId }, 'Connected to Slack'); + } catch (err) { + logger.warn( + { err }, + 'Connected to Slack but failed to get bot user ID', + ); + } + + this.connected = true; + + // Flush any messages queued before connection + await this.flushOutgoingQueue(); + + // Sync channel names on startup + await this.syncChannelMetadata(); + } + + async sendMessage(jid: string, text: string): Promise { + const channelId = jid.replace(/^slack:/, ''); + + if (!this.connected) { + this.outgoingQueue.push({ jid, text }); + logger.info( + { jid, queueSize: this.outgoingQueue.length }, + 'Slack disconnected, message queued', + ); + return; + } + + try { + // Slack limits messages to ~4000 characters; split if needed + if (text.length <= MAX_MESSAGE_LENGTH) { + await this.app.client.chat.postMessage({ channel: channelId, text }); + } else { + for (let i = 0; i < text.length; i += MAX_MESSAGE_LENGTH) { + await this.app.client.chat.postMessage({ + channel: channelId, + text: text.slice(i, i + MAX_MESSAGE_LENGTH), + }); + } + } + logger.info({ jid, length: text.length }, 'Slack message sent'); + } catch (err) { + this.outgoingQueue.push({ jid, text }); + logger.warn( + { jid, err, queueSize: this.outgoingQueue.length }, + 'Failed to send Slack message, queued', + ); + } + } + + isConnected(): boolean { + return this.connected; + } + + ownsJid(jid: string): boolean { + return jid.startsWith('slack:'); + } + + async disconnect(): Promise { + this.connected = false; + await this.app.stop(); + } + + // Slack does not expose a typing indicator API for bots. + // This no-op satisfies the Channel interface so the orchestrator + // doesn't need channel-specific branching. + async setTyping(_jid: string, _isTyping: boolean): Promise { + // no-op: Slack Bot API has no typing indicator endpoint + } + + /** + * Sync channel metadata from Slack. + * Fetches channels the bot is a member of and stores their names in the DB. + */ + async syncChannelMetadata(): Promise { + try { + logger.info('Syncing channel metadata from Slack...'); + let cursor: string | undefined; + let count = 0; + + do { + const result = await this.app.client.conversations.list({ + types: 'public_channel,private_channel', + exclude_archived: true, + limit: 200, + cursor, + }); + + for (const ch of result.channels || []) { + if (ch.id && ch.name && ch.is_member) { + updateChatName(`slack:${ch.id}`, ch.name); + count++; + } + } + + cursor = result.response_metadata?.next_cursor || undefined; + } while (cursor); + + logger.info({ count }, 'Slack channel metadata synced'); + } catch (err) { + logger.error({ err }, 'Failed to sync Slack channel metadata'); + } + } + + private async resolveUserName( + userId: string, + ): Promise { + if (!userId) return undefined; + + const cached = this.userNameCache.get(userId); + if (cached) return cached; + + try { + const result = await this.app.client.users.info({ user: userId }); + const name = result.user?.real_name || result.user?.name; + if (name) this.userNameCache.set(userId, name); + return name; + } catch (err) { + logger.debug({ userId, err }, 'Failed to resolve Slack user name'); + return undefined; + } + } + + private async flushOutgoingQueue(): Promise { + if (this.flushing || this.outgoingQueue.length === 0) return; + this.flushing = true; + try { + logger.info( + { count: this.outgoingQueue.length }, + 'Flushing Slack outgoing queue', + ); + while (this.outgoingQueue.length > 0) { + const item = this.outgoingQueue.shift()!; + const channelId = item.jid.replace(/^slack:/, ''); + await this.app.client.chat.postMessage({ + channel: channelId, + text: item.text, + }); + logger.info( + { jid: item.jid, length: item.text.length }, + 'Queued Slack message sent', + ); + } + } finally { + this.flushing = false; + } + } +} diff --git a/.agent/skills/add-slack/manifest.yaml b/.agent/skills/add-slack/manifest.yaml new file mode 100644 index 0000000..8320bb3 --- /dev/null +++ b/.agent/skills/add-slack/manifest.yaml @@ -0,0 +1,21 @@ +skill: slack +version: 1.0.0 +description: "Slack Bot integration via @slack/bolt with Socket Mode" +core_version: 0.1.0 +adds: + - src/channels/slack.ts + - src/channels/slack.test.ts +modifies: + - src/index.ts + - src/config.ts + - src/routing.test.ts +structured: + npm_dependencies: + "@slack/bolt": "^4.6.0" + env_additions: + - SLACK_BOT_TOKEN + - SLACK_APP_TOKEN + - SLACK_ONLY +conflicts: [] +depends: [] +test: "npx vitest run src/channels/slack.test.ts" diff --git a/.agent/skills/add-slack/modify/src/config.ts b/.agent/skills/add-slack/modify/src/config.ts new file mode 100644 index 0000000..257adcb --- /dev/null +++ b/.agent/skills/add-slack/modify/src/config.ts @@ -0,0 +1,75 @@ +import path from 'path'; + +import { readEnvFile } from './env.js'; + +// Read config values from .env (falls back to process.env). +// Secrets are NOT read here — they stay on disk and are loaded only +// where needed (container-runner.ts) to avoid leaking to child processes. +const envConfig = readEnvFile([ + 'ASSISTANT_NAME', + 'ASSISTANT_HAS_OWN_NUMBER', + 'SLACK_ONLY', +]); + +export const ASSISTANT_NAME = + process.env.ASSISTANT_NAME || envConfig.ASSISTANT_NAME || 'Andy'; +export const ASSISTANT_HAS_OWN_NUMBER = + (process.env.ASSISTANT_HAS_OWN_NUMBER || envConfig.ASSISTANT_HAS_OWN_NUMBER) === 'true'; +export const POLL_INTERVAL = 2000; +export const SCHEDULER_POLL_INTERVAL = 60000; + +// Absolute paths needed for container mounts +const PROJECT_ROOT = process.cwd(); +const HOME_DIR = process.env.HOME || '/Users/user'; + +// Mount security: allowlist stored OUTSIDE project root, never mounted into containers +export const MOUNT_ALLOWLIST_PATH = path.join( + HOME_DIR, + '.config', + (process.env.AGENT_NAME || 'clawdie') + '-cp', + 'mount-allowlist.json', +); +export const STORE_DIR = path.resolve(PROJECT_ROOT, 'store'); +export const GROUPS_DIR = path.resolve(PROJECT_ROOT, 'groups'); +export const DATA_DIR = path.resolve(PROJECT_ROOT, 'data'); +export const MAIN_GROUP_FOLDER = 'main'; + +export const CONTAINER_IMAGE = + process.env.CONTAINER_IMAGE || (process.env.AGENT_NAME || 'clawdie') + '-cp-agent:latest'; +export const CONTAINER_TIMEOUT = parseInt( + process.env.CONTAINER_TIMEOUT || '1800000', + 10, +); +export const CONTAINER_MAX_OUTPUT_SIZE = parseInt( + process.env.CONTAINER_MAX_OUTPUT_SIZE || '10485760', + 10, +); // 10MB default +export const IPC_POLL_INTERVAL = 1000; +export const IDLE_TIMEOUT = parseInt( + process.env.IDLE_TIMEOUT || '1800000', + 10, +); // 30min default — how long to keep container alive after last result +export const MAX_CONCURRENT_CONTAINERS = Math.max( + 1, + parseInt(process.env.MAX_CONCURRENT_CONTAINERS || '5', 10) || 5, +); + +function escapeRegex(str: string): string { + return str.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); +} + +export const TRIGGER_PATTERN = new RegExp( + `^@${escapeRegex(ASSISTANT_NAME)}\\b`, + 'i', +); + +// Timezone for scheduled tasks (cron expressions, etc.) +// Uses system timezone by default +export const TIMEZONE = + process.env.TZ || Intl.DateTimeFormat().resolvedOptions().timeZone; + +// Slack configuration +// SLACK_BOT_TOKEN and SLACK_APP_TOKEN are read directly by SlackChannel +// from .env via readEnvFile() to keep secrets off process.env. +export const SLACK_ONLY = + (process.env.SLACK_ONLY || envConfig.SLACK_ONLY) === 'true'; diff --git a/.agent/skills/add-slack/modify/src/config.ts.intent.md b/.agent/skills/add-slack/modify/src/config.ts.intent.md new file mode 100644 index 0000000..e92c690 --- /dev/null +++ b/.agent/skills/add-slack/modify/src/config.ts.intent.md @@ -0,0 +1,25 @@ +# Intent: src/config.ts modifications + +## What changed + +Added SLACK_ONLY configuration export for Slack channel support. + +## Key sections + +- **readEnvFile call**: Must include `SLACK_ONLY` in the keys array. NanoClaw does NOT load `.env` into `process.env` — all `.env` values must be explicitly requested via `readEnvFile()`. +- **SLACK_ONLY**: Boolean flag from `process.env` or `envConfig`, when `true` disables WhatsApp channel creation +- **Note**: SLACK_BOT_TOKEN and SLACK_APP_TOKEN are NOT read here. They are read directly by SlackChannel via `readEnvFile()` in `slack.ts` to keep secrets off the config module entirely (same pattern as ANTHROPIC_API_KEY in container-runner.ts). + +## Invariants + +- All existing config exports remain unchanged +- New Slack key is added to the `readEnvFile` call alongside existing keys +- New export is appended at the end of the file +- No existing behavior is modified — Slack config is additive only +- Both `process.env` and `envConfig` are checked (same pattern as `ASSISTANT_NAME`) + +## Must-keep + +- All existing exports (`ASSISTANT_NAME`, `POLL_INTERVAL`, `TRIGGER_PATTERN`, etc.) +- The `readEnvFile` pattern — ALL config read from `.env` must go through this function +- The `escapeRegex` helper and `TRIGGER_PATTERN` construction diff --git a/.agent/skills/add-slack/modify/src/index.ts b/.agent/skills/add-slack/modify/src/index.ts new file mode 100644 index 0000000..50212e1 --- /dev/null +++ b/.agent/skills/add-slack/modify/src/index.ts @@ -0,0 +1,498 @@ +import fs from 'fs'; +import path from 'path'; + +import { + ASSISTANT_NAME, + DATA_DIR, + IDLE_TIMEOUT, + MAIN_GROUP_FOLDER, + POLL_INTERVAL, + SLACK_ONLY, + TRIGGER_PATTERN, +} from './config.js'; +import { WhatsAppChannel } from './channels/whatsapp.js'; +import { SlackChannel } from './channels/slack.js'; +import { + ContainerOutput, + runContainerAgent, + writeGroupsSnapshot, + writeTasksSnapshot, +} from './container-runner.js'; +import { cleanupOrphans, ensureContainerRuntimeRunning } from './container-runtime.js'; +import { + getAllChats, + getAllRegisteredGroups, + getAllSessions, + getAllTasks, + getMessagesSince, + getNewMessages, + getRouterState, + initDatabase, + setRegisteredGroup, + setRouterState, + setSession, + storeChatMetadata, + storeMessage, +} from './db.js'; +import { GroupQueue } from './group-queue.js'; +import { startIpcWatcher } from './ipc.js'; +import { findChannel, formatMessages, formatOutbound } from './router.js'; +import { startSchedulerLoop } from './task-scheduler.js'; +import { Channel, NewMessage, RegisteredGroup } from './types.js'; +import { logger } from './logger.js'; +import { readEnvFile } from './env.js'; + +// Re-export for backwards compatibility during refactor +export { escapeXml, formatMessages } from './router.js'; + +let lastTimestamp = ''; +let sessions: Record = {}; +let registeredGroups: Record = {}; +let lastAgentTimestamp: Record = {}; +let messageLoopRunning = false; + +let whatsapp: WhatsAppChannel; +let slack: SlackChannel | undefined; +const channels: Channel[] = []; +const queue = new GroupQueue(); + +function loadState(): void { + lastTimestamp = getRouterState('last_timestamp') || ''; + const agentTs = getRouterState('last_agent_timestamp'); + try { + lastAgentTimestamp = agentTs ? JSON.parse(agentTs) : {}; + } catch { + logger.warn('Corrupted last_agent_timestamp in DB, resetting'); + lastAgentTimestamp = {}; + } + sessions = getAllSessions(); + registeredGroups = getAllRegisteredGroups(); + logger.info( + { groupCount: Object.keys(registeredGroups).length }, + 'State loaded', + ); +} + +function saveState(): void { + setRouterState('last_timestamp', lastTimestamp); + setRouterState( + 'last_agent_timestamp', + JSON.stringify(lastAgentTimestamp), + ); +} + +function registerGroup(jid: string, group: RegisteredGroup): void { + registeredGroups[jid] = group; + setRegisteredGroup(jid, group); + + // Create group folder + const groupDir = path.join(DATA_DIR, '..', 'groups', group.folder); + fs.mkdirSync(path.join(groupDir, 'logs'), { recursive: true }); + + logger.info( + { jid, name: group.name, folder: group.folder }, + 'Group registered', + ); +} + +/** + * Get available groups list for the agent. + * Returns groups ordered by most recent activity. + */ +export function getAvailableGroups(): import('./container-runner.js').AvailableGroup[] { + const chats = getAllChats(); + const registeredJids = new Set(Object.keys(registeredGroups)); + + return chats + .filter((c) => c.jid !== '__group_sync__' && c.is_group) + .map((c) => ({ + jid: c.jid, + name: c.name, + lastActivity: c.last_message_time, + isRegistered: registeredJids.has(c.jid), + })); +} + +/** @internal - exported for testing */ +export function _setRegisteredGroups(groups: Record): void { + registeredGroups = groups; +} + +/** + * Process all pending messages for a group. + * Called by the GroupQueue when it's this group's turn. + */ +async function processGroupMessages(chatJid: string): Promise { + const group = registeredGroups[chatJid]; + if (!group) return true; + + const channel = findChannel(channels, chatJid); + if (!channel) { + console.log(`Warning: no channel owns JID ${chatJid}, skipping messages`); + return true; + } + + const isMainGroup = group.folder === MAIN_GROUP_FOLDER; + + const sinceTimestamp = lastAgentTimestamp[chatJid] || ''; + const missedMessages = getMessagesSince(chatJid, sinceTimestamp, ASSISTANT_NAME); + + if (missedMessages.length === 0) return true; + + // For non-main groups, check if trigger is required and present + if (!isMainGroup && group.requiresTrigger !== false) { + const hasTrigger = missedMessages.some((m) => + TRIGGER_PATTERN.test(m.content.trim()), + ); + if (!hasTrigger) return true; + } + + const prompt = formatMessages(missedMessages); + + // Advance cursor so the piping path in startMessageLoop won't re-fetch + // these messages. Save the old cursor so we can roll back on error. + const previousCursor = lastAgentTimestamp[chatJid] || ''; + lastAgentTimestamp[chatJid] = + missedMessages[missedMessages.length - 1].timestamp; + saveState(); + + logger.info( + { group: group.name, messageCount: missedMessages.length }, + 'Processing messages', + ); + + // Track idle timer for closing stdin when agent is idle + let idleTimer: ReturnType | null = null; + + const resetIdleTimer = () => { + if (idleTimer) clearTimeout(idleTimer); + idleTimer = setTimeout(() => { + logger.debug({ group: group.name }, 'Idle timeout, closing container stdin'); + queue.closeStdin(chatJid); + }, IDLE_TIMEOUT); + }; + + await channel.setTyping?.(chatJid, true); + let hadError = false; + let outputSentToUser = false; + + const output = await runAgent(group, prompt, chatJid, async (result) => { + // Streaming output callback — called for each agent result + if (result.result) { + const raw = typeof result.result === 'string' ? result.result : JSON.stringify(result.result); + // Strip ... blocks — agent uses these for internal reasoning + const text = raw.replace(/[\s\S]*?<\/internal>/g, '').trim(); + logger.info({ group: group.name }, `Agent output: ${raw.slice(0, 200)}`); + if (text) { + await channel.sendMessage(chatJid, text); + outputSentToUser = true; + } + // Only reset idle timer on actual results, not session-update markers (result: null) + resetIdleTimer(); + } + + if (result.status === 'error') { + hadError = true; + } + }); + + await channel.setTyping?.(chatJid, false); + if (idleTimer) clearTimeout(idleTimer); + + if (output === 'error' || hadError) { + // If we already sent output to the user, don't roll back the cursor — + // the user got their response and re-processing would send duplicates. + if (outputSentToUser) { + logger.warn({ group: group.name }, 'Agent error after output was sent, skipping cursor rollback to prevent duplicates'); + return true; + } + // Roll back cursor so retries can re-process these messages + lastAgentTimestamp[chatJid] = previousCursor; + saveState(); + logger.warn({ group: group.name }, 'Agent error, rolled back message cursor for retry'); + return false; + } + + return true; +} + +async function runAgent( + group: RegisteredGroup, + prompt: string, + chatJid: string, + onOutput?: (output: ContainerOutput) => Promise, +): Promise<'success' | 'error'> { + const isMain = group.folder === MAIN_GROUP_FOLDER; + const sessionId = sessions[group.folder]; + + // Update tasks snapshot for container to read (filtered by group) + const tasks = getAllTasks(); + writeTasksSnapshot( + group.folder, + isMain, + tasks.map((t) => ({ + id: t.id, + groupFolder: t.group_folder, + prompt: t.prompt, + schedule_type: t.schedule_type, + schedule_value: t.schedule_value, + status: t.status, + next_run: t.next_run, + })), + ); + + // Update available groups snapshot (main group only can see all groups) + const availableGroups = getAvailableGroups(); + writeGroupsSnapshot( + group.folder, + isMain, + availableGroups, + new Set(Object.keys(registeredGroups)), + ); + + // Wrap onOutput to track session ID from streamed results + const wrappedOnOutput = onOutput + ? async (output: ContainerOutput) => { + if (output.newSessionId) { + sessions[group.folder] = output.newSessionId; + setSession(group.folder, output.newSessionId); + } + await onOutput(output); + } + : undefined; + + try { + const output = await runContainerAgent( + group, + { + prompt, + sessionId, + groupFolder: group.folder, + chatJid, + isMain, + }, + (proc, containerName) => queue.registerProcess(chatJid, proc, containerName, group.folder), + wrappedOnOutput, + ); + + if (output.newSessionId) { + sessions[group.folder] = output.newSessionId; + setSession(group.folder, output.newSessionId); + } + + if (output.status === 'error') { + logger.error( + { group: group.name, error: output.error }, + 'Container agent error', + ); + return 'error'; + } + + return 'success'; + } catch (err) { + logger.error({ group: group.name, err }, 'Agent error'); + return 'error'; + } +} + +async function startMessageLoop(): Promise { + if (messageLoopRunning) { + logger.debug('Message loop already running, skipping duplicate start'); + return; + } + messageLoopRunning = true; + + logger.info(`NanoClaw running (trigger: @${ASSISTANT_NAME})`); + + while (true) { + try { + const jids = Object.keys(registeredGroups); + const { messages, newTimestamp } = getNewMessages(jids, lastTimestamp, ASSISTANT_NAME); + + if (messages.length > 0) { + logger.info({ count: messages.length }, 'New messages'); + + // Advance the "seen" cursor for all messages immediately + lastTimestamp = newTimestamp; + saveState(); + + // Deduplicate by group + const messagesByGroup = new Map(); + for (const msg of messages) { + const existing = messagesByGroup.get(msg.chat_jid); + if (existing) { + existing.push(msg); + } else { + messagesByGroup.set(msg.chat_jid, [msg]); + } + } + + for (const [chatJid, groupMessages] of messagesByGroup) { + const group = registeredGroups[chatJid]; + if (!group) continue; + + const channel = findChannel(channels, chatJid); + if (!channel) { + console.log(`Warning: no channel owns JID ${chatJid}, skipping messages`); + continue; + } + + const isMainGroup = group.folder === MAIN_GROUP_FOLDER; + const needsTrigger = !isMainGroup && group.requiresTrigger !== false; + + // For non-main groups, only act on trigger messages. + // Non-trigger messages accumulate in DB and get pulled as + // context when a trigger eventually arrives. + if (needsTrigger) { + const hasTrigger = groupMessages.some((m) => + TRIGGER_PATTERN.test(m.content.trim()), + ); + if (!hasTrigger) continue; + } + + // Pull all messages since lastAgentTimestamp so non-trigger + // context that accumulated between triggers is included. + const allPending = getMessagesSince( + chatJid, + lastAgentTimestamp[chatJid] || '', + ASSISTANT_NAME, + ); + const messagesToSend = + allPending.length > 0 ? allPending : groupMessages; + const formatted = formatMessages(messagesToSend); + + if (queue.sendMessage(chatJid, formatted)) { + logger.debug( + { chatJid, count: messagesToSend.length }, + 'Piped messages to active container', + ); + lastAgentTimestamp[chatJid] = + messagesToSend[messagesToSend.length - 1].timestamp; + saveState(); + // Show typing indicator while the container processes the piped message + channel.setTyping?.(chatJid, true); + } else { + // No active container — enqueue for a new one + queue.enqueueMessageCheck(chatJid); + } + } + } + } catch (err) { + logger.error({ err }, 'Error in message loop'); + } + await new Promise((resolve) => setTimeout(resolve, POLL_INTERVAL)); + } +} + +/** + * Startup recovery: check for unprocessed messages in registered groups. + * Handles crash between advancing lastTimestamp and processing messages. + */ +function recoverPendingMessages(): void { + for (const [chatJid, group] of Object.entries(registeredGroups)) { + const sinceTimestamp = lastAgentTimestamp[chatJid] || ''; + const pending = getMessagesSince(chatJid, sinceTimestamp, ASSISTANT_NAME); + if (pending.length > 0) { + logger.info( + { group: group.name, pendingCount: pending.length }, + 'Recovery: found unprocessed messages', + ); + queue.enqueueMessageCheck(chatJid); + } + } +} + +function ensureContainerSystemRunning(): void { + ensureContainerRuntimeRunning(); + cleanupOrphans(); +} + +async function main(): Promise { + ensureContainerSystemRunning(); + initDatabase(); + logger.info('Database initialized'); + loadState(); + + // Graceful shutdown handlers + const shutdown = async (signal: string) => { + logger.info({ signal }, 'Shutdown signal received'); + await queue.shutdown(10000); + for (const ch of channels) await ch.disconnect(); + process.exit(0); + }; + process.on('SIGTERM', () => shutdown('SIGTERM')); + process.on('SIGINT', () => shutdown('SIGINT')); + + // Channel callbacks (shared by all channels) + const channelOpts = { + onMessage: (_chatJid: string, msg: NewMessage) => storeMessage(msg), + onChatMetadata: (chatJid: string, timestamp: string, name?: string, channel?: string, isGroup?: boolean) => + storeChatMetadata(chatJid, timestamp, name, channel, isGroup), + registeredGroups: () => registeredGroups, + }; + + // Create and connect channels + // Check if Slack tokens are configured + const slackEnv = readEnvFile(['SLACK_BOT_TOKEN', 'SLACK_APP_TOKEN']); + const hasSlackTokens = !!(slackEnv.SLACK_BOT_TOKEN && slackEnv.SLACK_APP_TOKEN); + + if (!SLACK_ONLY) { + whatsapp = new WhatsAppChannel(channelOpts); + channels.push(whatsapp); + await whatsapp.connect(); + } + + if (hasSlackTokens) { + slack = new SlackChannel(channelOpts); + channels.push(slack); + await slack.connect(); + } + + // Start subsystems (independently of connection handler) + startSchedulerLoop({ + registeredGroups: () => registeredGroups, + getSessions: () => sessions, + queue, + onProcess: (groupJid, proc, containerName, groupFolder) => queue.registerProcess(groupJid, proc, containerName, groupFolder), + sendMessage: async (jid, rawText) => { + const channel = findChannel(channels, jid); + if (!channel) { + console.log(`Warning: no channel owns JID ${jid}, cannot send message`); + return; + } + const text = formatOutbound(rawText); + if (text) await channel.sendMessage(jid, text); + }, + }); + startIpcWatcher({ + sendMessage: (jid, text) => { + const channel = findChannel(channels, jid); + if (!channel) throw new Error(`No channel for JID: ${jid}`); + return channel.sendMessage(jid, text); + }, + registeredGroups: () => registeredGroups, + registerGroup, + syncGroupMetadata: async (force) => { + // Sync metadata across all active channels + if (whatsapp) await whatsapp.syncGroupMetadata(force); + if (slack) await slack.syncChannelMetadata(); + }, + getAvailableGroups, + writeGroupsSnapshot: (gf, im, ag, rj) => writeGroupsSnapshot(gf, im, ag, rj), + }); + queue.setProcessMessagesFn(processGroupMessages); + recoverPendingMessages(); + startMessageLoop(); +} + +// Guard: only run when executed directly, not when imported by tests +const isDirectRun = + process.argv[1] && + new URL(import.meta.url).pathname === new URL(`file://${process.argv[1]}`).pathname; + +if (isDirectRun) { + main().catch((err) => { + logger.error({ err }, 'Failed to start NanoClaw'); + process.exit(1); + }); +} diff --git a/.agent/skills/add-slack/modify/src/index.ts.intent.md b/.agent/skills/add-slack/modify/src/index.ts.intent.md new file mode 100644 index 0000000..a7deec7 --- /dev/null +++ b/.agent/skills/add-slack/modify/src/index.ts.intent.md @@ -0,0 +1,70 @@ +# Intent: src/index.ts modifications + +## What changed + +Refactored from single WhatsApp channel to multi-channel architecture supporting Slack alongside WhatsApp. + +## Key sections + +### Imports (top of file) + +- Added: `SlackChannel` from `./channels/slack.js` +- Added: `SLACK_ONLY` from `./config.js` +- Added: `readEnvFile` from `./env.js` +- Existing: `findChannel` from `./router.js` and `Channel` type from `./types.js` are already present + +### Module-level state + +- Kept: `let whatsapp: WhatsAppChannel` — still needed for `syncGroupMetadata` reference +- Added: `let slack: SlackChannel | undefined` — direct reference for `syncChannelMetadata` +- Kept: `const channels: Channel[] = []` — array of all active channels + +### processGroupMessages() + +- Uses `findChannel(channels, chatJid)` lookup (already exists in base) +- Uses `channel.setTyping?.()` and `channel.sendMessage()` (already exists in base) + +### startMessageLoop() + +- Uses `findChannel(channels, chatJid)` per group (already exists in base) +- Uses `channel.setTyping?.()` for typing indicators (already exists in base) + +### main() + +- Added: Reads Slack tokens via `readEnvFile()` to check if Slack is configured +- Added: conditional WhatsApp creation (`if (!SLACK_ONLY)`) +- Added: conditional Slack creation (`if (hasSlackTokens)`) +- Changed: scheduler `sendMessage` uses `findChannel()` → `channel.sendMessage()` +- Changed: IPC `syncGroupMetadata` syncs both WhatsApp and Slack metadata +- Changed: IPC `sendMessage` uses `findChannel()` → `channel.sendMessage()` + +### Shutdown handler + +- Changed from `await whatsapp.disconnect()` to `for (const ch of channels) await ch.disconnect()` +- Disconnects all active channels (WhatsApp, Slack, or any future channels) on SIGTERM/SIGINT + +## Invariants + +- All existing message processing logic (triggers, cursors, idle timers) is preserved +- The `runAgent` function is completely unchanged +- State management (loadState/saveState) is unchanged +- Recovery logic is unchanged +- Container runtime check is unchanged (ensureContainerSystemRunning) + +## Design decisions + +### Double readEnvFile for Slack tokens + +`main()` in index.ts reads `SLACK_BOT_TOKEN`/`SLACK_APP_TOKEN` via `readEnvFile()` to check +whether Slack is configured (controls whether to instantiate SlackChannel). The SlackChannel +constructor reads them again independently. This is intentional — index.ts needs to decide +_whether_ to create the channel, while SlackChannel needs the actual token values. Keeping +both reads follows the security pattern of not passing secrets through intermediate variables. + +## Must-keep + +- The `escapeXml` and `formatMessages` re-exports +- The `_setRegisteredGroups` test helper +- The `isDirectRun` guard at bottom +- All error handling and cursor rollback logic in processGroupMessages +- The outgoing queue flush and reconnection logic (in each channel, not here) diff --git a/.agent/skills/add-slack/modify/src/routing.test.ts b/.agent/skills/add-slack/modify/src/routing.test.ts new file mode 100644 index 0000000..3a7f7ff --- /dev/null +++ b/.agent/skills/add-slack/modify/src/routing.test.ts @@ -0,0 +1,161 @@ +import { describe, it, expect, beforeEach } from 'vitest'; + +import { _initTestDatabase, getAllChats, storeChatMetadata } from './db.js'; +import { getAvailableGroups, _setRegisteredGroups } from './index.js'; + +beforeEach(() => { + _initTestDatabase(); + _setRegisteredGroups({}); +}); + +// --- JID ownership patterns --- + +describe('JID ownership patterns', () => { + // These test the patterns that will become ownsJid() on the Channel interface + + it('WhatsApp group JID: ends with @g.us', () => { + const jid = '12345678@g.us'; + expect(jid.endsWith('@g.us')).toBe(true); + }); + + it('WhatsApp DM JID: ends with @s.whatsapp.net', () => { + const jid = '12345678@s.whatsapp.net'; + expect(jid.endsWith('@s.whatsapp.net')).toBe(true); + }); + + it('Slack channel JID: starts with slack:', () => { + const jid = 'slack:C0123456789'; + expect(jid.startsWith('slack:')).toBe(true); + }); + + it('Slack DM JID: starts with slack:D', () => { + const jid = 'slack:D0123456789'; + expect(jid.startsWith('slack:')).toBe(true); + }); +}); + +// --- getAvailableGroups --- + +describe('getAvailableGroups', () => { + it('returns only groups, excludes DMs', () => { + storeChatMetadata('group1@g.us', '2024-01-01T00:00:01.000Z', 'Group 1', 'whatsapp', true); + storeChatMetadata('user@s.whatsapp.net', '2024-01-01T00:00:02.000Z', 'User DM', 'whatsapp', false); + storeChatMetadata('group2@g.us', '2024-01-01T00:00:03.000Z', 'Group 2', 'whatsapp', true); + + const groups = getAvailableGroups(); + expect(groups).toHaveLength(2); + expect(groups.map((g) => g.jid)).toContain('group1@g.us'); + expect(groups.map((g) => g.jid)).toContain('group2@g.us'); + expect(groups.map((g) => g.jid)).not.toContain('user@s.whatsapp.net'); + }); + + it('excludes __group_sync__ sentinel', () => { + storeChatMetadata('__group_sync__', '2024-01-01T00:00:00.000Z'); + storeChatMetadata('group@g.us', '2024-01-01T00:00:01.000Z', 'Group', 'whatsapp', true); + + const groups = getAvailableGroups(); + expect(groups).toHaveLength(1); + expect(groups[0].jid).toBe('group@g.us'); + }); + + it('marks registered groups correctly', () => { + storeChatMetadata('reg@g.us', '2024-01-01T00:00:01.000Z', 'Registered', 'whatsapp', true); + storeChatMetadata('unreg@g.us', '2024-01-01T00:00:02.000Z', 'Unregistered', 'whatsapp', true); + + _setRegisteredGroups({ + 'reg@g.us': { + name: 'Registered', + folder: 'registered', + trigger: '@Andy', + added_at: '2024-01-01T00:00:00.000Z', + }, + }); + + const groups = getAvailableGroups(); + const reg = groups.find((g) => g.jid === 'reg@g.us'); + const unreg = groups.find((g) => g.jid === 'unreg@g.us'); + + expect(reg?.isRegistered).toBe(true); + expect(unreg?.isRegistered).toBe(false); + }); + + it('returns groups ordered by most recent activity', () => { + storeChatMetadata('old@g.us', '2024-01-01T00:00:01.000Z', 'Old', 'whatsapp', true); + storeChatMetadata('new@g.us', '2024-01-01T00:00:05.000Z', 'New', 'whatsapp', true); + storeChatMetadata('mid@g.us', '2024-01-01T00:00:03.000Z', 'Mid', 'whatsapp', true); + + const groups = getAvailableGroups(); + expect(groups[0].jid).toBe('new@g.us'); + expect(groups[1].jid).toBe('mid@g.us'); + expect(groups[2].jid).toBe('old@g.us'); + }); + + it('excludes non-group chats regardless of JID format', () => { + // Unknown JID format stored without is_group should not appear + storeChatMetadata('unknown-format-123', '2024-01-01T00:00:01.000Z', 'Unknown'); + // Explicitly non-group with unusual JID + storeChatMetadata('custom:abc', '2024-01-01T00:00:02.000Z', 'Custom DM', 'custom', false); + // A real group for contrast + storeChatMetadata('group@g.us', '2024-01-01T00:00:03.000Z', 'Group', 'whatsapp', true); + + const groups = getAvailableGroups(); + expect(groups).toHaveLength(1); + expect(groups[0].jid).toBe('group@g.us'); + }); + + it('returns empty array when no chats exist', () => { + const groups = getAvailableGroups(); + expect(groups).toHaveLength(0); + }); + + it('includes Slack channel JIDs', () => { + storeChatMetadata('slack:C0123456789', '2024-01-01T00:00:01.000Z', 'Slack Channel', 'slack', true); + storeChatMetadata('user@s.whatsapp.net', '2024-01-01T00:00:02.000Z', 'User DM', 'whatsapp', false); + + const groups = getAvailableGroups(); + expect(groups).toHaveLength(1); + expect(groups[0].jid).toBe('slack:C0123456789'); + }); + + it('returns Slack DM JIDs as groups when is_group is true', () => { + storeChatMetadata('slack:D0123456789', '2024-01-01T00:00:01.000Z', 'Slack DM', 'slack', true); + + const groups = getAvailableGroups(); + expect(groups).toHaveLength(1); + expect(groups[0].jid).toBe('slack:D0123456789'); + expect(groups[0].name).toBe('Slack DM'); + }); + + it('marks registered Slack channels correctly', () => { + storeChatMetadata('slack:C0123456789', '2024-01-01T00:00:01.000Z', 'Slack Registered', 'slack', true); + storeChatMetadata('slack:C9999999999', '2024-01-01T00:00:02.000Z', 'Slack Unregistered', 'slack', true); + + _setRegisteredGroups({ + 'slack:C0123456789': { + name: 'Slack Registered', + folder: 'slack-registered', + trigger: '@Andy', + added_at: '2024-01-01T00:00:00.000Z', + }, + }); + + const groups = getAvailableGroups(); + const slackReg = groups.find((g) => g.jid === 'slack:C0123456789'); + const slackUnreg = groups.find((g) => g.jid === 'slack:C9999999999'); + + expect(slackReg?.isRegistered).toBe(true); + expect(slackUnreg?.isRegistered).toBe(false); + }); + + it('mixes WhatsApp and Slack chats ordered by activity', () => { + storeChatMetadata('wa@g.us', '2024-01-01T00:00:01.000Z', 'WhatsApp', 'whatsapp', true); + storeChatMetadata('slack:C100', '2024-01-01T00:00:03.000Z', 'Slack', 'slack', true); + storeChatMetadata('wa2@g.us', '2024-01-01T00:00:02.000Z', 'WhatsApp 2', 'whatsapp', true); + + const groups = getAvailableGroups(); + expect(groups).toHaveLength(3); + expect(groups[0].jid).toBe('slack:C100'); + expect(groups[1].jid).toBe('wa2@g.us'); + expect(groups[2].jid).toBe('wa@g.us'); + }); +}); diff --git a/.agent/skills/add-slack/modify/src/routing.test.ts.intent.md b/.agent/skills/add-slack/modify/src/routing.test.ts.intent.md new file mode 100644 index 0000000..4f310be --- /dev/null +++ b/.agent/skills/add-slack/modify/src/routing.test.ts.intent.md @@ -0,0 +1,21 @@ +# Intent: src/routing.test.ts modifications + +## What changed + +Added Slack JID pattern tests and Slack-specific getAvailableGroups tests. + +## Key sections + +- **JID ownership patterns**: Added Slack channel JID (`slack:C...`) and Slack DM JID (`slack:D...`) pattern tests +- **getAvailableGroups**: Added tests for Slack channel inclusion, Slack DM handling, registered Slack channels, and mixed WhatsApp + Slack ordering + +## Invariants + +- All existing WhatsApp JID pattern tests remain unchanged +- All existing getAvailableGroups tests remain unchanged +- New tests follow the same patterns as existing tests + +## Must-keep + +- All existing WhatsApp tests (group JID, DM JID patterns) +- All existing getAvailableGroups tests (DM exclusion, sentinel exclusion, registration, ordering, non-group exclusion, empty array) diff --git a/.agent/skills/add-slack/tests/slack.test.ts b/.agent/skills/add-slack/tests/slack.test.ts new file mode 100644 index 0000000..7e8d946 --- /dev/null +++ b/.agent/skills/add-slack/tests/slack.test.ts @@ -0,0 +1,171 @@ +import { describe, expect, it } from 'vitest'; +import fs from 'fs'; +import path from 'path'; + +describe('slack skill package', () => { + const skillDir = path.resolve(__dirname, '..'); + + it('has a valid manifest', () => { + const manifestPath = path.join(skillDir, 'manifest.yaml'); + expect(fs.existsSync(manifestPath)).toBe(true); + + const content = fs.readFileSync(manifestPath, 'utf-8'); + expect(content).toContain('skill: slack'); + expect(content).toContain('version: 1.0.0'); + expect(content).toContain('@slack/bolt'); + }); + + it('has all files declared in adds', () => { + const addFile = path.join(skillDir, 'add', 'src', 'channels', 'slack.ts'); + expect(fs.existsSync(addFile)).toBe(true); + + const content = fs.readFileSync(addFile, 'utf-8'); + expect(content).toContain('class SlackChannel'); + expect(content).toContain('implements Channel'); + + // Test file for the channel + const testFile = path.join(skillDir, 'add', 'src', 'channels', 'slack.test.ts'); + expect(fs.existsSync(testFile)).toBe(true); + + const testContent = fs.readFileSync(testFile, 'utf-8'); + expect(testContent).toContain("describe('SlackChannel'"); + }); + + it('has all files declared in modifies', () => { + const indexFile = path.join(skillDir, 'modify', 'src', 'index.ts'); + const configFile = path.join(skillDir, 'modify', 'src', 'config.ts'); + const routingTestFile = path.join(skillDir, 'modify', 'src', 'routing.test.ts'); + + expect(fs.existsSync(indexFile)).toBe(true); + expect(fs.existsSync(configFile)).toBe(true); + expect(fs.existsSync(routingTestFile)).toBe(true); + + const indexContent = fs.readFileSync(indexFile, 'utf-8'); + expect(indexContent).toContain('SlackChannel'); + expect(indexContent).toContain('SLACK_ONLY'); + expect(indexContent).toContain('findChannel'); + expect(indexContent).toContain('channels: Channel[]'); + + const configContent = fs.readFileSync(configFile, 'utf-8'); + expect(configContent).toContain('SLACK_ONLY'); + }); + + it('has intent files for modified files', () => { + expect(fs.existsSync(path.join(skillDir, 'modify', 'src', 'index.ts.intent.md'))).toBe(true); + expect(fs.existsSync(path.join(skillDir, 'modify', 'src', 'config.ts.intent.md'))).toBe(true); + expect(fs.existsSync(path.join(skillDir, 'modify', 'src', 'routing.test.ts.intent.md'))).toBe(true); + }); + + it('has setup documentation', () => { + expect(fs.existsSync(path.join(skillDir, 'SKILL.md'))).toBe(true); + expect(fs.existsSync(path.join(skillDir, 'SLACK_SETUP.md'))).toBe(true); + }); + + it('modified index.ts preserves core structure', () => { + const content = fs.readFileSync( + path.join(skillDir, 'modify', 'src', 'index.ts'), + 'utf-8', + ); + + // Core functions still present + expect(content).toContain('function loadState()'); + expect(content).toContain('function saveState()'); + expect(content).toContain('function registerGroup('); + expect(content).toContain('function getAvailableGroups()'); + expect(content).toContain('function processGroupMessages('); + expect(content).toContain('function runAgent('); + expect(content).toContain('function startMessageLoop()'); + expect(content).toContain('function recoverPendingMessages()'); + expect(content).toContain('function ensureContainerSystemRunning()'); + expect(content).toContain('async function main()'); + + // Test helper preserved + expect(content).toContain('_setRegisteredGroups'); + + // Direct-run guard preserved + expect(content).toContain('isDirectRun'); + }); + + it('modified index.ts includes Slack channel creation', () => { + const content = fs.readFileSync( + path.join(skillDir, 'modify', 'src', 'index.ts'), + 'utf-8', + ); + + // Multi-channel architecture + expect(content).toContain('const channels: Channel[] = []'); + expect(content).toContain('channels.push(whatsapp)'); + expect(content).toContain('channels.push(slack)'); + + // Conditional channel creation + expect(content).toContain('if (!SLACK_ONLY)'); + expect(content).toContain('new SlackChannel(channelOpts)'); + + // Shutdown disconnects all channels + expect(content).toContain('for (const ch of channels) await ch.disconnect()'); + }); + + it('modified config.ts preserves all existing exports', () => { + const content = fs.readFileSync( + path.join(skillDir, 'modify', 'src', 'config.ts'), + 'utf-8', + ); + + // All original exports preserved + expect(content).toContain('export const ASSISTANT_NAME'); + expect(content).toContain('export const POLL_INTERVAL'); + expect(content).toContain('export const TRIGGER_PATTERN'); + expect(content).toContain('export const CONTAINER_IMAGE'); + expect(content).toContain('export const DATA_DIR'); + expect(content).toContain('export const TIMEZONE'); + + // Slack config added + expect(content).toContain('export const SLACK_ONLY'); + }); + + it('modified routing.test.ts includes Slack JID tests', () => { + const content = fs.readFileSync( + path.join(skillDir, 'modify', 'src', 'routing.test.ts'), + 'utf-8', + ); + + // Slack JID pattern tests + expect(content).toContain('slack:C'); + expect(content).toContain('slack:D'); + + // Mixed ordering test + expect(content).toContain('mixes WhatsApp and Slack'); + + // All original WhatsApp tests preserved + expect(content).toContain('@g.us'); + expect(content).toContain('@s.whatsapp.net'); + expect(content).toContain('__group_sync__'); + }); + + it('slack.ts implements required Channel interface methods', () => { + const content = fs.readFileSync( + path.join(skillDir, 'add', 'src', 'channels', 'slack.ts'), + 'utf-8', + ); + + // Channel interface methods + expect(content).toContain('async connect()'); + expect(content).toContain('async sendMessage('); + expect(content).toContain('isConnected()'); + expect(content).toContain('ownsJid('); + expect(content).toContain('async disconnect()'); + expect(content).toContain('async setTyping('); + + // Security pattern: reads tokens from .env, not process.env + expect(content).toContain('readEnvFile'); + expect(content).not.toContain('process.env.SLACK_BOT_TOKEN'); + expect(content).not.toContain('process.env.SLACK_APP_TOKEN'); + + // Key behaviors + expect(content).toContain('socketMode: true'); + expect(content).toContain('MAX_MESSAGE_LENGTH'); + expect(content).toContain('thread_ts'); + expect(content).toContain('TRIGGER_PATTERN'); + expect(content).toContain('userNameCache'); + }); +}); diff --git a/.agent/skills/add-stripe/SKILL.md b/.agent/skills/add-stripe/SKILL.md new file mode 100644 index 0000000..9342654 --- /dev/null +++ b/.agent/skills/add-stripe/SKILL.md @@ -0,0 +1,84 @@ +--- +name: add-stripe +description: Reference guide for Stripe integration in Clawdie. Stripe ships in core on main — use this skill to understand the Stripe tool surface, webhook handling, and payment flows. +--- + +# Skill: add-stripe + +Current `main` already ships Stripe in core. This file remains as historical +reference for older branches and for understanding the Stripe tool surface. + +## Prerequisites + +- A Stripe account +- A **Restricted API Key** (RAK) with only the permissions you need + +## Setup + +### 1. Create a Restricted API Key + +1. Go to Stripe Dashboard → Developers → API Keys → Restricted Keys +2. Create a new key with only what your agent needs: + - **Customers**: Read + - **Payment Intents**: Read + - **Payment Links**: Write (if creating links) + - **Invoices**: Read + - **Subscriptions**: Read + - **Refunds**: Write (if issuing refunds) + - **Balance**: Read +3. Copy the key (starts with `rk_live_` or `rk_test_`) + +### 2. Add to .env + +``` +STRIPE_SECRET_KEY=rk_test_your_key_here +STRIPE_ENABLE_REFUNDS=NO +``` + +Current onboarding can configure a restricted test key (`rk_test_...`) or skip Stripe for now. + +### 3. Current main behavior + +There is no manual apply-skill step on current `main`. Stripe tools are built +into `jail/agent-runner` and become available when `STRIPE_SECRET_KEY` is set. + +## Tools the agent gets + +| Tool | Description | +| ----------------------------- | ---------------------------------------------------------------------------------- | +| `stripe_get_balance` | Current account balance | +| `stripe_list_customers` | List customers, filter by email | +| `stripe_get_customer` | Get a customer by ID | +| `stripe_list_payment_intents` | Recent payment intents | +| `stripe_create_payment_link` | Create a payment link for a price | +| `stripe_list_invoices` | List invoices, filter by customer/status | +| `stripe_create_refund` | Issue a refund by payment intent or charge (only when `STRIPE_ENABLE_REFUNDS=YES`) | +| `stripe_list_subscriptions` | List subscriptions, filter by customer/status | + +Tools are registered only when `STRIPE_SECRET_KEY` is set. Refunds remain off by +default until `STRIPE_ENABLE_REFUNDS=YES`. + +## Example conversations + +> "How much did we make this month?" +> → Agent calls `stripe_get_balance` and lists recent payment intents + +> "Create a payment link for the Pro plan" +> → Agent calls `stripe_create_payment_link` with the price ID + +> "John wants a refund for last month's invoice" +> → Agent calls `stripe_list_customers` to find John, then `stripe_list_invoices`, +> then `stripe_create_refund` + +## Security + +- Use a Restricted API Key — never your full secret key +- Limit permissions to what the agent actually needs +- `STRIPE_SECRET_KEY` is passed in the jailed stdin payload, never written as a mounted file +- Refunds stay disabled until you deliberately enable them + +## Upgrading to full channel + +Future versions of this skill may add inbound webhook handling so the agent +can react to Stripe events (payment succeeded, subscription cancelled, etc.) +automatically. For now, all interactions are agent-initiated. diff --git a/.agent/skills/add-stripe/add/jail/agent-runner/src/stripe-tools.ts b/.agent/skills/add-stripe/add/jail/agent-runner/src/stripe-tools.ts new file mode 100644 index 0000000..7e694c0 --- /dev/null +++ b/.agent/skills/add-stripe/add/jail/agent-runner/src/stripe-tools.ts @@ -0,0 +1,275 @@ +/** + * Stripe MCP tools for Clawdie agent. + * Registered into the existing ipc-mcp-stdio server when STRIPE_SECRET_KEY is set. + * Uses a Restricted API Key — never the full secret key. + */ +import type { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js'; +import type Stripe from 'stripe'; +import { z } from 'zod'; + +export function registerStripeTools(server: McpServer): void { + const key = process.env.STRIPE_SECRET_KEY; + if (!key) return; + + let stripeInstance: Stripe | null = null; + + const getStripe = async (): Promise => { + if (!stripeInstance) { + const { default: StripeClass } = await import('stripe') as { default: typeof Stripe }; + stripeInstance = new StripeClass(key); + } + return stripeInstance; + }; + + server.tool( + 'stripe_get_balance', + 'Get the current Stripe account balance (available and pending)', + {}, + async () => { + try { + const stripe = await getStripe(); + const balance = await stripe.balance.retrieve(); + const lines = balance.available.map( + (b) => `${b.currency.toUpperCase()} available: ${(b.amount / 100).toFixed(2)}`, + ); + const pending = balance.pending.map( + (b) => `${b.currency.toUpperCase()} pending: ${(b.amount / 100).toFixed(2)}`, + ); + return { + content: [{ type: 'text' as const, text: [...lines, ...pending].join('\n') }], + }; + } catch (err) { + return { + content: [{ type: 'text' as const, text: `Stripe error: ${err instanceof Error ? err.message : String(err)}` }], + isError: true, + }; + } + }, + ); + + server.tool( + 'stripe_list_customers', + 'List Stripe customers. Optionally filter by email.', + { + email: z.string().optional().describe('Filter by exact email address'), + limit: z.number().int().min(1).max(100).default(10).optional().describe('Max results, default 10'), + }, + async (args) => { + try { + const stripe = await getStripe(); + const params: Stripe.CustomerListParams = { limit: args.limit ?? 10 }; + if (args.email) params.email = args.email; + const customers = await stripe.customers.list(params); + const lines = customers.data.map( + (c) => `${c.id} | ${c.email || 'no email'} | ${c.name || 'no name'} | created: ${new Date(c.created * 1000).toISOString()}`, + ); + return { + content: [{ type: 'text' as const, text: lines.length > 0 ? lines.join('\n') : 'No customers found.' }], + }; + } catch (err) { + return { + content: [{ type: 'text' as const, text: `Stripe error: ${err instanceof Error ? err.message : String(err)}` }], + isError: true, + }; + } + }, + ); + + server.tool( + 'stripe_get_customer', + 'Get a Stripe customer by their customer ID', + { + customer_id: z.string().describe('Stripe customer ID (cus_...)'), + }, + async (args) => { + try { + const stripe = await getStripe(); + const customer = await stripe.customers.retrieve(args.customer_id); + return { + content: [{ type: 'text' as const, text: JSON.stringify(customer, null, 2) }], + }; + } catch (err) { + return { + content: [{ type: 'text' as const, text: `Stripe error: ${err instanceof Error ? err.message : String(err)}` }], + isError: true, + }; + } + }, + ); + + server.tool( + 'stripe_list_payment_intents', + 'List recent Stripe payment intents, optionally filtered by customer', + { + customer_id: z.string().optional().describe('Filter by customer ID (cus_...)'), + limit: z.number().int().min(1).max(100).default(10).optional().describe('Max results, default 10'), + }, + async (args) => { + try { + const stripe = await getStripe(); + const params: Stripe.PaymentIntentListParams = { limit: args.limit ?? 10 }; + if (args.customer_id) params.customer = args.customer_id; + const intents = await stripe.paymentIntents.list(params); + const lines = intents.data.map( + (pi) => + `${pi.id} | ${((pi.amount ?? 0) / 100).toFixed(2)} ${(pi.currency ?? '').toUpperCase()} | ${pi.status} | ${new Date(pi.created * 1000).toISOString()}`, + ); + return { + content: [{ type: 'text' as const, text: lines.length > 0 ? lines.join('\n') : 'No payment intents found.' }], + }; + } catch (err) { + return { + content: [{ type: 'text' as const, text: `Stripe error: ${err instanceof Error ? err.message : String(err)}` }], + isError: true, + }; + } + }, + ); + + server.tool( + 'stripe_create_payment_link', + 'Create a Stripe payment link for a product price. Returns the URL to share with customers.', + { + price_id: z.string().describe('Stripe price ID (price_...) — find in Products → Prices'), + quantity: z.number().int().min(1).default(1).optional().describe('Quantity, default 1'), + }, + async (args) => { + try { + const stripe = await getStripe(); + const link = await stripe.paymentLinks.create({ + line_items: [{ price: args.price_id, quantity: args.quantity ?? 1 }], + }); + return { + content: [{ type: 'text' as const, text: `Payment link created:\nURL: ${link.url}\nID: ${link.id}\nActive: ${link.active}` }], + }; + } catch (err) { + return { + content: [{ type: 'text' as const, text: `Stripe error: ${err instanceof Error ? err.message : String(err)}` }], + isError: true, + }; + } + }, + ); + + server.tool( + 'stripe_list_invoices', + 'List Stripe invoices, optionally filtered by customer or status', + { + customer_id: z.string().optional().describe('Filter by customer ID (cus_...)'), + status: z + .enum(['draft', 'open', 'paid', 'uncollectible', 'void']) + .optional() + .describe('Filter by invoice status'), + limit: z.number().int().min(1).max(100).default(10).optional().describe('Max results, default 10'), + }, + async (args) => { + try { + const stripe = await getStripe(); + const params: Stripe.InvoiceListParams = { limit: args.limit ?? 10 }; + if (args.customer_id) params.customer = args.customer_id; + if (args.status) params.status = args.status; + const invoices = await stripe.invoices.list(params); + const lines = invoices.data.map( + (inv) => + `${inv.id} | ${inv.customer_email || inv.customer} | ${((inv.amount_due ?? 0) / 100).toFixed(2)} ${(inv.currency ?? '').toUpperCase()} | ${inv.status} | ${new Date(inv.created * 1000).toISOString()}${inv.hosted_invoice_url ? ` | ${inv.hosted_invoice_url}` : ''}`, + ); + return { + content: [{ type: 'text' as const, text: lines.length > 0 ? lines.join('\n') : 'No invoices found.' }], + }; + } catch (err) { + return { + content: [{ type: 'text' as const, text: `Stripe error: ${err instanceof Error ? err.message : String(err)}` }], + isError: true, + }; + } + }, + ); + + server.tool( + 'stripe_create_refund', + 'Issue a full or partial refund for a payment intent or charge', + { + payment_intent_id: z + .string() + .optional() + .describe('Payment intent ID to refund (pi_...). Provide this OR charge_id.'), + charge_id: z + .string() + .optional() + .describe('Charge ID to refund (ch_...). Provide this OR payment_intent_id.'), + amount: z + .number() + .int() + .optional() + .describe('Amount in smallest currency unit (e.g. cents). Omit for full refund.'), + reason: z + .enum(['duplicate', 'fraudulent', 'requested_by_customer']) + .optional() + .describe('Reason for refund'), + }, + async (args) => { + if (!args.payment_intent_id && !args.charge_id) { + return { + content: [{ type: 'text' as const, text: 'Provide either payment_intent_id or charge_id.' }], + isError: true, + }; + } + try { + const stripe = await getStripe(); + const params: Stripe.RefundCreateParams = {}; + if (args.payment_intent_id) params.payment_intent = args.payment_intent_id; + if (args.charge_id) params.charge = args.charge_id; + if (args.amount) params.amount = args.amount; + if (args.reason) params.reason = args.reason; + const refund = await stripe.refunds.create(params); + return { + content: [{ + type: 'text' as const, + text: `Refund created:\nID: ${refund.id}\nAmount: ${((refund.amount ?? 0) / 100).toFixed(2)} ${(refund.currency ?? '').toUpperCase()}\nStatus: ${refund.status}`, + }], + }; + } catch (err) { + return { + content: [{ type: 'text' as const, text: `Stripe error: ${err instanceof Error ? err.message : String(err)}` }], + isError: true, + }; + } + }, + ); + + server.tool( + 'stripe_list_subscriptions', + 'List Stripe subscriptions, optionally filtered by customer or status', + { + customer_id: z.string().optional().describe('Filter by customer ID (cus_...)'), + status: z + .enum(['active', 'past_due', 'unpaid', 'canceled', 'incomplete', 'trialing', 'all']) + .optional() + .describe('Filter by status. Default: active'), + limit: z.number().int().min(1).max(100).default(10).optional().describe('Max results, default 10'), + }, + async (args) => { + try { + const stripe = await getStripe(); + const params: Stripe.SubscriptionListParams = { + limit: args.limit ?? 10, + status: (args.status as Stripe.SubscriptionListParams['status']) ?? 'active', + }; + if (args.customer_id) params.customer = args.customer_id; + const subs = await stripe.subscriptions.list(params); + const lines = subs.data.map( + (s) => + `${s.id} | customer: ${typeof s.customer === 'string' ? s.customer : s.customer?.id} | ${s.status} | renews: ${new Date(s.current_period_end * 1000).toISOString()}${s.cancel_at_period_end ? ' (cancels at period end)' : ''}`, + ); + return { + content: [{ type: 'text' as const, text: lines.length > 0 ? lines.join('\n') : 'No subscriptions found.' }], + }; + } catch (err) { + return { + content: [{ type: 'text' as const, text: `Stripe error: ${err instanceof Error ? err.message : String(err)}` }], + isError: true, + }; + } + }, + ); +} diff --git a/.agent/skills/add-stripe/manifest.yaml b/.agent/skills/add-stripe/manifest.yaml new file mode 100644 index 0000000..8cd2013 --- /dev/null +++ b/.agent/skills/add-stripe/manifest.yaml @@ -0,0 +1,16 @@ +skill: stripe +version: 1.0.0 +description: "Stripe payments, customers, invoices, subscriptions, refunds, and payment links" +core_version: 0.4.0 +adds: + - jail/agent-runner/src/stripe-tools.ts +modifies: + - src/jail-runner.ts + - jail/agent-runner/src/ipc-mcp-stdio.ts + - jail/agent-runner/package.json +structured: + npm_dependencies: + stripe: "^17.0.0" +conflicts: [] +depends: [] +test: "" diff --git a/.agent/skills/add-stripe/modify/jail/agent-runner/package.json.intent.md b/.agent/skills/add-stripe/modify/jail/agent-runner/package.json.intent.md new file mode 100644 index 0000000..14fede7 --- /dev/null +++ b/.agent/skills/add-stripe/modify/jail/agent-runner/package.json.intent.md @@ -0,0 +1,28 @@ +# Intent: jail/agent-runner/package.json modifications + +## What changed + +Added `stripe` as a runtime dependency. + +## Key sections + +### dependencies + +- Added: `"stripe": "^17.0.0"` + +```json +{ + "dependencies": { + "@modelcontextprotocol/sdk": "^1.12.1", + "cron-parser": "^5.0.0", + "stripe": "^17.0.0", + "zod": "^4.0.0" + } +} +``` + +## Invariants + +- All existing dependencies remain unchanged +- devDependencies are unchanged +- package name, version, scripts are unchanged diff --git a/.agent/skills/add-stripe/modify/jail/agent-runner/src/ipc-mcp-stdio.ts.intent.md b/.agent/skills/add-stripe/modify/jail/agent-runner/src/ipc-mcp-stdio.ts.intent.md new file mode 100644 index 0000000..41f66fb --- /dev/null +++ b/.agent/skills/add-stripe/modify/jail/agent-runner/src/ipc-mcp-stdio.ts.intent.md @@ -0,0 +1,36 @@ +# Intent: jail/agent-runner/src/ipc-mcp-stdio.ts modifications + +## What changed + +Imported and registered the Stripe MCP tools so the agent can access Stripe +operations (customers, invoices, payment links, refunds, subscriptions, balance). +Tools are only registered when `STRIPE_SECRET_KEY` is present in the environment. + +## Key sections + +### Imports (top of file) + +- Added: `import { registerStripeTools } from './stripe-tools.js';` + +### After all existing server.tool() calls, before the transport connection + +- Added: `registerStripeTools(server);` + +```typescript +// Add near the bottom, before: +// const transport = new StdioServerTransport(); + +registerStripeTools(server); + +// Existing: +const transport = new StdioServerTransport(); +await server.connect(transport); +``` + +## Invariants + +- All existing tools (send_message, schedule_task, list_tasks, etc.) are unchanged +- Server name and version are unchanged +- IPC directory constants are unchanged +- Transport connection is unchanged +- If `STRIPE_SECRET_KEY` is absent, `registerStripeTools` is a no-op diff --git a/.agent/skills/add-stripe/modify/src/jail-runner.ts.intent.md b/.agent/skills/add-stripe/modify/src/jail-runner.ts.intent.md new file mode 100644 index 0000000..df034ac --- /dev/null +++ b/.agent/skills/add-stripe/modify/src/jail-runner.ts.intent.md @@ -0,0 +1,37 @@ +# Intent: src/jail-runner.ts modifications + +## What changed + +Added `STRIPE_SECRET_KEY` to the secrets allowlist so the Stripe API key is +passed to the jail agent securely via stdin — never via environment variables +or files. + +## Key sections + +### readSecrets() function (around line 216) + +- Added: `'STRIPE_SECRET_KEY'` to the `readEnvFile` array + +```typescript +// Before: +return readEnvFile([ + 'ANTHROPIC_API_KEY', + ... + 'KIMI_API_KEY', +]); + +// After: +return readEnvFile([ + 'ANTHROPIC_API_KEY', + ... + 'KIMI_API_KEY', + 'STRIPE_SECRET_KEY', +]); +``` + +## Invariants + +- All existing API keys remain in the list unchanged +- `readEnvFile()` signature and behavior are unchanged +- Secrets are still passed via stdin JSON, never written to disk or env +- If `STRIPE_SECRET_KEY` is absent from `.env`, it is silently skipped diff --git a/.agent/skills/add-telegram-swarm/SKILL.md b/.agent/skills/add-telegram-swarm/SKILL.md new file mode 100644 index 0000000..2dfa795 --- /dev/null +++ b/.agent/skills/add-telegram-swarm/SKILL.md @@ -0,0 +1,388 @@ +--- +name: add-telegram-swarm +description: Add Agent Swarm (Teams) support to Telegram. Each subagent gets its own bot identity in the group. Requires Telegram channel to be set up first (use /add-telegram). Triggers on "agent swarm", "agent teams telegram", "telegram swarm", "bot pool". +--- + +# Add Agent Swarm to Telegram + +This skill adds Agent Teams (Swarm) support to an existing Telegram channel. Each subagent in a team gets its own bot identity in the Telegram group, so users can visually distinguish which agent is speaking. + +**Prerequisite**: Telegram must already be set up via the `/add-telegram` skill. If `src/telegram.ts` does not exist or `TELEGRAM_BOT_TOKEN` is not configured, tell the user to run `/add-telegram` first. + +## How It Works + +- The **main bot** receives messages and sends lead agent responses (already set up by `/add-telegram`) +- **Pool bots** are send-only — each gets a Grammy `Api` instance (no polling) +- When a subagent calls `send_message` with a `sender` parameter, the host assigns a pool bot and renames it to match the sender's role +- Messages appear in Telegram from different bot identities + +``` +Subagent calls send_message(text: "Found 3 results", sender: "Researcher") + → MCP writes IPC file with sender field + → Host IPC watcher picks it up + → Assigns pool bot #2 to "Researcher" (round-robin, stable per-group) + → Renames pool bot #2 to "Researcher" via setMyName + → Sends message via pool bot #2's Api instance + → Appears in Telegram from "Researcher" bot +``` + +## Prerequisites + +### 1. Create Pool Bots + +Tell the user: + +> I need you to create 3-5 Telegram bots to use as the agent pool. These will be renamed dynamically to match agent roles. +> +> 1. Open Telegram and search for `@BotFather` +> 2. Send `/newbot` for each bot: +> - Give them any placeholder name (e.g., "Bot 1", "Bot 2") +> - Usernames like `myproject_swarm_1_bot`, `myproject_swarm_2_bot`, etc. +> 3. Copy all the tokens +> 4. Add all bots to your Telegram group(s) where you want agent teams + +Wait for user to provide the tokens. + +### 2. Disable Group Privacy for Pool Bots + +Tell the user: + +> **Important**: Each pool bot needs Group Privacy disabled so it can send messages in groups. +> +> For each pool bot in `@BotFather`: +> +> 1. Send `/mybots` and select the bot +> 2. Go to **Bot Settings** > **Group Privacy** > **Turn off** +> +> Then add all pool bots to your Telegram group(s). + +## Implementation + +### Step 1: Update Configuration + +Read `src/config.ts` and add the bot pool config near the other Telegram exports: + +```typescript +export const TELEGRAM_BOT_POOL = (process.env.TELEGRAM_BOT_POOL || '') + .split(',') + .map((t) => t.trim()) + .filter(Boolean); +``` + +### Step 2: Add Bot Pool to Telegram Module + +Read `src/telegram.ts` and add the following: + +1. **Update imports** — add `Api` to the Grammy import: + +```typescript +import { Api, Bot } from 'grammy'; +``` + +2. **Add pool state** after the existing `let bot` declaration: + +```typescript +// Bot pool for agent teams: send-only Api instances (no polling) +const poolApis: Api[] = []; +// Maps "{groupFolder}:{senderName}" → pool Api index for stable assignment +const senderBotMap = new Map(); +let nextPoolIndex = 0; +``` + +3. **Add pool functions** — place these before the `isTelegramConnected` function: + +```typescript +/** + * Initialize send-only Api instances for the bot pool. + * Each pool bot can send messages but doesn't poll for updates. + */ +export async function initBotPool(tokens: string[]): Promise { + for (const token of tokens) { + try { + const api = new Api(token); + const me = await api.getMe(); + poolApis.push(api); + logger.info( + { username: me.username, id: me.id, poolSize: poolApis.length }, + 'Pool bot initialized', + ); + } catch (err) { + logger.error({ err }, 'Failed to initialize pool bot'); + } + } + if (poolApis.length > 0) { + logger.info({ count: poolApis.length }, 'Telegram bot pool ready'); + } +} + +/** + * Send a message via a pool bot assigned to the given sender name. + * Assigns bots round-robin on first use; subsequent messages from the + * same sender in the same group always use the same bot. + * On first assignment, renames the bot to match the sender's role. + */ +export async function sendPoolMessage( + chatId: string, + text: string, + sender: string, + groupFolder: string, +): Promise { + if (poolApis.length === 0) { + // No pool bots — fall back to main bot + await sendTelegramMessage(chatId, text); + return; + } + + const key = `${groupFolder}:${sender}`; + let idx = senderBotMap.get(key); + if (idx === undefined) { + idx = nextPoolIndex % poolApis.length; + nextPoolIndex++; + senderBotMap.set(key, idx); + // Rename the bot to match the sender's role, then wait for Telegram to propagate + try { + await poolApis[idx].setMyName(sender); + await new Promise((r) => setTimeout(r, 2000)); + logger.info({ sender, groupFolder, poolIndex: idx }, 'Assigned and renamed pool bot'); + } catch (err) { + logger.warn({ sender, err }, 'Failed to rename pool bot (sending anyway)'); + } + } + + const api = poolApis[idx]; + try { + const numericId = chatId.replace(/^tg:/, ''); + const MAX_LENGTH = 4096; + if (text.length <= MAX_LENGTH) { + await api.sendMessage(numericId, text); + } else { + for (let i = 0; i < text.length; i += MAX_LENGTH) { + await api.sendMessage(numericId, text.slice(i, i + MAX_LENGTH)); + } + } + logger.info({ chatId, sender, poolIndex: idx, length: text.length }, 'Pool message sent'); + } catch (err) { + logger.error({ chatId, sender, err }, 'Failed to send pool message'); + } +} +``` + +### Step 3: Add sender Parameter to MCP Tool + +Read `container/agent-runner/src/ipc-mcp-stdio.ts` and update the `send_message` tool to accept an optional `sender` parameter: + +Change the tool's schema from: + +```typescript +{ text: z.string().describe('The message text to send') }, +``` + +To: + +```typescript +{ + text: z.string().describe('The message text to send'), + sender: z.string().optional().describe('Your role/identity name (e.g. "Researcher"). When set, messages appear from a dedicated bot in Telegram.'), +}, +``` + +And update the handler to include `sender` in the IPC data: + +```typescript +async (args) => { + const data: Record = { + type: 'message', + chatJid, + text: args.text, + sender: args.sender || undefined, + groupFolder, + timestamp: new Date().toISOString(), + }; + + writeIpcFile(MESSAGES_DIR, data); + + return { content: [{ type: 'text' as const, text: 'Message sent.' }] }; + }, +``` + +### Step 4: Update Host IPC Routing + +Read `src/ipc.ts` and make these changes: + +1. **Add imports** — add `sendPoolMessage` and `initBotPool` from the Telegram swarm module, and `TELEGRAM_BOT_POOL` from config. + +2. **Update IPC message routing** — in `src/ipc.ts`, find where the `sendMessage` dependency is called to deliver IPC messages (inside `processIpcFiles`). The `sendMessage` is passed in via the `IpcDeps` parameter. Wrap it to route Telegram swarm messages through the bot pool: + +```typescript +if (data.sender && data.chatJid.startsWith('tg:')) { + await sendPoolMessage( + data.chatJid, + data.text, + data.sender, + sourceGroup, + ); +} else { + await deps.sendMessage(data.chatJid, data.text); +} +``` + +Note: The assistant name prefix is handled by `formatOutbound()` in the router — Telegram channels have `prefixAssistantName = false` so no prefix is added for `tg:` JIDs. + +3. **Initialize pool in `main()` in `src/index.ts`** — after creating the Telegram channel, add: + +```typescript +if (TELEGRAM_BOT_POOL.length > 0) { + await initBotPool(TELEGRAM_BOT_POOL); +} +``` + +### Step 5: Update AGENT.md Files + +#### 5a. Add global message formatting rules + +Read `groups/global/AGENT.md` and add a Message Formatting section: + +````markdown +## Message Formatting + +NEVER use markdown. Only use WhatsApp/Telegram formatting: +- *single asterisks* for bold (NEVER **double asterisks**) +- _underscores_ for italic +- • bullet points +- ```triple backticks``` for code + +No ## headings. No [links](url). No **double stars**. +```` + +#### 5b. Update existing group AGENT.md headings + +In any group AGENT.md that has a "WhatsApp Formatting" section (e.g. `groups/main/AGENT.md`), rename the heading to reflect multi-channel support: + +``` +## WhatsApp Formatting (and other messaging apps) +``` + +#### 5c. Add Agent Teams instructions to Telegram groups + +For each Telegram group that will use agent teams, create or update its `groups/{folder}/AGENT.md` with these instructions. Read the existing AGENT.md first (or `groups/global/AGENT.md` as a base) and add the Agent Teams section: + +````markdown +## Agent Teams + +When creating a team to tackle a complex task, follow these rules: + +### CRITICAL: Follow the user's prompt exactly + +Create *exactly* the team the user asked for — same number of agents, same roles, same names. Do NOT add extra agents, rename roles, or use generic names like "Researcher 1". If the user says "a marine biologist, a physicist, and Alexander Hamilton", create exactly those three agents with those exact names. + +### Team member instructions + +Each team member MUST be instructed to: + +1. *Share progress in the group* via `mcp__nanoclaw__send_message` with a `sender` parameter matching their exact role/character name (e.g., `sender: "Marine Biologist"` or `sender: "Alexander Hamilton"`). This makes their messages appear from a dedicated bot in the Telegram group. +2. *Also communicate with teammates* via `SendMessage` as normal for coordination. +3. Keep group messages *short* — 2-4 sentences max per message. Break longer content into multiple `send_message` calls. No walls of text. +4. Use the `sender` parameter consistently — always the same name so the bot identity stays stable. +5. NEVER use markdown formatting. Use ONLY WhatsApp/Telegram formatting: single *asterisks* for bold (NOT **double**), _underscores_ for italic, • for bullets, ```backticks``` for code. No ## headings, no [links](url), no **double asterisks**. + +### Example team creation prompt + +When creating a teammate, include instructions like: + +\``` +You are the Marine Biologist. When you have findings or updates for the user, send them to the group using mcp__nanoclaw__send_message with sender set to "Marine Biologist". Keep each message short (2-4 sentences max). Use emojis for strong reactions. ONLY use single *asterisks* for bold (never **double**), _underscores_ for italic, • for bullets. No markdown. Also communicate with teammates via SendMessage. +\``` + +### Lead agent behavior + +As the lead agent who created the team: + +- You do NOT need to react to or relay every teammate message. The user sees those directly from the teammate bots. +- Send your own messages only to comment, share thoughts, synthesize, or direct the team. +- When processing an internal update from a teammate that doesn't need a user-facing response, wrap your *entire* output in `` tags. +- Focus on high-level coordination and the final synthesis. +```` + +### Step 6: Update Environment + +Add pool tokens to `.env`: + +```bash +TELEGRAM_BOT_POOL=TOKEN1,TOKEN2,TOKEN3,... +``` + +**Important**: Sync to all required locations: + +```bash +cp .env data/env/env +``` + +Also add `TELEGRAM_BOT_POOL` to the launchd plist (`~/Library/LaunchAgents/com.nanoclaw.plist`) in the `EnvironmentVariables` dict if using launchd. + +### Step 7: Rebuild and Restart + +```bash +npm run build +./container/build.sh # Required — MCP tool changed +# macOS: +launchctl unload ~/Library/LaunchAgents/com.nanoclaw.plist +launchctl load ~/Library/LaunchAgents/com.nanoclaw.plist +# Linux: +# systemctl --user restart nanoclaw +``` + +Must use `unload/load` (macOS) or `restart` (Linux) because the service env vars changed. + +### Step 8: Test + +Tell the user: + +> Send a message in your Telegram group asking for a multi-agent task, e.g.: +> "Assemble a team of a researcher and a coder to build me a hello world app" +> +> You should see: +> +> - The lead agent (main bot) acknowledging and creating the team +> - Each subagent messaging from a different bot, renamed to their role +> - Short, scannable messages from each agent +> +> Check logs: `tail -f logs/nanoclaw.log | grep -i pool` + +## Architecture Notes + +- Pool bots use Grammy's `Api` class — lightweight, no polling, just send +- Bot names are set via `setMyName` — changes are global to the bot, not per-chat +- A 2-second delay after `setMyName` allows Telegram to propagate the name change before the first message +- Sender→bot mapping is stable within a group (keyed as `{groupFolder}:{senderName}`) +- Mapping resets on service restart — pool bots get reassigned fresh +- If pool runs out, bots are reused (round-robin wraps) + +## Troubleshooting + +### Pool bots not sending messages + +1. Verify tokens: `curl -s "https://api.telegram.org/botTOKEN/getMe"` +2. Check pool initialized: `grep "Pool bot" logs/nanoclaw.log` +3. Ensure all pool bots are members of the Telegram group +4. Check Group Privacy is disabled for each pool bot + +### Bot names not updating + +Telegram caches bot names client-side. The 2-second delay after `setMyName` helps, but users may need to restart their Telegram client to see updated names immediately. + +### Subagents not using send_message + +Check the group's `AGENT.md` has the Agent Teams instructions. The lead agent reads this when creating teammates and must include the `send_message` + `sender` instructions in each teammate's prompt. + +## Removal + +To remove Agent Swarm support while keeping basic Telegram: + +1. Remove `TELEGRAM_BOT_POOL` from `src/config.ts` +2. Remove pool code from `src/telegram.ts` (`poolApis`, `senderBotMap`, `initBotPool`, `sendPoolMessage`) +3. Remove pool routing from IPC handler in `src/index.ts` (revert to plain `sendMessage`) +4. Remove `initBotPool` call from `main()` +5. Remove `sender` param from MCP tool in `container/agent-runner/src/ipc-mcp-stdio.ts` +6. Remove Agent Teams section from group AGENT.md files +7. Remove `TELEGRAM_BOT_POOL` from `.env`, `data/env/env`, and launchd plist/systemd unit +8. Rebuild: `npm run build && ./container/build.sh && launchctl unload ~/Library/LaunchAgents/com.nanoclaw.plist && launchctl load ~/Library/LaunchAgents/com.nanoclaw.plist` (macOS) or `npm run build && ./container/build.sh && systemctl --user restart nanoclaw` (Linux) diff --git a/.agent/skills/add-telegram/SKILL.md b/.agent/skills/add-telegram/SKILL.md new file mode 100644 index 0000000..eda8903 --- /dev/null +++ b/.agent/skills/add-telegram/SKILL.md @@ -0,0 +1,251 @@ +--- +name: add-telegram +description: Add Telegram as a channel. Can replace WhatsApp entirely or run alongside it. Also configurable as a control-only channel (triggers actions) or passive channel (receives notifications only). +--- + +# Add Telegram Channel + +This skill adds Telegram support to NanoClaw using the skills engine for deterministic code changes, then walks through interactive setup. + +## Phase 1: Pre-flight + +### Check if already applied + +Read `.nanoclaw/state.yaml`. If `telegram` is in `applied_skills`, skip to Phase 3 (Setup). The code changes are already in place. + +### Ask the user + +Use `AskUserQuestion` to collect configuration: + +AskUserQuestion: Should Telegram replace WhatsApp or run alongside it? + +- **Replace WhatsApp** - Telegram will be the only channel (sets TELEGRAM_ONLY=true) +- **Alongside** - Both Telegram and WhatsApp channels active + +AskUserQuestion: Do you have a Telegram bot token, or do you need to create one? + +If they have one, collect it now. If not, we'll create one in Phase 3. + +## Phase 2: Apply Code Changes + +Run the skills engine to apply this skill's code package. The package files are in this directory alongside this SKILL.md. + +### Initialize skills system (if needed) + +If `.nanoclaw/` directory doesn't exist yet: + +```bash +npx tsx scripts/apply-skill.ts --init +``` + +Or call `initSkillsSystem()` from `skills-engine/migrate.ts`. + +### Apply the skill + +```bash +npx tsx scripts/apply-skill.ts .agent/skills/add-telegram +``` + +This deterministically: + +- Adds `src/channels/telegram.ts` (TelegramChannel class implementing Channel interface) +- Adds `src/channels/telegram.test.ts` (46 unit tests) +- Three-way merges Telegram support into `src/index.ts` (multi-channel support, findChannel routing) +- Three-way merges Telegram config into `src/config.ts` (TELEGRAM_BOT_TOKEN, TELEGRAM_ONLY exports) +- Three-way merges updated routing tests into `src/routing.test.ts` +- Installs the `grammy` npm dependency +- Updates `.env.example` with `TELEGRAM_BOT_TOKEN` and `TELEGRAM_ONLY` +- Records the application in `.nanoclaw/state.yaml` + +If the apply reports merge conflicts, read the intent files: + +- `modify/src/index.ts.intent.md` — what changed and invariants for index.ts +- `modify/src/config.ts.intent.md` — what changed for config.ts + +### Validate code changes + +```bash +npm test +npm run build +``` + +All tests must pass (including the new telegram tests) and build must be clean before proceeding. + +## Phase 3: Setup + +### Create Telegram Bot (if needed) + +If the user doesn't have a bot token, tell them: + +> I need you to create a Telegram bot: +> +> 1. Open Telegram and search for `@BotFather` +> 2. Send `/newbot` and follow prompts: +> - Bot name: Something friendly (e.g., "Andy Assistant") +> - Bot username: Must end with "bot" (e.g., "andy_ai_bot") +> 3. Copy the bot token (looks like `123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11`) + +Wait for the user to provide the token. + +### Configure environment + +Add to `.env`: + +```bash +TELEGRAM_BOT_TOKEN= +``` + +If they chose to replace WhatsApp: + +```bash +TELEGRAM_ONLY=true +``` + +Sync to container environment: + +```bash +mkdir -p data/env && cp .env data/env/env +``` + +The container reads environment from `data/env/env`, not `.env` directly. + +### Disable Group Privacy (for group chats) + +Tell the user: + +> **Important for group chats**: By default, Telegram bots only see @mentions and commands in groups. To let the bot see all messages: +> +> 1. Open Telegram and search for `@BotFather` +> 2. Send `/mybots` and select your bot +> 3. Go to **Bot Settings** > **Group Privacy** > **Turn off** +> +> This is optional if you only want trigger-based responses via @mentioning the bot. + +### Build and restart + +```bash +npm run build +launchctl kickstart -k gui/$(id -u)/com.nanoclaw # macOS +# Linux: systemctl --user restart nanoclaw +``` + +## Phase 4: Registration + +### Get Chat ID + +Tell the user: + +> 1. Open your bot in Telegram (search for its username) +> 2. Send `/chatid` — it will reply with the chat ID +> 3. For groups: add the bot to the group first, then send `/chatid` in the group + +Wait for the user to provide the chat ID (format: `tg:123456789` or `tg:-1001234567890`). + +### Register the chat + +Use the IPC register flow or register directly. The chat ID, name, and folder name are needed. + +For a main chat (responds to all messages, uses the `main` folder): + +```typescript +registerGroup('tg:', { + name: '', + folder: 'main', + trigger: `@${ASSISTANT_NAME}`, + added_at: new Date().toISOString(), + requiresTrigger: false, +}); +``` + +For additional chats (trigger-only): + +```typescript +registerGroup('tg:', { + name: '', + folder: '', + trigger: `@${ASSISTANT_NAME}`, + added_at: new Date().toISOString(), + requiresTrigger: true, +}); +``` + +## Phase 5: Verify + +### Test the connection + +Tell the user: + +> Send a message to your registered Telegram chat: +> +> - For main chat: Any message works +> - For non-main: `@Andy hello` or @mention the bot +> +> The bot should respond within a few seconds. + +### Check logs if needed + +```bash +tail -f logs/nanoclaw.log +``` + +## Troubleshooting + +### Bot not responding + +Check: + +1. `TELEGRAM_BOT_TOKEN` is set in `.env` AND synced to `data/env/env` +2. Chat is registered in the ops database (check with: `psql "$OPS_DB_URL" -c "SELECT * FROM registered_groups WHERE jid LIKE 'tg:%'"`) +3. For non-main chats: message includes trigger pattern +4. Service is running: `launchctl list | grep nanoclaw` (macOS) or `systemctl --user status nanoclaw` (Linux) + +### Bot only responds to @mentions in groups + +Group Privacy is enabled (default). Fix: + +1. `@BotFather` > `/mybots` > select bot > **Bot Settings** > **Group Privacy** > **Turn off** +2. Remove and re-add the bot to the group (required for the change to take effect) + +### Getting chat ID + +If `/chatid` doesn't work: + +- Verify token: `curl -s "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/getMe"` +- Check bot is started: `tail -f logs/nanoclaw.log` + +## After Setup + +If running `npm run dev` while the service is active: + +```bash +# macOS: +launchctl unload ~/Library/LaunchAgents/com.nanoclaw.plist +npm run dev +# When done testing: +launchctl load ~/Library/LaunchAgents/com.nanoclaw.plist +# Linux: +# systemctl --user stop nanoclaw +# npm run dev +# systemctl --user start nanoclaw +``` + +## Agent Swarms (Teams) + +After completing the Telegram setup, use `AskUserQuestion`: + +AskUserQuestion: Would you like to add Agent Swarm support? Without it, Agent Teams still work — they just operate behind the scenes. With Swarm support, each subagent appears as a different bot in the Telegram group so you can see who's saying what and have interactive team sessions. + +If they say yes, invoke the `/add-telegram-swarm` skill. + +## Removal + +To remove Telegram integration: + +1. Delete `src/channels/telegram.ts` +2. Remove `TelegramChannel` import and creation from `src/index.ts` +3. Remove `channels` array and revert to using `whatsapp` directly in `processGroupMessages`, scheduler deps, and IPC deps +4. Revert `getAvailableGroups()` filter to only include `@g.us` chats +5. Remove Telegram config (`TELEGRAM_BOT_TOKEN`, `TELEGRAM_ONLY`) from `src/config.ts` +6. Remove Telegram registrations from ops database: `psql "$OPS_DB_URL" -c "DELETE FROM registered_groups WHERE jid LIKE 'tg:%'"` +7. Uninstall: `npm uninstall grammy` +8. Rebuild: `npm run build && launchctl kickstart -k gui/$(id -u)/com.nanoclaw` (macOS) or `npm run build && systemctl --user restart nanoclaw` (Linux) diff --git a/.agent/skills/add-telegram/add/src/channels/telegram.test.ts b/.agent/skills/add-telegram/add/src/channels/telegram.test.ts new file mode 100644 index 0000000..950b607 --- /dev/null +++ b/.agent/skills/add-telegram/add/src/channels/telegram.test.ts @@ -0,0 +1,926 @@ +import { describe, it, expect, beforeEach, vi, afterEach } from 'vitest'; + +// --- Mocks --- + +// Mock config +vi.mock('../config.js', () => ({ + ASSISTANT_NAME: 'Andy', + TRIGGER_PATTERN: /^@Andy\b/i, +})); + +// Mock logger +vi.mock('../logger.js', () => ({ + logger: { + debug: vi.fn(), + info: vi.fn(), + warn: vi.fn(), + error: vi.fn(), + }, +})); + +// --- Grammy mock --- + +type Handler = (...args: any[]) => any; + +const botRef = vi.hoisted(() => ({ current: null as any })); + +vi.mock('grammy', () => ({ + Bot: class MockBot { + token: string; + commandHandlers = new Map(); + filterHandlers = new Map(); + errorHandler: Handler | null = null; + + api = { + sendMessage: vi.fn().mockResolvedValue(undefined), + sendChatAction: vi.fn().mockResolvedValue(undefined), + }; + + constructor(token: string) { + this.token = token; + botRef.current = this; + } + + command(name: string, handler: Handler) { + this.commandHandlers.set(name, handler); + } + + on(filter: string, handler: Handler) { + const existing = this.filterHandlers.get(filter) || []; + existing.push(handler); + this.filterHandlers.set(filter, existing); + } + + catch(handler: Handler) { + this.errorHandler = handler; + } + + start(opts: { onStart: (botInfo: any) => void }) { + opts.onStart({ username: 'andy_ai_bot', id: 12345 }); + } + + stop() {} + }, +})); + +import { TelegramChannel, TelegramChannelOpts } from './telegram.js'; + +// --- Test helpers --- + +function createTestOpts( + overrides?: Partial, +): TelegramChannelOpts { + return { + onMessage: vi.fn(), + onChatMetadata: vi.fn(), + registeredGroups: vi.fn(() => ({ + 'tg:100200300': { + name: 'Test Group', + folder: 'test-group', + trigger: '@Andy', + added_at: '2024-01-01T00:00:00.000Z', + }, + })), + ...overrides, + }; +} + +function createTextCtx(overrides: { + chatId?: number; + chatType?: string; + chatTitle?: string; + text: string; + fromId?: number; + firstName?: string; + username?: string; + messageId?: number; + date?: number; + entities?: any[]; +}) { + const chatId = overrides.chatId ?? 100200300; + const chatType = overrides.chatType ?? 'group'; + return { + chat: { + id: chatId, + type: chatType, + title: overrides.chatTitle ?? 'Test Group', + }, + from: { + id: overrides.fromId ?? 99001, + first_name: overrides.firstName ?? 'Alice', + username: overrides.username ?? 'alice_user', + }, + message: { + text: overrides.text, + date: overrides.date ?? Math.floor(Date.now() / 1000), + message_id: overrides.messageId ?? 1, + entities: overrides.entities ?? [], + }, + me: { username: 'andy_ai_bot' }, + reply: vi.fn(), + }; +} + +function createMediaCtx(overrides: { + chatId?: number; + chatType?: string; + fromId?: number; + firstName?: string; + date?: number; + messageId?: number; + caption?: string; + extra?: Record; +}) { + const chatId = overrides.chatId ?? 100200300; + return { + chat: { + id: chatId, + type: overrides.chatType ?? 'group', + title: 'Test Group', + }, + from: { + id: overrides.fromId ?? 99001, + first_name: overrides.firstName ?? 'Alice', + username: 'alice_user', + }, + message: { + date: overrides.date ?? Math.floor(Date.now() / 1000), + message_id: overrides.messageId ?? 1, + caption: overrides.caption, + ...(overrides.extra || {}), + }, + me: { username: 'andy_ai_bot' }, + }; +} + +function currentBot() { + return botRef.current; +} + +async function triggerTextMessage(ctx: ReturnType) { + const handlers = currentBot().filterHandlers.get('message:text') || []; + for (const h of handlers) await h(ctx); +} + +async function triggerMediaMessage( + filter: string, + ctx: ReturnType, +) { + const handlers = currentBot().filterHandlers.get(filter) || []; + for (const h of handlers) await h(ctx); +} + +// --- Tests --- + +describe('TelegramChannel', () => { + beforeEach(() => { + vi.clearAllMocks(); + }); + + afterEach(() => { + vi.restoreAllMocks(); + }); + + // --- Connection lifecycle --- + + describe('connection lifecycle', () => { + it('resolves connect() when bot starts', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + + await channel.connect(); + + expect(channel.isConnected()).toBe(true); + }); + + it('registers command and message handlers on connect', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + + await channel.connect(); + + expect(currentBot().commandHandlers.has('chatid')).toBe(true); + expect(currentBot().commandHandlers.has('ping')).toBe(true); + expect(currentBot().filterHandlers.has('message:text')).toBe(true); + expect(currentBot().filterHandlers.has('message:photo')).toBe(true); + expect(currentBot().filterHandlers.has('message:video')).toBe(true); + expect(currentBot().filterHandlers.has('message:voice')).toBe(true); + expect(currentBot().filterHandlers.has('message:audio')).toBe(true); + expect(currentBot().filterHandlers.has('message:document')).toBe(true); + expect(currentBot().filterHandlers.has('message:sticker')).toBe(true); + expect(currentBot().filterHandlers.has('message:location')).toBe(true); + expect(currentBot().filterHandlers.has('message:contact')).toBe(true); + }); + + it('registers error handler on connect', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + + await channel.connect(); + + expect(currentBot().errorHandler).not.toBeNull(); + }); + + it('disconnects cleanly', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + + await channel.connect(); + expect(channel.isConnected()).toBe(true); + + await channel.disconnect(); + expect(channel.isConnected()).toBe(false); + }); + + it('isConnected() returns false before connect', () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + + expect(channel.isConnected()).toBe(false); + }); + }); + + // --- Text message handling --- + + describe('text message handling', () => { + it('delivers message for registered group', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + const ctx = createTextCtx({ text: 'Hello everyone' }); + await triggerTextMessage(ctx); + + expect(opts.onChatMetadata).toHaveBeenCalledWith( + 'tg:100200300', + expect.any(String), + 'Test Group', + 'telegram', + true, + ); + expect(opts.onMessage).toHaveBeenCalledWith( + 'tg:100200300', + expect.objectContaining({ + id: '1', + chat_jid: 'tg:100200300', + sender: '99001', + sender_name: 'Alice', + content: 'Hello everyone', + is_from_me: false, + }), + ); + }); + + it('only emits metadata for unregistered chats', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + const ctx = createTextCtx({ chatId: 999999, text: 'Unknown chat' }); + await triggerTextMessage(ctx); + + expect(opts.onChatMetadata).toHaveBeenCalledWith( + 'tg:999999', + expect.any(String), + 'Test Group', + 'telegram', + true, + ); + expect(opts.onMessage).not.toHaveBeenCalled(); + }); + + it('skips command messages (starting with /)', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + const ctx = createTextCtx({ text: '/start' }); + await triggerTextMessage(ctx); + + expect(opts.onMessage).not.toHaveBeenCalled(); + expect(opts.onChatMetadata).not.toHaveBeenCalled(); + }); + + it('extracts sender name from first_name', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + const ctx = createTextCtx({ text: 'Hi', firstName: 'Bob' }); + await triggerTextMessage(ctx); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'tg:100200300', + expect.objectContaining({ sender_name: 'Bob' }), + ); + }); + + it('falls back to username when first_name missing', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + const ctx = createTextCtx({ text: 'Hi' }); + ctx.from.first_name = undefined as any; + await triggerTextMessage(ctx); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'tg:100200300', + expect.objectContaining({ sender_name: 'alice_user' }), + ); + }); + + it('falls back to user ID when name and username missing', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + const ctx = createTextCtx({ text: 'Hi', fromId: 42 }); + ctx.from.first_name = undefined as any; + ctx.from.username = undefined as any; + await triggerTextMessage(ctx); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'tg:100200300', + expect.objectContaining({ sender_name: '42' }), + ); + }); + + it('uses sender name as chat name for private chats', async () => { + const opts = createTestOpts({ + registeredGroups: vi.fn(() => ({ + 'tg:100200300': { + name: 'Private', + folder: 'private', + trigger: '@Andy', + added_at: '2024-01-01T00:00:00.000Z', + }, + })), + }); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + const ctx = createTextCtx({ + text: 'Hello', + chatType: 'private', + firstName: 'Alice', + }); + await triggerTextMessage(ctx); + + expect(opts.onChatMetadata).toHaveBeenCalledWith( + 'tg:100200300', + expect.any(String), + 'Alice', // Private chats use sender name + 'telegram', + false, + ); + }); + + it('uses chat title as name for group chats', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + const ctx = createTextCtx({ + text: 'Hello', + chatType: 'supergroup', + chatTitle: 'Project Team', + }); + await triggerTextMessage(ctx); + + expect(opts.onChatMetadata).toHaveBeenCalledWith( + 'tg:100200300', + expect.any(String), + 'Project Team', + 'telegram', + true, + ); + }); + + it('converts message.date to ISO timestamp', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + const unixTime = 1704067200; // 2024-01-01T00:00:00.000Z + const ctx = createTextCtx({ text: 'Hello', date: unixTime }); + await triggerTextMessage(ctx); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'tg:100200300', + expect.objectContaining({ + timestamp: '2024-01-01T00:00:00.000Z', + }), + ); + }); + }); + + // --- @mention translation --- + + describe('@mention translation', () => { + it('translates @bot_username mention to trigger format', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + const ctx = createTextCtx({ + text: '@andy_ai_bot what time is it?', + entities: [{ type: 'mention', offset: 0, length: 12 }], + }); + await triggerTextMessage(ctx); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'tg:100200300', + expect.objectContaining({ + content: '@Andy @andy_ai_bot what time is it?', + }), + ); + }); + + it('does not translate if message already matches trigger', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + const ctx = createTextCtx({ + text: '@Andy @andy_ai_bot hello', + entities: [{ type: 'mention', offset: 6, length: 12 }], + }); + await triggerTextMessage(ctx); + + // Should NOT double-prepend — already starts with @Andy + expect(opts.onMessage).toHaveBeenCalledWith( + 'tg:100200300', + expect.objectContaining({ + content: '@Andy @andy_ai_bot hello', + }), + ); + }); + + it('does not translate mentions of other bots', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + const ctx = createTextCtx({ + text: '@some_other_bot hi', + entities: [{ type: 'mention', offset: 0, length: 15 }], + }); + await triggerTextMessage(ctx); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'tg:100200300', + expect.objectContaining({ + content: '@some_other_bot hi', // No translation + }), + ); + }); + + it('handles mention in middle of message', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + const ctx = createTextCtx({ + text: 'hey @andy_ai_bot check this', + entities: [{ type: 'mention', offset: 4, length: 12 }], + }); + await triggerTextMessage(ctx); + + // Bot is mentioned, message doesn't match trigger → prepend trigger + expect(opts.onMessage).toHaveBeenCalledWith( + 'tg:100200300', + expect.objectContaining({ + content: '@Andy hey @andy_ai_bot check this', + }), + ); + }); + + it('handles message with no entities', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + const ctx = createTextCtx({ text: 'plain message' }); + await triggerTextMessage(ctx); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'tg:100200300', + expect.objectContaining({ + content: 'plain message', + }), + ); + }); + + it('ignores non-mention entities', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + const ctx = createTextCtx({ + text: 'check https://example.com', + entities: [{ type: 'url', offset: 6, length: 19 }], + }); + await triggerTextMessage(ctx); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'tg:100200300', + expect.objectContaining({ + content: 'check https://example.com', + }), + ); + }); + }); + + // --- Non-text messages --- + + describe('non-text messages', () => { + it('stores photo with placeholder', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + const ctx = createMediaCtx({}); + await triggerMediaMessage('message:photo', ctx); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'tg:100200300', + expect.objectContaining({ content: '[Photo]' }), + ); + }); + + it('stores photo with caption', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + const ctx = createMediaCtx({ caption: 'Look at this' }); + await triggerMediaMessage('message:photo', ctx); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'tg:100200300', + expect.objectContaining({ content: '[Photo] Look at this' }), + ); + }); + + it('stores video with placeholder', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + const ctx = createMediaCtx({}); + await triggerMediaMessage('message:video', ctx); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'tg:100200300', + expect.objectContaining({ content: '[Video]' }), + ); + }); + + it('stores voice message with placeholder', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + const ctx = createMediaCtx({}); + await triggerMediaMessage('message:voice', ctx); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'tg:100200300', + expect.objectContaining({ content: '[Voice message]' }), + ); + }); + + it('stores audio with placeholder', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + const ctx = createMediaCtx({}); + await triggerMediaMessage('message:audio', ctx); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'tg:100200300', + expect.objectContaining({ content: '[Audio]' }), + ); + }); + + it('stores document with filename', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + const ctx = createMediaCtx({ + extra: { document: { file_name: 'report.pdf' } }, + }); + await triggerMediaMessage('message:document', ctx); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'tg:100200300', + expect.objectContaining({ content: '[Document: report.pdf]' }), + ); + }); + + it('stores document with fallback name when filename missing', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + const ctx = createMediaCtx({ extra: { document: {} } }); + await triggerMediaMessage('message:document', ctx); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'tg:100200300', + expect.objectContaining({ content: '[Document: file]' }), + ); + }); + + it('stores sticker with emoji', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + const ctx = createMediaCtx({ + extra: { sticker: { emoji: '😂' } }, + }); + await triggerMediaMessage('message:sticker', ctx); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'tg:100200300', + expect.objectContaining({ content: '[Sticker 😂]' }), + ); + }); + + it('stores location with placeholder', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + const ctx = createMediaCtx({}); + await triggerMediaMessage('message:location', ctx); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'tg:100200300', + expect.objectContaining({ content: '[Location]' }), + ); + }); + + it('stores contact with placeholder', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + const ctx = createMediaCtx({}); + await triggerMediaMessage('message:contact', ctx); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'tg:100200300', + expect.objectContaining({ content: '[Contact]' }), + ); + }); + + it('ignores non-text messages from unregistered chats', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + const ctx = createMediaCtx({ chatId: 999999 }); + await triggerMediaMessage('message:photo', ctx); + + expect(opts.onMessage).not.toHaveBeenCalled(); + }); + }); + + // --- sendMessage --- + + describe('sendMessage', () => { + it('sends message via bot API', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + await channel.sendMessage('tg:100200300', 'Hello'); + + expect(currentBot().api.sendMessage).toHaveBeenCalledWith( + '100200300', + 'Hello', + ); + }); + + it('strips tg: prefix from JID', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + await channel.sendMessage('tg:-1001234567890', 'Group message'); + + expect(currentBot().api.sendMessage).toHaveBeenCalledWith( + '-1001234567890', + 'Group message', + ); + }); + + it('splits messages exceeding 4096 characters', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + const longText = 'x'.repeat(5000); + await channel.sendMessage('tg:100200300', longText); + + expect(currentBot().api.sendMessage).toHaveBeenCalledTimes(2); + expect(currentBot().api.sendMessage).toHaveBeenNthCalledWith( + 1, + '100200300', + 'x'.repeat(4096), + ); + expect(currentBot().api.sendMessage).toHaveBeenNthCalledWith( + 2, + '100200300', + 'x'.repeat(904), + ); + }); + + it('sends exactly one message at 4096 characters', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + const exactText = 'y'.repeat(4096); + await channel.sendMessage('tg:100200300', exactText); + + expect(currentBot().api.sendMessage).toHaveBeenCalledTimes(1); + }); + + it('handles send failure gracefully', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + currentBot().api.sendMessage.mockRejectedValueOnce( + new Error('Network error'), + ); + + // Should not throw + await expect( + channel.sendMessage('tg:100200300', 'Will fail'), + ).resolves.toBeUndefined(); + }); + + it('does nothing when bot is not initialized', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + + // Don't connect — bot is null + await channel.sendMessage('tg:100200300', 'No bot'); + + // No error, no API call + }); + }); + + // --- ownsJid --- + + describe('ownsJid', () => { + it('owns tg: JIDs', () => { + const channel = new TelegramChannel('test-token', createTestOpts()); + expect(channel.ownsJid('tg:123456')).toBe(true); + }); + + it('owns tg: JIDs with negative IDs (groups)', () => { + const channel = new TelegramChannel('test-token', createTestOpts()); + expect(channel.ownsJid('tg:-1001234567890')).toBe(true); + }); + + it('does not own WhatsApp group JIDs', () => { + const channel = new TelegramChannel('test-token', createTestOpts()); + expect(channel.ownsJid('12345@g.us')).toBe(false); + }); + + it('does not own WhatsApp DM JIDs', () => { + const channel = new TelegramChannel('test-token', createTestOpts()); + expect(channel.ownsJid('12345@s.whatsapp.net')).toBe(false); + }); + + it('does not own unknown JID formats', () => { + const channel = new TelegramChannel('test-token', createTestOpts()); + expect(channel.ownsJid('random-string')).toBe(false); + }); + }); + + // --- setTyping --- + + describe('setTyping', () => { + it('sends typing action when isTyping is true', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + await channel.setTyping('tg:100200300', true); + + expect(currentBot().api.sendChatAction).toHaveBeenCalledWith( + '100200300', + 'typing', + ); + }); + + it('does nothing when isTyping is false', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + await channel.setTyping('tg:100200300', false); + + expect(currentBot().api.sendChatAction).not.toHaveBeenCalled(); + }); + + it('does nothing when bot is not initialized', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + + // Don't connect + await channel.setTyping('tg:100200300', true); + + // No error, no API call + }); + + it('handles typing indicator failure gracefully', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + currentBot().api.sendChatAction.mockRejectedValueOnce( + new Error('Rate limited'), + ); + + await expect( + channel.setTyping('tg:100200300', true), + ).resolves.toBeUndefined(); + }); + }); + + // --- Bot commands --- + + describe('bot commands', () => { + it('/chatid replies with chat ID and metadata', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + const handler = currentBot().commandHandlers.get('chatid')!; + const ctx = { + chat: { id: 100200300, type: 'group' as const }, + from: { first_name: 'Alice' }, + reply: vi.fn(), + }; + + await handler(ctx); + + expect(ctx.reply).toHaveBeenCalledWith( + expect.stringContaining('tg:100200300'), + expect.objectContaining({ parse_mode: 'Markdown' }), + ); + }); + + it('/chatid shows chat type', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + const handler = currentBot().commandHandlers.get('chatid')!; + const ctx = { + chat: { id: 555, type: 'private' as const }, + from: { first_name: 'Bob' }, + reply: vi.fn(), + }; + + await handler(ctx); + + expect(ctx.reply).toHaveBeenCalledWith( + expect.stringContaining('private'), + expect.any(Object), + ); + }); + + it('/ping replies with bot status', async () => { + const opts = createTestOpts(); + const channel = new TelegramChannel('test-token', opts); + await channel.connect(); + + const handler = currentBot().commandHandlers.get('ping')!; + const ctx = { reply: vi.fn() }; + + await handler(ctx); + + expect(ctx.reply).toHaveBeenCalledWith('Andy is online.'); + }); + }); + + // --- Channel properties --- + + describe('channel properties', () => { + it('has name "telegram"', () => { + const channel = new TelegramChannel('test-token', createTestOpts()); + expect(channel.name).toBe('telegram'); + }); + }); +}); diff --git a/.agent/skills/add-telegram/add/src/channels/telegram.ts b/.agent/skills/add-telegram/add/src/channels/telegram.ts new file mode 100644 index 0000000..43a6266 --- /dev/null +++ b/.agent/skills/add-telegram/add/src/channels/telegram.ts @@ -0,0 +1,244 @@ +import { Bot } from 'grammy'; + +import { ASSISTANT_NAME, TRIGGER_PATTERN } from '../config.js'; +import { logger } from '../logger.js'; +import { + Channel, + OnChatMetadata, + OnInboundMessage, + RegisteredGroup, +} from '../types.js'; + +export interface TelegramChannelOpts { + onMessage: OnInboundMessage; + onChatMetadata: OnChatMetadata; + registeredGroups: () => Record; +} + +export class TelegramChannel implements Channel { + name = 'telegram'; + + private bot: Bot | null = null; + private opts: TelegramChannelOpts; + private botToken: string; + + constructor(botToken: string, opts: TelegramChannelOpts) { + this.botToken = botToken; + this.opts = opts; + } + + async connect(): Promise { + this.bot = new Bot(this.botToken); + + // Command to get chat ID (useful for registration) + this.bot.command('chatid', (ctx) => { + const chatId = ctx.chat.id; + const chatType = ctx.chat.type; + const chatName = + chatType === 'private' + ? ctx.from?.first_name || 'Private' + : (ctx.chat as any).title || 'Unknown'; + + ctx.reply( + `Chat ID: \`tg:${chatId}\`\nName: ${chatName}\nType: ${chatType}`, + { parse_mode: 'Markdown' }, + ); + }); + + // Command to check bot status + this.bot.command('ping', (ctx) => { + ctx.reply(`${ASSISTANT_NAME} is online.`); + }); + + this.bot.on('message:text', async (ctx) => { + // Skip commands + if (ctx.message.text.startsWith('/')) return; + + const chatJid = `tg:${ctx.chat.id}`; + let content = ctx.message.text; + const timestamp = new Date(ctx.message.date * 1000).toISOString(); + const senderName = + ctx.from?.first_name || + ctx.from?.username || + ctx.from?.id.toString() || + 'Unknown'; + const sender = ctx.from?.id.toString() || ''; + const msgId = ctx.message.message_id.toString(); + + // Determine chat name + const chatName = + ctx.chat.type === 'private' + ? senderName + : (ctx.chat as any).title || chatJid; + + // Translate Telegram @bot_username mentions into TRIGGER_PATTERN format. + // Telegram @mentions (e.g., @andy_ai_bot) won't match TRIGGER_PATTERN + // (e.g., ^@Andy\b), so we prepend the trigger when the bot is @mentioned. + const botUsername = ctx.me?.username?.toLowerCase(); + if (botUsername) { + const entities = ctx.message.entities || []; + const isBotMentioned = entities.some((entity) => { + if (entity.type === 'mention') { + const mentionText = content + .substring(entity.offset, entity.offset + entity.length) + .toLowerCase(); + return mentionText === `@${botUsername}`; + } + return false; + }); + if (isBotMentioned && !TRIGGER_PATTERN.test(content)) { + content = `@${ASSISTANT_NAME} ${content}`; + } + } + + // Store chat metadata for discovery + const isGroup = ctx.chat.type === 'group' || ctx.chat.type === 'supergroup'; + this.opts.onChatMetadata(chatJid, timestamp, chatName, 'telegram', isGroup); + + // Only deliver full message for registered groups + const group = this.opts.registeredGroups()[chatJid]; + if (!group) { + logger.debug( + { chatJid, chatName }, + 'Message from unregistered Telegram chat', + ); + return; + } + + // Deliver message — startMessageLoop() will pick it up + this.opts.onMessage(chatJid, { + id: msgId, + chat_jid: chatJid, + sender, + sender_name: senderName, + content, + timestamp, + is_from_me: false, + }); + + logger.info( + { chatJid, chatName, sender: senderName }, + 'Telegram message stored', + ); + }); + + // Handle non-text messages with placeholders so the agent knows something was sent + const storeNonText = (ctx: any, placeholder: string) => { + const chatJid = `tg:${ctx.chat.id}`; + const group = this.opts.registeredGroups()[chatJid]; + if (!group) return; + + const timestamp = new Date(ctx.message.date * 1000).toISOString(); + const senderName = + ctx.from?.first_name || + ctx.from?.username || + ctx.from?.id?.toString() || + 'Unknown'; + const caption = ctx.message.caption ? ` ${ctx.message.caption}` : ''; + + const isGroup = ctx.chat.type === 'group' || ctx.chat.type === 'supergroup'; + this.opts.onChatMetadata(chatJid, timestamp, undefined, 'telegram', isGroup); + this.opts.onMessage(chatJid, { + id: ctx.message.message_id.toString(), + chat_jid: chatJid, + sender: ctx.from?.id?.toString() || '', + sender_name: senderName, + content: `${placeholder}${caption}`, + timestamp, + is_from_me: false, + }); + }; + + this.bot.on('message:photo', (ctx) => storeNonText(ctx, '[Photo]')); + this.bot.on('message:video', (ctx) => storeNonText(ctx, '[Video]')); + this.bot.on('message:voice', (ctx) => + storeNonText(ctx, '[Voice message]'), + ); + this.bot.on('message:audio', (ctx) => storeNonText(ctx, '[Audio]')); + this.bot.on('message:document', (ctx) => { + const name = ctx.message.document?.file_name || 'file'; + storeNonText(ctx, `[Document: ${name}]`); + }); + this.bot.on('message:sticker', (ctx) => { + const emoji = ctx.message.sticker?.emoji || ''; + storeNonText(ctx, `[Sticker ${emoji}]`); + }); + this.bot.on('message:location', (ctx) => storeNonText(ctx, '[Location]')); + this.bot.on('message:contact', (ctx) => storeNonText(ctx, '[Contact]')); + + // Handle errors gracefully + this.bot.catch((err) => { + logger.error({ err: err.message }, 'Telegram bot error'); + }); + + // Start polling — returns a Promise that resolves when started + return new Promise((resolve) => { + this.bot!.start({ + onStart: (botInfo) => { + logger.info( + { username: botInfo.username, id: botInfo.id }, + 'Telegram bot connected', + ); + console.log(`\n Telegram bot: @${botInfo.username}`); + console.log( + ` Send /chatid to the bot to get a chat's registration ID\n`, + ); + resolve(); + }, + }); + }); + } + + async sendMessage(jid: string, text: string): Promise { + if (!this.bot) { + logger.warn('Telegram bot not initialized'); + return; + } + + try { + const numericId = jid.replace(/^tg:/, ''); + + // Telegram has a 4096 character limit per message — split if needed + const MAX_LENGTH = 4096; + if (text.length <= MAX_LENGTH) { + await this.bot.api.sendMessage(numericId, text); + } else { + for (let i = 0; i < text.length; i += MAX_LENGTH) { + await this.bot.api.sendMessage( + numericId, + text.slice(i, i + MAX_LENGTH), + ); + } + } + logger.info({ jid, length: text.length }, 'Telegram message sent'); + } catch (err) { + logger.error({ jid, err }, 'Failed to send Telegram message'); + } + } + + isConnected(): boolean { + return this.bot !== null; + } + + ownsJid(jid: string): boolean { + return jid.startsWith('tg:'); + } + + async disconnect(): Promise { + if (this.bot) { + this.bot.stop(); + this.bot = null; + logger.info('Telegram bot stopped'); + } + } + + async setTyping(jid: string, isTyping: boolean): Promise { + if (!this.bot || !isTyping) return; + try { + const numericId = jid.replace(/^tg:/, ''); + await this.bot.api.sendChatAction(numericId, 'typing'); + } catch (err) { + logger.debug({ jid, err }, 'Failed to send Telegram typing indicator'); + } + } +} diff --git a/.agent/skills/add-telegram/manifest.yaml b/.agent/skills/add-telegram/manifest.yaml new file mode 100644 index 0000000..fe7a36a --- /dev/null +++ b/.agent/skills/add-telegram/manifest.yaml @@ -0,0 +1,20 @@ +skill: telegram +version: 1.0.0 +description: "Telegram Bot API integration via Grammy" +core_version: 0.1.0 +adds: + - src/channels/telegram.ts + - src/channels/telegram.test.ts +modifies: + - src/index.ts + - src/config.ts + - src/routing.test.ts +structured: + npm_dependencies: + grammy: "^1.39.3" + env_additions: + - TELEGRAM_BOT_TOKEN + - TELEGRAM_ONLY +conflicts: [] +depends: [] +test: "npx vitest run src/channels/telegram.test.ts" diff --git a/.agent/skills/add-telegram/modify/src/config.ts b/.agent/skills/add-telegram/modify/src/config.ts new file mode 100644 index 0000000..b0de64c --- /dev/null +++ b/.agent/skills/add-telegram/modify/src/config.ts @@ -0,0 +1,77 @@ +import os from 'os'; +import path from 'path'; + +import { readEnvFile } from './env.js'; + +// Read config values from .env (falls back to process.env). +// Secrets are NOT read here — they stay on disk and are loaded only +// where needed (container-runner.ts) to avoid leaking to child processes. +const envConfig = readEnvFile([ + 'ASSISTANT_NAME', + 'ASSISTANT_HAS_OWN_NUMBER', + 'TELEGRAM_BOT_TOKEN', + 'TELEGRAM_ONLY', +]); + +export const ASSISTANT_NAME = + process.env.ASSISTANT_NAME || envConfig.ASSISTANT_NAME || 'Andy'; +export const ASSISTANT_HAS_OWN_NUMBER = + (process.env.ASSISTANT_HAS_OWN_NUMBER || envConfig.ASSISTANT_HAS_OWN_NUMBER) === 'true'; +export const POLL_INTERVAL = 2000; +export const SCHEDULER_POLL_INTERVAL = 60000; + +// Absolute paths needed for container mounts +const PROJECT_ROOT = process.cwd(); +const HOME_DIR = process.env.HOME || os.homedir(); + +// Mount security: allowlist stored OUTSIDE project root, never mounted into containers +export const MOUNT_ALLOWLIST_PATH = path.join( + HOME_DIR, + '.config', + (process.env.AGENT_NAME || 'clawdie') + '-cp', + 'mount-allowlist.json', +); +export const STORE_DIR = path.resolve(PROJECT_ROOT, 'store'); +export const GROUPS_DIR = path.resolve(PROJECT_ROOT, 'groups'); +export const DATA_DIR = path.resolve(PROJECT_ROOT, 'data'); +export const MAIN_GROUP_FOLDER = 'main'; + +export const CONTAINER_IMAGE = + process.env.CONTAINER_IMAGE || (process.env.AGENT_NAME || 'clawdie') + '-cp-agent:latest'; +export const CONTAINER_TIMEOUT = parseInt( + process.env.CONTAINER_TIMEOUT || '1800000', + 10, +); +export const CONTAINER_MAX_OUTPUT_SIZE = parseInt( + process.env.CONTAINER_MAX_OUTPUT_SIZE || '10485760', + 10, +); // 10MB default +export const IPC_POLL_INTERVAL = 1000; +export const IDLE_TIMEOUT = parseInt( + process.env.IDLE_TIMEOUT || '1800000', + 10, +); // 30min default — how long to keep container alive after last result +export const MAX_CONCURRENT_CONTAINERS = Math.max( + 1, + parseInt(process.env.MAX_CONCURRENT_CONTAINERS || '5', 10) || 5, +); + +function escapeRegex(str: string): string { + return str.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); +} + +export const TRIGGER_PATTERN = new RegExp( + `^@${escapeRegex(ASSISTANT_NAME)}\\b`, + 'i', +); + +// Timezone for scheduled tasks (cron expressions, etc.) +// Uses system timezone by default +export const TIMEZONE = + process.env.TZ || Intl.DateTimeFormat().resolvedOptions().timeZone; + +// Telegram configuration +export const TELEGRAM_BOT_TOKEN = + process.env.TELEGRAM_BOT_TOKEN || envConfig.TELEGRAM_BOT_TOKEN || ''; +export const TELEGRAM_ONLY = + (process.env.TELEGRAM_ONLY || envConfig.TELEGRAM_ONLY) === 'true'; diff --git a/.agent/skills/add-telegram/modify/src/config.ts.intent.md b/.agent/skills/add-telegram/modify/src/config.ts.intent.md new file mode 100644 index 0000000..1057ae4 --- /dev/null +++ b/.agent/skills/add-telegram/modify/src/config.ts.intent.md @@ -0,0 +1,25 @@ +# Intent: src/config.ts modifications + +## What changed + +Added two new configuration exports for Telegram channel support. + +## Key sections + +- **readEnvFile call**: Must include `TELEGRAM_BOT_TOKEN` and `TELEGRAM_ONLY` in the keys array. NanoClaw does NOT load `.env` into `process.env` — all `.env` values must be explicitly requested via `readEnvFile()`. +- **TELEGRAM_BOT_TOKEN**: Read from `process.env` first, then `envConfig` fallback, defaults to empty string (channel disabled when empty) +- **TELEGRAM_ONLY**: Boolean flag from `process.env` or `envConfig`, when `true` disables WhatsApp channel creation + +## Invariants + +- All existing config exports remain unchanged +- New Telegram keys are added to the `readEnvFile` call alongside existing keys +- New exports are appended at the end of the file +- No existing behavior is modified — Telegram config is additive only +- Both `process.env` and `envConfig` are checked (same pattern as `ASSISTANT_NAME`) + +## Must-keep + +- All existing exports (`ASSISTANT_NAME`, `POLL_INTERVAL`, `TRIGGER_PATTERN`, etc.) +- The `readEnvFile` pattern — ALL config read from `.env` must go through this function +- The `escapeRegex` helper and `TRIGGER_PATTERN` construction diff --git a/.agent/skills/add-telegram/modify/src/index.ts b/.agent/skills/add-telegram/modify/src/index.ts new file mode 100644 index 0000000..b91e244 --- /dev/null +++ b/.agent/skills/add-telegram/modify/src/index.ts @@ -0,0 +1,509 @@ +import fs from 'fs'; +import path from 'path'; + +import { + ASSISTANT_NAME, + IDLE_TIMEOUT, + MAIN_GROUP_FOLDER, + POLL_INTERVAL, + TELEGRAM_BOT_TOKEN, + TELEGRAM_ONLY, + TRIGGER_PATTERN, +} from './config.js'; +import { TelegramChannel } from './channels/telegram.js'; +import { WhatsAppChannel } from './channels/whatsapp.js'; +import { + ContainerOutput, + runContainerAgent, + writeGroupsSnapshot, + writeTasksSnapshot, +} from './container-runner.js'; +import { cleanupOrphans, ensureContainerRuntimeRunning } from './container-runtime.js'; +import { + getAllChats, + getAllRegisteredGroups, + getAllSessions, + getAllTasks, + getMessagesSince, + getNewMessages, + getRouterState, + initDatabase, + setRegisteredGroup, + setRouterState, + setSession, + storeChatMetadata, + storeMessage, +} from './db.js'; +import { GroupQueue } from './group-queue.js'; +import { resolveGroupFolderPath } from './group-folder.js'; +import { startIpcWatcher } from './ipc.js'; +import { findChannel, formatMessages, formatOutbound } from './router.js'; +import { startSchedulerLoop } from './task-scheduler.js'; +import { Channel, NewMessage, RegisteredGroup } from './types.js'; +import { logger } from './logger.js'; + +// Re-export for backwards compatibility during refactor +export { escapeXml, formatMessages } from './router.js'; + +let lastTimestamp = ''; +let sessions: Record = {}; +let registeredGroups: Record = {}; +let lastAgentTimestamp: Record = {}; +let messageLoopRunning = false; + +let whatsapp: WhatsAppChannel; +const channels: Channel[] = []; +const queue = new GroupQueue(); + +function loadState(): void { + lastTimestamp = getRouterState('last_timestamp') || ''; + const agentTs = getRouterState('last_agent_timestamp'); + try { + lastAgentTimestamp = agentTs ? JSON.parse(agentTs) : {}; + } catch { + logger.warn('Corrupted last_agent_timestamp in DB, resetting'); + lastAgentTimestamp = {}; + } + sessions = getAllSessions(); + registeredGroups = getAllRegisteredGroups(); + logger.info( + { groupCount: Object.keys(registeredGroups).length }, + 'State loaded', + ); +} + +function saveState(): void { + setRouterState('last_timestamp', lastTimestamp); + setRouterState( + 'last_agent_timestamp', + JSON.stringify(lastAgentTimestamp), + ); +} + +function registerGroup(jid: string, group: RegisteredGroup): void { + let groupDir: string; + try { + groupDir = resolveGroupFolderPath(group.folder); + } catch (err) { + logger.warn( + { jid, folder: group.folder, err }, + 'Rejecting group registration with invalid folder', + ); + return; + } + + registeredGroups[jid] = group; + setRegisteredGroup(jid, group); + + // Create group folder + fs.mkdirSync(path.join(groupDir, 'logs'), { recursive: true }); + + logger.info( + { jid, name: group.name, folder: group.folder }, + 'Group registered', + ); +} + +/** + * Get available groups list for the agent. + * Returns groups ordered by most recent activity. + */ +export function getAvailableGroups(): import('./container-runner.js').AvailableGroup[] { + const chats = getAllChats(); + const registeredJids = new Set(Object.keys(registeredGroups)); + + return chats + .filter((c) => c.jid !== '__group_sync__' && c.is_group) + .map((c) => ({ + jid: c.jid, + name: c.name, + lastActivity: c.last_message_time, + isRegistered: registeredJids.has(c.jid), + })); +} + +/** @internal - exported for testing */ +export function _setRegisteredGroups(groups: Record): void { + registeredGroups = groups; +} + +/** + * Process all pending messages for a group. + * Called by the GroupQueue when it's this group's turn. + */ +async function processGroupMessages(chatJid: string): Promise { + const group = registeredGroups[chatJid]; + if (!group) return true; + + const channel = findChannel(channels, chatJid); + if (!channel) { + console.log(`Warning: no channel owns JID ${chatJid}, skipping messages`); + return true; + } + + const isMainGroup = group.folder === MAIN_GROUP_FOLDER; + + const sinceTimestamp = lastAgentTimestamp[chatJid] || ''; + const missedMessages = getMessagesSince(chatJid, sinceTimestamp, ASSISTANT_NAME); + + if (missedMessages.length === 0) return true; + + // For non-main groups, check if trigger is required and present + if (!isMainGroup && group.requiresTrigger !== false) { + const hasTrigger = missedMessages.some((m) => + TRIGGER_PATTERN.test(m.content.trim()), + ); + if (!hasTrigger) return true; + } + + const prompt = formatMessages(missedMessages); + + // Advance cursor so the piping path in startMessageLoop won't re-fetch + // these messages. Save the old cursor so we can roll back on error. + const previousCursor = lastAgentTimestamp[chatJid] || ''; + lastAgentTimestamp[chatJid] = + missedMessages[missedMessages.length - 1].timestamp; + saveState(); + + logger.info( + { group: group.name, messageCount: missedMessages.length }, + 'Processing messages', + ); + + // Track idle timer for closing stdin when agent is idle + let idleTimer: ReturnType | null = null; + + const resetIdleTimer = () => { + if (idleTimer) clearTimeout(idleTimer); + idleTimer = setTimeout(() => { + logger.debug({ group: group.name }, 'Idle timeout, closing container stdin'); + queue.closeStdin(chatJid); + }, IDLE_TIMEOUT); + }; + + await channel.setTyping?.(chatJid, true); + let hadError = false; + let outputSentToUser = false; + + const output = await runAgent(group, prompt, chatJid, async (result) => { + // Streaming output callback — called for each agent result + if (result.result) { + const raw = typeof result.result === 'string' ? result.result : JSON.stringify(result.result); + // Strip ... blocks — agent uses these for internal reasoning + const text = raw.replace(/[\s\S]*?<\/internal>/g, '').trim(); + logger.info({ group: group.name }, `Agent output: ${raw.slice(0, 200)}`); + if (text) { + await channel.sendMessage(chatJid, text); + outputSentToUser = true; + } + // Only reset idle timer on actual results, not session-update markers (result: null) + resetIdleTimer(); + } + + if (result.status === 'success') { + queue.notifyIdle(chatJid); + } + + if (result.status === 'error') { + hadError = true; + } + }); + + await channel.setTyping?.(chatJid, false); + if (idleTimer) clearTimeout(idleTimer); + + if (output === 'error' || hadError) { + // If we already sent output to the user, don't roll back the cursor — + // the user got their response and re-processing would send duplicates. + if (outputSentToUser) { + logger.warn({ group: group.name }, 'Agent error after output was sent, skipping cursor rollback to prevent duplicates'); + return true; + } + // Roll back cursor so retries can re-process these messages + lastAgentTimestamp[chatJid] = previousCursor; + saveState(); + logger.warn({ group: group.name }, 'Agent error, rolled back message cursor for retry'); + return false; + } + + return true; +} + +async function runAgent( + group: RegisteredGroup, + prompt: string, + chatJid: string, + onOutput?: (output: ContainerOutput) => Promise, +): Promise<'success' | 'error'> { + const isMain = group.folder === MAIN_GROUP_FOLDER; + const sessionId = sessions[group.folder]; + + // Update tasks snapshot for container to read (filtered by group) + const tasks = getAllTasks(); + writeTasksSnapshot( + group.folder, + isMain, + tasks.map((t) => ({ + id: t.id, + groupFolder: t.group_folder, + prompt: t.prompt, + schedule_type: t.schedule_type, + schedule_value: t.schedule_value, + status: t.status, + next_run: t.next_run, + })), + ); + + // Update available groups snapshot (main group only can see all groups) + const availableGroups = getAvailableGroups(); + writeGroupsSnapshot( + group.folder, + isMain, + availableGroups, + new Set(Object.keys(registeredGroups)), + ); + + // Wrap onOutput to track session ID from streamed results + const wrappedOnOutput = onOutput + ? async (output: ContainerOutput) => { + if (output.newSessionId) { + sessions[group.folder] = output.newSessionId; + setSession(group.folder, output.newSessionId); + } + await onOutput(output); + } + : undefined; + + try { + const output = await runContainerAgent( + group, + { + prompt, + sessionId, + groupFolder: group.folder, + chatJid, + isMain, + assistantName: ASSISTANT_NAME, + }, + (proc, containerName) => queue.registerProcess(chatJid, proc, containerName, group.folder), + wrappedOnOutput, + ); + + if (output.newSessionId) { + sessions[group.folder] = output.newSessionId; + setSession(group.folder, output.newSessionId); + } + + if (output.status === 'error') { + logger.error( + { group: group.name, error: output.error }, + 'Container agent error', + ); + return 'error'; + } + + return 'success'; + } catch (err) { + logger.error({ group: group.name, err }, 'Agent error'); + return 'error'; + } +} + +async function startMessageLoop(): Promise { + if (messageLoopRunning) { + logger.debug('Message loop already running, skipping duplicate start'); + return; + } + messageLoopRunning = true; + + logger.info(`NanoClaw running (trigger: @${ASSISTANT_NAME})`); + + while (true) { + try { + const jids = Object.keys(registeredGroups); + const { messages, newTimestamp } = getNewMessages(jids, lastTimestamp, ASSISTANT_NAME); + + if (messages.length > 0) { + logger.info({ count: messages.length }, 'New messages'); + + // Advance the "seen" cursor for all messages immediately + lastTimestamp = newTimestamp; + saveState(); + + // Deduplicate by group + const messagesByGroup = new Map(); + for (const msg of messages) { + const existing = messagesByGroup.get(msg.chat_jid); + if (existing) { + existing.push(msg); + } else { + messagesByGroup.set(msg.chat_jid, [msg]); + } + } + + for (const [chatJid, groupMessages] of messagesByGroup) { + const group = registeredGroups[chatJid]; + if (!group) continue; + + const channel = findChannel(channels, chatJid); + if (!channel) { + console.log(`Warning: no channel owns JID ${chatJid}, skipping messages`); + continue; + } + + const isMainGroup = group.folder === MAIN_GROUP_FOLDER; + const needsTrigger = !isMainGroup && group.requiresTrigger !== false; + + // For non-main groups, only act on trigger messages. + // Non-trigger messages accumulate in DB and get pulled as + // context when a trigger eventually arrives. + if (needsTrigger) { + const hasTrigger = groupMessages.some((m) => + TRIGGER_PATTERN.test(m.content.trim()), + ); + if (!hasTrigger) continue; + } + + // Pull all messages since lastAgentTimestamp so non-trigger + // context that accumulated between triggers is included. + const allPending = getMessagesSince( + chatJid, + lastAgentTimestamp[chatJid] || '', + ASSISTANT_NAME, + ); + const messagesToSend = + allPending.length > 0 ? allPending : groupMessages; + const formatted = formatMessages(messagesToSend); + + if (queue.sendMessage(chatJid, formatted)) { + logger.debug( + { chatJid, count: messagesToSend.length }, + 'Piped messages to active container', + ); + lastAgentTimestamp[chatJid] = + messagesToSend[messagesToSend.length - 1].timestamp; + saveState(); + // Show typing indicator while the container processes the piped message + channel.setTyping?.(chatJid, true)?.catch((err) => + logger.warn({ chatJid, err }, 'Failed to set typing indicator'), + ); + } else { + // No active container — enqueue for a new one + queue.enqueueMessageCheck(chatJid); + } + } + } + } catch (err) { + logger.error({ err }, 'Error in message loop'); + } + await new Promise((resolve) => setTimeout(resolve, POLL_INTERVAL)); + } +} + +/** + * Startup recovery: check for unprocessed messages in registered groups. + * Handles crash between advancing lastTimestamp and processing messages. + */ +function recoverPendingMessages(): void { + for (const [chatJid, group] of Object.entries(registeredGroups)) { + const sinceTimestamp = lastAgentTimestamp[chatJid] || ''; + const pending = getMessagesSince(chatJid, sinceTimestamp, ASSISTANT_NAME); + if (pending.length > 0) { + logger.info( + { group: group.name, pendingCount: pending.length }, + 'Recovery: found unprocessed messages', + ); + queue.enqueueMessageCheck(chatJid); + } + } +} + +function ensureContainerSystemRunning(): void { + ensureContainerRuntimeRunning(); + cleanupOrphans(); +} + +async function main(): Promise { + ensureContainerSystemRunning(); + initDatabase(); + logger.info('Database initialized'); + loadState(); + + // Graceful shutdown handlers + const shutdown = async (signal: string) => { + logger.info({ signal }, 'Shutdown signal received'); + await queue.shutdown(10000); + for (const ch of channels) await ch.disconnect(); + process.exit(0); + }; + process.on('SIGTERM', () => shutdown('SIGTERM')); + process.on('SIGINT', () => shutdown('SIGINT')); + + // Channel callbacks (shared by all channels) + const channelOpts = { + onMessage: (_chatJid: string, msg: NewMessage) => storeMessage(msg), + onChatMetadata: (chatJid: string, timestamp: string, name?: string, channel?: string, isGroup?: boolean) => + storeChatMetadata(chatJid, timestamp, name, channel, isGroup), + registeredGroups: () => registeredGroups, + }; + + // Create and connect channels + if (TELEGRAM_BOT_TOKEN) { + const telegram = new TelegramChannel(TELEGRAM_BOT_TOKEN, channelOpts); + channels.push(telegram); + await telegram.connect(); + } + + if (!TELEGRAM_ONLY) { + whatsapp = new WhatsAppChannel(channelOpts); + channels.push(whatsapp); + await whatsapp.connect(); + } + + // Start subsystems (independently of connection handler) + startSchedulerLoop({ + registeredGroups: () => registeredGroups, + getSessions: () => sessions, + queue, + onProcess: (groupJid, proc, containerName, groupFolder) => queue.registerProcess(groupJid, proc, containerName, groupFolder), + sendMessage: async (jid, rawText) => { + const channel = findChannel(channels, jid); + if (!channel) { + console.log(`Warning: no channel owns JID ${jid}, cannot send message`); + return; + } + const text = formatOutbound(rawText); + if (text) await channel.sendMessage(jid, text); + }, + }); + startIpcWatcher({ + sendMessage: (jid, text) => { + const channel = findChannel(channels, jid); + if (!channel) throw new Error(`No channel for JID: ${jid}`); + return channel.sendMessage(jid, text); + }, + registeredGroups: () => registeredGroups, + registerGroup, + syncGroupMetadata: (force) => whatsapp?.syncGroupMetadata(force) ?? Promise.resolve(), + getAvailableGroups, + writeGroupsSnapshot: (gf, im, ag, rj) => writeGroupsSnapshot(gf, im, ag, rj), + }); + queue.setProcessMessagesFn(processGroupMessages); + recoverPendingMessages(); + startMessageLoop().catch((err) => { + logger.fatal({ err }, 'Message loop crashed unexpectedly'); + process.exit(1); + }); +} + +// Guard: only run when executed directly, not when imported by tests +const isDirectRun = + process.argv[1] && + new URL(import.meta.url).pathname === new URL(`file://${process.argv[1]}`).pathname; + +if (isDirectRun) { + main().catch((err) => { + logger.error({ err }, 'Failed to start NanoClaw'); + process.exit(1); + }); +} diff --git a/.agent/skills/add-telegram/modify/src/index.ts.intent.md b/.agent/skills/add-telegram/modify/src/index.ts.intent.md new file mode 100644 index 0000000..16d2d1f --- /dev/null +++ b/.agent/skills/add-telegram/modify/src/index.ts.intent.md @@ -0,0 +1,59 @@ +# Intent: src/index.ts modifications + +## What changed + +Refactored from single WhatsApp channel to multi-channel architecture using the `Channel` interface. + +## Key sections + +### Imports (top of file) + +- Added: `TelegramChannel` from `./channels/telegram.js` +- Added: `TELEGRAM_BOT_TOKEN`, `TELEGRAM_ONLY` from `./config.js` +- Added: `findChannel` from `./router.js` +- Added: `Channel` type from `./types.js` + +### Module-level state + +- Added: `const channels: Channel[] = []` — array of all active channels +- Kept: `let whatsapp: WhatsAppChannel` — still needed for `syncGroupMetadata` reference + +### processGroupMessages() + +- Added: `findChannel(channels, chatJid)` lookup at the start +- Changed: `whatsapp.setTyping()` → `channel.setTyping?.()` (optional chaining) +- Changed: `whatsapp.sendMessage()` → `channel.sendMessage()` in output callback + +### getAvailableGroups() + +- Unchanged: uses `c.is_group` filter from base (Telegram channels pass `isGroup=true` via `onChatMetadata`) + +### startMessageLoop() + +- Added: `findChannel(channels, chatJid)` lookup per group in message processing +- Changed: `whatsapp.setTyping()` → `channel.setTyping?.()` for typing indicators + +### main() + +- Changed: shutdown disconnects all channels via `for (const ch of channels)` +- Added: shared `channelOpts` object for channel callbacks +- Added: conditional WhatsApp creation (`if (!TELEGRAM_ONLY)`) +- Added: conditional Telegram creation (`if (TELEGRAM_BOT_TOKEN)`) +- Changed: scheduler `sendMessage` uses `findChannel()` → `channel.sendMessage()` +- Changed: IPC `sendMessage` uses `findChannel()` → `channel.sendMessage()` + +## Invariants + +- All existing message processing logic (triggers, cursors, idle timers) is preserved +- The `runAgent` function is completely unchanged +- State management (loadState/saveState) is unchanged +- Recovery logic is unchanged +- Container runtime check is unchanged (ensureContainerSystemRunning) + +## Must-keep + +- The `escapeXml` and `formatMessages` re-exports +- The `_setRegisteredGroups` test helper +- The `isDirectRun` guard at bottom +- All error handling and cursor rollback logic in processGroupMessages +- The outgoing queue flush and reconnection logic (in WhatsAppChannel, not here) diff --git a/.agent/skills/add-telegram/modify/src/routing.test.ts b/.agent/skills/add-telegram/modify/src/routing.test.ts new file mode 100644 index 0000000..5b44063 --- /dev/null +++ b/.agent/skills/add-telegram/modify/src/routing.test.ts @@ -0,0 +1,161 @@ +import { describe, it, expect, beforeEach } from 'vitest'; + +import { _initTestDatabase, getAllChats, storeChatMetadata } from './db.js'; +import { getAvailableGroups, _setRegisteredGroups } from './index.js'; + +beforeEach(() => { + _initTestDatabase(); + _setRegisteredGroups({}); +}); + +// --- JID ownership patterns --- + +describe('JID ownership patterns', () => { + // These test the patterns that will become ownsJid() on the Channel interface + + it('WhatsApp group JID: ends with @g.us', () => { + const jid = '12345678@g.us'; + expect(jid.endsWith('@g.us')).toBe(true); + }); + + it('WhatsApp DM JID: ends with @s.whatsapp.net', () => { + const jid = '12345678@s.whatsapp.net'; + expect(jid.endsWith('@s.whatsapp.net')).toBe(true); + }); + + it('Telegram JID: starts with tg:', () => { + const jid = 'tg:123456789'; + expect(jid.startsWith('tg:')).toBe(true); + }); + + it('Telegram group JID: starts with tg: and has negative ID', () => { + const jid = 'tg:-1001234567890'; + expect(jid.startsWith('tg:')).toBe(true); + }); +}); + +// --- getAvailableGroups --- + +describe('getAvailableGroups', () => { + it('returns only groups, excludes DMs', () => { + storeChatMetadata('group1@g.us', '2024-01-01T00:00:01.000Z', 'Group 1', 'whatsapp', true); + storeChatMetadata('user@s.whatsapp.net', '2024-01-01T00:00:02.000Z', 'User DM', 'whatsapp', false); + storeChatMetadata('group2@g.us', '2024-01-01T00:00:03.000Z', 'Group 2', 'whatsapp', true); + + const groups = getAvailableGroups(); + expect(groups).toHaveLength(2); + expect(groups.map((g) => g.jid)).toContain('group1@g.us'); + expect(groups.map((g) => g.jid)).toContain('group2@g.us'); + expect(groups.map((g) => g.jid)).not.toContain('user@s.whatsapp.net'); + }); + + it('excludes __group_sync__ sentinel', () => { + storeChatMetadata('__group_sync__', '2024-01-01T00:00:00.000Z'); + storeChatMetadata('group@g.us', '2024-01-01T00:00:01.000Z', 'Group', 'whatsapp', true); + + const groups = getAvailableGroups(); + expect(groups).toHaveLength(1); + expect(groups[0].jid).toBe('group@g.us'); + }); + + it('marks registered groups correctly', () => { + storeChatMetadata('reg@g.us', '2024-01-01T00:00:01.000Z', 'Registered', 'whatsapp', true); + storeChatMetadata('unreg@g.us', '2024-01-01T00:00:02.000Z', 'Unregistered', 'whatsapp', true); + + _setRegisteredGroups({ + 'reg@g.us': { + name: 'Registered', + folder: 'registered', + trigger: '@Andy', + added_at: '2024-01-01T00:00:00.000Z', + }, + }); + + const groups = getAvailableGroups(); + const reg = groups.find((g) => g.jid === 'reg@g.us'); + const unreg = groups.find((g) => g.jid === 'unreg@g.us'); + + expect(reg?.isRegistered).toBe(true); + expect(unreg?.isRegistered).toBe(false); + }); + + it('returns groups ordered by most recent activity', () => { + storeChatMetadata('old@g.us', '2024-01-01T00:00:01.000Z', 'Old', 'whatsapp', true); + storeChatMetadata('new@g.us', '2024-01-01T00:00:05.000Z', 'New', 'whatsapp', true); + storeChatMetadata('mid@g.us', '2024-01-01T00:00:03.000Z', 'Mid', 'whatsapp', true); + + const groups = getAvailableGroups(); + expect(groups[0].jid).toBe('new@g.us'); + expect(groups[1].jid).toBe('mid@g.us'); + expect(groups[2].jid).toBe('old@g.us'); + }); + + it('excludes non-group chats regardless of JID format', () => { + // Unknown JID format stored without is_group should not appear + storeChatMetadata('unknown-format-123', '2024-01-01T00:00:01.000Z', 'Unknown'); + // Explicitly non-group with unusual JID + storeChatMetadata('custom:abc', '2024-01-01T00:00:02.000Z', 'Custom DM', 'custom', false); + // A real group for contrast + storeChatMetadata('group@g.us', '2024-01-01T00:00:03.000Z', 'Group', 'whatsapp', true); + + const groups = getAvailableGroups(); + expect(groups).toHaveLength(1); + expect(groups[0].jid).toBe('group@g.us'); + }); + + it('returns empty array when no chats exist', () => { + const groups = getAvailableGroups(); + expect(groups).toHaveLength(0); + }); + + it('includes Telegram chat JIDs', () => { + storeChatMetadata('tg:100200300', '2024-01-01T00:00:01.000Z', 'Telegram Chat', 'telegram', true); + storeChatMetadata('user@s.whatsapp.net', '2024-01-01T00:00:02.000Z', 'User DM', 'whatsapp', false); + + const groups = getAvailableGroups(); + expect(groups).toHaveLength(1); + expect(groups[0].jid).toBe('tg:100200300'); + }); + + it('returns Telegram group JIDs with negative IDs', () => { + storeChatMetadata('tg:-1001234567890', '2024-01-01T00:00:01.000Z', 'TG Group', 'telegram', true); + + const groups = getAvailableGroups(); + expect(groups).toHaveLength(1); + expect(groups[0].jid).toBe('tg:-1001234567890'); + expect(groups[0].name).toBe('TG Group'); + }); + + it('marks registered Telegram chats correctly', () => { + storeChatMetadata('tg:100200300', '2024-01-01T00:00:01.000Z', 'TG Registered', 'telegram', true); + storeChatMetadata('tg:999999', '2024-01-01T00:00:02.000Z', 'TG Unregistered', 'telegram', true); + + _setRegisteredGroups({ + 'tg:100200300': { + name: 'TG Registered', + folder: 'tg-registered', + trigger: '@Andy', + added_at: '2024-01-01T00:00:00.000Z', + }, + }); + + const groups = getAvailableGroups(); + const tgReg = groups.find((g) => g.jid === 'tg:100200300'); + const tgUnreg = groups.find((g) => g.jid === 'tg:999999'); + + expect(tgReg?.isRegistered).toBe(true); + expect(tgUnreg?.isRegistered).toBe(false); + }); + + it('mixes WhatsApp and Telegram chats ordered by activity', () => { + storeChatMetadata('wa@g.us', '2024-01-01T00:00:01.000Z', 'WhatsApp', 'whatsapp', true); + storeChatMetadata('tg:100', '2024-01-01T00:00:03.000Z', 'Telegram', 'telegram', true); + storeChatMetadata('wa2@g.us', '2024-01-01T00:00:02.000Z', 'WhatsApp 2', 'whatsapp', true); + + const groups = getAvailableGroups(); + expect(groups).toHaveLength(3); + expect(groups[0].jid).toBe('tg:100'); + expect(groups[1].jid).toBe('wa2@g.us'); + expect(groups[2].jid).toBe('wa@g.us'); + }); +}); diff --git a/.agent/skills/add-telegram/tests/telegram.test.ts b/.agent/skills/add-telegram/tests/telegram.test.ts new file mode 100644 index 0000000..50dd599 --- /dev/null +++ b/.agent/skills/add-telegram/tests/telegram.test.ts @@ -0,0 +1,118 @@ +import { describe, expect, it } from 'vitest'; +import fs from 'fs'; +import path from 'path'; + +describe('telegram skill package', () => { + const skillDir = path.resolve(__dirname, '..'); + + it('has a valid manifest', () => { + const manifestPath = path.join(skillDir, 'manifest.yaml'); + expect(fs.existsSync(manifestPath)).toBe(true); + + const content = fs.readFileSync(manifestPath, 'utf-8'); + expect(content).toContain('skill: telegram'); + expect(content).toContain('version: 1.0.0'); + expect(content).toContain('grammy'); + }); + + it('has all files declared in adds', () => { + const addFile = path.join(skillDir, 'add', 'src', 'channels', 'telegram.ts'); + expect(fs.existsSync(addFile)).toBe(true); + + const content = fs.readFileSync(addFile, 'utf-8'); + expect(content).toContain('class TelegramChannel'); + expect(content).toContain('implements Channel'); + + // Test file for the channel + const testFile = path.join(skillDir, 'add', 'src', 'channels', 'telegram.test.ts'); + expect(fs.existsSync(testFile)).toBe(true); + + const testContent = fs.readFileSync(testFile, 'utf-8'); + expect(testContent).toContain("describe('TelegramChannel'"); + }); + + it('has all files declared in modifies', () => { + const indexFile = path.join(skillDir, 'modify', 'src', 'index.ts'); + const configFile = path.join(skillDir, 'modify', 'src', 'config.ts'); + const routingTestFile = path.join(skillDir, 'modify', 'src', 'routing.test.ts'); + + expect(fs.existsSync(indexFile)).toBe(true); + expect(fs.existsSync(configFile)).toBe(true); + expect(fs.existsSync(routingTestFile)).toBe(true); + + const indexContent = fs.readFileSync(indexFile, 'utf-8'); + expect(indexContent).toContain('TelegramChannel'); + expect(indexContent).toContain('TELEGRAM_BOT_TOKEN'); + expect(indexContent).toContain('TELEGRAM_ONLY'); + expect(indexContent).toContain('findChannel'); + expect(indexContent).toContain('channels: Channel[]'); + + const configContent = fs.readFileSync(configFile, 'utf-8'); + expect(configContent).toContain('TELEGRAM_BOT_TOKEN'); + expect(configContent).toContain('TELEGRAM_ONLY'); + }); + + it('has intent files for modified files', () => { + expect(fs.existsSync(path.join(skillDir, 'modify', 'src', 'index.ts.intent.md'))).toBe(true); + expect(fs.existsSync(path.join(skillDir, 'modify', 'src', 'config.ts.intent.md'))).toBe(true); + }); + + it('modified index.ts preserves core structure', () => { + const content = fs.readFileSync( + path.join(skillDir, 'modify', 'src', 'index.ts'), + 'utf-8', + ); + + // Core functions still present + expect(content).toContain('function loadState()'); + expect(content).toContain('function saveState()'); + expect(content).toContain('function registerGroup('); + expect(content).toContain('function getAvailableGroups()'); + expect(content).toContain('function processGroupMessages('); + expect(content).toContain('function runAgent('); + expect(content).toContain('function startMessageLoop()'); + expect(content).toContain('function recoverPendingMessages()'); + expect(content).toContain('function ensureContainerSystemRunning()'); + expect(content).toContain('async function main()'); + + // Test helper preserved + expect(content).toContain('_setRegisteredGroups'); + + // Direct-run guard preserved + expect(content).toContain('isDirectRun'); + }); + + it('modified index.ts includes Telegram channel creation', () => { + const content = fs.readFileSync( + path.join(skillDir, 'modify', 'src', 'index.ts'), + 'utf-8', + ); + + // Multi-channel architecture + expect(content).toContain('const channels: Channel[] = []'); + expect(content).toContain('channels.push(whatsapp)'); + expect(content).toContain('channels.push(telegram)'); + + // Conditional channel creation + expect(content).toContain('if (!TELEGRAM_ONLY)'); + expect(content).toContain('if (TELEGRAM_BOT_TOKEN)'); + + // Shutdown disconnects all channels + expect(content).toContain('for (const ch of channels) await ch.disconnect()'); + }); + + it('modified config.ts preserves all existing exports', () => { + const content = fs.readFileSync( + path.join(skillDir, 'modify', 'src', 'config.ts'), + 'utf-8', + ); + + // All original exports preserved + expect(content).toContain('export const ASSISTANT_NAME'); + expect(content).toContain('export const POLL_INTERVAL'); + expect(content).toContain('export const TRIGGER_PATTERN'); + expect(content).toContain('export const CONTAINER_IMAGE'); + expect(content).toContain('export const DATA_DIR'); + expect(content).toContain('export const TIMEZONE'); + }); +}); diff --git a/.agent/skills/add-voice-transcription/SKILL.md b/.agent/skills/add-voice-transcription/SKILL.md new file mode 100644 index 0000000..9997ff4 --- /dev/null +++ b/.agent/skills/add-voice-transcription/SKILL.md @@ -0,0 +1,145 @@ +--- +name: add-voice-transcription +description: Add voice message transcription to NanoClaw using OpenAI's Whisper API. Automatically transcribes WhatsApp voice notes so the agent can read and respond to them. +--- + +# Add Voice Transcription + +This skill adds automatic voice message transcription to NanoClaw's WhatsApp channel using OpenAI's Whisper API. When a voice note arrives, it is downloaded, transcribed, and delivered to the agent as `[Voice: ]`. + +## Phase 1: Pre-flight + +### Check if already applied + +Read `.nanoclaw/state.yaml`. If `voice-transcription` is in `applied_skills`, skip to Phase 3 (Configure). The code changes are already in place. + +### Ask the user + +Use `AskUserQuestion` to collect information: + +AskUserQuestion: Do you have an OpenAI API key for Whisper transcription? + +If yes, collect it now. If no, direct them to create one at https://platform.openai.com/api-keys. + +## Phase 2: Apply Code Changes + +Run the skills engine to apply this skill's code package. + +### Initialize skills system (if needed) + +If `.nanoclaw/` directory doesn't exist yet: + +```bash +npx tsx scripts/apply-skill.ts --init +``` + +### Apply the skill + +```bash +npx tsx scripts/apply-skill.ts .agent/skills/add-voice-transcription +``` + +This deterministically: + +- Adds `src/transcription.ts` (voice transcription module using OpenAI Whisper) +- Three-way merges voice handling into `src/channels/whatsapp.ts` (isVoiceMessage check, transcribeAudioMessage call) +- Three-way merges transcription tests into `src/channels/whatsapp.test.ts` (mock + 3 test cases) +- Installs the `openai` npm dependency +- Updates `.env.example` with `OPENAI_API_KEY` +- Records the application in `.nanoclaw/state.yaml` + +If the apply reports merge conflicts, read the intent files: + +- `modify/src/channels/whatsapp.ts.intent.md` — what changed and invariants for whatsapp.ts +- `modify/src/channels/whatsapp.test.ts.intent.md` — what changed for whatsapp.test.ts + +### Validate code changes + +```bash +npm test +npm run build +``` + +All tests must pass (including the 3 new voice transcription tests) and build must be clean before proceeding. + +## Phase 3: Configure + +### Get OpenAI API key (if needed) + +If the user doesn't have an API key: + +> I need you to create an OpenAI API key: +> +> 1. Go to https://platform.openai.com/api-keys +> 2. Click "Create new secret key" +> 3. Give it a name (e.g., "NanoClaw Transcription") +> 4. Copy the key (starts with `sk-`) +> +> Cost: ~$0.006 per minute of audio (~$0.003 per typical 30-second voice note) + +Wait for the user to provide the key. + +### Add to environment + +Add to `.env`: + +```bash +OPENAI_API_KEY= +``` + +Sync to container environment: + +```bash +mkdir -p data/env && cp .env data/env/env +``` + +The container reads environment from `data/env/env`, not `.env` directly. + +### Build and restart + +```bash +npm run build +launchctl kickstart -k gui/$(id -u)/com.nanoclaw # macOS +# Linux: systemctl --user restart nanoclaw +``` + +## Phase 4: Verify + +### Test with a voice note + +Tell the user: + +> Send a voice note in any registered WhatsApp chat. The agent should receive it as `[Voice: ]` and respond to its content. + +### Check logs if needed + +```bash +tail -f logs/nanoclaw.log | grep -i voice +``` + +Look for: + +- `Transcribed voice message` — successful transcription with character count +- `OPENAI_API_KEY not set` — key missing from `.env` +- `OpenAI transcription failed` — API error (check key validity, billing) +- `Failed to download audio message` — media download issue + +## Troubleshooting + +### Voice notes show "[Voice Message - transcription unavailable]" + +1. Check `OPENAI_API_KEY` is set in `.env` AND synced to `data/env/env` +2. Verify key works: `curl -s https://api.openai.com/v1/models -H "Authorization: Bearer $OPENAI_API_KEY" | head -c 200` +3. Check OpenAI billing — Whisper requires a funded account + +### Voice notes show "[Voice Message - transcription failed]" + +Check logs for the specific error. Common causes: + +- Network timeout — transient, will work on next message +- Invalid API key — regenerate at https://platform.openai.com/api-keys +- Rate limiting — wait and retry + +### Agent doesn't respond to voice notes + +Verify the chat is registered and the agent is running. Voice transcription only runs for registered groups. diff --git a/.agent/skills/add-voice-transcription/add/src/transcription.ts b/.agent/skills/add-voice-transcription/add/src/transcription.ts new file mode 100644 index 0000000..91c5e7f --- /dev/null +++ b/.agent/skills/add-voice-transcription/add/src/transcription.ts @@ -0,0 +1,98 @@ +import { downloadMediaMessage } from '@whiskeysockets/baileys'; +import { WAMessage, WASocket } from '@whiskeysockets/baileys'; + +import { readEnvFile } from './env.js'; + +interface TranscriptionConfig { + model: string; + enabled: boolean; + fallbackMessage: string; +} + +const DEFAULT_CONFIG: TranscriptionConfig = { + model: 'whisper-1', + enabled: true, + fallbackMessage: '[Voice Message - transcription unavailable]', +}; + +async function transcribeWithOpenAI( + audioBuffer: Buffer, + config: TranscriptionConfig, +): Promise { + const env = readEnvFile(['OPENAI_API_KEY']); + const apiKey = env.OPENAI_API_KEY; + + if (!apiKey) { + console.warn('OPENAI_API_KEY not set in .env'); + return null; + } + + try { + const openaiModule = await import('openai'); + const OpenAI = openaiModule.default; + const toFile = openaiModule.toFile; + + const openai = new OpenAI({ apiKey }); + + const file = await toFile(audioBuffer, 'voice.ogg', { + type: 'audio/ogg', + }); + + const transcription = await openai.audio.transcriptions.create({ + file: file, + model: config.model, + response_format: 'text', + }); + + // When response_format is 'text', the API returns a plain string + return transcription as unknown as string; + } catch (err) { + console.error('OpenAI transcription failed:', err); + return null; + } +} + +export async function transcribeAudioMessage( + msg: WAMessage, + sock: WASocket, +): Promise { + const config = DEFAULT_CONFIG; + + if (!config.enabled) { + return config.fallbackMessage; + } + + try { + const buffer = (await downloadMediaMessage( + msg, + 'buffer', + {}, + { + logger: console as any, + reuploadRequest: sock.updateMediaMessage, + }, + )) as Buffer; + + if (!buffer || buffer.length === 0) { + console.error('Failed to download audio message'); + return config.fallbackMessage; + } + + console.log(`Downloaded audio message: ${buffer.length} bytes`); + + const transcript = await transcribeWithOpenAI(buffer, config); + + if (!transcript) { + return config.fallbackMessage; + } + + return transcript.trim(); + } catch (err) { + console.error('Transcription error:', err); + return config.fallbackMessage; + } +} + +export function isVoiceMessage(msg: WAMessage): boolean { + return msg.message?.audioMessage?.ptt === true; +} diff --git a/.agent/skills/add-voice-transcription/manifest.yaml b/.agent/skills/add-voice-transcription/manifest.yaml new file mode 100644 index 0000000..cb4d587 --- /dev/null +++ b/.agent/skills/add-voice-transcription/manifest.yaml @@ -0,0 +1,17 @@ +skill: voice-transcription +version: 1.0.0 +description: "Voice message transcription via OpenAI Whisper" +core_version: 0.1.0 +adds: + - src/transcription.ts +modifies: + - src/channels/whatsapp.ts + - src/channels/whatsapp.test.ts +structured: + npm_dependencies: + openai: "^4.77.0" + env_additions: + - OPENAI_API_KEY +conflicts: [] +depends: [] +test: "npx vitest run src/channels/whatsapp.test.ts" diff --git a/.agent/skills/add-voice-transcription/modify/src/channels/whatsapp.test.ts b/.agent/skills/add-voice-transcription/modify/src/channels/whatsapp.test.ts new file mode 100644 index 0000000..30e79b0 --- /dev/null +++ b/.agent/skills/add-voice-transcription/modify/src/channels/whatsapp.test.ts @@ -0,0 +1,963 @@ +import { describe, it, expect, beforeEach, vi, afterEach } from 'vitest'; +import { EventEmitter } from 'events'; + +// --- Mocks --- + +// Mock config +vi.mock('../config.js', () => ({ + STORE_DIR: '/tmp/clawdie-cp-test-store', + ASSISTANT_NAME: 'Andy', + ASSISTANT_HAS_OWN_NUMBER: false, +})); + +// Mock logger +vi.mock('../logger.js', () => ({ + logger: { + debug: vi.fn(), + info: vi.fn(), + warn: vi.fn(), + error: vi.fn(), + }, +})); + +// Mock db +vi.mock('../db.js', () => ({ + getLastGroupSync: vi.fn(() => null), + setLastGroupSync: vi.fn(), + updateChatName: vi.fn(), +})); + +// Mock transcription +vi.mock('../transcription.js', () => ({ + isVoiceMessage: vi.fn((msg: any) => msg.message?.audioMessage?.ptt === true), + transcribeAudioMessage: vi.fn().mockResolvedValue('Hello this is a voice message'), +})); + +// Mock fs +vi.mock('fs', async () => { + const actual = await vi.importActual('fs'); + return { + ...actual, + default: { + ...actual, + existsSync: vi.fn(() => true), + mkdirSync: vi.fn(), + }, + }; +}); + +// Mock child_process (used for osascript notification) +vi.mock('child_process', () => ({ + exec: vi.fn(), +})); + +// Build a fake WASocket that's an EventEmitter with the methods we need +function createFakeSocket() { + const ev = new EventEmitter(); + const sock = { + ev: { + on: (event: string, handler: (...args: unknown[]) => void) => { + ev.on(event, handler); + }, + }, + user: { + id: '1234567890:1@s.whatsapp.net', + lid: '9876543210:1@lid', + }, + sendMessage: vi.fn().mockResolvedValue(undefined), + sendPresenceUpdate: vi.fn().mockResolvedValue(undefined), + groupFetchAllParticipating: vi.fn().mockResolvedValue({}), + end: vi.fn(), + // Expose the event emitter for triggering events in tests + _ev: ev, + }; + return sock; +} + +let fakeSocket: ReturnType; + +// Mock Baileys +vi.mock('@whiskeysockets/baileys', () => { + return { + default: vi.fn(() => fakeSocket), + Browsers: { macOS: vi.fn(() => ['macOS', 'Chrome', '']) }, + DisconnectReason: { + loggedOut: 401, + badSession: 500, + connectionClosed: 428, + connectionLost: 408, + connectionReplaced: 440, + timedOut: 408, + restartRequired: 515, + }, + makeCacheableSignalKeyStore: vi.fn((keys: unknown) => keys), + useMultiFileAuthState: vi.fn().mockResolvedValue({ + state: { + creds: {}, + keys: {}, + }, + saveCreds: vi.fn(), + }), + }; +}); + +import { WhatsAppChannel, WhatsAppChannelOpts } from './whatsapp.js'; +import { getLastGroupSync, updateChatName, setLastGroupSync } from '../db.js'; +import { transcribeAudioMessage } from '../transcription.js'; + +// --- Test helpers --- + +function createTestOpts(overrides?: Partial): WhatsAppChannelOpts { + return { + onMessage: vi.fn(), + onChatMetadata: vi.fn(), + registeredGroups: vi.fn(() => ({ + 'registered@g.us': { + name: 'Test Group', + folder: 'test-group', + trigger: '@Andy', + added_at: '2024-01-01T00:00:00.000Z', + }, + })), + ...overrides, + }; +} + +function triggerConnection(state: string, extra?: Record) { + fakeSocket._ev.emit('connection.update', { connection: state, ...extra }); +} + +function triggerDisconnect(statusCode: number) { + fakeSocket._ev.emit('connection.update', { + connection: 'close', + lastDisconnect: { + error: { output: { statusCode } }, + }, + }); +} + +async function triggerMessages(messages: unknown[]) { + fakeSocket._ev.emit('messages.upsert', { messages }); + // Flush microtasks so the async messages.upsert handler completes + await new Promise((r) => setTimeout(r, 0)); +} + +// --- Tests --- + +describe('WhatsAppChannel', () => { + beforeEach(() => { + fakeSocket = createFakeSocket(); + vi.mocked(getLastGroupSync).mockReturnValue(null); + }); + + afterEach(() => { + vi.restoreAllMocks(); + }); + + /** + * Helper: start connect, flush microtasks so event handlers are registered, + * then trigger the connection open event. Returns the resolved promise. + */ + async function connectChannel(channel: WhatsAppChannel): Promise { + const p = channel.connect(); + // Flush microtasks so connectInternal completes its await and registers handlers + await new Promise((r) => setTimeout(r, 0)); + triggerConnection('open'); + return p; + } + + // --- Connection lifecycle --- + + describe('connection lifecycle', () => { + it('resolves connect() when connection opens', async () => { + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + expect(channel.isConnected()).toBe(true); + }); + + it('sets up LID to phone mapping on open', async () => { + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + // The channel should have mapped the LID from sock.user + // We can verify by sending a message from a LID JID + // and checking the translated JID in the callback + }); + + it('flushes outgoing queue on reconnect', async () => { + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + // Disconnect + (channel as any).connected = false; + + // Queue a message while disconnected + await channel.sendMessage('test@g.us', 'Queued message'); + expect(fakeSocket.sendMessage).not.toHaveBeenCalled(); + + // Reconnect + (channel as any).connected = true; + await (channel as any).flushOutgoingQueue(); + + // Group messages get prefixed when flushed + expect(fakeSocket.sendMessage).toHaveBeenCalledWith( + 'test@g.us', + { text: 'Andy: Queued message' }, + ); + }); + + it('disconnects cleanly', async () => { + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + await channel.disconnect(); + expect(channel.isConnected()).toBe(false); + expect(fakeSocket.end).toHaveBeenCalled(); + }); + }); + + // --- QR code and auth --- + + describe('authentication', () => { + it('exits process when QR code is emitted (no auth state)', async () => { + vi.useFakeTimers(); + const mockExit = vi.spyOn(process, 'exit').mockImplementation(() => undefined as never); + + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + // Start connect but don't await (it won't resolve - process exits) + channel.connect().catch(() => {}); + + // Flush microtasks so connectInternal registers handlers + await vi.advanceTimersByTimeAsync(0); + + // Emit QR code event + fakeSocket._ev.emit('connection.update', { qr: 'some-qr-data' }); + + // Advance timer past the 1000ms setTimeout before exit + await vi.advanceTimersByTimeAsync(1500); + + expect(mockExit).toHaveBeenCalledWith(1); + mockExit.mockRestore(); + vi.useRealTimers(); + }); + }); + + // --- Reconnection behavior --- + + describe('reconnection', () => { + it('reconnects on non-loggedOut disconnect', async () => { + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + expect(channel.isConnected()).toBe(true); + + // Disconnect with a non-loggedOut reason (e.g., connectionClosed = 428) + triggerDisconnect(428); + + expect(channel.isConnected()).toBe(false); + // The channel should attempt to reconnect (calls connectInternal again) + }); + + it('exits on loggedOut disconnect', async () => { + const mockExit = vi.spyOn(process, 'exit').mockImplementation(() => undefined as never); + + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + // Disconnect with loggedOut reason (401) + triggerDisconnect(401); + + expect(channel.isConnected()).toBe(false); + expect(mockExit).toHaveBeenCalledWith(0); + mockExit.mockRestore(); + }); + + it('retries reconnection after 5s on failure', async () => { + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + // Disconnect with stream error 515 + triggerDisconnect(515); + + // The channel sets a 5s retry — just verify it doesn't crash + await new Promise((r) => setTimeout(r, 100)); + }); + }); + + // --- Message handling --- + + describe('message handling', () => { + it('delivers message for registered group', async () => { + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + await triggerMessages([ + { + key: { + id: 'msg-1', + remoteJid: 'registered@g.us', + participant: '5551234@s.whatsapp.net', + fromMe: false, + }, + message: { conversation: 'Hello Andy' }, + pushName: 'Alice', + messageTimestamp: Math.floor(Date.now() / 1000), + }, + ]); + + expect(opts.onChatMetadata).toHaveBeenCalledWith( + 'registered@g.us', + expect.any(String), + undefined, + 'whatsapp', + true, + ); + expect(opts.onMessage).toHaveBeenCalledWith( + 'registered@g.us', + expect.objectContaining({ + id: 'msg-1', + content: 'Hello Andy', + sender_name: 'Alice', + is_from_me: false, + }), + ); + }); + + it('only emits metadata for unregistered groups', async () => { + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + await triggerMessages([ + { + key: { + id: 'msg-2', + remoteJid: 'unregistered@g.us', + participant: '5551234@s.whatsapp.net', + fromMe: false, + }, + message: { conversation: 'Hello' }, + pushName: 'Bob', + messageTimestamp: Math.floor(Date.now() / 1000), + }, + ]); + + expect(opts.onChatMetadata).toHaveBeenCalledWith( + 'unregistered@g.us', + expect.any(String), + undefined, + 'whatsapp', + true, + ); + expect(opts.onMessage).not.toHaveBeenCalled(); + }); + + it('ignores status@broadcast messages', async () => { + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + await triggerMessages([ + { + key: { + id: 'msg-3', + remoteJid: 'status@broadcast', + fromMe: false, + }, + message: { conversation: 'Status update' }, + messageTimestamp: Math.floor(Date.now() / 1000), + }, + ]); + + expect(opts.onChatMetadata).not.toHaveBeenCalled(); + expect(opts.onMessage).not.toHaveBeenCalled(); + }); + + it('ignores messages with no content', async () => { + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + await triggerMessages([ + { + key: { + id: 'msg-4', + remoteJid: 'registered@g.us', + fromMe: false, + }, + message: null, + messageTimestamp: Math.floor(Date.now() / 1000), + }, + ]); + + expect(opts.onMessage).not.toHaveBeenCalled(); + }); + + it('extracts text from extendedTextMessage', async () => { + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + await triggerMessages([ + { + key: { + id: 'msg-5', + remoteJid: 'registered@g.us', + participant: '5551234@s.whatsapp.net', + fromMe: false, + }, + message: { + extendedTextMessage: { text: 'A reply message' }, + }, + pushName: 'Charlie', + messageTimestamp: Math.floor(Date.now() / 1000), + }, + ]); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'registered@g.us', + expect.objectContaining({ content: 'A reply message' }), + ); + }); + + it('extracts caption from imageMessage', async () => { + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + await triggerMessages([ + { + key: { + id: 'msg-6', + remoteJid: 'registered@g.us', + participant: '5551234@s.whatsapp.net', + fromMe: false, + }, + message: { + imageMessage: { caption: 'Check this photo', mimetype: 'image/jpeg' }, + }, + pushName: 'Diana', + messageTimestamp: Math.floor(Date.now() / 1000), + }, + ]); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'registered@g.us', + expect.objectContaining({ content: 'Check this photo' }), + ); + }); + + it('extracts caption from videoMessage', async () => { + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + await triggerMessages([ + { + key: { + id: 'msg-7', + remoteJid: 'registered@g.us', + participant: '5551234@s.whatsapp.net', + fromMe: false, + }, + message: { + videoMessage: { caption: 'Watch this', mimetype: 'video/mp4' }, + }, + pushName: 'Eve', + messageTimestamp: Math.floor(Date.now() / 1000), + }, + ]); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'registered@g.us', + expect.objectContaining({ content: 'Watch this' }), + ); + }); + + it('transcribes voice messages', async () => { + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + await triggerMessages([ + { + key: { + id: 'msg-8', + remoteJid: 'registered@g.us', + participant: '5551234@s.whatsapp.net', + fromMe: false, + }, + message: { + audioMessage: { mimetype: 'audio/ogg; codecs=opus', ptt: true }, + }, + pushName: 'Frank', + messageTimestamp: Math.floor(Date.now() / 1000), + }, + ]); + + expect(transcribeAudioMessage).toHaveBeenCalled(); + expect(opts.onMessage).toHaveBeenCalledTimes(1); + expect(opts.onMessage).toHaveBeenCalledWith( + 'registered@g.us', + expect.objectContaining({ content: '[Voice: Hello this is a voice message]' }), + ); + }); + + it('falls back when transcription returns null', async () => { + vi.mocked(transcribeAudioMessage).mockResolvedValueOnce(null); + + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + await triggerMessages([ + { + key: { + id: 'msg-8b', + remoteJid: 'registered@g.us', + participant: '5551234@s.whatsapp.net', + fromMe: false, + }, + message: { + audioMessage: { mimetype: 'audio/ogg; codecs=opus', ptt: true }, + }, + pushName: 'Frank', + messageTimestamp: Math.floor(Date.now() / 1000), + }, + ]); + + expect(opts.onMessage).toHaveBeenCalledTimes(1); + expect(opts.onMessage).toHaveBeenCalledWith( + 'registered@g.us', + expect.objectContaining({ content: '[Voice Message - transcription unavailable]' }), + ); + }); + + it('falls back when transcription throws', async () => { + vi.mocked(transcribeAudioMessage).mockRejectedValueOnce(new Error('API error')); + + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + await triggerMessages([ + { + key: { + id: 'msg-8c', + remoteJid: 'registered@g.us', + participant: '5551234@s.whatsapp.net', + fromMe: false, + }, + message: { + audioMessage: { mimetype: 'audio/ogg; codecs=opus', ptt: true }, + }, + pushName: 'Frank', + messageTimestamp: Math.floor(Date.now() / 1000), + }, + ]); + + expect(opts.onMessage).toHaveBeenCalledTimes(1); + expect(opts.onMessage).toHaveBeenCalledWith( + 'registered@g.us', + expect.objectContaining({ content: '[Voice Message - transcription failed]' }), + ); + }); + + it('uses sender JID when pushName is absent', async () => { + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + await triggerMessages([ + { + key: { + id: 'msg-9', + remoteJid: 'registered@g.us', + participant: '5551234@s.whatsapp.net', + fromMe: false, + }, + message: { conversation: 'No push name' }, + // pushName is undefined + messageTimestamp: Math.floor(Date.now() / 1000), + }, + ]); + + expect(opts.onMessage).toHaveBeenCalledWith( + 'registered@g.us', + expect.objectContaining({ sender_name: '5551234' }), + ); + }); + }); + + // --- LID ↔ JID translation --- + + describe('LID to JID translation', () => { + it('translates known LID to phone JID', async () => { + const opts = createTestOpts({ + registeredGroups: vi.fn(() => ({ + '1234567890@s.whatsapp.net': { + name: 'Self Chat', + folder: 'self-chat', + trigger: '@Andy', + added_at: '2024-01-01T00:00:00.000Z', + }, + })), + }); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + // The socket has lid '9876543210:1@lid' → phone '1234567890@s.whatsapp.net' + // Send a message from the LID + await triggerMessages([ + { + key: { + id: 'msg-lid', + remoteJid: '9876543210@lid', + fromMe: false, + }, + message: { conversation: 'From LID' }, + pushName: 'Self', + messageTimestamp: Math.floor(Date.now() / 1000), + }, + ]); + + // Should be translated to phone JID + expect(opts.onChatMetadata).toHaveBeenCalledWith( + '1234567890@s.whatsapp.net', + expect.any(String), + undefined, + 'whatsapp', + false, + ); + }); + + it('passes through non-LID JIDs unchanged', async () => { + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + await triggerMessages([ + { + key: { + id: 'msg-normal', + remoteJid: 'registered@g.us', + participant: '5551234@s.whatsapp.net', + fromMe: false, + }, + message: { conversation: 'Normal JID' }, + pushName: 'Grace', + messageTimestamp: Math.floor(Date.now() / 1000), + }, + ]); + + expect(opts.onChatMetadata).toHaveBeenCalledWith( + 'registered@g.us', + expect.any(String), + undefined, + 'whatsapp', + true, + ); + }); + + it('passes through unknown LID JIDs unchanged', async () => { + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + await triggerMessages([ + { + key: { + id: 'msg-unknown-lid', + remoteJid: '0000000000@lid', + fromMe: false, + }, + message: { conversation: 'Unknown LID' }, + pushName: 'Unknown', + messageTimestamp: Math.floor(Date.now() / 1000), + }, + ]); + + // Unknown LID passes through unchanged + expect(opts.onChatMetadata).toHaveBeenCalledWith( + '0000000000@lid', + expect.any(String), + undefined, + 'whatsapp', + false, + ); + }); + }); + + // --- Outgoing message queue --- + + describe('outgoing message queue', () => { + it('sends message directly when connected', async () => { + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + await channel.sendMessage('test@g.us', 'Hello'); + // Group messages get prefixed with assistant name + expect(fakeSocket.sendMessage).toHaveBeenCalledWith('test@g.us', { text: 'Andy: Hello' }); + }); + + it('prefixes direct chat messages on shared number', async () => { + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + await channel.sendMessage('123@s.whatsapp.net', 'Hello'); + // Shared number: DMs also get prefixed (needed for self-chat distinction) + expect(fakeSocket.sendMessage).toHaveBeenCalledWith('123@s.whatsapp.net', { text: 'Andy: Hello' }); + }); + + it('queues message when disconnected', async () => { + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + // Don't connect — channel starts disconnected + await channel.sendMessage('test@g.us', 'Queued'); + expect(fakeSocket.sendMessage).not.toHaveBeenCalled(); + }); + + it('queues message on send failure', async () => { + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + // Make sendMessage fail + fakeSocket.sendMessage.mockRejectedValueOnce(new Error('Network error')); + + await channel.sendMessage('test@g.us', 'Will fail'); + + // Should not throw, message queued for retry + // The queue should have the message + }); + + it('flushes multiple queued messages in order', async () => { + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + // Queue messages while disconnected + await channel.sendMessage('test@g.us', 'First'); + await channel.sendMessage('test@g.us', 'Second'); + await channel.sendMessage('test@g.us', 'Third'); + + // Connect — flush happens automatically on open + await connectChannel(channel); + + // Give the async flush time to complete + await new Promise((r) => setTimeout(r, 50)); + + expect(fakeSocket.sendMessage).toHaveBeenCalledTimes(3); + // Group messages get prefixed + expect(fakeSocket.sendMessage).toHaveBeenNthCalledWith(1, 'test@g.us', { text: 'Andy: First' }); + expect(fakeSocket.sendMessage).toHaveBeenNthCalledWith(2, 'test@g.us', { text: 'Andy: Second' }); + expect(fakeSocket.sendMessage).toHaveBeenNthCalledWith(3, 'test@g.us', { text: 'Andy: Third' }); + }); + }); + + // --- Group metadata sync --- + + describe('group metadata sync', () => { + it('syncs group metadata on first connection', async () => { + fakeSocket.groupFetchAllParticipating.mockResolvedValue({ + 'group1@g.us': { subject: 'Group One' }, + 'group2@g.us': { subject: 'Group Two' }, + }); + + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + // Wait for async sync to complete + await new Promise((r) => setTimeout(r, 50)); + + expect(fakeSocket.groupFetchAllParticipating).toHaveBeenCalled(); + expect(updateChatName).toHaveBeenCalledWith('group1@g.us', 'Group One'); + expect(updateChatName).toHaveBeenCalledWith('group2@g.us', 'Group Two'); + expect(setLastGroupSync).toHaveBeenCalled(); + }); + + it('skips sync when synced recently', async () => { + // Last sync was 1 hour ago (within 24h threshold) + vi.mocked(getLastGroupSync).mockReturnValue( + new Date(Date.now() - 60 * 60 * 1000).toISOString(), + ); + + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + await new Promise((r) => setTimeout(r, 50)); + + expect(fakeSocket.groupFetchAllParticipating).not.toHaveBeenCalled(); + }); + + it('forces sync regardless of cache', async () => { + vi.mocked(getLastGroupSync).mockReturnValue( + new Date(Date.now() - 60 * 60 * 1000).toISOString(), + ); + + fakeSocket.groupFetchAllParticipating.mockResolvedValue({ + 'group@g.us': { subject: 'Forced Group' }, + }); + + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + await channel.syncGroupMetadata(true); + + expect(fakeSocket.groupFetchAllParticipating).toHaveBeenCalled(); + expect(updateChatName).toHaveBeenCalledWith('group@g.us', 'Forced Group'); + }); + + it('handles group sync failure gracefully', async () => { + fakeSocket.groupFetchAllParticipating.mockRejectedValue( + new Error('Network timeout'), + ); + + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + // Should not throw + await expect(channel.syncGroupMetadata(true)).resolves.toBeUndefined(); + }); + + it('skips groups with no subject', async () => { + fakeSocket.groupFetchAllParticipating.mockResolvedValue({ + 'group1@g.us': { subject: 'Has Subject' }, + 'group2@g.us': { subject: '' }, + 'group3@g.us': {}, + }); + + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + // Clear any calls from the automatic sync on connect + vi.mocked(updateChatName).mockClear(); + + await channel.syncGroupMetadata(true); + + expect(updateChatName).toHaveBeenCalledTimes(1); + expect(updateChatName).toHaveBeenCalledWith('group1@g.us', 'Has Subject'); + }); + }); + + // --- JID ownership --- + + describe('ownsJid', () => { + it('owns @g.us JIDs (WhatsApp groups)', () => { + const channel = new WhatsAppChannel(createTestOpts()); + expect(channel.ownsJid('12345@g.us')).toBe(true); + }); + + it('owns @s.whatsapp.net JIDs (WhatsApp DMs)', () => { + const channel = new WhatsAppChannel(createTestOpts()); + expect(channel.ownsJid('12345@s.whatsapp.net')).toBe(true); + }); + + it('does not own Telegram JIDs', () => { + const channel = new WhatsAppChannel(createTestOpts()); + expect(channel.ownsJid('tg:12345')).toBe(false); + }); + + it('does not own unknown JID formats', () => { + const channel = new WhatsAppChannel(createTestOpts()); + expect(channel.ownsJid('random-string')).toBe(false); + }); + }); + + // --- Typing indicator --- + + describe('setTyping', () => { + it('sends composing presence when typing', async () => { + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + await channel.setTyping('test@g.us', true); + expect(fakeSocket.sendPresenceUpdate).toHaveBeenCalledWith('composing', 'test@g.us'); + }); + + it('sends paused presence when stopping', async () => { + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + await channel.setTyping('test@g.us', false); + expect(fakeSocket.sendPresenceUpdate).toHaveBeenCalledWith('paused', 'test@g.us'); + }); + + it('handles typing indicator failure gracefully', async () => { + const opts = createTestOpts(); + const channel = new WhatsAppChannel(opts); + + await connectChannel(channel); + + fakeSocket.sendPresenceUpdate.mockRejectedValueOnce(new Error('Failed')); + + // Should not throw + await expect(channel.setTyping('test@g.us', true)).resolves.toBeUndefined(); + }); + }); + + // --- Channel properties --- + + describe('channel properties', () => { + it('has name "whatsapp"', () => { + const channel = new WhatsAppChannel(createTestOpts()); + expect(channel.name).toBe('whatsapp'); + }); + + it('does not expose prefixAssistantName (prefix handled internally)', () => { + const channel = new WhatsAppChannel(createTestOpts()); + expect('prefixAssistantName' in channel).toBe(false); + }); + }); +}); diff --git a/.agent/skills/add-voice-transcription/modify/src/channels/whatsapp.test.ts.intent.md b/.agent/skills/add-voice-transcription/modify/src/channels/whatsapp.test.ts.intent.md new file mode 100644 index 0000000..1607ddb --- /dev/null +++ b/.agent/skills/add-voice-transcription/modify/src/channels/whatsapp.test.ts.intent.md @@ -0,0 +1,30 @@ +# Intent: src/channels/whatsapp.test.ts modifications + +## What changed + +Added mock for the transcription module and 3 new test cases for voice message handling. + +## Key sections + +### Mocks (top of file) + +- Added: `vi.mock('../transcription.js', ...)` with `isVoiceMessage` and `transcribeAudioMessage` mocks +- Added: `import { transcribeAudioMessage } from '../transcription.js'` for test assertions + +### Test cases (inside "message handling" describe block) + +- Changed: "handles message with no extractable text (e.g. voice note without caption)" → "transcribes voice messages" + - Now expects `[Voice: Hello this is a voice message]` instead of empty content +- Added: "falls back when transcription returns null" — expects `[Voice Message - transcription unavailable]` +- Added: "falls back when transcription throws" — expects `[Voice Message - transcription failed]` + +## Invariants (must-keep) + +- All existing test cases for text, extendedTextMessage, imageMessage, videoMessage unchanged +- All connection lifecycle tests unchanged +- All LID translation tests unchanged +- All outgoing queue tests unchanged +- All group metadata sync tests unchanged +- All ownsJid and setTyping tests unchanged +- All existing mocks (config, logger, db, fs, child_process, baileys) unchanged +- Test helpers (createTestOpts, triggerConnection, triggerDisconnect, triggerMessages, connectChannel) unchanged diff --git a/.agent/skills/add-voice-transcription/modify/src/channels/whatsapp.ts b/.agent/skills/add-voice-transcription/modify/src/channels/whatsapp.ts new file mode 100644 index 0000000..9fc832d --- /dev/null +++ b/.agent/skills/add-voice-transcription/modify/src/channels/whatsapp.ts @@ -0,0 +1,400 @@ +import { exec } from 'child_process'; +import fs from 'fs'; +import path from 'path'; + +import makeWASocket, { + Browsers, + DisconnectReason, + WASocket, + fetchLatestWaWebVersion, + makeCacheableSignalKeyStore, + useMultiFileAuthState, +} from '@whiskeysockets/baileys'; + +import { + ASSISTANT_HAS_OWN_NUMBER, + ASSISTANT_NAME, + STORE_DIR, +} from '../config.js'; +import { getLastGroupSync, setLastGroupSync, updateChatName } from '../db.js'; +import { logger } from '../logger.js'; +import { isVoiceMessage, transcribeAudioMessage } from '../transcription.js'; +import { + Channel, + OnInboundMessage, + OnChatMetadata, + RegisteredGroup, +} from '../types.js'; + +const GROUP_SYNC_INTERVAL_MS = 24 * 60 * 60 * 1000; // 24 hours + +export interface WhatsAppChannelOpts { + onMessage: OnInboundMessage; + onChatMetadata: OnChatMetadata; + registeredGroups: () => Record; +} + +export class WhatsAppChannel implements Channel { + name = 'whatsapp'; + + private sock!: WASocket; + private connected = false; + private lidToPhoneMap: Record = {}; + private outgoingQueue: Array<{ jid: string; text: string }> = []; + private flushing = false; + private groupSyncTimerStarted = false; + + private opts: WhatsAppChannelOpts; + + constructor(opts: WhatsAppChannelOpts) { + this.opts = opts; + } + + async connect(): Promise { + return new Promise((resolve, reject) => { + this.connectInternal(resolve).catch(reject); + }); + } + + private async connectInternal(onFirstOpen?: () => void): Promise { + const authDir = path.join(STORE_DIR, 'auth'); + fs.mkdirSync(authDir, { recursive: true }); + + const { state, saveCreds } = await useMultiFileAuthState(authDir); + + const { version } = await fetchLatestWaWebVersion({}).catch((err) => { + logger.warn( + { err }, + 'Failed to fetch latest WA Web version, using default', + ); + return { version: undefined }; + }); + this.sock = makeWASocket({ + version, + auth: { + creds: state.creds, + keys: makeCacheableSignalKeyStore(state.keys, logger), + }, + printQRInTerminal: false, + logger, + browser: Browsers.macOS('Chrome'), + }); + + this.sock.ev.on('connection.update', (update) => { + const { connection, lastDisconnect, qr } = update; + + if (qr) { + const msg = + 'WhatsApp authentication required. Run /setup to re-authenticate.'; + logger.error(msg); + exec( + `osascript -e 'display notification "${msg}" with title "NanoClaw" sound name "Basso"'`, + ); + setTimeout(() => process.exit(1), 1000); + } + + if (connection === 'close') { + this.connected = false; + const reason = ( + lastDisconnect?.error as { output?: { statusCode?: number } } + )?.output?.statusCode; + const shouldReconnect = reason !== DisconnectReason.loggedOut; + logger.info( + { + reason, + shouldReconnect, + queuedMessages: this.outgoingQueue.length, + }, + 'Connection closed', + ); + + if (shouldReconnect) { + logger.info('Reconnecting...'); + this.connectInternal().catch((err) => { + logger.error({ err }, 'Failed to reconnect, retrying in 5s'); + setTimeout(() => { + this.connectInternal().catch((err2) => { + logger.error({ err: err2 }, 'Reconnection retry failed'); + }); + }, 5000); + }); + } else { + logger.info('Logged out. Run /setup to re-authenticate.'); + process.exit(0); + } + } else if (connection === 'open') { + this.connected = true; + logger.info('Connected to WhatsApp'); + + // Announce availability so WhatsApp relays subsequent presence updates (typing indicators) + this.sock.sendPresenceUpdate('available').catch((err) => { + logger.warn({ err }, 'Failed to send presence update'); + }); + + // Build LID to phone mapping from auth state for self-chat translation + if (this.sock.user) { + const phoneUser = this.sock.user.id.split(':')[0]; + const lidUser = this.sock.user.lid?.split(':')[0]; + if (lidUser && phoneUser) { + this.lidToPhoneMap[lidUser] = `${phoneUser}@s.whatsapp.net`; + logger.debug({ lidUser, phoneUser }, 'LID to phone mapping set'); + } + } + + // Flush any messages queued while disconnected + this.flushOutgoingQueue().catch((err) => + logger.error({ err }, 'Failed to flush outgoing queue'), + ); + + // Sync group metadata on startup (respects 24h cache) + this.syncGroupMetadata().catch((err) => + logger.error({ err }, 'Initial group sync failed'), + ); + // Set up daily sync timer (only once) + if (!this.groupSyncTimerStarted) { + this.groupSyncTimerStarted = true; + setInterval(() => { + this.syncGroupMetadata().catch((err) => + logger.error({ err }, 'Periodic group sync failed'), + ); + }, GROUP_SYNC_INTERVAL_MS); + } + + // Signal first connection to caller + if (onFirstOpen) { + onFirstOpen(); + onFirstOpen = undefined; + } + } + }); + + this.sock.ev.on('creds.update', saveCreds); + + this.sock.ev.on('messages.upsert', async ({ messages }) => { + for (const msg of messages) { + if (!msg.message) continue; + const rawJid = msg.key.remoteJid; + if (!rawJid || rawJid === 'status@broadcast') continue; + + // Translate LID JID to phone JID if applicable + const chatJid = await this.translateJid(rawJid); + + const timestamp = new Date( + Number(msg.messageTimestamp) * 1000, + ).toISOString(); + + // Always notify about chat metadata for group discovery + const isGroup = chatJid.endsWith('@g.us'); + this.opts.onChatMetadata( + chatJid, + timestamp, + undefined, + 'whatsapp', + isGroup, + ); + + // Only deliver full message for registered groups + const groups = this.opts.registeredGroups(); + if (groups[chatJid]) { + const content = + msg.message?.conversation || + msg.message?.extendedTextMessage?.text || + msg.message?.imageMessage?.caption || + msg.message?.videoMessage?.caption || + ''; + + // Skip protocol messages with no text content (encryption keys, read receipts, etc.) + // but allow voice messages through for transcription + if (!content && !isVoiceMessage(msg)) continue; + + const sender = msg.key.participant || msg.key.remoteJid || ''; + const senderName = msg.pushName || sender.split('@')[0]; + + const fromMe = msg.key.fromMe || false; + // Detect bot messages: with own number, fromMe is reliable + // since only the bot sends from that number. + // With shared number, bot messages carry the assistant name prefix + // (even in DMs/self-chat) so we check for that. + const isBotMessage = ASSISTANT_HAS_OWN_NUMBER + ? fromMe + : content.startsWith(`${ASSISTANT_NAME}:`); + + // Transcribe voice messages before storing + let finalContent = content; + if (isVoiceMessage(msg)) { + try { + const transcript = await transcribeAudioMessage(msg, this.sock); + if (transcript) { + finalContent = `[Voice: ${transcript}]`; + logger.info( + { chatJid, length: transcript.length }, + 'Transcribed voice message', + ); + } else { + finalContent = '[Voice Message - transcription unavailable]'; + } + } catch (err) { + logger.error({ err }, 'Voice transcription error'); + finalContent = '[Voice Message - transcription failed]'; + } + } + + this.opts.onMessage(chatJid, { + id: msg.key.id || '', + chat_jid: chatJid, + sender, + sender_name: senderName, + content: finalContent, + timestamp, + is_from_me: fromMe, + is_bot_message: isBotMessage, + }); + } + } + }); + } + + async sendMessage(jid: string, text: string): Promise { + // Prefix bot messages with assistant name so users know who's speaking. + // On a shared number, prefix is also needed in DMs (including self-chat) + // to distinguish bot output from user messages. + // Skip only when the assistant has its own dedicated phone number. + const prefixed = ASSISTANT_HAS_OWN_NUMBER + ? text + : `${ASSISTANT_NAME}: ${text}`; + + if (!this.connected) { + this.outgoingQueue.push({ jid, text: prefixed }); + logger.info( + { jid, length: prefixed.length, queueSize: this.outgoingQueue.length }, + 'WA disconnected, message queued', + ); + return; + } + try { + await this.sock.sendMessage(jid, { text: prefixed }); + logger.info({ jid, length: prefixed.length }, 'Message sent'); + } catch (err) { + // If send fails, queue it for retry on reconnect + this.outgoingQueue.push({ jid, text: prefixed }); + logger.warn( + { jid, err, queueSize: this.outgoingQueue.length }, + 'Failed to send, message queued', + ); + } + } + + isConnected(): boolean { + return this.connected; + } + + ownsJid(jid: string): boolean { + return jid.endsWith('@g.us') || jid.endsWith('@s.whatsapp.net'); + } + + async disconnect(): Promise { + this.connected = false; + this.sock?.end(undefined); + } + + async setTyping(jid: string, isTyping: boolean): Promise { + try { + const status = isTyping ? 'composing' : 'paused'; + logger.debug({ jid, status }, 'Sending presence update'); + await this.sock.sendPresenceUpdate(status, jid); + } catch (err) { + logger.debug({ jid, err }, 'Failed to update typing status'); + } + } + + /** + * Sync group metadata from WhatsApp. + * Fetches all participating groups and stores their names in the database. + * Called on startup, daily, and on-demand via IPC. + */ + async syncGroupMetadata(force = false): Promise { + if (!force) { + const lastSync = getLastGroupSync(); + if (lastSync) { + const lastSyncTime = new Date(lastSync).getTime(); + if (Date.now() - lastSyncTime < GROUP_SYNC_INTERVAL_MS) { + logger.debug({ lastSync }, 'Skipping group sync - synced recently'); + return; + } + } + } + + try { + logger.info('Syncing group metadata from WhatsApp...'); + const groups = await this.sock.groupFetchAllParticipating(); + + let count = 0; + for (const [jid, metadata] of Object.entries(groups)) { + if (metadata.subject) { + updateChatName(jid, metadata.subject); + count++; + } + } + + setLastGroupSync(); + logger.info({ count }, 'Group metadata synced'); + } catch (err) { + logger.error({ err }, 'Failed to sync group metadata'); + } + } + + private async translateJid(jid: string): Promise { + if (!jid.endsWith('@lid')) return jid; + const lidUser = jid.split('@')[0].split(':')[0]; + + // Check local cache first + const cached = this.lidToPhoneMap[lidUser]; + if (cached) { + logger.debug( + { lidJid: jid, phoneJid: cached }, + 'Translated LID to phone JID (cached)', + ); + return cached; + } + + // Query Baileys' signal repository for the mapping + try { + const pn = await this.sock.signalRepository?.lidMapping?.getPNForLID(jid); + if (pn) { + const phoneJid = `${pn.split('@')[0].split(':')[0]}@s.whatsapp.net`; + this.lidToPhoneMap[lidUser] = phoneJid; + logger.info( + { lidJid: jid, phoneJid }, + 'Translated LID to phone JID (signalRepository)', + ); + return phoneJid; + } + } catch (err) { + logger.debug({ err, jid }, 'Failed to resolve LID via signalRepository'); + } + + return jid; + } + + private async flushOutgoingQueue(): Promise { + if (this.flushing || this.outgoingQueue.length === 0) return; + this.flushing = true; + try { + logger.info( + { count: this.outgoingQueue.length }, + 'Flushing outgoing message queue', + ); + while (this.outgoingQueue.length > 0) { + const item = this.outgoingQueue.shift()!; + // Send directly — queued items are already prefixed by sendMessage + await this.sock.sendMessage(item.jid, { text: item.text }); + logger.info( + { jid: item.jid, length: item.text.length }, + 'Queued message sent', + ); + } + } finally { + this.flushing = false; + } + } +} diff --git a/.agent/skills/add-voice-transcription/modify/src/channels/whatsapp.ts.intent.md b/.agent/skills/add-voice-transcription/modify/src/channels/whatsapp.ts.intent.md new file mode 100644 index 0000000..67a5719 --- /dev/null +++ b/.agent/skills/add-voice-transcription/modify/src/channels/whatsapp.ts.intent.md @@ -0,0 +1,31 @@ +# Intent: src/channels/whatsapp.ts modifications + +## What changed + +Added voice message transcription support. When a WhatsApp voice note (PTT audio) arrives, it is downloaded and transcribed via OpenAI Whisper before being stored as message content. + +## Key sections + +### Imports (top of file) + +- Added: `isVoiceMessage`, `transcribeAudioMessage` from `../transcription.js` + +### messages.upsert handler (inside connectInternal) + +- Added: `let finalContent = content` variable to allow voice transcription to override text content +- Added: `isVoiceMessage(msg)` check after content extraction +- Added: try/catch block calling `transcribeAudioMessage(msg, this.sock)` + - Success: `finalContent = '[Voice: ]'` + - Null result: `finalContent = '[Voice Message - transcription unavailable]'` + - Error: `finalContent = '[Voice Message - transcription failed]'` +- Changed: `this.opts.onMessage()` call uses `finalContent` instead of `content` + +## Invariants (must-keep) + +- All existing message handling (conversation, extendedTextMessage, imageMessage, videoMessage) unchanged +- Connection lifecycle (connect, reconnect, disconnect) unchanged +- LID translation logic unchanged +- Outgoing message queue unchanged +- Group metadata sync unchanged +- sendMessage prefix logic unchanged +- setTyping, ownsJid, isConnected — all unchanged diff --git a/.agent/skills/add-voice-transcription/tests/voice-transcription.test.ts b/.agent/skills/add-voice-transcription/tests/voice-transcription.test.ts new file mode 100644 index 0000000..76ebd0d --- /dev/null +++ b/.agent/skills/add-voice-transcription/tests/voice-transcription.test.ts @@ -0,0 +1,123 @@ +import { describe, expect, it } from 'vitest'; +import fs from 'fs'; +import path from 'path'; + +describe('voice-transcription skill package', () => { + const skillDir = path.resolve(__dirname, '..'); + + it('has a valid manifest', () => { + const manifestPath = path.join(skillDir, 'manifest.yaml'); + expect(fs.existsSync(manifestPath)).toBe(true); + + const content = fs.readFileSync(manifestPath, 'utf-8'); + expect(content).toContain('skill: voice-transcription'); + expect(content).toContain('version: 1.0.0'); + expect(content).toContain('openai'); + expect(content).toContain('OPENAI_API_KEY'); + }); + + it('has all files declared in adds', () => { + const transcriptionFile = path.join(skillDir, 'add', 'src', 'transcription.ts'); + expect(fs.existsSync(transcriptionFile)).toBe(true); + + const content = fs.readFileSync(transcriptionFile, 'utf-8'); + expect(content).toContain('transcribeAudioMessage'); + expect(content).toContain('isVoiceMessage'); + expect(content).toContain('transcribeWithOpenAI'); + expect(content).toContain('downloadMediaMessage'); + expect(content).toContain('readEnvFile'); + }); + + it('has all files declared in modifies', () => { + const whatsappFile = path.join(skillDir, 'modify', 'src', 'channels', 'whatsapp.ts'); + const whatsappTestFile = path.join(skillDir, 'modify', 'src', 'channels', 'whatsapp.test.ts'); + + expect(fs.existsSync(whatsappFile)).toBe(true); + expect(fs.existsSync(whatsappTestFile)).toBe(true); + }); + + it('has intent files for modified files', () => { + expect(fs.existsSync(path.join(skillDir, 'modify', 'src', 'channels', 'whatsapp.ts.intent.md'))).toBe(true); + expect(fs.existsSync(path.join(skillDir, 'modify', 'src', 'channels', 'whatsapp.test.ts.intent.md'))).toBe(true); + }); + + it('modified whatsapp.ts preserves core structure', () => { + const content = fs.readFileSync( + path.join(skillDir, 'modify', 'src', 'channels', 'whatsapp.ts'), + 'utf-8', + ); + + // Core class and methods preserved + expect(content).toContain('class WhatsAppChannel'); + expect(content).toContain('implements Channel'); + expect(content).toContain('async connect()'); + expect(content).toContain('async sendMessage('); + expect(content).toContain('isConnected()'); + expect(content).toContain('ownsJid('); + expect(content).toContain('async disconnect()'); + expect(content).toContain('async setTyping('); + expect(content).toContain('async syncGroupMetadata('); + expect(content).toContain('private async translateJid('); + expect(content).toContain('private async flushOutgoingQueue('); + + // Core imports preserved + expect(content).toContain('ASSISTANT_HAS_OWN_NUMBER'); + expect(content).toContain('ASSISTANT_NAME'); + expect(content).toContain('STORE_DIR'); + }); + + it('modified whatsapp.ts includes transcription integration', () => { + const content = fs.readFileSync( + path.join(skillDir, 'modify', 'src', 'channels', 'whatsapp.ts'), + 'utf-8', + ); + + // Transcription imports + expect(content).toContain("import { isVoiceMessage, transcribeAudioMessage } from '../transcription.js'"); + + // Voice message handling + expect(content).toContain('isVoiceMessage(msg)'); + expect(content).toContain('transcribeAudioMessage(msg, this.sock)'); + expect(content).toContain('finalContent'); + expect(content).toContain('[Voice:'); + expect(content).toContain('[Voice Message - transcription unavailable]'); + expect(content).toContain('[Voice Message - transcription failed]'); + }); + + it('modified whatsapp.test.ts includes transcription mock and tests', () => { + const content = fs.readFileSync( + path.join(skillDir, 'modify', 'src', 'channels', 'whatsapp.test.ts'), + 'utf-8', + ); + + // Transcription mock + expect(content).toContain("vi.mock('../transcription.js'"); + expect(content).toContain('isVoiceMessage'); + expect(content).toContain('transcribeAudioMessage'); + + // Voice transcription test cases + expect(content).toContain('transcribes voice messages'); + expect(content).toContain('falls back when transcription returns null'); + expect(content).toContain('falls back when transcription throws'); + expect(content).toContain('[Voice: Hello this is a voice message]'); + }); + + it('modified whatsapp.test.ts preserves all existing test sections', () => { + const content = fs.readFileSync( + path.join(skillDir, 'modify', 'src', 'channels', 'whatsapp.test.ts'), + 'utf-8', + ); + + // All existing test describe blocks preserved + expect(content).toContain("describe('connection lifecycle'"); + expect(content).toContain("describe('authentication'"); + expect(content).toContain("describe('reconnection'"); + expect(content).toContain("describe('message handling'"); + expect(content).toContain("describe('LID to JID translation'"); + expect(content).toContain("describe('outgoing message queue'"); + expect(content).toContain("describe('group metadata sync'"); + expect(content).toContain("describe('ownsJid'"); + expect(content).toContain("describe('setTyping'"); + expect(content).toContain("describe('channel properties'"); + }); +}); diff --git a/.agent/skills/agent-setup/SKILL.md b/.agent/skills/agent-setup/SKILL.md new file mode 100644 index 0000000..726ba58 --- /dev/null +++ b/.agent/skills/agent-setup/SKILL.md @@ -0,0 +1,88 @@ +--- +name: agent-setup +description: Set up the current Clawdie runtime on FreeBSD using the host orchestrator plus db/git/cms/worker jails. Use when bringing up a fresh FreeBSD host or aligning an install with current main. +--- + +# Agent FreeBSD Setup + +Use this skill for the active FreeBSD deployment model on current `main`. + +## Current architecture + +Clawdie no longer uses a dedicated operator jail. + +Current layout: + +- host: orchestrator, service management, setup flow +- `{agent}-db` jail: PostgreSQL for Agent System Skills and User/Agent Memory +- `{agent}-git` jail: local bare repositories +- `{agent}-cms` jail: nginx + Astro (Strapi is optional when used) +- `{agent}-worker` jail: baseline sandbox at `${AGENT_SUBNET_BASE}.101` + +## Canonical addresses + +- `warden0`: `${AGENT_SUBNET_BASE}.1/24` +- `.2`: reserved compatibility slot (legacy controlplane naming) +- `{agent}-db`: `${AGENT_SUBNET_BASE}.3` +- `{agent}-cms`: `${AGENT_SUBNET_BASE}.4` +- `{agent}-ollama` or `{agent}-llamacpp`: `${AGENT_SUBNET_BASE}.5` (optional) +- `{agent}-git`: `${AGENT_SUBNET_BASE}.6` (optional) +- `{agent}-worker`: `${AGENT_SUBNET_BASE}.101` +- browser/gui: `${AGENT_SUBNET_BASE}.150` (reserved) + +Notes: + +- Service jails should be **agent-prefixed** to avoid name collisions when a + second agent is added later. +- If you intentionally want to share a local LLM jail across agents, keep a + single `ollama`/`llamacpp` jail and point multiple agents at it explicitly. +- For the “add a second agent later” procedure, follow `docs/public/reference/multi-agent.md`. + +## Canonical setup flow + +```sh +./setup.sh +just install +``` + +## Host prerequisites + +```sh +freebsd-version +zpool status zroot +sysctl net.inet.ip.forwarding +sysctl kern.racct.enable +``` + +Install the baseline: + +```sh +sudo pkg install -y bash git bsddialog bastille node24 npm tmux python311 uv ripgrep fd-find rsync postgresql18-client py311-pillow dejavu rust sanoid +npm install -g @earendil-works/pi-coding-agent +``` + +## tmux baseline for `pi` + +```sh +cat >> ~/.tmux.conf << 'EOF' +set -g extended-keys on +set -g extended-keys-format csi-u +EOF +tmux source-file ~/.tmux.conf 2>/dev/null || true +``` + +## ZFS and networking + +Delegate to: + +- `/warden-zfs` for dataset layout and snapshots +- `/bastille-network` for `warden0` +- `/warden-pf` for NAT and port forwards + +## Safe defaults + +- keep SSH on the host +- keep Ansible aimed at the host, not inside jails +- keep `db` mandatory for split-brain +- keep `git` and `cms` as first-class service jails +- keep `.home.arpa` for internal naming and `.invalid` as the public placeholder diff --git a/.agent/skills/ansible-freebsd/SKILL.md b/.agent/skills/ansible-freebsd/SKILL.md new file mode 100644 index 0000000..d63853f --- /dev/null +++ b/.agent/skills/ansible-freebsd/SKILL.md @@ -0,0 +1,306 @@ +--- +name: ansible-freebsd +description: Use Ansible for repeatable FreeBSD host and Bastille jail operations in Clawdie. Use when turning FreeBSD admin work into playbooks, inventories, and roles, including nginx, host networking, git/db/cms jail bootstrap, and Astro/Strapi deployment. +--- + +# Ansible FreeBSD + +Use this skill when a FreeBSD host workflow should become repeatable Ansible +automation instead of ad hoc shell commands. + +## Addressing note + +All IPs are derived from `AGENT_SUBNET_BASE` (set in `.env`). Default for agent 1 +is `10.0.1`. Each additional agent gets its own `/24`: agent 2 → `10.0.2`, etc. + +Fixed layout within each agent subnet: + +- `.1` = warden0 gateway +- `.2` = `{agent}-controlplane` (reserved) +- `.3` = `{agent}-db` (PostgreSQL) +- `.4` = `{agent}-cms` (Astro + Strapi) +- `.5` = `{agent}-ollama` or `{agent}-llamacpp` (optional local LLM) +- `.6` = `{agent}-git` (optional, `FEATURE_GIT=true`) + +Examples below use `10.0.1.x` (agent 1). Use `{{ agent_subnet_base }}.x` in playbooks. + +This skill complements existing host/jail skills: + +- `freebsd-admin` defines host-level intent and safety rules +- `warden-bootstrap` defines Bastille jail bootstrap shape +- `nginx` defines web-serving behavior +- `astro` and `strapi` define the cms application architecture + +This skill turns those operational patterns into Ansible structure. + +## Scope + +This skill covers: + +- Ansible inventory layout for the FreeBSD host +- host playbooks for packages, `sysrc`, `sysctl`, services, and config files +- Bastille jail creation orchestration through explicit commands +- `git` jail bootstrap for local code storage (optional, `FEATURE_GIT=true`) +- `{agent}-cms` jail bootstrap for Astro + Strapi on `${AGENT_SUBNET_BASE}.4` +- nginx deployment handoff for static Astro output + +This skill does not replace: + +- `freebsd-admin` for host policy decisions +- `warden-pf` for firewall design +- `warden-zfs` for dataset design +- `nginx` for site architecture decisions + +## Core rules + +- keep Ansible boring and explicit +- prefer idempotent tasks, but do not force fake abstractions over Bastille or ZFS +- shell out for `bastille`, `zfs`, and `bastille cmd` when that is the clearest path +- keep host tasks and jail tasks clearly separated +- validate after each major phase +- connect Ansible to the FreeBSD host by SSH, not to Bastille jails by default + +## Initial repository shape + +Create infrastructure under: + +```text +infra/ansible/ + inventories/ + production/ + hosts.yaml + group_vars/ + host_vars/ + playbooks/ + host-preflight.yaml + base-host.yaml + host-nginx.yaml + host-pf-baseline.yaml + db-memory-bootstrap.yaml + git-jail-bootstrap.yaml + jail-cms-create.yaml + jail-cms-bootstrap.yaml + cms-strapi-bootstrap.yaml + cms-astro-bootstrap.yaml + cms-deploy.yaml + roles/ + freebsd_base/ + freebsd_network/ + nginx_host/ + bastille_jail/ + cms_jail/ +``` + +Start small. The first useful milestone is not full automation. It is: + +1. host preflight +2. `cms` jail creation +3. `cms` jail package bootstrap +4. optional `git` jail bootstrap for local code storage + +## Workflow + +1. Read `references/install.md` before assuming Ansible is available on the host +2. Read `references/layout.md` +3. Read `references/cms-astro-strapi-plan.md` if the target is the `cms` jail on `${AGENT_SUBNET_BASE}.4` +4. Read `references/host-encrypted-dataset.md` before assuming screenshot encryption can be bootstrapped automatically +5. Reuse existing skill assumptions instead of inventing new architecture +6. Create or update only the smallest playbooks needed for the current step +7. Keep commands auditable and close to existing manual docs + +## Recommended first Ansible targets + +### `host-preflight.yaml` + +Validate: + +- FreeBSD host +- Bastille installed +- `15.0-RELEASE` available +- `warden0` exists with `{{ agent_subnet_base }}.1/24` +- forwarding enabled or clearly missing +- encrypted screenshot dataset exists and is mounted correctly + +Important: + +- `host-preflight.yaml` does not create the encrypted ZFS dataset +- use `references/host-encrypted-dataset.md` for the one-time host bootstrap + +### `base-host.yaml` + +Apply the first repeatable host baseline for current `main`: + +- host package baseline: `bash`, `bsddialog`, `bastille`, `git`, `tmux`, + `python311`, `uv`, `ripgrep`, `fd-find`, `rsync`, `postgresql18-client`, + `node24`, `npm`, `py311-pillow`, `dejavu` +- `gateway_enable="YES"` +- immediate forwarding enablement +- Bastille resolver baseline file +- host validation after change + +This playbook is the default Ansible handoff target for `freebsd-admin`. + +### `jail-cms-create.yaml` + +Create the `cms` jail with the proven pattern: + +```text +bastille create -T -B -g ${AGENT_SUBNET_BASE}.1 ${AGENT_NAME}-cms 15.0-RELEASE {{ agent_subnet_base }}.4/24 warden0 +``` + +Then: + +- set hostname to `cms..home.arpa` +- restart jail +- validate route and reachability to `db` at `${AGENT_SUBNET_BASE}.3` + +### `jail-cms-bootstrap.yaml` + +Inside `cms`: + +- install `bash`, `nginx`, `node24`, `npm`, `git`, `rsync`, `postgresql18-client` +- ensure `clawdie` user exists +- create `/home/clawdie/strapi` +- create `/home/clawdie/clawdie-docs` + +### `db-memory-bootstrap.yaml` + +Bootstrap the `db` jail as the PostgreSQL 18 memory backend: + +- install `postgresql18-server`, `postgresql18-client`, `postgresql18-pgvector`, and `postgresql18-contrib` +- initialize `data` directory if missing +- start PostgreSQL +- create the `clawdie` database +- create and password the split-brain PostgreSQL roles plus `strapi_cms` +- enable `pgcrypto`, `uuid-ossp`, and `vector` +- set `listen_addresses` and `pg_hba.conf` +- validate with `sockstat`, `show listen_addresses`, and `\\dx` + +### `git-jail-bootstrap.yaml` + +Bootstrap a dedicated `git` jail for plain local git storage: + +- create the `git` jail when `FEATURE_GIT=true` +- assign the reserved address from `AGENT_SUBNET_BASE` (`.6` by default, configurable) +- install `git` +- create `/srv/git` +- optionally create `Clawdie-AI.git` +- validate hostname and package presence + +This is intentionally the plain-git first stage: + +- no Gitea +- no public web UI +- no CI + +See [GIT-STORAGE.md](../../../docs/public/operate/git-storage.md) for +the current scope and workflow. The installer may reserve `Local Gitea` as a +future mode, but this playbook currently bootstraps only bare local git +storage. + +### `host-nginx.yaml` + +Apply the basic host nginx baseline: + +- install package +- persist service enablement +- validate config +- reload and check status + +### `host-pf-baseline.yaml` + +Apply the minimal Warden PF include and validation path: + +- write one Warden include +- ensure `pf.conf` includes it +- validate with `pfctl -nf` +- reload PF and inspect NAT/filter state + +### `cms-strapi-bootstrap.yaml` + +Coordinate: + +- PostgreSQL setup in `db` +- prefer `db-memory-bootstrap.yaml` first when the `db` jail is not already prepared +- `pg_hba.conf` entry for `{{ agent_subnet_base }}.3/32` +- Strapi install in `cms` +- initial env/config + +### `cms-astro-bootstrap.yaml` + +Inside `cms`: + +- create Astro project +- install required packages +- prepare build/deploy script + +### `cms-deploy.yaml` + +Deploy Astro output to the `cms` jail webroot: + +- back up `/srv/www/` +- sync built output from `cms` +- validate local jail nginx first, then the chosen edge mode + +## Safe defaults + +- prefer FreeBSD `pkg` install for host Ansible unless there is a clear reason not to +- treat `uv tool install ansible` as a fallback path that may fail on FreeBSD/Python packaging +- keep one SSH trust path: control machine -> FreeBSD host +- manage `cms`, `db`, and other jails from the host through `bastille`, `bastille cmd`, and jail root paths +- do not enable jail `sshd` just to satisfy Ansible unless there is a real separate-ops need +- keep host nginx optional; default Clawdie web serving lives in `cms` +- keep `db` at `${AGENT_SUBNET_BASE}.3` for PostgreSQL only +- keep `git` optional (`FEATURE_GIT=true`), does not occupy a fixed address +- keep `cms` at `${AGENT_SUBNET_BASE}.4` for Astro + Strapi only +- do not expose Strapi admin publicly by default +- keep rollback simple with snapshots and webroot backup + +## SSH model + +Default model: + +- Ansible SSHes into the FreeBSD host +- the host manages Bastille jails locally +- jail tasks run through `bastille`, `bastille cmd`, package commands, and direct file placement in jail roots + +Why: + +- avoids SSH key sprawl across jails +- avoids enabling `sshd` in service jails unnecessarily +- keeps jails as host-managed runtime units rather than pretending they are separate servers + +Optional model (separate ops boundary): + +- enable `sshd` inside service jails and install the operator key with + `infra/ansible/playbooks/jails-ssh-baseline.yaml` +- optionally expose jail SSH via PF with predictable host ports (derived from + the jail IP last octet): `.2 → :2222`, `.3 → :2223`, `.4 → :2224`, `.5 → :2225`, `.6 → :2226` + (see `infra/ansible/playbooks/host-pf-baseline.yaml` + `jails_ssh_expose_via_pf`) + +This is intentionally opt-in; the default Clawdie model remains host-managed +jails without direct SSH access. + +## When to use Ansible here + +Use this skill when: + +- the same FreeBSD admin action will be repeated +- jail creation should become reproducible +- `cms` setup is stabilizing into known phases +- nginx/site deploys should stop depending on manual copy-paste + +When the task starts as host-level policy or troubleshooting intent, read +`freebsd-admin` first, then use this skill to codify the proven host change. + +Do not use Ansible just to hide one command. Use it when the workflow benefits +from repeatability, validation, and versioned structure. + +## File naming convention + +Use `.yaml` consistently for all Ansible files in this repository: + +- inventories +- playbooks +- vars files +- role task files diff --git a/.agent/skills/ansible-freebsd/references/cms-astro-strapi-plan.md b/.agent/skills/ansible-freebsd/references/cms-astro-strapi-plan.md new file mode 100644 index 0000000..6354edb --- /dev/null +++ b/.agent/skills/ansible-freebsd/references/cms-astro-strapi-plan.md @@ -0,0 +1,142 @@ +# CMS Jail Astro / Strapi Ansible Plan + +Use this reference when implementing Ansible for the Astro + Strapi deployment +path in the `cms` jail at `{{ agent_subnet_base }}.4`. + +## Architecture baseline + +- cms jail nginx is the default Clawdie-owned web server +- `db` jail at `{{ agent_subnet_base }}.3` runs PostgreSQL only +- `cms` jail at `{{ agent_subnet_base }}.4` runs nginx + Strapi + Astro +- public delivery is an edge-mode choice, not a host-nginx requirement +- `cms..home.arpa` can proxy to Strapi admin and remain Tailscale-only + +## Implementation phases + +### Phase 1. Host preflight + +Create `playbooks/host-preflight.yaml`. + +Validate: + +- FreeBSD host +- Bastille present +- release `15.0-RELEASE` available +- `warden0` exists +- `warden0` has `{{ agent_subnet_base }}.1/24` +- `gateway_enable` and `net.inet.ip.forwarding` + +Output should clearly identify missing prerequisites before any jail work. + +### Phase 2. Create the cms jail + +Create `playbooks/jail-cms-create.yaml`. + +Tasks: + +- optional snapshot before creation +- detect whether `cms` already exists +- create with: + - thick jail + - VNET + - bridge `warden0` + - gateway `{{ agent_subnet_base }}.1` + - IP `{{ agent_subnet_base }}.4/24` +- set hostname to `cms..home.arpa` +- restart jail +- validate: + - `hostname` + - `ifconfig` + - `netstat -rn` + - ping `{{ agent_subnet_base }}.3` + +### Phase 3. Bootstrap cms packages + +Create `playbooks/jail-cms-bootstrap.yaml`. + +Inside `cms`: + +- install `node24` +- install `npm` +- install `git` +- install `postgresql18-client` +- ensure `clawdie` user and home +- create: + - `/home/clawdie/strapi` + - `/home/clawdie/clawdie-docs` + +### Phase 4. Bootstrap Strapi + +Create `playbooks/cms-strapi-bootstrap.yaml`. + +Host-side work against `db` and `cms`: + +- create `strapi_cms` database in `db` +- create `strapi_cms` +- set password from Ansible vars +- grant privileges +- add `pg_hba.conf` line for `{{ agent_subnet_base }}.4/32` +- reload PostgreSQL +- install Strapi in `/home/clawdie/strapi` +- write Strapi env/config + +Validation: + +- `psql` from `cms` to `db` +- Strapi starts on `{{ agent_subnet_base }}.4:1337` + +### Phase 5. Bootstrap Astro + +Create `playbooks/cms-astro-bootstrap.yaml`. + +Inside `cms`: + +- create Astro project at `/home/clawdie/clawdie-docs` +- install Astro dependencies +- prepare deploy script +- configure Strapi endpoint as `http://localhost:1337/api` + +### Phase 6. Deploy to the cms jail web stack + +Create `playbooks/cms-deploy.yaml`. + +Tasks: + +- verify Astro project builds +- back up `/srv/www/` to `/srv/www.bak/` +- sync built output into `/srv/www/` +- validate local cms nginx first +- optionally configure one edge mode later + +## Variable model + +Use a small starting variable set: + +```yaml +cms_jail_name: cms +cms_jail_ip: {{ agent_subnet_base }}.4 +cms_hostname: cms..home.arpa +db_jail_ip: {{ agent_subnet_base }}.3 +warden_bridge: warden0 +warden_gateway: {{ agent_subnet_base }}.1 +astro_site_path: /home/clawdie/clawdie-docs +strapi_path: /home/clawdie/strapi +clawdie_webroot: /srv/www +``` + +## First milestone + +The first successful Ansible milestone is: + +1. `host-preflight.yaml` +2. `jail-cms-create.yaml` +3. `jail-cms-bootstrap.yaml` + +Do not try to automate nginx, Strapi, Astro, and deployment all at once. + +## Success criteria + +- `cms` jail reproducibly created on `{{ agent_subnet_base }}.4` +- `cms` can reach `db` on `{{ agent_subnet_base }}.3` +- package bootstrap is repeatable +- later app bootstrap can build on stable infrastructure diff --git a/.agent/skills/ansible-freebsd/references/host-encrypted-dataset.md b/.agent/skills/ansible-freebsd/references/host-encrypted-dataset.md new file mode 100644 index 0000000..fdf06f5 --- /dev/null +++ b/.agent/skills/ansible-freebsd/references/host-encrypted-dataset.md @@ -0,0 +1,80 @@ +# Host Encrypted Dataset + +Use this one-time host bootstrap to create the encrypted ZFS dataset used for +private screenshot storage. + +Do not try to create this dataset from `host-preflight.yaml`. + +`host-preflight.yaml` only validates that: + +- `zroot//encrypted` exists +- its mountpoint is `/encrypted` +- `encrypted/screenshots/` exists and is writable by the operator user + +## Why Manual Bootstrap + +This dataset uses passphrase-based ZFS encryption. + +Creating it correctly requires an explicit host-side operator step: + +- choose a passphrase interactively +- confirm the mountpoint +- set ownership for the operator user + +That is not a good fit for non-interactive Ansible preflight. + +## One-Time Creation + +Run on the FreeBSD host as `root`. + +Replace: + +- `` with the actual ZFS prefix, for example `clawdie-runtime` +- `` with the repo path, for example `/home/clawdija/clawdie-ai` +- `` with the operator user, for example `clawdie` + +```sh +zfs list zroot//encrypted >/dev/null 2>&1 || \ +zfs create \ + -o encryption=on \ + -o keyformat=passphrase \ + -o keylocation=prompt \ + -o mountpoint=/encrypted \ + zroot//encrypted +``` + +Create the private screenshots directory and make it writable by the operator: + +```sh +install -d -o -g wheel -m 0750 /encrypted/screenshots +``` + +## Validation + +Check dataset and mountpoint: + +```sh +zfs list zroot//encrypted +zfs get mountpoint zroot//encrypted +``` + +Check ownership: + +```sh +ls -ld /encrypted +ls -ld /encrypted/screenshots +``` + +Expected: + +- dataset exists +- mountpoint is `/encrypted` +- `encrypted/screenshots` is owned by the operator user + +## Operational Rule + +If the encrypted dataset is missing or mounted elsewhere: + +- do not run screenshot capture expecting private copies +- fix the host dataset first +- rerun `host-preflight.yaml` diff --git a/.agent/skills/ansible-freebsd/references/install.md b/.agent/skills/ansible-freebsd/references/install.md new file mode 100644 index 0000000..ceb80be --- /dev/null +++ b/.agent/skills/ansible-freebsd/references/install.md @@ -0,0 +1,41 @@ +# Installing Ansible on the FreeBSD Host + +Use this reference before building or running Clawdie Ansible playbooks. + +## Recommended default + +Prefer the native FreeBSD package: + +```sh +sudo pkg update +sudo pkg install sysutils/ansible +ansible --version +ansible-playbook --version +``` + +Why: + +- boring host path +- avoids Python build surprises +- fits the host-managed Clawdie model +- avoids inventing a jail-local Ansible control node + +## Fallback path + +`uv` can work, but it is not the default on FreeBSD: + +```sh +sudo uv tool install --python 3.11 ansible +``` + +Only use that path if native `pkg` is blocked and the host Python line is +already understood. + +## Operational rule + +Keep one SSH trust path: + +- Ansible SSHes into the FreeBSD host +- the host manages Bastille jails locally + +Do not install or enable jail `sshd` just to satisfy Ansible. diff --git a/.agent/skills/ansible-freebsd/references/layout.md b/.agent/skills/ansible-freebsd/references/layout.md new file mode 100644 index 0000000..d136e6f --- /dev/null +++ b/.agent/skills/ansible-freebsd/references/layout.md @@ -0,0 +1,85 @@ +# Ansible Layout for Clawdie FreeBSD Operations + +Use this reference when creating the first Ansible structure in the repository. + +## Goal + +Keep the layout obvious for both humans and agents. + +```text +infra/ansible/ + inventories/ + production/ + hosts.yaml + group_vars/ + host_vars/ + playbooks/ + roles/ +``` + +Use `.yaml` consistently throughout this tree. + +## Inventory model + +Recommended starting model: + +- one production host +- no premature multi-host complexity +- jail operations still target the host and shell into Bastille/bastille cmd where needed +- do not model `cms` or `db` as separate SSH inventory hosts by default + +Example inventory shape: + +```yaml +all: + hosts: + clawdie_host: + ansible_host: your-hostname-or-ip + ansible_user: your-admin-user +``` + +Optional: + +```yaml + ansible_ssh_private_key_file: ~/.ssh/clawdie-ansible +``` + +The important rule is that Ansible reaches the host, and the host reaches the +jails. + +## Playbook boundaries + +Keep playbooks phase-oriented: + +- host preflight +- host networking +- host nginx +- jail creation +- jail bootstrap +- application bootstrap +- deployment + +Do not create one giant playbook for everything. + +## Role boundaries + +Keep roles generic enough to reuse, but not abstract to the point of hiding +what the host actually does. + +Good first roles: + +- `freebsd_base` +- `freebsd_network` +- `nginx_host` +- `bastille_jail` +- `cms_jail` + +## Implementation style + +Prefer: + +- `community.general.sysrc` +- package/service/file/template modules where they fit +- explicit `command` or `shell` for `bastille`, `zfs`, and `bastille cmd` + +The goal is boring reliability, not maximum abstraction. diff --git a/.agent/skills/astro-wiki-deploy/SKILL.md b/.agent/skills/astro-wiki-deploy/SKILL.md new file mode 100644 index 0000000..e320a06 --- /dev/null +++ b/.agent/skills/astro-wiki-deploy/SKILL.md @@ -0,0 +1,181 @@ +--- +name: astro-wiki-deploy +description: Deploy the Colibri wiki (astro/wiki/) to wiki.clawdie.si via the CMS jail. Plain Astro — no Starlight. Covers content staging, build, deploy, and the 6 pitfalls discovered during the initial deployment (26.jun.2026). +--- + +# Astro Wiki Deploy + +Deploy the plain-Astro Colibri wiki to `https://wiki.clawdie.si`. + +## Architecture + +``` +colibri/ + docs/wiki/ ← canonical content (23 EN + 23 SL .md files) + astro/wiki/ ← Astro build pipeline (minimal, no Starlight) + src/pages/ + index.astro — EN landing (flat list) + [...slug].astro — EN dynamic route (reads src/content/*.md) + sl/index.astro — SL landing + sl/[...slug].astro — SL dynamic route (reads src/content/sl/*.md) +``` + +CMS jail builds the site, host nginx serves it. Same pattern as docs.clawdie.si. + +## Full deploy flow + +```sh +# 1. Stage content into Astro project (source of truth → build input) +cd /home/clawdie/ai/colibri +rm -rf astro/wiki/src/content +cp -r docs/wiki astro/wiki/src/content + +# 2. Copy source into CMS jail +sudo cp -r astro/wiki/src \ + /usr/local/bastille/jails/cms/root/usr/home/clawdie/clawdie-wiki/ + +# 3. Build inside jail (rm -rf dist for clean state — see pitfall #4) +sudo bastille cmd cms sh -c ' + cd /usr/home/clawdie/clawdie-wiki && + rm -rf dist node_modules/.astro && + ASTRO_SITE_URL="https://wiki.clawdie.si" npm run build' + +# 4. Deploy to jail webroot +sudo bastille cmd cms sh -c ' + rm -rf /usr/local/www/wiki.clawdie.si && + mkdir -p /usr/local/www/wiki.clawdie.si && + cp -r /usr/home/clawdie/clawdie-wiki/dist/* /usr/local/www/wiki.clawdie.si/' + +# 5. Cross jail boundary to host webroot (tar is the reliable bridge) +sudo bastille cmd cms sh -c \ + 'tar -czf /tmp/wiki-dist.tar.gz -C /usr/local/www/wiki.clawdie.si .' +sudo cp /usr/local/bastille/jails/cms/root/tmp/wiki-dist.tar.gz /tmp/ +sudo rm -rf /usr/local/www/wiki.clawdie.si +sudo mkdir -p /usr/local/www/wiki.clawdie.si +sudo tar -xzf /tmp/wiki-dist.tar.gz -C /usr/local/www/wiki.clawdie.si/ +sudo chown -R clawdie:clawdie /usr/local/www/wiki.clawdie.si + +# 6. Verify +curl -sk --resolve wiki.clawdie.si:443:127.0.0.1 https://wiki.clawdie.si/ | head -5 +curl -sk --resolve wiki.clawdie.si:443:127.0.0.1 https://wiki.clawdie.si/sl/ | head -5 +``` + +## First-time setup (CMS jail) + +```sh +# Stage the Astro project into the jail once +sudo mkdir -p /usr/local/bastille/jails/cms/root/usr/home/clawdie/clawdie-wiki +sudo cp -r astro/wiki/package.json astro/wiki/astro.config.mjs \ + /usr/local/bastille/jails/cms/root/usr/home/clawdie/clawdie-wiki/ + +# Install dependencies inside jail +sudo bastille cmd cms sh -c 'cd /usr/home/clawdie/clawdie-wiki && npm install' +``` + +## Build checklist (7 steps to a clean deploy) + +Run through these in order. Each step produces a known-good intermediate state. + +### 1. Quote all YAML frontmatter values + +**Why:** Colons, double-quotes, and em dashes in Slovenian (or English) +titles can be read as YAML syntax. Quoting prevents parse failures. + +**What:** Wrap every title and description containing `:`, `"`, or `—` in +double quotes. Use single-quote wrappers when the value itself contains +double-quotes. + +```yaml +# ✅ Works +title: "Agentska vprega: pi, zot & Colibri" +description: 'Sprememba ni "končana" brez uspešnega preverjanja.' +``` + +**Result:** `npm run build` starts without YAML parse errors. + +### 2. Declare content-path constants inside `getStaticPaths()` + +**Why:** Astro's SSR compiler moves top-level `const` declarations into a +different scope. Variables needed for route generation must live inside +the function that uses them. + +**What:** Move `const WIKI_DIR`, `const EXCLUDE`, and the `walk()` helper +into the `getStaticPaths()` body. + +```js +// ✅ Works — everything inside the function +export function getStaticPaths() { + const WIKI_DIR = path.resolve("src/content"); + const EXCLUDE = [".git", "index.md"]; + function walk(dir, prefix = "") { ... } + return walk(WIKI_DIR); +} +``` + +**Result:** `WIKI_DIR` is defined at route generation time. No "is not +defined" errors. + +### 3. Create explicit routes for SL content + +**Why:** A `src/content/sl/` directory triggers Astro's content-collection +auto-generation, which creates routes with hardcoded wrong paths. Explicit +routes read from the same directory correctly. + +**What:** Create `src/pages/sl/index.astro` and `src/pages/sl/[...slug].astro` +that read directly from `src/content/sl/`. No content collection config file +needed. + +**Result:** SL pages build with correct paths and no auto-generated artifacts. + +### 4. Clean the dist cache before every build + +**Why:** A failed or interrupted build leaves compiled `.mjs` files in +`dist/` and `.astro/` that shadow fixed source code on the next attempt. + +**What:** Always delete the cache before building: + +```sh +rm -rf dist node_modules/.astro && npm run build +``` + +**Result:** Every build starts from a blank slate. Fixed source → fixed output, +no stale artifacts. + +### 5. Resolve content paths with `path.resolve("src/content")` + +**Why:** `import.meta.url` in Astro SSR points to the compiled `.mjs` module +in `dist/`, not the project root. `path.resolve` with a relative path uses +`process.cwd()`, which IS the project root during build. + +**What:** All content path resolution uses `path.resolve("src/content")` +for the EN content and `path.resolve("src/content/sl")` for SL. + +**Result:** File reads find the actual `.md` source files, not compiled +module stubs. + +### 6. Extract markdown H1 as a title fallback + +**Why:** Pages with only a markdown H1 (`# Title`) and no YAML frontmatter +show the URL slug in ``. The H1 is a better human-readable fallback. + +**What:** Three-tier fallback: + +```js +const title = (frontmatter.title || content.match(/^#\s+(.+)$/m)?.[1] || slug) + .replace(/^["']|["']$/g, ""); +``` + +**Result:** Every page has a human-readable `<title>` tag — frontmatter first, +H1 second, slug last. + +### 7. Placeholder TLS cert for first deploy (reference) + +**Why:** nginx refuses to start when `ssl_certificate` files don't exist, +but acme.sh needs nginx running to validate the domain via HTTP-01. + +**What:** Create a self-signed placeholder cert, start nginx, issue the real +cert with acme.sh. The real cert overwrites the placeholder automatically. + +**Result:** nginx starts on first attempt, acme.sh validates, real cert +replaces placeholder. Full steps in the **nginx** skill §"Adding a new +public static HTTPS site." diff --git a/.agent/skills/astro/SKILL.md b/.agent/skills/astro/SKILL.md new file mode 100644 index 0000000..3afd21e --- /dev/null +++ b/.agent/skills/astro/SKILL.md @@ -0,0 +1,407 @@ +--- +name: astro +description: Manage the Astro static site that powers Clawdie web surfaces. Use when building pages, configuring Strapi integration, serving from the cms jail web stack, or managing the frontend project. Triggers on "astro", "frontend", "build site", "static site", "clawdie.invalid build". +--- + +# Astro + +Use this skill for the static site generator that builds tenant homes and tenant sites from Strapi and repo content. + +This skill is also responsible for the transition from the current plain HTML +site to the future Astro + Strapi setup in the `cms` jail. + +## Scope + +This skill covers: + +- Astro project setup and configuration +- Strapi content integration (REST API) +- Page and layout creation +- Build and deployment to the `cms` jail webroot +- Tenant home (`<tenant>.<base>`) and tenant-site (`<site>.<tenant>.<base>`) web output +- Design system porting from current static HTML +- Skill page generation from SKILL.md files +- Build automation and webhooks + +This skill does not replace: + +- `strapi` for content management +- `nginx` for web server configuration + +## Architecture + +``` +cms jail (${AGENT_SUBNET_BASE}.4) + ├── /home/clawdie/clawdie-si/ ← public clawdie.si landing site + │ ├── source: bootstrap/cms/clawdie-si/ + │ └── dist/ → /usr/local/www/clawdie-si/ + │ + ├── /home/clawdie/clawdie-docs/ ← docs.clawdie.si Starlight site + │ ├── source: bootstrap/cms/clawdie-docs/ + │ ├── docs source of truth: repo docs/public/ + │ └── dist/ → /usr/local/www/clawdie/ + │ + └── optional Strapi runtime (localhost:1337), not public by default +``` + +Astro output is pure static HTML/CSS/JS served by nginx in the `cms` jail and +fronted by the selected edge mode. The current landing/docs sites are +repo-native Astro projects; Strapi remains optional and should not be assumed +for a landing-page deploy. + +For Starlight docs, `docs/public/` is the source of truth. The +`bootstrap/cms/clawdie-docs/src/content/docs/` tree is generated by the +`prebuild` sync and should not be hand-edited or committed just because a local +build regenerated it. + +## Surface model + +- Operator app: `ai.<base>` — separate stack, not Astro here +- Shared CMS host: `cms.<base>` — shared admin/API, not a tenant app +- Tenant home: `<tenant>.<base>` — Astro/static app +- Tenant site: `<site>.<tenant>.<base>` — Astro/static app + +Default internal base: `home.arpa`. + +## Migration goal + +The immediate goal is not a redesign. It is to let the agent transform the +current live static site into an Astro-based clone first, then gradually move +editable content into Strapi. + +Recommended sequence: + +1. replicate the current site structure and design in Astro +2. keep public exposure decisions separate from the content migration +3. classify content into Astro-managed vs Strapi-managed +4. move selected content into Strapi in batches +5. serve Astro output from the `cms` jail webroot under tenant-home and tenant-site hostnames + +For the first FreeBSD deployment, keep the Astro site deliberately minimal: + +- do not add `sharp` +- do not rely on Astro image optimization +- keep images in `public/` or serve prebuilt static assets directly +- treat `examples/astro-cv/` as a prototype/reference, not the bootstrap source + +## Canonical paths + +Repo paths: + +- Landing source: `bootstrap/cms/clawdie-si/` +- Docs source: `bootstrap/cms/clawdie-docs/` +- Public docs source of truth: `docs/public/` + +Inside the `cms` jail: + +- Landing project: `/home/clawdie/clawdie-si/` +- Landing build output: `/home/clawdie/clawdie-si/dist/` +- Landing deploy target: `/usr/local/www/clawdie-si/` +- Docs project: `/home/clawdie/clawdie-docs/` +- Docs build output: `/home/clawdie/clawdie-docs/dist/` +- Docs deploy target: `/usr/local/www/clawdie/` +- Strapi endpoint, when enabled: `http://localhost:1337/api` + +Deployment scripts: + +- Landing: `bootstrap/cms/clawdie-si/scripts/deploy.mjs` +- Docs: `bootstrap/cms/clawdie-docs/scripts/deploy-docs.mjs` + +Build defaults: + +- Use `passthroughImageService()`; do not require `sharp`. +- Keep images under `public/` as static assets. +- Use Node 24 where possible; Node 22 has been sufficient for current Astro builds, but Node 24 is the target runtime. + +## Design system + +Port the existing public Clawdie visual identity: + +### Fonts + +- Headings: `Cormorant Garamond` (300, 400, 600 weights) +- Monospace: `DM Mono` (300, 400 weights) + +### Colors (dark theme) + +```css +--cream: #0d1117; /* page background */ +--amber: #00b4d8; /* accent / links */ +--amber-dark: #0096b7; /* hover states */ +--charcoal: #e2e8f0; /* headings */ +--grey: #8b949e; /* secondary text */ +--grey-light: #21262d; /* borders / dividers */ +--paper: #161b22; /* card backgrounds */ +--ink: #c9d1d9; /* body text */ +``` + +### Components to port + +- Hero statement block (dark background, accent text) +- Status note (left border, label) +- Comparison grid (two-column, cloud vs clawdie) +- Ecosystem cards (grid, featured variant) +- Principle rows (numbered, icon + text) +- CTA block (centered, button row) +- Divider (gradient line) +- Top nav (monospace, uppercase) + +## Workflow + +### Phase 1: Project setup + +1. Read `references/setup.md` +2. Read `references/static-migration.md` +3. Create Astro project with Strapi integration +4. Port design system CSS +5. Create base layout matching current site + +### Phase 1.5: Static-site inventory + +Before changing content sources: + +1. inspect the current live/static pages +2. map each route to an Astro page +3. identify shared layout pieces (nav, footer, hero, cards, guides) +4. reproduce the current visual design as closely as practical + +### Phase 2: Page templates + +Create Astro page templates: + +| Template | Source | Route | +| --------------------------- | --------------------------- | ------------------------ | +| `pages/index.astro` | Strapi: Page (tenant home) | `/` on `<tenant>.<base>` | +| `pages/docs/index.astro` | Strapi: Page (docs) | `/docs/` | +| `pages/guides/[slug].astro` | Strapi: Guide collection | `/guides/{slug}` | +| `pages/skills/[slug].astro` | Local SKILL.md files | `/skills/{slug}` | +| `pages/skills/index.astro` | Generated from skills list | `/skills/` | +| `pages/blog/[slug].astro` | Strapi: BlogPost collection | `/blog/{slug}` | + +### Phase 3: Skill page generation + +Skills are imported at build time from the local filesystem: + +```astro +// src/pages/skills/[slug].astro +import { getCollection } from 'astro:content'; + +// Read .agent/skills/*/SKILL.md files +// Parse YAML frontmatter for metadata +// Render markdown body as HTML +``` + +This keeps skills in sync with code — no manual CMS updates needed. + +### Phase 3.5: Content split + +Move only the right content into Strapi. + +Good Strapi candidates: + +- homepage editable sections +- guides +- docs landing pages +- future blog/project pages + +Keep in Astro/repo: + +- skill pages generated from `SKILL.md` +- highly technical generated docs +- content that should always follow repository state + +### Phase 4: Build and deploy + +For the public landing page: + +```sh +cd /home/clawdie/clawdie-si +npm run build +npm run deploy +``` + +For documentation: + +```sh +cd /home/clawdie/clawdie-docs +npm run build +npm run deploy +``` + +Before deploying from an agent session: + +1. Confirm the repo is on `main` and clean. +2. Build locally or in the jail first. +3. Snapshot or back up the target webroot unless the operator explicitly says to skip it because of disk pressure. +4. Deploy only `dist/` output; never delete the project source. +5. Validate via jail-local HTTP with the correct `Host:` header and public HTTPS. + +### Phase 5: Automation + +Build triggers (choose one): + +**Option A: Strapi webhook** + +- Strapi fires webhook on content publish +- Host script receives webhook and runs build +- Best for content-driven updates + +**Option B: Cron rebuild** + +- `cron` runs `astro build && rsync` every 15 minutes +- Simple, no webhook infrastructure needed +- Good enough for low-frequency updates + +**Option C: Manual build** + +- Operator or agent runs `npm run build` when needed +- Simplest, full control + +Recommended: Start with Option C, move to B, then A as needed. + +Keep hostname routing and nginx server_name policy in the `nginx` skill and setup code. Astro owns page output, not host classification. + +## FreeBSD jail install snags + +### sharp has no pre-built binary for FreeBSD + +`sharp` is an optional Astro image optimization library. It tries to build from +source on FreeBSD, which requires `node-addon-api` to be bundled. + +**Symptoms:** + +``` +npm error sharp: Attempting to build from source via node-gyp +npm error sharp: Please add node-addon-api to your dependencies +``` + +**Fix — two steps:** + +1. Install without native scripts: + + ```sh + npm install --ignore-scripts + ``` + +2. Configure Astro to use the passthrough image service (no optimization): + ```js + // astro.config.mjs + import { defineConfig, passthroughImageService } from 'astro/config'; + export default defineConfig({ + image: { service: passthroughImageService() }, + // ... rest of config + }); + ``` + +This is fine for static/docs sites. Keep images in `public/` as static assets. +If you actually need image optimization (WebP conversion, resize), `vips` is +available via pkg (`pkg install vips`) but getting sharp to link against it +requires additional build steps. + +### npm config "python" warning + +You may see `Unknown env config "python"` during npm install. This is a harmless +warning from a stale global npm config. Ignore it. + +### Starlight confirmed working on FreeBSD + Node 22 + +With `--ignore-scripts` + `passthroughImageService()`, Starlight builds cleanly. +Scaffold command: `npm create astro@latest . -- --template starlight --no-install --no-git` + +## Safe defaults + +- Always ZFS snapshot before deploying a new build +- Keep the previous build as `.bak/` fallback before deploying new dist +- Test build locally before deploying: `npm run preview` +- Never delete the Astro project source — only the `dist/` output is expendable +- Run `nginx -t` inside the cms jail after nginx config changes +- Aim for route and visual parity before changing too much content structure +- Always use `npm install --ignore-scripts` on FreeBSD +- Always configure `passthroughImageService()` — never rely on sharp + +## Build commands + +Landing site: + +```sh +cd bootstrap/cms/clawdie-si +npm run build +npm run preview +npm run deploy +``` + +Docs site: + +```sh +cd bootstrap/cms/clawdie-docs +npm run build +npm run preview +npm run deploy +``` + +Inside the jail the project paths are usually `/home/clawdie/clawdie-si` and +`/home/clawdie/clawdie-docs`; in the repository they live under +`bootstrap/cms/`. + +## Deployment verification + +When testing the `cms` jail nginx directly, always send the public host header. +The default jail vhost intentionally returns `404`, so a plain +`http://127.0.0.1/sl/` check is a false negative for `clawdie.si`. + +Landing site checks: + +```sh +sudo bastille cmd cms service nginx onestatus +sudo bastille cmd cms curl -sI -H 'Host: clawdie.si' http://127.0.0.1/sl/ +sudo bastille cmd cms curl -sI -H 'Host: clawdie.si' http://127.0.0.1/en/ +curl -sI https://clawdie.si/sl/ +curl -sI https://clawdie.si/en/ +curl -s https://clawdie.si/sl/ | grep -F 'Ustvarjaj · Poseduj · Skaliraj' +``` + +Expected results: + +- jail-local `curl -H 'Host: clawdie.si'` returns `200` +- public HTTPS `/sl/` and `/en/` return `200` +- `/` may redirect to `/en/` depending on the landing nginx config + +Do not use FreeBSD `fetch` for the host-header check; `fetch` does not support a +simple `--header` option. Use `curl`, which is installed in the `cms` jail. + +## Troubleshooting + +### Build fails with Strapi fetch error + +- Check Strapi is running: `curl -s http://10.0.0.3:1337/api` +- Check network: can host reach jail IP? +- Check API permissions in Strapi admin +- Astro can build with fallback content if Strapi is down (graceful degradation) + +### Styles look wrong after build + +- Check CSS variables are ported correctly +- Compare with the checked-in bridge HTML under `html/clawdie/` +- Check Google Fonts imports in layout + +### Skill pages missing or stale + +- Skills are read at build time from `.agent/skills/*/SKILL.md` +- Rebuild to pick up new or updated skills +- Check file path pattern matches in Astro config + +### Deploy doesn't update site + +- Static files are served immediately by nginx — hard refresh browser. +- Check rsync output for errors. +- Landing: verify files landed in `/usr/local/www/clawdie-si/` inside `cms`. +- Docs: verify files landed in `/usr/local/www/clawdie/` inside `cms`. +- Check file permissions with `ls -la` on the target webroot. +- If jail-local direct HTTP returns `404`, retry with the expected host header: `curl -sI -H 'Host: clawdie.si' http://127.0.0.1/sl/`. +- If docs build dirties `bootstrap/cms/clawdie-docs/src/content/docs/`, remember that tree is generated from `docs/public/`; do not commit it unless intentionally changing generated-source policy. + +### Sharp or image build errors + +- Use `npm install --ignore-scripts` — never let sharp try to build from source +- Add `passthroughImageService()` to `astro.config.mjs` — see FreeBSD snags above +- Keep images under `public/` as static assets, skip Astro image optimization diff --git a/.agent/skills/astro/references/setup.md b/.agent/skills/astro/references/setup.md new file mode 100644 index 0000000..81128d8 --- /dev/null +++ b/.agent/skills/astro/references/setup.md @@ -0,0 +1,174 @@ +# Astro Project Setup + +## Prerequisites + +- `cms` jail running at ${AGENT_SUBNET_BASE}.4 with Node.js 24 +- Strapi running in cms jail (or plan to set up later) +- nginx running inside `cms` and serving `/srv/www/` + +## Step 1: Create project (inside cms jail) + +```sh +sudo bastille cmd cms su - clawdie -c "cd /home/clawdie && npm create astro@latest clawdie-docs -- --template minimal --no-install" +sudo bastille cmd cms su - clawdie -c "cd /home/clawdie/clawdie-docs && npm install" +``` + +## Step 2: Install integrations + +```sh +npm install @astrojs/mdx +npm install gray-matter # for parsing SKILL.md frontmatter +npm install marked # for rendering markdown +``` + +## Step 3: Configure Astro + +Edit `astro.config.mjs`: + +```js +import { defineConfig } from 'astro/config'; +import mdx from '@astrojs/mdx'; + +export default defineConfig({ + site: 'https://clawdie.invalid', + output: 'static', + integrations: [mdx()], + build: { + assets: 'assets', + }, +}); +``` + +## Step 4: Create base layout + +Create `src/layouts/Base.astro` with: + +- Google Fonts imports (Cormorant Garamond, DM Mono) +- CSS variables from current design system +- Hex background pattern SVG +- Top nav component +- Footer component + +Port directly from the current `/usr/local/www/clawdie/index.html` styles. + +## Step 5: Create Strapi fetch utility + +Create `src/lib/strapi.ts`: + +```ts +const STRAPI_URL = import.meta.env.STRAPI_URL || 'http://localhost:1337'; + +export async function fetchAPI(path: string) { + const res = await fetch(`${STRAPI_URL}/api${path}`); + if (!res.ok) { + console.error(`Strapi fetch failed: ${path} (${res.status})`); + return null; + } + const json = await res.json(); + return json.data; +} + +export async function getPage(slug: string) { + return fetchAPI(`/pages?filters[slug][$eq]=${slug}&populate=*`); +} + +export async function getGuides() { + return fetchAPI('/guides?sort=order:asc&populate=*'); +} + +export async function getGuide(slug: string) { + return fetchAPI(`/guides?filters[slug][$eq]=${slug}&populate=*`); +} +``` + +## Step 6: Create skill loader + +Create `src/lib/skills.ts`: + +```ts +import fs from 'node:fs'; +import path from 'node:path'; +import matter from 'gray-matter'; +import { marked } from 'marked'; + +// nullfs-mounted from host into cms jail +const SKILLS_DIR = '/mnt/skills'; + +export interface Skill { + name: string; + slug: string; + description: string; + content: string; + html: string; +} + +export function getAllSkills(): Skill[] { + const dirs = fs.readdirSync(SKILLS_DIR); + return dirs + .filter(d => fs.existsSync(path.join(SKILLS_DIR, d, 'SKILL.md'))) + .map(d => { + const raw = fs.readFileSync(path.join(SKILLS_DIR, d, 'SKILL.md'), 'utf-8'); + const { data, content } = matter(raw); + return { + name: data.name || d, + slug: d, + description: data.description || '', + content, + html: marked(content) as string, + }; + }) + .sort((a, b) => a.name.localeCompare(b.name)); +} + +export function getSkill(slug: string): Skill | undefined { + return getAllSkills().find(s => s.slug === slug); +} +``` + +## Step 7: Create deploy script + +Add to `package.json`: + +```json +{ + "scripts": { + "deploy": "npm run build && rsync -av --delete dist/ /srv/www/" + } +} +``` + +## Step 8: Environment file + +Create `.env` inside cms jail: + +``` +STRAPI_URL=http://localhost:1337 +SITE_URL=https://clawdie.invalid +NODE_OPTIONS=--max-old-space-size=512 +``` + +## Step 9: Test build + +```sh +npm run build +ls dist/ +``` + +Verify output contains `index.html` and expected pages. + +## Step 10: First deploy + +```sh +# snapshot before deploy +sudo zfs snapshot -nv zroot/ROOT/default@pre-astro-deploy +sudo zfs snapshot zroot/ROOT/default@pre-astro-deploy + +# backup current jail webroot +cp -r /srv/www /srv/www.bak + +# deploy +npm run deploy + +# verify +curl -sI http://127.0.0.1 | head -5 +``` diff --git a/.agent/skills/astro/references/static-migration.md b/.agent/skills/astro/references/static-migration.md new file mode 100644 index 0000000..ec2ddce --- /dev/null +++ b/.agent/skills/astro/references/static-migration.md @@ -0,0 +1,70 @@ +# Static HTML to Astro / Strapi Migration + +Use this reference when converting the current Clawdie public site from +hand-edited static HTML into an Astro site with Strapi-backed editable content. + +## Core principle + +Clone first, restructure second. + +Do not start with a redesign. First reproduce the current public site closely +enough that switching the build source does not feel like launching a different +website. + +## Migration phases + +### 1. Inventory the current site + +Read the current static pages in `/usr/local/www/clawdie/` and classify them: + +- page should become an Astro template backed by Strapi +- page should stay Astro/repo-managed +- page can remain simple static output + +## 2. Capture the design system + +Extract and preserve: + +- typography +- colors +- spacing +- layout rhythm +- navigation structure +- footer and CTA blocks + +## 3. Recreate route parity + +Target the same routes first: + +- `/` +- `/docs/` +- `/guides/*` + +Route parity matters before deeper content modeling. + +## 4. Move content selectively + +Good candidates for Strapi: + +- homepage sections needing editorial updates +- guides +- docs landing pages + +Bad candidates for Strapi: + +- skill pages from repository `SKILL.md` +- generated technical references +- content tightly coupled to code state + +## 5. Decouple content migration from edge ownership + +Build and serve the site inside `cms` first. Public edge choice comes later. + +That means the cutover is: + +- old source: hand-edited HTML bridge pages +- new source: Astro build output served from `/srv/www/` inside `cms` +- public delivery: existing reverse proxy, host PF redirect, direct jail IP, or + internal-only + +This keeps content migration separate from the operator's host web stack. diff --git a/.agent/skills/backup-db/SKILL.md b/.agent/skills/backup-db/SKILL.md new file mode 100644 index 0000000..8e7b500 --- /dev/null +++ b/.agent/skills/backup-db/SKILL.md @@ -0,0 +1,65 @@ +--- +name: backup-db +description: Create a full PostgreSQL database backup (pg_dump) and store in project-relative backup directory +compatibility: FreeBSD 15.x +invoke_patterns: + - "Back up the database" + - "Back up * database" + - "Database backup" + - "Dump * database" + - "Create backup" + - "Back up before migration" +estimated_tokens: 800-1200 +--- + +# backup-db + +Full PostgreSQL dump using `pg_dump`. Backs up to `data/backups/` under project root. Never uses `/tmp/`. + +## Usage + +```bash +# Full dump, compressed +bastille cmd db pg_dump -U clawdie -Fc clawdie_ai_public \ + > /home/clawdie/clawdie-ai/data/backups/clawdie_ai_public_$(date +%Y-%m-%d_%H%M%S).dump + +# Verify backup +bastille cmd db pg_restore --list /home/clawdie/clawdie-ai/data/backups/<file>.dump | head -20 +``` + +## Output + +``` +Backup created: data/backups/clawdie_ai_public_2026-04-07_103045.dump +Size: 2.3 GB +Compression ratio: 4.2:1 +Duration: 18m 3s +Verify: OK (1842 objects listed) +``` + +## Retention + +- Keep 7 daily backups minimum +- Before any migration: always create a backup first +- Before any schema change: same + +## Directory + +``` +data/backups/ +├── clawdie_ai_public_2026-04-07_103045.dump +├── clawdie_2026-04-07_103200.dump +└── ... +``` + +## When to Use + +- Before running `db-migrate` +- Before any risky schema change +- On-demand from CEO or operator instruction + +## Escalate If + +- Backup file size is 0 or unexpectedly small → escalate to CEO +- Backup takes >60 min (possible lock contention) → escalate to DBA +- Disk full before backup completes → escalate immediately diff --git a/.agent/skills/coding-agent/SKILL.md b/.agent/skills/coding-agent/SKILL.md new file mode 100644 index 0000000..a8f3618 --- /dev/null +++ b/.agent/skills/coding-agent/SKILL.md @@ -0,0 +1,166 @@ +--- +name: coding-agent +description: Configure or debug pi (the LLM coding agent subprocess) in Clawdie. Use when changing the AI provider, model, or pi settings; when pi fails to respond; or when setting up pi for the first time. pi is @earendil-works/pi-coding-agent from the Codeberg pi project, installed at /opt/npm/bin/pi inside the clawdie-controlplane jail. +--- + +# Coding Agent (pi) Configuration + +Clawdie uses **pi** (`@earendil-works/pi-coding-agent`) as its LLM execution engine. For each incoming message, `src/agent-runner.ts` spawns `/opt/npm/bin/pi` as a subprocess in `--print` (non-interactive) mode and streams the response back. + +**Current setup:** ZAI provider (`PI_TUI_PROVIDER=zai`), model GLM-5.1 (`PI_TUI_MODEL=GLM-5.1`), binary at `/opt/npm/bin/pi` inside the `clawdie-controlplane` jail. + +## .env Variables + +```bash +PI_TUI_BIN=/opt/npm/bin/pi # Path to pi binary (inside jail) +PI_TUI_PROVIDER=zai # LLM provider (zai, anthropic, openai, openrouter, ollama) +PI_TUI_MODEL=GLM-5.1 # Model name as pi expects it +ZAI_API_KEY=<key> # API key for the ZAI provider +OPENROUTER_API_KEY=<key> # If switching to OpenRouter +ANTHROPIC_API_KEY=<key> # If switching to Anthropic/Claude +``` + +## Checking Current pi Setup + +```bash +# Verify binary exists and version +sudo bastille cmd clawdie-controlplane /opt/npm/bin/pi --version + +# Check pi auth config (inside jail as root) +sudo bastille cmd clawdie-controlplane cat /root/.pi/agent/auth.json + +# Check pi settings (default provider/model) +sudo bastille cmd clawdie-controlplane cat /root/.pi/agent/settings.json + +# Verify skills symlink +sudo bastille cmd clawdie-controlplane ls /root/.pi/agent/skills/ +``` + +## Switching Provider or Model + +Edit `.env` (on the host — nullfs-mounted into jail): + +```bash +# Example: switch to OpenRouter with Qwen +PI_TUI_PROVIDER=openrouter +PI_TUI_MODEL=qwen/qwen3-14b-instruct +OPENROUTER_API_KEY=sk-or-v1-... +``` + +Then restart the agent: + +```bash +sudo bastille cmd clawdie-controlplane service clawdie restart +``` + +Send a test message on Telegram and check `logs/clawdie.log` for the new provider name in spawn logs. + +## Installing / Reinstalling pi + +**IMPORTANT:** pi is published as `@earendil-works/pi-coding-agent`. +Older docs or installs may mention legacy package names; use the `pi-update` +skill for package-rename migrations. In the controlplane jail, install from the +host's pre-built copy so the jail does not need to rebuild the multi-package pi +project: + +```bash +# Set npm prefix first (required — fresh jail defaults to /usr/local) +sudo bastille cmd clawdie-controlplane npm config set prefix /opt/npm + +# Install from host's already-built copy +sudo bastille cmd clawdie-controlplane \ + npm install -g /usr/home/clawdie/.npm-global/lib/node_modules/@earendil-works/pi-coding-agent + +# Verify +sudo bastille cmd clawdie-controlplane /opt/npm/bin/pi --version +``` + +After reinstall, copy auth and settings: + +```bash +sudo bastille cmd clawdie-controlplane mkdir -p /root/.pi/agent +sudo bastille cmd clawdie-controlplane \ + cp /usr/home/clawdie/.pi/agent/auth.json /root/.pi/agent/auth.json +sudo bastille cmd clawdie-controlplane \ + cp /usr/home/clawdie/.pi/agent/settings.json /root/.pi/agent/settings.json +sudo bastille cmd clawdie-controlplane \ + ln -sf /usr/home/clawdie/clawdie-ai/.agent/skills /root/.pi/agent/skills +``` + +## Troubleshooting pi + +### pi not found + +```bash +sudo bastille cmd clawdie-controlplane ls /opt/npm/bin/pi +# If missing: reinstall (see above) +``` + +### pi exits with no output + +Read the per-run log: + +```bash +ls -t groups/*/logs/agent-*.log | head -3 +tail -100 groups/Samo/logs/agent-<runId>.log +``` + +Common causes: + +- Wrong `PI_TUI_MODEL` name — check provider's model list +- API key expired or missing in `/root/.pi/agent/auth.json` +- `PI_TUI_BIN` points to wrong path + +### Adding a new provider to pi auth + +```bash +# Edit /root/.pi/agent/auth.json inside jail +sudo bastille cmd clawdie-controlplane \ + sh -c 'cat /root/.pi/agent/auth.json' +# Then edit on host (nullfs): +# nano /usr/local/bastille/jails/clawdie-controlplane/root/root/.pi/agent/auth.json +``` + +Format: + +```json +{ + "zai": { "type": "api_key", "key": "..." }, + "openrouter": { "type": "api_key", "key": "sk-or-v1-..." }, + "anthropic": { "type": "api_key", "key": "sk-ant-..." } +} +``` + +## Architecture + +``` +Telegram message + │ + ▼ +src/index.ts (main loop) + │ + ▼ +src/agent-runner.ts + │ spawns subprocess + ▼ +/opt/npm/bin/pi --print --provider zai --model GLM-5.1 ... + │ + ▼ +ZAI API (GLM-5.1) + │ + ▼ +pi stdout → agent-runner collects → Telegram reply +``` + +## Verify End-to-End + +```bash +# Check agent is running +sudo bastille cmd clawdie-controlplane service clawdie status + +# Watch logs while sending a test message on Telegram +tail -f logs/clawdie.log + +# Look for: "Spawning pi agent" then "pi agent completed" +grep -E 'Spawning pi|pi agent' logs/clawdie.log | tail -5 +``` diff --git a/.agent/skills/coding-agent/add/src/providers/anthropic.ts b/.agent/skills/coding-agent/add/src/providers/anthropic.ts new file mode 100644 index 0000000..4996a87 --- /dev/null +++ b/.agent/skills/coding-agent/add/src/providers/anthropic.ts @@ -0,0 +1,194 @@ +import Anthropic from '@anthropic-ai/sdk'; +import { + AIProvider, + ProviderOptions, + ProviderResult, + StreamChunk, + ProviderConfig, + ProviderError, + ProviderUnavailableError, + ProviderRateLimitError, + ToolDefinition, + ToolCall, +} from './provider.js'; + +export class AnthropicProvider implements AIProvider { + readonly name = 'Anthropic'; + readonly type = 'claude' as const; + + private client: Anthropic; + private config: ProviderConfig; + private defaultModel: string; + + constructor(config: ProviderConfig = {}) { + this.config = config; + this.defaultModel = config.model || 'claude-sonnet-4-20250514'; + + this.client = new Anthropic({ + apiKey: config.apiKey || process.env.ANTHROPIC_API_KEY, + baseURL: config.baseUrl, + timeout: config.timeout || 120000, + maxRetries: config.maxRetries || 2, + }); + } + + async execute( + prompt: string, + options?: ProviderOptions, + ): Promise<ProviderResult> { + if (!this.isAvailable()) { + throw new ProviderUnavailableError(this.name, 'API key not configured'); + } + + try { + const response = await this.client.messages.create({ + model: options?.model || this.defaultModel, + max_tokens: options?.maxTokens || 4096, + temperature: options?.temperature, + system: options?.systemPrompt, + stop_sequences: options?.stopSequences, + messages: [{ role: 'user', content: prompt }], + tools: options?.tools ? this.convertTools(options.tools) : undefined, + }); + + return this.parseResponse(response); + } catch (error) { + throw this.handleError(error); + } + } + + async *stream( + prompt: string, + options?: ProviderOptions, + ): AsyncIterable<StreamChunk> { + if (!this.isAvailable()) { + throw new ProviderUnavailableError(this.name, 'API key not configured'); + } + + try { + const stream = this.client.messages.stream({ + model: options?.model || this.defaultModel, + max_tokens: options?.maxTokens || 4096, + temperature: options?.temperature, + system: options?.systemPrompt, + stop_sequences: options?.stopSequences, + messages: [{ role: 'user', content: prompt }], + tools: options?.tools ? this.convertTools(options.tools) : undefined, + }); + + let inputTokens = 0; + let outputTokens = 0; + + for await (const event of stream) { + if ( + event.type === 'content_block_delta' && + event.delta.type === 'text_delta' + ) { + yield { type: 'text', delta: event.delta.text }; + } else if ( + event.type === 'content_block_start' && + event.content_block.type === 'tool_use' + ) { + yield { + type: 'tool_use', + toolCall: { + id: event.content_block.id, + name: event.content_block.name, + arguments: event.content_block.input as Record<string, unknown>, + }, + }; + } else if (event.type === 'message_start') { + inputTokens = event.message.usage.input_tokens; + } else if (event.type === 'message_delta') { + outputTokens = event.usage?.output_tokens || 0; + } + } + + yield { type: 'usage', usage: { inputTokens, outputTokens } }; + } catch (error) { + throw this.handleError(error); + } + } + + isAvailable(): boolean { + return !!( + this.config.apiKey || + process.env.ANTHROPIC_API_KEY || + process.env.AGENT_CODE_OAUTH_TOKEN + ); + } + + async getModels(): Promise<string[]> { + return [ + 'claude-opus-4-20250514', + 'claude-sonnet-4-20250514', + 'claude-haiku-3-5-20241022', + 'claude-3-5-sonnet-20241022', + 'claude-3-5-haiku-20241022', + 'claude-3-opus-20240229', + 'claude-3-sonnet-20240229', + 'claude-3-haiku-20240307', + ]; + } + + getDefaultModel(): string { + return this.defaultModel; + } + + private convertTools(tools: ToolDefinition[]): Anthropic.Tool[] { + return tools.map((tool) => ({ + name: tool.name, + description: tool.description, + input_schema: tool.parameters as Anthropic.Tool['input_schema'], + })); + } + + private parseResponse(response: Anthropic.Message): ProviderResult { + let content = ''; + const toolCalls: ToolCall[] = []; + + for (const block of response.content) { + if (block.type === 'text') { + content += block.text; + } else if (block.type === 'tool_use') { + toolCalls.push({ + id: block.id, + name: block.name, + arguments: block.input as Record<string, unknown>, + }); + } + } + + return { + content, + toolCalls: toolCalls.length > 0 ? toolCalls : undefined, + usage: { + inputTokens: response.usage.input_tokens, + outputTokens: response.usage.output_tokens, + }, + stopReason: response.stop_reason as ProviderResult['stopReason'], + }; + } + + private handleError(error: unknown): Error { + if (error instanceof Anthropic.APIError) { + if (error.status === 429) { + const retryAfter = error.headers?.['retry-after']; + return new ProviderRateLimitError( + this.name, + retryAfter ? parseInt(retryAfter, 10) : undefined, + ); + } + return new ProviderError( + error.message, + this.name, + error.code, + error.status, + ); + } + if (error instanceof Error) { + return new ProviderError(error.message, this.name); + } + return new ProviderError('Unknown error', this.name); + } +} diff --git a/.agent/skills/coding-agent/add/src/providers/coding-agent.ts b/.agent/skills/coding-agent/add/src/providers/coding-agent.ts new file mode 100644 index 0000000..ddf77ac --- /dev/null +++ b/.agent/skills/coding-agent/add/src/providers/coding-agent.ts @@ -0,0 +1,176 @@ +import { spawn, ChildProcess } from 'child_process'; +import { + AIProvider, + ProviderOptions, + ProviderResult, + StreamChunk, + ProviderConfig, + ProviderError, + ProviderUnavailableError, +} from './provider.js'; + +export class CodingAgentProvider implements AIProvider { + readonly name = 'coding-agent'; + readonly type = 'claude' as const; // coding-agent uses Claude by default + + private config: ProviderConfig; + private defaultModel: string; + private process: ChildProcess | null = null; + + constructor(config: ProviderConfig = {}) { + this.config = config; + this.defaultModel = config.model || 'claude-sonnet-4-20250514'; + } + + async execute( + prompt: string, + options?: ProviderOptions, + ): Promise<ProviderResult> { + if (!this.isAvailable()) { + throw new ProviderUnavailableError( + this.name, + 'coding-agent binary not found in PATH', + ); + } + + return new Promise((resolve, reject) => { + const args = this.buildArgs(prompt, options); + + this.process = spawn('coding-agent', args, { + cwd: this.config.baseUrl || process.cwd(), + env: { + ...process.env, + ANTHROPIC_API_KEY: + this.config.apiKey || process.env.ANTHROPIC_API_KEY, + AGENT_CODE_OAUTH_TOKEN: process.env.AGENT_CODE_OAUTH_TOKEN, + }, + }); + + let stdout = ''; + let stderr = ''; + + this.process.stdout?.on('data', (data) => { + stdout += data.toString(); + }); + + this.process.stderr?.on('data', (data) => { + stderr += data.toString(); + }); + + this.process.on('close', (code) => { + if (code === 0) { + resolve({ + content: stdout.trim(), + usage: undefined, // coding-agent doesn't report token usage + }); + } else { + reject( + new ProviderError( + `coding-agent exited with code ${code}: ${stderr}`, + this.name, + 'PROCESS_ERROR', + code ?? undefined, + ), + ); + } + }); + + this.process.on('error', (error) => { + reject( + new ProviderError( + `Failed to spawn coding-agent: ${error.message}`, + this.name, + ), + ); + }); + + // Send prompt via stdin + this.process.stdin?.write(prompt); + this.process.stdin?.end(); + }); + } + + async *stream( + prompt: string, + options?: ProviderOptions, + ): AsyncIterable<StreamChunk> { + // coding-agent doesn't support streaming in the same way + // Execute and yield the complete result + const result = await this.execute(prompt, options); + + // Yield text in chunks for consistency + const chunkSize = 100; + for (let i = 0; i < result.content.length; i += chunkSize) { + yield { + type: 'text', + delta: result.content.slice(i, i + chunkSize), + }; + } + + if (result.usage) { + yield { type: 'usage', usage: result.usage }; + } + } + + isAvailable(): boolean { + // Check if coding-agent binary exists + try { + const result = spawn('which', ['coding-agent'], { shell: true }); + return result.exitCode === 0; + } catch { + return false; + } + } + + async getModels(): Promise<string[]> { + // coding-agent uses Claude models + return [ + 'claude-opus-4-20250514', + 'claude-sonnet-4-20250514', + 'claude-haiku-3-5-20241022', + ]; + } + + getDefaultModel(): string { + return this.defaultModel; + } + + private buildArgs(prompt: string, options?: ProviderOptions): string[] { + const args: string[] = [ + '--non-interactive', + '--model', + options?.model || this.defaultModel, + ]; + + if (options?.systemPrompt) { + args.push('--system', options.systemPrompt); + } + + if (options?.maxTokens) { + args.push('--max-tokens', options.maxTokens.toString()); + } + + if (options?.temperature !== undefined) { + args.push('--temperature', options.temperature.toString()); + } + + // Read from stdin + args.push('--stdin'); + + return args; + } + + kill(): void { + if (this.process) { + this.process.kill(); + this.process = null; + } + } +} + +// Factory function for convenience +export function createCodingAgentProvider( + config?: ProviderConfig, +): CodingAgentProvider { + return new CodingAgentProvider(config); +} diff --git a/.agent/skills/coding-agent/add/src/providers/gemini.ts b/.agent/skills/coding-agent/add/src/providers/gemini.ts new file mode 100644 index 0000000..08602fe --- /dev/null +++ b/.agent/skills/coding-agent/add/src/providers/gemini.ts @@ -0,0 +1,147 @@ +import { GoogleGenerativeAI } from '@google/generative-ai'; +import { + AIProvider, + ProviderOptions, + ProviderResult, + StreamChunk, + ProviderConfig, + ProviderError, + ProviderUnavailableError, + ToolDefinition, + ToolCall, +} from './provider.js'; + +export class GeminiProvider implements AIProvider { + readonly name = 'Gemini'; + readonly type = 'gemini' as const; + + private client: GoogleGenerativeAI; + private config: ProviderConfig; + private defaultModel: string; + + constructor(config: ProviderConfig = {}) { + this.config = config; + this.defaultModel = config.model || process.env.GEMINI_MODEL || 'gemini-pro'; + + this.client = new GoogleGenerativeAI( + config.apiKey || process.env.GOOGLE_API_KEY || '' + ); + } + + async execute(prompt: string, options?: ProviderOptions): Promise<ProviderResult> { + if (!this.isAvailable()) { + throw new ProviderUnavailableError(this.name, 'API key not configured'); + } + + try { + const model = this.client.getGenerativeModel({ + model: options?.model || this.defaultModel, + }); + + const result = await model.generateContent({ + contents: [{ role: 'user', parts: [{ text: prompt }] }], + generationConfig: { + maxOutputTokens: options?.maxTokens || 4096, + temperature: options?.temperature, + stopSequences: options?.stopSequences, + }, + systemInstruction: options?.systemPrompt + ? { parts: [{ text: options.systemPrompt }] } + : undefined, + }); + + return this.parseResponse(result); + } catch (error) { + throw this.handleError(error); + } + } + + async *stream(prompt: string, options?: ProviderOptions): AsyncIterable<StreamChunk> { + if (!this.isAvailable()) { + throw new ProviderUnavailableError(this.name, 'API key not configured'); + } + + try { + const model = this.client.getGenerativeModel({ + model: options?.model || this.defaultModel, + }); + + const result = await model.generateContentStream({ + contents: [{ role: 'user', parts: [{ text: prompt }] }], + generationConfig: { + maxOutputTokens: options?.maxTokens || 4096, + temperature: options?.temperature, + stopSequences: options?.stopSequences, + }, + systemInstruction: options?.systemPrompt + ? { parts: [{ text: options.systemPrompt }] } + : undefined, + }); + + let inputTokens = 0; + let outputTokens = 0; + + for await (const chunk of result.stream) { + const text = chunk.text(); + if (text) { + yield { type: 'text', delta: text }; + } + + if (chunk.usageMetadata) { + inputTokens = chunk.usageMetadata.promptTokenCount; + outputTokens = chunk.usageMetadata.candidatesTokenCount; + } + } + + yield { type: 'usage', usage: { inputTokens, outputTokens } }; + } catch (error) { + throw this.handleError(error); + } + } + + isAvailable(): boolean { + return !!(this.config.apiKey || process.env.GOOGLE_API_KEY); + } + + async getModels(): Promise<string[]> { + return [ + 'gemini-pro', + 'gemini-pro-vision', + 'gemini-1.5-pro', + 'gemini-1.5-flash', + ]; + } + + getDefaultModel(): string { + return this.defaultModel; + } + + private parseResponse(result: Awaited<ReturnType<GoogleGenerativeAI['getGenerativeModel'] extends (...args: any) => any ? ReturnType<ReturnType<GoogleGenerativeAI['getGenerativeModel']>['generateContent']> : never>): ProviderResult { + const response = result.response; + const content = response.text(); + + return { + content, + usage: response.usageMetadata + ? { + inputTokens: response.usageMetadata.promptTokenCount, + outputTokens: response.usageMetadata.candidatesTokenCount, + } + : undefined, + stopReason: response.candidates?.[0]?.finishReason as ProviderResult['stopReason'], + }; + } + + private handleError(error: unknown): Error { + if (error instanceof Error) { + if (error.message.includes('429') || error.message.includes('quota')) { + return new ProviderError(error.message, this.name, 'RATE_LIMIT', 429); + } + return new ProviderError(error.message, this.name); + } + return new ProviderError('Unknown error', this.name); + } +} + +// Note: Tool use support in Gemini is more limited than Claude/OpenAI +// Full tool/function calling requires additional configuration diff --git a/.agent/skills/coding-agent/add/src/providers/index.ts b/.agent/skills/coding-agent/add/src/providers/index.ts new file mode 100644 index 0000000..7bd82f7 --- /dev/null +++ b/.agent/skills/coding-agent/add/src/providers/index.ts @@ -0,0 +1,143 @@ +import { + AIProvider, + ProviderType, + ExecutionMode, + ProviderConfig, + ProviderError, +} from './provider.js'; +import { AnthropicProvider } from './anthropic.js'; +import { OpenAIProvider } from './openai.js'; +import { GeminiProvider } from './gemini.js'; +import { CodingAgentProvider } from './coding-agent.js'; + +export { + ProviderError, + ProviderUnavailableError, + ProviderRateLimitError, + type AIProvider, + type ProviderOptions, + type ProviderResult, + type StreamChunk, + type ToolDefinition, + type ToolCall, + type ProviderType, + type ExecutionMode, + type ProviderConfig, +}; + +export { AnthropicProvider } from './anthropic.js'; +export { OpenAIProvider } from './openai.js'; +export { GeminiProvider } from './gemini.js'; +export { CodingAgentProvider } from './coding-agent.js'; + +const PROVIDERS: Record<ProviderType, new (config: ProviderConfig) => AIProvider> = { + claude: AnthropicProvider, + openai: OpenAIProvider, + gemini: GeminiProvider, +}; + +export function createProvider( + type: ProviderType, + config: ProviderConfig = {} +): AIProvider { + const ProviderClass = PROVIDERS[type]; + if (!ProviderClass) { + throw new ProviderError(`Unknown provider type: ${type}`, 'factory'); + } + return new ProviderClass(config); +} + +export function createProviderFromEnv(): AIProvider { + const providerType = (process.env.AI_PROVIDER || 'claude') as ProviderType; + + if (!['claude', 'openai', 'gemini'].includes(providerType)) { + throw new ProviderError( + `Invalid AI_PROVIDER: ${providerType}. Must be claude, openai, or gemini`, + 'factory' + ); + } + + const config: ProviderConfig = { + model: process.env.CLAUDE_MODEL || process.env.OPENAI_MODEL || process.env.GEMINI_MODEL, + }; + + switch (providerType) { + case 'claude': + config.apiKey = process.env.ANTHROPIC_API_KEY; + break; + case 'openai': + config.apiKey = process.env.OPENAI_API_KEY; + config.organizationId = process.env.OPENAI_ORG_ID; + break; + case 'gemini': + config.apiKey = process.env.GOOGLE_API_KEY; + break; + } + + return createProvider(providerType, config); +} + +export function createExecutionProvider( + mode: ExecutionMode = 'container', + providerType?: ProviderType +): AIProvider { + const actualMode = mode || (process.env.EXECUTION_MODE as ExecutionMode) || 'container'; + const actualType = providerType || (process.env.AI_PROVIDER as ProviderType) || 'claude'; + + switch (actualMode) { + case 'tui': + return new CodingAgentProvider({ + model: process.env.CLAUDE_MODEL, + }); + + case 'direct': + return createProviderFromEnv(); + + case 'container': + default: + // Container mode uses the provider config passed to the container + return createProvider(actualType, { + apiKey: process.env.ANTHROPIC_API_KEY || process.env.OPENAI_API_KEY || process.env.GOOGLE_API_KEY, + model: process.env.CLAUDE_MODEL || process.env.OPENAI_MODEL || process.env.GEMINI_MODEL, + }); + } +} + +export function getAvailableProviders(): ProviderType[] { + const available: ProviderType[] = []; + + if (process.env.ANTHROPIC_API_KEY || process.env.AGENT_CODE_OAUTH_TOKEN) { + available.push('claude'); + } + + if (process.env.OPENAI_API_KEY) { + available.push('openai'); + } + + if (process.env.GOOGLE_API_KEY) { + available.push('gemini'); + } + + return available; +} + +export function getDefaultProvider(): ProviderType { + const available = getAvailableProviders(); + + if (available.includes('claude')) { + return 'claude'; + } + + if (available.includes('openai')) { + return 'openai'; + } + + if (available.includes('gemini')) { + return 'gemini'; + } + + throw new ProviderError( + 'No AI provider configured. Set ANTHROPIC_API_KEY, OPENAI_API_KEY, or GOOGLE_API_KEY', + 'factory' + ); +} diff --git a/.agent/skills/coding-agent/add/src/providers/openai.ts b/.agent/skills/coding-agent/add/src/providers/openai.ts new file mode 100644 index 0000000..c5ca8da --- /dev/null +++ b/.agent/skills/coding-agent/add/src/providers/openai.ts @@ -0,0 +1,225 @@ +import OpenAI from 'openai'; +import { + AIProvider, + ProviderOptions, + ProviderResult, + StreamChunk, + ProviderConfig, + ProviderError, + ProviderUnavailableError, + ProviderRateLimitError, + ToolDefinition, + ToolCall, +} from './provider.js'; + +export class OpenAIProvider implements AIProvider { + readonly name = 'OpenAI'; + readonly type = 'openai' as const; + + private client: OpenAI; + private config: ProviderConfig; + private defaultModel: string; + + constructor(config: ProviderConfig = {}) { + this.config = config; + this.defaultModel = config.model || process.env.OPENAI_MODEL || 'gpt-4o'; + + this.client = new OpenAI({ + apiKey: config.apiKey || process.env.OPENAI_API_KEY, + baseURL: config.baseUrl, + organization: config.organizationId || process.env.OPENAI_ORG_ID, + timeout: config.timeout || 120000, + maxRetries: config.maxRetries || 2, + }); + } + + async execute( + prompt: string, + options?: ProviderOptions, + ): Promise<ProviderResult> { + if (!this.isAvailable()) { + throw new ProviderUnavailableError(this.name, 'API key not configured'); + } + + try { + const response = await this.client.chat.completions.create({ + model: options?.model || this.defaultModel, + max_tokens: options?.maxTokens || 4096, + temperature: options?.temperature, + messages: options?.systemPrompt + ? [ + { role: 'system', content: options.systemPrompt }, + { role: 'user', content: prompt }, + ] + : [{ role: 'user', content: prompt }], + tools: options?.tools ? this.convertTools(options.tools) : undefined, + stop: options?.stopSequences, + }); + + return this.parseResponse(response); + } catch (error) { + throw this.handleError(error); + } + } + + async *stream( + prompt: string, + options?: ProviderOptions, + ): AsyncIterable<StreamChunk> { + if (!this.isAvailable()) { + throw new ProviderUnavailableError(this.name, 'API key not configured'); + } + + try { + const stream = await this.client.chat.completions.create({ + model: options?.model || this.defaultModel, + max_tokens: options?.maxTokens || 4096, + temperature: options?.temperature, + messages: options?.systemPrompt + ? [ + { role: 'system', content: options.systemPrompt }, + { role: 'user', content: prompt }, + ] + : [{ role: 'user', content: prompt }], + tools: options?.tools ? this.convertTools(options.tools) : undefined, + stop: options?.stopSequences, + stream: true, + }); + + let inputTokens = 0; + let outputTokens = 0; + + for await (const chunk of stream) { + const delta = chunk.choices[0]?.delta; + + if (delta?.content) { + yield { type: 'text', delta: delta.content }; + } + + if (delta?.tool_calls) { + for (const toolCall of delta.tool_calls) { + if (toolCall.function?.name) { + yield { + type: 'tool_use', + toolCall: { + id: toolCall.id || '', + name: toolCall.function.name, + arguments: JSON.parse(toolCall.function.arguments || '{}'), + }, + }; + } + } + } + + if (chunk.usage) { + inputTokens = chunk.usage.prompt_tokens; + outputTokens = chunk.usage.completion_tokens; + } + } + + yield { type: 'usage', usage: { inputTokens, outputTokens } }; + } catch (error) { + throw this.handleError(error); + } + } + + isAvailable(): boolean { + return !!(this.config.apiKey || process.env.OPENAI_API_KEY); + } + + async getModels(): Promise<string[]> { + return [ + 'gpt-4o', + 'gpt-4o-mini', + 'gpt-4-turbo', + 'gpt-4', + 'gpt-3.5-turbo', + 'o1-preview', + 'o1-mini', + ]; + } + + getDefaultModel(): string { + return this.defaultModel; + } + + private convertTools( + tools: ToolDefinition[], + ): OpenAI.Chat.ChatCompletionTool[] { + return tools.map((tool) => ({ + type: 'function' as const, + function: { + name: tool.name, + description: tool.description, + parameters: tool.parameters, + }, + })); + } + + private parseResponse(response: OpenAI.Chat.ChatCompletion): ProviderResult { + const choice = response.choices[0]; + let content = ''; + const toolCalls: ToolCall[] = []; + + if (choice.message.content) { + content = choice.message.content; + } + + if (choice.message.tool_calls) { + for (const toolCall of choice.message.tool_calls) { + toolCalls.push({ + id: toolCall.id, + name: toolCall.function.name, + arguments: JSON.parse(toolCall.function.arguments), + }); + } + } + + return { + content, + toolCalls: toolCalls.length > 0 ? toolCalls : undefined, + usage: response.usage + ? { + inputTokens: response.usage.prompt_tokens, + outputTokens: response.usage.completion_tokens, + } + : undefined, + stopReason: this.mapStopReason(choice.finish_reason), + }; + } + + private mapStopReason(reason: string | null): ProviderResult['stopReason'] { + switch (reason) { + case 'stop': + return 'end_turn'; + case 'length': + return 'max_tokens'; + case 'tool_calls': + return 'tool_use'; + default: + return 'end_turn'; + } + } + + private handleError(error: unknown): Error { + if (error instanceof OpenAI.APIError) { + if (error.status === 429) { + const retryAfter = error.headers?.['retry-after']; + return new ProviderRateLimitError( + this.name, + retryAfter ? parseInt(retryAfter, 10) : undefined, + ); + } + return new ProviderError( + error.message, + this.name, + error.code, + error.status, + ); + } + if (error instanceof Error) { + return new ProviderError(error.message, this.name); + } + return new ProviderError('Unknown error', this.name); + } +} diff --git a/.agent/skills/coding-agent/add/src/providers/provider.ts b/.agent/skills/coding-agent/add/src/providers/provider.ts new file mode 100644 index 0000000..b8cb8d4 --- /dev/null +++ b/.agent/skills/coding-agent/add/src/providers/provider.ts @@ -0,0 +1,91 @@ +export interface ToolDefinition { + name: string; + description: string; + parameters: Record<string, unknown>; +} + +export interface ToolCall { + id: string; + name: string; + arguments: Record<string, unknown>; +} + +export interface ProviderOptions { + model?: string; + maxTokens?: number; + temperature?: number; + systemPrompt?: string; + tools?: ToolDefinition[]; + stopSequences?: string[]; +} + +export interface ProviderResult { + content: string; + toolCalls?: ToolCall[]; + usage?: { + inputTokens: number; + outputTokens: number; + }; + stopReason?: 'end_turn' | 'max_tokens' | 'stop_sequence' | 'tool_use'; +} + +export interface StreamChunk { + type: 'text' | 'tool_use' | 'usage'; + delta?: string; + toolCall?: ToolCall; + usage?: { inputTokens: number; outputTokens: number }; +} + +export type ProviderType = 'claude' | 'openai' | 'gemini'; +export type ExecutionMode = 'container' | 'tui' | 'direct'; + +export interface AIProvider { + readonly name: string; + readonly type: ProviderType; + + execute(prompt: string, options?: ProviderOptions): Promise<ProviderResult>; + stream(prompt: string, options?: ProviderOptions): AsyncIterable<StreamChunk>; + isAvailable(): boolean; + getModels(): Promise<string[]>; + getDefaultModel(): string; +} + +export interface ProviderConfig { + apiKey?: string; + model?: string; + baseUrl?: string; + organizationId?: string; + timeout?: number; + maxRetries?: number; +} + +export class ProviderError extends Error { + constructor( + message: string, + public readonly provider: string, + public readonly code?: string, + public readonly statusCode?: number + ) { + super(message); + this.name = 'ProviderError'; + } +} + +export class ProviderUnavailableError extends ProviderError { + constructor(provider: string, reason: string) { + super(`Provider ${provider} unavailable: ${reason}`, provider, 'UNAVAILABLE'); + this.name = 'ProviderUnavailableError'; + } +} + +export class ProviderRateLimitError extends ProviderError { + constructor(provider: string, retryAfter?: number) { + super( + `Provider ${provider} rate limited${retryAfter ? `, retry after ${retryAfter}s` : ''}`, + provider, + 'RATE_LIMIT', + 429 + ); + this.name = 'ProviderRateLimitError'; + } +} diff --git a/.agent/skills/coding-agent/manifest.yaml b/.agent/skills/coding-agent/manifest.yaml new file mode 100644 index 0000000..0cd22c0 --- /dev/null +++ b/.agent/skills/coding-agent/manifest.yaml @@ -0,0 +1,36 @@ +skill: coding-agent +version: 1.0.0 +description: 'Provider abstraction layer for AI providers (Claude, OpenAI, Gemini) and TUI integration via coding-agent' +core_version: 0.6.0 +adds: + - src/providers/provider.ts + - src/providers/anthropic.ts + - src/providers/openai.ts + - src/providers/gemini.ts + - src/providers/coding-agent.ts + - src/providers/index.ts + - src/providers/provider.test.ts +modifies: + - src/config.ts + - src/container-runner.ts + - .env.example +structured: + npm_dependencies: + '@anthropic-ai/sdk': '^0.39.0' + 'openai': '^4.90.0' + '@google/generative-ai': '^0.21.0' + env_additions: + - AI_PROVIDER + - EXECUTION_MODE + - ANTHROPIC_API_KEY + - CLAUDE_MODEL + - OPENAI_API_KEY + - OPENAI_MODEL + - OPENAI_ORG_ID + - GOOGLE_API_KEY + - GEMINI_MODEL + - FALLBACK_PROVIDER + - FALLBACK_MODEL +conflicts: [] +depends: [] +test: 'npx vitest run src/providers/provider.test.ts' diff --git a/.agent/skills/coding-agent/modify/.env.example b/.agent/skills/coding-agent/modify/.env.example new file mode 100644 index 0000000..11267c9 --- /dev/null +++ b/.agent/skills/coding-agent/modify/.env.example @@ -0,0 +1,60 @@ +# Clawdie Environment Configuration + +# === Assistant Configuration === +ASSISTANT_NAME=Clawdie + +# === Provider Selection === +# Which AI provider to use: claude, openai, or gemini +AI_PROVIDER=claude + +# How to execute requests: container (default), tui, or direct +EXECUTION_MODE=container + +# === Claude/Anthropic Configuration === +# Get API key from https://console.anthropic.com/ +# Or use OAuth token from: claude setup-token +ANTHROPIC_API_KEY=sk-ant-your-key-here +# AGENT_CODE_OAUTH_TOKEN=your-oauth-token + +# Claude model selection +CLAUDE_MODEL=claude-sonnet-4-20250514 + +# === OpenAI Configuration === +# Get API key from https://platform.openai.com/api-keys +# OPENAI_API_KEY=sk-your-key-here +# OPENAI_ORG_ID=org-your-org-id + +# OpenAI model selection +# OPENAI_MODEL=gpt-4o + +# === Gemini Configuration === +# Get API key from https://aistudio.google.com/app/apikey +# GOOGLE_API_KEY=your-key-here + +# Gemini model selection +# GEMINI_MODEL=gemini-pro + +# === Fallback Provider (Optional) === +# If primary provider fails, use this provider +# FALLBACK_PROVIDER=openai +# FALLBACK_MODEL=gpt-4o + +# === Telegram Configuration === +# Get bot token from @BotFather on Telegram +TELEGRAM_BOT_TOKEN=your-bot-token-here + +# Set to true if Telegram should be the only channel +TELEGRAM_ONLY=true + +# === Container Configuration === +CONTAINER_IMAGE=clawdie-cp-agent:latest +CONTAINER_TIMEOUT=1800000 +CONTAINER_MAX_OUTPUT_SIZE=10485760 +MAX_CONCURRENT_CONTAINERS=5 + +# === Other Settings === +# Timezone for scheduled tasks +# TZ=Europe/Ljubljana + +# Idle timeout for containers (30 min default) +IDLE_TIMEOUT=1800000 diff --git a/.agent/skills/coding-agent/modify/src/config.ts b/.agent/skills/coding-agent/modify/src/config.ts new file mode 100644 index 0000000..66c2432 --- /dev/null +++ b/.agent/skills/coding-agent/modify/src/config.ts @@ -0,0 +1,121 @@ +import os from 'os'; +import path from 'path'; + +import { readEnvFile } from './env.js'; + +// Read config values from .env (falls back to process.env). +// Secrets are NOT read here — they stay on disk and are loaded only +// where needed (container-runner.ts) to avoid leaking to child processes. +const envConfig = readEnvFile(['ASSISTANT_NAME', 'ASSISTANT_HAS_OWN_NUMBER']); + +export const ASSISTANT_NAME = + process.env.ASSISTANT_NAME || envConfig.ASSISTANT_NAME || 'Andy'; +export const ASSISTANT_HAS_OWN_NUMBER = + (process.env.ASSISTANT_HAS_OWN_NUMBER || + envConfig.ASSISTANT_HAS_OWN_NUMBER) === 'true'; +export const POLL_INTERVAL = 2000; +export const SCHEDULER_POLL_INTERVAL = 60000; + +// Absolute paths needed for container mounts +const PROJECT_ROOT = process.cwd(); +const HOME_DIR = process.env.HOME || os.homedir(); + +// Mount security: allowlist stored OUTSIDE project root, never mounted into containers +export const MOUNT_ALLOWLIST_PATH = path.join( + HOME_DIR, + '.config', + (process.env.AGENT_NAME || 'clawdie') + '-cp', + 'mount-allowlist.json', +); +export const STORE_DIR = path.resolve(PROJECT_ROOT, 'store'); +export const GROUPS_DIR = path.resolve(PROJECT_ROOT, 'groups'); +export const DATA_DIR = path.resolve(PROJECT_ROOT, 'data'); +export const MAIN_GROUP_FOLDER = 'main'; + +export const CONTAINER_IMAGE = + process.env.CONTAINER_IMAGE || (process.env.AGENT_NAME || 'clawdie') + '-cp-agent:latest'; +export const CONTAINER_TIMEOUT = parseInt( + process.env.CONTAINER_TIMEOUT || '1800000', + 10, +); +export const CONTAINER_MAX_OUTPUT_SIZE = parseInt( + process.env.CONTAINER_MAX_OUTPUT_SIZE || '10485760', + 10, +); // 10MB default +export const IPC_POLL_INTERVAL = 1000; +export const IDLE_TIMEOUT = parseInt(process.env.IDLE_TIMEOUT || '1800000', 10); // 30min default — how long to keep container alive after last result +export const MAX_CONCURRENT_CONTAINERS = Math.max( + 1, + parseInt(process.env.MAX_CONCURRENT_CONTAINERS || '5', 10) || 5, +); + +function escapeRegex(str: string): string { + return str.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); +} + +export const TRIGGER_PATTERN = new RegExp( + `^@${escapeRegex(ASSISTANT_NAME)}\\b`, + 'i', +); + +// Timezone for scheduled tasks (cron expressions, etc.) +// Uses system timezone by default +export const TIMEZONE = + process.env.TZ || Intl.DateTimeFormat().resolvedOptions().timeZone; + +// === Provider Configuration === +// These settings control which AI provider to use and how to execute requests + +export type ProviderType = 'claude' | 'openai' | 'gemini'; +export type ExecutionMode = 'container' | 'tui' | 'direct'; + +export const AI_PROVIDER = (process.env.AI_PROVIDER || + 'claude') as ProviderType; +export const EXECUTION_MODE = (process.env.EXECUTION_MODE || + 'container') as ExecutionMode; + +// Claude/Anthropic configuration +export const CLAUDE_MODEL = + process.env.CLAUDE_MODEL || 'claude-sonnet-4-20250514'; + +// OpenAI configuration +export const OPENAI_MODEL = process.env.OPENAI_MODEL || 'gpt-4o'; + +// Gemini configuration +export const GEMINI_MODEL = process.env.GEMINI_MODEL || 'gemini-pro'; + +// Fallback provider (optional) +export const FALLBACK_PROVIDER = process.env.FALLBACK_PROVIDER as + | ProviderType + | undefined; +export const FALLBACK_MODEL = process.env.FALLBACK_MODEL; + +// Validate provider type +if (!['claude', 'openai', 'gemini'].includes(AI_PROVIDER)) { + console.warn( + `Warning: Invalid AI_PROVIDER "${AI_PROVIDER}", defaulting to "claude"`, + ); +} + +// Validate execution mode +if (!['container', 'tui', 'direct'].includes(EXECUTION_MODE)) { + console.warn( + `Warning: Invalid EXECUTION_MODE "${EXECUTION_MODE}", defaulting to "container"`, + ); +} + +// Get the effective model for the current provider +export function getModelForProvider( + provider: ProviderType = AI_PROVIDER, +): string { + switch (provider) { + case 'claude': + return CLAUDE_MODEL; + case 'openai': + return OPENAI_MODEL; + case 'gemini': + return GEMINI_MODEL; + default: + return CLAUDE_MODEL; + } +} diff --git a/.agent/skills/coding-agent/modify/src/config.ts.intent.md b/.agent/skills/coding-agent/modify/src/config.ts.intent.md new file mode 100644 index 0000000..3917f1d --- /dev/null +++ b/.agent/skills/coding-agent/modify/src/config.ts.intent.md @@ -0,0 +1,72 @@ +# src/config.ts Modification Intent + +## What Changed + +Added provider configuration settings to enable multi-provider support (Claude, OpenAI, Gemini) and execution mode selection (container, TUI, direct API). + +## Changes Made + +1. **Type Definitions** + - Added `ProviderType` type: `'claude' | 'openai' | 'gemini'` + - Added `ExecutionMode` type: `'container' | 'tui' | 'direct'` + +2. **New Configuration Exports** + - `AI_PROVIDER` - Which AI provider to use (default: 'claude') + - `EXECUTION_MODE` - How to execute requests (default: 'container') + - `CLAUDE_MODEL` - Claude model name (default: 'claude-sonnet-4-20250514') + - `OPENAI_MODEL` - OpenAI model name (default: 'gpt-4o') + - `GEMINI_MODEL` - Gemini model name (default: 'gemini-pro') + - `FALLBACK_PROVIDER` - Optional secondary provider + - `FALLBACK_MODEL` - Optional fallback model + +3. **Helper Function** + - `getModelForProvider(provider)` - Returns the appropriate model name for a given provider + +4. **Validation** + - Warns if invalid `AI_PROVIDER` is set + - Warns if invalid `EXECUTION_MODE` is set + +## Invariants + +1. **Backwards Compatibility**: All existing exports remain unchanged. New settings are additive. + +2. **Default Values**: If environment variables are not set, sensible defaults are used: + - Provider defaults to 'claude' (existing behavior) + - Mode defaults to 'container' (existing behavior) + +3. **No Secret Exposure**: API keys are NOT read here. They remain in environment and are read only where needed (container-runner.ts, providers). + +4. **Type Safety**: Provider and mode types are exported for use throughout the codebase. + +## Environment Variables + +```bash +# Provider Selection +AI_PROVIDER=claude # claude | openai | gemini +EXECUTION_MODE=container # container | tui | direct + +# Model Selection (provider-specific) +CLAUDE_MODEL=claude-sonnet-4-20250514 +OPENAI_MODEL=gpt-4o +GEMINI_MODEL=gemini-pro + +# Fallback (optional) +FALLBACK_PROVIDER=openai +FALLBACK_MODEL=gpt-4o +``` + +## Usage + +```typescript +import { AI_PROVIDER, EXECUTION_MODE, getModelForProvider } from './config.js'; + +const model = getModelForProvider(AI_PROVIDER); +console.log(`Using ${AI_PROVIDER} with model ${model} in ${EXECUTION_MODE} mode`); +``` + +## Testing Considerations + +- Test with each provider type +- Test with missing environment variables +- Test fallback logic +- Verify warnings are logged for invalid values diff --git a/.agent/skills/coding-agent/modify/src/container-runner.ts b/.agent/skills/coding-agent/modify/src/container-runner.ts new file mode 100644 index 0000000..2659e99 --- /dev/null +++ b/.agent/skills/coding-agent/modify/src/container-runner.ts @@ -0,0 +1,17 @@ +# Placeholder for container-runner.ts modifications +# The actual three-way merge will be handled by the skills engine +# +# Key additions to make in src/container-runner.ts: +# +# 1. Import provider system: +# import { createExecutionProvider, AIProvider } from './providers/index.js'; +# import { AI_PROVIDER, EXECUTION_MODE } from './config.js'; +# +# 2. Add provider instance to container context +# 3. Modify runInContainer() to check EXECUTION_MODE: +# - If 'direct': Use provider.execute() directly +# - If 'tui': Use CodingAgentProvider +# - If 'container': Use existing container logic +# +# 4. Pass provider config to container environment +# 5. Add fallback provider logic if FALLBACK_PROVIDER is set diff --git a/.agent/skills/coding-agent/modify/src/container-runner.ts.intent.md b/.agent/skills/coding-agent/modify/src/container-runner.ts.intent.md new file mode 100644 index 0000000..966d0a8 --- /dev/null +++ b/.agent/skills/coding-agent/modify/src/container-runner.ts.intent.md @@ -0,0 +1,195 @@ +# src/container-runner.ts Modification Intent + +## What Changed + +Modified the container runner to support multiple execution modes (container, TUI, direct API) and provider abstraction. + +## Changes Made + +### 1. New Imports + +```typescript +import { + createExecutionProvider, + AIProvider, + ProviderError, + ProviderUnavailableError, +} from './providers/index.js'; +import { + AI_PROVIDER, + EXECUTION_MODE, + getModelForProvider, + FALLBACK_PROVIDER, + FALLBACK_MODEL, +} from './config.js'; +``` + +### 2. Provider Instance Caching + +```typescript +let cachedProvider: AIProvider | null = null; + +function getProvider(): AIProvider { + if (!cachedProvider) { + cachedProvider = createExecutionProvider(EXECUTION_MODE, AI_PROVIDER); + } + return cachedProvider; +} +``` + +### 3. Modified runInContainer() + +The function now checks `EXECUTION_MODE` before deciding how to execute: + +```typescript +export async function runInContainer( + input: ContainerInput, + group: RegisteredGroup, +): Promise<ContainerOutput> { + switch (EXECUTION_MODE) { + case 'direct': + return runDirect(input); + case 'tui': + return runWithTUI(input); + case 'container': + default: + return runInContainerProcess(input, group); + } +} +``` + +### 4. Direct Execution Mode + +```typescript +async function runDirect(input: ContainerInput): Promise<ContainerOutput> { + try { + const provider = getProvider(); + const result = await provider.execute(input.prompt, { + model: getModelForProvider(), + systemPrompt: buildSystemPrompt(input), + }); + + return { + status: 'success', + result: result.content, + }; + } catch (error) { + if (error instanceof ProviderError) { + // Try fallback provider if configured + if (FALLBACK_PROVIDER) { + return runWithFallback(input, error); + } + } + return { + status: 'error', + error: error instanceof Error ? error.message : 'Unknown error', + }; + } +} +``` + +### 5. TUI Execution Mode + +```typescript +async function runWithTUI(input: ContainerInput): Promise<ContainerOutput> { + const provider = new CodingAgentProvider({ + model: getModelForProvider(), + }); + + // TUI mode spawns coding-agent process + return runDirect(input); // Same logic, different provider +} +``` + +### 6. Fallback Provider Logic + +```typescript +async function runWithFallback( + input: ContainerInput, + originalError: ProviderError, +): Promise<ContainerOutput> { + logger.warn( + `Primary provider failed: ${originalError.message}. Trying fallback...`, + ); + + const fallbackProvider = createExecutionProvider( + EXECUTION_MODE, + FALLBACK_PROVIDER, + ); + + try { + const result = await fallbackProvider.execute(input.prompt, { + model: FALLBACK_MODEL || getModelForProvider(FALLBACK_PROVIDER), + systemPrompt: buildSystemPrompt(input), + }); + + return { + status: 'success', + result: result.content, + }; + } catch (fallbackError) { + logger.error(`Fallback provider also failed: ${fallbackError}`); + return { + status: 'error', + error: originalError.message, + }; + } +} +``` + +### 7. Environment Variable Passing + +When `EXECUTION_MODE=container`, pass provider config to container: + +```typescript +const envVars: Record<string, string> = { + // ... existing vars + AI_PROVIDER, + EXECUTION_MODE, + CLAUDE_MODEL, + OPENAI_MODEL, + GEMINI_MODEL, +}; +``` + +## Invariants + +1. **Backwards Compatibility**: Existing container-based execution is the default. No behavior change unless `EXECUTION_MODE` is explicitly set. + +2. **Error Handling**: All execution modes return the same `ContainerOutput` type for consistency. + +3. **Logging**: Mode selection and fallbacks are logged for debugging. + +4. **Session Management**: Direct and TUI modes don't support session persistence (no sessionId returned). + +5. **Security**: API keys are only passed to containers via environment, never logged. + +## Execution Mode Comparison + +| Mode | Isolation | Overhead | Features | +| --------- | ------------------ | -------- | ------------------------------------- | +| container | Full (jail/Docker) | Moderate | All features, session persistence | +| tui | Process isolation | Low | Interactive debugging, no persistence | +| direct | None | Minimal | Fastest, no persistence, no isolation | + +## Testing Considerations + +1. Test each execution mode +2. Test fallback provider switching +3. Test error propagation +4. Test session handling differences +5. Test environment variable passing + +## Migration Path + +Existing deployments: + +1. No changes required - defaults to 'container' mode +2. To use TUI mode: Set `EXECUTION_MODE=tui` and install coding-agent +3. To use direct API: Set `EXECUTION_MODE=direct` + +New deployments: + +1. Choose execution mode based on requirements +2. Set appropriate provider API keys +3. Configure fallback if desired diff --git a/.agent/skills/coding-agent/tests/provider.test.ts b/.agent/skills/coding-agent/tests/provider.test.ts new file mode 100644 index 0000000..9ea088f --- /dev/null +++ b/.agent/skills/coding-agent/tests/provider.test.ts @@ -0,0 +1,283 @@ +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; +import { + AIProvider, + ProviderOptions, + ProviderResult, + ProviderError, + ProviderUnavailableError, +} from '../add/src/providers/provider.js'; +import { AnthropicProvider } from '../add/src/providers/anthropic.js'; +import { OpenAIProvider } from '../add/src/providers/openai.js'; +import { GeminiProvider } from '../add/src/providers/gemini.js'; +import { + createProvider, + createProviderFromEnv, + getAvailableProviders, + getDefaultProvider, +} from '../add/src/providers/index.js'; + +describe('Provider Interface', () => { + it('should define required methods', () => { + const mockProvider: AIProvider = { + name: 'mock', + type: 'claude', + execute: vi.fn(), + stream: vi.fn(), + isAvailable: vi.fn(), + getModels: vi.fn(), + getDefaultModel: vi.fn(), + }; + + expect(mockProvider.name).toBe('mock'); + expect(mockProvider.type).toBe('claude'); + expect(typeof mockProvider.execute).toBe('function'); + expect(typeof mockProvider.stream).toBe('function'); + expect(typeof mockProvider.isAvailable).toBe('function'); + expect(typeof mockProvider.getModels).toBe('function'); + expect(typeof mockProvider.getDefaultModel).toBe('function'); + }); +}); + +describe('AnthropicProvider', () => { + let originalEnv: NodeJS.ProcessEnv; + + beforeEach(() => { + originalEnv = { ...process.env }; + delete process.env.ANTHROPIC_API_KEY; + delete process.env.AGENT_CODE_OAUTH_TOKEN; + }); + + afterEach(() => { + process.env = originalEnv; + }); + + it('should not be available without API key', () => { + const provider = new AnthropicProvider(); + expect(provider.isAvailable()).toBe(false); + }); + + it('should be available with API key', () => { + process.env.ANTHROPIC_API_KEY = 'sk-ant-test'; + const provider = new AnthropicProvider(); + expect(provider.isAvailable()).toBe(true); + }); + + it('should be available with OAuth token', () => { + process.env.AGENT_CODE_OAUTH_TOKEN = 'test-token'; + const provider = new AnthropicProvider(); + expect(provider.isAvailable()).toBe(true); + }); + + it('should return correct default model', () => { + const provider = new AnthropicProvider(); + expect(provider.getDefaultModel()).toBe('claude-sonnet-4-20250514'); + }); + + it('should return list of models', async () => { + const provider = new AnthropicProvider(); + const models = await provider.getModels(); + expect(models).toContain('claude-sonnet-4-20250514'); + expect(models).toContain('claude-opus-4-20250514'); + }); + + it('should throw when not available', async () => { + const provider = new AnthropicProvider(); + await expect(provider.execute('test')).rejects.toThrow( + ProviderUnavailableError, + ); + }); +}); + +describe('OpenAIProvider', () => { + let originalEnv: NodeJS.ProcessEnv; + + beforeEach(() => { + originalEnv = { ...process.env }; + delete process.env.OPENAI_API_KEY; + }); + + afterEach(() => { + process.env = originalEnv; + }); + + it('should not be available without API key', () => { + const provider = new OpenAIProvider(); + expect(provider.isAvailable()).toBe(false); + }); + + it('should be available with API key', () => { + process.env.OPENAI_API_KEY = 'sk-test'; + const provider = new OpenAIProvider(); + expect(provider.isAvailable()).toBe(true); + }); + + it('should return correct default model', () => { + const provider = new OpenAIProvider(); + expect(provider.getDefaultModel()).toBe('gpt-4o'); + }); + + it('should return list of models', async () => { + const provider = new OpenAIProvider(); + const models = await provider.getModels(); + expect(models).toContain('gpt-4o'); + expect(models).toContain('gpt-4-turbo'); + }); + + it('should throw when not available', async () => { + const provider = new OpenAIProvider(); + await expect(provider.execute('test')).rejects.toThrow( + ProviderUnavailableError, + ); + }); +}); + +describe('GeminiProvider', () => { + let originalEnv: NodeJS.ProcessEnv; + + beforeEach(() => { + originalEnv = { ...process.env }; + delete process.env.GOOGLE_API_KEY; + }); + + afterEach(() => { + process.env = originalEnv; + }); + + it('should not be available without API key', () => { + const provider = new GeminiProvider(); + expect(provider.isAvailable()).toBe(false); + }); + + it('should be available with API key', () => { + process.env.GOOGLE_API_KEY = 'test-key'; + const provider = new GeminiProvider(); + expect(provider.isAvailable()).toBe(true); + }); + + it('should return correct default model', () => { + const provider = new GeminiProvider(); + expect(provider.getDefaultModel()).toBe('gemini-pro'); + }); + + it('should return list of models', async () => { + const provider = new GeminiProvider(); + const models = await provider.getModels(); + expect(models).toContain('gemini-pro'); + expect(models).toContain('gemini-1.5-pro'); + }); + + it('should throw when not available', async () => { + const provider = new GeminiProvider(); + await expect(provider.execute('test')).rejects.toThrow( + ProviderUnavailableError, + ); + }); +}); + +describe('Provider Factory', () => { + let originalEnv: NodeJS.ProcessEnv; + + beforeEach(() => { + originalEnv = { ...process.env }; + delete process.env.ANTHROPIC_API_KEY; + delete process.env.OPENAI_API_KEY; + delete process.env.GOOGLE_API_KEY; + delete process.env.AI_PROVIDER; + }); + + afterEach(() => { + process.env = originalEnv; + }); + + it('should create provider by type', () => { + const claude = createProvider('claude'); + expect(claude.name).toBe('Anthropic'); + expect(claude.type).toBe('claude'); + + const openai = createProvider('openai'); + expect(openai.name).toBe('OpenAI'); + expect(openai.type).toBe('openai'); + + const gemini = createProvider('gemini'); + expect(gemini.name).toBe('Gemini'); + expect(gemini.type).toBe('gemini'); + }); + + it('should get available providers', () => { + expect(getAvailableProviders()).toEqual([]); + + process.env.ANTHROPIC_API_KEY = 'sk-ant-test'; + expect(getAvailableProviders()).toEqual(['claude']); + + process.env.OPENAI_API_KEY = 'sk-test'; + expect(getAvailableProviders()).toEqual(['claude', 'openai']); + + process.env.GOOGLE_API_KEY = 'test-key'; + expect(getAvailableProviders()).toEqual(['claude', 'openai', 'gemini']); + }); + + it('should get default provider', () => { + process.env.ANTHROPIC_API_KEY = 'sk-ant-test'; + expect(getDefaultProvider()).toBe('claude'); + + delete process.env.ANTHROPIC_API_KEY; + process.env.OPENAI_API_KEY = 'sk-test'; + expect(getDefaultProvider()).toBe('openai'); + + delete process.env.OPENAI_API_KEY; + process.env.GOOGLE_API_KEY = 'test-key'; + expect(getDefaultProvider()).toBe('gemini'); + }); + + it('should throw when no providers available', () => { + expect(() => getDefaultProvider()).toThrow(ProviderError); + }); + + it('should create provider from environment', () => { + process.env.ANTHROPIC_API_KEY = 'sk-ant-test'; + process.env.AI_PROVIDER = 'claude'; + + const provider = createProviderFromEnv(); + expect(provider.name).toBe('Anthropic'); + expect(provider.type).toBe('claude'); + }); + + it('should throw on invalid provider type', () => { + process.env.AI_PROVIDER = 'invalid'; + expect(() => createProviderFromEnv()).toThrow(ProviderError); + }); +}); + +describe('ProviderError', () => { + it('should create error with all fields', () => { + const error = new ProviderError('Test error', 'claude', 'TEST_CODE', 500); + expect(error.message).toBe('Test error'); + expect(error.provider).toBe('claude'); + expect(error.code).toBe('TEST_CODE'); + expect(error.statusCode).toBe(500); + expect(error.name).toBe('ProviderError'); + }); + + it('should create error with minimal fields', () => { + const error = new ProviderError('Test error', 'claude'); + expect(error.message).toBe('Test error'); + expect(error.provider).toBe('claude'); + expect(error.code).toBeUndefined(); + expect(error.statusCode).toBeUndefined(); + }); +}); + +describe('ProviderUnavailableError', () => { + it('should create unavailable error', () => { + const error = new ProviderUnavailableError( + 'claude', + 'API key not configured', + ); + expect(error.message).toBe( + 'Provider claude unavailable: API key not configured', + ); + expect(error.provider).toBe('claude'); + expect(error.code).toBe('UNAVAILABLE'); + expect(error.name).toBe('ProviderUnavailableError'); + }); +}); diff --git a/.agent/skills/customize/SKILL.md b/.agent/skills/customize/SKILL.md new file mode 100644 index 0000000..c416e9a --- /dev/null +++ b/.agent/skills/customize/SKILL.md @@ -0,0 +1,120 @@ +--- +name: customize +description: Add new capabilities or modify NanoClaw behavior. Use when user wants to add channels (Telegram, Slack, email input), change triggers, add integrations, modify the router, or make any other customizations. This is an interactive skill that asks questions to understand what the user wants. +--- + +# NanoClaw Customization + +This skill helps users add capabilities or modify behavior. Use AskUserQuestion to understand what they want before making changes. + +## Workflow + +1. **Understand the request** - Ask clarifying questions +2. **Plan the changes** - Identify files to modify +3. **Implement** - Make changes directly to the code +4. **Test guidance** - Tell user how to verify + +## Key Files + +| File | Purpose | +| -------------------------- | --------------------------------------------------- | +| `src/index.ts` | Orchestrator: state, message loop, agent invocation | +| `src/channels/whatsapp.ts` | WhatsApp connection, auth, send/receive | +| `src/ipc.ts` | IPC watcher and task processing | +| `src/router.ts` | Message formatting and outbound routing | +| `src/types.ts` | TypeScript interfaces (includes Channel) | +| `src/config.ts` | Assistant name, trigger pattern, directories | +| `src/db.ts` | Database initialization and queries | +| `src/whatsapp-auth.ts` | Standalone WhatsApp authentication script | +| `groups/AGENT.md` | Global memory/persona | + +## Common Customization Patterns + +### Adding a New Input Channel (e.g., Telegram, Slack, Email) + +Questions to ask: + +- Which channel? (Telegram, Slack, Discord, email, SMS, etc.) +- Same trigger word or different? +- Same memory hierarchy or separate? +- Should messages from this channel go to existing groups or new ones? + +Implementation pattern: + +1. Create `src/channels/{name}.ts` implementing the `Channel` interface from `src/types.ts` (see `src/channels/whatsapp.ts` for reference) +2. Add the channel instance to `main()` in `src/index.ts` and wire callbacks (`onMessage`, `onChatMetadata`) +3. Messages are stored via the `onMessage` callback; routing is automatic via `ownsJid()` + +### Adding a New MCP Integration + +Questions to ask: + +- What service? (Calendar, Notion, database, etc.) +- What operations needed? (read, write, both) +- Which groups should have access? + +Implementation: + +1. Add MCP server config to the container settings (see `src/container-runner.ts` for how MCP servers are mounted) +2. Document available tools in `groups/AGENT.md` + +### Changing Assistant Behavior + +Questions to ask: + +- What aspect? (name, trigger, persona, response style) +- Apply to all groups or specific ones? + +Simple changes → edit `src/config.ts` +Persona changes → edit `groups/AGENT.md` +Per-group behavior → edit specific group's `AGENT.md` + +### Adding New Commands + +Questions to ask: + +- What should the command do? +- Available in all groups or main only? +- Does it need new MCP tools? + +Implementation: + +1. Commands are handled by the agent naturally — add instructions to `groups/AGENT.md` or the group's `AGENT.md` +2. For trigger-level routing changes, modify `processGroupMessages()` in `src/index.ts` + +### Changing Deployment + +Questions to ask: + +- Target platform? (Linux server, Docker, different Mac) +- Service manager? (systemd, Docker, supervisord) + +Implementation: + +1. Create appropriate service files +2. Update paths in config +3. Provide setup instructions + +## After Changes + +Always tell the user: + +```bash +# Rebuild and restart +npm run build +# macOS: +launchctl unload ~/Library/LaunchAgents/com.nanoclaw.plist +launchctl load ~/Library/LaunchAgents/com.nanoclaw.plist +# Linux: +# systemctl --user restart nanoclaw +``` + +## Example Interaction + +User: "Add Telegram as an input channel" + +1. Ask: "Should Telegram use the same @Andy trigger, or a different one?" +2. Ask: "Should Telegram messages create separate conversation contexts, or share with WhatsApp groups?" +3. Create `src/channels/telegram.ts` implementing the `Channel` interface (see `src/channels/whatsapp.ts`) +4. Add the channel to `main()` in `src/index.ts` +5. Tell user how to authenticate and test diff --git a/.agent/skills/db-analyze/SKILL.md b/.agent/skills/db-analyze/SKILL.md new file mode 100644 index 0000000..a70760e --- /dev/null +++ b/.agent/skills/db-analyze/SKILL.md @@ -0,0 +1,45 @@ +--- +name: db-analyze +description: Run ANALYZE on PostgreSQL to update table statistics used by the query planner +compatibility: FreeBSD 15.x +invoke_patterns: + - "Run analyze" + - "Analyze the database" + - "Update statistics" + - "Query planner stats" + - "Slow queries" +estimated_tokens: 400-600 +--- + +# db-analyze + +Update PostgreSQL query planner statistics. Fast, non-blocking, safe at any time. Usually run after bulk inserts or as part of regular maintenance. + +## Usage + +```bash +# Analyze entire database +bastille cmd db psql -U clawdie -d clawdie_ai_public -c "ANALYZE VERBOSE;" + +# Single table +bastille cmd db psql -U clawdie -d clawdie_ai_public -c "ANALYZE VERBOSE agent_activity;" +``` + +## Output + +``` +INFO: analyzing "public.agent_activity" +INFO: "agent_activity": scanned 30000 of 30000 pages, containing 15043 live rows +ANALYZE +``` + +## When to Use + +- After bulk inserts (embedding generation, log imports) +- When queries are running slower than expected +- As part of weekly DBA maintenance +- After `db-migrate` adds new tables or indexes + +## Escalate If + +- Analyze reveals severely bloated tables → recommend `db-vacuum` to DBA/CEO diff --git a/.agent/skills/db-migrate/SKILL.md b/.agent/skills/db-migrate/SKILL.md new file mode 100644 index 0000000..63e47f8 --- /dev/null +++ b/.agent/skills/db-migrate/SKILL.md @@ -0,0 +1,65 @@ +--- +name: db-migrate +description: Apply pending PostgreSQL schema migrations — always backs up first, reports what changed +compatibility: FreeBSD 15.x +invoke_patterns: + - "Apply migrations" + - "Run migrations" + - "Database migration" + - "Apply pending schema changes" + - "Migrate the database" +estimated_tokens: 1000-3000 +--- + +# db-migrate + +Apply pending schema migrations to PostgreSQL. **Always creates a backup and ZFS snapshot before running.** Reports exactly what changed. + +## Pre-Migration Checklist + +Before running migrations, always: + +1. Run `backup-db` skill → full dump +2. Run `zfs-snapshot` skill → `dataset@before-migration` +3. Check for active connections (avoid migration under load) + +## Usage + +```bash +# Check pending migrations (dry run) +bastille cmd db psql -U clawdie -d clawdie_ai_public \ + -c "SELECT id, name, applied_at FROM schema_migrations ORDER BY id;" + +# Apply migrations (project uses custom migration runner or pg-migrate) +# Check setup/db.ts for the current migration approach +``` + +## Typical Migration Flow + +``` +1. Snapshot: zroot/bastille/jails/db@before-migration-2026-04-07 +2. Backup: data/backups/clawdie_ai_public_2026-04-07_pre-migration.dump +3. Apply: run migration scripts in order +4. Verify: check table counts, spot-check key tables +5. Report: "Migration complete. 3 tables altered, 2 indexes added." +``` + +## Rollback + +If migration fails: + +```bash +# Restore from ZFS snapshot (requires operator approval — data destructive) +hostd zfs rollback zroot/bastille/jails/db@before-migration-2026-04-07 +``` + +## When to Use + +- Explicit request from CEO or operator +- Never run autonomously without board approval + +## Escalate If + +- Migration script has destructive operations (DROP TABLE, etc.) → **always escalate first** +- Migration takes >30 min → report progress to CEO +- Any error during migration → stop immediately, escalate diff --git a/.agent/skills/db-sync-check/SKILL.md b/.agent/skills/db-sync-check/SKILL.md new file mode 100644 index 0000000..88f2af9 --- /dev/null +++ b/.agent/skills/db-sync-check/SKILL.md @@ -0,0 +1,58 @@ +--- +name: db-sync-check +description: Verify replication lag between local PostgreSQL and Supabase (persistent memory sync) +compatibility: FreeBSD 15.x +invoke_patterns: + - "Check replication lag" + - "Is Supabase in sync" + - "Replication status" + - "Memory sync status" + - "Check sync" +estimated_tokens: 200-400 +--- + +# db-sync-check + +Check that local PostgreSQL and Supabase are in sync. Verifies replication lag and row counts on key tables. + +## Usage + +```bash +# Check local row count +bastille cmd db psql -U clawdie -d clawdie_ai_public \ + -c "SELECT COUNT(*) FROM agent_activity;" + +# Compare with Supabase (if SUPABASE_URL is set) +curl -s "${SUPABASE_URL}/rest/v1/agent_activity?select=count" \ + -H "apikey: ${SUPABASE_KEY}" \ + -H "Prefer: count=exact" | head -c 200 + +# Check replication slot lag (if using logical replication) +bastille cmd db psql -U clawdie -d clawdie_ai_public \ + -c "SELECT slot_name, active, lag FROM pg_replication_slots;" +``` + +## Output + +``` +Local agent_activity: 15,043 rows +Supabase agent_activity: 15,040 rows +Lag: 3 rows (acceptable — <60s sync interval) +``` + +## Acceptable Lag + +- < 100 rows: normal +- 100-1000 rows: elevated, monitor +- > 1000 rows: alert — investigate replication pipeline + +## When to Use + +- Weekly DBA check +- After Supabase connectivity issues +- After bulk inserts to verify sync is catching up + +## Escalate If + +- Lag > 1000 rows → escalate to CEO for investigation +- Replication slot inactive → escalate immediately diff --git a/.agent/skills/db-vacuum/SKILL.md b/.agent/skills/db-vacuum/SKILL.md new file mode 100644 index 0000000..3122f9c --- /dev/null +++ b/.agent/skills/db-vacuum/SKILL.md @@ -0,0 +1,56 @@ +--- +name: db-vacuum +description: Run VACUUM (and optionally ANALYZE) on PostgreSQL tables to reclaim dead row space +compatibility: FreeBSD 15.x +invoke_patterns: + - "Run vacuum" + - "Vacuum the database" + - "Vacuum *" + - "Reclaim disk space" + - "Dead rows cleanup" + - "Database maintenance" +estimated_tokens: 600-900 +--- + +# db-vacuum + +PostgreSQL VACUUM to reclaim space from dead rows. Safe to run on live databases — does not lock tables (VACUUM FULL does, avoid it without operator approval). + +## Usage + +```bash +# Standard vacuum (non-blocking, safe on live DB) +bastille cmd db psql -U clawdie -d clawdie_ai_public -c "VACUUM VERBOSE;" + +# Vacuum + analyze (recommended — updates query planner stats) +bastille cmd db psql -U clawdie -d clawdie_ai_public -c "VACUUM ANALYZE VERBOSE;" + +# Single table +bastille cmd db psql -U clawdie -d clawdie_ai_public -c "VACUUM ANALYZE agent_activity;" +``` + +## Output + +``` +INFO: vacuuming "public.agent_activity" +INFO: scanned index "agent_activity_pkey" to remove 1842 row versions +INFO: "agent_activity": removed 1842 row versions in 23 pages +INFO: "agent_activity": found 1842 removable, 15043 nonremovable row versions +VACUUM +``` + +## Do NOT Run + +- `VACUUM FULL` — locks the entire table; requires operator approval +- During high-traffic periods without checking active connections first + +## When to Use + +- Weekly maintenance (DBA on-demand) +- After large deletes or updates +- When `pg_stat_user_tables` shows high `n_dead_tup` + +## Escalate If + +- Vacuum blocked by long-running query → escalate to CEO (kill query decision) +- `VACUUM FULL` needed → requires operator approval diff --git a/.agent/skills/debug/SKILL.md b/.agent/skills/debug/SKILL.md new file mode 100644 index 0000000..76af6ca --- /dev/null +++ b/.agent/skills/debug/SKILL.md @@ -0,0 +1,282 @@ +--- +name: debug +description: Debug Clawdie agent issues on the current FreeBSD host runtime. Use when the agent is not responding, the service is restarting, Telegram messages go unanswered, subprocesses stall, or when verifying the live runtime shape. Covers host service management, logs, optional jail state, memory DB reachability, and common resolution patterns. +--- + +# Clawdie Agent Debugging (FreeBSD host runtime) + +All commands run on the host unless explicitly noted otherwise. + +## Scope and source of truth + +Use this skill for live runtime diagnosis. Do not use it as the canonical +architecture explainer. For current runtime defaults, prefer: + +- `src/config.ts` +- `setup/db.ts` +- `docs/internal/POSTGRES-MEMORY.md` +- `just doctor` + +Do not assume any of the following unless current config proves it: + +- a `clawdie-controlplane` jail +- `10.0.1.2` / `10.0.1.3` +- `clawdie_brain` +- a hardcoded pi path like `/opt/npm/bin/pi` + +## Current runtime recap + +Default runtime today: + +- main service: host rc.d service `clawdie` +- privileged sidecar: host rc.d service `clawdie_hostd` +- main process: `dist/index.js` +- main logs: `logs/clawdie.log`, `logs/clawdie.error.log` +- per-run pi logs: `groups/{folder}/logs/agent-*.log` +- default database mode: `DB_RUNTIME=host` +- default DB host for jails: `${SUBNET_BASE}.1` + +Optional / install-specific pieces: + +- `DB_RUNTIME=jail` uses the `db` jail role from `infra/jails.yaml` +- current repo jail registry defaults to `10.0.1.0/24`, with `db` on `.5` +- `PI_TUI_BIN` may override the pi path; otherwise use `command -v pi` + +## Log locations + +| Log | Path | Content | +| ----------- | ---------------------------------------- | --------------------------------------------- | +| Main app | `logs/clawdie.log` | Startup, Telegram events, scheduler, watchdog | +| Main errors | `logs/clawdie.error.log` | Uncaught exceptions and fatal errors | +| Per-run pi | `groups/{folder}/logs/agent-{runId}.log` | Full pi subprocess output | +| Heartbeat | `logs/heartbeat.log` | Controlplane checks and LLM reachability | +| Embed | `logs/embed-docs.log` | Knowledge embedding runs | + +## 1. Check service status + +```bash +sudo service clawdie status +sudo service clawdie_hostd status +pgrep -laf 'node.*/dist/index.js' +``` + +If the install explicitly uses `DB_RUNTIME=jail`, check that jail separately: + +```bash +sudo bastille list +``` + +## 2. Read logs + +```bash +tail -f logs/clawdie.log +tail -50 logs/clawdie.error.log +ls groups/*/logs/agent-*.log | tail -5 +``` + +After a run completes with an error, read the newest per-run log: + +```bash +ls -t groups/*/logs/agent-*.log | head -3 +tail -100 groups/main/logs/agent-<runId>.log +``` + +## 3. Restart / stop / start + +```bash +sudo service clawdie restart +sudo service clawdie stop +sudo service clawdie start +sudo service clawdie onestart +``` + +If hostd itself is suspect: + +```bash +sudo service clawdie_hostd restart +``` + +## 4. Common failure modes + +### Agent not responding to Telegram + +1. Check the main service is running. +2. Check the bot token exists: + +```bash +grep '^TELEGRAM_BOT_TOKEN=' .env +``` + +3. Look for Telegram / Grammy errors: + +```bash +grep -i 'telegram\|grammy\|409\|403' logs/clawdie.log | tail -20 +``` + +`409 Conflict` usually means two instances are long-polling at once. Check for +duplicate Node processes before restarting. + +### pi subprocess fails or produces no answer + +Check the newest per-run log first: + +```bash +ls -t groups/*/logs/agent-*.log | head -3 +tail -100 groups/main/logs/agent-<runId>.log +``` + +Then verify the configured pi path rather than assuming one: + +```bash +grep '^PI_TUI_BIN=' .env +command -v pi +pi --version +``` + +What to check first: + +- provider key missing or expired +- `PI_TUI_BIN` points at a dead path +- pi process died under memory pressure + +For memory pressure signals: + +```bash +dmesg | grep -i 'kill\|oom' | tail -10 +``` + +### Memory DB unreachable + +Check the live DB mode and target first: + +```bash +grep -E '^(DB_RUNTIME|DB_HOST|MEMORY_DB_URL)=' .env +node -e "import('./dist/config.js').then((m) => console.log(JSON.stringify({ DB_RUNTIME: m.DB_RUNTIME, DB_HOST: m.DB_HOST, MEMORY_DB_NAME: m.MEMORY_DB_NAME }, null, 2)))" +``` + +Probe the resolved host: + +```bash +pg_isready -h "$(node -e 'import("./dist/config.js").then((m) => process.stdout.write(m.DB_HOST))')" -p 5432 +``` + +If `DB_RUNTIME=jail`, verify the optional db jail exists and use its actual +jail name from `bastille list`: + +```bash +sudo bastille list | grep db +sudo bastille cmd <db-jail-name> service postgresql status +``` + +### Service keeps crashing + +Check for rapid restart loops in the log: + +```bash +grep -E 'fatal|error|exit|SIGTERM|SIGKILL' logs/clawdie.log | tail -20 +grep '409\|duplicate\|conflict' logs/clawdie.log | tail -10 +``` + +Pause the service to stop the restart loop, resolve the underlying issue, then +restart. + +### Watchdog or controlplane resets the agent + +```bash +grep -i watchdog logs/clawdie.log | tail -10 +grep -i controlplane logs/heartbeat.log | tail -20 +``` + +### Build errors after a code change + +```bash +npm run build +just doctor +``` + +Restart the service after a successful build if the running process needs to +pick up new code. + +## 5. Quick diagnostic + +Prefer the built-in check first: + +```bash +just doctor +``` + +If you need a manual snapshot: + +```bash +sudo service clawdie status +sudo service clawdie_hostd status +grep -E '^(DB_RUNTIME|DB_HOST|TELEGRAM_BOT_TOKEN|PI_TUI_BIN)=' .env +tail -10 logs/clawdie.log +tail -10 logs/clawdie.error.log +``` + +## 6. Optional jail shell access + +For jail-backed installs: + +```bash +sudo bastille console db +sudo bastille cmd db service postgresql status +``` + +Do not assume a controlplane jail exists. + +## 7. Enable debug logging + +Set `LOG_LEVEL=debug` in `.env`, then restart: + +```bash +sudo service clawdie restart +tail -f logs/clawdie.log +``` + +## 8. Metrics + +The metrics endpoint is exposed by the main runtime: + +```bash +curl -s http://127.0.0.1:9100/metrics | head -20 +curl -s http://127.0.0.1:9100/healthz +``` + +Use `/healthz` only as a listener check. Use `just doctor` for actual runtime +diagnosis. + +## 9. Autostart configuration + +```bash +sudo sysrc clawdie_enable +sudo sysrc clawdie_enable=AUTO +sudo sysrc clawdie_enable=YES +sudo sysrc clawdie_enable=NONE +``` + +Hostd has its own rcvar: + +```bash +sudo sysrc clawdie_hostd_enable +``` + +## 10. Session state + +Per-group state lives in `groups/{folder}/`: + +```bash +ls groups/ +ls groups/main/logs/ +ls groups/main/ipc/ 2>/dev/null +``` + +Run logs are temporary diagnostic files: + +```bash +rm -f groups/main/logs/agent-*.log +``` + +Do not delete active session files or the whole `groups/` tree while the agent +is running. diff --git a/.agent/skills/disk-usage/SKILL.md b/.agent/skills/disk-usage/SKILL.md new file mode 100644 index 0000000..9cc7e93 --- /dev/null +++ b/.agent/skills/disk-usage/SKILL.md @@ -0,0 +1,61 @@ +--- +name: disk-usage +description: Check disk space on host and inside jails — ZFS pool usage, dataset quotas, free space +compatibility: FreeBSD 15.x +invoke_patterns: + - "How much free disk" + - "Disk space" + - "Check disk" + - "Is disk full" + - "Storage usage" + - "ZFS usage" + - "Free up space" +estimated_tokens: 300-500 +--- + +# disk-usage + +Check disk and ZFS storage usage on the host and inside jails. + +## Usage + +```bash +# ZFS pool overview +zpool status +zpool list + +# Dataset usage and quotas +zfs list -o name,used,avail,quota,mountpoint + +# Specific dataset +zfs list tank/bastille/jails/db + +# Inside a jail +bastille cmd db df -h +``` + +## Output + +``` +NAME USED AVAIL QUOTA MOUNTPOINT +zroot 45G 890G - / +zroot/bastille/jails/db 2.3G 18G 20G /usr/local/bastille/jails/db/root +zroot/bastille/jails/cms 5.1G 15G 20G /usr/local/bastille/jails/cms/root +``` + +## Warning Thresholds + +- **>80% used:** Report to CEO, non-urgent +- **>90% used:** Escalate immediately — risk of write failures +- **Quota exceeded:** Jail may stop accepting writes — urgent + +## When to Use + +- Daily Sysadmin heartbeat +- Before running large backups or migrations +- After receiving low-disk alert + +## Escalate If + +- Any dataset >90% full → escalate to CEO for capacity decision +- ZFS pool degraded (DEGRADED status) → escalate immediately diff --git a/.agent/skills/docs-deployment/CROWDIN-SETUP.md b/.agent/skills/docs-deployment/CROWDIN-SETUP.md new file mode 100644 index 0000000..c076e97 --- /dev/null +++ b/.agent/skills/docs-deployment/CROWDIN-SETUP.md @@ -0,0 +1,363 @@ +# Crowdin Setup Guide + +**Objective:** Configure Crowdin project for clawdie-ai documentation translation workflow. + +**Time Required:** ~30 minutes + +**Prerequisites:** + +- GitHub account with push access to clawdie-ai repo +- Crowdin account (free tier is sufficient) + +--- + +## Step 1: Create Crowdin Project + +### 1a. Create on Crowdin + +1. Go to https://crowdin.com/ +2. **Sign in** or create a free account +3. Click **"Create Project"** button +4. **Project name:** `Clawdie-AI` +5. **Project identifier:** `clawdie-ai` (this becomes the URL slug) +6. **Source language:** English +7. **Target languages:** + - Slovenian (sl) + - German (de) + - French (fr) + - Spanish (es) +8. **Visibility:** Public (optional, but allows community contributions) +9. Click **"Create"** + +### 1b. Note Your Project ID and API Token + +After project creation, you'll need: + +- **Project ID** (visible in Project Settings → General → Project ID) +- **Personal API Token** (your Crowdin account → Settings → API Tokens → Create New Token) + +--- + +## Step 2: Connect GitHub + +### 2a. Install Crowdin GitHub App + +1. Go to: https://github.com/apps/crowdin +2. Click **"Install"** +3. Select the organization/account that owns `clawdie-ai` +4. Grant permissions (repo access, webhooks) +5. Confirm installation + +### 2b. Connect Crowdin Project to GitHub + +1. In Crowdin project → **Integrations** +2. Click **GitHub** tile +3. **Authorize** with GitHub +4. **Select repository:** `Clawdie/Clawdie-AI` +5. **Verify branch:** `main` +6. Click **"Connect"** + +--- + +## Step 3: Configure File Mapping + +### 3a. Upload .crowdin.yml + +The `.crowdin.yml` file in the repo root tells Crowdin how to map files: + +```yaml +project_id: YOUR_PROJECT_ID +api_token_env: CROWDIN_PERSONAL_TOKEN +base_path: / + +files: + - source: /docs/**.md + dest: /docs/{two_letters_code} + translation_update: crowdin_move + languages_mapping: + two_letters_code: + sl: sl + de: de + fr: fr + es: es + +commit_message: 'translations: update {language} translations from Crowdin' +auto_commit: true +``` + +**To use:** + +1. Edit `.crowdin.yml` in repo root +2. Replace `YOUR_PROJECT_ID` with your Crowdin project ID +3. Commit and push to `main` +4. Crowdin will automatically use this config + +### 3b. Trigger Initial Sync + +1. In Crowdin project → **GitHub Integration** +2. Click **"Sync Now"** (or wait for webhook) +3. All `.md` files from `docs/` will be pulled as source strings + +--- + +## Step 4: Add Crowdin API Token as GitHub Secret + +This allows GitHub Actions (optional) to integrate with Crowdin: + +### 4a. Create GitHub Secret + +1. Go to GitHub repo → **Settings** → **Secrets and variables** → **Actions** +2. Click **"New repository secret"** +3. **Name:** `CROWDIN_PERSONAL_TOKEN` +4. **Value:** Your Crowdin Personal API Token (from Step 1b) +5. Click **"Add secret"** + +### 4b. Alternative: Set Environment Variable on Host + +If using cron instead of GitHub Actions: + +```bash +# On FreeBSD host (where docs-sync.cron.sh runs) +# Add to /root/.bashrc or /root/.zshrc: + +export CROWDIN_PERSONAL_TOKEN="your_token_here" + +# Or set in crontab: +CROWDIN_PERSONAL_TOKEN=your_token_here +0 5 * * * /home/clawdie/clawdie-ai/scripts/docs-sync.cron.sh +``` + +--- + +## Step 5: Test the Workflow + +### 5a. Commit a Test English String + +```bash +cd /home/clawdie/clawdie-ai + +# Create a test doc +cat > docs/TEST-TRANSLATION.md <<'EOF' +# Translation Test + +This is a test document for Crowdin integration. + +## Content + +Please translate me! +EOF + +git add docs/TEST-TRANSLATION.md +git commit -m "docs: Add test translation document" +git push origin main +``` + +### 5b. Verify Crowdin Received It + +1. Go to Crowdin project +2. Check **Dashboard** or **Files** tab +3. You should see `TEST-TRANSLATION.md` as a new source file +4. All strings should be available for translation + +### 5c. Test Translation (Manual) + +1. In Crowdin, go to **Files** → `TEST-TRANSLATION.md` +2. Click on **German (de)** +3. Translate the strings manually (just for testing) +4. Mark the file as **100% reviewed** + +### 5d. Verify Translation Commit + +1. Wait ~5 minutes or trigger sync in Crowdin UI +2. Check GitHub repo +3. Look for a new commit from **crowdin[bot]** with: + - File: `docs/de/TEST-TRANSLATION.md` + - Content: Your German translation + +### 5e. Verify Compile + +```bash +# On FreeBSD host (or locally): +cd /home/clawdie/clawdie-ai + +# Test docs-compile.sh with German +./scripts/docs-compile.sh --language de /tmp/test-output/ + +# Check output +ls /tmp/test-output/docs-v*/de/ +# Should see: test-translation.html +``` + +--- + +## Step 6: Enable Translators + +### 6a. Invite Team Members + +1. In Crowdin project → **Members** +2. Click **"Invite Members"** +3. Add email addresses of translators +4. Assign them to languages (de, fr, es, sl) +5. Send invitations + +### 6b. Set Up Machine Translation (Optional) + +For faster initial translations: + +1. Project → **Automation** +2. Enable **Machine Translation** (if available in your plan) +3. Choose provider (Microsoft Translator, Google Translate, etc.) +4. Set languages to use machine translation by default + +--- + +## Step 7: Schedule Daily Sync + +### Option A: Using Cron (Recommended for FreeBSD) + +On the deployed host: + +```bash +# Create cron job (runs daily at 05:00 UTC) +0 5 * * * /home/clawdie/clawdie-ai/scripts/docs-sync.cron.sh + +# Verify cron log +tail -f /var/log/clawdie-docs-sync.log +``` + +**What it does:** + +1. `git pull` — fetches Crowdin bot commits +2. `docs-compile.sh --languages sl,en,de,fr,es` — compiles all languages +3. Symlink swap — zero-downtime deployment +4. Cleanup — deletes versions older than 30 days + +### Option B: GitHub Actions (Optional) + +Create `.github/workflows/crowdin-sync.yml`: + +```yaml +name: Crowdin Sync +on: + schedule: + - cron: '0 5 * * *' # 05:00 UTC + workflow_dispatch: # Allow manual trigger + +jobs: + sync: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + with: + token: ${{ secrets.GITHUB_TOKEN }} + + - name: Download translations from Crowdin + uses: crowdin/github-action@v1 + with: + upload_sources: false + download_translations: true + env: + CROWDIN_PROJECT_ID: ${{ secrets.CROWDIN_PROJECT_ID }} + CROWDIN_PERSONAL_TOKEN: ${{ secrets.CROWDIN_PERSONAL_TOKEN }} +``` + +--- + +## Troubleshooting + +### Crowdin Not Pulling from GitHub + +**Symptoms:** Commit new English doc, but Crowdin doesn't see it. + +**Check:** + +1. ✅ GitHub Crowdin app is installed in repo +2. ✅ `.crowdin.yml` exists and is correctly formatted +3. ✅ Crowdin project ID matches in `.crowdin.yml` +4. ✅ Check Crowdin → Integrations → GitHub → Logs + +**Fix:** + +```bash +# Manual trigger (if you have API token): +curl -X POST https://api.crowdin.com/api/v2/projects/{projectId}/remote-files/sync \ + -H "Authorization: Bearer $CROWDIN_PERSONAL_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"targetLanguageIds": ["sl", "de", "fr", "es"]}' +``` + +### GitHub Not Receiving Crowdin Commits + +**Symptoms:** Crowdin shows 100% translated, but no commit appears in GitHub. + +**Check:** + +1. ✅ `auto_commit: true` in `.crowdin.yml` +2. ✅ Crowdin bot has write access (check GitHub → Settings → Collaborators) +3. ✅ Language is set to 100% reviewed (not just translated) + +**Fix:** + +1. In Crowdin → **Files**, click **language (de)** +2. Select all strings → **Mark as reviewed** +3. Wait 5 minutes, check GitHub commits + +### .crowdin.yml Not Being Recognized + +**Symptoms:** Crowdin says "No files configured" even though `.crowdin.yml` is in repo. + +**Check:** + +1. ✅ File is in repo root (`/home/clawdie/clawdie-ai/.crowdin.yml`) +2. ✅ File is committed and pushed to `main` +3. ✅ Crowdin project ID in file matches actual project + +**Fix:** + +1. Push `.crowdin.yml` to main +2. In Crowdin → Settings → Import Configuration +3. Check "Use configuration file from repository" +4. Click "Update" + +--- + +## Security + +### Protect Your API Token + +❌ **Never commit** `CROWDIN_PERSONAL_TOKEN` to Git +✅ **Use GitHub Secrets** for CI/CD +✅ **Use environment variables** on host machine +✅ **Rotate tokens** if exposed + +### Crowdin Bot Permissions + +The Crowdin GitHub app needs: + +- ✅ Read access to your repo (pull source files) +- ✅ Write access to your repo (commit translations) +- ✅ Webhook access (react to GitHub events) + +These are standard and safe. Revoke if Crowdin is no longer used. + +--- + +## Next Steps + +1. ✅ Project created on Crowdin +2. ✅ GitHub connected via app +3. ✅ `.crowdin.yml` configured +4. ✅ Test workflow verified +5. ✅ Translators invited +6. ✅ Daily sync scheduled +7. 📚 **Ready for production translation workflow!** + +--- + +## Support + +- **Crowdin Docs:** https://support.crowdin.com/ +- **GitHub Crowdin App:** https://github.com/apps/crowdin +- **clawdie-ai Crowdin Project:** https://crowdin.com/project/clawdie-ai +- **Translation Workflow:** See `CROWDIN.md` (root of repo) +- **Integration Guide:** `.agent/skills/docs-deployment/INTEGRATION.md` diff --git a/.agent/skills/docs-deployment/INTEGRATION.md b/.agent/skills/docs-deployment/INTEGRATION.md new file mode 100644 index 0000000..db0c8d1 --- /dev/null +++ b/.agent/skills/docs-deployment/INTEGRATION.md @@ -0,0 +1,488 @@ +# Clawdie-AI: Build to Deployment Integration Guide + +**Phase:** 3.0+ +**Status:** Cross-repo reference (clawdie-ai + clawdie-shell) +**Created:** 24.mar.2026 + +--- + +## Overview + +This guide shows how **clawdie-shell** (ISO building) and **clawdie-ai** (deployment & operations) work together to deliver Clawdie-AI as a complete stack. + +``` +clawdie-shell (Build) clawdie-ai (Deploy) +───────────────────────────────────────────────────────── +build-iso skill → docs-deployment skill +build.sh (orchestrate) → docs-sync.cron.sh +firstboot.sh (provision) → nginx (serve) +shell-*.sh modules → Live system +``` + +--- + +## The Journey: Build → Deploy → Serve + +### Phase 1: ISO Creation (clawdie-shell repo) + +**Where:** `clawdie-shell/` +**Skill:** `build-iso` +**What happens:** + +```bash +cd /home/clawdie/clawdie-shell + +# Trigger build +npm run build-iso + +# What build-iso does: +# 1. Clones clawdie-iso from git jail +# 2. Runs build.sh with cloud/baremetal variants +# 3. Injects config (--target cloud, --domain, --tz, etc.) +# 4. Creates versioned ISO: clawdie-iso-cloud-24.mar.2026.img +# 5. Publishes to https://clawdie.si/downloads/ +``` + +**Output:** + +``` +clawdie-iso-cloud-24.mar.2026.img (cloud VPS variant) +clawdie-iso-amd-24.mar.2026.img (baremetal AMD GPU) +clawdie-iso-intel-24.mar.2026.img (baremetal Intel GPU) +clawdie-iso-baremetal-24.mar.2026.img (auto-detect) +``` + +**Baked into ISO:** + +- `build.cfg` — TARGET, ASSISTANT_NAME, AGENT_DOMAIN, TZ, SSH_PUBLIC_KEY, etc. +- `scripts/docs-compile.sh` — markdown → HTML compiler +- `scripts/docs-sync.cron.sh` — daily sync orchestrator +- Shell modules — gpu.sh, pkg.sh, env.sh, deploy.sh, etc. + +--- + +### Phase 2: ISO Boot & Firstboot (FreeBSD host) + +**Where:** User downloads ISO and boots on target hardware +**Scripts:** `firstboot.sh` + shell modules (in ISO payload) +**What happens:** + +``` +User boots ISO + ↓ +bsddialog wizard (baremetal only) +├─ Basic config (name, domain, timezone) +├─ ZFS RAID selection +├─ Backup destination (optional) +└─ Review + confirm + ↓ +firstboot.sh orchestrator runs: +├─ clawdie-shell-gpu.sh (GPU detection, cloud: skips) +├─ clawdie-shell-nvidia.sh (NVIDIA setup if needed) +├─ clawdie-shell-pkg.sh (Package repo config, offline cache) +├─ clawdie-shell-env.sh (Generate .env with secrets) +├─ clawdie-shell-system.sh (hostname, rc.conf, services) +└─ clawdie-shell-deploy.sh (Extract tarball, npm install, jails) + ↓ +System boots into FreeBSD with: +✓ Jails configured +✓ Services running +✓ Documentation source in docs/ +✓ Deployment scripts ready +✓ API keys in .env (blank, ready for web UI) +``` + +--- + +### Phase 3: Host-Level Operations (clawdie-ai repo) + +**Where:** `/home/clawdie/clawdie-ai/` (on deployed host) +**Skill:** `docs-deployment` +**What happens:** + +#### 3a. Author writes documentation + +```bash +cd /home/clawdie/clawdie-ai + +# Write English docs +vim docs/public/install/install.md +git add docs/public/install/install.md +git commit -m "docs: Add installation guide" +git push origin main + +# Crowdin webhook detects change +# Translators start working +``` + +#### 3b. Crowdin syncs translations + +``` +Crowdin auto-fetches (webhook) + ↓ +Translators work (crowdin.com/project/clawdie-ai) + ↓ +Translations complete (24-48 hours) + ↓ +Crowdin auto-commits: + docs/de/INSTALL.md (German) + docs/fr/INSTALL.md (French) + docs/es/INSTALL.md (Spanish) + ↓ +Git pulls on host +``` + +#### 3c. Daily sync @ 05:00 UTC (cron) + +```bash +# docs-sync.cron.sh runs automatically + +1. git pull + ✓ Gets latest markdown + translations + +2. docs-compile.sh --languages en,de,fr,es + ✓ Compiles all languages to versioned dirs: + docs-v0.9.0_24.mar.2026/ + ├─ en/ (29 HTML files) + ├─ de/ (29 HTML files) + ├─ fr/ (29 HTML files) + └─ es/ (29 HTML files) + +3. Validate output + ✓ Check all language indexes exist + ✓ Verify HTML structure + +4. Atomic symlink swap (zero-downtime) + en-current → docs-v0.9.0_24.mar.2026/en/ + de-current → docs-v0.9.0_24.mar.2026/de/ + fr-current → docs-v0.9.0_24.mar.2026/fr/ + es-current → docs-v0.9.0_24.mar.2026/es/ + +5. nginx reload + ✓ Serves new docs (no downtime) + +6. Cleanup + ✓ Delete versions older than 30 days + ✓ Update metadata +``` + +#### 3d. Users access documentation + +``` +Browser: https://docs.clawdie.si/ + ↓ +Language selector (root page) + ↓ +User picks language: /en/, /de/, /fr/, /es/ + ↓ +Nginx resolves symlink: + en-current → docs-v0.9.0_24.mar.2026/en/ + ↓ +User sees English docs (compiled from markdown) + ↓ +Zero-downtime updates: + - Author commits docs in clawdie-ai + - Cron syncs automatically @ 05:00 UTC + - Symlink swaps (no 404s, no downtime) +``` + +--- + +## Key Integration Points + +### 1. ISO Payload Contains Deployment Scripts + +**In clawdie-shell build.sh:** + +```bash +# Inject deployment scripts into ISO +cp /home/clawdie/clawdie-ai/scripts/docs-compile.sh \ + /mnt/media/scripts/ +cp /home/clawdie/clawdie-ai/scripts/docs-sync.cron.sh \ + /mnt/media/scripts/ +``` + +**In firstboot.sh:** + +```bash +# Scripts are available on first boot +/usr/local/share/clawdie-iso/scripts/docs-compile.sh --help +/usr/local/share/clawdie-iso/scripts/docs-sync.cron.sh +``` + +### 2. build.cfg Bridges Both Repos + +**In clawdie-shell (build-time):** + +```bash +# build.cfg is baked into ISO +TARGET="cloud" # cloud | baremetal +ASSISTANT_NAME="natasha" # pre-configured +AGENT_DOMAIN="natasha.internal" +TZ="Europe/Ljubljana" +``` + +**In clawdie-ai (runtime):** + +```bash +# scripts/docs-compile.sh and docs-sync.cron.sh read LANGUAGES +# Slovenian is primary (prototype default), others follow +LANGUAGES="sl,en,de,fr,es" # Slovenian primary, then English, German, French, Spanish +# Can be overridden by build.cfg or environment for cloud builds: +# LANGUAGES="sl,en" # Cloud: Slovenian primary + English only +``` + +### 3. Crowdin ↔ Documentation Sync + +**Source of truth:** `clawdie-ai` repo (markdown) + +``` +docs/public/install/install.md (English) ──→ Crowdin ──→ docs/sl/INSTALL.md (Slovenian primary) + ├→ docs/de/INSTALL.md (German) + ├→ docs/fr/INSTALL.md (French) + └→ docs/es/INSTALL.md (Spanish) +``` + +**Deployment:** Both repos pull from clawdie-ai + +``` +clawdie-shell (build-iso) + ├─ References clawdie-ai/docs/ for docs in ISO? + └─ Or: Operators maintain separate docs for ISO releases + +clawdie-ai (docs-deployment) + ├─ Compiles docs/ to HTML + ├─ Deploys daily @ 05:00 UTC + └─ Crowdin keeps languages in sync +``` + +--- + +## Workflow Scenarios + +### Scenario 1: Release a New ISO (clawdie-shell) + +``` +Operator in clawdie-shell repo: + +1. Update build.cfg with new version +2. Run: npm run build-iso +3. build-iso skill: + ├─ Clones clawdie-iso + ├─ Runs build.sh + ├─ Injects docs scripts (from clawdie-ai) + ├─ Creates ISO: clawdie-iso-cloud-24.mar.2026.img + └─ Publishes to https://clawdie.si/downloads/ + +4. Users download and boot ISO +``` + +### Scenario 2: Update Documentation (clawdie-ai) + +``` +Author in clawdie-ai repo: + +1. Write English docs: docs/public/install/install.md +2. Commit & push +3. Crowdin detects (webhook) +4. Translators work (24-48 hours) +5. Crowdin commits: docs/de/, docs/fr/, docs/es/ +6. Cron @ 05:00 UTC: + ├─ git pull (gets translations) + ├─ Compile all languages + ├─ Symlink swap (zero-downtime) + └─ nginx reload + +7. Users see updated docs in all languages +``` + +### Scenario 3: Add New Language to Docs + +``` +Process: + +1. Admin adds language to Crowdin project + crowdin.com/project/clawdie-ai → Add: Italian (it) + +2. Author adds to docs-sync.cron.sh: + LANGUAGES="en,de,fr,es,it" + +3. Crowdin auto-syncs translations to docs/it/ + +4. Next cron run (05:00 UTC): + ├─ Compile includes Italian + ├─ Create: docs-v0.9.0_24.mar.2026/it/ + └─ Symlink: it-current → docs-v0.9.0_24.mar.2026/it/ + +5. Nginx location block added (manually or via template): + location /it/ { ... } + +6. Users access: https://docs.clawdie.si/it/ +``` + +--- + +## Dependency Chain + +``` +clawdie-shell (Build) +├─ build-iso skill +├─ build.sh +│ └─ Fetches from clawdie-iso repo +│ └─ Contains: firstboot.sh + shell-*.sh modules +│ └─ Injects into ISO: +│ ├─ build.cfg (TARGET, ASSISTANT_NAME, etc.) +│ ├─ docs-compile.sh (from clawdie-ai) +│ ├─ docs-sync.cron.sh (from clawdie-ai) +│ └─ shell-*.sh modules +│ +└─ Output: clawdie-iso-{cloud,amd,baremetal}-24.mar.2026.img + └─ Users download & boot + └─ firstboot.sh runs shell modules + └─ System boots with deployment scripts ready + +clawdie-ai (Deploy) +├─ docs-deployment skill +├─ docs-compile.sh (used in ISO and on host) +├─ docs-sync.cron.sh (runs daily @ 05:00 UTC on host) +├─ Crowdin integration (markdown → translations) +│ └─ docs/ + docs/de/, docs/fr/, docs/es/ +│ +└─ Output: Zero-downtime documented system + └─ Users access: https://docs.clawdie.si/{en,de,fr,es}/ +``` + +--- + +## File Cross-References + +### clawdie-shell must know about: + +- `clawdie-ai/scripts/docs-compile.sh` — Injected into ISO +- `clawdie-ai/scripts/docs-sync.cron.sh` — Injected into ISO +- `clawdie-ai/docs/` — Source of truth for documentation +- `clawdie-ai/.agent/skills/docs-deployment/` — Integration guide (this file) + +### clawdie-ai must know about: + +- `clawdie-shell/skills/build-iso/` — How ISOs are created +- `clawdie-shell/build.sh` — What gets baked into ISO +- `clawdie-shell/build.cfg` — Template for cloud/baremetal variants + +### Both repos reference: + +- `clawdie-iso/` — FreeBSD ISO build system (not in this guide, external) +- `crowdin.com/project/clawdie-ai` — Translation management + +--- + +## Deployment Architectures + +### Cloud Deployment (VPS) + +``` +1. User downloads: clawdie-iso-cloud-24.mar.2026.img +2. Boots in VPS (no GUI, headless) +3. Firstboot runs: + ├─ Cloud preset (no GPU detection, no bsddialog) + ├─ All values pre-baked (ASSISTANT_NAME, DOMAIN, TZ) + ├─ System configures fully automated + └─ Ready for API key injection via web UI +4. Documentation: + └─ docs-sync.cron.sh @ 05:00 UTC + ├─ English only (cloud: reduced languages) + └─ Zero-downtime updates + +Docs serve: https://docs.natasha.internal/en/ +``` + +### Baremetal Deployment (Physical Hardware) + +``` +1. User downloads: clawdie-iso-amd-24.mar.2026.img (GPU-specific) +2. Boots on physical hardware (with GUI desktop) +3. Firstboot runs: + ├─ Interactive wizard (bsddialog) + ├─ User selects: name, domain, timezone, ZFS RAID config + ├─ GPU auto-detected (AMD/Intel/NVIDIA) + ├─ System configures interactively + └─ Desktop available for API key entry (browser) +4. Documentation: + └─ docs-sync.cron.sh @ 05:00 UTC + ├─ All languages (en, de, fr, es) + └─ Zero-downtime updates + +Docs serve: https://docs.natasha.internal/en/ (default) + or /de/, /fr/, /es/ (via language selector) +``` + +--- + +## Version Timeline + +### v0.9.0 Release (Target) + +**clawdie-shell:** + +- ✅ build-iso skill (publish ISO) +- ✅ Cloud vs baremetal variants +- ✅ SSH key + password configuration +- ✅ ZFS RAID selection + +**clawdie-ai:** + +- ✅ docs-deployment skill +- ✅ Multi-language (en, de, fr, es) +- ✅ Crowdin integration ready +- ✅ Zero-downtime deployment +- ✅ 30-day version retention + +**Together:** + +- ✅ Complete ISO → deployment pipeline +- ✅ Build (clawdie-shell) → Deploy (clawdie-ai) +- ✅ Documentation in 4 languages +- ✅ Automated daily sync @ 05:00 UTC + +--- + +## Quick Links + +**clawdie-shell:** + +- `.agent/skills/build-iso/SKILL.md` — ISO building +- `build.sh` — Build orchestrator +- `build.cfg` — Build configuration + +**clawdie-ai:** + +- `.agent/skills/docs-deployment/SKILL.md` — This skill ⬅️ +- `scripts/docs-compile.sh` — Markdown → HTML +- `scripts/docs-sync.cron.sh` — Daily sync orchestrator +- `CROWDIN.md` — Translation workflow +- `docs/DOCUMENTATION-SYNC-RUNBOOK.md` — Operator manual + +**External:** + +- Crowdin: https://crowdin.com/project/clawdie-ai +- ISO Downloads: https://clawdie.si/downloads/ +- Docs: https://docs.clawdie.si/ + +--- + +## Questions & Support + +**For ISO building issues:** See `clawdie-shell/.agent/skills/build-iso/SKILL.md` + +**For documentation deployment:** See `clawdie-ai/.agent/skills/docs-deployment/SKILL.md` + +**For troubleshooting:** See `clawdie-ai/docs/DOCUMENTATION-SYNC-RUNBOOK.md` + +**For translation workflow:** See `clawdie-ai/CROWDIN.md` + +**For Crowdin setup (admin):** See `clawdie-ai/.agent/skills/docs-deployment/CROWDIN-SETUP.md` + +--- + +**Last Updated:** 24.mar.2026 +**Phase:** 3.5 (Crowdin integration — Phase 3.0 foundation complete, ready for v0.9.0 release) diff --git a/.agent/skills/docs-deployment/SKILL.md b/.agent/skills/docs-deployment/SKILL.md new file mode 100644 index 0000000..d7416b1 --- /dev/null +++ b/.agent/skills/docs-deployment/SKILL.md @@ -0,0 +1,430 @@ +--- +name: docs-deployment +description: | + Manage multi-language documentation deployment on FreeBSD host. + Handles Crowdin integration, markdown compilation, symlink-based + zero-downtime deployment, and nginx language routing. + + Use when: setting up docs.clawdie.si, multi-language support, + Crowdin integration, documentation deployment, language routing, + translation workflow, documentation monitoring. +--- + +# Documentation Deployment System + +**Phase:** 3.0+ +**Status:** Production-ready (v0.9.0+) +**Created:** 24.mar.2026 + +--- + +## Overview + +This skill covers the **host-level documentation deployment system** for Clawdie-AI. + +Handles multi-language markdown→HTML compilation, Crowdin translation integration, zero-downtime symlink-based deployments, and automated daily syncing @ 05:00 UTC. + +**Not in scope:** CMS jail nginx (see `nginx` skill for cms jail vhost management) + +--- + +## Architecture at a Glance + +``` +Source (git) Compilation Deployment Serving +───────────────────────────────────────────────────────────────────────────────── +docs/public/install/install.md ──→ docs-compile.sh ──→ docs-v0.9.0_24.mar.2026/ ──→ nginx +docs/de/INSTALL.md (28 files) ├─ en/ (symlink) /en/ +docs/fr/INSTALL.md (4 langs) ├─ de/ (symlink) /de/ +docs/es/INSTALL.md ├─ fr/ (symlink) /fr/ + └─ es/ (symlink) /es/ + ↓ +Crowdin auto-syncs translations +(crowdin.com/project/clawdie-ai) + ↓ +docs-sync.cron.sh @ 05:00 UTC +├─ git pull +├─ compile (all 4 languages) +├─ atomic symlink swap (zero-downtime) +├─ nginx reload +└─ cleanup (30-day retention) +``` + +--- + +## Key Components + +### 1. Markdown Source (Git) + +- **English source:** `docs/*.md` (you maintain) +- **Translations:** `docs/de/*.md`, `docs/fr/*.md`, `docs/es/*.md` (Crowdin maintains) +- **Single source of truth:** English markdown is authoritative +- **Git strategy:** Markdown committed, HTML never committed + +**Locations:** + +``` +/home/clawdie/clawdie-ai/docs/ +├─ INSTALL.md (English) +├─ REQUIREMENTS.md (English) +├─ de/ +│ ├─ INSTALL.md (German — auto-synced by Crowdin) +│ └─ REQUIREMENTS.md +├─ fr/ +│ ├─ INSTALL.md (French) +│ └─ REQUIREMENTS.md +└─ es/ + ├─ INSTALL.md (Spanish) + └─ REQUIREMENTS.md +``` + +### 2. Compilation Pipeline (docs-compile.sh) + +- **Input:** Markdown files from `docs/` (filtered by `.docignore`) +- **Output:** Versioned HTML directories `docs-v0.9.0_24.mar.2026/{en,de,fr,es}/` +- **Features:** + - No external dependencies (pure shell, no pandoc/markdown packages needed) + - Filters internal/sensitive docs via `.docignore` + - Auto-generates language-specific indexes + - Version naming: `docs-v{SEMVER}_{DD.mon.YYYY}` (user-friendly Slovenian date format) + - Supports any number of languages + +**Usage:** + +```bash +# Single language (backward compatible) +scripts/docs-compile.sh --semver 0.9.0 /usr/local/www/docs.clawdie.si/ + +# Multi-language +scripts/docs-compile.sh --semver 0.9.0 --languages en,de,fr,es \ + /usr/local/www/docs.clawdie.si/ +``` + +**Performance:** ~10s for 4 languages, 28 markdown files + +### 3. Sync Orchestrator (docs-sync.cron.sh) + +- **Runs:** Daily @ 05:00 UTC via cron +- **Steps:** + 1. Git pull (fetch latest markdown + translations from Crowdin) + 2. Compile markdown → HTML (all languages) + 3. Validate output (check all language indexes exist) + 4. **Atomic symlink swap** (zero-downtime deployment) + 5. Nginx reload (serve new content) + 6. Cleanup old versions (30-day retention) + +**Execution time:** ~15-20s total + +### 4. Symlink-Based Deployment + +- **Strategy:** Atomic swaps for zero-downtime updates +- **How it works:** + + ```bash + # Before swap + en-current → docs-v0.9.0_20260317/en/ (old version) + + # Atomic swap (ln -sfn is atomic on modern filesystems) + ln -sfn docs-v0.9.0_24.mar.2026/en en-current + + # After swap + en-current → docs-v0.9.0_24.mar.2026/en/ (new version) + + # Total downtime: < 1 millisecond + ``` + +- **Rollback:** Also < 1 second (just repoint symlink to previous version) +- **Retention:** Keep 30 days of versions (~60MB for 3 versions) + +### 5. Crowdin Integration + +- **Project:** https://crowdin.com/project/clawdie-ai +- **Workflow:** + 1. You write English docs (docs/\*.md) + 2. Crowdin auto-fetches via GitHub webhook + 3. Translators work in Crowdin UI (no git knowledge needed) + 4. Crowdin auto-commits translations (docs/de/, docs/fr/, etc.) + 5. Cron pulls translations automatically + +- **No manual file editing:** Translations are managed by Crowdin, not manually +- **Translation memory:** Crowdin reuses past translations for consistency +- **Community:** Open to community translators + +### 6. Nginx Configuration + +- **Vhost:** `/usr/local/etc/nginx/vhosts/docs.clawdie.si.conf` +- **Language routing:** + + ```nginx + location /en/ { try_files $uri /en/index.html; } # English + location /de/ { try_files $uri /de/index.html; } # German + location /fr/ { try_files $uri /fr/index.html; } # French + location /es/ { try_files $uri /es/index.html; } # Spanish + location = / { return 301 /en/; } # Default to English + ``` + +- **Symlink following:** Nginx follows symlinks (en-current, de-current, etc.) +- **Language selector:** Root URL (`docs.clawdie.si/`) shows language picker + +--- + +## Deployment Checklist + +```bash +# 1. Crowdin Setup (manual, 15 min) +[ ] Create account: https://crowdin.com +[ ] Create project: "Clawdie-AI" +[ ] Connect GitHub (OAuth) +[ ] Add languages: de, fr, es +[ ] Configure file mappings: + Source: /docs/*.md + German: /docs/de/%filename% + French: /docs/fr/%filename% + Spanish: /docs/es/%filename% + +# 2. Repository Structure (10 min) +[ ] mkdir -p docs/{de,fr,es} +[ ] Add to .gitignore (language dirs auto-synced by Crowdin) +[ ] Create CROWDIN.md (workflow guide) +[ ] Update .docignore (filter language dirs from English build) + +# 3. Code Modifications (3 hours with testing) +[ ] Modify docs-compile.sh (add --languages flag) +[ ] Modify docs-sync.cron.sh (language-aware sync) +[ ] Test: sh docs-compile.sh --languages en,de,fr tmp/test +[ ] Verify: ls -la tmp/test/docs-v*/en/ /de/ /fr/ + +# 4. Nginx Configuration (20 min) +[ ] Update docs.clawdie.si.conf (language locations) +[ ] Create language selector HTML (root page) +[ ] Test: sudo nginx -t && sudo service nginx reload +[ ] Verify: curl https://docs.clawdie.si/{en,de,fr} + +# 5. Testing (1 hour) +[ ] Manual compile test +[ ] Nginx routing test +[ ] Crowdin integration test (commit test file, wait for sync) +[ ] Symlink swap test +[ ] Rollback test + +# 6. Git Commits (10 min) +[ ] 6 commits (structure, compile.sh, sync.sh, nginx, docs, metadata) +[ ] Push to origin +``` + +--- + +## Quick Reference + +| Task | Command | Time | +| -------------------------- | ----------------------------------------------------------------------------------------------------- | --------- | +| **Check sync status** | `tail -f /var/log/clawdie-docs-sync.log` | Real-time | +| **View current version** | `ls -l /usr/local/www/docs.clawdie.si/en-current` | Instant | +| **Manual compile** | `scripts/docs-compile.sh --languages en,de,fr /usr/local/www/docs.clawdie.si/` | ~10s | +| **Trigger immediate sync** | `sudo scripts/docs-sync.cron.sh` | ~15-20s | +| **Instant rollback** | `ln -sfn docs-v0.9.0_20260317 /usr/local/www/docs.clawdie.si/en-current && sudo service nginx reload` | <1s | +| **List versions** | `ls -lt /usr/local/www/docs.clawdie.si/docs-v* \| head -3` | Instant | + +--- + +## Common Tasks + +### Publishing New Documentation + +```bash +cd /home/clawdie/clawdie-ai + +# Write English doc +vim docs/MY-FEATURE.md +git add docs/MY-FEATURE.md +git commit -m "docs: Add MY-FEATURE guide" +git push origin main + +# Crowdin detects change (webhook) +# Translators translate (within 24 hours) +# Cron syncs automatically (05:00 UTC next day) +# Users see docs in all languages (zero-downtime) +``` + +### Excluding Docs from Public Sites + +Files matching `.docignore` patterns are NOT deployed: + +```bash +# Current filters (docs/.docignore) +cat /home/clawdie/clawdie-ai/docs/.docignore + +# Examples: +# POSTGRES-MEMORY.md (internal notes) +# *-INTERNAL.md (internal docs) +# */private/* (private directories) +``` + +To exclude new content: + +```bash +echo "*-DRAFT.md" >> /home/clawdie/clawdie-ai/docs/.docignore + +# File will be in git but NOT synced to public sites +``` + +### Instant Rollback + +If the new version needs rollback: + +```bash +# Find previous working version +ls -lt /usr/local/www/docs.clawdie.si/docs-v* | head -3 + +# Revert to previous +sudo ln -sfn docs-v0.9.0_20260317 /usr/local/www/docs.clawdie.si/en-current +sudo ln -sfn docs-v0.9.0_20260317 /usr/local/www/docs.clawdie.si/de-current +# ... (for all languages) + +# Reload nginx +sudo service nginx reload + +# Total time: < 1 second +``` + +### Monitor Sync Health + +```bash +# Last sync timestamp +jq '.last_sync' /home/clawdie/clawdie-ai/docs/.sync-metadata.json + +# Current versions deployed +ls -l /usr/local/www/docs.clawdie.si/{en,de,fr,es}-current + +# Sync logs (watch real-time) +tail -f /var/log/clawdie-docs-sync.log + +# Verify symlinks +find /usr/local/www/docs.clawdie.si -maxdepth 1 -type l ! -exec test -d {} \; -print +# Should be empty +``` + +--- + +## Troubleshooting + +### Symlink Broken (Points to Missing Directory) + +```bash +# Check +ls -l /usr/local/www/docs.clawdie.si/en-current +# en-current -> docs-v0.9.0_24.mar.2026 (BROKEN LINK) + +# Fix: Find working version +ls -d /usr/local/www/docs.clawdie.si/docs-v* + +# Repoint +sudo ln -sfn docs-v0.9.0_20260317 /usr/local/www/docs.clawdie.si/en-current +sudo service nginx reload +``` + +### Nginx 404 After Symlink Swap + +```bash +# Check nginx config +sudo nginx -t + +# Check permissions (nginx needs read+execute) +stat /usr/local/www/docs.clawdie.si/docs-v0.9.0_24.mar.2026 +# Should show: (0755) drwxr-xr-x + +# Fix if needed +sudo chmod -R a+rx /usr/local/www/docs.clawdie.si/docs-v*/ + +# Reload +sudo service nginx reload +``` + +### Cron Sync Not Running + +```bash +# Check cron job +sudo crontab -l | grep docs-sync +# Should show: 0 5 * * * root /home/clawdie/clawdie-ai/scripts/docs-sync.cron.sh + +# Check system logs +sudo grep docs-sync /var/log/cron | tail -10 + +# If missing, reinstall: +sudo crontab -e +# Add: 0 5 * * * root /home/clawdie/clawdie-ai/scripts/docs-sync.cron.sh >> /var/log/clawdie-docs-sync.log 2>&1 +``` + +### Git Pull Failing in Sync + +```bash +# Check git status +cd /home/clawdie/clawdie-ai +git status + +# Check for merge conflicts +git diff HEAD origin/implementation -- docs/ + +# Resolve manually +git fetch origin +git merge origin/implementation +# Fix conflicts, commit, push + +# Retry sync +sudo /home/clawdie/clawdie-ai/scripts/docs-sync.cron.sh +``` + +--- + +## Performance & Capacity + +| Metric | Value | Notes | +| --------------------- | ---------- | ------------------------------------- | +| **Compile time** | ~10s | 4 languages, 28 markdown files | +| **Sync time** | ~15-20s | git pull + compile + deploy + cleanup | +| **Symlink swap** | <1ms | Atomic operation | +| **Rollback time** | <1s | Repoint symlink + reload nginx | +| **Version size** | ~20MB each | Per language version | +| **3-version storage** | ~60MB | 30-day retention | +| **Daily bandwidth** | ~100-200MB | git pull + sync logs | + +--- + +## Integration Points + +**Related files in repo:** + +- `scripts/docs-compile.sh` — Compilation implementation +- `scripts/docs-sync.cron.sh` — Sync orchestrator +- `docs/DOCUMENTATION-POLICY.md` — Policy and governance +- `docs/DOCUMENTATION-SYNC-RUNBOOK.md` — Operator manual (detailed) +- `docs/.docignore` — Filter rules +- `docs/.sync-metadata.json` — Deployment metadata +- `CROWDIN.md` — Translation workflow + +**External services:** + +- Crowdin (https://crowdin.com/project/clawdie-ai) — Translation management +- GitHub (oauth for Crowdin) — Source control + +**Host files:** + +- `/usr/local/etc/nginx/vhosts/docs.clawdie.si.conf` — Nginx config +- `/usr/local/www/docs.clawdie.si/` — Webroot +- `/var/log/clawdie-docs-sync.log` — Sync logs +- Cron job (root crontab) — Daily trigger @ 05:00 UTC + +--- + +## See Also + +- [DOCUMENTATION-SYNC-RUNBOOK.md](../../docs/DOCUMENTATION-SYNC-RUNBOOK.md) — Detailed operator manual with all troubleshooting steps +- [CROWDIN.md](../../CROWDIN.md) — Translation workflow guide for authors and translators +- [html/docs-clawdie-si/VERSIONING.md](../../html/docs-clawdie-si/VERSIONING.md) — Deep dive into symlink-based versioning architecture +- `nginx` skill — Handles cms jail nginx (not host-level docs) + +--- + +**Last Updated:** 24.mar.2026 +**Skill Version:** 1.0 +**Phase:** 3.0+ (Production-ready) diff --git a/.agent/skills/docs-deployment/templates/language-selector.html b/.agent/skills/docs-deployment/templates/language-selector.html new file mode 100644 index 0000000..739a017 --- /dev/null +++ b/.agent/skills/docs-deployment/templates/language-selector.html @@ -0,0 +1,209 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <meta charset="UTF-8"> + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + <title>Clawdie-AI Documentation + + + +
+

📚 Clawdie-AI

+

Select your language to view documentation

+ +
+ +
🇬🇧
+
English
+ + +
🇩🇪
+
Deutsch
+ + +
🇫🇷
+
Français
+ + +
🇪🇸
+
Español
+ +
+ +
+ +
+ 💬 Help translate:
+ Documentation is maintained via Crowdin. + Join our translation community at + crowdin.com/project/clawdie-ai +
+ +
+

+ Clawdie-AI • FreeBSD-based AI Assistant
+ clawdie.si • Documentation v0.9.0+ +

+
+
+ + diff --git a/.agent/skills/docs-deployment/templates/nginx-vhost-template.conf b/.agent/skills/docs-deployment/templates/nginx-vhost-template.conf new file mode 100644 index 0000000..e9b835d --- /dev/null +++ b/.agent/skills/docs-deployment/templates/nginx-vhost-template.conf @@ -0,0 +1,142 @@ +# docs.clawdie.si — Multi-Language Documentation Vhost +# Template for host-level documentation serving with language routing +# +# Location: /usr/local/etc/nginx/vhosts/docs.clawdie.si.conf +# Symlinks: en-current, de-current, fr-current, es-current +# (each points to docs-v0.9.0_24.mar.2026/{lang}/) +# +# Features: +# - Language routing: /en/, /de/, /fr/, /es/ +# - Language selector: docs.clawdie.si/ (root redirects to language picker) +# - Zero-downtime updates: Atomic symlink swaps +# - SSL/TLS: Let's Encrypt + +# HTTP → HTTPS redirect +server { + listen 80; + listen [::]:80; + server_name docs.clawdie.si; + return 301 https://docs.clawdie.si$request_uri; +} + +# HTTPS server (multi-language) +server { + listen 443 ssl http2; + listen [::]:443 ssl http2; + server_name docs.clawdie.si; + + # SSL configuration + ssl_certificate /usr/local/etc/nginx/ssl/docs/fullchain.cer; + ssl_certificate_key /usr/local/etc/nginx/ssl/docs/docs.key; + ssl_protocols TLSv1.2 TLSv1.3; + ssl_ciphers HIGH:!aNULL:!MD5; + ssl_prefer_server_ciphers on; + + # Security headers + add_header X-Content-Type-Options nosniff always; + add_header X-Frame-Options SAMEORIGIN always; + add_header X-XSS-Protection "1; mode=block" always; + add_header Referrer-Policy strict-origin-when-cross-origin always; + add_header Permissions-Policy "geolocation=(), microphone=(), camera=()" always; + + # Logging + access_log /var/log/nginx/docs.access.log combined; + error_log /var/log/nginx/docs.error.log warn; + + # ──────────────────────────────────────────────────────────────────────────── + # Language-Specific Routes + # ──────────────────────────────────────────────────────────────────────────── + + # English documentation + location /en/ { + root /usr/local/www/docs.clawdie.si; + # Points to: en-current symlink → docs-v0.9.0_24.mar.2026/en/ + try_files $uri $uri/ /en/index.html =404; + expires 7d; + add_header Cache-Control "public, max-age=604800"; + } + + # German documentation + location /de/ { + root /usr/local/www/docs.clawdie.si; + # Points to: de-current symlink → docs-v0.9.0_24.mar.2026/de/ + try_files $uri $uri/ /de/index.html =404; + expires 7d; + add_header Cache-Control "public, max-age=604800"; + } + + # French documentation + location /fr/ { + root /usr/local/www/docs.clawdie.si; + # Points to: fr-current symlink → docs-v0.9.0_24.mar.2026/fr/ + try_files $uri $uri/ /fr/index.html =404; + expires 7d; + add_header Cache-Control "public, max-age=604800"; + } + + # Spanish documentation + location /es/ { + root /usr/local/www/docs.clawdie.si; + # Points to: es-current symlink → docs-v0.9.0_24.mar.2026/es/ + try_files $uri $uri/ /es/index.html =404; + expires 7d; + add_header Cache-Control "public, max-age=604800"; + } + + # ──────────────────────────────────────────────────────────────────────────── + # Root: Language Selector + # ──────────────────────────────────────────────────────────────────────────── + + # Root serves language picker HTML + location = / { + root /usr/local/www/docs.clawdie.si; + try_files /index.html =404; + } + + # ──────────────────────────────────────────────────────────────────────────── + # Static Assets (CSS, JS, Images) + # ──────────────────────────────────────────────────────────────────────────── + + location /css/ { + root /usr/local/www/docs.clawdie.si; + try_files $uri =404; + expires 30d; + add_header Cache-Control "public, max-age=2592000, immutable"; + } + + location /js/ { + root /usr/local/www/docs.clawdie.si; + try_files $uri =404; + expires 30d; + add_header Cache-Control "public, max-age=2592000, immutable"; + } + + location /images/ { + root /usr/local/www/docs.clawdie.si; + try_files $uri =404; + expires 30d; + add_header Cache-Control "public, max-age=2592000, immutable"; + } + + # ──────────────────────────────────────────────────────────────────────────── + # Deny Access to Hidden Files & Versioned Directories + # ──────────────────────────────────────────────────────────────────────────── + + # Block access to version directories (users should access via /en/, /de/, etc.) + location ~ ^/docs-v[0-9]+ { + return 403; + } + + # Block hidden files (.git, .env, etc.) + location ~ /\. { + return 403; + } + + # ──────────────────────────────────────────────────────────────────────────── + # Catch-all: Deny undefined paths + # ──────────────────────────────────────────────────────────────────────────── + + location / { + return 404; + } +} diff --git a/.agent/skills/docs-localization-pipeline/README.md b/.agent/skills/docs-localization-pipeline/README.md new file mode 100644 index 0000000..97bd19e --- /dev/null +++ b/.agent/skills/docs-localization-pipeline/README.md @@ -0,0 +1,315 @@ +# Docs Localization Pipeline Skill + +**Codifies the three-bird documentation architecture for Pi/Aider automation.** + +## Overview + +This skill automates the complete documentation localization and deployment pipeline: + +``` +English Docs (docs/public/) + ↓ +Crowdin (Project 883714) + ↓ [Push source, Pull translations] +Translated Docs (docs/public/{lang}/) + ↓ +Astro Starlight Build + ↓ [npm run build] +Static Site (dist/) + ↓ +Deploy to nginx + ↓ +Public: https://docs.clawdie.si +``` + +## Quick Start + +### 1. Initial Setup (First Time Only) + +```bash +# Setup Astro Starlight project +./.agent/skills/docs-localization-pipeline/setup-clawdie-docs.sh + +# Verify setup +ls -la /home/clawdie/clawdie-docs/ +cd /home/clawdie/clawdie-docs && npm run dev +# Browse: http://localhost:4321 +``` + +### 2. Manual Pipeline Execution + +```bash +# Full pipeline (all stages) +./.agent/skills/docs-localization-pipeline/orchestrate-pipeline.sh --all + +# Individual stages +./.agent/skills/docs-localization-pipeline/orchestrate-pipeline.sh --push # Push sources to Crowdin +./.agent/skills/docs-localization-pipeline/orchestrate-pipeline.sh --pull # Pull translations +./.agent/skills/docs-localization-pipeline/orchestrate-pipeline.sh --build # Pull + Build +./.agent/skills/docs-localization-pipeline/orchestrate-pipeline.sh --deploy # Deploy built site +./.agent/skills/docs-localization-pipeline/orchestrate-pipeline.sh --status # Check Crowdin status +``` + +### 3. Pi/Aider Commands + +```bash +# Via Pi TUI +pi "sync and rebuild documentation site" +pi "publish docs translations" +pi "check documentation translation progress" + +# Via Aider +aider --auto-commits --message "Sync translations and rebuild docs" docs/ +``` + +## File Structure + +``` +.agent/skills/docs-localization-pipeline/ +├── SKILL.md # Architecture & detailed reference +├── README.md # This file +├── setup-clawdie-docs.sh # Bootstrap Astro Starlight project +└── orchestrate-pipeline.sh # Orchestrate full Crowdin→Astro→Deploy pipeline +``` + +## Scripts + +### setup-clawdie-docs.sh + +**Purpose:** Bootstrap the Astro Starlight documentation site. + +**Usage:** + +```bash +./setup-clawdie-docs.sh # Default: /home/clawdie/clawdie-docs +./setup-clawdie-docs.sh /custom/path # Custom path +``` + +**What it does:** + +1. Creates Astro project structure +2. Installs dependencies (npm install) +3. Generates `astro.config.mjs` with multi-language configuration +4. Sets up content collections config +5. Creates symlinks to `docs/public/{lang}/` for translations +6. Creates i18n UI translation JSON files + +**Prerequisites:** + +- Node.js >=20 +- npm +- Docs source at `/home/clawdie/clawdie-ai/docs/public/` + +**Output:** + +``` +/home/clawdie/clawdie-docs/ +├── package.json +├── astro.config.mjs +├── tsconfig.json +├── src/ +│ ├── content/config.ts +│ ├── docs/ +│ │ ├── en → /home/clawdie/clawdie-ai/docs/public/en +│ │ ├── de → /home/clawdie/clawdie-ai/docs/public/de +│ │ └── [other languages] +│ └── i18n/ +│ ├── en.json +│ ├── de.json +│ └── sl.json +└── dist/ (after npm run build) +``` + +### orchestrate-pipeline.sh + +**Purpose:** Automate the full Crowdin→Astro→Deploy pipeline. + +**Usage:** + +```bash +./orchestrate-pipeline.sh --all # Full pipeline +./orchestrate-pipeline.sh --push # Stage 1: Push English sources +./orchestrate-pipeline.sh --pull # Stage 2: Pull translations +./orchestrate-pipeline.sh --build # Stages 2+3: Pull + Build +./orchestrate-pipeline.sh --build-only # Stage 3: Build only +./orchestrate-pipeline.sh --deploy-only # Stage 4: Deploy +./orchestrate-pipeline.sh --status # Check Crowdin status +./orchestrate-pipeline.sh --help # Show help +``` + +**Stages:** + +| Stage | Command | What It Does | +| ----- | ---------- | ------------------------------------------------ | +| 1 | `--push` | Upload English `docs/public/*.md` to Crowdin | +| 2 | `--pull` | Download translations into `docs/public/{lang}/` | +| 3 | `--build` | Run `npm run build` to generate static site | +| 4 | `--deploy` | Copy `dist/` to nginx webroot | + +**Environment Variables:** + +```bash +# Set in .env or export before running +CROWDIN_PERSONAL_TOKEN=tskey-xxx # Crowdin API token (required for push/pull) +CMS_DOCS_SITE_PATH=/home/clawdie/clawdie-docs # Astro project path +DOCS_DEPLOY_TARGET=/usr/local/www/clawdie/docs # nginx webroot +DOCS_MIN_COMPLETION=80 # Min % translation completion before deploy +``` + +**Output Example:** + +``` +=== Stage 1: Push English Sources to Crowdin === +[INFO] Uploading English sources to Crowdin... + [1] Uploading: install/index.md + [2] Uploading: install/requirements.md + ... +[INFO] Upload complete: 42 files processed, 0 failed + +=== Stage 2: Pull Translations from Crowdin === +[INFO] Downloading translations from Crowdin... +[INFO] Translation completion: 85% ✓ +Language: sl + Downloaded: /tmp/crowdin-sl-1712951234.zip + ✓ Extracted 42 files to sl/ + +=== Stage 3: Build Astro Site === +[INFO] Building Astro site... +✓ Site built: 217 files + +=== Stage 4: Deploy to Nginx === +[INFO] Copying files to /usr/local/www/clawdie/docs +✓ Nginx reloaded +✓ Deploy complete: https://docs.clawdie.si +``` + +## Configuration + +### .env Setup + +```bash +# Crowdin API token (https://crowdin.com/settings/api/tokens) +CROWDIN_PERSONAL_TOKEN=tskey-your-token-here + +# Astro project path (optional, defaults to /home/clawdie/clawdie-docs) +CMS_DOCS_SITE_PATH=/home/clawdie/clawdie-docs + +# nginx deployment target (optional, defaults to /usr/local/www/clawdie/docs) +DOCS_DEPLOY_TARGET=/usr/local/www/clawdie/docs + +# Minimum translation completion % before deploying (default: 80) +DOCS_MIN_COMPLETION=80 +``` + +### Crowdin Project + +- **Project ID:** 883714 +- **Configuration:** `.crowdin.yml` (repo root) +- **Languages:** SL (default), EN, DE, HR, SR, RU, EL, IT, MK, SK, BS +- **Source:** `docs/public/**.md` +- **Output:** `docs/public/{lang}/` + +## Workflows + +### Weekly Sync (Recommended) + +```bash +# Add to crontab (runs Monday 9am) +0 9 * * MON /path/to/.agent/skills/docs-localization-pipeline/orchestrate-pipeline.sh --pull && \ + cd /path/to && \ + /path/to/.agent/skills/docs-localization-pipeline/orchestrate-pipeline.sh --build-only && \ + /path/to/.agent/skills/docs-localization-pipeline/orchestrate-pipeline.sh --deploy-only +``` + +### On-Demand via Pi + +```bash +# Pi will interpret natural language and call orchestrate-pipeline.sh +pi "sync docs translations and rebuild site" +# → Executes: orchestrate-pipeline.sh --all +``` + +### GitHub Actions (Optional) + +See `.github/workflows/crowdin-sync.yml` for automated CI/CD setup. + +## Troubleshooting + +### "Crowdin token not found" + +**Fix:** + +```bash +# Option A: Create token file +mkdir -p ~/.config/clawdie +echo "tskey-your-token" > ~/.config/clawdie/crowdin-token +chmod 600 ~/.config/clawdie/crowdin-token + +# Option B: Set env var +export CROWDIN_PERSONAL_TOKEN=tskey-your-token + +# Option C: Add to .env +echo 'CROWDIN_PERSONAL_TOKEN=tskey-your-token' >> .env +``` + +### "Astro project not found" + +**Fix:** + +```bash +# Run setup script +./.agent/skills/docs-localization-pipeline/setup-clawdie-docs.sh +``` + +### "npm: command not found" + +**Fix:** Install Node.js and npm for FreeBSD: + +```bash +sudo pkg install node24 npm-node24 +``` + +### "Translation completion too low" + +**Fix:** Adjust `DOCS_MIN_COMPLETION` or wait for translators: + +```bash +DOCS_MIN_COMPLETION=50 ./orchestrate-pipeline.sh --all +``` + +### "Astro build fails with missing locale" + +**Fix:** Ensure translations are pulled before build: + +```bash +# Manual steps +./scripts/crowdin-sync.sh --pull +cd /home/clawdie/clawdie-docs && npm run build +``` + +### "Deploy fails: permission denied" + +**Fix:** Ensure sudo access to webroot: + +```bash +# Add to sudoers (FreeBSD): +sudo visudo +# Add line: clawdie ALL=(ALL) NOPASSWD: /bin/cp -r, /usr/sbin/service nginx +``` + +## Next Steps + +1. **Crowdin Setup** → Create translators and assign languages +2. **First Push** → `./orchestrate-pipeline.sh --push` to seed English sources +3. **Translation** → Wait for translators to complete milestone (80%+) +4. **First Build** → `./orchestrate-pipeline.sh --build` to test build +5. **Deploy** → `./orchestrate-pipeline.sh --deploy` to go live +6. **Automate** → Setup cron or GitHub Actions for weekly syncs + +## References + +- **Crowdin Project:** https://crowdin.com/project/clawdie-ai +- **Crowdin API:** https://developer.crowdin.com/api/ +- **Astro Docs:** https://docs.astro.build/ +- **Starlight Docs:** https://starlight.astro.build/ +- **Skill Architecture:** `SKILL.md` (in this directory) diff --git a/.agent/skills/docs-localization-pipeline/SKILL.md b/.agent/skills/docs-localization-pipeline/SKILL.md new file mode 100644 index 0000000..836d445 --- /dev/null +++ b/.agent/skills/docs-localization-pipeline/SKILL.md @@ -0,0 +1,406 @@ +--- +name: docs-localization-pipeline +description: Automate the Crowdin → Astro Starlight → deploy localization pipeline for Clawdie docs. +--- + +# Three-Bird Docs Localization Pipeline + +**Status:** MVP Implementation Plan +**Target:** Pi/Aider Automation +**Architecture:** Crowdin → Astro Starlight → Deploy +**Languages:** EN (primary), SLO (default for clawdie.si), DE/SRB/CRO/BIH/RU/ZN/BR (roadmap) + +## Overview + +The **three-bird architecture** separates concerns for documentation, localization, and content delivery: + +1. **Crowdin** — Single source of truth for translations. English docs are pushed, Crowdin maintains translations in 9 target languages. +2. **Astro Starlight** — Static site generator. Builds multilingual docs site from Crowdin translations + English source. +3. **Strapi CMS** (optional) — Content backend for rich media, structured content (deferred due to FreeBSD native binding issues; see `doc/STRAPI-FREEBSD-GOTCHA.md`). + +**MVP Approach:** Markdown-only pipeline (Crowdin → Astro). Ships EN + SLO immediately. Can defer CMS until image upload requirements emerge. + +--- + +## Phase 1: Crowdin → Astro Build Pipeline + +### Architecture + +``` +docs/public/ # English source (Markdown) + ├── install/ + ├── architecture/ + ├── reference/ + └── [other sections] + +Crowdin (Project 883714) # Localization source of truth + ├── Source: docs/public/**.md + ├── Languages: sl, de, hr, sr, ru, el, it, mk, sk, bs + └── Output: docs/public/{lang}/** + +docs/public/{lang}/ # Pulled translations + ├── sl/ # Slovenian + ├── de/ # German + ├── hr/ # Croatian + └── [others...] + +Astro Starlight Site # Multi-language static build + ├── Root (SLO): / + ├── English: /en/ + ├── German: /de/ + ├── Croatian: /hr/ + └── [others...] + +Deployment + └── nginx → docs.clawdie.si +``` + +### Pipeline Steps (Automation-Ready for Pi/Aider) + +#### Step 1: Push English Sources to Crowdin + +```bash +./scripts/crowdin-sync.sh --push +``` + +**What it does:** + +- Finds all `.md` files in `docs/public/` (excluding translation directories) +- Uploads each file to Crowdin storage +- Updates project files via Crowdin API +- Waits for translators to work on updated strings + +**Automation notes:** + +- Run whenever `docs/public/` English files change +- Safe to run multiple times (idempotent) +- Requires `CROWDIN_PERSONAL_TOKEN` env var or `~/.config/clawdie/crowdin-token` + +#### Step 2: Download Translations from Crowdin + +```bash +./scripts/crowdin-sync.sh --pull +``` + +**What it does:** + +- For each language (sl, de, hr, sr, ru, el, it, mk, sk, bs): + - Requests translation export from Crowdin + - Downloads ZIP of translated files + - Extracts to `docs/public/{lang}/` +- Creates `docs/public/{lang}/` directories if missing + +**Automation notes:** + +- Run after translations reach >80% completion +- Safe to run multiple times (overwrites old translations) +- Requires same token as push + +#### Step 3: Build Astro Starlight Site + +```bash +# In Astro project root +npm run build +``` + +**What it does:** + +- Reads English source from `docs/public/` +- Reads translations from `docs/public/{lang}/` +- Generates static site at `dist/` +- Multi-language navigation configured in `astro.config.mjs` + +**Output structure:** + +``` +dist/ + ├── index.html # SLO root + ├── en/ # English + ├── de/ # German + ├── hr/ # Croatian + └── [others...] +``` + +**Automation notes:** + +- Requires Node.js >=20 + npm +- ~30-60 seconds per build +- Idempotent (safe to run multiple times) + +#### Step 4: Deploy to Production + +```bash +# Deploy to nginx webroot (rsync: incremental, verified, cleaned up) +sudo rsync -av --delete --checksum /home/clawdie/clawdie-docs/dist/ /usr/local/www/clawdie/docs/ + +# Reload nginx +sudo nginx -s reload +``` + +**Why rsync (instead of cp -r):** + +- **Incremental:** Only changed files copied (faster on repeated deploys) +- **Cleanup:** `--delete` removes stale assets automatically +- **Verified:** `--checksum` ensures file integrity +- **Resumable:** Can re-run if interrupted + +**Automation notes:** + +- Requires sudo access (setup in .env or via wheel group) +- rsync available on FreeBSD (`/usr/local/bin/rsync`) +- Preview changes first: add `--dry-run` flag before deploying +- Alternative: git push to deployment repo with webhook triggers + +--- + +## Phase 2: Astro Starlight Project Setup (Required) + +### Directory Structure + +Create `/home/clawdie/clawdie-docs/` (or configure `CMS_DOCS_SITE_PATH` in `.env`): + +``` +/home/clawdie/clawdie-docs/ + ├── astro.config.mjs # Multi-locale Starlight config + ├── package.json + ├── src/ + │ ├── content/ + │ │ ├── config.ts # Content collections config + │ │ ├── docs/ + │ │ │ ├── index.mdx # SLO root + │ │ │ ├── en/ # English docs (symlink from docs/public/en/) + │ │ │ ├── de/ # German (symlink from docs/public/de/) + │ │ │ └── [others...] + │ │ └── i18n/ # UI translations (t() function) + │ │ ├── en.json + │ │ ├── de.json + │ │ └── sl.json + │ ├── components/ # Custom overrides + │ └── styles/ + └── dist/ # Build output +``` + +### astro.config.mjs Example + +```javascript +// astro.config.mjs +import { defineConfig } from 'astro/config'; +import starlight from '@astrojs/starlight'; + +export default defineConfig({ + site: 'https://docs.clawdie.si', + integrations: [ + starlight({ + title: 'Clawdie AI', + defaultLocale: 'sl', + locales: { + sl: { label: '🇸🇮 Slovenščina', lang: 'sl' }, + en: { label: '🇬🇧 English', lang: 'en' }, + de: { label: '🇩🇪 Deutsch', lang: 'de' }, + hr: { label: '🇭🇷 Hrvatski', lang: 'hr' }, + sr: { label: '🇷🇸 Српски', lang: 'sr' }, + ru: { label: '🇷🇺 Русский', lang: 'ru' }, + }, + sidebar: [ + { + label: 'Namestitev', + translations: { + en: 'Installation', + de: 'Installation', + hr: 'Instalacija', + sr: 'Инсталација', + ru: 'Установка', + }, + items: [ + { slug: 'install', label: 'Pregled', translations: { en: 'Overview' } }, + { slug: 'install/requirements', label: 'Zahteve' }, + ], + }, + { + label: 'Arhitektura', + translations: { + en: 'Architecture', + de: 'Architektur', + hr: 'Arhitektura', + sr: 'Архитектура', + ru: 'Архитектура', + }, + items: [ + { slug: 'architecture', label: 'Pregled' }, + { slug: 'architecture/jail-networking' }, + ], + }, + ], + }), + ], +}); +``` + +### Symlink Strategy + +Instead of copying translated docs, use symlinks to avoid duplication: + +```bash +cd /home/clawdie/clawdie-docs/src/content/docs/ +ln -s /home/clawdie/clawdie-ai/docs/public/en en +ln -s /home/clawdie/clawdie-ai/docs/public/de de +ln -s /home/clawdie/clawdie-ai/docs/public/hr hr +# ... etc +``` + +This way: + +- Crowdin pulls translations to one location +- Astro builds from symlinks +- No duplication, easier to update + +--- + +## Phase 3: Crowdin Project Configuration + +### Create Project (if not exists) + +Visit https://crowdin.com/dashboard and create new project: + +- **Project Name:** Clawdie-AI +- **Default Language:** English +- **Target Languages:** SL, DE, HR, SR, RU, EL, IT, MK, SK, BS + +### Configure .crowdin.yml + +Already configured in repo: + +```yaml +project_id: 883714 +api_token_env: CROWDIN_PERSONAL_TOKEN +base_path: ./ +files: + - source: docs/public/**.md + dest: docs/public/%two_letters_code% + translation_update: crowdin_move +``` + +### API Token Setup + +1. Create personal token at https://crowdin.com/settings/api/tokens +2. Store in **one** of: + - `.env` file: `CROWDIN_PERSONAL_TOKEN=xxx` + - `~/.config/clawdie/crowdin-token`: `xxx` + - GitHub Actions secret: `CROWDIN_PERSONAL_TOKEN` + +--- + +## Phase 4: Automation for Pi/Aider + +### Cron-Based Sync (Daily/Weekly) + +Add to `.env`: + +```bash +# Crowdin sync schedule (cron) +# "0 9 * * MON" = Every Monday at 9am +DOCS_SYNC_SCHEDULE="0 9 * * MON" +``` + +### Manual Execution (via Pi) + +Pi skill command: + +```bash +pi "sync docs translations and rebuild the site" +``` + +Aider harness command: + +```bash +aider --auto-commits \ + --message "Sync translations from Crowdin and rebuild docs" \ + "Crowdin sync and Astro rebuild required" +``` + +### GitHub Actions Workflow (Optional) + +Already created: `.github/workflows/crowdin-sync.yml` + +Triggers: + +- **Manual** (workflow_dispatch): Click "Run workflow" to sync manually +- **Scheduled**: Every Monday at 9 UTC +- **On push**: When `docs/public/` changes + +--- + +## MVP Checklist + +- [ ] **Crowdin Project (883714)** - Verify project exists and is configured +- [ ] **English Source Upload** - Run `./scripts/crowdin-sync.sh --push` to seed project +- [ ] **Translator Setup** - Add translators to Crowdin, assign languages +- [ ] **Astro Starlight Project** - Create at `/home/clawdie/clawdie-docs/` with starlight theme +- [ ] **astro.config.mjs** - Configure all target locales and sidebar +- [ ] **Symlinks** - Link `src/content/docs/{en,de,hr,sr,ru,el,it,mk,sk,bs}` to `docs/public/{lang}` +- [ ] **Local Build Test** - `npm run build` produces multilingual static site +- [ ] **Crowdin → Pull Test** - `./scripts/crowdin-sync.sh --pull` downloads SLO + EN +- [ ] **Full Pipeline Test** - Push → Pull → Build → Deploy in sequence +- [ ] **Pi Automation** - Create Pi skill to execute full pipeline +- [ ] **Deploy Target** - Configure nginx at `docs.clawdie.si` to serve `dist/` +- [ ] **Monitoring** - Check Crowdin progress weekly, trigger rebuild when >80% complete + +--- + +## Troubleshooting + +### "No translations available for {lang}" + +**Cause:** Language not configured in Crowdin or no translations yet. +**Fix:** Add language to Crowdin project settings, ensure translators have assignments. + +### "Failed to upload to storage" + +**Cause:** Invalid Crowdin token or API rate limit. +**Fix:** Verify token in `.env` or `~/.config/clawdie/crowdin-token`, wait 1 minute, retry. + +### "Astro build fails with missing locale" + +**Cause:** Translation directory not downloaded yet. +**Fix:** Run `./scripts/crowdin-sync.sh --pull` before build, or exclude incomplete locales from `astro.config.mjs`. + +### "Symlinks not found by Astro" + +**Cause:** Symlinks created in wrong location or Astro reading before symlinks established. +**Fix:** Verify symlinks exist at `src/content/docs/{lang}/`, restart dev server with `npm run dev`. + +--- + +## Next Steps (Post-MVP) + +### Phase 5: Strapi CMS Integration (Deferred) + +Once FreeBSD native binding issues are resolved (see `doc/STRAPI-FREEBSD-GOTCHA.md`): + +- Strapi as backend for rich media content (images, videos, structured data) +- Crowdin sync to Strapi API endpoints +- Astro fetches from Strapi for dynamic components +- CMS content types: Page, Guide, Skill, BlogPost + +Current blockers: + +- `sharp` (image processing) lacks FreeBSD prebuilt binary +- `esbuild`, `@swc/core` same issue +- Workaround: Compile libvips from /usr/ports, but time-intensive + +**Recommendation:** Keep Strapi in roadmap, ship markdown-only MVP first. + +### Phase 6: Real-Time Collaboration + +- Webhook from Crowdin to rebuild site immediately after translation milestone +- Slack notification on translation progress +- Discord bot for translator coordination + +--- + +## References + +- **Crowdin API:** https://developer.crowdin.com/api/ +- **Astro Starlight:** https://starlight.astro.build/ +- **Crowdin Sync Script:** `./scripts/crowdin-sync.sh` +- **FreeBSD CMS Gotcha:** `doc/STRAPI-FREEBSD-GOTCHA.md` diff --git a/.agent/skills/docs-localization-pipeline/orchestrate-pipeline.sh b/.agent/skills/docs-localization-pipeline/orchestrate-pipeline.sh new file mode 100755 index 0000000..c9f40f5 --- /dev/null +++ b/.agent/skills/docs-localization-pipeline/orchestrate-pipeline.sh @@ -0,0 +1,294 @@ +#!/bin/sh +# Three-Bird Docs Localization Pipeline Orchestrator +# Automates: Crowdin sync → Astro build → Deploy +# +# Usage: +# ./orchestrate-pipeline.sh --all # Full pipeline: push → pull → build → deploy +# ./orchestrate-pipeline.sh --pull # Pull translations only (no push) +# ./orchestrate-pipeline.sh --build # Pull + build only (no push/deploy) +# ./orchestrate-pipeline.sh --build-only # Build only (skip pull) +# ./orchestrate-pipeline.sh --deploy-only # Deploy existing build +# ./orchestrate-pipeline.sh --status # Check Crowdin status only +# +# Environment variables (from .env or CLI): +# CROWDIN_PERSONAL_TOKEN - Crowdin API token (required for push/pull) +# CMS_DOCS_SITE_PATH - Path to Astro project (default: /home/clawdie/clawdie-docs) +# DOCS_DEPLOY_TARGET - nginx webroot (default: /usr/local/www/clawdie/docs) +# DOCS_MIN_COMPLETION - Min % completion before deploy (default: 80) + +set -e + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +REPO_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)" +CMS_DOCS_SITE_PATH="${CMS_DOCS_SITE_PATH:-/home/clawdie/clawdie-docs}" +DOCS_DEPLOY_TARGET="${DOCS_DEPLOY_TARGET:-/usr/local/www/clawdie/docs}" +DOCS_MIN_COMPLETION="${DOCS_MIN_COMPLETION:-80}" + +# Colors +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +NC='\033[0m' + +log_info() { echo "${GREEN}[INFO]${NC} $1"; } +log_warn() { echo "${YELLOW}[WARN]${NC} $1"; } +log_err() { echo "${RED}[ERROR]${NC} $1" >&2; exit 1; } +log_section() { echo ""; echo "${BLUE}=== $1 ===${NC}"; } + +# ──────────────────────────────────────────────────────────────────────────── +# Helper functions +# ──────────────────────────────────────────────────────────────────────────── + +check_prereqs() { + local missing=0 + + if ! command -v node >/dev/null 2>&1; then + log_warn "Node.js not found" + missing=1 + fi + + if ! command -v npm >/dev/null 2>&1; then + log_warn "npm not found" + missing=1 + fi + + if [ ! -d "$REPO_ROOT" ]; then + log_err "Repository root not found: $REPO_ROOT" + fi + + if [ $missing -eq 1 ]; then + log_err "Missing prerequisites. Install with: pkg install node24 npm-node24" + fi +} + +check_crowdin_token() { + if [ -n "$CROWDIN_PERSONAL_TOKEN" ]; then + return 0 + elif [ -f ~/.config/clawdie/crowdin-token ]; then + return 0 + elif [ -f "$REPO_ROOT/.env" ] && grep -q CROWDIN_PERSONAL_TOKEN "$REPO_ROOT/.env"; then + return 0 + else + log_err "Crowdin token not found. Set CROWDIN_PERSONAL_TOKEN or create ~/.config/clawdie/crowdin-token" + fi +} + +get_crowdin_completion() { + # Returns average completion % across all languages + # Used to decide if deploy is safe + if ! check_crowdin_token 2>/dev/null; then + log_warn "Crowdin token not available, skipping completion check" + return 0 + fi + + cd "$REPO_ROOT" + local completion + completion=$("$REPO_ROOT/scripts/crowdin-sync.sh" --status 2>/dev/null | grep -oP '\(\K[0-9]+(?=%)' | awk '{sum+=$1; count++} END {if (count>0) printf "%.0f", sum/count; else print "0"}') + echo "$completion" +} + +# ──────────────────────────────────────────────────────────────────────────── +# Stage 1: Crowdin Push (upload English sources) +# ──────────────────────────────────────────────────────────────────────────── + +stage_crowdin_push() { + log_section "Stage 1: Push English Sources to Crowdin" + + check_crowdin_token || return 1 + + cd "$REPO_ROOT" + if ./scripts/crowdin-sync.sh --push; then + log_info "✓ English sources uploaded" + return 0 + else + log_err "✗ Failed to push sources" + return 1 + fi +} + +# ──────────────────────────────────────────────────────────────────────────── +# Stage 2: Crowdin Pull (download translations) +# ──────────────────────────────────────────────────────────────────────────── + +stage_crowdin_pull() { + log_section "Stage 2: Pull Translations from Crowdin" + + check_crowdin_token || return 1 + + # Check completion % + local completion + completion=$(get_crowdin_completion) + + if [ "$completion" -lt "$DOCS_MIN_COMPLETION" ]; then + log_warn "Translation completion: ${completion}% (minimum: ${DOCS_MIN_COMPLETION}%)" + log_warn "Proceeding anyway (use --force to skip this check)" + else + log_info "Translation completion: ${completion}% ✓" + fi + + cd "$REPO_ROOT" + if ./scripts/crowdin-sync.sh --pull; then + log_info "✓ Translations downloaded" + return 0 + else + log_err "✗ Failed to pull translations" + return 1 + fi +} + +# ──────────────────────────────────────────────────────────────────────────── +# Stage 3: Astro Build +# ──────────────────────────────────────────────────────────────────────────── + +stage_astro_build() { + log_section "Stage 3: Build Astro Site" + + if [ ! -d "$CMS_DOCS_SITE_PATH" ]; then + log_warn "Astro project not found at: $CMS_DOCS_SITE_PATH" + log_info "Setting up Astro Starlight..." + + if [ -x "$SCRIPT_DIR/setup-clawdie-docs.sh" ]; then + "$SCRIPT_DIR/setup-clawdie-docs.sh" "$CMS_DOCS_SITE_PATH" || log_err "Failed to setup Astro" + else + log_err "Setup script not found: $SCRIPT_DIR/setup-clawdie-docs.sh" + fi + fi + + if [ ! -d "$CMS_DOCS_SITE_PATH" ]; then + log_err "Astro path still missing: $CMS_DOCS_SITE_PATH" + fi + + log_info "Building Astro site at: $CMS_DOCS_SITE_PATH" + + cd "$CMS_DOCS_SITE_PATH" + + if [ ! -f package.json ]; then + log_err "package.json not found in Astro project" + fi + + if ! npm run build; then + log_err "✗ Astro build failed" + return 1 + fi + + if [ ! -d dist ]; then + log_err "Build output not found: $CMS_DOCS_SITE_PATH/dist" + fi + + log_info "✓ Site built: $(find dist -type f | wc -l) files" + return 0 +} + +# ──────────────────────────────────────────────────────────────────────────── +# Stage 4: Deploy +# ──────────────────────────────────────────────────────────────────────────── + +stage_deploy() { + log_section "Stage 4: Deploy to Nginx" + + if [ ! -d "$CMS_DOCS_SITE_PATH/dist" ]; then + log_err "Build output not found. Run build first: $CMS_DOCS_SITE_PATH/dist" + fi + + mkdir -p "$DOCS_DEPLOY_TARGET" + + log_info "Syncing files from $CMS_DOCS_SITE_PATH/dist/ → $DOCS_DEPLOY_TARGET" + log_info "(Using rsync for incremental updates and integrity checking)" + + # Dry-run first to show what will change + log_info "Preview (dry-run):" + if ! sudo rsync -av --delete --checksum --dry-run "$CMS_DOCS_SITE_PATH/dist/" "$DOCS_DEPLOY_TARGET" 2>/dev/null | head -20; then + log_warn "⚠ rsync preview failed, attempting actual sync anyway" + fi + + # Actual sync with verification + if sudo rsync -av --delete --checksum "$CMS_DOCS_SITE_PATH/dist/" "$DOCS_DEPLOY_TARGET"; then + log_info "✓ Files deployed (rsync: incremental, verified, cleaned up stale assets)" + else + log_err "✗ Deploy failed (check sudo permissions or rsync availability)" + fi + + # Reload nginx (optional) + if command -v nginx >/dev/null 2>&1 && sudo nginx -t 2>/dev/null; then + if sudo nginx -s reload 2>/dev/null; then + log_info "✓ Nginx reloaded" + else + log_warn "⚠ Failed to reload nginx (may need manual: sudo nginx -s reload)" + fi + else + log_warn "⚠ nginx not found or not running" + fi + + log_info "✓ Deploy complete: https://docs.clawdie.si" + return 0 +} + +# ──────────────────────────────────────────────────────────────────────────── +# Main +# ──────────────────────────────────────────────────────────────────────────── + +main() { + local action="${1:---all}" + + check_prereqs + + case "$action" in + --all) + log_info "Running full pipeline: push → pull → build → deploy" + stage_crowdin_push || exit 1 + stage_crowdin_pull || exit 1 + stage_astro_build || exit 1 + stage_deploy || exit 1 + log_section "Pipeline Complete" + log_info "✓ Docs published at https://docs.clawdie.si" + ;; + --push) + stage_crowdin_push || exit 1 + ;; + --pull) + stage_crowdin_pull || exit 1 + ;; + --build) + stage_crowdin_pull || exit 1 + stage_astro_build || exit 1 + ;; + --build-only) + stage_astro_build || exit 1 + ;; + --deploy-only) + stage_deploy || exit 1 + ;; + --status) + log_section "Crowdin Status" + check_crowdin_token && cd "$REPO_ROOT" && ./scripts/crowdin-sync.sh --status || log_warn "Crowdin token not available" + ;; + --help|-h) + echo "Three-Bird Docs Pipeline Orchestrator" + echo "" + echo "Usage: $0 {--all|--push|--pull|--build|--build-only|--deploy-only|--status}" + echo "" + echo "Stages:" + echo " --push Push English sources to Crowdin" + echo " --pull Pull translations from Crowdin" + echo " --build Pull translations + build Astro site" + echo " --build-only Build Astro site only (skip pull)" + echo " --deploy-only Deploy existing build" + echo " --all Full pipeline (push → pull → build → deploy)" + echo " --status Check Crowdin status only" + echo "" + echo "Environment variables:" + echo " CROWDIN_PERSONAL_TOKEN - Crowdin API token" + echo " CMS_DOCS_SITE_PATH - Path to Astro project (default: /home/clawdie/clawdie-docs)" + echo " DOCS_DEPLOY_TARGET - Deploy path (default: /usr/local/www/clawdie/docs)" + echo " DOCS_MIN_COMPLETION - Min % completion to deploy (default: 80)" + echo "" + exit 0 + ;; + *) + log_err "Unknown action: $action (use --help for usage)" + ;; + esac +} + +main "$@" diff --git a/.agent/skills/docs-localization-pipeline/setup-clawdie-docs.sh b/.agent/skills/docs-localization-pipeline/setup-clawdie-docs.sh new file mode 100755 index 0000000..558c7a2 --- /dev/null +++ b/.agent/skills/docs-localization-pipeline/setup-clawdie-docs.sh @@ -0,0 +1,377 @@ +#!/bin/sh +# Setup Astro Starlight Docs Site +# Creates the Astro project structure, configures Starlight, and sets up symlinks to Crowdin translations +# +# Usage: +# ./setup-clawdie-docs.sh # Use default path /home/clawdie/clawdie-docs +# ./setup-clawdie-docs.sh /custom/path # Use custom installation path +# +# Prerequisites: +# - Node.js >=20, npm +# - Docs source at /home/clawdie/clawdie-ai/docs/public/ +# - Crowdin translated dirs at /home/clawdie/clawdie-ai/docs/public/{en,de,hr,sr,ru,el,it,mk,sk,bs} + +set -e + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +REPO_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)" +ASTRO_PATH="${1:-/home/clawdie/clawdie-docs}" +DOCS_SOURCE="${REPO_ROOT}/docs/public" + +log_info() { echo "[INFO] $1"; } +log_err() { echo "[ERROR] $1" >&2; exit 1; } + +# ──────────────────────────────────────────────────────────────────────────── +# Verify prerequisites +# ──────────────────────────────────────────────────────────────────────────── + +if ! command -v node >/dev/null 2>&1; then + log_err "Node.js not found. Install with: pkg install node24" +fi + +if ! command -v npm >/dev/null 2>&1; then + log_err "npm not found. Install with: pkg install npm-node24" +fi + +if [ ! -d "$DOCS_SOURCE" ]; then + log_err "Docs source not found: $DOCS_SOURCE" +fi + +log_info "Prerequisites OK" + +# ──────────────────────────────────────────────────────────────────────────── +# Create Astro project +# ──────────────────────────────────────────────────────────────────────────── + +if [ -d "$ASTRO_PATH" ]; then + log_info "Astro path already exists: $ASTRO_PATH" + log_info "Updating configuration (skipping npm install)..." + UPDATE_ONLY=1 +else + log_info "Creating Astro project at: $ASTRO_PATH" + mkdir -p "$ASTRO_PATH" +fi + +cd "$ASTRO_PATH" + +# ──────────────────────────────────────────────────────────────────────────── +# Create package.json (minimal) +# ──────────────────────────────────────────────────────────────────────────── + +if [ ! -f package.json ] || [ -z "$UPDATE_ONLY" ]; then + log_info "Creating package.json..." + cat > package.json << 'EOF' +{ + "name": "clawdie-docs", + "version": "1.0.0", + "description": "Clawdie AI Documentation Site", + "type": "module", + "scripts": { + "dev": "astro dev", + "build": "astro build", + "preview": "astro preview" + }, + "dependencies": { + "astro": "^4.13.0", + "@astrojs/starlight": "^0.23.0", + "typescript": "^5.0.0" + } +} +EOF + log_info "Installing dependencies..." + npm install +fi + +# ──────────────────────────────────────────────────────────────────────────── +# Create astro.config.mjs +# ──────────────────────────────────────────────────────────────────────────── + +log_info "Creating astro.config.mjs..." +cat > astro.config.mjs << 'EOF' +// astro.config.mjs +import { defineConfig } from 'astro/config'; +import starlight from '@astrojs/starlight'; + +export default defineConfig({ + site: 'https://docs.clawdie.si', + integrations: [ + starlight({ + title: 'Clawdie AI', + description: 'Personal AI Assistant Documentation', + defaultLocale: 'sl', + locales: { + sl: { + label: '🇸🇮 Slovenščina', + lang: 'sl', + }, + en: { + label: '🇬🇧 English', + lang: 'en', + }, + de: { + label: '🇩🇪 Deutsch', + lang: 'de', + }, + hr: { + label: '🇭🇷 Hrvatski', + lang: 'hr', + }, + sr: { + label: '🇷🇸 Српски', + lang: 'sr', + }, + ru: { + label: '🇷🇺 Русский', + lang: 'ru', + }, + el: { + label: '🇬🇷 Ελληνικά', + lang: 'el', + }, + it: { + label: '🇮🇹 Italiano', + lang: 'it', + }, + mk: { + label: '🇲🇰 Македонски', + lang: 'mk', + }, + sk: { + label: '🇸🇰 Slovenčina', + lang: 'sk', + }, + bs: { + label: '🇧🇦 Bosanski', + lang: 'bs', + }, + }, + sidebar: [ + { + label: 'Namestitev', + translations: { + en: 'Installation', + de: 'Installation', + hr: 'Instalacija', + sr: 'Инсталација', + ru: 'Установка', + el: 'Εγκατάσταση', + it: 'Installazione', + mk: 'Инсталација', + sk: 'Inštalácia', + bs: 'Instalacija', + }, + items: [ + { + slug: 'install', + label: 'Pregled', + translations: { + en: 'Overview', + de: 'Überblick', + hr: 'Pregled', + sr: 'Преглед', + ru: 'Обзор', + }, + }, + { + slug: 'install/requirements', + label: 'Zahteve', + translations: { + en: 'Requirements', + de: 'Anforderungen', + hr: 'Zahtjevi', + }, + }, + { + slug: 'install/install', + label: 'Namestitev', + translations: { + en: 'Installation', + }, + }, + { + slug: 'install/iso', + label: 'Clawdie-ISO', + translations: { + en: 'Clawdie-ISO', + }, + }, + ], + }, + { + label: 'Arhitektura', + translations: { + en: 'Architecture', + de: 'Architektur', + hr: 'Arhitektura', + sr: 'Архитектура', + ru: 'Архитектура', + }, + items: [ + { + slug: 'architecture', + label: 'Pregled', + translations: { + en: 'Overview', + }, + }, + { + slug: 'architecture/jail-networking', + label: 'Jail Networking', + }, + { + slug: 'architecture/bastille', + label: 'Bastille', + }, + { + slug: 'architecture/controlplane', + label: 'Controlplane', + }, + ], + }, + { + label: 'Referenca', + translations: { + en: 'Reference', + de: 'Referenz', + hr: 'Referencija', + sr: 'Референца', + ru: 'Справка', + }, + items: [ + { + slug: 'reference', + label: 'Pregled', + translations: { + en: 'Overview', + }, + }, + ], + }, + ], + social: { + github: 'https://codeberg.org/Clawdie/Clawdie-AI', + }, + customCss: [], + }), + ], +}); +EOF +log_info "Created astro.config.mjs" + +# ──────────────────────────────────────────────────────────────────────────── +# Create directory structure +# ──────────────────────────────────────────────────────────────────────────── + +log_info "Creating directory structure..." +mkdir -p src/content/docs +mkdir -p src/content/i18n +mkdir -p src/components +mkdir -p src/styles + +# ──────────────────────────────────────────────────────────────────────────── +# Create content config +# ──────────────────────────────────────────────────────────────────────────── + +log_info "Creating content collections config..." +cat > src/content/config.ts << 'EOF' +// src/content/config.ts +import { docsLoader, i18nLoader } from '@astrojs/starlight/loaders'; +import { docsSchema, i18nSchema } from '@astrojs/starlight/schema'; +import { defineCollection } from 'astro:content'; + +export const collections = { + docs: defineCollection({ + loader: docsLoader(), + schema: docsSchema(), + }), + i18n: defineCollection({ + loader: i18nLoader(), + schema: i18nSchema(), + }), +}; +EOF +log_info "Created src/content/config.ts" + +# ──────────────────────────────────────────────────────────────────────────── +# Create symlinks to translation directories +# ──────────────────────────────────────────────────────────────────────────── + +log_info "Setting up symlinks to Crowdin translations..." + +cd "src/content/docs" + +for lang in en de hr sr ru el it mk sk bs; do + if [ -L "$lang" ]; then + log_info " $lang: symlink already exists" + elif [ -d "$lang" ]; then + log_info " $lang: directory exists, removing for symlink..." + rm -rf "$lang" + ln -s "${DOCS_SOURCE}/${lang}" "$lang" + log_info " $lang: created symlink" + else + log_info " $lang: creating symlink..." + ln -s "${DOCS_SOURCE}/${lang}" "$lang" 2>/dev/null || log_err "Failed to create symlink for $lang" + log_info " $lang: created" + fi +done + +log_info "Symlinks created" + +# ──────────────────────────────────────────────────────────────────────────── +# Create i18n translations +# ──────────────────────────────────────────────────────────────────────────── + +cd "$ASTRO_PATH" + +log_info "Creating UI translations..." +cat > src/content/i18n/sl.json << 'EOF' +{ + "search.label": "Iskanje", + "search.cancelLabel": "Prekini", + "themeSelect.accessibleLabel": "Izbira teme", + "tableOfContents.onThisPage": "Na tej strani" +} +EOF + +cat > src/content/i18n/en.json << 'EOF' +{ + "search.label": "Search", + "search.cancelLabel": "Cancel", + "themeSelect.accessibleLabel": "Select theme", + "tableOfContents.onThisPage": "On this page" +} +EOF + +cat > src/content/i18n/de.json << 'EOF' +{ + "search.label": "Suche", + "search.cancelLabel": "Abbrechen", + "themeSelect.accessibleLabel": "Thema wählen", + "tableOfContents.onThisPage": "Auf dieser Seite" +} +EOF + +log_info "Created i18n translations" + +# ──────────────────────────────────────────────────────────────────────────── +# Create tsconfig +# ──────────────────────────────────────────────────────────────────────────── + +log_info "Creating tsconfig.json..." +cat > tsconfig.json << 'EOF' +{ + "extends": "astro/tsconfigs/strict" +} +EOF + +log_info "Setup complete!" +log_info "" +log_info "Next steps:" +log_info " 1. cd $ASTRO_PATH" +log_info " 2. npm run dev # Start dev server at http://localhost:4321" +log_info " 3. npm run build # Build for production" +log_info "" +log_info "To sync translations from Crowdin first:" +log_info " cd ${REPO_ROOT}" +log_info " ./scripts/crowdin-sync.sh --pull # Downloads translated docs" +log_info "" diff --git a/.agent/skills/freebsd-admin/SKILL.md b/.agent/skills/freebsd-admin/SKILL.md new file mode 100644 index 0000000..b37cc77 --- /dev/null +++ b/.agent/skills/freebsd-admin/SKILL.md @@ -0,0 +1,92 @@ +--- +name: freebsd-admin +description: Make host-level FreeBSD system changes for Clawdie. Use when changing rc.conf, sysctl state, service lifecycle, host routing, gateway_enable, bridge persistence, or other machine-wide settings that should stay outside the jail and VM-specific skills. +--- + +# FreeBSD Admin + +Use this skill for host-level administrative changes on the FreeBSD machine. + +This skill is intentionally separate from jail and VM skills. It owns system +state, not workload-specific provisioning. + +Use it to decide what host state should exist. If the same change should become +repeatable, hand execution off to `ansible-freebsd`. + +## Scope + +This skill covers: + +- `sysrc` +- `service` +- `sysctl` +- `rsync` (file sync and deployment — see `/rsync` skill for patterns) +- `gateway_enable` +- host routing and forwarding state +- persistent host resolver and network base state needed by Bastille jails +- reboot-safe host persistence checks +- host-side service identities needed for shared jail-mounted tooling +- bhyve/VMM kernel module setup for VM support +- RCTL kernel configuration for resource limits +- host locale configuration (`~/.login_conf`, `cap_mkdb`) +- FreeBSD base update reboot readiness and post-reboot validation + +This skill does not replace: + +- `nginx` or jail-specific service skills for workload-level changes +- `service-restart` for targeted service lifecycle work +- runtime/setup source docs for current jail bootstrap and bridge layout +- `ansible-freebsd` for repeatable host execution +- product- or VM-specific docs for guest design work + +## Privileged delegation boundary + +At runtime, privileged host operations from the agent go through +`hostd` — a root daemon on `/var/run/-hostd.sock`. The agent +user calls `hostd(op, params)` from `src/hostd/client.ts`; the daemon runs the +whitelisted op handler as root. + +When making a one-off host change interactively (outside the agent runtime), +use `sudo` directly as usual. When deciding whether a host change should become +a hostd op, consider: will the agent need to trigger this at runtime without +operator input? If yes, add it as a whitelisted op in `src/hostd/privileged-commands.ts`. + +For update-status questions, use the existing read-only hostd audit ops +(`pkg-version`, `pkg-audit`, `freebsd-update-status`, `freebsd-version`) via +the sysadmin update-report path. Do not expose `freebsd-update fetch` or run +mutating update commands for a status report. + +## Tailscale controlplane exposure + +When the controlplane API/dashboard is only exposed on Tailscale: + +- allow `tailscale0` ingress to ports `3100` (direct API) and `443` (nginx proxy) +- validate PF before reload (`sudo pfctl -nf /etc/pf.conf`) and then `sudo service pf reload` + +## Workflow + +1. Read `references/system-changes.md` +2. Read `references/network-forwarding.md` +3. If the host uses Tailscale or specialized DNS, read `references/resolver-baseline.md` +4. Read `references/execution-modes.md` +5. If the work should move into Ansible, read `references/ansible-handoff.md` +6. If the host needs shared service identities, read `references/service-identities.md` +7. If rollback matters before a change, read `references/rollback-patterns.md` +8. If needed, read `references/validation.md` +9. If configuring VM support or RCTL, read `references/bhyve-prerequisites.md` +10. If setting host locale after onboarding, read `references/locale-setup.md` +11. If memory budgeting for jails/VMs, read `references/memory-budget.md` +12. If modifying `/boot/loader.conf`, read `references/loader-conf.md` +13. If modifying `/etc/rc.conf`, read `references/rc-conf.md` +14. If a FreeBSD base or package update may require reboot, read `references/freebsd-update-reboot.md` +15. Prefer the scripts in `scripts/` for canonical host-side command bundles + +## Safe defaults + +- keep host state changes explicit and reversible +- use `sysrc` for persistent changes +- use `sysctl` for immediate runtime changes +- validate after every host change before blaming Bastille or the jail +- never reboot autonomously; capture state, explain why reboot is needed, and wait for explicit operator go-ahead +- prefer Ansible once a host change stops being one-off and becomes standard ops +- keep service-user UID/GID choices explicit when host-mounted jail paths depend on numeric ownership diff --git a/.agent/skills/freebsd-admin/references/ansible-handoff.md b/.agent/skills/freebsd-admin/references/ansible-handoff.md new file mode 100644 index 0000000..5b0ddf3 --- /dev/null +++ b/.agent/skills/freebsd-admin/references/ansible-handoff.md @@ -0,0 +1,42 @@ +# Ansible Handoff + +Use this file when a host-side FreeBSD fix should become repeatable through +Ansible. + +## Current preferred playbooks + +- `infra/ansible/playbooks/base-host.yaml` +- `infra/ansible/playbooks/host-preflight.yaml` + +## Promotion path + +1. prove the host command sequence manually +2. keep the change host-scoped if it touches `rc.conf`, `sysctl`, PF, Bastille, or resolver state +3. move the stable commands into the smallest matching playbook +4. validate from the host, not from inside a jail + +## Good Ansible handoff candidates + +- `gateway_enable="YES"` +- forwarding persistence +- Bastille resolver baseline +- host package baseline +- `warden0` validation +- PF include and reload validation +- jail creation and bootstrap through `bastille` / `bastille cmd` + +## Not an Ansible handoff target by default + +- enabling jail `sshd` +- creating an operator jail +- resurrecting the removed operator-jail model +- mixing host PF policy with app-level nginx design in one playbook + +## Next likely playbooks + +- `base-host.yaml` +- `host-pf-baseline.yaml` +- `git-jail-bootstrap.yaml` +- `jail-cms-create.yaml` +- `jail-cms-bootstrap.yaml` +- `cms-strapi-bootstrap.yaml` diff --git a/.agent/skills/freebsd-admin/references/bhyve-prerequisites.md b/.agent/skills/freebsd-admin/references/bhyve-prerequisites.md new file mode 100644 index 0000000..5c6e3be --- /dev/null +++ b/.agent/skills/freebsd-admin/references/bhyve-prerequisites.md @@ -0,0 +1,101 @@ +# Bhyve/VMM Prerequisites + +Required kernel modules and configuration for running bhyve VMs on the FreeBSD host. + +## Kernel Modules + +### Required (loader.conf) + +```bash +# /boot/loader.conf +vmm_load="YES" # Hypervisor kernel module +nmdm_load="YES" # Null modem for VM serial console +``` + +### Verification + +```bash +# Check if VMM is loaded +kldstat | grep vmm + +# Check VMM capabilities +sysctl hw.vmm + +# Check for VMX/SVM support +dmesg | grep -iE "VMX|SVM|EPT" +``` + +## Resource Requirements + +| Resource | Minimum | Recommended | +| --------- | ------- | ----------- | +| CPU cores | 2 | 4+ | +| RAM | 4GB | 8GB+ | +| Disk | 20GB | 50GB+ | + +## Nested Virtualization Notes + +If running on a KVM guest (nested virtualization): + +| Feature | Works | +| ----------------- | ---------------- | +| Headless VMs | ✅ | +| VNC/SPICE display | ✅ | +| GPU passthrough | ❌ | +| Windows GUI VMs | ✅ (via VNC/RDP) | + +## ZFS Dataset Layout + +Recommended structure for VM storage: + +```bash +sudo zfs create -o mountpoint=/var/bhyve zroot/bhyve +sudo zfs create -o mountpoint=/var/bhyve/disks zroot/bhyve/disks +sudo zfs create -o mountpoint=/var/bhyve/iso zroot/bhyve/iso +``` + +## Networking Options + +### Option A: Share existing bridge + +```bash +# Add VM tap interface to existing warden0 bridge +ifconfig warden0 addm tap0 +``` + +### Option B: Dedicated VM bridge + +```bash +# /etc/rc.conf +cloned_interfaces+="vm0" +ifconfig_vm0="inet 10.0.2.1/24" +``` + +## RCTL Memory Limits for VMs + +Same RCTL kernel requirement as jail memory limits: + +```bash +# /boot/loader.conf +kern.racct.enable="1" +``` + +Apply limits to bhyve processes: + +```bash +# Example: 4GB limit for a VM +rctl -a process:bhyve:memoryuse:deny=4G +``` + +## Related Skills + +- `browser-vm` - Linux browser VM planning +- `freebsd-admin` - Host-level changes +- `warden-zfs` - ZFS dataset management + +## References + +- FreeBSD Handbook: [Virtualization](https://docs.freebsd.org/en/books/handbook/virtualization/) +- bhyve(8) +- vmm(4) +- nmdm(4) diff --git a/.agent/skills/freebsd-admin/references/execution-modes.md b/.agent/skills/freebsd-admin/references/execution-modes.md new file mode 100644 index 0000000..c60577c --- /dev/null +++ b/.agent/skills/freebsd-admin/references/execution-modes.md @@ -0,0 +1,45 @@ +# Execution Modes + +Use this file to decide whether a host change should stay a direct command or +move into Ansible. + +## 1. Direct host mode + +Use direct commands when: + +- diagnosing a live problem +- proving a fix for the first time +- checking current host state +- making a one-off emergency repair + +Examples: + +- `sysctl net.inet.ip.forwarding` +- `service pf status` +- `ifconfig warden0` +- `sysrc gateway_enable="YES"` + +## 2. Ansible mode + +Use Ansible when: + +- the same host change will be repeated +- the host baseline should become documented policy +- the change belongs in standard host operations +- validation should be reproducible + +Examples: + +- forwarding baseline +- resolver baseline for Bastille +- nginx package/service state +- future repeatable host baseline or service-jail bootstrap + +## Rule of thumb + +1. inspect manually +2. prove the fix +3. codify it in Ansible if it should persist as standard practice + +Do not hide unexplained host behavior behind Ansible. Understand the host fix +first, then automate it. diff --git a/.agent/skills/freebsd-admin/references/freebsd-update-reboot.md b/.agent/skills/freebsd-admin/references/freebsd-update-reboot.md new file mode 100644 index 0000000..101f001 --- /dev/null +++ b/.agent/skills/freebsd-admin/references/freebsd-update-reboot.md @@ -0,0 +1,95 @@ +# FreeBSD Base Update Reboot Handoff + +Use this reference after FreeBSD base or package updates, and whenever the +operator asks whether a reboot is required. + +## Reboot-needed rule + +A reboot is required when the installed kernel/userland version is newer than +the running kernel: + +```sh +freebsd-version -k # installed kernel +freebsd-version -u # installed userland +uname -r # running kernel +``` + +If `freebsd-version -k` or `freebsd-version -u` reports a newer patch level than +`uname -r`, the update is staged but not fully active. Report that plainly and +ask the operator for an explicit reboot go-ahead. Do not reboot autonomously. + +Example interpretation: + +```text +freebsd-version -k: 15.0-RELEASE-p9 +freebsd-version -u: 15.0-RELEASE-p9 +uname -r: 15.0-RELEASE-p8 +``` + +This means the system has p9 installed but is still running a p8 kernel. A reboot +is required to complete the update. + +## Pre-reboot status capture + +Before recommending or handing off a reboot, capture enough state for the next +agent/operator to compare after boot: + +```sh +hostname +freebsd-version -k +freebsd-version -u +uname -r +/usr/sbin/service clawdie status +/usr/sbin/service nginx status +/usr/sbin/service postgresql status +/usr/sbin/jls +/sbin/pfctl -s info +``` + +Use absolute paths for base-system tools when the agent shell has a narrow PATH. +Unprivileged agents may see permission errors for service internals, PostgreSQL, +or PF. Record those as permission-limited checks rather than claiming the service +is down. + +## Package/service considerations + +A same-major PostgreSQL package upgrade, such as `postgresql18-server` 18.3 → +18.4, does not require dump/restore. It still benefits from a reboot or service +restart so the new binaries are loaded. + +If `nginx`, `tailscale`, PostgreSQL, or other long-running daemons were upgraded, +prefer a controlled reboot over piecemeal restarts unless the operator asks for a +minimal-disruption restart plan. + +## Post-reboot verification + +After the operator confirms the host is back, verify: + +```sh +freebsd-version -k +freebsd-version -u +uname -r +/usr/sbin/service clawdie status +/usr/sbin/service nginx status +/usr/sbin/service postgresql status +/usr/sbin/jls +/sbin/pfctl -s info +``` + +Expected after a successful reboot: `freebsd-version -k`, `freebsd-version -u`, +and `uname -r` all report the same patch level. + +Also verify application-specific readiness that matters for the current work: + +- Clawdie control plane reachable/running +- Forgejo reachable if git work is active +- jails are running (`cms`, `worker`, or whatever the host normally owns) +- PF enabled and rules loaded +- Tailscale reachable if remote agents depend on it + +## Vulnerability audit wording + +If `pkg audit` still reports vulnerable packages after an upgrade, do not imply +the upgrade failed. Say that the applied upgrade completed, but unrelated +packages remain vulnerable until fixed packages are available or the operator +chooses a ports/package remediation path. diff --git a/.agent/skills/freebsd-admin/references/loader-conf.md b/.agent/skills/freebsd-admin/references/loader-conf.md new file mode 100644 index 0000000..9298fa3 --- /dev/null +++ b/.agent/skills/freebsd-admin/references/loader-conf.md @@ -0,0 +1,124 @@ +# /boot/loader.conf Settings + +Key loader.conf settings for Clawdie FreeBSD hosts. + +## Boot Behavior + +### `beastie_disable="YES"` + +**NOT related to Bastille jails!** Beastie is the FreeBSD mascot. + +| Setting | Effect | +| ----------------------- | --------------------------------------- | +| `beastie_disable="YES"` | Skip graphical boot menu, boot directly | +| `beastie_disable="NO"` | Show beastie boot menu with options | + +**Use `YES` for headless servers:** + +- No local display/monitor +- Faster boot (no menu timeout) +- Cleaner console output +- Remote SSH management only + +**Use `NO` for:** + +- Workstations with local display +- Systems that need boot menu access (single user mode, etc.) + +### `autoboot_delay="-1"` + +| Value | Behavior | +| ----- | -------------------------------- | +| `-1` | No delay, boot immediately | +| `0` | No delay, but check for keypress | +| `N` | Wait N seconds before booting | + +### `console="comconsole,vidconsole"` + +Enables console output on both serial (`comconsole`) and video (`vidconsole`). Essential for headless servers with serial access. + +--- + +## Kernel Modules + +### `zfs_load="YES"` + +Loads ZFS kernel module early in boot. Required for ZFS root pools. + +### `vmm_load="YES"` + +Loads bhyve hypervisor kernel module. Required for running VMs. + +### `nmdm_load="YES"` + +Loads null modem module for VM serial console access via `/dev/nmdm*`. + +--- + +## Resource Management + +### `kern.racct.enable="1"` + +Enables RCTL (Resource Controls) for: + +- Jail memory limits (`rctl -a jail:name:memoryuse:deny=4G`) +- Process-level resource constraints +- VM memory limits + +**Requires reboot** to change. Cannot be toggled at runtime. + +### `vfs.zfs.arc_max="4294967296"` + +ZFS ARC (Adaptive Replacement Cache) maximum size in bytes. +Example: `4294967296` = 4GB + +Set appropriately based on total RAM: + +- 8GB system: 2-3GB ARC +- 16GB system: 4-6GB ARC +- 32GB system: 8-12GB ARC + +--- + +## Storage + +### `kern.geom.label.disk_ident.enable="0"` + +Disables GEOM disk identity labels. Prevents device name churn from disk identifiers. + +--- + +## Example Full Configuration + +``` +# Storage +zfs_load=YES +kern.geom.label.disk_ident.enable=0 + +# Boot behavior (headless server) +autoboot_delay="-1" +beastie_disable="YES" +loader_logo="none" +console="comconsole,vidconsole" + +# ZFS ARC limit (4GB) +vfs.zfs.arc_max="4294967296" + +# Resource controls for jails/VMs +kern.racct.enable=1 + +# Bhyve virtualization +vmm_load="YES" +nmdm_load="YES" +``` + +--- + +## Common Confusion: Beastie vs Bastille + +| Term | What It Is | +| ------------ | ---------------------------------------------------------- | +| **Beastie** | FreeBSD mascot (red daemon), boot menu | +| **Bastille** | Jail management tool (`bastille create`, `bastille start`) | + +`beastie_disable="YES"` does **NOT** affect Bastille jails in any way. diff --git a/.agent/skills/freebsd-admin/references/locale-setup.md b/.agent/skills/freebsd-admin/references/locale-setup.md new file mode 100644 index 0000000..137a458 --- /dev/null +++ b/.agent/skills/freebsd-admin/references/locale-setup.md @@ -0,0 +1,95 @@ +# Host Locale Setup + +Set the operator locale on the FreeBSD host after onboarding selection. + +## Why login.conf, not /etc/profile + +FreeBSD does not use `/etc/environment` or `/etc/profile` for locale the way +Linux does. The canonical mechanism is `login.conf`. Without it, every new +session reverts to `C.UTF-8` regardless of what is in `.env`. + +## Pattern + +### User scope (no sudo required — default for Clawdie setup) + +```sh +# 1. Write ~/.login_conf +cat > ~/.login_conf << 'EOF' +me:\ + :charset=UTF-8:\ + :lang=de_DE.UTF-8: +EOF + +# 2. Rebuild the login.conf database +cap_mkdb ~/.login_conf +``` + +### System-wide (requires root — only if the whole machine needs it) + +```sh +# Edit /etc/login.conf — find the default: class and add: +# :lang=de_DE.UTF-8:\ +# :charset=UTF-8:\ +sudo cap_mkdb /etc/login.conf +``` + +Prefer user scope for Clawdie deployments. Only use system-wide if multiple +users share the machine and all need the same locale. + +## Activation + +`login.conf` changes take effect at the **next login or tmux server restart**. +They do NOT change the current shell session automatically. + +Since Clawdie setup runs inside tmux, always tell the operator after locale change: + +> Respawn tmux to activate: `tmux kill-server && tmux` + +To verify after respawn: + +```sh +locale +# LANG=de_DE.UTF-8 +echo $LANG +``` + +## Soft approach (no respawn) + +When you cannot interrupt the session: + +```sh +# Patch tmux server env — takes effect in new windows/panes +tmux setenv -g LANG de_DE.UTF-8 +tmux setenv -g LC_ALL de_DE.UTF-8 + +# Patch current pane immediately +export LANG=de_DE.UTF-8 +export LC_ALL=de_DE.UTF-8 +``` + +See `/tmux-screenshot` skill → Session Lifecycle for the full decision table. + +## Charset extraction + +The charset comes from the system locale string: + +| System locale | charset | +| ------------- | --------------------------------- | +| `de_DE.UTF-8` | `UTF-8` | +| `it_IT.UTF-8` | `UTF-8` | +| `sl_SI.UTF-8` | `UTF-8` | +| `ru_RU.UTF-8` | `UTF-8` | +| `C.UTF-8` | neutral — do not write login.conf | + +If the locale is neutral (`C`, `POSIX`, `C.UTF-8`), skip writing `~/.login_conf`. +Nothing useful to set. + +## Rollback + +To revert to the default neutral locale: + +```sh +rm -f ~/.login_conf ~/.login_conf.db +``` + +New sessions will inherit `C.UTF-8` from the system default. diff --git a/.agent/skills/freebsd-admin/references/memory-budget.md b/.agent/skills/freebsd-admin/references/memory-budget.md new file mode 100644 index 0000000..f992d87 --- /dev/null +++ b/.agent/skills/freebsd-admin/references/memory-budget.md @@ -0,0 +1,77 @@ +# Host Memory Budget + +Total physical RAM: 12 GB + +## Allocation + +| Component | Budget | Notes | +| ------------------------ | ------ | ------------------------------------------------ | +| ZFS ARC | 4 GB | `vfs.zfs.arc_max=4294967296` in loader.conf | +| Host OS + nginx + system | 1.5 GB | base system, sshd, tailscale, cron | +| git jail | 512 MB | local bare repositories, minimal service surface | +| db jail | 1.5 GB | PostgreSQL 18 + pgvector | +| cms jail | 1 GB | Strapi + Astro builds | +| Headroom | 2 GB | burst capacity, agent processes | + +## Enforcement + +### ZFS ARC (enforced) + +Set live: + +```sh +sudo sysctl vfs.zfs.arc_max=4294967296 +``` + +Persisted in `/boot/loader.conf`: + +``` +vfs.zfs.arc_max="4294967296" # 4 GB +``` + +### Jail memory limits (not yet enforced) + +`kern.racct.enable` is currently `0`. To enable RCTL-based jail memory limits: + +1. Add to `/boot/loader.conf`: + ``` + kern.racct.enable="1" + ``` +2. Reboot +3. Then set limits: + ```sh + rctl -a jail:git:memoryuse:deny=512M + rctl -a jail:db:memoryuse:deny=1536M + rctl -a jail:cms:memoryuse:deny=1G + ``` + +Until RACCT is enabled, use application-level limits: + +- PostgreSQL: `shared_buffers = 384MB` in `postgresql.conf` +- Strapi: `NODE_OPTIONS="--max-old-space-size=512"` in Strapi env +- Astro builds: `NODE_OPTIONS="--max-old-space-size=512"` during build + +### Monitoring + +```sh +# overall memory +top -b -d 1 | head -10 + +# ZFS ARC usage +sysctl vfs.zfs.arc_max vfs.zfs.arc_min kstat.zfs.misc.arcstats.size + +# per-jail (requires RACCT) +rctl -h jail:git +rctl -h jail:db +rctl -h jail:cms +``` + +## Future: RACCT enablement + +Enabling RACCT requires a reboot. Plan this during a maintenance window: + +1. ZFS snapshot all jails +2. Add `kern.racct.enable="1"` to `/boot/loader.conf` +3. Reboot +4. Apply rctl rules +5. Persist rules in `/etc/rctl.conf` diff --git a/.agent/skills/freebsd-admin/references/network-forwarding.md b/.agent/skills/freebsd-admin/references/network-forwarding.md new file mode 100644 index 0000000..68dc9eb --- /dev/null +++ b/.agent/skills/freebsd-admin/references/network-forwarding.md @@ -0,0 +1,33 @@ +# Host Network Forwarding + +Use this file for the canonical forwarding change on the FreeBSD host. + +## Persistent setting + +```sh +sudo sysrc gateway_enable="YES" +``` + +## Immediate runtime setting + +```sh +sudo sysctl net.inet.ip.forwarding=1 +``` + +## Validation + +```sh +sysctl net.inet.ip.forwarding +``` + +Expected: + +- `net.inet.ip.forwarding: 1` + +## Why it matters + +VNET jails on `warden0` cannot route traffic through the host until forwarding +is enabled. + +This is a host prerequisite. It is not a Bastille template issue and not a jail +runtime bug. diff --git a/.agent/skills/freebsd-admin/references/rc-conf.md b/.agent/skills/freebsd-admin/references/rc-conf.md new file mode 100644 index 0000000..aa3b3b1 --- /dev/null +++ b/.agent/skills/freebsd-admin/references/rc-conf.md @@ -0,0 +1,211 @@ +# /etc/rc.conf Settings + +Key rc.conf settings for Clawdie FreeBSD hosts. Use `sysrc` to modify - never edit by hand in scripts. + +## Management Commands + +```bash +# Read a value +sysrc + +# Set a value (idempotent) +sysrc = + +# Check if set +sysrc -c + +# Show all values +sysrc -a + +# Delete a key +sysrc -x +``` + +--- + +## Required Settings by Category + +### System Core + +| Key | Value | Purpose | +| ------------ | ---------------- | ----------------- | +| `hostname` | `your.host.name` | System hostname | +| `zfs_enable` | `YES` | Enable ZFS | +| `dumpdev` | `AUTO` | Crash dump device | + +### Networking + +| Key | Value | Purpose | +| ----------------------- | ----------------------------------- | --------------------------- | +| `gateway_enable` | `YES` | IP forwarding for jails/VMs | +| `cloned_interfaces` | `bridge0` | Create bridge interface | +| `ifconfig_bridge0_name` | `warden0` | Rename to warden0 | +| `ifconfig_warden0` | `inet ${AGENT_SUBNET_BASE}.1/24 up` | Jail bridge gateway | + +### Security + +| Key | Value | Purpose | +| ----------------- | ----- | ------------------------- | +| `pf_enable` | `YES` | Packet filter firewall | +| `sshd_enable` | `YES` | SSH daemon | +| `sshd_rsa_enable` | `NO` | Disable obsolete RSA keys | + +### Services + +| Key | Value | Purpose | +| ------------------------- | ----- | ------------------------ | +| `nginx_enable` | `YES` | Reverse proxy | +| `tailscaled_enable` | `YES` | Tailscale VPN (if used) | +| `qemu_guest_agent_enable` | `YES` | QEMU guest agent (if VM) | + +### Resource Management + +| Key | Value | Purpose | +| ------------- | ----- | ----------------------- | +| `rctl_enable` | `YES` | RCTL for jail/VM limits | + +### Bastille/Jails + +| Key | Value | Purpose | +| ----------------- | ----- | --------------------- | +| `bastille_enable` | `YES` | Bastille jail manager | + +--- + +## Clawdie Host Canonical rc.conf + +```sh +# System +hostname="your.host.name" +zfs_enable="YES" +dumpdev="AUTO" + +# Networking for jails +gateway_enable="YES" +cloned_interfaces="bridge0" +ifconfig_bridge0_name="warden0" +ifconfig_warden0="inet ${AGENT_SUBNET_BASE}.1/24 up" + +# Security +sshd_enable="YES" +sshd_rsa_enable="NO" +pf_enable="YES" + +# Services +nginx_enable="YES" +rctl_enable="YES" + +# Optional (cloud/VM) +qemu_guest_agent_enable="YES" +tailscaled_enable="YES" +``` + +--- + +## Validation Commands + +```bash +# Check gateway enabled +sysrc gateway_enable + +# Check bridge config +sysrc -a | grep -E 'cloned_interfaces|warden0|bridge0' + +# Check all services +sysrc -a | grep _enable + +# Verify forwarding runtime +sysctl net.inet.ip.forwarding +``` + +--- + +## Common Patterns + +### Add jail bridge networking + +```bash +sudo sysrc gateway_enable="YES" +sudo sysrc cloned_interfaces+="bridge0" +sudo sysrc ifconfig_bridge0_name="warden0" +sudo sysrc ifconfig_warden0="inet ${AGENT_SUBNET_BASE}.1/24 up" +``` + +### Enable a service + +```bash +sudo sysrc nginx_enable="YES" +``` + +### Check before setting + +```bash +# Idempotent - only sets if different +sudo sysrc pf_enable="YES" +# Output: pf_enable: NO -> YES +``` + +--- + +## Jail rc.conf (Inside Jails) + +Jails have their own `/etc/rc.conf` with jail-specific settings: + +| Jail | Key Settings | +| ----- | ---------------------------------------- | +| `db` | `postgresql_enable=YES` | +| `cms` | `strapi_enable=YES` (if service defined) | + +Modify jail rc.conf via: + +```bash +sudo bastille cmd sysrc = +``` + +--- + +## Related Files + +| File | Purpose | +| ----------------------- | ----------------------------------- | +| `/etc/defaults/rc.conf` | System defaults (don't edit) | +| `/etc/rc.conf` | Host configuration | +| `/etc/rc.conf.local` | Local overrides (optional) | +| `/boot/loader.conf` | Kernel module loading (before init) | + +--- + +## Troubleshooting + +### Setting not persisting + +```bash +# Check if in rc.conf +grep /etc/rc.conf + +# Check for typos +sysrc -c +``` + +### Service won't start + +```bash +# Check if enabled +sysrc _enable + +# Check rc script exists +ls /etc/rc.d/ /usr/local/etc/rc.d/ +``` + +### Network interface not created + +```bash +# Check cloned_interfaces +sysrc cloned_interfaces + +# Check ifconfig line +sysrc ifconfig_ + +# Manually create +ifconfig create +``` diff --git a/.agent/skills/freebsd-admin/references/resolver-baseline.md b/.agent/skills/freebsd-admin/references/resolver-baseline.md new file mode 100644 index 0000000..3b453f6 --- /dev/null +++ b/.agent/skills/freebsd-admin/references/resolver-baseline.md @@ -0,0 +1,56 @@ +# Host Resolver Baseline + +Use this file when the host uses Tailscale DNS or another specialized resolver +that should not be copied directly into Bastille jails. + +## Problem shape + +The host may legitimately use: + +```conf +nameserver 100.100.100.100 +``` + +through Tailscale MagicDNS. + +Real observed shape on this host: + +```conf +# Generated by resolvconf +search taile682b7.ts.net openstacklocal +nameserver 100.100.100.100 +``` + +That does not make it a good default resolver file for Warden VNET jails. + +## Recommended split + +- host keeps its own resolver model +- Bastille jails use a dedicated resolver file + +Recommended file: + +```sh +/usr/local/etc/bastille/resolv.conf +``` + +Recommended contents: + +```conf +nameserver 1.1.1.1 +nameserver 9.9.9.9 +``` + +Then set: + +```sh +bastille_resolv_conf="/usr/local/etc/bastille/resolv.conf" +``` + +in `/usr/local/etc/bastille/bastille.conf`. + +## Why + +- avoids leaking host-specific DNS assumptions into jails +- keeps jail network debugging simpler +- avoids treating a host Tailscale resolver as a jail prerequisite diff --git a/.agent/skills/freebsd-admin/references/rollback-patterns.md b/.agent/skills/freebsd-admin/references/rollback-patterns.md new file mode 100644 index 0000000..122dc54 --- /dev/null +++ b/.agent/skills/freebsd-admin/references/rollback-patterns.md @@ -0,0 +1,95 @@ +# Rollback Patterns + +Use this file when a host change needs a simple reversal path before more work +continues. + +## Rule + +Prefer the smallest rollback that returns the host to the last known-good state. +Do not mix unrelated host rollback steps into one big command bundle. + +## `sysrc` + +Check current value: + +```sh +sysrc gateway_enable +``` + +Revert: + +```sh +sudo sysrc gateway_enable=NO +``` + +## `sysctl` + +Check live value: + +```sh +sysctl net.inet.ip.forwarding +``` + +Revert live state: + +```sh +sudo sysctl net.inet.ip.forwarding=0 +``` + +## nginx + +Validate before reload: + +```sh +sudo nginx -t +``` + +Reload: + +```sh +sudo service nginx reload +``` + +If a config change was bad: + +1. restore the previous vhost or config file +2. run `sudo nginx -t` +3. reload nginx again + +## `pf` + +Always syntax check first: + +```sh +sudo pfctl -nf /etc/pf.conf +``` + +Reload: + +```sh +sudo service pf reload +``` + +If the new rules are bad: + +1. restore the previous `pf.conf` or include file +2. run `sudo pfctl -nf /etc/pf.conf` +3. reload again + +## Bastille resolver baseline + +If a jail resolver baseline caused confusion: + +1. restore the previous `/usr/local/etc/bastille/resolv.conf` +2. check `/usr/local/etc/bastille/bastille.conf` +3. restart only the affected jail + +## Loader and reboot-bound changes + +For `/boot/loader.conf` or RACCT changes: + +1. record the previous line before editing +2. keep the change isolated in one commit or one operator step +3. schedule reboot-bound rollback separately from runtime rollback + +These changes are not the same class as a `sysctl` or `service reload`. diff --git a/.agent/skills/freebsd-admin/references/service-identities.md b/.agent/skills/freebsd-admin/references/service-identities.md new file mode 100644 index 0000000..49bdc92 --- /dev/null +++ b/.agent/skills/freebsd-admin/references/service-identities.md @@ -0,0 +1,72 @@ +# Service Identities + +Use this file when a host-level service account must match ownership on a +host-mounted jail path. + +## Why this matters + +Host-mounted jail paths use numeric ownership. + +That means: + +- matching usernames alone are not enough +- host and jail must agree on UID/GID when they share mounted storage +- ownership bugs are easier to prevent than to repair later + +## Current standard shared tool identity + +For shared promoted Node.js tooling, standardize: + +```text +user: node +group: node +uid: 3000 +gid: 3000 +home: /var/db/node +shell: /usr/sbin/nologin +``` + +This identity should exist on: + +- the FreeBSD host +- any persistent jail that mounts and executes the shared prefix + +Current likely candidate: + +- `cms` + +## Shared prefix model + +Shared dataset: + +```text +zroot/clawdie-runtime/shared/npm-global +``` + +Mounted path inside a consuming jail: + +```text +/opt/npm +``` + +Ownership: + +```text +node:node +3000:3000 +``` + +## Host-side commands + +```sh +pw groupshow node >/dev/null 2>&1 || pw groupadd node -g 3000 +pw usershow node >/dev/null 2>&1 || pw useradd node -u 3000 -g 3000 -d /var/db/node -m -s /usr/sbin/nologin -c "Shared npm tool owner" +install -d -o 3000 -g 3000 -m 755 /var/db/node +``` + +## Safe defaults + +- keep the interactive operator separate from the shared tool owner +- keep `/opt/npm` owned by `node:node` +- keep host-mounted shared prefixes explicit +- do not assume every jail needs this identity diff --git a/.agent/skills/freebsd-admin/references/system-changes.md b/.agent/skills/freebsd-admin/references/system-changes.md new file mode 100644 index 0000000..2f1cabd --- /dev/null +++ b/.agent/skills/freebsd-admin/references/system-changes.md @@ -0,0 +1,19 @@ +# Host System Changes + +Use this file when deciding whether a change belongs to the host. + +## Host-owned changes + +Examples: + +- `gateway_enable="YES"` +- `net.inet.ip.forwarding=1` +- `service pf restart` +- `service netif restart` +- `sysrc ifconfig_bridge0_name="warden0"` + +## Why this separation matters + +- host failures should not be misdiagnosed as jail failures +- agents need one place for machine-wide state changes +- Warden skills should focus on their domain instead of becoming generic host admin guides diff --git a/.agent/skills/freebsd-admin/references/validation.md b/.agent/skills/freebsd-admin/references/validation.md new file mode 100644 index 0000000..380ed7e --- /dev/null +++ b/.agent/skills/freebsd-admin/references/validation.md @@ -0,0 +1,26 @@ +# Host Validation + +Use this file after making host-side changes. + +## Canonical checks + +```sh +/sbin/sysctl net.inet.ip.forwarding +/sbin/ifconfig warden0 +/usr/sbin/service pf status +/usr/sbin/jls +/sbin/zpool status +/sbin/zdb -C zroot | grep ashift +``` + +Use absolute paths for FreeBSD base-system tools when an agent shell has a +minimal PATH. If a status command fails with `Permission denied`, record the +permission boundary instead of treating it as proof that the service is down. + +## Interpretation + +- forwarding off: host routing problem +- `warden0` missing: host bridge persistence problem +- `pf` stopped: NAT/filtering not active +- jail missing: not a host forwarding problem, go back to `warden-bootstrap` +- suspicious `ashift`: storage validation problem for database workloads diff --git a/.agent/skills/freebsd-admin/scripts/render_forwarding_commands.sh b/.agent/skills/freebsd-admin/scripts/render_forwarding_commands.sh new file mode 100755 index 0000000..8da73c6 --- /dev/null +++ b/.agent/skills/freebsd-admin/scripts/render_forwarding_commands.sh @@ -0,0 +1,8 @@ +#!/bin/sh +set -eu + +cat <<'EOF' +sudo sysrc gateway_enable="YES" +sudo sysctl net.inet.ip.forwarding=1 +sysctl net.inet.ip.forwarding +EOF diff --git a/.agent/skills/freebsd-admin/scripts/render_host_validation.sh b/.agent/skills/freebsd-admin/scripts/render_host_validation.sh new file mode 100755 index 0000000..b7c24e7 --- /dev/null +++ b/.agent/skills/freebsd-admin/scripts/render_host_validation.sh @@ -0,0 +1,9 @@ +#!/bin/sh +set -eu + +cat <<'EOF' +sysctl net.inet.ip.forwarding +ifconfig warden0 +service pf status +jls +EOF diff --git a/.agent/skills/git-branch-protect/SKILL.md b/.agent/skills/git-branch-protect/SKILL.md new file mode 100644 index 0000000..97b12f3 --- /dev/null +++ b/.agent/skills/git-branch-protect/SKILL.md @@ -0,0 +1,65 @@ +--- +name: git-branch-protect +description: Check or apply branch protection rules — verify main is protected, create hotfix branches +compatibility: FreeBSD 15.x +invoke_patterns: + - 'Protect branch' + - 'Branch protection' + - 'Check branch rules' + - 'Create hotfix branch' + - 'Lock main branch' +estimated_tokens: 200-400 +--- + +# git-branch-protect + +Check branch protection status and create protected branches (e.g. hotfix +branches). Does not modify Codeberg's remote protection rules — those require +operator access to Codeberg UI. + +## Check Local Protection + +```bash +cd $AGENT_REPO_DIR + +# What hooks are configured? +ls -la .git/hooks/ + +# Check current branch policies +git config --list | grep -E "branch|push" +``` + +## Create Hotfix Branch + +```bash +cd $AGENT_REPO_DIR + +# From a specific tag +git fetch local --tags +git checkout -b hotfix/security-patch v0.10.0 +git push local hotfix/security-patch + +# Verify +git log --oneline hotfix/security-patch ^main | head -5 +``` + +## Remote Protection (Codeberg) + +Branch protection rules on Codeberg (require PR review, no force push) must be +set in the Codeberg UI: + +``` +Settings → Branches → Add rule → main → Require pull request +``` + +Escalate to operator to configure remote protection. + +## When to Use + +- Creating a hotfix branch from a previous stable tag +- Verifying local git config is secure before a release + +## Escalate If + +- Remote branch protection rules are missing → escalate to operator to configure via Codeberg UI +- Force push to main is requested → always escalate, never do it without explicit operator instruction diff --git a/.agent/skills/git-merge/SKILL.md b/.agent/skills/git-merge/SKILL.md new file mode 100644 index 0000000..0324a39 --- /dev/null +++ b/.agent/skills/git-merge/SKILL.md @@ -0,0 +1,88 @@ +--- +name: git-merge +description: Merge a branch into a target branch — detects conflicts and escalates rather than auto-resolving +compatibility: FreeBSD 15.x +invoke_patterns: + - 'Merge PR *' + - 'Merge * into *' + - 'Merge branch' + - 'Merge pull request' +estimated_tokens: 500-1500 +--- + +# git-merge + +Merge a source branch into a target branch. If conflicts are detected, **stop +and escalate** — never auto-resolve. + +## Remotes + +| Name | URL | Purpose | +| ---------- | ---------------------------- | ------------------ | +| `local` | `$GIT_LOCAL_URL` (git jail) | Default push/pull | +| `upstream` | `$REMOTE_GIT_URL` (Codeberg) | Explicit sync only | + +## Usage + +```bash +cd $AGENT_REPO_DIR + +# Fetch latest from local +git fetch local + +# Checkout target branch +git checkout main +git pull --rebase local main + +# Merge source branch +git merge --no-ff feature/my-branch + +# Check for conflicts +git status +``` + +## Happy Path Output + +``` +Merge made by the 'ort' strategy. + src/index.ts | 42 ++++++++++ + 1 file changed, 42 insertions(+) +Merged commit: abc123def +``` + +## Conflict Output (escalate immediately) + +``` +CONFLICT (content): Merge conflict in src/index.ts +Automatic merge failed; fix conflicts and then commit the result. +``` + +If conflicts → `git merge --abort` → post `approval_request` with: + +- Which files have conflicts +- What changed on each side (brief summary) +- Suggested resolution approaches + +## After Merge + +```bash +# Push merged result to local +git push local main +``` + +## Safety Rules + +- Never force-push (`--force`) without explicit operator instruction +- Never `git reset --hard` without explicit operator instruction +- Always verify CI is green on source branch before merging + +## When to Use + +- Explicit request from operator with branch name and target +- Never run autonomously on `main` without task assignment + +## Escalate If + +- Any merge conflict → always escalate +- Target is `main` → post approval_request regardless of conflicts +- CI failing on source branch → block merge, escalate to operator diff --git a/.agent/skills/git-pull/SKILL.md b/.agent/skills/git-pull/SKILL.md new file mode 100644 index 0000000..7578f49 --- /dev/null +++ b/.agent/skills/git-pull/SKILL.md @@ -0,0 +1,84 @@ +--- +name: git-pull +description: Pull latest changes from local git jail with rebase strategy — safe, no merge commits +compatibility: FreeBSD 15.x +invoke_patterns: + - 'Pull latest' + - 'Pull from origin' + - 'Pull from local' + - 'Update repo' + - 'Sync from local' + - 'Get latest changes' +estimated_tokens: 200-300 +--- + +# git-pull + +Pull latest changes from the local git jail using rebase. Never creates merge +commits on main. + +## Architecture + +The **local git jail** is the primary remote. Agents always pull from `local` +first. The Codeberg `upstream` remote is for explicit sync only. + +``` +upstream (Codeberg) → local (git jail) → agent working copy +``` + +## Remotes + +| Name | URL | Purpose | +| ---------- | ---------------------------- | ------------------ | +| `local` | `$GIT_LOCAL_URL` (git jail) | Default push/pull | +| `upstream` | `$REMOTE_GIT_URL` (Codeberg) | Explicit sync only | + +## Usage + +```bash +cd $AGENT_REPO_DIR + +# Pull from local git jail (default) +git pull --rebase local main + +# Verify +git log --oneline -5 +``` + +## Pull from Codeberg (explicit) + +Only when instructed to sync from upstream: + +```bash +git fetch upstream +git pull --rebase upstream main +# Then push to local so other agents see it: +git push local main +``` + +## Output + +``` +Current branch main is up to date. +-- or -- +Successfully rebased and updated refs/heads/main. +3 commits pulled: abc123, def456, ghi789 +``` + +## Safety Rules + +- Always use `--rebase` (not default merge pull) to keep history clean +- Never run if there are uncommitted local changes +- Check `git status` first — if working tree is dirty, escalate + +## When to Use + +- Before starting any new work +- As part of release preparation (`git-release-tag`) +- Scheduled sync (weekly via Git Admin) + +## Escalate If + +- Rebase fails due to conflicts → escalate to operator +- Local commits exist that aren't on local remote → escalate (unexpected divergence) +- `GIT_LOCAL_URL` not set → check `$AGENT_REPO_DIR/.env` or ask operator diff --git a/.agent/skills/git-push-mirror/SKILL.md b/.agent/skills/git-push-mirror/SKILL.md new file mode 100644 index 0000000..6484fa9 --- /dev/null +++ b/.agent/skills/git-push-mirror/SKILL.md @@ -0,0 +1,71 @@ +--- +name: git-push-mirror +description: Push to upstream Codeberg mirror from local git jail and verify sync +compatibility: FreeBSD 15.x +invoke_patterns: + - 'Push to mirror' + - 'Push to Codeberg' + - 'Sync to upstream' + - 'Mirror repository' + - 'Push upstream' +estimated_tokens: 300-500 +--- + +# git-push-mirror + +Push the local git jail repository to the upstream Codeberg mirror and verify +the push succeeded. + +## Architecture + +``` +Agent working copy → local (git jail, $GIT_LOCAL_URL) → upstream (Codeberg, $REMOTE_GIT_URL) +``` + +Agents push to `local` by default (the git jail). Codeberg sync is explicit +and on-demand via this skill. + +## Remotes + +| Name | URL | Purpose | +| ---------- | ---------------------------- | ------------------ | +| `local` | `$GIT_LOCAL_URL` (git jail) | Default push/pull | +| `upstream` | `$REMOTE_GIT_URL` (Codeberg) | Explicit sync only | + +## Usage + +```bash +cd $AGENT_REPO_DIR + +# First ensure local is up to date +git push local main --tags + +# Then push to upstream (Codeberg) +git push upstream main --tags + +# Verify sync +git ls-remote upstream HEAD +git rev-parse HEAD +# Both should show same commit hash +``` + +## Branch Model + +- `main` — shared base, mirrors Codeberg upstream +- `${AGENT_NAME}` — this agent's working branch (e.g. `mevy`) +- Other agent branches — read-only unless explicitly merging + +When pushing upstream, push the branch the operator requests. Do not force-push +unless explicitly instructed. + +## When to Use + +- After a release tag is created +- After major commits the operator wants backed up to Codeberg +- Explicit "push to Codeberg" instruction from operator + +## Escalate If + +- Push fails with auth error → SSH key or token may need refresh +- Upstream is behind by >100 commits → investigate why sync stopped +- Force push requested on `main` — always escalate for confirmation diff --git a/.agent/skills/git-push-upstream/SKILL.md b/.agent/skills/git-push-upstream/SKILL.md new file mode 100644 index 0000000..62970d3 --- /dev/null +++ b/.agent/skills/git-push-upstream/SKILL.md @@ -0,0 +1,71 @@ +--- +name: git-push-upstream +description: Push current branch to Codeberg upstream — explicit one-way sync from local to public mirror +compatibility: FreeBSD 15.x +invoke_patterns: + - 'Push to Codeberg' + - 'Push to upstream' + - 'Sync to Codeberg' + - 'Publish to upstream' + - 'Backup to upstream' +estimated_tokens: 200-400 +--- + +# git-push-upstream + +Push the current branch from the local git jail to the Codeberg upstream +mirror. This is an **explicit, one-way sync** — it does not run automatically. + +## Architecture + +``` +Agent working copy → local (git jail) → upstream (Codeberg) +``` + +All agent work targets `local` (the git jail) by default. This skill is for +when the operator explicitly wants commits on Codeberg. + +## Remotes + +| Name | URL | Purpose | +| ---------- | ---------------------------- | ------------------ | +| `local` | `$GIT_LOCAL_URL` (git jail) | Default push/pull | +| `upstream` | `$REMOTE_GIT_URL` (Codeberg) | Explicit sync only | + +## Usage + +```bash +cd $AGENT_REPO_DIR + +# Ensure local is up to date first +git push local HEAD + +# Push current branch to upstream +git push upstream HEAD + +# Push all tags +git push upstream --tags + +# Verify +git ls-remote upstream HEAD +git rev-parse HEAD +``` + +## Safety Rules + +- Always push to `local` first, then to `upstream` +- Never force-push to upstream without explicit operator instruction +- Check `git log --oneline local..upstream/main` to see what will be pushed +- If upstream is ahead, pull and rebase first + +## When to Use + +- Operator explicitly requests "push to Codeberg" +- After a release tag is created (usually paired with `git-release-tag`) +- Periodic backup of important work + +## Escalate If + +- Push rejected (upstream ahead) → pull --rebase first, then retry +- Auth failure → SSH key or deploy token may need refresh, escalate to operator +- Force push requested on `main` → always escalate for confirmation diff --git a/.agent/skills/git-release-tag/SKILL.md b/.agent/skills/git-release-tag/SKILL.md new file mode 100644 index 0000000..d354891 --- /dev/null +++ b/.agent/skills/git-release-tag/SKILL.md @@ -0,0 +1,73 @@ +--- +name: git-release-tag +description: Create an annotated git tag for a release and push to local + upstream +compatibility: FreeBSD 15.x +invoke_patterns: + - 'Tag version *' + - 'Tag release *' + - 'Create release' + - 'Release v*' + - 'Tag and release' +estimated_tokens: 300-500 +--- + +# git-release-tag + +Create an annotated git tag and push to local (git jail) and upstream +(Codeberg). Verifies CHANGELOG.md is updated before tagging. + +## Remotes + +| Name | URL | Purpose | +| ---------- | ---------------------------- | ------------------ | +| `local` | `$GIT_LOCAL_URL` (git jail) | Default push/pull | +| `upstream` | `$REMOTE_GIT_URL` (Codeberg) | Explicit sync only | + +## Pre-Tag Checklist + +1. Is `CHANGELOG.md` updated with an entry for this version? +2. Is `package.json` `version` field updated? +3. Is the target commit on `main` (not a feature branch)? + +If any are missing → escalate to operator before tagging. + +## Usage + +```bash +cd $AGENT_REPO_DIR + +# Create annotated tag +git tag -a v0.11.0 -m "v0.11.0" + +# Push tag to local (git jail) +git push local v0.11.0 + +# Push tag to upstream (Codeberg) +git push upstream v0.11.0 +``` + +## Output + +``` +Tag v0.11.0 created at commit abc123def +Pushed to local (git jail) +Pushed to upstream (codeberg.org:Clawdie/Clawdie-AI) +``` + +## Naming Convention + +- Format: `v{MAJOR}.{MINOR}.{PATCH}` (semver) +- Examples: `v0.11.0`, `v1.0.0`, `v1.2.3` +- Only tag on minor/major bumps — patches don't get tagged (see AGENTS.md) + +## When to Use + +- After CHANGELOG.md and package.json are updated +- After all tests pass on main +- Explicit release instruction from operator + +## Escalate If + +- CHANGELOG not updated → escalate to operator to write entry first +- package.json version not bumped → escalate to operator +- Tagging a commit that isn't on main → escalate for confirmation diff --git a/.agent/skills/jail-status/SKILL.md b/.agent/skills/jail-status/SKILL.md new file mode 100644 index 0000000..0836e39 --- /dev/null +++ b/.agent/skills/jail-status/SKILL.md @@ -0,0 +1,53 @@ +--- +name: jail-status +description: Check status of one or all Bastille jails — running, stopped, uptime, CPU, RAM +compatibility: FreeBSD 15.x +invoke_patterns: + - "Check if * jail is running" + - "Is * jail up" + - "Is * up" + - "Jail status" + - "Are jails running" + - "Check * health" +estimated_tokens: 300-500 +--- + +# jail-status + +Check the status of FreeBSD jails managed by Bastille. + +## Usage + +```bash +# All jails +bastille list + +# Specific jail +bastille list | grep + +# Detailed resource usage +rctl -hu jail: +``` + +## Output + +``` +JID IP Address Hostname Path + 5 10.0.1.5 clawdie-db /usr/local/bastille/jails/clawdie-db/root + 6 10.0.1.3 clawdie-cms /usr/local/bastille/jails/clawdie-cms/root + 7 10.0.1.2 clawdie-git /usr/local/bastille/jails/clawdie-git/root +``` + +Treat the above as a shape example, not a hardcoded truth. Resolve live jail +names and IPs from `bastille list` and the current jail registry. + +## When to Use + +- Daily Sysadmin heartbeat: check all critical jails +- After host reboot: verify jails came back up +- When DB/CMS/Git is unreachable: confirm jail is running before deeper diagnosis + +## Escalate If + +- A jail is stopped unexpectedly → `service-restart` skill or escalate to CEO +- A jail shows high CPU/RAM → report to CEO for capacity review diff --git a/.agent/skills/llama-cpp-embeddings/SKILL.md b/.agent/skills/llama-cpp-embeddings/SKILL.md new file mode 100644 index 0000000..127e0c8 --- /dev/null +++ b/.agent/skills/llama-cpp-embeddings/SKILL.md @@ -0,0 +1,67 @@ +--- +name: llama-cpp-embeddings +description: Embedding configuration for Clawdie's pgvector memory. Currently uses OpenRouter (BAAI/bge-m3, 1024 dims). The llamacpp jail runs a chat model (GLM-5.1), NOT an embedding model. Triggers on "embeddings", "llama.cpp", "bge-m3", "pgvector setup", "embed". +--- + +# Embeddings — Current Setup + +## Active configuration (as of 2026-03-29) + +Embeddings are handled by **OpenRouter**, not the local llamacpp jail. + +```sh +EMBED_BASE_URL=https://openrouter.ai/api/v1 +EMBED_MODEL=BAAI/bge-m3 +EMBED_DIMENSIONS=1024 +EMBED_API_KEY= +``` + +DB schema: `vector(1024)` in `memory_embeddings.embedding` column. + +**Why not local llamacpp?** The llamacpp jail runs a chat model (GLM-5.1 via z.ai). +A chat model returns 501 for `/v1/embeddings` — it does not support the embeddings +endpoint. Running a separate embedding model in the llamacpp jail is possible but +was deprioritised in favour of OpenRouter for reliability. + +## Verify embeddings are working + +```sh +curl -s -X POST https://openrouter.ai/api/v1/embeddings \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer $OPENROUTER_API_KEY" \ + -d '{"model":"BAAI/bge-m3","input":"test"}' | jq '.data[0].embedding | length' +# Should print: 1024 +``` + +## DB schema dimension + +The `memory_embeddings` table uses `vector(1024)`. If you change the embedding +model to one with different dimensions, you must drop and recreate the table: + +```sql +DROP TABLE memory_embeddings; +-- then re-run schema migration to recreate with correct vector(N) +``` + +Embeddings are regenerated from stored text — no source data is lost. + +## Fallback: FTS-only mode + +Set `EMBED_BASE_URL=` (empty) to disable embeddings entirely. +Memory search falls back to full-text search (tsvector). Quality is lower +but the system works. Re-enable later by setting the URL and restarting. + +## Future: local embedding model in llamacpp jail + +To run embeddings locally, the llamacpp jail would need a **separate** llama-server +instance running in `--embedding` mode with a dedicated embedding model (e.g. +`nomic-embed-text-v1.5`, 768 dims, or `BAAI/bge-m3` with `--embedding` flag). + +Key constraints: + +- Cannot reuse the chat server process — embedding mode and chat mode are mutually exclusive +- Must run on a different port (e.g. 8081 for chat, 8082 for embed) +- DB schema dimension must match the model (768 for nomic, 1024 for bge-m3) +- All existing embeddings must be regenerated if switching models + +Until this is needed, use OpenRouter. diff --git a/.agent/skills/network-throughput/SKILL.md b/.agent/skills/network-throughput/SKILL.md new file mode 100644 index 0000000..a682b05 --- /dev/null +++ b/.agent/skills/network-throughput/SKILL.md @@ -0,0 +1,538 @@ +--- +name: network-throughput +description: Run a clean osa-to-debby HTTPS network throughput test with packet captures, counters, cleanup, and Hermes graph-ready artifacts +compatibility: FreeBSD 15.x server, Linux client +invoke_patterns: + - "Run throughput test" + - "Run network throughput test" + - "Test download speed" + - "Capture pcaps on both sides" + - "Diagnose image download" + - "Network throughput test" + - "Network troughput test" +estimated_tokens: 1400-2200 +--- + +# network-throughput + +Run one clean HTTPS image download from osa FreeBSD server to debby, with packet capture and counters on both sides, to separate PF/PMTU/MSS/server behavior from access-path/bufferbloat behavior. + +Main rule: + +> One download. Captures first. No overwrite. Summaries first. Cleanup after. + +## Definitions + +Record these before the run so there is no ambiguity: + +- `client`: debby +- `client network`: house Wi-Fi | hotspot | wired | other +- `server`: osa.smilepowered.org +- `server public IP`: 51.83.197.148 +- `server Tailscale IP`: 100.72.229.63 +- `URL`: exact image URL under test +- `downloader`: curl or wget only; prefer curl with `--http1.1` + +Terminology: + +- "osa hotspot" means the phone hotspot/mobile Wi-Fi path. +- "osa server" means `osa.smilepowered.org`, FreeBSD 15, public `51.83.197.148`, Tailscale `100.72.229.63`. + +## Safety Rules + +- Ask before starting any privileged capture or firewall-visible test. +- Use one test download only. Do not run Firefox/browser downloads in parallel. +- On FreeBSD/Clawdie, use project-local scratch directories, not system `/tmp` or `/var/tmp`. +- On Linux/debby/Hermes, prefer `~/.local/state/hermes/net-tests/`; project-root `tmp/` is also acceptable when running inside a specific repo. +- On Linux/debby/Hermes, root `tcpdump` may stage in `/tmp/osa-pcap-$TEST_ID` when direct writes into user-owned artifact dirs are awkward; move/chown the captures into `CLIENT_DIR` after the run. +- On FreeBSD, run root commands in the visible tmux root window when one is available. +- Do not leave large pcaps or duplicate downloaded images behind after analysis. +- Do not change PF during the measurement unless the purpose of the test is explicitly PF comparison. +- Use UTC timestamps at before-counters, pcap start, download start, download stop, and after-counters. + +## Variables + +Set these before the run. + +### Server / osa + +```sh +TEST_ID="osa-clean-download-$(date -u +%Y%m%dT%H%M%SZ)" +SERVER_IF="vtnet0" +SERVER_DIR="/home/clawdie/clawdie-iso/tmp/network-tests/$TEST_ID" +CLIENT_IP="" +SERVER_IP="51.83.197.148" +SERVER_TS_IP="100.72.229.63" +DURATION_SEC="600" +``` + +### Linux client / debby / Hermes + +Prefer the Hermes state directory for Hermes-run artifacts. If running inside a specific project repo, project-root `tmp/network-tests/$TEST_ID` is also acceptable. + +```sh +TEST_ID="osa-clean-download-$(date -u +%Y%m%dT%H%M%SZ)" +CLIENT_DIR="$HOME/.local/state/hermes/net-tests/$TEST_ID" +# Alternative when running inside a repo: +# CLIENT_DIR="$PWD/tmp/network-tests/$TEST_ID" +URL="https://osa.smilepowered.org/downloads/iso/clawdie-xfce-operator-usb-fbsd15.0-amd64-15.maj.2026.img.gz" +SHA_URL="$URL.sha256" +SERVER_IP="51.83.197.148" +SERVER_TS_IP="100.72.229.63" +DURATION_SEC="600" +``` + +## Preflight + +### Server before-counters + +```sh +mkdir -p "$SERVER_DIR" +date -u '+before_counters_utc=%Y-%m-%dT%H:%M:%SZ' | tee "$SERVER_DIR/timestamps.txt" +df -h /home/clawdie > "$SERVER_DIR/df-before.txt" +ifconfig "$SERVER_IF" > "$SERVER_DIR/ifconfig-server-if.txt" +netstat -rn > "$SERVER_DIR/routes.txt" +/sbin/pfctl -si -v > "$SERVER_DIR/pf-before.txt" +netstat -s -p tcp > "$SERVER_DIR/tcp-before.txt" +netstat -s -p ip > "$SERVER_DIR/ip-before.txt" +netstat -s -p icmp > "$SERVER_DIR/icmp-before.txt" +netstat -s -p icmp6 > "$SERVER_DIR/icmp6-before.txt" +``` + +Also save the TCP lines we care about most: + +```sh +netstat -s -p tcp | grep -E 'retransmitted|retransmit timeouts|resend initiated by MTU discovery|Path MTU|black hole' > "$SERVER_DIR/tcp-before-key-lines.txt" +``` + +### Linux client preflight + +```sh +mkdir -p "$CLIENT_DIR" +date -u '+client_preflight_utc=%Y-%m-%dT%H:%M:%SZ' | tee "$CLIENT_DIR/timestamps.txt" +df -h . > "$CLIENT_DIR/df-before.txt" +ip addr > "$CLIENT_DIR/ip-addr.txt" +ip route > "$CLIENT_DIR/ip-route.txt" +curl -4 -sS --max-time 10 https://ifconfig.me > "$CLIENT_DIR/public-ip.txt" || true +ip route get "$SERVER_IP" > "$CLIENT_DIR/route-to-osa-public.txt" +``` + +Pick the outbound interface from `route-to-osa-public.txt`: + +```sh +CLIENT_IF="" +``` + +## Packet Capture + +Start both captures before the download. Wait until tcpdump prints `listening on ...` on both sides. + +Operational order: + +1. start server pcap +2. start client pcap +3. start client latency monitor +4. start download +5. stop download if bounded +6. stop latency monitor +7. stop pcaps +8. collect after-counters + +Avoid tiny rotating pcap rings. For a 5-10 minute test, either use a single non-overwriting pcap or enough ring files to preserve the beginning/SYN. + +### Server capture + +If client public IP is known, prefer this filter: + +```sh +date -u '+server_pcap_start_utc=%Y-%m-%dT%H:%M:%SZ' | tee -a "$SERVER_DIR/timestamps.txt" +/usr/bin/timeout 900 /usr/sbin/tcpdump -ni "$SERVER_IF" \ + -s 0 \ + -C 200 -W 20 \ + -w "$SERVER_DIR/osa-vtnet0-download.pcap" \ + "(host $CLIENT_IP and tcp port 443) or icmp or icmp6" +``` + +If client public IP is not known yet, temporarily capture all HTTPS plus ICMP/PTB signals: + +```sh +date -u '+server_pcap_start_utc=%Y-%m-%dT%H:%M:%SZ' | tee -a "$SERVER_DIR/timestamps.txt" +/usr/bin/timeout 900 /usr/sbin/tcpdump -ni "$SERVER_IF" \ + -s 0 \ + -C 200 -W 20 \ + -w "$SERVER_DIR/osa-vtnet0-download.pcap" \ + "tcp port 443 or icmp or icmp6" +``` + +### Linux client capture + +Run in a root shell or with sudo. Prefer writing directly to `CLIENT_DIR`; if permissions get in the way, stage in a root-writable capture directory and then move/chown into `CLIENT_DIR` after capture. + +Direct-to-artifact-dir form: + +```sh +date -u '+client_pcap_start_utc=%Y-%m-%dT%H:%M:%SZ' | tee -a "$CLIENT_DIR/timestamps.txt" +sudo /usr/bin/timeout 900 tcpdump -ni "$CLIENT_IF" \ + -s 0 \ + -C 200 -W 20 \ + -w "$CLIENT_DIR/debby-osa-public.pcap" \ + "host $SERVER_IP and (tcp port 443 or icmp or icmp6)" +``` + +Staged form, if root tcpdump cannot write directly to `CLIENT_DIR`: + +```sh +CAPTURE_TMP="/tmp/osa-pcap-$TEST_ID" +mkdir -p "$CAPTURE_TMP" +date -u '+client_pcap_start_utc=%Y-%m-%dT%H:%M:%SZ' | tee -a "$CLIENT_DIR/timestamps.txt" +sudo /usr/bin/timeout 900 tcpdump -ni "$CLIENT_IF" \ + -s 0 \ + -C 200 -W 20 \ + -w "$CAPTURE_TMP/debby-osa-public.pcap" \ + "host $SERVER_IP and (tcp port 443 or icmp or icmp6)" +sudo chown "$USER:$USER" "$CAPTURE_TMP"/debby-osa-public.pcap* +mv "$CAPTURE_TMP"/debby-osa-public.pcap* "$CLIENT_DIR"/ +rmdir "$CAPTURE_TMP" 2>/dev/null || true +``` + +## Client Latency Monitor + +Start this on debby before the download and stop it after the download stops. It helps identify access-path/bufferbloat symptoms. + +```sh +( + GW=$(ip route show default | awk '{print $3; exit}') + while true; do + date -u '+utc=%Y-%m-%dT%H:%M:%SZ' + for t in "$GW" 1.1.1.1 "$SERVER_TS_IP" 100.103.255.41; do + echo "### $t" + ping -c 5 -i 0.2 -W 2 "$t" + done + sleep 5 + done +) | tee "$CLIENT_DIR/ping-monitor.log" & +PING_MON_PID=$! +echo "$PING_MON_PID" > "$CLIENT_DIR/ping-monitor.pid" +``` + +Stop it after the download stops: + +```sh +kill "$(cat "$CLIENT_DIR/ping-monitor.pid")" 2>/dev/null || true +``` + +## Download Test + +Use a CLI downloader, not a browser. For diagnosis, default to a fresh output file and no resume. Prefer `curl --http1.1` so the test is one simple TCP flow and avoids HTTP/2 multiplexing noise. + +### Full artifact download + +```sh +cd "$CLIENT_DIR" +date -u '+download_start_utc=%Y-%m-%dT%H:%M:%SZ' | tee -a timestamps.txt +curl -L \ + --fail \ + --http1.1 \ + --output clawdie-test.img.gz \ + --write-out '\nremote_ip=%{remote_ip}\nremote_port=%{remote_port}\nlocal_ip=%{local_ip}\nlocal_port=%{local_port}\ntime_total=%{time_total}\nspeed_download=%{speed_download}\nsize_download=%{size_download}\nhttp_code=%{http_code}\n' \ + "$URL" \ + 2>&1 | tee curl-download.log +date -u '+download_end_utc=%Y-%m-%dT%H:%M:%SZ' | tee -a timestamps.txt +``` + +### Bounded 5-10 minute sample + +Use this when disk or time is limited. A timeout exit is acceptable; we care about behavior during the window. + +```sh +cd "$CLIENT_DIR" +date -u '+download_start_utc=%Y-%m-%dT%H:%M:%SZ' | tee -a timestamps.txt +timeout "$DURATION_SEC" curl -L \ + --fail \ + --http1.1 \ + --output clawdie-test.img.gz \ + --write-out '\nremote_ip=%{remote_ip}\nremote_port=%{remote_port}\nlocal_ip=%{local_ip}\nlocal_port=%{local_port}\ntime_total=%{time_total}\nspeed_download=%{speed_download}\nsize_download=%{size_download}\nhttp_code=%{http_code}\n' \ + "$URL" \ + 2>&1 | tee curl-download.log +date -u '+download_end_utc=%Y-%m-%dT%H:%M:%SZ' | tee -a timestamps.txt +``` + +## Postflight + +Stop captures first, then collect after-counters. + +### Server after-counters + +```sh +date -u '+after_counters_utc=%Y-%m-%dT%H:%M:%SZ' | tee -a "$SERVER_DIR/timestamps.txt" +/sbin/pfctl -si -v > "$SERVER_DIR/pf-after.txt" +netstat -s -p tcp > "$SERVER_DIR/tcp-after.txt" +netstat -s -p ip > "$SERVER_DIR/ip-after.txt" +netstat -s -p icmp > "$SERVER_DIR/icmp-after.txt" +netstat -s -p icmp6 > "$SERVER_DIR/icmp6-after.txt" +netstat -s -p tcp | grep -E 'retransmitted|retransmit timeouts|resend initiated by MTU discovery|Path MTU|black hole' > "$SERVER_DIR/tcp-after-key-lines.txt" +ls -lh "$SERVER_DIR" > "$SERVER_DIR/ls.txt" +``` + +### Linux client postflight + +```sh +ls -lh "$CLIENT_DIR" > "$CLIENT_DIR/ls.txt" +curl -L --fail --max-time 30 -o "$CLIENT_DIR/expected.sha256" "$SHA_URL" +``` + +If the full image completed, verify checksum without depending on the filename inside the `.sha256` file: + +```sh +EXPECTED=$(awk '{print $1; exit}' "$CLIENT_DIR/expected.sha256") +ACTUAL=$(sha256sum "$CLIENT_DIR/clawdie-test.img.gz" | awk '{print $1}') +printf 'expected=%s\nactual=%s\n' "$EXPECTED" "$ACTUAL" | tee "$CLIENT_DIR/checksum-result.txt" +test "$EXPECTED" = "$ACTUAL" +``` + +## Summary-First Exchange + +Before copying raw pcaps around, generate text summaries on each side. Exchange summaries first; only copy raw pcaps if something suspicious needs deeper inspection. + +```sh +PCAP="/path/to/file.pcap" +OUT="$PCAP.summary.txt" +{ + echo "## capinfos" + capinfos "$PCAP" + + echo + echo "## tcp conversations" + tshark -r "$PCAP" -q -z conv,tcp + + echo + echo "## 1s io stats" + tshark -r "$PCAP" -q \ + -z io,stat,1,'tcp','icmp || icmpv6','tcp.analysis.retransmission || tcp.analysis.fast_retransmission','tcp.analysis.duplicate_ack','tcp.analysis.zero_window' + + echo + echo "## SYN MSS/window options" + tshark -r "$PCAP" -Y 'tcp.flags.syn == 1' -T fields \ + -e frame.time_relative \ + -e ip.src -e ip.dst \ + -e tcp.srcport -e tcp.dstport \ + -e tcp.flags.ack \ + -e tcp.options.mss_val \ + -e tcp.window_size_value \ + -e tcp.options.wscale.shift \ + -e tcp.options.sack_perm \ + | head -100 + + echo + echo "## interesting TCP/ICMP" + tshark -r "$PCAP" \ + -Y 'tcp.analysis.retransmission or tcp.analysis.fast_retransmission or tcp.analysis.duplicate_ack or tcp.analysis.zero_window or tcp.analysis.out_of_order or tcp.analysis.lost_segment or icmp or icmpv6' \ + -T fields \ + -e frame.time_relative \ + -e frame.number \ + -e ip.src -e ip.dst \ + -e _ws.col.Protocol \ + -e tcp.srcport -e tcp.dstport \ + -e tcp.seq -e tcp.ack -e tcp.len \ + -e tcp.analysis.retransmission \ + -e tcp.analysis.fast_retransmission \ + -e tcp.analysis.duplicate_ack \ + -e tcp.analysis.duplicate_ack_num \ + -e tcp.analysis.zero_window \ + -e tcp.analysis.out_of_order \ + -e tcp.analysis.lost_segment \ + -e icmp.type -e icmp.code \ + -e icmpv6.type -e icmpv6.code \ + | head -300 + + echo + echo "## expert" + tshark -r "$PCAP" -q -z expert | head -300 +} > "$OUT" +``` + +## Hermes Graph Inputs + +Hermes should process locally from the client-side artifacts plus copied server-side summaries/artifacts: + +- `curl-download.log` +- `ping-monitor.log` +- `timestamps.txt` +- `pf-before.txt`, `pf-after.txt` +- `tcp-before.txt`, `tcp-after.txt` +- `ip-before.txt`, `ip-after.txt` +- `icmp-before.txt`, `icmp-after.txt` +- `icmp6-before.txt`, `icmp6-after.txt` +- `*.pcap.summary.txt` +- raw `*.pcap*` only if needed + +Recommended graph outputs: + +- throughput over time +- retransmissions over time +- duplicate ACKs over time +- RTT estimate over time, if available +- TCP window/zero-window events over time +- bytes by flow +- gateway/internet/Tailscale latency during download +- timeline of stalls or timeout bursts + +## Cleanup + +Cleanup is mandatory after the report/dashboard exists. + +Keep: + +- curl log +- ping monitor log +- counter before/after text +- pcap summaries +- final dashboard/report + +Remove: + +- downloaded test image +- partial image +- raw pcaps, unless still needed + +### Client cleanup + +```sh +rm -f "$CLIENT_DIR/clawdie-test.img.gz" +rm -f "$CLIENT_DIR"/*.partial +``` + +If raw pcaps are no longer needed, delete only raw capture files and keep `*.summary.txt`: + +```sh +for f in "$CLIENT_DIR"/*.pcap "$CLIENT_DIR"/*.pcap[0-9]*; do + [ -e "$f" ] || continue + case "$f" in + *.summary.txt|*.gz) continue ;; + esac + rm -f "$f" +done +``` + +If suspicious and raw pcaps must be kept temporarily, compress only raw capture files: + +```sh +for f in "$CLIENT_DIR"/*.pcap "$CLIENT_DIR"/*.pcap[0-9]*; do + [ -e "$f" ] || continue + case "$f" in + *.summary.txt|*.gz) continue ;; + esac + gzip -9 "$f" +done +``` + +### Server cleanup + +If raw pcaps are no longer needed, delete only raw capture files and keep `*.summary.txt`: + +```sh +for f in "$SERVER_DIR"/*.pcap "$SERVER_DIR"/*.pcap[0-9]*; do + [ -e "$f" ] || continue + case "$f" in + *.summary.txt|*.gz) continue ;; + esac + rm -f "$f" +done +``` + +If suspicious and raw pcaps must be kept temporarily, compress only raw capture files: + +```sh +for f in "$SERVER_DIR"/*.pcap "$SERVER_DIR"/*.pcap[0-9]*; do + [ -e "$f" ] || continue + case "$f" in + *.summary.txt|*.gz) continue ;; + esac + gzip -9 "$f" +done +``` + +Then confirm space recovered: + +```sh +df -h /home/clawdie +``` + +## Result Summary Template + +```text +Test ID: +Date/time UTC: +Client: debby +Client network path: house Wi-Fi | hotspot | wired | other +Server: osa.smilepowered.org +Server public IP: 51.83.197.148 +Server Tailscale IP: 100.72.229.63 +URL: +Downloader: curl --http1.1 | wget | other +Parallel downloads: no | yes, describe +Server capture duration: +Client capture duration: +Downloaded bytes: +Average throughput: +Checksum result: pass | fail | partial only + +Latency: +- Gateway latency during download: +- 1.1.1.1 latency during download: +- osa Tailscale latency during download: +- domedog latency during download: + +PF/PMTU/MSS: +- PF state/search/fragment counters changed: +- ICMP frag-needed / IPv6 Packet Too Big seen: +- PMTU black-hole counters: +- SYN MSS values: + +TCP: +- Server retransmitted data delta: +- Server retransmit timeout delta: +- Pcap retransmissions: +- Fast retransmissions: +- Duplicate ACKs: +- Zero-window events: + +Conclusion: +- PF likely root cause? yes | no | inconclusive +- PMTU/MSS likely root cause? yes | no | inconclusive +- Access path/bufferbloat suspect? yes | no | inconclusive +- Next action: + +Cleanup: +- Large client image removed: yes | no +- Large server pcap removed/compressed: yes | no +- Raw pcap copies deleted after conclusion: yes | no +- Disk checked after cleanup: yes | no +``` + +## Shared Intent Text + +```text +Intent for next throughput test: + +We want one clean HTTPS download from osa.smilepowered.org to debby, with no Firefox and no parallel downloads. The purpose is to separate server/PF/PMTU/MSS behavior from client/access-path/bufferbloat behavior. + +Definitions: +- “osa hotspot” means phone hotspot/mobile Wi-Fi path. +- “osa server” means osa.smilepowered.org, FreeBSD 15, public 51.83.197.148, Tailscale 100.72.229.63. + +Rules: +1. Start pcaps on both sides before the download. +2. Use one curl/wget download only, preferably curl --http1.1. +3. Use UTC timestamps for before/start/stop/after. +4. Server collects TCP/PF/ICMP/ICMPv6 counters before and after. +5. Pcaps must not overwrite the beginning of the test; use enough -W or shorten runtime. +6. Exchange pcap summaries first, not raw pcaps. +7. Keep final logs/summaries/dashboard, then clean raw pcaps and downloaded test image. + +Success criteria: +- We know bytes/time/throughput. +- We know whether retransmits/dupACKs/timeouts were normal or abnormal. +- We know whether ICMP/PTB/fragmentation-needed/PMTU looked suspicious. +- We know whether latency spikes were local/access-path or server-side. +``` diff --git a/.agent/skills/nginx-glasspane/references/cache-policy.md b/.agent/skills/nginx-glasspane/references/cache-policy.md new file mode 100644 index 0000000..85acc2a --- /dev/null +++ b/.agent/skills/nginx-glasspane/references/cache-policy.md @@ -0,0 +1,15 @@ +# Nginx Cache Policy for Glasspane + +Use this file when deciding what should and should not be cached. + +## Recommended split + +- `index.html` / latest page: low or no cache +- `latest.json`: no cache +- `viewer.html`: low cache +- `*.png`, `*.txt`, `*.json` screenshot artifacts: cache allowed + +## Why + +- latest operator state must refresh reliably +- archived artifacts are immutable once written diff --git a/.agent/skills/nginx-glasspane/scripts/render_latest_json.sh b/.agent/skills/nginx-glasspane/scripts/render_latest_json.sh new file mode 100755 index 0000000..39e7e1c --- /dev/null +++ b/.agent/skills/nginx-glasspane/scripts/render_latest_json.sh @@ -0,0 +1,14 @@ +#!/bin/sh +set -eu + +UUID="${1:-exampleuuid1234}" +SESSION="${2:-tmux-session}" +CAPTURED_AT="${3:-2026-03-08T12:00:00.000Z}" + +cat <` — not served by cms jail nginx +- Shared CMS admin/API: `cms.` — shared service surface +- Shared code admin: `git.` — separate service surface, not cms nginx +- Tenant home: `.` — served by cms jail nginx +- Tenant site: `..` — served by cms jail nginx + +For the internal default install, `` is `home.arpa`. + +## Actual architecture (as deployed) + +```text +operator at ai. + → controlplane app (not cms nginx) + +tenant/browser at . or .. + → host nginx (optional public SSL terminator / proxy) + → cms jail nginx at ${AGENT_SUBNET_BASE}.4 + → tenant home + tenant sites (static output) + +operator/editor at cms. + → host nginx (optional public/internal proxy) + → cms jail nginx + → Strapi admin + CMS API +``` + +**Host nginx** terminates SSL and proxies to the jail when public exposure exists. +**Jail nginx** is the real web server for tenant homes, tenant sites, and the shared CMS surface. + +This diverges from the original design (PF RDR → jail nginx handling SSL itself). The reason: +host nginx was already running for clawdie.si, so PF RDR to the jail would break the existing +site. Host-proxy is the correct pattern when multiple domains share the same host. + +## Scope + +This skill covers: + +- Host nginx SSL proxy vhosts (SSL termination, proxy_pass to jail) +- Jail nginx server_name routing for `cms.`, `.`, and `..` +- SSL certificate management for public-exposed surfaces +- ACME challenge pattern when host nginx proxies to jail +- Strapi admin/API reverse proxy on the shared CMS surface +- Tenant home and tenant-site static serving + +This skill does not replace: + +- `warden-pf` for PF firewall rules +- `freebsd-admin` for host-level system changes + +## SSL certificate tools (two tools on this host) + +| Domain | Tool | Location | +| ------------------------------------------------------- | ------- | ------------------------------------------- | +| `clawdie.si`, `docs.clawdie.si`, `osa.smilepowered.org` | acme.sh | `/root/.acme.sh/_ecc/` | +| `samob.smilepowered.org` | certbot | `/usr/local/etc/letsencrypt/live//` | + +Keep the tools separate. Do not migrate certbot domains to acme.sh without a plan. + +`just doctor` audits the canonical acme.sh-backed public cert paths and reports `TLS_