agent-shadow-worker

Shadow worker agent — parallel task execution

agent-shadow-worker is a KADI “shadow” agent that monitors and maintains shadow worktrees for worker agents (artist, designer, programmer). It ensures worker playtrees exist, auto-creates git worktrees/branches when missing, and emits backup/monitoring events (e.g., shadow-{role}.backup.completed / shadow-{role}.backup.failed). Role selection is flexible via AGENT_ROLE env or config.toml and it integrates with vault secrets and broker networks.

Architecture

• Data flow:

  • Reads runtime config (config.toml) and role-specific config files via ShadowRoleLoader.
  • Connects to a broker (local or remote) using BaseAgent/createShadowAgent from agents-library and publishes events on shadow-{role} topics.
  • Optionally loads provider credentials from a secrets vault (loadVaultCredentials) for integrations (Anthropic, Model Manager).
  • Monitors the worker worktree and shadow worktree; auto-initializes main playground repo and git worktrees when missing using child_process.execSync.
  • Writes logs via ability-log and persists metadata to arcadedb (configured via [arcadedb]).

• Key components:

  • ShadowRoleLoader (roles/ShadowRoleLoader.js): loads role-specific paths, branches, and monitoring interval.
  • agents-library helpers: createShadowAgent, BaseAgent, readConfig, loadVaultCredentials, logger, timer.
  • Git worktree management code in src/index.ts that creates repos and worktrees when missing.
  • Broker(s): local (ws://localhost:8080/kadi) and remote (wss://broker.dadavidtseng.com/kadi).

• How it fits in AGENTS ecosystem:

  • Acts as a companion to primary worker agents: keeps a shadow copy of a worker’s workspace and publishes backup/monitoring events other agents or services can subscribe to.
  • Uses KADI brokering and abilities (secret-ability for vaults, ability-file-local for file operations, ability-log for logging).
  • Deploy manifests in agent.json define role-specific service runs (start:artist / start:programmer / start:designer).

Tools / API

The package relies on and uses the following exported helpers / tools (from agents-library) and local classes. If these tools are registered as abilities in the runtime, their usage is described below.

Tool / Export	Description	Key parameters / return
createShadowAgent	Factory to create an agent instance wired for shadow behaviors (broker connect, topic naming, lifecycle).	(config: BaseAgentConfig) => Promise (typical usage: createShadowAgent(cfg))
BaseAgent	Core agent class for broker connection and message handling.	new BaseAgent(cfg) — exposes publish/subscribe methods, lifecycle controls.
loadVaultCredentials	Loads vault secrets (KADI vault integration).	() => Promise<Record<string,string>> — fallback for ANTHROPIC_API_KEY, MODEL_MANAGER_* keys.
readConfig	Reads config.toml into a config accessor used in this agent.	() => config object with .string(key), .has(key), .strings(key)
setLogLevel	Set global logging level used by logger.	setLogLevel(‘info'
setAgentTag	Set agent identifier tag for structured logging.	setAgentTag(agentId: string)
logger	Structured logging helper (info/debug/error).	logger.info(tag, message, elapsed?)
timer	Lightweight timer utility used for elapsed logging.	timer.start(key), timer.elapsed(key)
loadDirective	Helper for loading runtime directives (not used directly in shown code).	loadDirective(…)
ShadowRoleLoader	Local role loader to read config/roles/{role}.toml and return role settings.	new ShadowRoleLoader(projectRoot).loadRole(roleName) => { role, workerWorktreePath, shadowWorktreePath, workerBranch, shadowBranch, monitoringInterval, mainRepoPath, … }

If the source registers any tools/abilities at runtime, they will appear in the agent’s ability manifest (agent.json includes secret-ability, ability-file-local, ability-log).

Configuration

Configuration is primarily sourced from config.toml, environment variables, and vault secrets.

• config.toml fields (present in package root)

  • [agent]
    • ID — agent identifier (e.g., “agent-shadow-worker”)
    • ROLE — default role (artist/designer/programmer)
    • VERSION — agent version
  • [logging]
    • LEVEL — log level (info/debug)
  • [broker.local]
    • URL — broker websocket URL (e.g., ws://localhost:8080/kadi)
    • NETWORKS — array of network names to join (e.g., [“git”,“qa”,“file”])
  • [broker.remote]
    • URL — remote broker websocket URL
    • NETWORKS — networks for remote broker
  • [worktree]
    • WORKER_PATH — example worker path (used by role configs)
    • SHADOW_PATH — shadow worktree path
    • WORKER_BRANCH — worker branch name
    • SHADOW_BRANCH — shadow branch name
  • [secrets]
    • VAULTS — list of vault names referenced ([“anthropic”,“model-manager”,“arcadedb”])
    • KEYS — secret keys expected in vault
  • [arcadedb]
    • HOST, PORT, USERNAME, DATABASE — arcade DB connection metadata

• Environment variables

  • AGENT_ROLE — override role at runtime (preferred for kadi run dev:artist)
  • ANTHROPIC_API_KEY, MODEL_MANAGER_API_KEY, MODEL_MANAGER_BASE_URL — provider keys (can be set via vault or env)
  • ARCADE_HOST, ARCADE_PORT — arcadedb host/port (used in agent.json deploy env)
  • NODE_ENV — production/development

• Secrets / Vault

  • The agent uses loadVaultCredentials() to fetch secrets from configured vaults. Deploy manifests (agent.json) expect these vaults and keys:
    • anthropic: ANTHROPIC_API_KEY
    • model-manager: MODEL_MANAGER_API_KEY, MODEL_MANAGER_BASE_URL
    • arcadedb: ARCADE_USERNAME, ARCADE_PASSWORD
  • agent.json “deploy” entries show kadi secret receive —vault anthropic —vault model-manager —vault arcadedb used during container start.

• Start scripts (package.json highlights)

  • start — node dist/index.js
  • start:artist — AGENT_ROLE=artist node dist/index.js
  • start:designer — AGENT_ROLE=designer node dist/index.js
  • start:programmer — AGENT_ROLE=programmer node dist/index.js
  • dev variants use npx tsx watch src/index.ts for live reload.

Code Examples

Below are key patterns copied from src/index.ts showing role loading, vault credential loading, and auto-creation of worktrees.

import fs from 'fs';
import path from 'path';
import { fileURLToPath } from 'url';
import { createShadowAgent, BaseAgent, loadVaultCredentials, readConfig, setLogLevel, setAgentTag, logger, timer, loadDirective } from 'agents-library';
import type { BaseAgentConfig } from 'agents-library';
import { ShadowRoleLoader } from './roles/ShadowRoleLoader.js';

const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);

const cfg = readConfig();

Role selection and broker resolution:

// Role: env var takes priority (for kadi run dev:artist), fallback to config.toml
const roleName = process.env.AGENT_ROLE ?? cfg.string('agent.ROLE');

// Broker resolution
const hasLocal = cfg.has('broker.local.URL');
const hasRemote = cfg.has('broker.remote.URL');
if (!hasLocal && !hasRemote) {
  throw new Error('At least one broker required: set [broker.local] or [broker.remote] in config.toml');
}

const brokerUrl = hasLocal
  ? cfg.string('broker.local.URL')
  : cfg.string('broker.remote.URL');
const networks = hasLocal
  ? cfg.strings('broker.local.NETWORKS')
  : cfg.strings('broker.remote.NETWORKS');

Loading vault credentials (fallback to env vars):

// Credentials: vault for provider keys (optional — shadow agents may not need LLM)
const vault = await loadVaultCredentials();
const anthropicApiKey = process.env.ANTHROPIC_API_KEY || vault.ANTHROPIC_API_KEY;
const modelManagerBaseUrl = process.env.MODEL_MANAGER_BASE_URL || vault.MODEL_MANAGER_BASE_URL;
const modelManagerApiKey = process.env.MODEL_MANAGER_API_KEY || vault.MODEL_MANAGER_API_KEY;

Role loader and worktree initialization (auto-create main repo and worktrees):

// Load role configuration from config/roles/{role}.toml
logger.info(agentId, `Loading shadow role configuration: ${roleName}`, timer.elapsed('main'));
const projectRoot = path.resolve(__dirname, '..');
const roleLoader = new ShadowRoleLoader(projectRoot);
const roleConfig = roleLoader.loadRole(roleName);

const mainRepoPath = roleConfig.mainRepoPath;
if (mainRepoPath) {
  const { execSync } = await import('child_process');

  // Auto-init main playground repo if missing
  if (!fs.existsSync(mainRepoPath)) {
    logger.info(agentId, `Initializing playground repo at ${mainRepoPath}...`, timer.elapsed('main'));
    fs.mkdirSync(mainRepoPath, { recursive: true });
    execSync('git init', { cwd: mainRepoPath, stdio: 'pipe' });
    execSync(`git config user.name "shadow-agent"`, { cwd: mainRepoPath, stdio: 'pipe' });
    execSync(`git config user.email "shadow-agent@dadavidtseng.com"`, { cwd: mainRepoPath, stdio: 'pipe' });
    execSync('git commit --allow-empty -m "init: agent-playground"', {
      cwd: mainRepoPath,
      stdio: 'pipe',
    });
    logger.info(agentId, 'Playground repo initialized', timer.elapsed('main'));
  }

  for (const [wtPath, branch] of [
    [roleConfig.workerWorktreePath!, roleConfig.workerBranch],
    [roleConfig.shadowWorktreePath!, roleConfig.shadowBranch],
  ] as const) {
    if (!fs.existsSync(wtPath)) {
      logger.info(agentId, `Worktree not found at ${wtPath} — creating...`, timer.elapsed('main'));
      try {
        execSync(`git worktree add "${wtPath}" -b "${branch}"`, { cwd: mainRepoPath, stdio: 'pipe' });
        logger.info(agentId, `Worktree created: ${wtPath} (branch: ${branch})`, timer.elapsed('main'));
        // Set per-worktree git identity
        execSync(`git config user.name "shadow-agent-${roleConfig.role}"`, { cwd: wtPath, stdio: 'pipe' });
        execSync(`git config user.email "shadow-agent-${roleConfig.role}@dadavidtseng.com"`, { cwd: wtPath, stdio: 'pipe' });
      } catch {
        try {
          execSync(`git worktree add "${wtPath}" "${branch}"`, { cwd: mainRepoPath, stdio: 'pipe' });
          logger.info(agentId, `Worktree created (existing branch): ${wtPath}`, timer.elapsed('main'));
          execSync(`git config user.name "shadow-agent-${roleConfig.role}"`, { cwd: wtPath, stdio: 'pipe' });
          execSync(`git config user.email "shadow-agent-${roleConfig.role}@dadavidtseng.com"`, { cwd: wtPath, stdio: 'pipe' });
        } catch (retryError: any) {
          logger.error(agentId, `Failed to create worktree: ${retryError.message}`, timer.elapsed('main'));
          process.exit(1);
        }
      }
    } else {
      logger.debug(agentId, `Worktree exists: ${wtPath}`, timer.elapsed('main'));
    }
  }
}

Event publishing

• Agent emits events on topics:
  • shadow-{role}.backup.completed
  • shadow-{role}.backup.failed Subscribers can listen via the broker to pick up backup notifications.

Dependencies

• Runtime packages (from package.json “dependencies”):
  • agents-library (core helpers: createShadowAgent, BaseAgent, logger, timer, readConfig, loadVaultCredentials)
  • @anthropic-ai/sdk — optional, used if Anthropic integration is required
  • @modelcontextprotocol/sdk — Model Manager integration
  • chokidar — file watching (likely used by more detailed role logic)
  • zod — config validation
• Dev dependencies include typescript, tsx, vitest, eslint, etc.

Abilities declared in agent.json:

• secret-ability — access to KADI secret/vaults
• ability-file-local — file system operations (worktree creation)
• ability-log — structured logging

Who depends on agent-shadow-worker:

• Worker agents / orchestrators that rely on shadow worktrees and published backup events may subscribe to shadow-{role} topics. The deploy manifests show that the agent expects integration with arcadedb and model manager services for logging and model metadata.

Practical notes for developers

• Role files: check config/roles/{role}.toml for role-specific paths, branches, and monitoring intervals used by ShadowRoleLoader.
• To test a specific role locally, set AGENT_ROLE before starting (or use npm scripts start:artist / dev:artist).
• When modifying worktree logic, preserve the two-step git worktree creation (create -b then fallback to existing branch add) and set per-worktree git identity as shown.
• Vaults: maintain vault definitions in deploy blocks (agent.json) so CI/deploy pipelines supply required secrets. loadVaultCredentials() provides a common fallback source.

If you need help locating the ShadowRoleLoader implementation or role TOML schemas, open roles/ShadowRoleLoader.js and config/roles/ in the repository — the roleConfig object’s fields used here are role, workerWorktreePath, shadowWorktreePath, workerBranch, shadowBranch, monitoringInterval, mainRepoPath.