ability-backup

Backup and restore ability

ability-backup is a KADI “ability” that orchestrates backup and restore pipelines for ArcadeDB using brokered tools. It composes local arcadedb export/restore tools, file compression/decompression, cloud-storage tools, and an optional staging server + KĀDI tunnel to support both co-located and fully distributed topologies. It exposes tools for on-demand backups, restores, listing cloud backups, in-memory scheduling, and status reporting. Pipelines return progressive partial results so callers can surface intermediate state on failure.

Architecture

• Purpose: coordinate backup/export on the ArcadeDB host, transfer to cloud storage (direct or via staging server), and perform restores (signed URL preferred, staging fallback).
• Key components:
  • Orchestrator (this ability): registers tools with KadiClient and composes broker RPC calls to other abilities.
  • Arcadedb-ability (broker): performs database export (arcade-backup) and restore (arcade-restore) and serves backup files when co-located.
  • Cloud-storage-ability (broker): performs cloud upload, download, listing, and signed-url issuance.
  • File-manager (broker): compresses/decompresses files (file-compress / file-decompress).
  • Staging server + KĀDI tunnel: used as fallback when signed URLs are not available (creates an HTTP endpoint that cloud-storage-ability can PUT into).
  • Secret-ability: supplies tunnel tokens and other secrets.
• Data flow (backup):
  1. arcade-backup → create archive + start local file-serving endpoint.
  2. cloud-upload-from-url → cloud-storage-ability fetches the file from the arcadedb-host-served URL and uploads to provider.
  3. arcade-backup-cleanup → stop file-serving endpoint.
• Data flow (restore):
  • Primary: cloud-get-download-url → arcade-restore fetches directly from signed URL.
  • Fallback: create staging server + tunnel, cloud-download-to-url → PUT to staging server, arcade-restore pulls from staging URL.
• Progressive failure: each pipeline step returns partial results so callers can recover or inspect partial state.

Tools / API

The ability registers these broker tools with KadiClient. Each row lists the tool name, summary, and primary input fields.

Tool name	Description	Key input fields
backup-database	Back up a database and upload to cloud storage. Returns progressive partial results.	database?: string, provider?: string, compress?: boolean, verify?: boolean, skipUpload?: boolean
backup-list	List available backups in cloud storage. Returns remotePath values that can be passed to backup-restore.	database?: string, provider?: string, limit?: number
backup-restore	Restore a database from a cloud backup. Tries signed URL, falls back to staging server.	database?: string, provider?: string, remotePath: string, overwrite?: boolean
backup-schedule	Create/update/remove an in-memory periodic backup schedule (does not survive restarts).	database?: string, provider?: string, intervalHours?: number (1–720), enabled?: boolean
backup-status	List active in-memory schedules and recent cloud backups for a database.	database?: string

If you use the KadiClient API, these tools are registered by the following functions (see Code Examples below):

• registerBackupTool(client, getStagingServer)
• registerListTool(client)
• registerRestoreTool(client, getStagingServer)
• registerScheduleTool(client)
• registerStatusTool(client)

When orchestrating pipelines the orchestrator calls other broker tools using KadiClient.invokeRemote, for example: client.invokeRemote(‘cloud-list’, {…}) or client.invokeRemote(‘cloud-get-download-url’, {…}).

Configuration

ability-backup uses a combination of config.toml, environment variables, and secrets vaults. The code calls loadConfig(‘cloud’, ‘CLOUD’) and loadConfig(‘backup’, ‘BACKUP’), so environment variable prefixes CLOUD_* and BACKUP_* will be respected as overrides.

config.toml (example, provided with the package)

• broker.local.URL — URL used to connect to broker (default in config.toml: “wss://broker.dadavidtseng.com/kadi”)
• broker.local.NETWORKS — array of network tags (example: [“arcadedb”,“backup”,“global”])

Relevant config keys referenced in code

• cloud.default_provider — default cloud provider (dropbox, googledrive, box)
• cloud.cloud_backup_path — optional base path for cloud backups (used if backup.cloud_backup_path absent)
• backup.cloud_backup_path — preferred base path for backups (example used: /kadi-backups)

Environment variables

• CLOUD_* — overrides for cloud config (e.g., CLOUD_DEFAULT_PROVIDER)
• BACKUP_* — overrides for backup config (e.g., BACKUP_CLOUD_BACKUP_PATH)
• NODE_ENV — used by runtime/deploy scripts (example used in deploy.env in agent.json)
• DASHBOARD_PORT, DASHBOARD_USERNAME, DASHBOARD_PASSWORD — used by deploy/service config (see secrets)

Secrets / Vaults

• There are two vaults referenced in agent.json deploy settings:
  • vault “tunnel”: required secret KADI_TUNNEL_TOKEN — used for establishing KĀDI tunnel for staging-server fallback.
  • vault “backup”: required secrets DASHBOARD_USERNAME and DASHBOARD_PASSWORD — used by the deployed service; stored in the backup secrets vault and delivered via broker in production deployments.
• The code uses secret-ability (declared in agent.json abilities) to fetch secrets when needed.

Deployment metadata (agent.json)

• brokers.remote — default broker endpoint wss://broker.dadavidtseng.com/kadi
• build/deploy instructions reference kadi secret receive —vault backup before kadi run start so secrets are available at container start.

Code Examples

Below are representative code snippets taken from the source to show registration and orchestration patterns.

• registerBackupTool (excerpt from src/tools/backup.ts)

export function registerBackupTool(
  client: KadiClient,
  getStagingServer: () => Promise<StagingServer>,
): void {
  client.registerTool(
    {
      name: 'backup-database',
      description:
        'Back up a database and upload to cloud storage. ' +
        'Works across distributed deployments — arcadedb-ability creates the ' +
        'backup and serves it over HTTP, cloud-storage-ability pulls from that URL. ' +
        'Returns progressive partial results on failure.',
      input: z.object({
        database: z
          .string()
          .optional()
          .describe('Database name to back up (default: "kadi")'),
        provider: z
          .string()
          .optional()
          .describe('Cloud provider (dropbox, googledrive, box). Uses config default if omitted'),
        compress: z
          .boolean()
          .optional()
          .describe('Compress the backup to .tar.gz (default: true)'),
        verify: z
          .boolean()
          .optional()
          .describe('Verify backup integrity (default: true)'),
        skipUpload: z
          .boolean()
          .optional()
          .describe('Skip the cloud upload step (default: false)'),
      }),
    },
    async (input) => {
      const totalStart = performance.now();
      const cloudConfig = loadConfig('cloud', 'CLOUD');
      const database = input.database || 'kadi';
      const provider = input.provider || cloudConfig.default_provider || 'dropbox';
      const shouldCompress = input.compress !== false;
      const verify = input.verify !== false;
      const skipUpload = input.skipUpload === true;

• registerListTool (excerpt from src/tools/list.ts)

export function registerListTool(client: KadiClient): void {
  client.registerTool(
    {
      name: 'backup-list',
      description:
        'List available database backups stored in the cloud. ' +
        'Returns backup files sorted newest-first with path, size, and date. ' +
        'The returned `remotePath` values can be passed directly to `backup-restore`.',
      input: z.object({
        database: z
          .string()
          .optional()
          .describe(
            'Filter by database name. If omitted, lists backups for all databases.',
          ),
        provider: z
          .string()
          .optional()
          .describe(
            'Cloud provider to query (dropbox, googledrive, box). ' +
            'Uses the configured default if omitted.',
          ),
        limit: z
          .number()
          .optional()
          .describe('Maximum number of results to return (default: 20)'),
      }),
    },
    async (input) => {
      try {
        // ── Resolve config ──────────────────────────────────────────
        const cloudConfig = loadConfig('cloud', 'CLOUD');
        const backupConfig = loadConfig('backup', 'BACKUP');
        const provider = input.provider || cloudConfig.default_provider || 'dropbox';
        const basePath = backupConfig.cloud_backup_path || cloudConfig.cloud_backup_path || '/kadi-backups';
        const limit = input.limit ?? 20;

• registerRestoreTool (excerpt from src/tools/restore.ts)

export function registerRestoreTool(
  client: KadiClient,
  getStagingServer: () => Promise<StagingServer>,
): void {
  client.registerTool(
    {
      name: 'backup-restore',
      description:
        'Restore a database from a cloud backup. ' +
        'Tries to get a signed download URL first (fastest path). ' +
        'Falls back to staging server + tunnel if the provider does not support signed URLs. ' +
        'Returns progressive partial results on failure.',
      input: z.object({
        database: z
          .string()
          .optional()
          .describe('Target database name (default: "kadi")'),
        provider: z
          .string()
          .optional()
          .describe('Cloud provider (dropbox, googledrive, box). Uses config default if omitted'),
        remotePath: z
          .string()
          .describe('Remote path of the backup file in cloud storage'),
        overwrite: z
          .boolean()
          .optional()
          .describe('Overwrite target database if it already exists (default: false)'),
      }),
    },
    async (input) => {
      const totalStart = performance.now();
      const cloudConfig = loadConfig('cloud', 'CLOUD');
      const database = input.database || 'kadi';
      const provider = input.provider || cloudConfig.default_provider || 'dropbox';
      const remotePath = input.remotePath;

• registerScheduleTool (excerpt from src/tools/schedule.ts)

export function registerScheduleTool(client: KadiClient): void {
  client.registerTool(
    {
      name: 'backup-schedule',
      description:
        'Create, update, or remove a periodic backup schedule. ' +
        'Schedules are in-memory only — they do not survive agent restarts. ' +
        'Set enabled: false to remove a schedule. ' +
        'Min interval: 1 hour, max interval: 720 hours (30 days).',
      input: z.object({
        database: z
          .string()
          .optional()
          .describe('Database to schedule backups for (default: "kadi")'),
        provider: z
          .string()
          .optional()
          .describe('Cloud provider for backup uploads (default: "dropbox")'),
        intervalHours: z
          .number()
          .min(1)
          .max(720)
          .optional()
          .describe('Hours between backups (default: 24, min: 1, max: 720)'),
        enabled: z
          .boolean()
          .optional()
          .describe('true to create/update schedule, false to remove (default: true)'),
      }),
    },
    async (input) => {
      try {
        const database = input.database || 'kadi';
        const provider = input.provider || 'dropbox';
        const intervalHours = input.intervalHours ?? 24;
        const enabled = input.enabled !== false;

        if (!enabled) {
          // Remove all schedules for this database
          const count = removeSchedulesByDatabase(database);
          if (count === 0) {
            return {
              success: true,
              status: 'removed',
              message: `No active schedules found for database "${database}"`,
              database,
              activeSchedules: listSchedules(),
            };
          }

• registerStatusTool (excerpt from src/tools/status.ts)

export function registerStatusTool(client: KadiClient): void {
  client.registerTool(
    {
      name: 'backup-status',
      description:
        'List active backup schedules and recent cloud backups for a database. ' +
        'Shows schedule metadata, recent backup files (newest 10), and current configuration.',
      input: z.object({
        database: z
          .string()
          .optional()
          .describe('Database to check status for (default: all databases)'),
      }),
    },
    async (input) => {
      try {
        const database = input.database;

        // ── Schedules ─────────────────────────────────────────────
        const schedules = database
          ? getSchedulesByDatabase(database)
          : listSchedules();

        // ── Recent cloud backups ──────────────────────────────────
        const cloudConfig = loadConfig('cloud', 'CLOUD');
        const backupConfig = loadConfig('backup', 'BACKUP');
        const defaultProvider = cloudConfig.default_provider || 'dropbox';
        const cloudBasePath = '/kadi-backups';

Pattern notes

• Tool registration uses KadiClient.registerTool with a Zod input schema (z.object).
• Orchestration invokes remote tools via client.invokeRemote(‘tool-name’, payload).
• loadConfig(‘cloud’,‘CLOUD’) and loadConfig(‘backup’,‘BACKUP’) are used to resolve defaults and paths.
• The code intentionally returns partial progress on failure to help callers diagnose or attempt recovery.

Dependencies

• Broker/ability dependencies (declared / expected to be present on the broker network):
  • arcadedb-ability (arcade-backup, arcade-restore, arcade-db-info)
  • cloud-storage-ability (cloud-upload, cloud-download, cloud-list, cloud-get-download-url, cloud-upload-from-url, cloud-download-to-url)
  • file-manager (file-compress, file-decompress)
  • secret-ability (secret-get) — declared in agent.json as an ability dependency
• Direct npm dependencies (from agent comment and package):
  • @kadi.build/core — KadiClient, Zod types, registerTool/invokeRemote primitives
  • @kadi.build/file-sharing — FileSharingServer for staging server (used by staging-server lib)
  • @kadi.build/tunnel-services — KĀDI tunnel client (staging-server fallback)
• What’s likely to call / depend on ability-backup:
  • UIs/dashboards and automation scripts which invoke these tools via the broker
  • Scheduled in-memory schedules (created via backup-schedule) will call backup-database periodically until the ability restarts (schedules are in-memory v1)
• What ability-backup depends on (from agent.json):
  • secret-ability (wildcard version)

Operational notes

• Schedules are in-memory only. For durable scheduling, persist schedule metadata externally and re-create on agent start.
• Backup base path conventions: backups stored under /kadi-backups/{database}/; list and restore tools expect remotePath values returned by cloud-list.
• Preferred restore path: providers that support signed URLs are used first (no staging/tunnel). Staging server + KADI tunnel is fallback for providers without signed URL or when direct access is not possible.
• Deploy scripts use “kadi secret receive —vault backup” to fetch runtime secrets via broker before running.

If you need more focused examples (invoking tools via KadiClient or implementing a custom cloud provider adapter), tell me which integration you want and I will provide targeted snippets and patterns.