ability-graph

Graph engine — N-signal hybrid search, chunking, entity dedup, ArcadeDB

ability-graph is a general-purpose graph storage and retrieval engine packaged as a Kadi “ability” (type: kadi-ability). It implements batched ingestion (embedding + metadata extraction), schema registry, hybrid N-signal recall (semantic/keyword/graph), neighbor expansion, and direct ArcadeDB SQL passthrough. It can run natively or be exposed as a remote broker ability and integrates with model-manager for embedding/chat calls and with a secrets vault for API credentials.

Architecture

• Data flow (ingest -> index -> recall):
  1. Ingest: client calls graph-batch-store with items. Items go through embedder (embedTexts) and extractor (extractMetadata) then upserted into ArcadeDB via graph lib functions (createVertex, upsertEntity, createEdge, updateVertex).
  2. Background jobs: long-running ingestion/extraction can be scheduled through jobManager (supports background operations and progress tracking).
  3. Index & signals: recall uses a hybridRecall that combines semantic embeddings (embedding model), keyword signals, and graph signals (via traversals and/or ArcadeDB queries).
  4. Recall + context expansion: graph-context performs recall then traverseGraph to include neighbors and edges for richer context for downstream agents.
  5. Mutations & queries: graph-command and graph-find/graph-count provide write and read SQL layers that invoke arcade-command / arcade-query broker tools via retry wrapper.
• Key components:
  • tools/*: tool registration functions that expose capabilities as Kadi tools (e.g., graph-batch-store, graph-chat, graph-find).
  • lib/embedder.js, lib/extractor.js: embedding & metadata extraction.
  • lib/graph.js: ArcadeDB-oriented graph primitives (vertex/edge upsert, traversal, orphan detection).
  • lib/signals/*: hybrid recall & signal implementations.
  • lib/retry.js: invokeWithRetry to call broker tools (arcade-command, arcade-query, chat-completion) with backoff.
  • schema-registry, job-manager: schema enforcement and background job orchestration.
• How it fits in the AGENTS ecosystem:
  • Provides persistent graph storage and recall services to other agents via Kadi broker tools.
  • Can call model-manager directly (HTTP) or through broker-based chat-completion/embedding abilities.
  • Declares dependency on secret-ability (for retrieving model-manager secrets), and on the “arcade” broker tools (arcade-command / arcade-query) for DB operations.

Tools / API

The ability registers the following tools (registered via client.registerTool). Each table row documents the tool name, exposing function, short description, and key parameters.

Tool name	Register function	Description	Key parameters
graph-batch-store	registerBatchStoreTool	Bulk ingest items with batched embedding + parallel extraction. Non-transactional per-vertex upserts.	items (content, vertexType, properties, topics, entities, edges), vertexType, database, concurrency, batchSize, onDuplicate (skip/replace/error), deduplicateBy, background
graph-chat	registerChatTool	Send chat completion requests via model-manager HTTP API or fallback to broker ‘chat-completion’.	messages (role/content array), model, temperature, max_tokens, api_key
graph-command	registerCommandTool	Execute write SQL commands (CREATE/UPDATE/DELETE) against ArcadeDB via ‘arcade-command’.	command (SQL), database
graph-context	registerContextTool	Recall vertices then expand via graph traversal to include neighbors and edges.	query, vertexType, depth, limit, filters, signals, database
graph-count	registerCountTool	Count vertices by type, optional WHERE filters and GROUP BY.	vertexType, filters, groupBy, database
graph-delete	registerDeleteTool	Delete a vertex by RID with optional cascade deletion of orphaned Topic/Entity vertices.	rid, cascade, database
graph-find	registerFindTool	Find vertices by type + filters; returns vertex properties (simpler than graph-query).	vertexType, filters, orderBy, limit, fields, database

Notes:

• Tools use zod schemas (z.object) to validate inputs.
• Tools call broker-side helpers via a SignalAbilities.invoke wrapper that maps to client.invokeRemote(tool, params).

Configuration

Primary configuration is in config.toml and secrets are expected in a secrets vault (secrets.toml delivered via Kadi secret broker).

Relevant config.toml fields:

• [broker.local]
  • URL: ws URL for local broker (example: “ws://localhost:8080/kadi”)
  • NETWORKS: list of networks ([“graph”])
  • MODE: “native” or other modes
• [broker.remote]
  • URL: remote broker URL (example: “wss://broker.dadavidtseng.com/kadi”)
  • NETWORKS: [“graph”]
  • MODE: “native”
• [graph]
  • database: default database name (example: “agents_memory”)
  • embedding_model: embedding model id (example: “text-embedding-3-small”)
  • extraction_model: extraction model id (example: “gpt-5-nano”)
  • chat_model: chat model id (example: “gpt-5-mini”)
  • default_agent: default agent id/name (example: “default”)
  • embedding_transport: “api” or “broker”
  • chat_transport: “api” or “broker”
  • (runtime) apiUrl: optional model-manager base URL (referenced in code as config.apiUrl)
  • (runtime) apiKey: optional model-manager API key (referenced as config.apiKey)

Environment variables and secrets:

• Secrets are delivered via the Kadi secret vault specified in agent.json deploy configuration:
  • Vault name: model-manager
  • Required keys (example): MODEL_MANAGER_BASE_URL, MODEL_MANAGER_API_KEY
• These secrets are consumed by graph-chat and embedding HTTP calls when chatTransport/embeddingTransport === ‘api’.
• Delivery method: broker (see agent.json.deploy.local.services.agent.command uses kadi secret receive —vault model-manager)

agent.json-specific configuration:

• “abilities”: { “secret-ability”: ”*” } — declares it depends on secret-ability to access secrets.
• “brokers”: local and remote endpoints are declared and mirrored in config.toml.
• “deploy” block shows the required secrets and how they are delivered.

Secrets best practices:

• Store model-manager credentials in the encrypted vault delivered by the broker.
• Do not commit secrets.toml to source control.

Code Examples

Below are representative snippets copied from the source demonstrating key patterns and actual function implementations you will modify.

1. graph-batch-store registration (ingest pattern, abilities wrapper):

import { KadiClient, z } from '@kadi.build/core';

import type { GraphConfig } from '../lib/config.js';
import { embedTexts, type EmbedResult } from '../lib/embedder.js';
import { extractMetadata } from '../lib/extractor.js';
import {
  createEdge,
  createVertex,
  extractRid,
  queryVertices,
  updateVertex,
  upsertEntity,
  upsertTopic,
} from '../lib/graph.js';
import { jobManager } from '../lib/job-manager.js';
import { schemaRegistry } from '../lib/schema-registry.js';
import { escapeSQL } from '../lib/sql.js';
import type { BatchItem, SignalAbilities } from '../lib/types.js';

export function registerBatchStoreTool(
  client: KadiClient,
  config: GraphConfig,
): void {
  const abilities: SignalAbilities = {
    invoke: <T>(tool: string, params: Record<string, unknown>) =>
      client.invokeRemote(tool, params) as Promise<T>,
  };

  client.registerTool(
    {
      name: 'graph-batch-store',
      description:
        'Bulk store multiple items with batched embedding and parallel extraction. ' +
        'Each vertex is created via individual DB writes (not transactional batch). ' +
        'Supports dedup strategies (skip, replace) and progress tracking.',
      input: z.object({
        items: z.array(z.object({
          content: z.string(),
          vertexType: z.string().optional(),
          properties: z.record(z.string(), z.unknown()).optional(),
          topics: z.array(z.string()).optional(),
          entities: z.array(z.object({ name: z.string(), type: z.string() })).optional(),
          edges: z.array(z.object({
            type: z.string(),
            direction: z.enum(['out', 'in']),
            targetRid: z.string().optional(),
            targetQuery: z.object({
              vertexType: z.string(),
              where: z.record(z.string(), z.unknown()),
            }).optional(),
            properties: z.record(z.string(), z.unknown()).optional(),
          })).optional(),
          skipExtraction: z.boolean().optional(),
          importance: z.number().optional(),
        })).describe('Items to store'),
        vertexType: z.string().optional().describe('Default vertex type for all items'),
        database: z.string().optional().describe('Target database'),
        background: z.boolean().optional().describe('Run as background job (default: false)'),
        concurrency: z.number().optional().describe('Parallel extraction (default: 5)'),
        batchSize: z.number().optional().describe('Embedding batch size (default: 100)'),
        onDuplicate: z.enum(['skip', 'replace', 'error']).optional()
          .describe('Dedup strategy (default: error)'),
        deduplicateBy: z.array(z.string()).optional()
          .describe('Properties for duplicate detection'),
      }),
    },
    async (input) => {
      const startTime = Date.now();
      const database = input.database ?? config.database;
      const concurrency = input.concurrency ?? 5;
      const batchSize = input.batchSize ?? 100;

2. graph-chat registration (model manager HTTP vs broker fallback):

import { KadiClient, z } from '@kadi.build/core';

import type { GraphConfig } from '../lib/config.js';
import { invokeWithRetry } from '../lib/retry.js';
import type { SignalAbilities } from '../lib/types.js';

export function registerChatTool(
  client: KadiClient,
  config: GraphConfig,
): void {
  const abilities: SignalAbilities = {
    invoke: <T>(tool: string, params: Record<string, unknown>) =>
      client.invokeRemote(tool, params) as Promise<T>,
  };

  client.registerTool(
    {
      name: 'graph-chat',
      description:
        'Send a chat completion request via the model manager. Supports system and user ' +
        'messages with configurable temperature and token limits.',
      input: z.object({
        messages: z.array(z.object({
          role: z.string().describe('Message role (e.g., system, user, assistant)'),
          content: z.string().describe('Message content'),
        })).describe('Chat messages to send'),
        model: z.string().optional().describe('Model to use (default: from config)'),
        temperature: z.number().optional().describe('Sampling temperature (default: 0.7)'),
        max_tokens: z.number().optional().describe('Maximum tokens to generate (default: 500)'),
        api_key: z.string().optional().describe('API key override'),
      }),
    },
    async (input) => {
      try {
        const apiKey = input.api_key ?? config.apiKey;
        const model = input.model ?? config.chatModel;
        const temperature = input.temperature ?? 0.7;
        const maxTokens = input.max_tokens ?? 500;

        // Direct HTTP call to model-manager (OpenAI-compatible)
        if (config.chatTransport === 'api' && config.apiUrl && apiKey) {
          const result = await callModelManagerHTTP(
            config.apiUrl, apiKey, model, input.messages, temperature, maxTokens,
          );
          return { success: true, result };
        }

        // Fallback: broker-based chat-completion tool
        const params: Record<string, unknown> = {
          model,
          messages: input.messages,
          temperature,
          max_tokens: maxTokens,
          api_key: apiKey,
        };

        const result = await invokeWithRetry(abilities, 'chat-completion', params);
        return { success: true, result };
      } catch (err: unknown) {
        const message = err instanceof Error ? err.message : String(err);
        return {
          success: false,
          error: `[graph-chat] ${message}`,
          tool: 'graph-chat',
        };
      }
    },
  );
}

3. graph-command registration (SQL write passthrough):

import { KadiClient, z } from '@kadi.build/core';

import type { GraphConfig } from '../lib/config.js';
import { invokeWithRetry } from '../lib/retry.js';
import type { ArcadeCommandResult, SignalAbilities } from '../lib/types.js';

export function registerCommandTool(
  client: KadiClient,
  config: GraphConfig,
): void {
  const abilities: SignalAbilities = {
    invoke: <T>(tool: string, params: Record<string, unknown>) =>
      client.invokeRemote(tool, params) as Promise<T>,
  };

  client.registerTool(
    {
      name: 'graph-command',
      description:
        'Execute a write SQL command against the graph database. Use for CREATE, UPDATE, ' +
        'DELETE, and other mutating operations.',
      input: z.object({
        command: z.string().describe('The SQL command to execute'),
        database: z.string().optional().describe('Target database (default: from config)'),
      }),
    },
    async (input) => {
      try {
        const database = input.database ?? config.database;

        const response = await invokeWithRetry<ArcadeCommandResult>(
          abilities,
          'arcade-command',
          { database, command: input.command },
        );

        return {
          success: true,
          result: response.result,
        };
      } catch (err: unknown) {
        const message = err instanceof Error ? err.message : String(err);
        return {
          success: false,
          error: `[graph-command] ${message}`,
          tool: 'graph-command',
        };
      }
    },
  );
}

4. graph-context registration (recall + expand pattern):

import { KadiClient, z } from '@kadi.build/core';

import type { GraphConfig } from '../lib/config.js';
import { traverseGraph } from '../lib/graph.js';
import { hybridRecall } from '../lib/signals/index.js';
import type { RecallRequest, SignalAbilities } from '../lib/types.js';

export function registerContextTool(
  client: KadiClient,
  config: GraphConfig,
): void {
  const abilities: SignalAbilities = {
    invoke: <T>(tool: string, params: Record<string, unknown>) =>
      client.invokeRemote(tool, params) as Promise<T>,
  };

  client.registerTool(
    {
      name: 'graph-context',
      description:
        'Recall vertices then expand via graph traversal for richer context. ' +
        'Returns both the recalled results and their connected neighbors.',
      input: z.object({
        query: z.string().describe('Search query'),
        vertexType: z.string().describe('Vertex type to search'),
        depth: z.number().optional().describe('Traversal depth (default: 1, max: 4)'),
        limit: z.number().optional().describe('Max recalled results to expand (default: 5)'),
        filters: z.record(z.string(), z.unknown()).optional().describe('Additional filters'),
        signals: z.array(z.string()).optional().describe('Recall signals'),
        database: z.string().optional().describe('Target database'),
      }),
    },
    async (input) => {
      try {
        const database = input.database ?? config.database;
        const depth = Math.max(1, Math.min(4, input.depth ?? 1));
        const limit = input.limit ?? 5;

        // Step 1: Recall top results
        const request: RecallRequest = {
          query: input.query,
          vertexType: input.vertexType,
          mode: 'hybrid',
          signals: input.signals ?? ['semantic', 'keyword', 'graph'],
          filters: input.filters,
          limit,
          database,
        };

        const recalled = await hybridRecall(request, abilities, config);

        // Step 2: Expand each result via graph traversal
        const contextResults: Array<Record<string, unknown>> = [];

        for (const result of recalled) {
          if (!result.rid) continue;

          const graph = await traverseGraph(
            abilities,
            database,
            result.rid,
            depth,
            input.filters,
          );

          contextResults.push({
            ...result,
            neighbors: graph.vertices.filter((v) => v.rid !== result.rid),
            edges: graph.edges,
          });
        }

        return {

Additional registration patterns (graph-count, graph-delete, graph-find) follow the same pattern: create SignalAbilities.invoke wrapper, client.registerTool with zod input, use invokeWithRetry to call arcade-query/arcade-command or graph lib helpers, and return standardized { success, result } or error shape.

Dependencies

• Declared ability dependencies:
  • secret-ability (declared in agent.json). Used to retrieve model-manager secrets from vault.
• External packages:
  • @kadi.build/core (KadiClient, zod integration, tool registration APIs)
  • kadi CLI / runtime (used in build/deploy scripts)
  • TypeScript (build)
• Broker tools / services invoked:
  • arcade-command (for write SQL): invoked via invokeWithRetry as ‘arcade-command’
  • arcade-query (for read SQL): invoked as ‘arcade-query’
  • chat-completion (fallback broker chat tool)
  • model-manager HTTP API (embedding/chat) when embedding_transport/chat_transport === ‘api’
• Internal modules (code-level dependencies):
  • lib/embedder.js, lib/extractor.js, lib/graph.js, lib/signals, lib/retry.js, lib/sql.js, lib/schema-registry.js, lib/job-manager.js
• What depends on ability-graph:
  • Any agents or abilities that require persistent graph storage, recall, or embedding-based search should call these registered ‘graph-*’ tools (via client.invokeRemote).
  • Example consumer patterns: a conversational agent retrieving context via graph-context, an ingestion pipeline calling graph-batch-store, or an admin UI calling graph-find/graph-count.

Agents: role and interactions

• Role of ability-graph:
  • Acts as the canonical graph storage/recall backend for the multi-agent system.
  • Exposes discrete tools as Kadi tools so any agent (local or remote) can call graph-* endpoints over the broker.
• How agents interact:
  • Agents create a KadiClient and call client.invokeRemote(‘graph-find’, { vertexType: ‘Memory’, filters: { … } }) or other graph-* tools.
  • For model operations, graph-chat prefers a direct HTTP call to model-manager using secrets delivered from secret-ability (MODEL_MANAGER_BASE_URL, MODEL_MANAGER_API_KEY); if not available or configured with chat_transport=‘broker’ it will call ‘chat-completion’ via the broker (invokeWithRetry).
  • For database writes/reads, graph-command and graph-count use invokeWithRetry to call ‘arcade-command’ and ‘arcade-query’ broker tools to ensure backoff and retries.
• Background jobs and long-running tasks:
  • Agents can request ingestion with background: true; jobManager will schedule and track progress allowing agents to poll job state rather than block.

Tips for maintainers

• When adding new tools, follow the established pattern:
  • Create SignalAbilities.invoke = client.invokeRemote wrapper.
  • Register tool with client.registerTool using zod schema for input validation.
  • Use invokeWithRetry for broker calls to arcade-* or chat-completion.
  • Return consistent { success, result } or { success: false, error, tool } shapes.
• If you change model-manager integration, update both config fields (config.apiUrl / config.apiKey) and ensure secrets are declared in agent.json.deploy.secrets for proper delivery.
• For schema changes, update schema-registry and ensure graph-batch-store uses upsertEntity/upsertTopic appropriately to avoid duplicate entity vertices.

If you need documentation for any specific tool’s full input schema or the internals of lib/graph.js (traversal, upsert semantics), tell me which file and I can extract the exact functions and signatures next.