ability-search

Search engine — chunking, embedding, hybrid text search (deprecated, use ability-graph)

ability-search is a KADI “ability” that provides chunking, embedding, and hybrid search over text content via eight broker-callable tools. It is intentionally a pure intermediary: all persistence is delegated to arcadedb-ability (via the broker) and all embeddings are produced by model-manager (via the broker). The ability exposes indexing, querying, and collection-inspection tools so other agents can index documents, run semantic/keyword/hybrid searches, and inspect collection statistics.

Architecture

• Entrypoint: src/index.ts constructs a KadiClient, loads configuration and secrets via loadSearchConfigWithVault(), registers tools, then connects to the broker.
• Key components:
  • KadiClient (from @kadi.build/core): registers tools and communicates with the broker.
  • Tools (src/tools/*): Grouped into Index, Query, and Collection categories. Each tool is registered via client.registerTool(…).
  • Chunker (src/lib/chunker.ts): Implements multiple chunking strategies (markdown-headers, code-blocks, paragraph, sliding-window, auto).
  • Embedder (src/lib/embedder.js): Wraps model-manager calls to produce embeddings (used by semantic/hybrid search and index).
  • Searcher (src/lib/searcher.js): Implements semanticSearch, keywordSearch, hybridSearch, similarSearch (hybrid uses Reciprocal Rank Fusion).
  • Schema / Vector Index helpers (src/lib/schema.js): ensureSchema, ensureVectorIndex, ensureDatabase: ensure ArcadeDB schema and vector index exist (invokes arcadedb-ability).
  • SQL helpers (src/lib/sql.js): escapeSQL and wrappers to call the arcade-query/arcade-command broker endpoints.
  • Config loader (src/lib/config.ts): loadSearchConfigWithVault() resolves config.yml, env vars, and secrets from the vault (secret-ability).
• Data flow:
  • Indexing: Caller -> search-index/search-index-file -> chunkContent -> embedTexts -> store chunks via arcade-command/arcade-query (arcadedb-ability).
  • Querying: Caller -> search-query/search-similar -> if semantic/hybrid: request embeddings via model-manager -> combine with ArcadeDB vector/keyword queries -> return fused / ranked results.
  • Collection inspection: Caller -> search-collections/search-collection-info -> query ArcadeDB for aggregates and metadata.
• How it fits in AGENTS:
  • An ability exposed on the broker that other agents and clients can invoke.
  • Depends on secret-ability (vault “models”), arcadedb-ability, and model-manager at runtime.
  • Designed to be stateless with all durable state inside ArcadeDB (via arcadedb-ability).

Tools / API

The ability registers eight broker-callable tools. Below is a concise table of each tool and its primary input/behavior.

Tool name	Description	Key input (shape)
search-collections	List all search collections with chunk counts and token statistics.	{} (no input)
search-collection-info	Get statistics and source list for a single collection.	{ collection: string }
search-index	Chunk, embed, and store documents into a collection. Creates schema & vector index automatically on first use.	{ collection: string, documents: Array<{source,title,content,metadata?}>, chunkStrategy?: string, maxTokens?: number, model?: string }
search-index-file	Read a file on disk, detect format, chunk & index its content into a collection.	{ collection: string, path: string, … }
search-reindex	Delete all chunks in a collection. Caller must re-index afterward.	{ collection: string }
search-delete	Delete chunks by collection, optionally filtered by source document.	{ collection: string, source?: string }
search-query	Search a collection using semantic, keyword, or hybrid mode (default: hybrid).	{ collection: string, query: string, limit?: number, mode?: ‘semantic'
search-similar	Find chunks similar to a given chunk id using its embedding vector.	{ collection: string, chunkId: string, limit?: number }

Note: The exact input validation is performed by z schemas when the tools are registered (see code snippets below for the concrete schemas used for search-collections, search-index and search-query).

If you need to call these tools from another agent, use the Kadi broker invoke pattern:

• client.invokeRemote('', inputPayload)

Registered tool grouping functions (API to the KadiClient):

• registerIndexTools(client: KadiClient, config: SearchConfig)
• registerQueryTools(client: KadiClient, config: SearchConfig)
• registerCollectionTools(client: KadiClient, config: SearchConfig)

Configuration

Configuration is resolved in this order (highest wins):

1. Environment variables (e.g. SEARCH_CHUNK_SIZE, SEARCH_API_KEY, SEARCH_EMBEDDING_API_URL)
2. Vault “models” (secrets.toml encrypted vault; loaded via secret-ability)
3. config.yml (walk-up from CWD)
4. Built-in defaults

Files and fields:

• agent.json (included) — contains metadata, brokers, deploy directives and required vault secrets:
  • agent.json.deploy.*.secrets.vault: “models”
  • Required secrets listed in agent.json.deploy.*.secrets.required: [“SEARCH_API_KEY”,“SEARCH_EMBEDDING_API_URL”]
• config.toml (example used by ability; not secrets):
  • [broker.local]
    • URL = “ws://localhost:8080/kadi”
    • NETWORKS = [“search”]
    • MODE = “native”
• config.yml (walk-up) — this is used for non-secret settings (common keys referenced by the code and comments):
  • chunk_size (default chunk size used by chunker)
  • embedding_model (default embedding model e.g., “nomic-embed-text”)
  • database (ArcadeDB database name)
  • embedding_transport / embedding_api_url (overrideable; secrets preferred)
  • other operational flags

Environment variables and secrets:

• SEARCH_CHUNK_SIZE — override chunk size
• SEARCH_API_KEY — required secret (in vault “models”); used to authenticate embedding API
• SEARCH_EMBEDDING_API_URL — required secret (vault)
• BROKER_URL — overrides broker URL resolution in src/index.ts
• NODE_ENV — used by deploy scripts (e.g., production for Akash)

Secrets vault:

• The ability expects secrets in the vault named “models” (agent.json). The startup command in deploys calls: kadi secret receive —vault models which makes SEARCH_API_KEY and SEARCH_EMBEDDING_API_URL available as environment variables or via the secret-ability loader.

Code Examples

Key startup & registration patterns are implemented in src/index.ts. This shows broker resolution, client creation, and tool registration:

// src/index.ts (excerpt)
import { KadiClient } from '@kadi.build/core';
import { loadSearchConfigWithVault } from './lib/config.js';
import { ensureDatabase } from './lib/schema.js';
import { registerCollectionTools } from './tools/collection-tools.js';
import { registerIndexTools } from './tools/index-tools.js';
import { registerQueryTools } from './tools/query-tools.js';

function resolveBrokerUrl(): string {
  if (process.env.BROKER_URL) return process.env.BROKER_URL;

  const agent = loadAgentJson();
  const brokers = agent.brokers ?? {};

  // Prefer remote → default (legacy) → local
  for (const key of ['remote', 'default', 'local']) {
    const entry = brokers[key];
    if (typeof entry === 'string') return entry;
    if (entry?.url) return entry.url;
  }

  throw new Error(
    'No broker URL found. Set BROKER_URL env var or add brokers.remote to agent.json.',
  );
}

const brokerUrl = resolveBrokerUrl();
const agentJson = loadAgentJson();

const client = new KadiClient({
  name: agentJson.name ?? 'search-ability',
  version: agentJson.version ?? '0.1.0',
  brokers: {
    default: { url: brokerUrl },
  },
});

Index tool pattern: chunk -> embed -> persist via arcadedb-ability (see search-index tool registration):

// src/tools/index-tools.ts (excerpt)
client.registerTool(
  {
    name: 'search-index',
    description:
      'Chunk, embed, and store documents in a search collection. Creates schema and vector index automatically on first use.',
    input: z.object({
      collection: z.string().describe('Collection name to index into'),
      documents: z
        .array(
          z.object({
            source: z.string().describe('Unique source identifier (file path, URL)'),
            title: z.string().describe('Human-readable title'),
            content: z.string().describe('Full text content to chunk and index'),
            metadata: z
              .record(z.string(), z.unknown())
              .optional()
              .describe('Arbitrary metadata attached to each chunk'),
          }),
        )
        .min(1)
        .describe('Documents to index'),
      chunkStrategy: z
        .string()
        .optional()
        .describe('Chunking strategy: markdown-headers, code-blocks, paragraph, sliding-window, auto (default: auto)'),
      maxTokens: z
        .number()
        .optional()
        .describe('Max tokens per chunk (default: from config or 500)'),
      model: z
        .string()
        .optional()
        .describe('Embedding model (default: from config or nomic-embed-text)'),
    }),
  },
  async (input) => {
    // implementation: ensureSchema, chunk, embedTexts, write chunks via arcade-command
  }
);

Query tool pattern: choose semantic/keyword/hybrid and dispatch to searcher implementations:

// src/tools/query-tools.ts (excerpt)
client.registerTool(
  {
    name: 'search-query',
    description:
      'Search a collection using semantic, keyword, or hybrid mode. Default mode is hybrid (combines both with Reciprocal Rank Fusion).',
    input: z.object({
      collection: z.string().describe('Collection to search'),
      query: z.string().describe('Search query text'),
      limit: z.number().optional().describe('Max results (default: 10)'),
      mode: z
        .string()
        .optional()
        .describe('Search mode: semantic, keyword, or hybrid (default: hybrid)'),
      model: z
        .string()
        .optional()
        .describe('Embedding model for semantic search (default: from config)'),
    }),
  },
  async (input) => {
    // implementation: switch on mode → semanticSearch|keywordSearch|hybridSearch
  },
);

Collection inspection tool example: search-collections (no input):

// src/tools/collection-tools.ts (excerpt)
client.registerTool(
  {
    name: 'search-collections',
    description:
      'List all search collections with chunk count and token statistics.',
    input: z.object({}),
  },
  async () => {
    try {
      const database = config.database;
      const sql = `SELECT collection, count(*) AS chunks, min(tokens) AS minTokens, max(tokens) AS maxTokens, avg(tokens) AS avgTokens FROM Chunk GROUP BY collection`;

      const response = (await client.invokeRemote('arcade-query', {
        database,
        query: sql,
      })) as ArcadeQueryResult;

      // handle response/result -> return { collections: [...] }
    } catch (err: unknown) {
      // error handling
    }
  },
);

Use these patterns when extending the ability:

• Use client.registerTool(…) with a z input schema to expose new tools.
• Use client.invokeRemote(‘arcade-query’/‘arcade-command’/‘model-manager’ etc.) to interact with other abilities.

Dependencies

• Internal/ability dependencies (declared in agent.json):
  • secret-ability ^0.9.4 — used to load encrypted vault “models” with SEARCH_API_KEY and SEARCH_EMBEDDING_API_URL.
• Runtime collaborators (must be available on the broker):
  • arcadedb-ability — used for persistence and vector/keyword queries via arcade-query and arcade-command RPCs.
  • model-manager (or equivalent embedding provider) — used by embedTexts / semantic/hybrid search paths to produce embedding vectors.
• NPM packages (from source imports and build):
  • @kadi.build/core — Kadi client library used to register tools and communicate over the broker.
  • js-yaml (used by config loader)
  • Others: standard Node libs (fs, path), and local lib modules in src/lib/
• What depends on ability-search:
  • Any agent that needs programmatic search/indexing capabilities in the AGENTS ecosystem can invoke ability-search tools via the broker.
  • Typical consumers: QA agents, retrieval-augmented generation agents, indexing pipelines, or UI agents that present collection statistics.

Notes for Developers / Extending

• Tool registration is centralized in src/tools/. Register additional tools using the existing registerTools(…) pattern so the main startup remains unchanged.
• Persistence is intentionally routed through the broker (arcade-query/arcade-command). Avoid direct DB drivers in this ability — use the arcade RPC endpoints so other deploys can swap DB implementations or scale arcadedb-ability independently.
• Secrets should not be stored in config.yml. Use the vault “models” and the secret-ability loader; see agent.json deploy entries and src/lib/config.ts for the loading behavior.
• By design this ability is stateless: restartable and scalable. All durable data is in ArcadeDB and embeddings are derived on demand.

If you need pointers to modify a specific area (chunker strategies, embedding transport, or search fusion), tell me which file or behavior you’d like to change and I can provide targeted guidance and code edits.