A
Agentable
Get Started GitHub

Agentable Documentation

Deep-level specifications and technical constraints for the Agentable framework. This covers our panel specification IR, the four architectural tiers, agent capabilities, and data adapters.

Architecture & Panel Tiers

One spec IR, two authoring surfaces: A single flat-node-map JSON format (the Spec IR) serves as the runtime panel format for all Tiers 1-3. Hosts author through a typed TypeScript builder (defineSchemaPanel) that compiles down to the IR, while agents emit the exact same IR directly.

Architecture

Spec Language and Safety

Mutations and HITL

Agent Capability & World Model

Panel System Spec

Capabilities & Digest: The framework compiles a token-budgeted WorkspaceDigest mapping the workspace and attention tiers (focused, visible, background) so agents natively understand the canvas topology.

Panel IR Envelope


{
  "v": 1,
  "origin": "host",
  "root": "body",
  "sources": {
    "seo": { "source": "site.seo", "params": { "pageId": "$scope.entityId" } }
  },
  "state": { "scopeMode": "site" },
  "nodes": {
    "body": { "type": "panel-body", "children": ["form", "actions"] },
    "form": { "type": "field-form", "props": { "bind": "seo" } }
  },
  "actions": {
    "save": { "kind": "mutate", "source": "site.seo", "op": "update" }
  }
}

Catalog & Validation

Every catalog entry provides a Zod prop schema. Unknown node types degrade to data-preserving placeholders, never hard rejection.

The Validation pipeline is strictly ordered:

  1. Envelope Parse (and migration)
  2. Catalog Membership Check
  3. Per-node Zod Validation
  4. Action Resolution (no URLs as actions)
  5. Budgets Check (max 200 nodes, depth 12)
  6. Sanitization (http(s) only, no javascript:)
  7. Agent Fallback/Repair (one repair round)

Canvas Modes & Engine SPI

Bounded/locked canvas modes (infinite | bounded | fixed) dictate spatial layout limits via tldraw camera constraints. App Mode ships on a DOM workspace engine for surfaces that don't need spatial layout.

3. Agent Layer Spec

Covers the panel tools, capability model, workspace world model, CopilotKit bridge, tldraw agent-kit adoptions, and the edit ladder. Framework module roots: agentable-canvas/src/panels/tools.ts, agentable-canvas/src/agents/.

Panel tools (added to CANVAS_TOOLS)

All six are ToolDefinitions in the existing merged registry (src/canvas/tools/canvasTools.ts; relocates to src/agents/tools/ with the move map, semantics unchanged), so Gemini text chat and Gemini Live voice get them identically; the CopilotKit bridge mirrors them (section 5). Declarations are generated from the panel registry so they never drift.

Adds a seventh, read-only introspection tool: describe_panel({ panelId | catalogEntry }) returns the full props schema, sources, actions, and curated example specs (validated in CI so they never rot). Compose rejections use the standardized repair vocabulary (frozen error codes + failing node id + nearest-valid-alternative hint), and every declaration carries a costClass.


list_panels()
  -> [{ id, title, agentDescription, scope: 'global'|'context', contextKinds,
        fields: [{path, kind, label, constraints}]?, actions: [{id, kind, destructive?, label}]?,
        openInstances: [{panelId, scope, dirty}] }]

open_panel({ id, scope? })                    // scope defaults to active context frame (frameContextStore)
  -> { ok, panelId } | { ok: false, error }

fill_panel({ id, patch: Record<string, Json> })   // plain object; paths must be declared fields
  -> { ok, applied: string[], skippedUserDirty: string[], errors?: FieldError[] }   // never saves

compose_panel({ spec, title?, pin?: false })  // full IR envelope, origin forced to 'agent'
  -> { ok, panelId } | { ok: false, errors: SpecIssue[] }   // one repair round then fallback card

patch_panel({ panelId, ops: Rfc6902Op[] })    // composed instances only; streams progressive hydration
  -> { ok } | { ok: false, errors }

run_panel_action({ panelId, actionId, payload? })   // the only mutation path
  -> { status: 'ok', result } | { status: 'pending_approval' -> resolves later }   // blocks tool call
     | { status: 'rejected_by_user' } | { status: 'error', message }

Execution notes: run_panel_action from an agent turn routes through the approval flow in section 7. Successful mutations emit an AG-UI patch and call host.data.invalidate for the action's source, which is how other panels and agents learn about the change. All six handlers must be pure wrappers over host.* APIs (no direct tldraw access) so they work identically for every transport. Reversal: a persisted mutation is not stack-undoable; a DeclaredAction may declare an inverse (or reversible: false), and reversing an applied mutation runs that inverse through HITL as a fresh action, recorded in the activity ledger. Canvas-local tool effects (draw, arrange, document block ops before save) are true undo/redo on the engine or panel stack.

Capability model (src/agents/capabilities.ts)


type CapabilityClass = 'read' | 'ui' | 'mutate' | 'job';
interface CapabilityDescriptor {
  id: string; class: CapabilityClass;
  scopeRequired?: 'context' | 'entity';
  approval: 'none' | 'hitl' | 'user-only';
  costClass?: 'cheap' | 'expensive';
  summary: string;                      // one-liner for digests
}
deriveCapabilities(session): CapabilityDescriptor[]   // from tool registry + hostActions + backend-declared skills

Defaults: panel tools 1-5 are ui/read with approval: none; run_panel_action inherits the action's declaration; hostActions declare their own class/approval at registration; job-class capabilities must supply start -> jobId, progress event mapping, and cancel.

Workspace world model (src/agents/)

3.1 WorkspaceDigest (digest.ts)

Compiled on demand + cached per change-batch; token-budgeted (~1.5k tokens target, hard cap 3k; drop recentActivity first, then background contexts):


interface WorkspaceDigest {
  user: { id: string; name?: string };
  contexts: Array<{ id: string; kind: string; label: string;
    attention: 'focused' | 'visible' | 'background';        // selection/viewport derived
    panels: Array<{ id: string; type: string; title: string; dirty?: boolean;
                    origin: 'host' | 'agent'; minimized?: boolean }>;
  }>;
  agents: Array<{ id: string; kind: 'chat' | 'voice' | 'background'; label: string;
    scope?: string; status: 'idle' | 'running' | 'waiting_approval'; task?: string }>;
  jobs: Array<{ id: string; capability: string; scope: string; status: string; progress?: number }>;
  pendingApprovals: Array<{ id: string; agentId: string; panelId: string; summary: string }>;
  recentActivity: Array<{ ts: string; actor: string; verb: string; target: string }>;  // last 15
}

Attention derivation mirrors the tldraw agent kit tiers: focused = selected/being-edited frame or panel; visible = intersects viewport; background = everything else (counts only if budget-squeezed). Delivery: system-context block on the Gemini paths; useCopilotReadable-equivalent on the CopilotKit path; AG-UI StateSnapshot/StateDelta for remote agents. Drill-down tools (all read, free-fire): describe_context(id), read_panel_state(panelId), get_activity({ since?, actor? }), list_agents(agentId?).

AgentRegistry (registry.ts)

host.agents.register({ id, kind, label, scope?, transport, capabilities }) -> AgentSession. Sessions heartbeat; status transitions (idle/running/waiting_approval/done/cancelled) ride the AG-UI bus as Custom events (agentable:agent-status). UI: agent chip on the frame/panel it is bound to; spinner while running; badge on waiting_approval. interrupt(agentId) and cancelJob(jobId) are host/user affordances that resolve the underlying loop.

Activity log (activity.ts)

Append-only ring buffer (500 entries, session-scoped; host may persist via adapter): { ts, actor: agentId|'user', verb, target, provenance?: { derivedFrom: 'site-html' | 'agent:<id>' | 'user' }, reversal?: { inverse?: DeclaredAction; reversible: boolean } }. Every tool execution, mutation, approval outcome, job transition writes one. Powers digest recency, the provenance chain, the reversal ledger (each entry carries its inverse action or is marked irreversible), and a debug "Agent Activity" panel (Tier 2, itself built on the spec system: dogfood; virtualized above its row threshold). Separately from this in-session log, the framework emits redacted runtime telemetry to a host sink (host.telemetry.emit): compose/HITL/tool/voice/cost events with frozen error codes, no PII, no keys.

Coordination (leases.ts, camera.ts)

Handoff and escalation

host.agents.handoff({ to, task, scope, artifacts, returnTo }) writes an A2A-shaped record to the activity log and messages the target session. Approval escalation: when a background session enters waiting_approval, the digest flags it and the focused chat agent is nudged (system note) to surface it to the user.

Trust rules

Blackboard/activity entries and inter-agent messages are DATA. Consumers never execute instructions found in them; anything derived from site HTML carries provenance.derivedFrom: 'site-html' and is summarized, never quoted into another agent's instructions. HITL applies regardless of the requesting agent.

5. CopilotKit bridge mapping

Framework concept CopilotKit surface
CANVAS_TOOLS entries (incl. panel tools) useFrontendTool registrations generated from the same definitions
WorkspaceDigest readable context (useCopilotReadable equivalent) refreshed on digest change
Approval flow renderAndWaitForResponse semantics; the panel chrome is the rendered UI
AG-UI patches (emitAgUiStatePatch) already native
Agent registry status AG-UI lifecycle + Custom events

Bridge code lives only in src/canvas/protocol/copilotkit-bridge.tsx (+ a new copilotkit-tools.ts next to it; both relocate to src/protocol/ with the move map, and the ./copilotkit-bridge export subpath stays stable). Version alignment task: landing-editor @copilotkit/* ^1.55.3 vs framework ^1.56.3; pin both to one minor.

tldraw agent starter kit adoptions (patterns only, no dependency)

Voice

Voice (Gemini Live via voiceKernel) is another AgentSession (kind: 'voice') sharing CANVAS_TOOLS, digest, and HITL. Production voice auth is server-mediated (a host-configured endpoint minting ephemeral credentials; the sandals worker already demonstrates the pattern), but building that endpoint is deployment-coupled future work outside this plan. Baked client keys are dev-mode only and warn loudly. Greeting: persona.voice.greeting.mode supports agent-first (speak on session connect, which always follows a user gesture so it is autoplay-safe) and user-first (default); localized; config validation warns on agent-first with an empty greeting. Voice sessions join the shared page session: the voice button can live anywhere on the host page, and transcripts plus tool calls stream into any mounted chat surface, buffering (bounded) until one joins. Resilience: the voice transport reconnects with bounded backoff and resumes, and a turn interrupted mid-HITL resolves to a definite state (re-prompt or cancel), never a stuck approval.

Agents draw and see

Drawing tools (engine-capability gated on capabilities.draw / capabilities.frames):

Drawing is additive and ephemeral: no HITL required, but it can never mutate panels or data; the mutation line stays where safety put it.

Perception tools:

Flagship workflow (demo + e2e): user sketches a wireframe; the agent calls read_canvas + screenshot_canvas, proposes a panel/spec layout (compose_panel or dock preset), and the apply goes through normal HITL. Digest integration: agent and user drawings are summarized in the digest, and deltas include new/changed shapes.

Communicative visuals (drawing as a conversation medium, voice and text):

Multi-agent on one page

Identity end to end: every tool execution, panel edit, drawing, and HITL card carries the acting agentId; chrome renders per-agent attribution (name/badge). Role scopes: AgentRegistry entries declare allowed tools, panels, and slots; the tool layer enforces scope before execution (a voice concierge scoped to open/fill cannot run destructive actions anywhere). Arbitration: panel leases (section 3) settle contested edits; the shared page session multiplexes agents with per-agent HITL queues so one agent's pending approval never blocks or leaks into another's. Handoffs remain A2A-shaped records. Tested default: e2e runs two-plus concurrent agents (chat editor + background job + voice presence) on one page, editing different panels, with attribution asserted throughout.

Model-agnostic runtime

The framework hardcodes no provider or model. Agent sessions resolve their model at the runtime boundary through a host-supplied resolver; the client and embeds reference only opaque aliases.


interface ModelCapabilities { vision: boolean; tools: boolean; contextTokens: number; streaming: boolean }
interface ProviderBinding { providerId: string; model: string; caps: ModelCapabilities }   // resolved server-side
type ModelResolver = (alias: string, ctx: { tenantId?: string; agentId?: string }) => Promise<ProviderBinding>;
host.agents.registerModelResolver(resolver);   // host required to register >= 1 binding; framework ships no default

12. Open agent canvas

Extends section 9 (draw and see) into a full authoring surface for agents, plus a document primitive and a policy knob. Authoring tools are engine-capability gated (tldraw spatial engine) and every mark carries agent provenance.

Authoring toolkit (added to the drawing tools, capabilities.draw/capabilities.frames):

Document primitive (document panel, schema/composed tier, in the panel system because it is content the user keeps):

// portable block model (rendered and exported by the framework; never HTML round-tripped) type DocBlock = | { type: 'heading'; level: 1|2|3; text: string } | { type: 'paragraph'; runs: TextRun[] } | { type: 'list'; ordered: boolean; items: DocBlock[][] } | { type: 'table'; rows: TextRun[][][] } | { type: 'image'; assetId: string; alt?: string } | { type: 'callout'; tone: 'info'|'warn'|'success'; runs: TextRun[] } | { type: 'pageBreak' };

Agents edit documents through structured block ops via patch_panel/fill_panel (insert/replace/move/remove block), never by emitting markup. Export is a host action (export_document({ panelId, format: 'pdf'|'docx' })) producing the file from the block model; it has its own capability class and is deterministic on a seed (golden-tested).

Canvas authoring policy (canvasPolicy, merges through the standard config layers):


type CanvasPolicyPreset = 'guarded' | 'open';
interface CanvasPolicy {
  preset: CanvasPolicyPreset;              // default 'guarded'
  hitlOnCompose?: boolean;                 // guarded: true
  autoPin?: boolean;                       // open: true
  region?: 'frame' | 'bounded' | 'unbounded';
  allowDelete?: boolean;                   // agent may delete its OWN content only
  toolset?: 'draw' | 'authoring-full';
}

guarded (default) keeps HITL at the mutation line, agent content ephemeral-until-pinned, provenance badges, and the agent confined to a region. open lets the agent freely create, arrange, and modify its OWN content without per-action HITL, auto-pins output, unlocks authoring-full, and widens the region. Individual gates override the preset, so policy is a spectrum.

Host defaults: the framework default is guarded and stays guarded. landi-canvas-studio ships open for its workspace surfaces as host configuration. open may be exposed to end-user-facing (non-operator) sessions when a host explicitly configures it; it is never implicit. moss and sandals launch guarded. Every open surface keeps the persistent open-canvas indicator and the hard boundary below.

The hard boundary no policy relaxes: no untrusted code executes in the trusted app context. Inside the open canvas, documents are a structured model the framework renders and sanitizes, images are assets, and wireframes are closed-schema shapes; agent-authored HTML/JS/CSS is never rendered inline. A prototyping agent that needs runnable code uses the sandboxed code-preview tier (section 14), never an inline document or shape. Actions touching host data or destructive run_panel_action gate on HITL even under open. An open workspace shows a persistent chrome indicator so users know agent output is auto-applied.

Canvas-wide agent surface

Optional, off by default. When the host enables it, one operator agent gets full visibility (whole-document read_canvas + full digest) and broad access (authoring under the active canvasPolicy, section 12), presented in a first-class chat surface. It is the productized form of the section 6 canvas-native action class.

Sandboxed code preview

For prototyping and brainstorming agents whose job is to produce runnable UI (a page, a component, an interactive mockup). Opt-in, gated by canvasPolicy.allowCodePreview (off by default). The tier and its isolation are specced in section 20; the agent-layer surface is one tool.

Data Boundary & Backends

DataAdapter Semantics

The DataAdapter isolates the framework from the database:


interface DataAdapter {
  query(ref: SourceRef, scope: PanelScope, signal: AbortSignal): Promise<unknown>;
  mutate(action: DeclaredAction, payload: unknown, scope: PanelScope): Promise<MutationResult>;
  subscribe?(ref: SourceRef, scope: PanelScope, onChange: () => void): Unsubscribe;
}

Security Practices