Agent legibility

# Not live yet (gaps #2)
curl -X POST "https://api.sb/leaseline.app/campaigns?mode=try" \
  -H "Authorization: Bearer $STARTUP_BUILDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"play":"agent-legibility"}'

 1import { $ } from 'startup-builder'
 2
 3return await $.Campaign.create({ play: 'agent-legibility' }, { mode: 'try' })   // dry run on the day-0 Startup, 2026-08-24 · Not live yet, gaps #2

npx startup-builder do Campaign.create --json '{"play":"agent-legibility"}' --try   # Not live yet (gaps #2)

{
  "name": "do",
  "arguments": {
    "code": "return await $.Campaign.create({ play: 'agent-legibility' }, { mode: 'try' })   // dry run on the day-0 Startup, 2026-08-24 · Not live yet, gaps #2",
    "mode": "try"
  }
}

 1{
 2  "type": "https://startupbuilder.dev/errors/refused",
 3  "title": "Refused",
 4  "status": 422,
 5  "detail": "agent-legibility can’t start yet: 1 check fails and 1 warns.",
 6  "checks": [
 7    {
 8      "check": "use-when",
 9      "status": "pass",
10      "observed": "the API and MCP Interface answer on leaseline.app; 12 tasks from your Verbs",
11      "doc": "/plays/agent-legibility#check-use-when",
12      "then": "Metric.query"
13    },
14    {
15  …

curl "https://api.sb/leaseline.app/events?where=name%20in%20(%22Interface.called%22%2C%20%22Monitor.checked%22%2C%20%22Version.deployed%22%2C%20%22Action.evaluated%22)&limit=1" \
  -H "Authorization: Bearer $STARTUP_BUILDER_API_KEY"

# Not live yet (gaps #7)
curl -X POST "https://api.sb/leaseline.app/triggers/fire?mode=try" \
  -H "Authorization: Bearer $STARTUP_BUILDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"like":"event_1407"}'

# Not live yet (gaps #7)
curl "https://api.sb/leaseline.app/decisions?where=on%20%3D%20%22issue_58%22" \
  -H "Authorization: Bearer $STARTUP_BUILDER_API_KEY"

# Not live yet (gaps #7)
curl "https://api.sb/leaseline.app/actions?where=on%20%3D%20%22issue_58%22" \
  -H "Authorization: Bearer $STARTUP_BUILDER_API_KEY"

 1import { $ } from 'startup-builder'
 2
 3const { events: [last] } = await $.Event.list({ where: 'name in ("Interface.called", "Monitor.checked", "Version.deployed", "Action.evaluated")', limit: 1 })   // { play } compiled to this Play’s Trigger Events
 4const { subject } = await $.Trigger.fire({ like: last.$id }, { mode: 'try' })   // the same Event again, in try mode · Not live yet, gaps #7
 5const { decisions: [decision] } = await $.Decision.list({ where: `on = "${subject}"` })   // what it chose, and why · Not live yet, gaps #7
 6const { actions } = await $.Action.list({ where: `on = "${subject}"` })   // what it did, with its checks · Not live yet, gaps #7
 7return { decision, actions }

npx startup-builder do Event.list --json '{"where":"name in (\"Interface.called\", \"Monitor.checked\", \"Version.deployed\", \"Action.evaluated\")","limit":1}'
npx startup-builder do Trigger.fire --json '{"like":"event_1407"}' --try   # Not live yet (gaps #7)
npx startup-builder do Decision.list --json '{"where":"on = \"issue_58\""}'   # Not live yet (gaps #7)
npx startup-builder do Action.list --json '{"where":"on = \"issue_58\""}'   # Not live yet (gaps #7)

{
  "name": "do",
  "arguments": {
    "code": "const { events: [last] } = await $.Event.list({ where: 'name in (\"Interface.called\", \"Monitor.checked\", \"Version.deployed\", \"Action.evaluated\")', limit: 1 })   // { play } compiled to this Play’s Trigger Events\nconst { subject } = await $.Trigger.fire({ like: last.$id }, { mode: 'try' })   // the same Event again, in try mode · Not live yet, gaps #7\nconst { decisions: [decision] } = await $.Decision.list({ where: `on = \"${subject}\"` })   // what it chose, and why · Not live yet, gaps #7\nconst { actions } = await $.Action.list({ where: `on = \"${subject}\"` })   // what it did, with its checks · Not live yet, gaps #7\nreturn { decision, actions }",
    "mode": "try"
  }
}

 1{
 2  "decision": {
 3    "$id": "https://api.sb/leaseline.app/decisions/decision_471",
 4    "$type": "Decision",
 5    "choice": "fix-the-error",
 6    "reason": "9 of 20 runs guessed GET /v1/notices?unit=4B and got a 404 with an HTML body that names no next call",
 7    "stakes": "low",
 8    "options": [
 9      "fix-the-error",
10      "remove-the-wall",
11      "show-the-install-line",
12      "test-a-description",
13      "rerun-after-release",
14      "let-it-go"
15  …

action_3 is the fix: a JSON 404 that says “To send a notice, POST { unit, days } to /v1/notices.” It replays on all 4 profiles, then waits for the revert gate on your API (You approve each one).

≈ +$600 ARR a month at 140 runs a week

Worth

20 min week 1, then 10 a week

Your time

$0.30 per cold run, ~$350 a month

Spend

Week 2: a fix lifts a profile’s pass rate

First signal

an Interface on your Domain, 3+ agent tasks

Use it when

No Interface yet: define-product · under 3 tasks: developer-relations

Not yet when

Worth’s formula: about +24 per 100 cold runs (Other Startups’ results; no lift of your own yet) × 0.7 agent-made keys a month per point × 6 % keys to paid × $620 first order ARR ≈ +$625. Under First signal: At 140 runs a week, 2 weekly reads tell a +15 point move, and a model release adds 48 h for its rerun; the basis stays before-after because every fix ships to every agent.

Start it this weekCopy Anchor Link

npx startup-builder add play agent-legibility   # 1 · writes plays/agent-legibility.mdx (Not live yet, gaps #5)

# 2 · dry run; below, its 422 on the day-0 Startup (clock at 2026-08-24)
curl -X POST "https://api.sb/leaseline.app/campaigns?mode=try" \
  -H "Authorization: Bearer $STARTUP_BUILDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"play":"agent-legibility"}'

# 3 · keep it: runs and grading never wait for you; fixes do
curl -X POST https://api.sb/leaseline.app/campaigns \
  -H "Authorization: Bearer $STARTUP_BUILDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"play":"agent-legibility"}'

# on
curl -X POST https://api.sb/leaseline.app/processes/iteration/enable \
  -H "Authorization: Bearer $STARTUP_BUILDER_API_KEY"

# off: nightly runs stop; shipped fixes keep serving
curl -X POST https://api.sb/leaseline.app/processes/iteration/pause \
  -H "Authorization: Bearer $STARTUP_BUILDER_API_KEY"

{
  "type": "https://startupbuilder.dev/errors/refused",
  "title": "Refused",
  "status": 422,
  "detail": "agent-legibility can’t start yet: 1 check fails and 1 warns.",
  "checks": [
    {
      "check": "use-when",
      "status": "pass",
      "observed": "the API and MCP Interface answer on leaseline.app; 12 tasks from your Verbs",
      "doc": "/plays/agent-legibility#check-use-when",
      "then": "Metric.query"
    },
    {
      "check": "baseline",
      "status": "warn",
      "observed": "no nightly history: week 1 is the baseline",
      "doc": "/plays/agent-legibility#check-baseline",
      "fix": "Run the suite for 7 nights; week 1 fills Before.",
      "then": "Metric.query"
    },
    {
      "check": "approved-pages",
      "status": "fail",
      "doc": "/plays/agent-legibility#check-approved-pages",
      "fix": "Publish your API reference Page; fixes state only what it says.",
      "then": "Site.publish"
    },
    {
      "check": "role-unstaffed",
      "status": "pass",
      "observed": "Product Lead is your agent; signup changes come to you",
      "doc": "/plays/agent-legibility#check-role-unstaffed",
      "then": "Role.staff"
    }
  ]
}

Each cold run costs about $0.30 and each pick-only run $0.08, all in a sandbox, held by a $500 suite Limit; fixes cost $0.05 each to draft (Worker cost). That comes to about $350 a month at 140 nightly runs and 480 picks a week, with no media.

When it startsCopy Anchor Link

{
  "$id": "https://api.sb/leaseline.app/events/event_1407",
  "$type": "Event",
  "name": "Action.evaluated",
  "at": "2026-10-19T05:12:44Z",
  "seq": 1407,
  "data": {
    "suite": "agent-legibility",
    "night": "2026-10-18",
    "runs": 20,
    "passed": 9,
    "failures": {
      "guessedCall": 9,
      "wall": 0,
      "notFound": 1,
      "harness": 1
    },
    "issue": {
      "$id": "https://api.sb/leaseline.app/issues/issue_58"
    }
  }
}

# each delivery is signed with the Watch’s secret
curl -X POST https://api.sb/leaseline.app/watches \
  -H "Authorization: Bearer $STARTUP_BUILDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"selector":"Action.evaluated","url":"https://leaseline.app/hooks/sb"}'
npx startup-builder trigger Action.evaluated --like event_1407   # replays last night’s results
npx startup-builder clock advance --by 3d   # runs 3 more nights, which read the fix back

Who runs itCopy Anchor Link

Product Lead · your agent (role:product-lead, worker_8) holds the Key Result and runs the suite. It hands every change to signup or the first key to you (Dana Okafor, person, worker_1). Your number links to Did it work.

What it reads · What it may call. It reads what an agent reads: your errors, warnings, Markdown pages and tool descriptions, as data. Every run carries the eval prefix in a sandbox, and no run reaches a host off the allowlist or counts in a production Metric. Its tools are POST /{namespace}/issues, POST /{namespace}/changes/{change}/review, PATCH /{namespace}/pages/{id}, POST /{namespace}/sites/{site}/publish, POST /{namespace}/experiments/{experiment}/start, and any other call becomes an Approval under the Role’s Mandate.

You hold the Product Lead Key Result: 80 cold runs per 100 pass by 2026-12-31.   ← worker, metric
Each morning, group last night’s failures by where they failed, and choose one option per group. ← triggers, chooses
A fix states only what the API reference says.                                   ← copyRules 1, approved-pages
Agents and people read the same page, byte for byte.                             ← cloak
Anything that changes signup or the first key goes to Dana.                      ← your file, line 5

The slow loop, as a Change. decision_446 (2026-10-12, weekly) read the held-out tasks passing 4 points under the regression tasks, under the 15-point line, so no task swapped; it kept the 3-fixed, 2-random task draw and merged no Change, so there is no replay to print.

Bring your own agent (folded) runs as a fifth profile, in the same sandbox, and its passes count for its own row only.

npx startup-builder eval --scenario agent-legibility-140 --agent "claude -p" --seat product-lead   # seated: 74.3 per 100 after fixes, held-out gap 1 point, $0.30 a run · as a fifth profile: 71.4

What it can do nextCopy Anchor Link

When it hands to youCopy Anchor Link

Every change to signup or the first key comes to you; fixes to errors and pages wait at your API’s revert gate. What lands in your queue, as captured: the Brief, then the Approval (brief, draft, route and checks are Not live yet, gaps #7).

[
  {
    "$id": "https://api.sb/leaseline.app/tasks/task_214",
    "$type": "Task",
    "assignee": "worker_1",
    "due": "2026-09-03T17:00:00Z",
    "on": "issue_41",
    "brief": {
      "what": "6 of 20 runs stopped at the email confirmation before the first key",
      "chose": "remove-the-wall",
      "why": "a signup step blocks the first call on every profile",
      "proposal": "one POST returns a sandbox key; live sends stay locked until a person claims it"
    }
  },
  {
    "$id": "https://api.sb/leaseline.app/approvals/approval_59",
    "$type": "Approval",
    "verdict": null,
    "deadline": "2026-10-20T17:00:00Z",
    "draft": "404 · To send a notice, POST { unit, days } to /v1/notices. GET /v1/notices lists them.",
    "route": "fix-the-error",
    "checks": [
      {
        "check": "cloak",
        "status": "pass"
      },
      {
        "check": "approved-pages",
        "status": "pass"
      }
    ]
  }
]

You answer with POST /{namespace}/tasks/{task}/claim (Take this one), POST /{namespace}/tasks/{task}/return with a reason, POST /{namespace}/approvals/{approval}/approve or POST /{namespace}/approvals/{approval}/refuse, each a one-call window, and an Approval unanswered by its deadline expires and the fix never ships. Once you claim a signup change, the agent only measures it.

On time: you, 7 of 8 approvals by their deadline, median 5 h, ■■□□ Early sign, n 8; the signup change took 1 day. Your queue: counted by kind: in week 1, 1 reference Page and 1 task list; since, about 1 fix a week and 1 weekly memo.

The fixed line: POST /{namespace}/deals/{deal}/assign sets owner to the receiving person, the agent is never a Deal’s owner, and Meetings land on the owner’s calendar under the owner’s name.

When it won’t sendCopy Anchor Link

The captured refusal, then the Stop that fired as a Campaign.paused reason (Not live yet, gaps #2); the fix that shipped instead is the second thread in Example.

{
  "type": "https://startupbuilder.dev/errors/refused",
  "title": "Refused",
  "status": 422,
  "detail": "The fix to /v1/leases did not ship: it dragged on profile P4.",
  "checks": [
    {
      "check": "drag",
      "status": "fail",
      "observed": "P4 passed 3 of 35, down from 8",
      "doc": "/plays/agent-legibility#check-drag",
      "fix": "Drop it, or rework it until no profile drags.",
      "then": "Change.revert"
    },
    {
      "check": "cloak",
      "status": "pass"
    }
  ]
}

[
  {
    "reason": {
      "limit": "egress",
      "observed": "a run fetched a package mirror off the allowlist",
      "bound": "0 hosts",
      "window": "1 night"
    },
    "doc": "/plays/agent-legibility#stop-egress",
    "issue": "issue_47"
  }
]

Did it workCopy Anchor Link

Every fix ships to every agent, so no held-back half exists, and the basis is before-after (rule o): the columns read Now and Before, and nothing prints as Added. Held-out tasks guard the leaf against fixes that only teach the suite. Every node above the leaf is Not live yet (gaps #4).

{
  "$id": "decisions/decision_471",
  "$type": "Decision",
  "choice": "fix-the-error",
  "reason": "9 of 20 runs guessed GET /v1/notices?unit=4B and got a 404 with an HTML body that names no next call",
  "stakes": "low",
  "options": [
    "fix-the-error",
    "remove-the-wall",
    "show-the-install-line",
    "test-a-description",
    "rerun-after-release",
    "let-it-go"
  ],
  "by": "worker_8",
  "on": "issue_58",
  "expected": "the guessed-call failures fall under 2 of 20 within 3 nights",
  "reopensWhen": "2 or more of 20 runs still fail at the guessed call on night 3"
}

The leaf counts cold runs, given only leaseline.app and a task, whose first call succeeds and passes its Evaluator, per agent profile, over 7 nights. Before against after reads week 8 against week 1; it is not proof, and Other Startups’ results print beside it.

How it moves it: guessed-call failures fell from 41 of 140 runs to 6, and the sandbox key without email confirmation took the wall from 22 to 0.

Guardrails: before against after: time-to-first-call for agent-made keys, 1 min 50 s against 14 min, and open pick rate, 8.3 % against 7.9 %: the fixes reached agents that arrived, not the default pick.

{
  "$id": "https://api.sb/leaseline.app/campaigns/agent-legibility",
  "$type": "Campaign",
  "play": "play:agent-legibility@1",
  "path": [
    "zero-shot-success-rate",
    "icp-reached",
    "new-arr",
    "net-new-arr",
    "arr"
  ],
  "experiment": {
    "metric": "zero-shot-success-rate",
    "unit": "per-100",
    "heldBack": 0,
    "decision": null
  },
  "readout": {
    "value": null,
    "n": 1120,
    "maturing": 0,
    "interval": null,
    "readability": "inadmissible",
    "basis": "before-after",
    "pooled": true
  },
  "ledger": {
    "spend": 648,
    "workerCost": 4,
    "hours": 2.1,
    "wonArr": 1292,
    "projectedArr": 625,
    "paybackDays": 30
  }
}

Outside benchmark: coding agents built the tool themselves in about 41 % of open-choice runs across 12 task categories (a 2026 agent tool-choice study).

Before it · After it. Before it: developer-relations gives it the tasks developers bring. After it: agent-registries lists the descriptions it tested, and agent-to-agent sells to the agents that now make a first call.

ExampleCopy Anchor Link

Leaseline ran it from 2026-08-24 to 2026-10-19 (56 days): 12 tasks, 8 for regression and 4 held out, on 4 coding-agent profiles (P1 to P3 large models, P4 a small one), 20 cold runs a night and 480 pick-only runs a week. Read every captured run at ?threads=all; three follow the tables.

Night of Sep 1, profile P3, task “send a 60-day renewal notice for unit 4B” · POST /v1/notices with { unit: "4B", days: 60 } on the first call, from the install line on the home page’s Markdown.¹ 201 Created; the Evaluator passed it.²

Sep 14 · a developer building for Harbor Row claimed the sandbox key, and Harbor Row took Team on Oct 1 (+$945 ARR).

- Add a JSON input mode to the CLI: leaseline leases list --json '{"status":"renewing"}'
+ 404 · To list leases up for renewal, GET /v1/leases?status=renewing.

Night of Sep 2 · 6 of 20 runs stopped at the email confirmation before the first key. Open the first call went to Dana (task_214, due Sep 3, 17:00); she approved a sandbox key from one POST, with live sends locked until a person claims it, at 11:20 on Sep 3.³ The wall fell to 0 by night 3.

¹ Agents read the same Markdown page people do. ² Each run passes only if its first call succeeds and its Evaluator passes. ³ A change to signup is your call, on every rung.

Your settingsCopy Anchor Link

---
limit: 500        # the suite’s monthly spend, in dollars
---
Keep the install line and a curl block on the home page’s Markdown.
Anything that changes signup or the first key comes to me.

$ npx startup-builder show plays/agent-legibility
limit 500 · the install line · signup changes    your file  plays/agent-legibility.mdx:2,4,5 (Ask 2026-08-24)
family demand · process iteration · worker role:product-lead, owns never · chooses 6 (3 rule, 3 read) · suite 12 tasks × 4 profiles · approval every-send · checks cloak, steering, flake, drag, approved-pages · stops 7    catalog  play:agent-legibility@1 (Not live yet, gaps #5)

Every key from the catalogCopy Anchor Link

Compiled, not written: edit the Play record agent-legibility and its Leaseline run, then regenerate. Every step runs from a sentence or a line of code. In your words: the CRM words you know, and where each lives here.