LangSmith - calado

Overview

If your agent is already traced with LangSmith, calado can consume the runs LangSmith is already collecting. You do not re-instrument your app and you do not add the calado SDK. A LangSmith Automation posts matching runs to calado, and the format: "langsmith" adapter regroups the flat run list by trace_id, rebuilds the run tree, and produces the same canonical conversation and step events as the direct SDK. Use this page if your agent is traced with LangSmith. If you run LangChain or LangGraph but do not use LangSmith, the LangChain callback handler is the path for you. For provider clients with no tracer, see the Node.js SDK or Direct API. There are two ways in:

Automation webhook

A LangSmith Automation rule posts matching runs to calado in real time. Zero code in your runtime. Recommended.

Forwarder script

A small script polls LangSmith’s run-query API on a cursor and POSTs each batch. Use it for backfill or where Automations are not an option.

Both paths hit POST /api/ingest with the same payload shape. Pick one.

Automation setup

LangSmith Automations do not fire on every run by default. Without a rule selecting the runs you care about, the webhook stays silent. This is the step customers most often miss.

Get your calado API key

In the calado dashboard, open your agent’s Settings page and create an API key. It is prefixed with cl_. You paste it into LangSmith in the next step.

Create the Automation rule

In LangSmith, open the project you want to mirror and go to Automations → + New Rule.Set a filter for the runs to forward:

Run type — chain is the typical root for agent invocations. Add llm and tool if you also want sub-runs delivered. calado rebuilds the tree either way, as long as runs share a trace_id.
Project — restrict to one or more projects.
Tags — narrow to a subset, for example production or agent.
Sampling — leave at 100% for full coverage; reduce it for high-volume projects.

Set the webhook action

Set the rule’s action to Webhook:

URL — https://app.calado.ai/api/ingest
Method — POST
Headers — Authorization: Bearer cl_<your_calado_api_key> and Content-Type: application/json
Body template — wrap LangSmith’s run payload in the calado envelope:
```
{
  "format": "langsmith",
  "conversations": {{ runs }}
}
```

Send a test trace

Run your agent once. The LangSmith Automation log should show a 202, and the conversation appears in calado within a few seconds.

Verifying it works

In calado, open Agents → (your agent) → Conversations. A forwarded trace shows up as a conversation, with the LangSmith project name, tags, and any attached feedback visible in its metadata. Raw conversations appear immediately. Analysis does not start automatically — click Run Analysis on the agent page when you are ready to classify.

How the flat run list becomes a conversation

LangSmith’s API returns runs as a flat list, not a nested tree. One webhook delivery can carry runs from several traces. The adapter:

Groups runs by trace_id. Each group becomes one conversation.
Finds the root of each group (the run with no parent) and rebuilds the {root, children} tree.
Maps llm and chat_model runs to messages, tool runs to tool calls, and retriever runs to retrievals.
Emits one conversation per trace_id, with each child run as a step.

A run that arrives before its parent (a webhook firing mid-trace) is passed through with a warning; calado links it once the parent arrives in a later delivery. A trace with no LLM activity is stored but marked as skipped for classification.

What gets captured

calado field	LangSmith source
`conversation.externalId` and `sessionId`	`trace_id`
`conversation.messages`	`llm` and `chat_model` run inputs and outputs
Step name and hierarchy	`Run.name`, `id`, `parent_run_id`
System prompt	`serialized.kwargs.messages`
Tool schemas	`extra.invocation_params.tools` and `serialized.kwargs.tools`
Per-step inline definition	A child run’s system-role input message, attached as that step’s `inlineDefinition` so the sub-agent is analyzed against its own prompt
Tool calls	`tool` runs
Retrievals	`retriever` runs
Errors	Run `error`
Latency	Computed from `start_time` and `end_time`
Project name	`metadata.langsmithProject`
User feedback	`metadata.langsmithFeedback`
Tags	`metadata.tags`

Forwarder script

Use the forwarder for backfills, or where the Automation route is not viable. It polls LangSmith’s /api/v1/runs/query endpoint on a cursor and POSTs each batch to calado.

export LANGSMITH_API_KEY=ls__...
export LANGSMITH_PROJECT="your-project-name"
export CALADO_API_KEY=cl_...
export CALADO_INGEST_URL="https://app.calado.ai/api/ingest"

npx tsx forward.ts

The script writes its cursor to .langsmith-cursor and resumes from there on re-run. Delete the file to backfill from the beginning. The full script is in the examples/langsmith-forwarder/ directory of the calado repo, ready to copy.

Caps

Limit	Value
Request body size	5 MB
Runs per `trace_id`	950
Conversations per request	1,000
Message content	1 MB

The runs cap is applied per trace_id group. If one trace in a multi-trace delivery exceeds it, calado returns a structured 400 too_many_child_runs naming that trace. The practical limit is usually the 5 MB body cap. If you hit either, narrow the Automation filter or split long-running work into shorter sessions.

Security

The webhook setup puts your calado API key into the LangSmith Automations UI, as a header value. LangSmith team members with access to that project’s Automation settings can read it. That is a slightly wider exposure surface than the direct-SDK path, where the key only ever lives in your runtime environment.

Treat the webhook key like any credential stored in a third-party tool. Create a dedicated calado key for the LangSmith Automation so you can rotate just that one, and rotate it whenever someone with LangSmith Automations access leaves your team. calado keys are one key per agent, so a leaked key can only POST to that single agent’s stream.

Webhook-scoped sub-keys, where a leaked webhook key cannot be used for anything else, are on the calado roadmap. Until then, the rotation discipline above is the mitigation.

Troubleshooting

The webhook returns 400 too_many_child_runs. One trace_id group exceeded the per-trace runs cap. Narrow the Automation filter or split the trace into shorter sessions. The webhook returns 401. The bearer token was rejected. Re-paste the key into the Automation header. The usual cause is a stray space or a missing Bearer prefix. No webhook activity at all. The Automation rule has no matching runs, or no rule exists. LangSmith Automations do not fire by default. See Automation setup, step 2.

Next: redacting sensitive data

Strip PII before runs leave your infrastructure.

​Overview

Automation webhook

Forwarder script

​Automation setup

​Verifying it works

​How the flat run list becomes a conversation

​What gets captured

​Forwarder script

​Caps

​Security

​Troubleshooting

Next: redacting sensitive data

Overview

Automation setup

Verifying it works

How the flat run list becomes a conversation

What gets captured

Forwarder script

Caps

Security

Troubleshooting