Tracing: Full Visibility

When an LLM workflow fails or produces bad output, you need to understand what happened — what data each step received, what the LLM returned, and where things went wrong. Tracing gives you that visibility. When tracing is enabled, each workflow run produces a trace: a tree of events showing the workflow and every step, evaluator, child workflow, LLM call, and HTTP call with their inputs, outputs, errors, and timing. Traces are written automatically when the worker runs. You don’t need to add any instrumentation to your code — just turn tracing on and tell it where to write.

What Gets Traced

Each traced workflow execution produces a trace tree. The root is the workflow, and children are everything that ran inside it:

Steps and evaluators — input, output (or error), start/end timestamps;
LLM calls - the loaded prompt (name, fileDir, variables, rendered config/messages), the result, raw token usage, and normalized usage/cost attributes. Text calls also include merged sources; image calls do not;
HTTP calls — method, URL, status code. Requests made with @outputai/http appear as http events; workflow sendHttpRequest calls appear as internal activities;
Internal activities — Output-managed activities such as trace destination lookup and workflow HTTP requests;
Child workflows — nested as their own trees with the same structure

This means you can open a trace file and see the full picture: which steps ran, what the LLM received and returned, how many tokens it used, and exactly where a failure happened. For prompts with skills, LLM trace input.prompt contains the loaded source prompt, including config.skills, but not the runtime-generated skill listing or load_skill tool added to the provider request.

Trace files are not a secure storage boundary. They can include workflow inputs and outputs, step inputs and outputs, LLM prompts and results, activity inputs and outputs, URLs, payloads, and opt-in HTTP response bodies. Misconfigured HTTP requests can leak keys, tokens, cookies, signed URLs, or other sensitive data into trace files and Temporal history. Avoid putting secrets directly in workflow inputs, URLs, payloads, or literal headers; use the documented $ENV_VAR_NAME header placeholders for sendHttpRequest request header secrets.

Enabling Tracing

Tracing is off by default. You enable it with environment variables. Local tracing writes JSON files to disk — no extra services needed. Remote tracing uploads to S3 when a run completes (requires Redis to correlate events). You can disable trace generation for specific workflows with options.disableTrace: true — see Workflow options. Local only (recommended to start):

OUTPUT_TRACE_LOCAL_ON=true

Trace files appear at logs/runs/<workflowName>/<timestamp>_<workflowId>.json under your project root. Local + remote:

OUTPUT_TRACE_LOCAL_ON=true
OUTPUT_TRACE_REMOTE_ON=true
OUTPUT_REDIS_URL=redis://localhost:6379
OUTPUT_TRACE_REMOTE_S3_BUCKET=my-traces
OUTPUT_AWS_REGION=us-east-1
OUTPUT_AWS_ACCESS_KEY_ID=...
OUTPUT_AWS_SECRET_ACCESS_KEY=...

All Environment Variables

Variable	Required when	Description
`OUTPUT_TRACE_LOCAL_ON`	Local tracing	Set to `true` to write trace files to disk
`OUTPUT_TRACE_REMOTE_ON`	Remote tracing	Set to `true` to upload traces to S3 on run completion
`OUTPUT_REDIS_URL`	Remote tracing	Redis address for correlating events before S3 upload
`OUTPUT_REDIS_TRACE_TTL`	Remote tracing	TTL in seconds for Redis keys that accumulate trace events before a workflow is uploaded. Default: 7 days (`604800`).
`OUTPUT_TRACE_REMOTE_S3_BUCKET`	Remote tracing	S3 bucket name
`OUTPUT_AWS_REGION`	Remote tracing	AWS region for the bucket
`OUTPUT_AWS_ACCESS_KEY_ID`	Remote tracing	AWS access key
`OUTPUT_AWS_SECRET_ACCESS_KEY`	Remote tracing	AWS secret key
`OUTPUT_TRACE_HOST_PATH`	Docker setups	Host path where trace files are mounted. The worker writes inside the container; this tells the API and CLI what path to report so you can open files on the host
`OUTPUT_TRACE_HTTP_VERBOSE`	HTTP client tracing	Set to `true` or `1` to include request/response headers and bodies for `@outputai/http` trace events. Headers are redacted by header name, but bodies are recorded as-is.

AWS Credential Permissions

The worker uploads each trace to S3 when the run completes, so its credentials (OUTPUT_AWS_ACCESS_KEY_ID / OUTPUT_AWS_SECRET_ACCESS_KEY) must allow write access to the trace bucket:

Action	Resource	Why
`s3:PutObject`	`arn:aws:s3:::<bucket>/*`	Upload the trace file
`s3:AbortMultipartUpload`	`arn:aws:s3:::<bucket>/*`	Clean up an incomplete multipart upload if one fails partway

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [ "s3:PutObject", "s3:AbortMultipartUpload" ],
      "Resource": "arn:aws:s3:::my-traces/*"
    }
  ]
}

The API server reads those same traces back to serve output workflow debug, cost, and dataset generate --download — see API Configuration for the read-side policy. The worker and API can share one key pair or use two identities each scoped to their own actions.

Reading a Trace

Each trace file is a single JSON object — the root workflow node. Every node in the tree has the same shape:

Field	Description
`id`	Unique identifier for this node
`kind`	Event kind. Common values include `workflow`, `step`, `evaluator`, `llm`, `http`, and `internal_step`. Other event kinds, such as `webhook`, can appear for specific workflow helpers.
`name`	Workflow name or activity type
`startedAt` / `endedAt`	Unix timestamps (ms)
`input`	What the node received
`output`	What the node returned (absent on error)
`attributes`	Structured metadata keyed by attribute type, including normalized LLM usage and cost
`error`	Error details with `name`, `message`, `stack` (absent on success)
`children`	Array of child nodes (same shape)

Step with an LLM Call

This is the most common trace shape you’ll see. The step has the LLM call as a child, so you can see the loaded prompt (including interpolation variables), what the LLM returned, and how many tokens it used:

{
  "id": "1",
  "kind": "step",
  "name": "lead_enrichment#generateSummary",
  "startedAt": 1761952643861,
  "endedAt": 1761952649808,
  "input": { "name": "Acme Corp", "industry": "SaaS", "size": 250 },
  "output": "Acme Corp is a B2B SaaS company...",
  "children": [
    {
      "id": "generateText-1761952643867",
      "kind": "llm",
      "name": "generateText",
      "startedAt": 1761952643867,
      "endedAt": 1761952649806,
      "input": {
        "prompt": {
          "name": "generate_summary@v1",
          "fileDir": "/prompts",
          "variables": { "companyName": "Acme Corp", "industry": "SaaS" },
          "config": { "provider": "anthropic", "model": "claude-sonnet-4-20250514" }
        }
      },
      "output": {
        "result": "Acme Corp is a B2B SaaS company...",
        "usage": { "inputTokens": 38, "outputTokens": 204, "totalTokens": 242 },
        "sources": []
      },
      "attributes": {
        "llm:generation:usage": {
          "type": "llm:generation:usage",
          "providerId": "anthropic",
          "modelId": "claude-sonnet-4-20250514",
          "status": "complete",
          "input": 38,
          "output": 204,
          "total": 242,
          "items": [
            { "group": "input", "label": null, "amount": 38 },
            { "group": "output", "label": null, "amount": 204 }
          ]
        },
        "llm:generation:cost": {
          "type": "llm:generation:cost",
          "providerId": "anthropic",
          "modelId": "claude-sonnet-4-20250514",
          "status": "precise",
          "input": 0.000114,
          "output": 0.00306,
          "total": 0.003174,
          "items": [
            { "group": "input", "label": null, "amount": 38, "ppm": 3, "total": 0.000114, "status": "ok" },
            { "group": "output", "label": null, "amount": 204, "ppm": 15, "total": 0.00306, "status": "ok" }
          ]
        },
        "llm:usage": {
          "type": "llm:usage",
          "modelId": "claude-sonnet-4-20250514",
          "usage": [
            { "type": "input", "ppm": 3, "amount": 38, "total": 0.000114 },
            { "type": "output", "ppm": 15, "amount": 204, "total": 0.00306 }
          ],
          "total": 0.003174,
          "tokensUsed": 242
        }
      },
      "children": []
    }
  ]
}

The deprecated llm:usage attribute remains on priced LLM traces for compatibility. New trace readers should use llm:generation:usage and llm:generation:cost, which preserve normalized usage and detailed pricing separately.

Failed Step

When a step fails, it has an error object instead of output. The error includes the name, message, and stack trace:

{
  "id": "1",
  "kind": "step",
  "name": "lead_enrichment#lookupCompany",
  "startedAt": 1766083745644,
  "endedAt": 1766083745645,
  "error": {
    "name": "FatalError",
    "message": "HubSpot API key is invalid",
    "stack": "FatalError: HubSpot API key is invalid\n    at fn (...steps.js:7:15)\n..."
  },
  "children": []
}

The parent workflow node also gets an error when the run fails, so you can see at the root level that something went wrong.

Full Workflow Trace

A complete trace puts it all together — the workflow root with all its step children:

{
  "id": "lead_enrichment-758d51b8-7bc0-40a4-b851-c692dff14910",
  "kind": "workflow",
  "name": "lead_enrichment",
  "startedAt": 1769527009211,
  "endedAt": 1769527009265,
  "input": { "companyDomain": "acme.com" },
  "output": { "company": "Acme Corp", "summary": "Acme Corp is a B2B SaaS company..." },
  "children": [
    {
      "id": "1",
      "kind": "step",
      "name": "lead_enrichment#lookupCompany",
      "startedAt": 1769527009222,
      "endedAt": 1769527009223,
      "input": "acme.com",
      "output": { "name": "Acme Corp", "industry": "SaaS", "size": 250 },
      "children": []
    },
    {
      "id": "2",
      "kind": "step",
      "name": "lead_enrichment#generateSummary",
      "input": { "name": "Acme Corp", "industry": "SaaS", "size": 250 },
      "output": "Acme Corp is a B2B SaaS company...",
      "children": [{ "kind": "llm", "..." : "..." }]
    },
    {
      "id": "3",
      "kind": "evaluator",
      "name": "lead_enrichment#judgeSummaryQuality",
      "input": { "summary": "Acme Corp is a B2B SaaS company...", "companyName": "Acme Corp" },
      "output": { "value": true, "confidence": 0.92, "reasoning": "Meets all criteria" },
      "children": [{ "kind": "llm", "..." : "..." }]
    }
  ]
}

Child Workflows

Child workflows appear as kind: "workflow" children with their own nested tree of steps:

{
  "id": "parent-44710c05-child-081d68e7",
  "kind": "workflow",
  "name": "child_workflow",
  "startedAt": 1766179342625,
  "endedAt": 1766179342636,
  "input": { "companyDomain": "acme.com" },
  "output": { "result": "..." },
  "children": [
    { "kind": "step", "name": "child_workflow#someStep", "..." : "..." }
  ]
}

Continue-as-New

When a workflow calls continueAsNew, the trace records "<<continued_as_new>>" as the output for that run. The new run keeps the same workflow ID but gets a new start time — each run produces its own trace file. To see the full chain, list trace files for that workflow name and match by workflow ID; sort by timestamp for order.

Accessing Traces

You can access traces in three ways:

Directly — Open the JSON files at logs/runs/<workflowName>/
Via API — See Get workflow trace log
Via CLI — See output workflow debug

The CLI never reads S3 itself. output workflow debug, output workflow cost, and output workflow dataset generate --download all fetch traces through the API’s trace-log endpoint, and the API reads remote traces from S3. The API’s AWS credentials therefore need s3:GetObject and s3:ListBucket on the trace bucket — see API Configuration.

Start Here

Workflows

Steps

Prompts

Clients

Costs

Evaluators

Production Readiness

Deployment

Integrate with Your App

Packages

Releases

What Gets Traced

Enabling Tracing

All Environment Variables

AWS Credential Permissions

Reading a Trace

Step with an LLM Call

Failed Step

Full Workflow Trace

Child Workflows

Continue-as-New

Accessing Traces

​What Gets Traced

​Enabling Tracing

​All Environment Variables

​AWS Credential Permissions

​Reading a Trace

​Step with an LLM Call

​Failed Step

​Full Workflow Trace

​Child Workflows

​Continue-as-New

​Accessing Traces

What Gets Traced

Enabling Tracing

All Environment Variables

AWS Credential Permissions

Reading a Trace

Step with an LLM Call

Failed Step

Full Workflow Trace

Child Workflows

Continue-as-New

Accessing Traces