Observability and traces

Every run produces logs, execution history, structured traces, latency breakdowns, and optional SDK spans, so you can debug gateway traffic, cron jobs, pipelines, and async jobs in one place.

Every invocation—a manual run, an API Gateway request, a job, a pipeline step, and so on—produces a trace: an ordered timeline of orchestrator work, optional custom spans from your code, structured log lines, and timing, so you can debug one run end to end.

A trace is scoped to a single run ID. Traces complement the metrics on the Functions page by answering what happened inside a specific execution, not only whether it succeeded. Observability is enabled per deployment via ENABLE_OBSERVABILITY; when it is off, runs still execute but no trace is recorded.

Where to read traces

Use whichever view matches how you are working:

• Executions — select any workspace run from the list, then open it to see the full timeline, status, duration, and payload or error details when the platform captured them.
• Function editor — the right-hand test panel lists recent runs for that function; open one to jump to the same execution detail view.
• A trace always includes the built-in orchestrator steps listed below. When your code uses the observability SDK, those spans and observe.log lines appear interleaved on the same timeline.

What the timeline shows

Steps appear in order with per-step durations. Parent spans wrap nested work, so you can distinguish cold-start time, handler time, and nested LLM calls at a glance.

Failed runs keep the spans emitted up to the point of failure, and the failing step records an error message when one is available. Successful runs show outputs where the runtime recorded them—all three runtimes capture span output: Node.js automatically (opt out with captureOutput: false), Python via set_output on a span context, and Go via the value passed to End.

Automatic orchestrator steps

Before your handler runs, the platform records infrastructure-level steps for every invocation:

• Build Environment — assembles environment variables and runtime configuration.
• Setup Layers — resolves and mounts the attached layers.
• Create Container — creates the Docker container (cold starts only).
• Acquire Container — selects a warm container (hot starts).
• Execute Function — runs your handler.

Console output

When observability is active for a run, Node.js console output is also forwarded into the trace as log lines, in addition to your process stdout stream: console.log and console.info map to level INFO, console.warn to WARN, console.error to ERROR, and console.debug to DEBUG. Prefer observe.log when you want to set the level explicitly plus attach structured metadata objects.

Wire protocol (__OBSERVABILITY__)

Custom spans and structured logs leave the container on stderr, one UTF-8 line per event. Each line must begin with the prefix __OBSERVABILITY__: (colon included), followed immediately by a single JSON object. The orchestrator demultiplexes Docker stderr and parses these lines. Function return payloads use stdout, so keep observability traffic on stderr only.

Use camelCase property names exactly as shown in the table below. The control plane matches each span_end to the correct open step using runId + spanId; duplicate spanId values within one run corrupt the timeline.

Nesting: every span_start opens a step, and parent/child linkage is explicit—an event carries a parentSpanId that the control plane matches to the parent's spanId. The Node.js and Python SDKs emit parentSpanId automatically; a runtime that omits it (the Go SDK) falls back to a legacy last-opened-span rule. Emit a span_end for every started span—the built-in SDKs do this on both success and failure.

Step input snapshot: on ingest, if input is present on span_start it is stored; otherwise metadata (emitted by both the Node.js and Python SDKs) is used when input is empty. Omit both for purely named spans. Captured step input and output are truncated at 32 KiB (the host and Node.js SDK) or 8 KiB (the Python SDK), with an oversize marker in place of the value.

If you build a custom runtime, you can emit the same JSON objects yourself. Invalid JSON after the prefix is ignored, and partial lines should be completed on the next write so that newline framing stays one object per line.

JSON event shapes (user events)

How spanType is stored

The orchestrator maps your spanType string to a persisted step type. activity → activity; generation, llm, ai → generation (always kept in the trace even when very fast); tool → tool; retrieval, database, db → activity; other known names (orchestrator, wait, timer, …) map directly. Unknown names default to a generic function step.

SDK: user-defined spans and logs

Add named spans around meaningful work so the timeline reflects your code structure. The SDK ships inside each runtime image, so you do not add it to package.json, requirements.txt, or your Go modules.

On Node.js, observe.span(name, fn, options) invokes fn with no arguments and captures its return value as the span output by default—pass captureOutput: false in options to opt out. For imperative control (multiple exit paths, partial results), use observe.startSpan(name, options) and call .end({ output, error }) when the work finishes.

On Python, use with observe.span(...) as a context manager. The yielded object supports set_output(...) for optional structured output on span end.

The Go package exposes lower-level StartSpan / End and Log; the value you pass to End is recorded as the span output. Shape your own helpers if you need nested spans similar to Node or Python.

Span types (semantic hints)

Set type (Node) or span_type (Python) so UIs and future tooling can classify spans. All types share the same capture pipeline.

Node.js

// Node.js — global observe object injected by the runtime
// observe.span(name, fn, options) calls fn with no arguments and captures its return
// value as the span output by default (pass captureOutput: false to opt out).
exports.handler = async (event, context) => {
  return await observe.span(
    'Fetch data',
    async () => {
      const data = await fetchSomething();

      const result = await observe.span(
        'Call LLM',
        async () => callLLM(data),
        { type: 'generation', input: { model: 'gpt-4o' } },
      );

      observe.log('INFO', 'Pipeline complete', { records: data.length });
      return { records: data.length, result };
    },
    { type: 'activity', captureOutput: true },
  );
};

Python

# Python — import from the runtime SDK
from observability import observe

async def handler(event, context):
    with observe.span('Fetch data', span_type='activity') as s:
        data = await fetch_something()
        s.set_output({'records': len(data)})

        with observe.span('Call LLM', span_type='generation'):
            result = await call_llm(data)

    observe.log('INFO', 'Pipeline complete', {'records': len(data)})
    return result

Go

// Go — import the runtime observability package
import obs "lambda/runtime/go122/observability"

func Handler(event, ctx map[string]interface{}) (interface{}, error) {
    span := obs.StartSpan("Fetch data")
    data, err := fetchSomething()
    if err != nil {
        span.End(nil, err.Error())
        return nil, err
    }
    span.End(map[string]interface{}{"records": len(data)}, "")

    obs.Log("INFO", "Pipeline complete", map[string]interface{}{"records": len(data)})
    return data, nil
}

Distributed tracing (<code>traceparent</code>)

When a client sends the W3C traceparent header on sync invoke, streaming invoke, or API Gateway requests, the platform can link the run’s root span to that remote trace (when distributed tracing is enabled for the deployment). The raw header is not stored; persisted run metadata includes upstreamTraceId and upstreamParentSpanId for correlation in the UI.

To link from your own service, propagate the same traceparent value you use downstream into these entry points. For async jobs you can also set context.__traceparent (or context.traceparent) on invoke-async; the HTTP header is copied into context.__traceparent when the body does not already specify a link. Invalid or missing values are ignored.

LLM fields on traces (control plane)

Integrations that call ObservabilityService on the host can attach generation metadata to steps: model name, provider, token counts, estimated cost, and optional parameters. Use step type generation for LLM-style work — those steps are always kept in the trace even when very short. Any step can set retainInTrace: true to opt out of the minimum-duration filter for tiny spans.

// Control plane (TypeScript) — ObservabilityService
const step = observability.addStep(runId, {
  name: 'OpenAI chat',
  type: 'generation',
  input: { messages },
  generation: { model: 'gpt-4o', provider: 'openai' },
});

observability.completeStep(runId, step.id, completion, undefined, durationMs, {
  generation: {
    inputTokens: usage.prompt_tokens,
    outputTokens: usage.completion_tokens,
    totalTokens: usage.total_tokens,
    costUsd: 0.0012,
  },
});

// Keep a very fast step in the trace (optional):
observability.addStep(runId, {
  name: 'Fast cache hit',
  type: 'activity',
  retainInTrace: true,
});

At step completion, pass a sixth argument to completeStep — { generation: { … } } — to merge token usage and cost after the model returns. Values go through the same redaction pipeline as other trace payloads.

Executions detail

On the trace tab for a run:

• Compare — shows side-by-side input and output JSON for a selected step.
• Generation — when a step carries generation metadata, a summary row shows the model, provider, token counts, and cost.
• Search — filters the step tree by step name and by generation fields.
• Upstream trace — when a traceparent header was accepted, a chip shows the linked upstream trace ID.

Structured logs

observe.log(level, message, metadata) writes a structured entry to the trace, not only to stdout. Levels are case-insensitive strings such as INFO, WARN, ERROR, and DEBUG.

Traces are retained for 30 days by default.

Retries & dead-letter

Async invocations run on a durable Postgres-backed queue. A job can set maxAttempts, and failed attempts are retried with exponential backoff. Delivery is at-least-once, not exactly-once, so keep handlers idempotent; an idempotency key deduplicates re-submits of the same job.

When a job exhausts its attempts, it moves to the dead-letter (DEAD) state instead of disappearing. From the Executions view you can inspect failed runs, replay a run (one at a time or as a batch), and requeue a dead-lettered job to try it again.