Observability and traces

Every run produces logs, execution history, structured traces, latency breakdowns, and optional SDK spans so you can debug gateway traffic, cron jobs, pipelines, and async jobs in one place.

Every invocation (manual run, API Gateway request, job, pipeline step, and so on) produces a trace: an ordered timeline of orchestrator work, optional custom spans from your code, structured log lines, and timing so you can debug one run end to end.

Traces are scoped to a single run ID. They complement metrics on the Functions page by answering what happened inside a specific execution — not only whether it succeeded.

Where to read traces

Use whichever view matches how you are working:

• Executions — pick any workspace run from the list, then open it for the full timeline, status, duration, and payload or error details when the platform captured them.
• Function editor — the right-hand test panel includes recent runs for that function; open one to jump to the same execution detail view.
• A trace always includes the built-in orchestrator steps below. When your code uses the observability SDK, those spans and observe.log lines appear interleaved on the same timeline.

What the timeline shows

Steps are shown in order with per-step durations. Parent spans wrap nested work so you can see cold start vs handler time vs nested LLM calls at a glance.

Failed runs keep spans emitted up to the failure; the failing step records an error message when one is available. Successful runs show outputs where the runtime recorded them (for example Node.js captureOutput or Python set_output on a span context).

Automatic orchestrator steps

Before your handler runs, the platform records infrastructure-level steps for every invocation:

• Build Environment — env vars and runtime config assembly
• Setup Layers — layer resolution and mounting
• Create Container — Docker container creation (cold start only)
• Acquire Container — warm container selection (hot start)
• Execute Function — the actual function execution

Console output

When observability is active for a run, console.log, console.warn, and console.error are also forwarded into the trace as log lines (in addition to your process stdout stream). Prefer observe.log when you want a stable level plus structured metadata objects.

Wire protocol (__OBSERVABILITY__)

Custom spans and structured logs leave the container on stderr, one UTF-8 line per event. Each line must begin with the prefix __OBSERVABILITY__: (colon included), followed immediately by a single JSON object. The orchestrator demultiplexes Docker stderr and parses these lines; function return payloads use stdout, so keep observe traffic on stderr only.

Use camelCase property names exactly as in the table below. The control plane matches span_end to the correct open step using runId + spanId; duplicate spanId values in one run will corrupt the timeline.

Nesting: every span_start opens a step. The next span_start emitted before that span’s span_end is treated as a child step in the UI. Emit span_end for every started span—built-in SDKs do this on both success and failure.

Step input snapshot: on ingest, if input is present on span_start it is stored; otherwise metadata (Node.js) is used when input is empty. Omit both for purely named spans.

If you build a custom runtime, you may emit the same JSON objects yourself. Invalid JSON after the prefix is ignored; partial lines should be completed on the next write so newline framing stays one object per line.

JSON event shapes (user events)

How spanType is stored

The orchestrator maps your spanType string to a persisted step type. activity → activity; generation, llm, ai → generation (always kept in the trace even when very fast); tool → tool; retrieval, database, db → activity; other known names (orchestrator, wait, timer, …) map directly. Unknown names default to a generic function step.

SDK: user-defined spans and logs

Add named spans around meaningful work so the timeline reflects your code structure. The SDK ships inside each runtime image — you do not add it to package.json, requirements.txt, or Go modules.

On Node.js, observe.span(name, fn, options) invokes fn with no arguments. To attach a JSON-friendly result to the span end event, return a value from fn and pass captureOutput: true in options. For imperative control (multiple exit paths, partial results), use observe.startSpan(name, options) and call .end({ output, error }) when the work finishes.

On Python, use with observe.span(...) as a context manager. The yielded object supports set_output(...) for optional structured output on span end.

The Go package exposes lower-level StartSpan / End and Log; shape your own helpers if you need nested spans similar to Node or Python.

Span types (semantic hints)

Set type (Node) or span_type (Python) so UIs and future tooling can classify spans. All types share the same capture pipeline.

Node.js

// Node.js — global observe object injected by the runtime
// observe.span(name, fn, options) calls fn with no arguments; use captureOutput or startSpan for outputs.
exports.handler = async (event, context) => {
  return await observe.span(
    'Fetch data',
    async () => {
      const data = await fetchSomething();

      const result = await observe.span(
        'Call LLM',
        async () => callLLM(data),
        { type: 'generation', input: { model: 'gpt-4o' } },
      );

      observe.log('INFO', 'Pipeline complete', { records: data.length });
      return { records: data.length, result };
    },
    { type: 'activity', captureOutput: true },
  );
};

Python

# Python — import from the runtime SDK
from observability import observe

async def handler(event, context):
    with observe.span('Fetch data', span_type='activity') as s:
        data = await fetch_something()
        s.set_output({'records': len(data)})

        with observe.span('Call LLM', span_type='generation'):
            result = await call_llm(data)

    observe.log('INFO', 'Pipeline complete', {'records': len(data)})
    return result

Go

// Go — import the runtime observability package
import obs "lambda/runtime/go122/observability"

func Handler(event, ctx map[string]interface{}) (interface{}, error) {
    span := obs.StartSpan("Fetch data")
    data, err := fetchSomething()
    if err != nil {
        span.End(nil, err.Error())
        return nil, err
    }
    span.End(map[string]interface{}{"records": len(data)}, "")

    obs.Log("INFO", "Pipeline complete", map[string]interface{}{"records": len(data)})
    return data, nil
}

Distributed tracing (<code>traceparent</code>)

When a client sends the W3C traceparent header on sync invoke, streaming invoke, or API Gateway requests, the platform can link the run’s root span to that remote trace (when distributed tracing is enabled for the deployment). The raw header is not stored; persisted run metadata includes upstreamTraceId and upstreamParentSpanId for correlation in the UI.

To link from your own service, propagate the same traceparent value you use downstream into these entry points. For async jobs you can also set context.__traceparent (or context.traceparent) on invoke-async; the HTTP header is copied into context.__traceparent when the body does not already specify a link. Invalid or missing values are ignored.

LLM fields on traces (control plane)

Integrations that call ObservabilityService on the host can attach generation metadata to steps: model name, provider, token counts, estimated cost, and optional parameters. Use step type generation for LLM-style work — those steps are always kept in the trace even when very short. Any step can set retainInTrace: true to opt out of the minimum-duration filter for tiny spans.

// Control plane (TypeScript) — ObservabilityService
const step = observability.addStep(runId, {
  name: 'OpenAI chat',
  type: 'generation',
  input: { messages },
  generation: { model: 'gpt-4o', provider: 'openai' },
});

observability.completeStep(runId, step.id, completion, undefined, durationMs, {
  generation: {
    inputTokens: usage.prompt_tokens,
    outputTokens: usage.completion_tokens,
    totalTokens: usage.total_tokens,
    costUsd: 0.0012,
  },
});

// Keep a very fast step in the trace (optional):
observability.addStep(runId, {
  name: 'Fast cache hit',
  type: 'activity',
  retainInTrace: true,
});

At step completion, pass a sixth argument to completeStep — { generation: { … } } — to merge token usage and cost after the model returns. Values go through the same redaction pipeline as other trace payloads.

Executions detail

On the trace tab for a run:

• Compare — side-by-side input and output JSON for a selected step.
• Generation — when a step carries generation metadata, a summary row shows model, provider, tokens, and cost.
• Search — filters the tree by step name and by generation fields.
• Upstream trace — when traceparent was accepted, a chip shows the linked upstream trace id.

Structured logs

observe.log(level, message, metadata) writes a structured entry to the trace (not only to stdout). Levels are case-insensitive strings such as INFO, WARN, ERROR, and DEBUG.

Traces are retained for 30 days by default.