ADR-0001 · Streaming tool-call extraction: native reassembly, not canonical normalization

Status: Accepted Date: 2026-07

Context

When an LLM streams its response chunk by chunk, MCS must reassemble those chunks into a complete message and then find any tool call in it. Two components cooperate:

• ExtractionStrategy -- the format expert. There is one per tool format (OpenAI Chat Completions, OpenAI Responses, Anthropic Messages, and the codec-based text format; a driver developer may add their own -- Hermes, Ollama, ...).
• LLMStreamBuffer -- a client-created accumulator. The client feeds it raw chunks; it hands itself to process_llm_response.
• ExtractionChain -- an ordered list of strategies.

The promise made to a driver developer is explicit in BaseDriver:

"Tool-call extraction is handled by a chain of ExtractionStrategy instances."

A developer who wants Hermes/Ollama support adds their strategy to that one list. The only reason the chain exists as a shared object is so the buffer (which reassembles) and the driver (which extracts) use the same strategies in the same order -- resolve the format once, use it everywhere.

An intermediate implementation broke this boundary. To let a single extract read every format, the buffer's accumulate normalised every wire (Anthropic, Responses, ...) into the canonical OpenAI message shape ({content, tool_calls}). The consequences:

1. The buffer stopped being a buffer. It no longer holds the message in the shape it arrives in; it translates it. as_dict() of an Anthropic stream returns an OpenAI message, not what stream=false would have returned from Anthropic.
2. The strategy abstraction degenerated. Because the assembled message is always canonical, the driver's chain only ever distinguishes "has tool_calls" (→ the OpenAI reader) from "content-only" (→ text). The Response/Anthropic strategies never claim an assembled message (their recognizes looks for streaming event markers that normalisation has erased). They live only inside the buffer, as accumulate -- half a strategy.
3. Resolution happens twice. The buffer resolves the wire format (to accumulate); the driver resolves again (to extract) -- the second resolve is nearly a no-op because everything is already canonical. The shared-chain rationale is defeated.

A developer adding a Hermes strategy would reasonably expect the driver's recognizes to inspect the message structure and pick their format -- but in the normalised world it never can. The boundaries were not kept.

Decision

The LLMStreamBuffer reassembles into the format's native message shape -- the exact shape the provider's SDK returns for stream=false. It does not normalise.

Responsibilities:

• ExtractionStrategy -- the format expert, end to end, in its own shape:

  • recognizes(shape) -- is this shape (a raw chunk or an assembled message) my format?
  • accumulate(acc, chunk) -- reassemble fragments into my native message shape.
  • is_done(chunk) -- my stream-completion signal.
  • extract(message) -- read the call(s) from my native message (e.g. AnthropicExtractionStrategy.extract reads content:[{type:"tool_use"}]).
  • result_messages(records) -- shape the history in my format.
  • forming_name(message) -- mid-stream name peek (for display hold).

• LLMStreamBuffer -- a pure accumulator, no format knowledge: it resolves the wire strategy once on the first chunk (via recognizes), delegates accumulate, and holds the native accumulator. as_dict() returns the native message; a separate display cursor tracks the content deltas each accumulate returns (so display text is format-agnostic even when content is a block list). It does not need to expose its strategy -- the driver resolves extraction itself (two axes, above).

• ExtractionChain -- the one shared ordered list. The buffer (resolve + accumulate) and the driver (extract) use the same instances in the same order. Adding a format is one entry that serves both. Native (envelope) strategies come before the text strategy.

• BaseDriver -- resolves the extraction axis on the native message. It calls chain.resolve(message) (streaming: message = buf.as_dict(); non-streaming: the str | dict). It does not reuse the buffer's strategy -- see the two axes below.

Two resolve axes -- not a redundant double resolve

Reassembly and extraction are different questions answered on different inputs:

axis	who	input	question
reassembly	buffer	a raw chunk	which wire format streams here? → accumulate
extraction	driver	the assembled message	which format is the call in? → extract

For a native call the two coincide (OpenAI chunk → OpenAI message → OpenAI extract). For a text-embedded call they diverge: the wire is OpenAI (content deltas) but the call sits as JSON in content, so extraction must be the text strategy. If the driver reused the buffer's wire strategy, OpenAICompletion.extract would find nothing and the embedded call would be lost. So the driver resolves extraction on the assembled message via the shared chain -- the same chain instances the buffer resolved reassembly on, but a genuinely different resolve. This is not the redundant double-resolve that alternative A had (where canonical normalisation made the second resolve degenerate -- always "tool_calls-vs-content"). What B removes is the normalisation, not the second resolve; on native shapes the extraction resolve is meaningful (it tells an Anthropic call from an Anthropic leak-in-text).

Two groups of strategies, mapped to the capabilities

The strategies split into two groups, and each maps to a capability:

Group	claims by	belongs to
Native (OpenAI Completions/Responses, Anthropic)	envelope / shape	SupportsNativeTools -- it contributes them to the chain and declares the backup
Text (the codec: JSON, and a developer's Hermes/XML/...)	content	the base (BaseDriver's codec) -- always present

The text strategy has two roles, one instance -- and that is not redundancy:

• Text mode (no native call): the text strategy in the chain is the primary extractor. A pure-text driver needs nothing else.
• Native mode, leak: the same text strategy is the backup. SupportsNativeTools declares an explicit backup (which must be a text strategy); the driver falls through to it when a native strategy claims the shape but extracts no call (see below).

The fallback is explicit, not implicit via ordering: if the driver's native path is not used, the backup does not fire, but the text strategy in the extractors still claims its content; if the native path is used, native wins and the backup catches leaks.

The chain reorders on match (restored in B)

The chain promotes the last strategy that matched to the front -- a stateless optimisation on the assumption that a stream keeps its format. This was removed under alternative A because canonical normalisation made every message look the same, so promoting the text strategy could have jumped it ahead of a native envelope and broken the "native wins" invariant. B makes it safe again: the shapes are distinguishable (the text strategy claims only a string content; native messages carry structured content -- tool_calls, Anthropic blocks -- so the text strategy can never claim a native shape). The reordering is not load-bearing for correctness; if it is dropped the result is identical, only slightly slower.

The text / leak fall-through

A model in native mode occasionally leaks its call as text (in the content channel) instead of the native slot. In native reassembly the assembled message is still a native message (e.g. an Anthropic message whose text block carries the JSON), so the native strategy claims it but extract finds no call. The driver therefore has an explicit rule: a native strategy that claims the shape but extracts nothing → fall through to the backup text strategy on the message's text. This is deliberate and visible, not an accidental by-product of normalisation.

The text lives in the native shape (an OpenAI content string, Anthropic text blocks, Responses output_text items), so each strategy exposes content_text(message) -> str -- format knowledge on the format expert. The driver hands that string to the backup's extract; if the backup finds a call, the backup (a text strategy) also shapes the result history -- because a model that leaked a call as text did not open a native tool_call_id and expects a text-style continuation, not a native tool answer.

Whether the fall-through may fire is a per-strategy flag, leaks_into_text:

• OpenAI Completions -- False. A leak arrives as a content-only message (no tool_calls key), so the strategy does not claim it and the chain routes it straight to text. When it does claim (a tool_calls key is present, even null), that key is authoritative "no call" -- the content is not second-guessed, so no fall-through. (This is the false-positive guard: {content: "<call-looking JSON>", tool_calls: null} must not execute.)
• Anthropic / Responses -- True. Their assembled message always carries the envelope (a content-block list, an output-item list) even for pure text, so the strategy claims every message of its shape; extract decides call-vs-text, and a text-only message falls through to the backup.

Alternatives considered

A -- Canonical intermediate (the rejected implementation). The buffer normalises every wire to the OpenAI shape; the driver's chain needs only an OpenAI reader + text.

• Pro: one extract for all; a leak collapses to a content-only message, so the text strategy catches it for free.
• Con: the buffer leaks the format concern (it translates, not buffers); the driver's strategy abstraction degenerates to tool_calls-vs-content; Response/Anthropic are half-strategies living only in the buffer; the format is resolved twice; a developer's added format never participates in the driver's recognizes. The shared-chain boundary is violated.

Rejected: the coupling and the concern leak outweigh the free leak handling.

Consequences

• The buffer is honest: as_dict() equals the provider's stream=false message. Easy to reason about, easy to test against real SDK payloads.
• Every strategy is a whole format expert (recognise + accumulate + extract + result_messages), each in its own shape. A developer adds Hermes/Ollama as one entry and it works for both reassembly and extraction.
• The format is resolved once (buffer), reused by the driver -- no double work.
• New cost: leak handling needs the explicit native→text fall-through (above). Previously canonical normalisation made it implicit; now it is spelled out.
• Migration required: the wire strategies' accumulate must build native shapes (not canonical tool_calls); their extract/recognizes/content_text must read the native message; the buffer tracks display via content deltas (not by reading content); the driver resolves extraction on the native message and implements the native→text fall-through gated by leaks_into_text. The buffer and driver share the one chain (via the SupportsStreaming factory), resolving different axes on it.

Known edge: interleaved leak-text and a native call

A model could start leaking a call as text (partial JSON accumulating in the content channel) and then, mid-stream, emit a native tool_call. The native strategy claims and extracts the native call, the driver runs it and reset()s the buffer -- discarding the partial leak-text, which was held (never shown) and is now gone; the model then continues the JSON into a fresh buffer. This is accepted as a known limitation: a model both leaking and natively calling in one turn is highly unlikely, and the native call is the authoritative one. Not worth special handling.

Resolved sub-decision -- how the buffer obtains the chain

The buffer stays independent (constructible standalone with any strategy list) -- that is what keeps fan-out working: the client feeds one buffer through a list of drivers, so it cannot be bound to a single driver.

SupportsStreaming offers this as convenience, not coupling:

• extraction_strategies -- exposes the driver's chain strategies.
• new_stream_buffer() -- returns a buffer pre-seeded with that chain.

streamer = DriverMeta.resolve_capability(driver, SupportsStreaming)
buf = streamer.new_stream_buffer()                    # convenience: chain pre-filled
# equivalently, by hand:
buf = LLMStreamBuffer(streamer.extraction_strategies)

The factory only constructs; it does not make the buffer a driver method or bind it. A client could build the chain itself and pass it to both driver and buffer -- the factory just removes that step. (A declarative alternative -- default extractors defined in config, loaded by both -- remains open but is not chosen here.)