The review queue was the bottleneck
Ondine distributed 2,000 concurrent LLM checks across workers

From a review queue to distributed verification

A returned answer is not yet a completed row

A verification request formats a prompt, calls a model and parses the answer. Across a dataset, calls can time out, responses can fail validation and providers can throttle bursts. A worker may stop after thousands of successful calls. The output still needs to be a table that can be filtered, audited and joined back to the source catalogue.

Async execution overlaps requests; it does not record which rows are safe to retry. A restart also needs completed results, failure state and accumulated cost. Ondine puts those concerns in the pipeline so application code does not have to reconstruct them from logs after each interruption.

A prompt behaves like a typed DataFrame transformation

Each row contained a proposed product pair and its evidence. The prompt produced three output columns: a decision, a confidence value and a reason. Keeping the operation inside the DataFrame made the input-to-output relationship explicit.

from pydantic import BaseModel, Field

class Verification(BaseModel):
    is_equivalent: bool
    confidence: float = Field(ge=0, le=1)
    reason: str

# Conceptual shape: input columns become typed output columns.
# proposed_product | reference_product | evidence
#        ↓ Ondine prompt + Verification schema
# is_equivalent | confidence | reason

Pydantic checks the response against that schema. A missing decision or an out-of-range confidence value fails validation. A valid response can still contain a wrong decision: neither the schema nor a confidence value between zero and one establishes that two products are equivalent.

Analysts can inspect the resulting columns alongside the source evidence, then approve, reject or route a substitution for human review. Prompt construction, provider calls and validation stay within the same data workflow.

One process was not the scaling unit

I partitioned the input work across multiple Ondine instances. Each instance owned a bounded number of asynchronous requests. LiteLLM provided a common model interface, and the requests were served by Azure OpenAI. Results were written back with their row identity so that finishing out of order did not scramble the final table.

Partitioning moved work beyond one Python process while keeping a separate concurrency ceiling on each worker. The reported 2,000 checks describe their combined activity. They do not establish the right worker setting for a different model, quota or request size.

Unbounded concurrency only moves the queue to the provider

HTTP 429 is a signal to reduce request pressure, not add more tasks to the queue. Each instance has a concurrency ceiling. Ondine handles the provider'sRetry-After guidance; its adaptive limiter can lower concurrency after throttling and raise it as requests succeed. These are execution capabilities, not measurements of the settings used in this deployment.

Per-worker limits are only half of the story in a fleet. Ten individually reasonable workers can still exceed one shared provider quota. Ondine includes a Redis-backed distributed rate-limiting option for deployments that need coordination across processes. Workers sharing a quota need the same limiter scope. Redis failure can fall back to local limits, which no longer coordinate the fleet. This option is not evidence that Redis was used in the production topology described here.

A failed worker must not erase completed checks

import os
from uuid import UUID
from ondine import PipelineBuilder
from pydantic import BaseModel, Field

class Verification(BaseModel):
    is_equivalent: bool
    confidence: float = Field(ge=0, le=1)
    reason: str

pipeline = (
    PipelineBuilder.create()
    .from_dataframe(
        df_in,
        input_columns=["proposed_product", "reference_product", "evidence"],
        output_columns=["is_equivalent", "confidence", "reason"],
    )
    .with_prompt(
        "Compare {proposed_product} with {reference_product}. "
        "Use only this evidence: {evidence}"
    )
    .with_llm(
        provider="azure_openai",
        model="<base-model>",
        azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
        azure_deployment=os.environ["AZURE_OPENAI_DEPLOYMENT"],
        use_managed_identity=True,
    )
    .with_structured_output(Verification)
    .with_concurrency(20)  # Per worker, tune to the deployment quota.
    .with_checkpoint_dir(".checkpoints")
    .with_checkpoint_interval(100)
    .with_max_budget(50.0)
    .build()
)

result = pipeline.execute()

# After a failed run, resume with the session UUID it printed.
result = pipeline.execute(resume_from=UUID("<session-uuid>"))

Ondine stores lightweight execution counters in compressed checkpoint files and completed responses in SQLite. SQLite runs in write-ahead logging mode, and each completed call is appended independently. If a process is killed, rows already written remain available. Passing the session UUID back to execute resumes the unfinished work while skipping stored responses. An answer billed by the provider but lost before it reaches the store may still be requested and charged again on retry.

Workers naturally complete out of order: a short response may return while an earlier request is still waiting. The response store retains the original row index, allowing the final output to be reconstructed in dataset order. Failed rows can also be retried separately instead of replaying a successful partition.

Cost is a property of the whole dataset

A cost estimate has to cover the dataset, not just one prompt. Retries and validation failures can add billable calls. Ondine can estimate cost before execution, track reported response cost while running and stop an instance after its configured threshold is crossed.

The limit therefore needs headroom for requests already in flight. The example above sets a per-instance budget; it does not reserve or enforce a shared budget across all workers.

A fleet total can hide a stuck partition

A distributed batch can appear healthy while one partition is repeatedly throttled or one schema failure consumes every retry. The useful view combines fleet throughput with worker detail: completed and pending rows, requests in flight, 429 rate, retry count, response latency, validation failures and accumulated cost.

Labels must remain bounded. Provider, model, worker and status are useful dimensions; product IDs and raw prompts are not. High-cardinality identifiers make metrics expensive and can leak business data. Row-level debugging belongs in the checkpoint or response store, connected to a controlled run identifier.

What is reported, and what remains unmeasured here

This page provides the architectural account and illustrative code, not a benchmark package. It does not publish the run duration, model and quota configuration, or raw traces behind the concurrency figure. Those would be needed to reproduce the result or compare it with another deployment.

The next boundary is between workers

The DataFrame interface gives each check an input row and a place for its result. Checkpointing preserves stored work when a worker stops. Neither mechanism, on its own, decides which worker owns the next partition or how much of a shared quota it may use.

That is where I would keep investing: shared work leasing, coordinated provider limits, a fleet-wide budget and an operational view that traces a failed result to its source row without exposing the prompt in metrics. These are the next design priorities, not additional results claimed for this deployment.