The Python prototype handled one ticket at a time, cleanly. It read the request, called a tool, waited on the model, returned an answer. Production sent twelve tickets at once, each one fanning out into three tool calls and a retry. The code didn’t change. The execution model did.
It’s what happens when a system built to answer one request at a time meets a workload that doesn’t arrive one at a time. The part of the system that needs to change under that load is narrower than most teams assume going in — not the model, not the whole stack, one specific layer, and increasingly that layer runs on the JVM rather than Python.
From prototype to partial failure
This isn’t a case against Python. Python built the agent ecosystem: LangChain, CrewAI, Pydantic, and the raw API clients that let a team go from idea to working demo in an afternoon. Nothing on the JVM side matches that iteration speed, and nothing in this article argues it should. The question that matters starts later: after the prototype works, when someone has to decide what happens when it takes real, concurrent production load.
Once the prototype meets production, concurrency is often where the first cracks appear — and it isn’t simply a throughput problem. Python 3.14 shipped free-threaded builds as officially supported through PEP 779, no longer experimental, and it materially improves Python’s CPU-parallelism story.
But the problem this article is about is broader than the GIL: preserving state and making safe decisions across concurrent, partially failed operations, which free-threading doesn’t solve on its own. (A related boundary problem — schema drift between a tool’s contract and what a model actually returns — is covered separately in our earlier piece on type safety for AI agents; this article picks up one layer up from that, at task state and dispatch.)
When a timeout becomes a duplicate action
Take a claims-processing agent as a concrete case. In the demo, it reads a document, extracts a claim number, and updates a record. In production, the same agent validates the claim against three other systems, checks for duplicates, retries a flaky downstream call, and has to know the difference between „still working” and „failed and needs a human.” That distinction usually stays invisible until the system accumulates real concurrency and partial failures.
Picture a downstream fraud-check service that starts responding slowly under load, not failing, just slow. The task tracker holds status as an untyped flag that stays "pending" until a callback flips it to "done" or "error." When the callback for the slow call never lands inside the timeout window the retry logic was written against, the tracker does the only thing it knows how to do: it assumes the call never happened and retries it, appending each attempt to the same record without ever reaching a state the code recognizes as failed. Nothing crashes.
The claim just accumulates duplicate fraud-check calls until a human notices the numbers don’t reconcile, as the sequence below shows — three distinct states collapsed into one boolean-shaped assumption. Typing the state model makes the retry decision explicit; durable execution coordinates that decision across failures; idempotency is what stops the repeated call from applying the same effect twice.
Same retry, two outcomes
Typed state makes the retry decision explicit. Idempotency prevents the retry from duplicating the effect.
Illustrative sequence, matching the fraud-check example above.
The layer that owns the outcome
The fraud-check example exposes a broader design question: which component owns the lifecycle of the work? Most „move it to the JVM” arguments skip that question, because they treat „the orchestration layer” as one undifferentiated thing. In practice it splits into two:
Dispatch and execution are different responsibilities
- The dispatch layer: decides which agent or skill handles an incoming request, tracks session-level context, and routes sub-tasks onward.
- The domain-agent execution layer: holds task state through a lifecycle, dispatches tool calls under a typed contract, and has to know, deterministically, when a step failed versus when it’s still running.
The second layer is where typed, concurrent runtimes earn their cost. It’s where a dropped exception, a silently retried call, or an untyped „it probably worked” assumption turns into a wrong answer a customer sees — not a claim that the whole application benefits from moving. Most of it doesn’t need to.
Public task state vs. internal execution state
To define that execution boundary precisely, we need to separate two different meanings of „task.” At the protocol boundary, the work can be represented as an A2A Task: a stateful unit with a task ID and an observable lifecycle. The same boundary could just as easily be exposed through a custom API or a Temporal workflow ID.
Behind that contract, the runtime still needs a separate internal state model for timeouts, retries, and uncertain outcomes — one the client never sees directly. A2A defines what the client can observe; it doesn’t prescribe how execution is implemented.
| Public protocol state (e.g. A2A) | Internal execution state |
|---|---|
| submitted working input-required completed failed | tool running outcome unknown retry scheduled effect confirmed human review |
These are two separate vocabularies, not a mapping — there’s no single internal state that „is” a given A2A state. The diagram below shows why: the architecture that carries a typed task contract from the edge to the domain-agent execution layer, and, inside that layer, the actual internal state machine that makes retry and timeout decisions, independently of whatever the public contract happens to be.
Architecture · where the typed layer sits
Edge stays light. The middle layer holds the state.
Edge
Python / TypeScript adapter
Fast to change. Translates the request into a tool call. No durable task-state ownership.
Typed task contractTask ID · command · statusA2A Task or equivalent
Domain-agent execution layer
Typed task-state service (JVM)
Owns the task lifecycle and makes retry, timeout, and cancellation decisions against a closed state model.
Backing systems
Fraud-check API · database · other agents
Can be slow or fail. Someone upstream has to decide what that means.
↪ inside the domain-agent execution layer · internal execution state (a branch point, not a linear sequence)
In one production system delivered by Scalac, the client architecture used Rust for top-level dispatch and the JVM for domain-agent execution. The simplified view above focuses on the JVM boundary discussed in this article, not on every valid way to divide the stack.
Simplified architecture and state model informed by Scalac's production recruitment-agent system and the DeductiveAI orchestration pattern.
Why the JVM fits
With that boundary defined, the JVM decision becomes more specific: it is not one decision, but three, bundled into „should this run on the JVM.” A protocol defines the lifecycle visible to the client — A2A is one example, not the only one. An execution runtime or workflow engine, such as Akka or Temporal, defines persistence, recovery, retries, and concurrency.
The implementation language and its type system — Scala, Java, Python, or Rust — determine which of those state and contract guarantees get enforced at compile time rather than left to convention. The previous section covered the first decision; this one is about the second and third.
Actors or structured concurrency?
The JVM offers more than one strong execution model for this layer, and they’re not the same thing. Akka provides actor-based state ownership, supervision, and persistence: a workflow or task can be represented by a persistent actor or entity that owns its state and recovery logic.
Plain Java or Scala services can instead reach for Virtual Threads and Structured Concurrency directly, to scope a set of related tool calls as one unit of work without an actor runtime at all. Both routes push more of the domain-agent layer’s guarantees into the compiler and runtime than a dynamically-typed dispatch loop does; which one a team picks is a separate decision from whether to use the JVM in the first place.
Recent JVM concurrency work makes the second route more directly applicable to agent-style fan-out and cancellation. JDK 26 continued the evolution of Structured Concurrency through its sixth preview (JEP 525), building on Virtual Threads, final since JDK 21. It maps well onto the domain-agent layer’s actual shape: an internal execution unit fans out into several tool calls that can be scoped to complete, fail, or time out together — harder to enforce by convention in an unstructured async loop than to express through a runtime built for it.
The next section compares the main architectural paths and then looks at public JVM implementations operating at production scale.
Evidence and alternatives
Durable orchestration is also available without leaving Python, through runtimes like LangGraph, Temporal, or Durable Task. The guarantees are real, though more of the typing lives in framework convention than in the compiler. The table below lines up the two paths on the axes that actually differ; these are architectural families rather than equivalent products, and individual runtimes within each side differ significantly in persistence and workflow semantics.
| Decision axis | Python-centric agent stack | JVM domain-execution service |
|---|---|---|
| Prototype and iteration speed | Very high, rich agent ecosystem and SDKs | Lower, tooling still maturing |
| Durable execution | Strong, e.g. LangGraph state or Temporal-style workflows | Strong, e.g. Akka actors and persistence |
| Typed task state | Possible, enforced mostly by framework convention | Compiler enforced, in Scala or Java |
| Concurrency model | Async and event-loop centric, structured by libraries | Actors, message passing and supervision; JVM services may also use virtual threads and structured concurrency outside the actor layer |
| Fit inside an existing JVM stack | An extra runtime and deployment surface | Native to JVM shops and tooling |
The JVM becomes the stronger fit specifically when a team wants that compiler-enforced state paired with an operational model already aligned with the rest of a JVM-heavy stack.
Three public systems show this pattern at different levels of maturity: a measured operational outcome, a complex multi-agent implementation delivered by a small team, and adoption by a regulated enterprise.
DeductiveAI is the measured outcome: a 10-person team’s autonomous root-cause-analysis system on Akka, coordinating event-based agent workflows with guardrails against runaway token usage — the same failure class as the retry loop above, one layer up. The published numbers: 90% faster root-cause analysis, a 70% reduction in time-to-mitigation, and 60% lower operating cost.
Llaama is the small-team implementation: a biopharma workflow engine running 16 agents across a petabyte of data, built by a two-person team over two months, using the same pattern of durable state and coordinated execution.
Manulife selected Akka as the runtime foundation for its enterprise agentic AI platform, built to support a high volume of business-critical applications. The platform is intended to provide reliable, scalable agent execution while embedding governance, safeguards, human oversight, and Responsible AI practices into the operating model.
Dispatch and domain-agent execution don’t have to share a technology stack, and one client implementation makes that concrete. In a recruitment-domain agent system delivered by Scalac, Rust handled top-level dispatch while an Akka-compatible JVM actor runtime owned domain-agent execution underneath it.
The two runtimes served different responsibilities within the same architecture, not two alternative answers to the same question — illustrating that the dispatch-layer choice and the domain-agent-execution-layer choice are independent decisions.
The full architecture and the reasoning behind it are covered in our production multi-agent architecture case study and its follow-up, Rust as the A2A Orchestrator, for anyone who wants the deeper technical case rather than a summary here.
Migration and cost
None of this argues for a rewrite. Python and TypeScript stay at the edges: the tool and model adapter layers, the parts that benefit most from fast iteration and a deep library ecosystem. The JVM layer takes over the parts that need typed state and exhaustive-match guarantees: task lifecycle tracking, tool-dispatch contracts, retries, and recovery. The real decision is a boundary one: which layer holds the state, not which stack wins outright.
Concretely, that boundary can be exposed as an A2A Task: a task ID that the edge polls or subscribes to as the work progresses. The edge doesn’t need to know how the task got done, only how to read its current state; it never reconstructs that state from individual tool calls.
Behind that boundary, the domain-agent runtime can own persisted commands, recovery, and workload coordination itself — which is why teams on this path often skip standing up a separate task queue for this specific workflow.
A broker such as Kafka or SQS may still be the right choice for cross-service event distribution, replay, or independent consumers. The claim isn’t that the runtime replaces messaging infrastructure outright, only that it can own the execution lifecycle internally.
None of this is free, and pretending otherwise would undercut the argument. Four trade-offs matter most:
| What you gain | What it costs |
|---|---|
| Compiler-enforced task state and exhaustive matching | ADK-Java requires explicit, manual task-state tracking that Python frameworks often abstract away by default |
| Durable execution, coordinated retries and recovery | A second runtime to operate, deploy, and put on-call |
| Structured concurrency and virtual threads | JVM agent tooling is still less mature than Python’s equivalents as of 2026 |
| A typed contract at the edge boundary | Circe and Jackson serialization friction at that same boundary |
These are the honest costs a team accepts for typed guarantees at the layer that needs them. It is a different discipline model, not a free upgrade.
That gap is narrower than it was even a year earlier: the JVM ecosystem now includes credible options such as Spring AI, LangChain4j, and ADK-Java, alongside a few specific, nameable gaps rather than a structurally immature ecosystem.
When the JVM is not worth it yet
There’s a real answer to „when should a team not do this,” and it’s worth stating as three concrete conditions rather than a vague „it depends”:
- Team size. If your agent system is owned by a very small team, asking them to operate a second runtime with its own deployment story and on-call is usually more expensive than accepting some convention-driven risk in Python.
- Concurrency. If the workload isn’t yet fanning out into concurrent multi-step tasks, there’s no fan-out cost to recover the JVM’s guarantees against.
- Reliability SLA. Without a committed uptime or correctness SLA, the engineering time it costs to earn typed guarantees is still more expensive than an occasional missed retry state.
None of these are permanent thresholds. DeductiveAI’s own system started as a small team’s internal tool before it became the guardrail-bearing production system described above. That’s the point at which the typed layer earned its cost back.
The question isn’t whether a Python prototype can survive contact with production — plenty do. It’s which specific part of the workflow a team is still willing to leave outside a typed contract once it’s coordinating tool calls a customer’s incident response depends on. DeductiveAI drew that line at the orchestration layer.
One client implementation delivered by Scalac drew the boundary differently, using separate runtimes for top-level dispatch and domain-agent execution. The JVM earns its place when the domain-agent execution layer becomes the reliability boundary: concurrent, stateful, and expensive to get wrong.
If your Python agent prototype is starting to take that kind of load, our engineering team can help harden the layer that’s under pressure, not rewrite what’s already working.
FAQ
Does moving the orchestration layer to the JVM mean rewriting the whole agent system?
No. The pattern that holds up in practice keeps Python and TypeScript at the edges: the MCP adapter layer and the model calling glue stay exactly where they are. Only the domain agent execution layer, the part that tracks task state, timeouts, and retries, moves to a typed runtime.
Is Akka the only way to build this layer on the JVM?
No. Spring AI, LangChain4j, and Google’s ADK Java are credible alternatives, alongside several younger projects whose workflow capabilities and maturity still vary. Akka is the example used here because DeductiveAI provides a named, metric backed production case. Scalac’s own recruitment agent system used Rust for the top level dispatch layer with an Akka compatible JVM runtime underneath, so the pattern holds beyond one vendor’s tooling.
When should a team not move the orchestration layer to the JVM yet?
Three conditions matter most. A one or two person team doesn’t have the bandwidth to also own a second runtime and its on call surface. A workload that isn’t yet fanning out into concurrent multi step tasks has no fan out cost to recover the JVM’s guarantees against. And without a committed uptime or correctness SLA, the engineering time it costs to earn typed guarantees is still more expensive than an occasional missed retry state.
Does this replace the need for a message broker like Kafka or SQS?
Often, yes, for this specific layer. The domain agent runtime already provides durable state, persisted commands, recovery, and workload coordination behind the typed task boundary, so teams frequently avoid standing up a separate broker just for this purpose. The broker’s job doesn’t disappear, it moves inside a runtime that’s already there.