Designing Enterprise Multi-Agent Voice/Text Systems: A Domain-Driven, Tool-Agnostic, Context-Persistent Architecture

A research note on building production-grade, maintainable multi-agent systems for enterprise IT operations support — covering both the business-domain model and the runtime architecture that lets a conversation move between voice and text without losing context.

Table of Contents

• Chapter 1. Introduction
  • 1.1 Problem Statement
  • 1.2 First Misconception: "RAG Agent" vs "API Agent"
  • 1.3 Second Misconception: Session Is Conversation
  • 1.4 Design Goals
  • 1.5 Scope of This Document
• Chapter 2. Core Concepts
  • 2.1 An Agent Is a Bounded Business Domain, Not a Persona
  • 2.2 A Tool Is a Capability, Not an Agent Type
  • 2.3 The Four Runtime Concepts: Session / Conversation / Agent / Model
  • 2.4 Two Independent Transitions: Domain Switch vs Mode Switch
• Chapter 3. Domain Model — Enterprise System Agents
  • 3.1 ASSET_SYSTEM
  • 3.2 ACCOUNTANT_SYSTEM
  • 3.3 HUMAN_RESOURCES_SYSTEM
  • 3.4 CONTRACT_SYSTEM
  • 3.5 Extensibility: Adding a New Domain Agent
• Chapter 4. The Unified Tool Layer
  • 4.1 Tool Category: API Call
  • 4.2 Tool Category: RAG Retrieval
  • 4.3 Tool Category: Web/Search
  • 4.4 The Tool Registry Pattern
  • 4.5 Runtime Tool Selection: Letting the LLM Decide
• Chapter 5. Conversation Architecture
  • 5.1 Why Session Must Not Own the Conversation
  • 5.2 ConversationState: The Portable Context
  • 5.3 ConversationManager: Orchestrating Transitions
  • 5.4 Transition Path A: Domain Switch (In-Place)
  • 5.5 Transition Path B: Mode Switch (Full Runtime Rebuild)
  • 5.6 Where the Conversation Actually Lives
• Chapter 6. System Architecture
  • 6.1 High-Level Component Diagram
  • 6.2 Full Lifecycle: Text → Voice → Domain Switch
  • 6.3 Generalized Tool Invocation Flow
• Chapter 7. Best Practices for Production
  • 7.1 Design Patterns Applied
  • 7.2 Maintainability
  • 7.3 Performance
  • 7.4 Concurrency Safety for Transitions
  • 7.5 Security and Data Governance
  • 7.6 Observability
  • 7.7 Testing Strategy
• Chapter 8. Failure Modes and Guardrails
  • 8.1 Tool-Call Failures
  • 8.2 Domain Lock (Fixed Mode)
  • 8.3 Mode-Switch Failure Modes
  • 8.4 Unbounded Context Growth
• Chapter 9. Roadmap
• References

Chapter 1. Introduction

1.1 Problem Statement

Enterprise IT organizations increasingly want a single conversational entry point in front of several internal systems: asset management, accounting, human resources, contract administration, and more. Two architectural mistakes recur when building this kind of system, and this document exists to correct both of them.

1.2 First Misconception: "RAG Agent" vs "API Agent"

It is tempting to model RAG (retrieval-augmented generation), API calls, and web search as three different kinds of agents. This is incorrect.

RAG, an API call, and a web search are all function calls. They are three implementations of the same interface: given a query, return information the LLM did not already have in its context window. The correct decomposition is:

An agent is defined by its business domain (what it is responsible for), not by which tool implementation it happens to use. Every agent has access to the full toolbox relevant to its domain, and the LLM decides at runtime which tool(s) to call — exactly as with any other function-calling task.

This is developed fully in Chapter 4.

1.3 Second Misconception: Session Is Conversation

A second, more subtle mistake is conflating the runtime transport (the live audio/text connection and the model currently attached to it) with the conversation (the actual sequence of things said, by whom, in what order).

When these two are fused into one object — as they typically are in a naive implementation, where "start a session" and "start a conversation" are the same function call — a consequence follows that is easy to miss until a real product requirement collides with it: the conversation cannot survive a change of transport. A user who starts in text chat and then wants to turn on their microphone is, in a fused design, forced to start an entirely new conversation, because "new transport" and "new conversation" were never actually two different things.

The fix is the same kind of decomposition as Chapter 1.2, applied one level up the stack:

A session is a runtime. A conversation is context. They are not the same object, they do not share a lifecycle, and treating them as one is what makes mid-conversation modality switching look impossible when it is not.

This is developed fully in Chapter 5.

1.4 Design Goals

1.5 Scope of This Document

This is an architecture and best-practices document. Code snippets illustrate the pattern; Chapter 9 lists the concrete implementation steps that follow from adopting it.

Chapter 2. Core Concepts

2.1 An Agent Is a Bounded Business Domain, Not a Persona

Borrowing Domain-Driven Design's bounded context [Evans, 2003], each agent corresponds to one enterprise system or business capability: ASSET_SYSTEM, ACCOUNTANT_SYSTEM, HUMAN_RESOURCES_SYSTEM, CONTRACT_SYSTEM. Each domain owns a system prompt, a set of relevant tools, and a handoff boundary — it can transfer the conversation to another domain, but it never "becomes" a different kind of thing.

2.2 A Tool Is a Capability, Not an Agent Type

A tool is a function_tool: a name, a docstring, a typed signature, an implementation. Tools are organized by category (API / RAG / Search) for engineering reasons only — never as a basis for splitting agents. Full treatment in Chapter 4.

2.3 The Four Runtime Concepts: Session / Conversation / Agent / Model

This separation is deliberate and load-bearing: Conversation is the only concept with a lifetime equal to the room's. Session, Agent, and Model are all disposable and get reconstructed around whatever the Conversation currently holds.

2.4 Two Independent Transitions: Domain Switch vs Mode Switch

Because Session/Agent/Model are coupled to each other (a given Session is built for a given mode, and a given Agent+Model pair is built for both a domain and a mode), there are exactly two kinds of transition a running conversation can undergo, and they have different costs:

Both transitions preserve the Conversation. Only their cost differs — a domain switch is essentially free (an in-place object swap), while a mode switch pays for a full runtime teardown/rebuild. This cost asymmetry is why the two are exposed as separate operations rather than a single generic "transition" call: callers should be able to reason about which one they're invoking.

Chapter 3. Domain Model — Enterprise System Agents

3.1 ASSET_SYSTEM

Responsibility: asset and equipment lifecycle — status, assignment history, approval workflow, location tracking.

Representative tools: API (get_asset_status, get_approval_history), RAG (search_asset_policy — SOPs, depreciation policy, approval matrix).

3.2 ACCOUNTANT_SYSTEM

Responsibility: accounting operations support — invoice status, journal entry lookups, closing-period questions. Support for the accounting system and its users, not general financial advice.

Representative tools: API (get_invoice_status, get_journal_entry), RAG (search_accounting_policy), Search (exchange-rate/regulatory lookups where relevant).

3.3 HUMAN_RESOURCES_SYSTEM

Responsibility: HR system support — leave balance, e-contract status, onboarding workflow, benefits questions.

Representative tools: API (get_leave_balance, get_econtract_status), RAG (search_hr_policy).

3.4 CONTRACT_SYSTEM

Responsibility: insurance contract administration support — contract status, high-face-value/high-premium reporting context, amendment history.

Representative tools: API (get_contract_status, get_amendment_history), RAG (search_contract_procedures).

3.5 Extensibility: Adding a New Domain Agent

1. Define scope and system prompt.
2. Declare which tools apply (many API/RAG tools are reusable across domains via namespaced parameters, e.g. search_knowledge_base(kb=...)).
3. Register the domain. No change is required to the tool layer or the conversation architecture (Chapter 5) — that is the direct payoff of keeping domain, tool, and runtime concerns decoupled.

Chapter 4. The Unified Tool Layer

4.1 Tool Category: API Call

A typed wrapper around an internal/external REST/RPC endpoint. Low latency, strongly typed, owned by the backend team for that system of record.

@function_tool
async def get_asset_status(ctx: RunContext, asset_id: str) -> dict:
    """Look up the current status of an asset by its ID."""
    return await asset_api_client.get(f"/assets/{asset_id}/status")

4.2 Tool Category: RAG Retrieval

Used for unstructured knowledge — policy documents, SOPs, runbooks — where the answer is synthesized from a document corpus [Lewis et al., 2020]. Higher latency than an API call; returns chunks, not a value; needs an ingestion pipeline as a separate operational concern.

@function_tool
async def search_asset_policy(ctx: RunContext, query: str) -> list[str]:
    """Search asset management policy and procedure documents."""
    embedding = await embed(query)
    return await vector_store.search(embedding, namespace="asset_policy", top_k=5)

4.3 Tool Category: Web/Search

Current, public information not owned by any internal system. Highest latency and least predictable of the three; provider-dependent (a realtime model may have a native search tool, while a text-LLM pipeline needs an explicit external search API).

@function_tool
async def search_web(ctx: RunContext, query: str) -> str:
    """Search the public web for current information."""
    return await web_search_provider.search(query)

4.4 The Tool Registry Pattern

Tools are declared centrally and assembled per domain via configuration, never hardcoded per agent class:

@dataclass(frozen=True)
class DomainToolConfig:
    api_tools: tuple[Callable, ...] = ()
    rag_namespaces: tuple[str, ...] = ()
    search_enabled: bool = False
 
ASSET_SYSTEM_TOOLS = DomainToolConfig(
    api_tools=(get_asset_status, get_approval_history),
    rag_namespaces=("asset_policy",),
    search_enabled=False,
)

This mirrors the Strategy pattern [Gamma et al., 1994]: the algorithm (which tool to invoke) is selected at runtime by the LLM; the available strategies (which tools exist for this domain) are configured declaratively per domain. The agent class contains no branching on tool category.

4.5 Runtime Tool Selection: Letting the LLM Decide

All tools are exposed to the LLM identically through function-calling; the model chooses based on name and docstring [OpenAI, 2023]. The engineering discipline that matters is writing precise tool docstrings — the docstring is the interface contract, not a category label in the code.

Chapter 5. Conversation Architecture

5.1 Why Session Must Not Own the Conversation

Restating 1.3 more concretely: an AgentSession in a realtime-voice framework is bound, at construction time, to a specific set of models (a realtime STS model, or an STT+LLM+TTS pipeline). Those parameters are not mutable after construction — switching from a duplex voice model to a text LLM is not a configuration change on an existing session, it is a different session entirely.

If the conversation is modeled as a property of the session object itself (e.g., the session's internal message list), then destroying the session to build a new one for the new mode necessarily destroys the conversation with it. The fix is to make the conversation a first-class object that sessions are built around, not a field inside a session.

5.2 ConversationState: The Portable Context

@dataclass
class ConversationState:
    agent_type: AgentType          # current domain
    session_mode: SessionMode      # current modality
    user_context: dict
    is_fixed_mode: bool
    chat_ctx: Optional[ChatContext]  # the actual message history

ConversationState has no dependency on any live Session, Agent, or Model object. It is a plain data holder — closer to the Memento pattern [Gamma et al., 1994] than to anything runtime-specific: a snapshot of "what has been said and what's currently active" that can be handed to a freshly constructed Agent to make it pick up mid-conversation.

5.3 ConversationManager: Orchestrating Transitions

ConversationManager is the single object, created once per room, that:

1. Holds the current ConversationState.
2. Holds references to the currently live Session and Agent (both disposable).
3. Exposes exactly two transition operations (2.4): switch_domain(target) and switch_mode(target).
4. Guards both operations with a lock, so a rapid double-trigger (e.g. a user double-tapping a mic button) cannot start two rebuilds concurrently and corrupt the state.

class ConversationManager:
    def __init__(self, ctx: JobContext, state: ConversationState):
        self.state = state
        self._session: Optional[AgentSession] = None
        self._agent: Optional[Agent] = None
        self._transition_lock = asyncio.Lock()

5.4 Transition Path A: Domain Switch (In-Place)

No AgentSession is destroyed here. The Model changes (different domain's llm/tts), but the Session's transport (audio or text) is untouched.

5.5 Transition Path B: Mode Switch (Full Runtime Rebuild)

This is strictly more expensive than a domain switch (full teardown and reconnection of the runtime), which is exactly why it is exposed as a distinct, explicit operation rather than folded into the same code path as a domain switch.

5.6 Where the Conversation Actually Lives

A subtlety worth making explicit: in the underlying voice-agent framework used here, the authoritative message history is a property of the Agent instance (a read-only chat_ctx), not of the AgentSession. This matters operationally: ConversationManager must read chat_ctx off the outgoing Agent before tearing down its Session, not off the Session itself — reading from the wrong object silently drops history. This is precisely the kind of framework-specific detail that should be isolated behind ConversationManager's interface (switch_domain/switch_mode) rather than leaked into every call site that wants to trigger a transition.

Chapter 6. System Architecture

6.1 High-Level Component Diagram

6.2 Full Lifecycle: Text → Voice → Domain Switch

6.3 Generalized Tool Invocation Flow

Chapter 7. Best Practices for Production

7.1 Design Patterns Applied

7.2 Maintainability

One file per domain, one file per tool, declarative tool configuration (4.4). The conversation architecture adds one more maintainability property: transitions are centralized. Any future transition type (e.g. a "summarize and restart" operation) is a new method on ConversationManager, not a new code path scattered across the call sites that need to trigger it.

7.3 Performance

• Latency budget per modality, as before.
• Parallel tool calls where independent.
• Mode-switch cost is real and should be measured: a full runtime rebuild (teardown + reconnect + first-token latency of the new model) is on the order of a connection round-trip, not a tool call. Products that expect frequent mode switching should treat this as a UX-visible operation (e.g. a brief "switching..." indicator), not a silent one.
• Domain switches, by contrast, should be near-instant, since no runtime teardown occurs.

7.4 Concurrency Safety for Transitions

Because switch_mode tears down and rebuilds shared runtime state, it must be serialized against itself (and, ideally, against switch_domain, since both mutate ConversationState). An asyncio.Lock around both transition entry points is the minimum bar; a system expecting high transition frequency should additionally make the RPC handler idempotent (a repeated "switch to REALTIME" while already in REALTIME is a no-op, not an error).

7.5 Security and Data Governance

Unchanged from the domain-model discussion: least-privilege API credentials per domain, PII boundaries on tool responses, audit logging of every tool invocation. One addition specific to the conversation architecture: the RPC endpoint that triggers a mode switch should be authenticated as the room's own participant, not exposed as an open/anonymous call, since it can trigger resource-costly runtime rebuilds.

7.6 Observability

Structured logs and metrics per tool invocation as before, plus:

• Log every transition (switch_domain/switch_mode) with before/after state and duration — this is a first-class user-facing event, not incidental debug output.
• A mode-switch failure (e.g. the new Session fails to start) should be distinguishable in metrics from a tool-call failure; they have different operational implications (one drops the user's connection, the other doesn't).

7.7 Testing Strategy

• Tool implementations: unit-testable in isolation, as before.
• ConversationManager: testable independently of any real transport, by faking the Session/Agent factory boundary — assert that chat_ctx is correctly captured from the outgoing Agent and passed to the incoming one, without needing a live voice connection.
• End-to-end: at least one integration test per supported transition path (text → realtime, realtime → text, pipeline → realtime, and a domain switch within each mode) asserting conversation history is intact after the transition.

Chapter 8. Failure Modes and Guardrails

8.1 Tool-Call Failures

Every tool call fails gracefully into a response the LLM can relay honestly ("I couldn't reach the asset system right now") rather than an unhandled exception terminating the turn.

8.2 Domain Lock (Fixed Mode)

A session can be locked to a single domain at creation time. Handoff requests are declined with a clear message; this guardrail is orthogonal to — and enforced independently of — whatever mode the session is currently in.

8.3 Mode-Switch Failure Modes

8.4 Unbounded Context Growth

Because ConversationState.chat_ctx now persists across every rebuild for the life of a room, a very long conversation (many domain and mode switches) accumulates history indefinitely. Long-conversation products should summarize or truncate chat_ctx periodically inside ConversationManager, the same way a single long-running Agent conversation would need to — this is not a new problem introduced by the architecture, but the architecture does make it more visible, since one ConversationState may now outlive several distinct Sessions.

Chapter 9. Roadmap

1. Finalize the domain list and system prompts (ASSET_SYSTEM, ACCOUNTANT_SYSTEM, HUMAN_RESOURCES_SYSTEM, CONTRACT_SYSTEM, and any additional capability domains).
2. Implement the Tool Registry pattern (§4.4) as a shared module.
3. Stand up the RAG ingestion pipeline (source → chunking → embedding → vector store), namespaced per domain, as an independent operational workstream.
4. Wire each domain's DomainToolConfig to its actual backends.
5. Implement ConversationManager/ConversationState (§5) and the client-facing mode-switch RPC contract.
6. Add integration tests for every supported transition path (§7.7) before exposing mode switching in a client UI.
7. Implement observability (§7.6) — transition logging in particular — before go-live.
8. Add context summarization/truncation (§8.4) once conversations in production are observed to run long.
9. Load-test each session mode independently, and separately measure mode-switch latency as a distinct budget from steady-state latency.

References

1. Evans, E. (2003). Domain-Driven Design: Tackling Complexity in the Heart of Software. Addison-Wesley.
2. Gamma, E., Helm, R., Johnson, R., Vlissides, J. (1994). Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley. (Template Method, Strategy, Registry-adjacent patterns, Memento, Mediator.)
3. Lewis, P., Perez, E., Piktus, A., et al. (2020). "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." Advances in Neural Information Processing Systems (NeurIPS).
4. OpenAI (2023). "Function calling and other API updates." https://openai.com/index/function-calling-and-other-api-updates/
5. LiveKit Documentation. "Agent sessions." https://docs.livekit.io/agents/build/sessions/
6. LiveKit Documentation. "Chat context — preserving conversation history across handoffs, summarizing context." https://docs.livekit.io/agents/logic/chat-context/
7. LiveKit Documentation. "Text and transcriptions — text-only sessions, disabling audio input/output." https://docs.livekit.io/agents/multimodality/text/
8. Fowler, M. "Bounded Context." martinfowler.com. https://martinfowler.com/bliki/BoundedContext.html