Hierarchical Agent Architectures: Teams of Teams for Complex Tasks

When a Flat Architecture Breaks Down

A single orchestrator managing five specialists works well for moderate complexity. But what happens when the task is "Produce a comprehensive market analysis report" with sections covering competitive landscape, financial projections, customer sentiment, technology trends, and regulatory environment? Each section requires its own research, analysis, and writing workflow. A single orchestrator managing fifteen specialists for all these subtasks becomes unwieldy.

Hierarchical architectures solve this by organizing agents into teams. A top-level orchestrator delegates major sections to sub-orchestrators, each managing their own team of specialists. This mirrors how large organizations work — the CEO does not manage every individual contributor. They manage VPs, who manage directors, who manage teams.

Two-Level Hierarchy: The Basic Pattern

Here is a practical example — a report-generation system with a top-level orchestrator and two team leads:

from agents import Agent, Runner, function_tool, handoff

# === Market Research Team ===

@function_tool
def search_market_data(query: str) -> str:
    """Search for market data and industry reports."""
    return f"Market data for '{query}': market size $4.2B, growing 12% YoY"

@function_tool
def analyze_competitors(industry: str) -> str:
    """Analyze competitive landscape for an industry."""
    return f"Top 3 competitors in {industry}: CompA (35%), CompB (28%), CompC (15%)"

market_researcher = Agent(
    name="Market Researcher",
    instructions="Research market size, growth, and trends using available tools.",
    tools=[search_market_data],
)

competitive_analyst = Agent(
    name="Competitive Analyst",
    instructions="Analyze the competitive landscape using available tools.",
    tools=[analyze_competitors],
)

market_team_lead = Agent(
    name="Market Research Lead",
    instructions="""You lead the market research team. When given a
    research request:
    1. Delegate market sizing to the Market Researcher
    2. Delegate competitive analysis to the Competitive Analyst
    3. Synthesize their findings into a market overview section

    Return a structured section with headers.""",
    handoffs=[handoff(market_researcher), handoff(competitive_analyst)],
)

# === Financial Analysis Team ===

@function_tool
def fetch_financial_data(company: str) -> str:
    """Fetch financial data for a company."""
    return f"{company}: Revenue $850M, EBITDA margin 22%, YoY growth 18%"

@function_tool
def build_projection(base_revenue: str, growth_rate: str) -> str:
    """Build a financial projection based on inputs."""
    return f"5-year projection: {base_revenue} growing at {growth_rate} = $1.9B by 2031"

financial_researcher = Agent(
    name="Financial Researcher",
    instructions="Gather financial data for companies using the fetch tool.",
    tools=[fetch_financial_data],
)

projection_analyst = Agent(
    name="Projection Analyst",
    instructions="Build financial projections using the projection tool.",
    tools=[build_projection],
)

finance_team_lead = Agent(
    name="Financial Analysis Lead",
    instructions="""You lead the financial analysis team. When given a
    request:
    1. Have Financial Researcher gather relevant financial data
    2. Have Projection Analyst build projections from that data
    3. Synthesize into a financial analysis section

    Return a structured section with headers.""",
    handoffs=[handoff(financial_researcher), handoff(projection_analyst)],
)

# === Top-Level Orchestrator ===

executive_orchestrator = Agent(
    name="Executive Orchestrator",
    instructions="""You produce comprehensive analysis reports. For any
    research request:
    1. Delegate market research to the Market Research Lead
    2. Delegate financial analysis to the Financial Analysis Lead
    3. Combine both sections into a final report with an executive summary

    You write the executive summary yourself after receiving both sections.""",
    handoffs=[handoff(market_team_lead), handoff(finance_team_lead)],
)

result = Runner.run_sync(
    executive_orchestrator,
    "Produce a market analysis for the AI voice agent industry",
)
print(result.final_output)

The executive orchestrator never calls a tool directly. It delegates to team leads, who delegate to specialists, who use tools. Each layer has a clear scope.

Span of Control: How Many Direct Reports?

Just like in human organizations, there is an optimal span of control for orchestrator agents. Research on human management suggests five to seven direct reports is ideal. For AI agents, the constraint is tighter because the model must reason about all its handoff options in a single inference call.

Recommended limits:

• Top-level orchestrator: 3-5 sub-orchestrators
• Sub-orchestrators: 3-5 specialist agents
• Keep total depth to 3 levels maximum

Beyond three levels, the conversation history becomes long enough that context window pressure causes quality degradation in the leaf agents.

Recursive Decomposition

Some tasks have a naturally recursive structure. "Summarize this 200-page document" can be decomposed into "summarize each chapter" which can be decomposed into "summarize each section." You can model this with a self-referential orchestrator pattern:

from agents import Agent, Runner, function_tool, handoff

@function_tool
def get_document_sections(document_id: str) -> str:
    """Get the sections of a document."""
    return "Sections: [Introduction, Methods, Results, Discussion, Conclusion]"

@function_tool
def get_section_text(document_id: str, section: str) -> str:
    """Get the text content of a specific section."""
    return f"Content of {section}: [800 words of detailed text about {section}...]"

section_summarizer = Agent(
    name="Section Summarizer",
    instructions="""You summarize individual document sections. Read the
    section content and produce a 2-3 sentence summary capturing the
    key points.""",
    tools=[get_section_text],
)

document_orchestrator = Agent(
    name="Document Orchestrator",
    instructions="""You coordinate document summarization:
    1. Get the list of sections using get_document_sections
    2. Hand off to Section Summarizer for each section
    3. After all sections are summarized, produce a unified
       executive summary combining all section summaries
    """,
    tools=[get_document_sections],
    handoffs=[handoff(section_summarizer)],
)

The orchestrator decomposes the document into sections and delegates each to the summarizer. This same pattern scales — if sections are too long, you can add a paragraph-level decomposition layer.

Context Flow in Hierarchical Systems

In a hierarchical system, context must flow downward (orchestrator provides task details to specialists) and upward (specialists return results to orchestrators). The SDK handles downward flow through conversation history — each handoff passes the accumulated context. Upward flow happens when a specialist completes its work and the orchestrator reads the result from the conversation.

For complex hierarchies, use RunContext to maintain structured state across all levels:

from dataclasses import dataclass, field

@dataclass
class ReportContext:
    market_section: str = ""
    finance_section: str = ""
    executive_summary: str = ""
    sections_completed: list[str] = field(default_factory=list)

Every agent in the hierarchy shares this context. When the market team lead finishes, it writes to market_section. When the executive orchestrator is ready to write the summary, it reads all sections from the context rather than parsing the conversation history.

Anti-Patterns to Avoid

Over-decomposition. If a task can be done well by two agents, do not split it across six. Hierarchy adds coordination overhead — latency from extra LLM calls and context from handoff messages.

Micromanagement. The orchestrator should delegate outcomes, not steps. "Research the market" is better than "First call search_market_data with query X, then call it again with query Y, then format as a table." Trust the specialist to figure out the steps.

Bypassing the chain of command. If the top-level orchestrator directly calls a leaf agent's tool, you have broken the hierarchy. This creates confusing conversation histories and makes the intermediate orchestrator irrelevant.

FAQ

How do I handle timeouts in deep hierarchies?

Set max_turns at each level of the hierarchy. The top-level Runner might allow 30 turns total, while each sub-orchestrator aims to complete within 10 turns. If a sub-team exceeds its budget, the sub-orchestrator should return a partial result rather than hang indefinitely.

Can different levels of the hierarchy use different models?

Yes, and they should. Top-level orchestrators need strong reasoning to decompose tasks correctly — use GPT-4o. Leaf agents performing focused tasks can often use GPT-4o-mini. This optimizes both cost and latency.

Is a hierarchical system always better than a flat one?

No. Flat systems are faster (fewer handoffs), cheaper (fewer LLM calls), and easier to debug. Use hierarchy only when the task genuinely requires multiple distinct teams with different tool sets. If a flat orchestrator with five specialists handles your use case, keep it flat.

---

#HierarchicalArchitecture #MultiAgentSystems #RecursiveDecomposition #OpenAIAgentsSDK #AgentOrchestration #AgenticAI #LearnAI #AIEngineering