As Large Language Model (LLM)-based Multi-Agent (LLM-MA) systems become increasingly prevalent in automating software engineering tasks, understanding their resource consumption is crucial for practical adoption. A recent study from arXiv provides a detailed analysis of token usage within these systems, offering valuable insights for developers and operations teams.
What Happened
Researchers analyzed the token consumption patterns of an LLM-MA system—specifically, the ChatDev framework using a GPT-5 reasoning model—across 30 software development tasks. They mapped the system's internal phases to standard software development stages: Design, Coding, Code Completion, Code Review, Testing, and Documentation. This allowed for a quantified comparison of token distribution (input, output, and reasoning) across these stages.
The key finding is that the Code Review phase accounts for the majority of token usage, consuming approximately 59.4% of all tokens. Furthermore, input tokens consistently represented the largest share of consumption, averaging 53.9% of the total. The study establishes a methodology for evaluating token usage in agentic software engineering workflows.
Why It Matters
These findings have significant implications for developers and organizations considering adopting LLM-MA systems. The disproportionate token consumption during Code Review suggests that current agent collaboration protocols may be inefficient, leading to higher operational costs. The dominance of input tokens indicates that the agents are spending a considerable amount of effort processing information, rather than generating novel code or outputs. This could be due to factors like verbose communication, redundant checks, or the need for extensive context.
For enterprises, this research enables more accurate cost prediction and workflow optimization. Understanding where tokens are consumed allows for targeted interventions, such as refining agent prompts, streamlining communication protocols, or exploring more efficient LLMs. It also shifts the understanding of cost drivers - away from initial code generation and towards the iterative refinement and verification phases. This insight is critical for budgeting and resource allocation.
What To Watch
This is a preliminary study, and further research is needed to validate these findings across different LLM-MA frameworks, LLMs, and software development tasks. It will be important to see how these token consumption patterns change as LLMs become more sophisticated and agent collaboration protocols evolve. Specifically, it remains to be seen whether techniques like knowledge distillation or more concise communication strategies can significantly reduce token usage without compromising code quality. Researchers should also investigate the trade-offs between token consumption and the effectiveness of the Code Review process. Finally, monitoring the impact of different LLMs (beyond GPT-5) on these patterns is essential.