Multi-Agent Systems (MASs) built on large language models (LLMs) have emerged as a promising framework for tackling complex, multi-step reasoning tasks through coordinated agent interactions. However, the additional complexity of these systems often increases unreliable outputs due to compounding uncertainty across agents, inconsistent reasoning, and overconfident incorrect responses. Existing refinement strategies, such as multi-agent debate, improve performance but rely on heuristic signals that do not robustly capture semantic agreement. This work introduces HASSUM, an extension of the HASHIRU MAS that incorporates semantic density and semantic entropy to quantify uncertainty and guide output refinement and decision-making. Both metrics quantify uncertainty at the level of meaning by analyzing consistency and concentration across multiple sampled responses. These metrics are primarily used by a coordinator (the CEO agent) to dynamically evaluate agent outputs, trigger reprompting, and select higher confidence responses. HASSUM was tested across a diverse suite of benchmarks, including TruthfulQA, StrategyQA, Tau-2 bench, and MMLU. The results demonstrate improvements over the original HASHIRU framework, particularly on open-ended and reasoning-intensive tasks. Ablation studies further highlight the complementary benefits of combining semantic entropy and density compared to using either metric alone. These findings suggest that semantic density and semantic entropy provide an effective signal for improving the reliability of MASs. More broadly, the work presented in this paper demonstrates the value of integrating uncertainty-aware reasoning into agentic systems and provides a foundation for future research on adaptive coordination and robust decision-making in multi-agent systems.

PDF, HTML

Masters Thesis, Department of Computer Sciences, The University of Texas at Austin, Austin, TX, April 2026.

@mastersthesis{knowlton:msthesis26, title={HASSUM: Hierarchical Agent System with Semantic Uncertainty Metrics}, author={John Knowlton}, month={April}, school={Department of Computer Sciences, The University of Texas at Austin}, address={Austin, TX}, url="http://nn.cs.utexas.edu/?knowlton:msthesis26", year={2026} }