TY - GEN
T1 - Rethinking the Bounds of LLM Reasoning
T2 - 62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024
AU - Wang, Qineng
AU - Wang, Zihao
AU - Su, Ying
AU - Tong, Hanghang
AU - Song, Yangqiu
N1 - The authors of this paper were supported by the NSFC Fund (U20B2053) from the NSFC of China, the RIF (R6020-19 and R6021-20), and the GRF (16211520 and 16205322) from RGC of Hong Kong. We also thank the support from the UGC Research Matching Grants (RMGS20EG01-D, RMGS20CR11, RMGS20CR12, RMGS20EG19, RMGS20EG21, RMGS23CR05, RMGS23EG08).
PY - 2024
Y1 - 2024
N2 - Recent progress in LLMs discussion suggests that multi-agent discussion improves the reasoning abilities of LLMs. In this work, we reevaluate this claim through systematic experiments, where we propose a novel group discussion framework to enrich the set of discussion mechanisms. Interestingly, our results show that a single-agent LLM with strong prompts can achieve almost the same performance as the best existing discussion approach on a wide range of reasoning tasks and backbone LLMs. We observe that the multi-agent discussion performs better than a single agent only when there is no demonstration in the prompt. Further study reveals the common interaction mechanisms of LLMs during the discussion.
AB - Recent progress in LLMs discussion suggests that multi-agent discussion improves the reasoning abilities of LLMs. In this work, we reevaluate this claim through systematic experiments, where we propose a novel group discussion framework to enrich the set of discussion mechanisms. Interestingly, our results show that a single-agent LLM with strong prompts can achieve almost the same performance as the best existing discussion approach on a wide range of reasoning tasks and backbone LLMs. We observe that the multi-agent discussion performs better than a single agent only when there is no demonstration in the prompt. Further study reveals the common interaction mechanisms of LLMs during the discussion.
UR - http://www.scopus.com/inward/record.url?scp=85204489998&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85204489998&partnerID=8YFLogxK
U2 - 10.18653/v1/2024.acl-long.331
DO - 10.18653/v1/2024.acl-long.331
M3 - Conference contribution
AN - SCOPUS:85204489998
T3 - Proceedings of the Annual Meeting of the Association for Computational Linguistics
SP - 6106
EP - 6131
BT - Long Papers
A2 - Ku, Lun-Wei
A2 - Martins, Andre F. T.
A2 - Srikumar, Vivek
PB - Association for Computational Linguistics (ACL)
Y2 - 11 August 2024 through 16 August 2024
ER -