- Meta’s AI, CICERO, dominated an online Diplomacy league, winning 20 out of 24 games.
- Despite its success, CICERO’s victories were driven by strategic acumen rather than communication skills.
- Researchers from USC Viterbi and other institutions analyzed 27,000 messages over 200 hours of gameplay.
- CICERO’s in-game communication often needed coherence and reflected its actual strategies.
- When communication abilities were restricted, CICERO’s performance remained unaffected, highlighting strategy as its core strength.
- Human players were more skilled in deception and persuasion, especially when they knew they were interacting with an AI.
- Findings suggest the potential for developing tools to counter AI-generated misinformation but also highlight AI’s current limitations in human-like negotiation.
Main AI News:
In 2022, Meta’s AI, CICERO, made waves by dominating human players in an online Diplomacy league, winning 20 out of 24 games. Diplomacy, a strategy game rooted in World War I, hinges on negotiation and trust, making CICERO’s success seem like a breakthrough in AI communication. However, a recent study reveals that AI’s victories are more about strategic acumen than conversational skills.
Researchers from USC Viterbi and teams from the University of Maryland, Princeton, and the University of Sydney examined CICERO’s performance in depth. Their findings, presented at the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), suggest that while CICERO excels in-game strategy, its communication falls short compared to humans.
Over 200 hours and 24 games, the researchers analyzed 27,000 in-game messages, using Abstract Meaning Representation (AMR) to determine if CICERO’s promises matched its actions. Despite its victories, CICERO’s messages were often incoherent and didn’t align with its gameplay, showing that its success wasn’t due to persuasive communication.
When the researchers limited CICERO’s ability to communicate, its performance barely changed, underscoring that strategy, not negotiation, was its strength. In contrast, human players were better at deception and persuasion, even lying more to CICERO once they recognized it as AI.
These findings highlight AI’s current limitations in mimicking human-like communication and suggest potential applications for developing tools to combat AI-generated misinformation. While CICERO’s strategic prowess in Diplomacy is notable, it reveals that true mastery of human interaction remains a challenge for AI, pointing to future areas of research and development.
Conclusion:Â The success of CICERO in the game of Diplomacy underscores the current state of AI as a powerful tool for strategic tasks rather than nuanced human communication. For the market, this indicates that while AI can excel in data-driven and strategic applications, it still struggles with tasks requiring sophisticated interpersonal interactions. Companies developing AI should focus on enhancing communication capabilities to unlock broader applications, particularly in customer service, negotiations, and other areas requiring high levels of human-like interaction. This factor also presents an opportunity for developing new solutions that address the limitations of AI in real-time, nuanced communication.
