- Moveo.AI’s custom LLM outperforms GPT-4-0613 in all CX evaluation dimensions except Markdown.
- Evaluation used a random sample from Moveo’s production data, with both models assessed on multiple CX factors.
- Moveo’s LLM showed better performance in hallucination, repetition, disambiguation, live agent handover, readability, language, and latency.
- GPT-4-0613 had superior Markdown formatting capabilities but performed worse in hallucination, potentially affecting customer experience.
- Moveo’s LLM has a response time of 5 seconds compared to GPT-4’s 18 seconds, enhancing support efficiency and customer satisfaction.
Main AI News:
Moveo.AI has reported that its proprietary LLM, fine-tuned for customer experience (CX), outperforms GPT-4-0613 in every evaluation dimension, with the exception of Markdown formatting. This comprehensive assessment utilized a random sample of hundreds of entries from Moveo’s production data, which neither their LLM nor GPT-4 had previously encountered. The evaluation process involved converting each entry into a prompt that included the user query, conversation history, relevant knowledge from documents, live instructions, and custom directives.
The performance of both Moveo’s LLM and GPT-4 was analyzed across eight critical CX dimensions:
- Hallucination
- Repetition
- Disambiguation
- Live agent handover
- Readability
- Language
- Markdown
- Latency
Each dimension was scored to determine the superior LLM. To facilitate this, Moveo utilized a separate GPT-4 instance as a grader, conducting individual API calls for each sample.
Notably, Moveo’s custom LLM excelled in all areas except Markdown, where GPT-4 demonstrated superior formatting capabilities. The most critical difference was in hallucination, where GPT-4 exhibited poorer performance, potentially impacting customer experience negatively. Incorrect product information provided by GPT-4 could result in customer dissatisfaction, increased support requests, and potential liabilities.
Moveo’s LLM operates with a response time of just 5 seconds, compared to GPT-4’s 18 seconds, enabling it to handle over four inquiries in the same period and significantly boosting support efficiency and customer satisfaction.
Panos Karagiannnis, CEO of Moveo.AI, emphasized, “Vertical-specific LLMs are crucial for enterprises as each customer interaction represents a chance to build trust and loyalty. Our LLM, with its reduced hallucination rates and real-time information integration, surpasses GPT-4, mitigates the risk of customer dissatisfaction and liabilities, and sets a new benchmark in customer experience.”
Conclusion:
Moveo.AI’s custom LLM offers significant advantages over GPT-4-0613 in key customer experience metrics, such as hallucination and response time. This enhanced performance suggests that companies prioritizing customer interaction quality and operational efficiency may benefit from adopting Moveo’s LLM. The ability to reduce errors and increase response speed can lead to improved customer satisfaction and lower support costs, setting a high standard for vertical-specific AI applications in customer service.
