AI Call Center Evaluations

SITUATION​

Our client, a large financial services firm, operated a call center to support customer inquiries. That call center received nearly 50 million calls each year. As part of regulatory compliance and quality initiatives, a percentage of those calls were evaluated for required language (agent identification and customer validation, for example) and appropriate agent conduct. Due to the time required to manually score calls, only a small percentage of calls were being evaluated. Apex was engaged to help complete an AI-enabled platform to evaluate and score calls, allowing the agent managers to identify problem calls more quickly for evaluation and coaching.​

Our team created a platform to test and prove the ability of a Large Language Model (LLM) to select appropriate data for scoring a call. As such, we needed to be able to change the LLM prompts, the number of questions, and the scoring mechanism for each scored item.  ​

We designed a flexible data structure for forms, questions, scoring algorithms, LLM prompts, and evidence generated by the LLM. We then created a data-driven UI for visualization of call scores by team or representative, review and editing of individual calls, and accuracy feedback from coaches. We were able to complete a functioning platform and demonstrate the ability to process and score calls with varying levels of accuracy. Additionally, we were able to provide a framework for continued prompt testing to improve the accuracy of individual questions.​

RESULT​

Apex was able to deliver a platform that would improve coverage of scored calls by orders of magnitude. The platform was able to process and score ~50,000 calls in a few batches in less than 24 hours of run time. This allowed the client to get better insights into calls by having a much larger sample size. Not only did we deliver what was asked, but we suggested several additional uses of the platform we had designed and developed to enable even greater potential value in the future.​