Case Study: Using a Large Language Model to power a Customer Relationship Management System

Introduction

In this case study, we explore the use of Large Language Model (LLM) to augment the capabilities of a Customer Relationship Management (CRM) system. We explore how LLMs can be used to understand the CRM system and help users with requests relating to the database. We also explore how the LLM can help with data validation given an external source file into the CRM database. The LLM is also capable of learning from user feedbacks in cases there is something wrong, and improve the performance in subsequent steps. This project is a collaboration between IPGN Solutions and the team from RGU which consists of Mr Truong Dang, Dr Thanh Nguyen and Prof. Eyad Elyan. Mr Truong Dang has extensive first-hand experience with working and implementing AI systems, while Dr Thanh Nguyen and Prof. Eyad Elyan have considerable background in AI.  

Background

IPGN Solutions specialises in developing highly configurable CRM solutions tailored for small and medium businesses. The team boasts extensive experience in energy industry spanning various operational departments such as procurement, logistics, projects, engineering, quality assurance etc. Currently, the company has a highly configurable CRM tailored for small and medium businesses, offering modules for sales, customer service, marketing, and project management. The system also supports extensive customisation and integrates with multiple third-party applications. The company is now exploring ways to integrate AI within their business, recognising the potential but also the significant costs and specialised skills required. Through research, the company was introduced to Professor Eyad, Dr Thanh Nguyen and Mr Truong Dang at Robert Gordon University (RGU). They suggested a collaborative project involving their research team, enabling the company to leverage academic expertise and resources for an effective and valuable integration of AI into their operations. After extensive discussion, the company and the team at RGU agreed upon building an AI-powered CRM system by leveraging the power of Large Language Model (LLM) via an Innovation Voucher.

Implementation

The project between the company and the team at RGU consisted of several stages. In the first stage, we explored the use of LLM for database manipulation with human supervision. More specifically, instead of crafting an SQL command to interact with the database, which requires technical knowledge, an user can use natural language to ask the LLM what they want and the LLM will do it for them. Functionalities such as insert, update or delete are supported, however the user needs to check the results afterwards. The user can also make queries about general information in the database. A separate telemetry database was also created which details the conversations between the user and the LLM as well as the commands performed and whether the task was successful or not. The purpose of this database is to store feedback data which would help the LLM to learn and improve its performance in the future.

The results of the first stage was reviewed by the company, and based on their feedbacks, we explored the advanced features for the AI-powered CRM system in the second stage. Firstly, the company wanted more than just an LLM which could interact with the database by issuing SQL commands – they wanted a whole new AI-based CRM system that smooths out much of the technical details which have been required until now. Normally people who buy the products of the company need to know about the interal workings of the system, which might require extensive training. Therefore the company wanted the AI to support with this so that new users to their products can immediately get the system up and running without much technical know-how. For example, when a customer buys the AI-based CRM system from the company and they want to configure it to their needs, they can ask the AI questions like “Where can I configure notification and reminder preferences?”. The main thing to notice is that it’s not just about the SQL commands, it’s about having the AI understand the whole system, from the database to the code base, graphical interface, etc.

Figure 1 – Our workflow for performing data validation on an external file

In the third stage, we tackle a more difficult challenge, namely to use LLM to help with data validation given an external source for the CRM system. Real-life scenarios present many challenges in integrating database systems with external data sources, since these sources can be very messy due to various reaons, including typographical mistakes, different naming conventions, implicit standards, etc. Given the capabilities of LLMs, they seemed to be the perfect candidate for the job. Our LLM was used to solve three main cases: 1) Mismatched, missing and incorrectly formatted data, where the external file might have a field named “Company name” while this is stored in the CRM system as “business_name”; 2) Implicit conventions, such as address in which there are many components such as street name, postcode, city name, etc. which needs to be parsed, and phone number, where people might type in different formats, etc.; 3) Flat file handling, where the LLM needs to split the data in the external source into different parts and send to different tables in the CRM system. Since LLMs are known to hallucinate, our approach is to decompose these tasks into smaller sub-tasks which allow the LLM to more reliably handle, and in our experiments the LLM generally don’t hallucinate when it is done this way. Another feature is that since there is always possibilities for errors, our system allows users to make a feedback to the LLM if it makes a mistake. In this case, the feedback will be stored in a separate database, and in the future, the LLM will learn from that dataset, thereby allowing it to improve itself.

Outcomes

After the project, we have created an AI-powered CRM system which leverages LLMs, which are the latest technology in AI, in order to handle the CRM system in an intelligent manner. Through a collaborative project between IPGN Solutions and researchers from RGU, the LLM was developed to handle natural language queries for database manipulation, support user interaction with system configuration, and perform complex data validation tasks. The system not only improves accessibility for non-technical users but also incorporates a feedback mechanism for continuous learning and improvement. Overall, the project showcases how LLMs can significantly streamline CRM operations and reduce the technical barrier for end users. The system is being deployed to test on real data from different customers, and an extended Innovation Voucher is being planned to further improve the system.