Researchers at Dakota State University, in partnership with regional insurance carrier Safety Insurance, devised an experimental chatbot called "Axlerod" to assist independent insurance agents. Whether that assistance was substantial is up for some debate.
Based on testing described in a pre-print paper [PDF], "Introducing Axlerod: An LLM-based Chatbot for Assisting Independent Insurance Agents," the chatbot can help auto insurance agents save time with certain information retrieval tasks.
The extent of that help and its value to insurance agents varies depending upon how much it's used and how much time is actually saved, underscoring the challenges businesses face when trying to implement AI systems.
The DSU researchers say that business-facing chatbots make more sense than consumer-facing ones because insurance agents, with their domain experience, are better suited to interpret subtle policy language and to spot hallucinations. And they argue AI technology can be consequential.
"This study highlights the transformative role of artificial intelligence in the insurance industry, demonstrating how conversational agents such as Axlerod can enhance operational efficiency by assisting independent insurance agents with policy retrieval and workflow automation," state authors Adam Bradley, John Hastings, and Khandaker Mamun Ahmed.
"Transformative," in this instance, means saving insurance agents 2.42 seconds on average for search-oriented tasks including finding client policy numbers, determining AutoPay eligibility, identifying covered vehicles, and determining the billing plan.
Axlerod consists of a lightweight wrapper around Google Gemini 2.5 Pro and some tools. It's a system prompt that describes how Gemini should behave. Gemini is linked to Safety Insurance's internal data sources through a middleware layer called LiteLLM that acts as a proxy to translate API requests using OpenAI's standard to Google's API and handle Vertex AI authentication. The chatbot also relies on a microframework for creating agentic applications dubbed "Smoltalk."
"With the help of Axlerod, human agents perform their task significantly faster comparing [sic] with the traditional way," the paper explains. "On average, without the chatbot a human agent takes 7.55 seconds where with chatbot the agent takes 5.13 seconds, and the 2.42 seconds faster search time would play a significant role in serving a large number of customers. With more complex tasks, where a user would need to navigate multiple screens or search for a user by name, Axlerod is notably faster."
Each chatbot inquiry costs on average $0.0075, according to the authors, who note that the cost can be discounted to account for the efficiency improvement of agents.
Whether the return on investment pans out depends on the usage scenario. Assuming an average insurance agent salary of $80,000, that translates to per-second pay of roughly $0.01, given 250 8-hour work days per year. With a savings of 2.42 seconds repeated 20 times per day, that's 48.4 seconds saved in one business day, or 12,100 seconds saved over 250 business days.
With 12,100 seconds priced at $0.01, that's a total annual savings of $121 in labor costs. However, let's consider that the employer isn't exactly going to get to deduct that $121 from the agent's annual pay, so it's $121 worth of extra time that may or may not turn into added productivity. It could just be another 48 seconds of drinking coffee each day.
A Google Gemini Business edition seat starts at $21 per month or $252 per year.
Khandaker Mamun Ahmed, a co-author of the paper and assistant professor at The Beacom College of Computer and Cyber Sciences at Dakota State University, told The Register in an email that AI technology has strong potential to transform insurance company operations.
"As noted in the paper, the US property and casualty market includes 4,100+ carriers with combined revenues exceeding $1T," Ahmed explained. "While we did not report a specific estimate for how many searches an agent might perform per day, agents routinely handle high daily activity volumes (often cited in the range of ~50–200 customer interactions per day). With an AI system built directly on the relevant data sources, the number of information lookups and searches could reasonably be much higher because the friction to retrieve answers is substantially reduced."
Using slightly different figures from The Register's inquiry, Ahmed said the ROI assuming 80 searches daily would be as follows:
Cost per Search: $0.0075
Daily Searches (10x/hr x 8 hrs): 80 searches
Total Daily Cost: 80 × $0.0075 = $0.60
Time Saved per Search: 2.42 seconds
Time Saved per Day: 193.6 seconds
Labor Value ($80k/yr, $0.01/sec): 2.42s × $0.01 = $0.0242 per search
Total Daily Savings: 80 × $0.0242 = $1.936
Net Daily Benefit: $1.936 (Savings) - $0.60 (Cost) = $1.336
Percentage: (Net Benefit/Cost) * 100=(1.336/0.60) *100 = 222% Daily ROI
The break-even point would be $0.0075/$0.0242 per search=0.31 searches, according to Ahmed.
Asked about how the study accounted for failures, Ahmed said, "Our system achieved a high success rate, with an average of 93.18 percent. In addition, Axlerod is designed to proactively request clarifying input when an initial query returns more than five results, enabling users to refine and narrow the search space.
"In practice, when refinement is needed, the interaction requires effort comparable to a standard lookup and is typically resolved through guided clarification rather than a separate correction workflow. Notably, we observed no failures during the testing period."
The Register spoke with Scott Johnson, of independent insurance brokerage Marindependent, to ask about how often he might make policy queries about basic data.
"About 10 or 20 times a day," Johnson said, adding that he's interested in potential uses for AI but said the chatbots he'd seen have limitations. He said that they often can't handle your question and miss common industry terms like requests for policy information or a declaration sheet.
Most independent agencies, he said, already rely on software like EZLynx that automates tasks like fetching insurance carrier data so it can be accessed on demand. That avoids maybe 80 percent of inquiries, he said, adding that when agents do have to reach out for information, questions tend to be complicated ones that chatbots wouldn't be able to answer.
Ahmed said, "Overall, we believe the opportunities outweigh the obstacles, particularly for agent-facing deployments where humans remain in the loop to validate outputs in a high-stakes environment. That said, robust field testing is still essential. Expanded evaluation with practicing agents will be critical to validate real-world utility, assess agent trust, and measure efficiency across both human-in-the-loop and more automated workflows." ®
Source: The register