Case Study: How ApertureData increased Chatbot accuracy by 15% with AIMon

Customer	A leading vector database that combines multimodal data, knowledge graphs, and vector search into a single solution.
Industry	Developer Tools
Primary Adopter	CTO, Software Engineering

Background

ApertureData is a leading vector database company that combines multimodal data, knowledge graphs, and vector search into a single solution.

Putting their customer experience first and foremost, they built an advanced chatbot to serve automated answers on their documentation website. The accuracy and relevance of their AI systems was of the highest importance. To achieve this, Aperture leveraged Retrieval-Augmented Generation (RAG) integrated with large language models (LLMs) to deliver precise and contextually relevant responses.

However, subtle inaccuracies and undetected hallucinations negatively impacted their outputs, potentially compromising their response quality and customer experience.

Challenges

1. The ApertureData Engineering team wanted clear visibility into the exact hallucination rate their customers were experiencing in production.
2. ApertureData wanted to discover problems with their RAG implementation including chunking and suboptimal retrievals.
3. The team also wanted to analyze the tone of the chatbot. They wanted to identify when the chatbot is being overly promotional and not adhere to a genuine tone of voice.

Gautam Saluja Senior Software Engineer

“AIMon provided us with comprehensive visibility into our entire LLM-RAG stack, clearly highlighting accuracy issues that we hadn’t previously identified. Its robust evaluation models enabled us to pinpoint exactly where improvements were needed, significantly enhancing the quality and reliability of our RAG and LLM outputs.”

How AIMon helps

AIMon helped Aperture effectively identify chatbot hallucinations and prioritize critical accuracy issues through its benchmark-leading hallucination evaluation model and the platform’s intuitive dashboard. It also provided valuable insights into optimizing chunk sizing, significantly improving response quality and retrieval relevance.

• Issue Identification and Prioritization:
  • AIMon’s dashboard effectively highlighted key problems, particularly hallucinated chatbot responses, enabling ApertureData to quickly prioritize and address critical queries that needed immediate improvement.
• Chunk Sizing Insights:
  • AIMon enabled ApertureData to realize that their existing chunking strategy was overly optimized (chunks were too small), which negatively impacted response accuracy. This led them to increase chunk sizes, improving retrieval relevance.
• Concrete Metrics and Actionable Feedback:
  • AIMon provided clear metrics (hallucination, conciseness, completeness scores), making it easier for ApertureData to understand chatbot performance issues and create specific action items and tickets to systematically enhance their chatbot.

Results

After deploying AIMon, the company was able to pinpoint problems and efficiently make build improvements to increase the accuracy in AI performance:

✅ApertureData was able to efficiently pinpoint issues, identify poorly answered queries, and prioritize improvements.
✅Identified issues included incorrect chunk sizing of content which were leading to suboptimal retrieval and hallucinations.
✅AIMon helped provide numerical measures (e.g., hallucination and conciseness scores) to prioritize troubleshooting.

Conclusion

By partnering with AIMon, ApertureData effectively identified and addressed critical accuracy challenges in their chatbot, achieving a remarkable 15% improvement in accuracy. AIMon’s precise visibility into hallucination rates, chunk sizing optimization, and actionable performance metrics enabled ApertureData’s engineering team to systematically enhance their chatbot’s relevance and reliability. The result was significantly improved customer experience, higher quality responses, and strengthened trust in their AI-powered documentation solution.

About AIMon AIMon helps its customers rapidly optimize their RAG, LLMs, and Agentic systems for higher accuracy, relevance, and adherence. It offers benchmark-leading, low-latency ML models that help find and correct issues in three different layers: data, retrieval, and output. Their most popular evaluation models are Hallucination detection, Context relevance, Instruction adherence, and Context quality.