On-premises AI assistant on self-hosted LLMs

Problem

COMAN Software wanted to give its users AI-powered access to product and company knowledge without violating data protection requirements. The chatbot previously in use, built on HubSpot, was costly to operate.

Solution

I designed and built a Q&A chatbot based on locally operated language models. The entire system runs on-premises: LM Studio and Ollama on an NVIDIA DGX Spark, Qdrant as the vector database, PostgreSQL for the application data. Sensitive data is not passed on to external cloud services. I measured answer quality and latencies through Langfuse traces and improved them iteratively on that basis.

Result

Result: the costly HubSpot chatbot was replaced. Costs have since been predictable and significantly lower, and answer quality is noticeably better, with far fewer hallucinations. The chatbot is today an integral part of the COMAN software.

Technical implementation

• RAG pipeline over product and company knowledge
• Integration with the HubSpot API
• Direct integration into COMAN's own software products
• Containerized deployment via Docker