Building a Custom AI Assistant for Your Business

Cassandra began as a Slack chatbot that could answer general prompts and draw on company knowledge from Notion, such as information about employees and projects. The team wanted to make internal information easier to find and reduce repetitive questions to HR, which ChatGPT alone could not answer.

The team first tested Llama and Mixtral using a laptop running llama.cpp, but responses were slow and requests queued instead of running asynchronously. Hosting a model themselves, locally or in the cloud, also meant taking on hardware, security, and operational work. They chose OpenAI as an external provider, while designing Cassandra so they could swap providers without rebuilding the rest of the system.

Kevin describes language model behavior as unpredictable, so the team tested common scenarios and checked response time and quality rather than expecting exact outputs. They used system prompts to set expectations, including asking Cassandra to admit when she does not know an answer and to be concise. At the time of the conversation, OpenAI usage had cost about $100 over more than six months, alongside roughly $10 for Google Cloud Functions.

Kevin found Slack’s API documentation difficult to navigate, including conflicting guidance on message length. The team built Markdown-to-Slack-block parsing to support longer, better formatted responses and planned to release it as open source. They were also developing a framework and Terraform setup to help others create assistants for Slack or Discord, with plugins for company-specific needs.

The team found that employees were initially cautious about using Cassandra and had questions about security, cost, and whether they could break anything. They said they do not inspect user messages as a matter of practice, and ask users to share message details when debugging. Kevin argues that people need guidance on using AI as a collaborator, while keeping their own judgment and creativity involved.