A University Chatbot for the University of Massachusetts Lowell that can answer a variety of questions about the university.
100% On-Prem.
- GitHub Actions CI
- Kubernetes
- Longhorn
- Ollama
- Redis on K8s
- AWS SSM Parameter Store
- Streamlit
- LangGraph
- LangChain
- FastMCP
- Grafana
- Prometheus
Available here on the club website.
Divided deployment on Kubernetes and on VM.
Inference model, embedding model, and docling chunker run directly on VM and expose an Ollama API endpoint and docling-serve API endpoint to call.
The orchestrator handles the tool calling loop between the model and the available tools. It operates using LangGraph and LangChain and is interfaced with a v1/chat endpoint running as a simple web server.
There is currently only synchronous endpoint support.
The design is intended to be fast and scalable. It leverage asynchronous API calls, multi-threading and a producer-consumer approach with thread-safe queues.