Self-Hosted AI Chatbots with Docker and Couchbase Capella

AI chatbots have become an essential tool for businesses and organizations. But most chatbot solutions depend on cloud-based models that introduce latency, API limitations, and perhaps most importantly, privacy concerns. What if you could run an AI chatbot entirely on your machine and still retain conversation history with a fully featured data platform?

In this post, we’ll walk through setting up a self-hosted AI chatbot using Docker Model Runner, a brand new feature from Docker that lets you run containerized models locally for inference and more, and Couchbase Capella to store, retrieve and search through conversations. The result is a fast, private and flexible chatbot that you control.

Ready to get started? Let’s get going!

Setting up Docker Model Runner

First, let’s ensure that your Docker Desktop and CLI version is up to date so that you have the Model Runner feature available to you. You do that by running docker model status from your terminal. If it is successful, you will see a success message outputted that Docker Model Runner is running. If you don’t, then you need to first get the latest version of Docker and install it. After you update Docker, you can run that command again and it should work.

Once you’ve done so, you will use Docker Model Runner to pull the container image with the Llama 3.2 model and make it available locally:

docker model pull ai/llama3.3

You can verify you successfully downloaded the Llama 3.2 model by running docker model list and you will see the model available to you for use:

{"object":"list","data":[{"id":"ai/llama3.3","object":"model","created":1741794281,"owned_by":"docker"}]}

Want to test it out? Opening up the model in interactive mode is quite easy! Just run docker model run ai/llama3.3 from the command line and you will enter the interactive mode:

Interactive chat mode started. Type ‘/bye’ to exit.
<

Now that you have Llama 3.2 downloaded and ready to use, it’s time to build a straightforward backend application that leverages the model for a self-hosted AI chatbot.

Creating the chatbot

The application you will create will accomplish the following tasks:

• Run Llama 3.2 locally via the docker model run CLI command
• Send user messages as prompts to the model
• Store chat history in Couchbase Capella
• Retrieve previous chats

Your application will be fully featured and ready to use immediately from your console as a robust AI powered chatbot. The code we build together here will provide you with the foundation to refactor it according to whatever needs you have. Perhaps, you wish to turn it into a backend for a web application. A few modifications will be all that is required to make that possible.

The application requires a couple of dependencies so from the project directory run npm install couchbase readline-sync. We use the Couchbase Node.js SDK to interact with our Couchbase Capella data store, and we use readline-sync to have the application interact with the user from the terminal.

Make sure you have set up a bucket in Capella to store the chat data, and have your Capella credentials at hand. As always, do not save your credentials in any version control. Use environment variables for local development to keep your credentials secure and not in public hands. 

We are building four functions in our application:

• askAI to encapsulate the process of feeding the message to our locally run AI model
• storeChat to send the chat history to Capella
• fetchChatHistory to retrieve chat chats
• 메인 to act as the primary interface for the application

Let’s start with the 메인 function that’ll wrap all the rest. This function will create a loop that the user can exit at any time offering a continuous chat experience:

As you can see, we have introduced functionality built on top of Capella’s data store, namely the ability to retrieve within the chat itself the previous chat history. This could be useful for a user to regain their chat context whenever they start up a new session.

Now that we have the 메인 function, let’s create the supporting functions that it invokes, starting with the askAI function:

Next, the storeChat function:

Finally, the fetchChatHistory function:

Once you have finished with the functions, make sure to add 요구하다 statements at the top of the file, and to create a connection to Couchbase Capella cluster:

Lastly, don’t forget to add a line at the end of your script that calls the 메인 function by inserting main(); on the last line.

Once finished, you have a fully ready AI chatbot hosted on your own machine retaining your privacy and simultaneously leveraging Capella for storage and retrieval.

Using your self-hosted AI chatbot

Your very own chatbot is ready to use! Every query you send it will be processed only locally on your machine using the Llama 3.2 model. Nothing will be sent to any AI provider remotely.

If you’re ready to give it a spin, do so by running the following:

node index.js # or whatever you named your file

Once you run it, you will see the following:

Connected to Couchbase Capella

Self-Hosted AI Chatbot (Llama 3.2 + Capella)

Type your message below. Type 'history' to view past chats or 'exit' to quit.

<

Go ahead and start asking questions and interacting with it. Here’s a brief example of what you can expect to see:

By using the scalability and security of Couchbase Capella along with Docker Model Runner’s privacy-first approach to running AI models locally, you can build dynamic AI applications that prioritize user privacy. Running models on your own machine also gives you faster inference, full control over prompt customization, the ability to store additional metadata, and the flexibility to fine-tune your chatbot’s behavior, all within your own environment.

This combination of Capella and Docker Model Runner gives you speed, control and the privacy needed to build AI applications without relying on external APIs. Whether you’re creating a chatbot, analyzing data or running AI-powered workflows, this setup ensures that anything you build will be efficient, scalable and fully under your control.

The only question is: What will you build?

• Connect with our developer community and show us what you are building!