AI chatbots have become an essential tool for businesses and organizations. But most chatbot solutions depend on cloud-based models that introduce latency, API limitations, and perhaps most importantly, privacy concerns. What if you could run an AI chatbot entirely on your machine and still retain conversation history with a fully featured data platform?<\/p>\n\n\n\n
In this post, we\u2019ll walk through setting up a self-hosted AI chatbot using Docker Model Runner, a brand new feature from Docker that lets you run containerized models locally for inference and more, and Couchbase Capella to store, retrieve and search through conversations. The result is a fast, private and flexible chatbot that you control.<\/p>\n\n\n\n
Ready to get started? Let\u2019s get going!<\/p>\n\n\n\n
First, let\u2019s ensure that your Docker Desktop and CLI version is up to date so that you have the Model Runner feature available to you. You do that by running Once you\u2019ve done so, you will use Docker Model Runner to pull the container image with the Llama 3.2 model and make it available locally:<\/p>\n\n\n\n You can verify you successfully downloaded the Llama 3.2 model by running Want to test it out? Opening up the model in interactive mode is quite easy! Just run \n Now that you have Llama 3.2 downloaded and ready to use, it\u2019s time to build a straightforward backend application that leverages the model for a self-hosted AI chatbot.<\/p>\n\n\n\n The application you will create will accomplish the following tasks:<\/p>\n\n\n\n Your application will be fully featured and ready to use immediately from your console as a robust AI powered chatbot. The code we build together here will provide you with the foundation to refactor it according to whatever needs you have. Perhaps, you wish to turn it into a backend for a web application. A few modifications will be all that is required to make that possible.<\/p>\n\n\n\n The application requires a couple of dependencies so from the project directory run Make sure you have set up a bucket in Capella to store the chat data, and have your Capella credentials at hand. As always, do not save your credentials in any version control. Use environment variables for local development to keep your credentials secure and not in public hands.\u00a0<\/p>\n\n\n\n We are building four functions in our application:<\/p>\n\n\n\n Let\u2019s start with the [crayon nums=”false” lang=”default” decode=”true”]async function main() { \u00a0\u00a0console.log(“Self-Hosted AI Chatbot (Llama 3.2 + Capella)”); \u00a0\u00a0while (true) { \u00a0\u00a0\u00a0\u00a0if (userMessage.toLowerCase() === “exit”) { \u00a0\u00a0\u00a0\u00a0if (userMessage.toLowerCase() === “history”) { \u00a0\u00a0\u00a0\u00a0console.log(“\ud83e\udd16 Thinking…”); \u00a0\u00a0\u00a0\u00a0await storeChat(collection, userMessage, aiResponse); As you can see, we have introduced functionality built on top of Capella\u2019s data store, namely the ability to retrieve within the chat itself the previous chat history. This could be useful for a user to regain their chat context whenever they start up a new session.<\/p>\n\n\n\n Now that we have the [crayon nums=”false” lang=”default” decode=”true”]async function askAI(prompt) { Next, the [crayon nums=”false” lang=”default” decode=”true”]async function storeChat(collection, userMessage, aiResponse) { \u00a0\u00a0await collection.upsert(`chat_${Date.now()}`, chatDoc); Finally, the [crayon nums=”false” lang=”default” decode=”true”]async function fetchChatHistory(cluster, limit = 5) { Once you have finished with the functions, make sure to add [crayon nums=”false” lang=”default” decode=”true”]const { exec } = require(“child_process”); async function connectToCouchbase() { Lastly, don’t forget to add a line at the end of your script that calls the Once finished, you have a fully ready AI chatbot hosted on your own machine retaining your privacy and simultaneously leveraging Capella for storage and retrieval.<\/p>\n\n\n\n Your very own chatbot is ready to use! Every query you send it will be processed only locally on your machine using the Llama 3.2 model. Nothing will be sent to any AI provider remotely.<\/p>\n\n\n\n If you\u2019re ready to give it a spin, do so by running the following:<\/p>\n\n\n\n \n Once you run it, you will see the following:<\/p>\n\n\n\n \n Go ahead and start asking questions and interacting with it. Here\u2019s a brief example of what you can expect to see:<\/p>\n\n\n [crayon nums=”false” lang=”default” decode=”true”]Connected to Couchbase Capella<\/p>\n \ud83d\ude80 Self-Hosted AI Chatbot (Llama 3.2 + Capella) > Should I pack a raincoat for Barcelona for a trip there at the end of March? Answer only with “yes” or “no”. > Tell me why I should pack a raincoat for Barcelona at the end of March. Limit your answer to 15 words or less. > exit By using the scalability and security of Couchbase Capella along with Docker Model Runner\u2019s privacy-first approach to running AI models locally, you can build dynamic AI applications that prioritize user privacy. Running models on your own machine also gives you faster inference, full control over prompt customization, the ability to store additional metadata, and the flexibility to fine-tune your chatbot\u2019s behavior, all within your own environment.<\/p>\n\n\n\n This combination of Capella and Docker Model Runner gives you speed, control and the privacy needed to build AI applications without relying on external APIs. Whether you\u2019re creating a chatbot, analyzing data or running AI-powered workflows, this setup ensures that anything you build will be efficient, scalable and fully under your control. <\/p>\n\n\n\n The only question is: What will you build?<\/p>\n\n\n\ndocker model status<\/code> from your terminal. If it is successful, you will see a success message outputted that Docker Model Runner is running<\/code>. If you don\u2019t, then you need to first get the latest version of Docker and install it. After you update Docker, you can run that command again and it should work.<\/p>\n\n\n\ndocker model pull ai\/llama3.3<\/code>\n<\/p>\n\n\n\ndocker model list<\/code> and you will see the model available to you for use:<\/p>\n\n\n\n{\"object\":\"list\",\"data\":[{\"id\":\"ai\/llama3.3\",\"object\":\"model\",\"created\":1741794281,\"owned_by\":\"docker\"}]}<\/code>\n<\/p>\n\n\n\ndocker model run ai\/llama3.3<\/code> from the command line and you will enter the interactive mode:<\/p>\n\n\n\nInteractive chat mode started. Type \u2018\/bye\u2019 to exit.
\n<\/code>><\/code><\/p>\n\n\n\nCreating the chatbot<\/h2>\n\n\n\n
\n
docker model run<\/code>\u00a0CLI command<\/li>\n\n\nnpm install couchbase readline-sync<\/code>. We use the Couchbase Node.js SDK to interact with our Couchbase Capella data store, and we use readline-sync<\/code> to have the application interact with the user from the terminal.<\/p>\n\n\n\n\n
askAI<\/code> to encapsulate the process of feeding the message to our locally run AI model<\/li>\n\n\nstoreChat<\/code> to send the chat history to Capella<\/li>\n\n\nfetchChatHistory<\/code> to retrieve chat chats<\/li>\n\n\nmain<\/code> to act as the primary interface for the application<\/li>\n\n<\/ul>\n\n\n\nmain<\/code> function that\u2019ll wrap all the rest. This function will create a loop that the user can exit at any time offering a continuous chat experience:<\/p>\n\n\n
\n\u00a0const { cluster, collection } = await connectToCouchbase();<\/p>\n
\n\u00a0\u00a0console.log(“Type your message below. Type ‘history’ to view past chats or ‘exit’ to quit.n”);<\/p>\n
\n\u00a0\u00a0\u00a0\u00a0const userMessage = readlineSync.question(“> “);<\/p>\n
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0console.log(“Goodbye!”);
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0break;
\n\u00a0\u00a0\u00a0\u00a0}<\/p>\n
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0const history = await fetchChatHistory(cluster);
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0console.log(“n\ud83d\udcdc Chat History:”);
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0history.forEach((chat) => {
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0console.log(`\ud83e\uddd1 ${chat.user}n\ud83e\udd16 ${chat.response}n`);
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0});
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0continue;
\n\u00a0\u00a0\u00a0\u00a0}<\/p>\n
\n\u00a0\u00a0\u00a0\u00a0const aiResponse = await askAI(userMessage);
\n\u00a0\u00a0\u00a0\u00a0console.log(`\ud83e\udd16 ${aiResponse}n`);<\/p>\n
\n\u00a0\u00a0}
\n}[\/crayon]<\/p>\n\n\n\nmain<\/code> function, let\u2019s create the supporting functions that it invokes, starting with the askAI<\/code> function:<\/p>\n\n\n
\n\u00a0\u00a0return new Promise((resolve, reject) => {
\n\u00a0\u00a0\u00a0\u00a0exec(
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0`docker model run ai\/llama3.3 “${prompt}”`,
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0(error, stdout, stderr) => {
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0if (error) {
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0console.error(`Error running model: ${error.message}`);
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0reject(error);
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0}
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0if (stderr) {
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0console.error(`Docker stderr: ${stderr}`);
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0}
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0resolve(stdout.trim()); \/\/ Return AI response
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0}
\n\u00a0\u00a0\u00a0\u00a0);
\n\u00a0\u00a0});
\n}[\/crayon]<\/p>\n\n\n\nstoreChat<\/code> function:<\/p>\n\n\n
\n\u00a0\u00a0const chatDoc = {
\n\u00a0\u00a0\u00a0\u00a0user: userMessage,
\n\u00a0\u00a0\u00a0\u00a0response: aiResponse,
\n\u00a0\u00a0\u00a0\u00a0timestamp: new Date().toISOString(),
\n\u00a0\u00a0};<\/p>\n
\n}[\/crayon]<\/p>\n\n\n\nfetchChatHistory<\/code> function:<\/p>\n\n\n
\n\u00a0\u00a0const query = `
\n\u00a0\u00a0\u00a0\u00a0SELECT user, response, timestamp FROM `chatbot`
\n\u00a0\u00a0\u00a0\u00a0ORDER BY timestamp DESC
\n\u00a0\u00a0\u00a0\u00a0LIMIT ${limit};
\n\u00a0\u00a0`;
\n\u00a0\u00a0const result = await cluster.query(query);
\n\u00a0\u00a0return result.rows;
\n}[\/crayon]<\/p>\n\n\n\nrequire<\/code> statements at the top of the file, and to create a connection to Couchbase Capella cluster:<\/p>\n\n\n
\nconst readlineSync = require(“readline-sync”);
\nconst couchbase = require(“couchbase”);
\nrequire(“dotenv”).config();<\/p>\n
\n\u00a0\u00a0try {
\n\u00a0\u00a0\u00a0\u00a0const cluster = await couchbase.connect(COUCHBASE_URL, {
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0username: COUCHBASE_USERNAME,
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0password: COUCHBASE_PASSWORD,
\n\u00a0\u00a0\u00a0\u00a0});
\n\u00a0\u00a0\u00a0\u00a0const bucket = cluster.bucket(“chatbot”);
\n\u00a0\u00a0\u00a0\u00a0const collection = bucket.defaultCollection();
\n\u00a0\u00a0\u00a0\u00a0console.log(“Connected to Couchbase Capella”);
\n\u00a0\u00a0\u00a0\u00a0return { cluster, collection };
\n\u00a0\u00a0} catch (err) {
\n\u00a0\u00a0\u00a0\u00a0console.error(“Failed to connect to Couchbase:”, err);
\n\u00a0\u00a0\u00a0\u00a0process.exit(1);
\n\u00a0\u00a0}
\n}[\/crayon]<\/p>\n\n\n\nmain<\/code> function by inserting main();<\/code> on the last line.<\/p>\n\n\n\nUsing your self-hosted AI chatbot
\n<\/h2>\n\n\n\nnode index.js # or whatever you named your file<\/code>\n<\/p>\n\n\n\nConnected to Couchbase Capella<\/code><\/p>\n\n\n\nSelf-Hosted AI Chatbot (Llama 3.2 + Capella)<\/code><\/p>\n\n\n\nType your message below. Type 'history' to view past chats or 'exit' to quit.<\/code><\/p>\n\n\n\n><\/code>\n<\/p>\n\n\n\n
\nType your message below. Type ‘history’ to view past chats or ‘exit’ to quit.<\/p>\n
\n\ud83e\udd16 Thinking…
\n\ud83e\udd16 Yes.<\/p>\n
\n\ud83e\udd16 Thinking…
\n\ud83e\udd16 You may need a raincoat as March weather in Barcelona can be unpredictable and rainy, up to 10C.<\/p>\n
\n\ud83d\udc4b Goodbye![\/crayon]<\/p>\n\n\n\n