{"id":4611,"date":"2025-03-24T11:06:51","date_gmt":"2025-03-24T18:06:51","guid":{"rendered":"https:\/\/www.couchbase.com\/blog\/chat-with-your-git-history-rag-couchbase-shell\/"},"modified":"2025-03-24T11:06:51","modified_gmt":"2025-03-24T18:06:51","slug":"chat-with-your-git-history-rag-couchbase-shell","status":"publish","type":"post","link":"https:\/\/www.couchbase.com\/blog\/pt\/chat-with-your-git-history-rag-couchbase-shell\/","title":{"rendered":"Chat With Your Git History, Thanks to RAG and Couchbase Shell"},"content":{"rendered":"\n

Don’t you love reading other people’s commit messages? No? Well, I do and as I was reading a very insightful commit message, I realized all the untapped content living in various Git logs (assuming the dev you follow are writing useful messages, of course). So, wouldn’t it be great if you could ask questions to a repo? Let’s see how this can be achieved doing RAG with Couchbase Shell.<\/p>\n\n\n\n

TL;DR<\/p>\n\n\n

[crayon wrap=”true” lang=”sh” decode=”true”]# with bash, extract your commit history to json
\nsource git-log-json.sh && git-log-json > commitlog.json<\/p>\n

# with cbsh, create scope, collection and collection Primary Index
\nscopes create gitlog; cb-env scope gitlog;collections create commits; cb-env collection commits; query “CREATE PRIMARY INDEX ON `default`:`cbsh`.`gitlog`.`commits`”<\/p>\n

# Import the doc in selected collection
\nopen commitlog.json | wrap content | insert id { |it| echo $it.content.commitHash } | doc upsert<\/p>\n

# Enrich the document with default model
\nquery “SELECT c.*, meta().id as id, c.subject || ‘ ‘ || c.body as text FROM `commits` as c” | wrap content| vector enrich-doc text | doc upsert<\/p>\n

# Create a Vector Index
\nvector create-index –similarity-metric dot_product commits textVector 1536<\/p>\n

# Run RAG
\nvector enrich-text “gemini” | vector search commits textVector –neighbors 20| select id |doc get| select content | reject -i content.textVector | par-each {|x| to json} | wrap content| ask “when and in which commit was gemini llm support added”[\/crayon]<\/p>\n\n\n\n

Couchbase Shell configuration<\/h2>\n\n\n\n

The initial step is to install and configure cbsh<\/em>. I am going to use my Capella instance. To get the config you can go under the Connect<\/strong> tab of your Capella cluster and select Couchbase Shell<\/strong>. This is the config under [[cluster]]<\/em>. To configure the model, take a look at what’s under [[llm]]<\/em>. I have chosen OpenAI but there are others. You need to define the model used for the embedding (that’s what turns text into a vector) and one for the Chat. This one takes the question and some additional context to answer the question. And of course you will need an API key.<\/p>\n\n\n

[crayon wrap=”true” lang=”yaml” decode=”true”]version = 1<\/p>\n

[[llm]]
\nidentifier = “OpenAI”
\nprovider = “OpenAI”
\nembed_model = “text-embedding-3-small”
\nchat_model = “gpt-3.5-turbo”
\napi_key = “sk-proj-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx”<\/p>\n

[[cluster]]
\nidentifier = “capella”
\nconnstr = “couchbases:\/\/cb.xxxxxxx.cloud.couchbase.com”
\nuser-display-name = “Laurent Doguin”
\nusername = “USER”
\npassword = “PASSWORD”
\ndefault-bucket = “cbsh”
\ndefault-scope = “gitlog”[\/crayon]<\/p>\n\n\n\n

You also need Git installed, then you should be all set.<\/p>\n\n\n\n

Import Git commit log<\/h2>\n\n\n\n

The first step is to get all the commits of the repo in JSON. Being lazy and old, and by old I mean not used to asking an AI, I searched for this on Google, found a number of Gists, that linked to other Gists, and I finally settled on this one<\/a>.<\/p>\n\n\n\n

I downloaded it, sourced it, went into my local couchbase-shell git repo and called it.<\/p>\n\n\n

[crayon nums=”false” lang=”default” decode=”true”]source git-log-json.sh && git-log-json > commitlog.json[\/crayon]<\/p>\n\n\n\n

But, for the benefit of the reader wondering if I made the right decision, let’s ask the configured model. Cbsh has an ask<\/em> command allowing you to to this:<\/p>\n\n\n

[crayon wrap=”true” lang=”sh” decode=”true”]\ud83d\udc64 Laurent Doguin \ud83c\udfe0 capella in \u2601\ufe0f cbsh.gitlog.commits
\n> ask “get the full commits in json for a git repo”<\/p>\n

To get the full commits in a Git repository as JSON, you can use the following command:<\/p>\n

git log –pretty=format:'{%n “commit”: “%H”,%n “author”: “%an <%ae>”,%n “date”: “%ad”,%n “message”: “%f”%n},’ –date=iso –reverse –all > commits.json
\n[\/crayon]<\/p>\n\n\n\n

This command will output each commit in the repository as a JSON object with the commit hash, author name and email, commit date, and commit message. The --all<\/code> flag ensures all branches are included. The --reverse<\/code> flag lists the commits in reverse chronological order. Finally, the output is redirected to a commits.json<\/code> file.<\/p>\n\n\n\n

Please make sure you run this command in the root directory of the Git repository you want to get the commits from.<\/p>\n\n\n\n

And as it turns out, it does not work out of the box (shocking I know). And it did not have all the info I needed, like the body part of the message. Of course we could spend time tuning this, but it’s very specific, with lots of edge cases.<\/p>\n\n\n\n

In any case I now have a list of commits in JSON format:<\/p>\n\n\n

[crayon lang=”js” decode=”true”][
\n {
\n ….
\n },
\n {
\n “author”: {
\n “name”: “Michael Nitschinger”,
\n “email”: “michael@nitschinger.at”,
\n “date”: “Thu, 20 Feb 2020 21:29:20 +0100”,
\n “dateISO8601”: “2020-02-20T21:29:20+01:00”
\n },
\n “body”: “”,
\n “commitHash”: “7a0d269fffd10045a63d40ca460deba944531890”,
\n “commitHashAbbreviated”: “7a0d269”,
\n “committer”: {
\n “name”: “Michael Nitschinger”,
\n “email”: “michael@nitschinger.at”,
\n “date”: “Thu, 20 Feb 2020 21:29:20 +0100”,
\n “dateISO8601”: “2020-02-20T21:29:20+01:00”
\n },
\n “encoding”: “”,
\n “notes”: “”,
\n “parent”: “”,
\n “parentAbbreviated”: “”,
\n “refs”: “”,
\n “signature”: {
\n “key”: “A6BCCB72D65B0D0F”,
\n “signer”: “”,
\n “verificationFlag”: “E”
\n },
\n “subject”: “Initial commit”,
\n “subjectSanitized”: “Initial-commit”,
\n “tree”: “3db442f3ef0438de58f72235e2658e5368a6752b”,
\n “treeAbbreviated”: “3db442f”
\n}][\/crayon]<\/p>\n\n\n\n

So what can you do with a JSON array of JSON objects? You can import it through the Capella UI or you can import them with Couchbase Shell. I first create the scope<\/em> and collection<\/em> and select them with cb-env, then create the SQL++ Index.<\/p>\n\n\n

[crayon wrap=”true” lang=”sh” decode=”true”]scopes create gitlog; cb-env scope gitlog; collections create commits; cb-env collection commits; query “CREATE PRIMARY INDEX ON `default`:`cbsh`.`gitlog`.`commits`”[\/crayon]<\/p>\n\n\n\n

Since cbsh is based on Nushell, the resulting JSON file can be easily opened, turned into a dataframe, transformed in a Couchbase document and inserted like so:<\/p>\n\n\n

[crayon line-height=”12″ nums=”false” lang=”default” decode=”true”]\ud83d\udc64 Laurent Doguin \ud83c\udfe0 capella in \u2601\ufe0f cbsh.gitlog.commits
\n> open commitlog.json |wrap content | insert id { |it| echo $it.content.commitHash }| doc upsert<\/p>\n

\u256d\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e
\n\u2502 # \u2502 processed \u2502 success \u2502 failed \u2502 failures \u2502 cluster \u2502
\n\u251c\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524
\n\u2502 0 \u2502 660 \u2502 660 \u2502 0 \u2502 \u2502 capella \u2502 \u2502
\n\u2570\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f
\n[\/crayon]<\/p>\n\n\n\n

Let’s get some documents just to see how it worked:<\/p>\n\n\n

[crayon line-height=”12″ nums=”false” lang=”default” decode=”true”]\ud83d\udc64 Laurent Doguin \ud83c\udfe0 capella in \u2601\ufe0f cbsh.gitlog.commits
\n> query “SELECT subject, body FROM `commits` LIMIT 1”
\n\u256d\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e
\n\u2502 # \u2502 subject \u2502 body \u2502 cluster \u2502
\n\u251c\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524
\n\u2502 0 \u2502 Bump Nushell \u2502 \u2502 capella \u2502
\n\u2570\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f<\/p>\n

[\/crayon]<\/p>\n\n\n\n

So this is content we could use for RAG. Time to enrich these docs.<\/p>\n\n\n\n

Enrich document with an AI model<\/h2>\n\n\n\n

To enrich the doc you need to have a model configured. Here I am using OpenAI and the enrich-doc<\/em> cbsh command:<\/p>\n\n\n

[crayon line-height=”12″ nums=”false” lang=”default” decode=”true”]\ud83d\udc64 Laurent Doguin \ud83c\udfe0 capella in \u2601\ufe0f cbsh.gitlog.commits
\n> query “SELECT c.*, meta().id as id, c.subject || ‘ ‘ || c.body as text FROM `commits` as c” | wrap content| vector enrich-doc text | doc upsert
\nEmbedding batch 1\/1
\n\u256d\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e
\n\u2502 # \u2502 processed \u2502 success \u2502 failed \u2502 failures \u2502 cluster \u2502
\n\u251c\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524
\n\u2502 0 \u2502 61 \u2502 61 \u2502 0 \u2502 \u2502 capella \u2502
\n\u2570\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f[\/crayon]<\/p>\n\n\n\n

The SELECT clause will return a JSON object with the content of the doc, and additional fields id<\/em> and text<\/em>. Text is the subject and body appended into one string. The object is wrapped in a content object and given to the vector enrich-doc<\/em> command, with text as a parameter, as it is the field that will be transformed in a vector. There should now be a textVector<\/em> field in each doc.<\/p>\n\n\n\n

Vector Search<\/h2>\n\n\n\n

In order to search through these vectors, we need to create a Vector Search index. It’s doable through the API or UI for something customizable. Here I am happy with default choices so I use cbsh instead:<\/p>\n\n\n

[crayon wrap=”true” lang=”default” decode=”true”]\ud83d\udc64 Laurent Doguin \ud83c\udfe0 capella in \u2601\ufe0f cbsh.gitlog.commits
\n> vector create-index –similarity-metric dot_product commits textVector 1536[\/crayon]<\/p>\n\n\n\n

The index created will use dot_product<\/em> as a similarity algorithm, vector dimensionality will be 1536, the name of the index is commit<\/em> and the indexed field is textVector<\/em>. The bucket, scope and collection are the one selected through cb-env<\/em>.<\/p>\n\n\n\n

To test vector search, the search query has to be turned in a vector, than piped to the search:<\/p>\n\n\n\n

[crayon wrap=”true” lang=”yaml” decode=”true”]version = 1<\/p>\n

[[llm]]
\nidentifier = “OpenAI”
\nprovider = “OpenAI”
\nembed_model = “text-embedding-3-small”
\nchat_model = “gpt-3.5-turbo”
\napi_key = “sk-proj-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx”<\/p>\n

[[cluster]]
\nidentifier = “capella”
\nconnstr = “couchbases:\/\/cb.xxxxxxx.cloud.couchbase.com”
\nuser-display-name = “Laurent Doguin”
\nusername = “USER”
\npassword = “PASSWORD”
\ndefault-bucket = “cbsh”
\ndefault-scope = “gitlog”[\/crayon]<\/p>\n0<\/p>\n\n\n\n

It returns 3 rows by default. Let’s extend it to see the content of the document. I am adding reject -i textVector<\/code> to remove the vector field, because no one needs a 1536 lines field in their terminal output:<\/p>\n\n\n\n

\"\"<\/p>\n\n\n\n

Ask your Git Repository<\/h2>\n\n\n\n

From here you have all the commits of a Git repository stored in Couchbase, enriched with an AI model, and all indexed and searchable. The last thing to do is call the model to run a query with RAG. It starts by a turning a question into a vector, pipe it to a vector search, get the full document from the return IDs, select the content object without the vector field, turn each object in a JSON doc (this way we can send the content and its structured metadata), wrap the jsonText<\/em> in a table and finally pipe it to the ask<\/em> command:<\/p>\n\n\n\n

[crayon wrap=”true” lang=”yaml” decode=”true”]version = 1<\/p>\n

[[llm]]
\nidentifier = “OpenAI”
\nprovider = “OpenAI”
\nembed_model = “text-embedding-3-small”
\nchat_model = “gpt-3.5-turbo”
\napi_key = “sk-proj-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx”<\/p>\n

[[cluster]]
\nidentifier = “capella”
\nconnstr = “couchbases:\/\/cb.xxxxxxx.cloud.couchbase.com”
\nuser-display-name = “Laurent Doguin”
\nusername = “USER”
\npassword = “PASSWORD”
\ndefault-bucket = “cbsh”
\ndefault-scope = “gitlog”[\/crayon]<\/p>\n1<\/p>\n\n\n\n

Asking the LLM when Gemini support was introduced. We get a date and a commit hash. It’s then easy to verify using git show<\/code>. There is a bit of repetition here so you can declare a variable for your question and reuse it:<\/p>\n\n\n\n

[crayon wrap=”true” lang=”yaml” decode=”true”]version = 1<\/p>\n

[[llm]]
\nidentifier = “OpenAI”
\nprovider = “OpenAI”
\nembed_model = “text-embedding-3-small”
\nchat_model = “gpt-3.5-turbo”
\napi_key = “sk-proj-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx”<\/p>\n

[[cluster]]
\nidentifier = “capella”
\nconnstr = “couchbases:\/\/cb.xxxxxxx.cloud.couchbase.com”
\nuser-display-name = “Laurent Doguin”
\nusername = “USER”
\npassword = “PASSWORD”
\ndefault-bucket = “cbsh”
\ndefault-scope = “gitlog”[\/crayon]<\/p>\n2<\/p>\n\n\n\n

And now we all know why the client crate had to be rewritten. It may not answer your own questions, but now you know how to get answers from any repo!<\/p>\n\n\n\n