3 Embeddings Tutorial
Anton Nesterov edited this page 2026-11-03 15:42:27 +01:00

Semantic Search for Git Repositories

Note

See source code at https://vski.sh/x/embeddings_tutorial

A tutorial on building a semantic code search service using VSKI's vector embeddings and Ollama. This guide walks you through creating a CLI tool that indexes your codebase and enables natural language search across it.

What You'll Build

A CLI tool with two commands:

  • index <path> - Indexes a directory, tracks git changes, generates embeddings
  • search <query> - Performs semantic search, returns ranked results with line numbers

Architecture

flowchart TD
    CLI["CLI tool<br/>(Python)"] --> V["VSKI server<br/>(Docker)"]
    V --> D[("SQLite + vectors<br/>(embedded)")]
    CLI -->|"HTTP"| O["Ollama<br/>(local embeddings)"]

Prerequisites

Quick Start

1. Start Ollama and Pull Model

# Ensure Ollama is running
ollama serve

# Pull the embedding model
ollama pull nomic-embed-text

2. Start VSKI Server

docker compose up -d

This starts VSKI server on http://localhost:8347

Create admin account (first time only)

Navigate to installer http://localhost:8347/installer and create your admin.

Or use curl:

curl -X POST http://localhost:8347/api/admins/init \
  -H "Content-Type: application/json" \
  -d '{"email":"admin@example.com","password":"yourpassword"}'

3. Initialize the Database

# Install the VSKI command line tool (includes the Python SDK)
pip install "vski-sdk[cli]" --extra-index-url https://vski.sh/api/packages/x/pypi/simple/

# Login via the command line
vski login --url http://localhost:8347 --email admin@example.com --password yourpassword

# Create database
vski db create semantic_search

# Apply the schema (the vski/ YAML from Step 2 below)
vski apply

4. Index Your Code

# Index current directory
python src/main.py index .

# Index a specific directory
python src/main.py index ./src
python src/main.py search "error handling"
python src/main.py search "database connection"
python src/main.py search "authentication middleware"

How It Works

Indexing Process

  1. File Discovery - Walks the directory tree, detects text files via MIME type
  2. Git Integration - Gets current branch, commit hash, and detects uncommitted changes
  3. Chunking - Splits files into ~500 token segments with 50 token overlap
  4. Line Tracking - Records line numbers for each chunk
  5. Embedding Generation - Sends chunks to Ollama for vector embedding
  6. Storage - Stores chunks and embeddings in VSKI

Search Process

  1. Query Embedding - Converts search query to vector via Ollama
  2. Vector Search - Queries VSKI for similar embeddings
  3. Ranking - Returns results sorted by similarity score
  4. Formatting - Displays file paths, line ranges, and content previews

Project Structure

docs/semantic_search/
├── README.md              # This file
├── docker-compose.yml     # Service definitions
├── vski/                  # Declarative schema (IaC YAML)
│   └── semantic_search/
│       └── collections.yaml
└── src/
    ├── main.py            # CLI entry point
    ├── indexer.py         # Indexing logic
    ├── embedder.py        # Ollama integration
    ├── searcher.py        # Search logic
    └── utils.py           # Utility functions

Step-by-Step Tutorial

Step 1: Docker Compose Setup

The docker-compose.yml defines the VSKI service:

services:
  vski:
    image: vski.sh/x/vski:latest-standalone
    ports:
      - "8347:8347"
    environment:
      - SERVER_PORT=8347
      - DATA_DIR=/app/data
      - JWT_SECRET=dev-secret-change-in-production
    volumes:
      - vski_data:/app/data

volumes:
  vski_data:

Ollama runs locally (not in Docker) for better GPU access.

Step 2: Define the Schema

The code_chunks collection stores indexed code segments. It is declared in the IaC YAML (see IaC) and created by vski apply via a server-side migration:

# vski/semantic_search/collections.yaml
version: 1
database: semantic_search

collections:
  code_chunks:
    type: base
    fields:
      - {name: file_path, type: text, required: true}
      - {name: content, type: text, required: true}
      - {name: line_start, type: number, required: true}
      - {name: line_end, type: number, required: true}
      - {name: branch, type: text}
      - {name: commit_hash, type: text}
      - {name: modified, type: bool}

Step 3: Embedding Generation

The embedder module handles communication with Ollama (its /api/embeddings endpoint takes {"model": ..., "prompt": text} and returns {"embedding": [...]}):

# src/embedder.py
import httpx

OLLAMA_URL = "http://localhost:11434"
EMBEDDING_MODEL = "nomic-embed-text"


async def generate_embedding(text: str) -> list[float]:
    async with httpx.AsyncClient() as http:
        response = await http.post(
            f"{OLLAMA_URL}/api/embeddings",
            json={"model": EMBEDDING_MODEL, "prompt": text},
        )
        response.raise_for_status()
        return response.json()["embedding"]

Step 4: File Indexing

The indexer walks directories, chunks text, and stores embeddings:

# src/indexer.py
from vski import VskiClient

from embedder import generate_embedding
from utils import chunk_text, discover_text_files, get_git_info


async def index_directory(client: VskiClient, dir_path: str) -> None:
    files = discover_text_files(dir_path)   # list of pathlib.Path
    git_info = get_git_info(dir_path)

    for file in files:
        content = file.read_text(encoding="utf-8", errors="ignore")

        for chunk in chunk_text(content, str(file)):
            embedding = await generate_embedding(chunk["content"])

            record = await client.collection("code_chunks").create({
                "file_path": chunk["file_path"],
                "content": chunk["content"],
                "line_start": chunk["line_start"],
                "line_end": chunk["line_end"],
                "branch": git_info["branch"],
                "commit_hash": git_info["commit"],
                "modified": git_info["has_changes"],
            })

            await client.embeddings.upsert("code_chunks", record["id"], embedding)

The searcher queries embeddings and formats results:

# src/searcher.py
from vski import VskiClient

from embedder import generate_embedding


async def search_code(
    client: VskiClient,
    query: str,
    limit: int = 10,
) -> list[dict]:
    embedding = await generate_embedding(query)

    results = await client.embeddings.search(
        "code_chunks", embedding, limit=limit, threshold=0.7
    )

    return [
        {
            "file_path": r["record"]["file_path"],
            "line_start": r["record"]["line_start"],
            "line_end": r["record"]["line_end"],
            "content": r["record"]["content"],
            "distance": r["distance"],
        }
        for r in results["results"]
    ]

Step 6: Command Line Entry Point

The main module ties everything together with a plain argparse CLI:

# src/main.py
import argparse
import asyncio
import os

from vski import VskiClient

from indexer import index_directory
from searcher import search_code


async def main(args: argparse.Namespace) -> None:
    async with VskiClient(os.getenv("VSKI_URL", "http://localhost:8347")) as client:
        res = await client.admins.auth_with_password(
            "admin@example.com", "yourpassword"
        )
        client.set_token(res["token"])
        client.set_db("semantic_search")

        if args.command == "index":
            await index_directory(client, args.path)
        else:  # search
            results = await search_code(client, args.query)
            display_results(results)


if __name__ == "__main__":
    parser = argparse.ArgumentParser(prog="semantic-search")
    sub = parser.add_subparsers(dest="command", required=True)

    index = sub.add_parser("index")
    index.add_argument("path")

    search = sub.add_parser("search")
    search.add_argument("query")

    asyncio.run(main(parser.parse_args()))

Configuration

Environment Variables

Variable Default Description
VSKI_URL http://localhost:8347 VSKI server URL
VSKI_TOKEN - Admin JWT token
OLLAMA_URL http://localhost:11434 Ollama server URL
EMBEDDING_MODEL nomic-embed-text Ollama model for embeddings

Chunking Parameters

Parameter Default Description
CHUNK_SIZE 500 Target tokens per chunk
CHUNK_OVERLAP 50 Token overlap between chunks

Advanced Usage

Index Only Changed Files

The indexer automatically detects uncommitted changes via git. You can filter:

# Only index modified files
files = discover_text_files(dir_path)
modified_files = [f for f in files if is_modified(str(f))]

Re-indexing

To re-index from scratch:

# Clear existing chunks
page = await client.collection("code_chunks").get_list(1, 10000)
for item in page["items"]:
    await client.collection("code_chunks").delete(item["id"])

# Re-index
await index_directory(client, dir_path)

Multiple Branches

Index multiple branches for comparison:

git checkout feature-branch
python src/main.py index .

git checkout main
python src/main.py index .

Then search with branch filter:

results = await client.collection("code_chunks").get_list(
    1, 20, filter='branch = "feature-branch"'
)

Troubleshooting

Ollama Model Not Found

# Pull the model manually
docker-compose exec ollama ollama pull nomic-embed-text

VSKI Authentication Failed

# Re-login
vski login --url http://localhost:8347

Slow Indexing

The indexer processes files sequentially. For large codebases, consider:

  • Reducing chunk size
  • Indexing specific directories
  • Using a GPU-enabled Ollama setup

Next Steps

  • Add file watching for automatic re-indexing
  • Implement incremental updates (only index changed files)
  • Add support for more embedding models
  • Build a web UI for search
  • Add syntax highlighting in previews

Resources