Table of contents
- Semantic Search for Git Repositories
- What You'll Build
- Architecture
- Prerequisites
- Quick Start
- 1. Start Ollama and Pull Model
- 2. Start VSKI Server
- 3. Initialize the Database
- 4. Index Your Code
- 5. Search
- How It Works
- Project Structure
- Step-by-Step Tutorial
- Step 1: Docker Compose Setup
- Step 2: Define the Schema
- Step 3: Embedding Generation
- Step 4: File Indexing
- Step 5: Semantic Search
- Step 6: Command Line Entry Point
- Configuration
- Advanced Usage
- Troubleshooting
- Next Steps
- Resources
Semantic Search for Git Repositories
Note
See source code at https://vski.sh/x/embeddings_tutorial
A tutorial on building a semantic code search service using VSKI's vector embeddings and Ollama. This guide walks you through creating a CLI tool that indexes your codebase and enables natural language search across it.
What You'll Build
A CLI tool with two commands:
index <path>- Indexes a directory, tracks git changes, generates embeddingssearch <query>- Performs semantic search, returns ranked results with line numbers
Architecture
flowchart TD
CLI["CLI tool<br/>(Python)"] --> V["VSKI server<br/>(Docker)"]
V --> D[("SQLite + vectors<br/>(embedded)")]
CLI -->|"HTTP"| O["Ollama<br/>(local embeddings)"]
Prerequisites
Quick Start
1. Start Ollama and Pull Model
# Ensure Ollama is running
ollama serve
# Pull the embedding model
ollama pull nomic-embed-text
2. Start VSKI Server
docker compose up -d
This starts VSKI server on http://localhost:8347
Create admin account (first time only)
Navigate to installer http://localhost:8347/installer and create your admin.
Or use curl:
curl -X POST http://localhost:8347/api/admins/init \
-H "Content-Type: application/json" \
-d '{"email":"admin@example.com","password":"yourpassword"}'
3. Initialize the Database
# Install the VSKI command line tool (includes the Python SDK)
pip install "vski-sdk[cli]" --extra-index-url https://vski.sh/api/packages/x/pypi/simple/
# Login via the command line
vski login --url http://localhost:8347 --email admin@example.com --password yourpassword
# Create database
vski db create semantic_search
# Apply the schema (the vski/ YAML from Step 2 below)
vski apply
4. Index Your Code
# Index current directory
python src/main.py index .
# Index a specific directory
python src/main.py index ./src
5. Search
python src/main.py search "error handling"
python src/main.py search "database connection"
python src/main.py search "authentication middleware"
How It Works
Indexing Process
- File Discovery - Walks the directory tree, detects text files via MIME type
- Git Integration - Gets current branch, commit hash, and detects uncommitted changes
- Chunking - Splits files into ~500 token segments with 50 token overlap
- Line Tracking - Records line numbers for each chunk
- Embedding Generation - Sends chunks to Ollama for vector embedding
- Storage - Stores chunks and embeddings in VSKI
Search Process
- Query Embedding - Converts search query to vector via Ollama
- Vector Search - Queries VSKI for similar embeddings
- Ranking - Returns results sorted by similarity score
- Formatting - Displays file paths, line ranges, and content previews
Project Structure
docs/semantic_search/
├── README.md # This file
├── docker-compose.yml # Service definitions
├── vski/ # Declarative schema (IaC YAML)
│ └── semantic_search/
│ └── collections.yaml
└── src/
├── main.py # CLI entry point
├── indexer.py # Indexing logic
├── embedder.py # Ollama integration
├── searcher.py # Search logic
└── utils.py # Utility functions
Step-by-Step Tutorial
Step 1: Docker Compose Setup
The docker-compose.yml defines the VSKI service:
services:
vski:
image: vski.sh/x/vski:latest-standalone
ports:
- "8347:8347"
environment:
- SERVER_PORT=8347
- DATA_DIR=/app/data
- JWT_SECRET=dev-secret-change-in-production
volumes:
- vski_data:/app/data
volumes:
vski_data:
Ollama runs locally (not in Docker) for better GPU access.
Step 2: Define the Schema
The code_chunks collection stores indexed code segments. It is declared
in the IaC YAML (see IaC) and created by vski apply via a
server-side migration:
# vski/semantic_search/collections.yaml
version: 1
database: semantic_search
collections:
code_chunks:
type: base
fields:
- {name: file_path, type: text, required: true}
- {name: content, type: text, required: true}
- {name: line_start, type: number, required: true}
- {name: line_end, type: number, required: true}
- {name: branch, type: text}
- {name: commit_hash, type: text}
- {name: modified, type: bool}
Step 3: Embedding Generation
The embedder module handles communication with Ollama (its /api/embeddings
endpoint takes {"model": ..., "prompt": text} and returns
{"embedding": [...]}):
# src/embedder.py
import httpx
OLLAMA_URL = "http://localhost:11434"
EMBEDDING_MODEL = "nomic-embed-text"
async def generate_embedding(text: str) -> list[float]:
async with httpx.AsyncClient() as http:
response = await http.post(
f"{OLLAMA_URL}/api/embeddings",
json={"model": EMBEDDING_MODEL, "prompt": text},
)
response.raise_for_status()
return response.json()["embedding"]
Step 4: File Indexing
The indexer walks directories, chunks text, and stores embeddings:
# src/indexer.py
from vski import VskiClient
from embedder import generate_embedding
from utils import chunk_text, discover_text_files, get_git_info
async def index_directory(client: VskiClient, dir_path: str) -> None:
files = discover_text_files(dir_path) # list of pathlib.Path
git_info = get_git_info(dir_path)
for file in files:
content = file.read_text(encoding="utf-8", errors="ignore")
for chunk in chunk_text(content, str(file)):
embedding = await generate_embedding(chunk["content"])
record = await client.collection("code_chunks").create({
"file_path": chunk["file_path"],
"content": chunk["content"],
"line_start": chunk["line_start"],
"line_end": chunk["line_end"],
"branch": git_info["branch"],
"commit_hash": git_info["commit"],
"modified": git_info["has_changes"],
})
await client.embeddings.upsert("code_chunks", record["id"], embedding)
Step 5: Semantic Search
The searcher queries embeddings and formats results:
# src/searcher.py
from vski import VskiClient
from embedder import generate_embedding
async def search_code(
client: VskiClient,
query: str,
limit: int = 10,
) -> list[dict]:
embedding = await generate_embedding(query)
results = await client.embeddings.search(
"code_chunks", embedding, limit=limit, threshold=0.7
)
return [
{
"file_path": r["record"]["file_path"],
"line_start": r["record"]["line_start"],
"line_end": r["record"]["line_end"],
"content": r["record"]["content"],
"distance": r["distance"],
}
for r in results["results"]
]
Step 6: Command Line Entry Point
The main module ties everything together with a plain argparse CLI:
# src/main.py
import argparse
import asyncio
import os
from vski import VskiClient
from indexer import index_directory
from searcher import search_code
async def main(args: argparse.Namespace) -> None:
async with VskiClient(os.getenv("VSKI_URL", "http://localhost:8347")) as client:
res = await client.admins.auth_with_password(
"admin@example.com", "yourpassword"
)
client.set_token(res["token"])
client.set_db("semantic_search")
if args.command == "index":
await index_directory(client, args.path)
else: # search
results = await search_code(client, args.query)
display_results(results)
if __name__ == "__main__":
parser = argparse.ArgumentParser(prog="semantic-search")
sub = parser.add_subparsers(dest="command", required=True)
index = sub.add_parser("index")
index.add_argument("path")
search = sub.add_parser("search")
search.add_argument("query")
asyncio.run(main(parser.parse_args()))
Configuration
Environment Variables
| Variable | Default | Description |
|---|---|---|
VSKI_URL |
http://localhost:8347 |
VSKI server URL |
VSKI_TOKEN |
- | Admin JWT token |
OLLAMA_URL |
http://localhost:11434 |
Ollama server URL |
EMBEDDING_MODEL |
nomic-embed-text |
Ollama model for embeddings |
Chunking Parameters
| Parameter | Default | Description |
|---|---|---|
CHUNK_SIZE |
500 |
Target tokens per chunk |
CHUNK_OVERLAP |
50 |
Token overlap between chunks |
Advanced Usage
Index Only Changed Files
The indexer automatically detects uncommitted changes via git. You can filter:
# Only index modified files
files = discover_text_files(dir_path)
modified_files = [f for f in files if is_modified(str(f))]
Re-indexing
To re-index from scratch:
# Clear existing chunks
page = await client.collection("code_chunks").get_list(1, 10000)
for item in page["items"]:
await client.collection("code_chunks").delete(item["id"])
# Re-index
await index_directory(client, dir_path)
Multiple Branches
Index multiple branches for comparison:
git checkout feature-branch
python src/main.py index .
git checkout main
python src/main.py index .
Then search with branch filter:
results = await client.collection("code_chunks").get_list(
1, 20, filter='branch = "feature-branch"'
)
Troubleshooting
Ollama Model Not Found
# Pull the model manually
docker-compose exec ollama ollama pull nomic-embed-text
VSKI Authentication Failed
# Re-login
vski login --url http://localhost:8347
Slow Indexing
The indexer processes files sequentially. For large codebases, consider:
- Reducing chunk size
- Indexing specific directories
- Using a GPU-enabled Ollama setup
Next Steps
- Add file watching for automatic re-indexing
- Implement incremental updates (only index changed files)
- Add support for more embedding models
- Build a web UI for search
- Add syntax highlighting in previews
Resources
- Introduction
- Performance
- Getting Started
- Coding Agents
- Command Line
- Infra as Code
- CI/CD Guide
- Project Boilerplate
- SDKs
- Authentication
- Databases
- Collections
- Gated Queries
- Graphs
- Rules and Access Control
- Granular Access Control
- Triggers
- Plugins
- Containers Plugin
- vski mcp
- Realtime
- Full Text Search
- Vector Embeddings
- KWH
- Workflows
- Pipelines
- Dashboard Widgets
- Connectors
- Cron
- Webhooks
- Gates
- Migrations
- Replication
- File Sync
- Backup
- Distributed Realtime
- Tutorial: ReactJS Todo
- Tutorial: Graph-Memory Agent
- Tutorial: Image Gallery
- Tutorial: Embeddings
- Tutorial: Graphs
- Recipe: Calendar Timetree
- Tutorial: Plugins
- Tutorial: MCP Server