Provider Implementation Documentation

Overview

The VSKI Agent framework now includes built-in provider implementations for OpenAI and Ollama. These providers enable the framework to communicate with AI model APIs using a standardized interface.

Architecture

Provider System Structure

src/provider/
├── types.cr           # Type definitions (ProviderConfig, ProviderRequest, ProviderResponse, etc.)
├── provider.cr        # Base class and Registry
├── factory.cr         # Factory for creating providers from ModelConfig
├── openai.cr          # OpenAI provider implementation
├── ollama.cr          # Ollama provider implementation
└── register.cr        # Auto-registration of default providers

Key Components

1. Provider::Base - Abstract base class that all providers must inherit from

   • Defines the interface: complete, stream, complete_async
   • Provides helper methods for validation and ID generation

2. Provider::Registry - Global registry for provider instances

   • Register and retrieve providers by name
   • Thread-safe with mutex protection

3. Provider::Factory - Factory for creating providers from configuration

   • Maps provider names to provider classes
   • Creates provider instances from ModelConfig
   • Caches provider instances for reuse

4. Provider::Register - Auto-registration module

   • Registers OpenAI and Ollama providers on application startup
   • Ensures providers are available without manual registration

Implemented Providers

1. OpenAI Provider

File: src/provider/openai.cr

Features:

• Full OpenAI API compatibility
• Bearer token authentication
• Streaming support via Server-Sent Events (SSE)
• Tool/function calling support
• Custom base URL support (for OpenAI-compatible APIs)
• Error handling with detailed messages

Default Configuration:

• Base URL: https://api.openai.com/v1
• Endpoint: /chat/completions
• Authentication: Bearer token via API key

Supported Operations:

• ✅ Synchronous completion (complete)
• ✅ Streaming completion (stream)
• ✅ Asynchronous completion (complete_async)
• ✅ Tool calling
• ✅ Custom temperature and max_tokens

Usage Example:

# Create from ModelConfig
config = Config::ModelConfig.new(
  title: "gpt-4",
  provider: "openai",
  model_name: "gpt-4-turbo-preview",
  api_key: "${OPENAI_API_KEY}",
  temperature: 0.7
)

provider = Provider::Factory.create(config)

# Make a completion request
request = ProviderRequest.new(
  model: "gpt-4-turbo-preview",
  messages: [
    ProviderMessage.new(role: "user", content: "Hello, world!")
  ]
)

response = provider.complete(request)
puts response.content

2. Ollama Provider

File: src/provider/ollama.cr

Features:

• OpenAI-compatible API (uses /v1/chat/completions endpoint)
• No authentication required (optional)
• Streaming support via Server-Sent Events (SSE)
• Tool/function calling support
• Local deployment friendly
• Error handling with detailed messages

Default Configuration:

• Base URL: http://localhost:11434/v1
• Endpoint: /chat/completions
• Authentication: Optional (typically not needed for local)

Supported Operations:

• ✅ Synchronous completion (complete)
• ✅ Streaming completion (stream)
• ✅ Asynchronous completion (complete_async)
• ✅ Tool calling
• ✅ Custom temperature and max_tokens

Usage Example:

# Create from ModelConfig
config = Config::ModelConfig.new(
  title: "llama2",
  provider: "ollama",
  model_name: "llama2",
  base_url: "http://localhost:11434/v1"
)

provider = Provider::Factory.create(config)

# Make a completion request
request = ProviderRequest.new(
  model: "llama2",
  messages: [
    ProviderMessage.new(role: "user", content: "Why is the sky blue?")
  ]
)

response = provider.complete(request)
puts response.content

Configuration

YAML Configuration

Add provider configurations to your agent.config.yaml:

models:
  # OpenAI Configuration
  - title: gpt-4
    provider: openai
    model_name: gpt-4-turbo-preview
    api_key: ${OPENAI_API_KEY}
    temperature: 0.7
    max_tokens: 4096

  # OpenAI-Compatible Local API (like LM Studio, vLLM, etc.)
  - title: local-model
    provider: openai
    model_name: custom-model-name
    base_url: http://localhost:1234/v1
    temperature: 0.8

  # Ollama Configuration
  - title: llama2
    provider: ollama
    model_name: llama2
    base_url: http://localhost:11434/v1
    temperature: 0.8

  # Ollama with custom host
  - title: llama3-remote
    provider: ollama
    model_name: llama3
    base_url: http://192.168.1.100:11434/v1

Environment Variables

API keys can be set via environment variables:

# OpenAI
export OPENAI_API_KEY="sk-your-key-here"

# Use in config with ${OPENAI_API_KEY}

Streaming Support

Both providers support streaming responses via channels:

# Create streaming request
request = ProviderRequest.new(
  model: "gpt-4-turbo-preview",
  messages: [
    ProviderMessage.new(role: "user", content: "Tell me a story")
  ]
)

# Get streaming channel
channel = provider.stream(request)

# Process stream events
while event = channel.receive?
  case event.event
  when ProviderEvent::Data
    print event.data if event.data
  when ProviderEvent::ToolCall
    puts "Tool call: #{event.tool_call}"
  when ProviderEvent::Done
    puts "\n[Complete]"
  when ProviderEvent::Error
    puts "Error: #{event.error}"
  end
end

Tool Calling

Both providers support OpenAI-style function calling:

# Define tools
tools = [
  ProviderTool.new(
    name: "get_weather",
    arguments: '{"type": "object", "properties": {"location": {"type": "string"}}}'
  )
]

# Create request with tools
request = ProviderRequest.new(
  model: "gpt-4-turbo-preview",
  messages: [
    ProviderMessage.new(role: "user", content: "What's the weather in Paris?")
  ],
  tools: tools
)

response = provider.complete(request)

# Check for tool calls
if tool_calls = response.tool_calls
  tool_calls.each do |tc|
    puts "Tool: #{tc.function.name}"
    puts "Args: #{tc.function.arguments}"
  end
end

Using with Provider Factory

Create Provider from Config

# From ModelConfig
provider = Provider::Factory.create(model_config)

# Or with caching (recommended)
provider = Provider::Factory.create_or_get(model_config)

Create and Register

# Create and register in Provider::Registry
Provider::Factory.create_and_register(model_config, registry_name: "my-provider")

# Later, retrieve from registry
provider = Provider::Registry.get("my-provider")

Batch Registration

# Register all models from configuration
loader = Config::Loader.new
app_config = loader.load_app_config
providers = Provider::Factory.create_and_register_all(app_config.models)

Custom Base URLs

Both providers support custom base URLs, making them compatible with:

OpenAI-Compatible Services

• Azure OpenAI
• vLLM
• LM Studio
• LocalAI
• Text Generation WebUI (oobabooga)
• Any OpenAI-compatible API

Example: Local OpenAI-Compatible API

models:
  - title: local-llm
    provider: openai
    model_name: qwen3-zero-coder-reasoning-0.8b-neo-ex
    base_url: http://localhost:1234/v1
    temperature: 0.7

This configuration works with any local LLM server that provides an OpenAI-compatible API.

Error Handling

Both providers include comprehensive error handling:

begin
  response = provider.complete(request)
rescue ex : Exception
  puts "Request failed: #{ex.message}"
  # Handle error appropriately
end

Common errors:

• 401 Unauthorized: Invalid or missing API key
• 404 Not Found: Invalid endpoint or model not found
• 429 Too Many Requests: Rate limit exceeded
• 500+ Server Error: Provider service error

Thread Safety

All provider operations are thread-safe:

• Provider::Factory: Uses mutex for provider class registration and instance caching
• Provider::Registry: Uses mutex for provider registration and retrieval
• Provider Instances: Can be safely shared across fibers

Performance Considerations

Provider Caching

The factory caches provider instances to avoid recreating them:

# First call creates and caches
provider1 = Provider::Factory.create_or_get(config)

# Second call returns cached instance
provider2 = Provider::Factory.create_or_get(config)

# provider1 and provider2 are the same object

Connection Reuse

HTTP connections are created per-request but can be optimized in the future with connection pooling.

Async Operations

Use complete_async for non-blocking operations:

# Non-blocking
channel = provider.complete_async(request)

# Do other work...

# Get result when ready
response = channel.receive

Testing

Verify provider registration:

# Check if providers are registered
Provider::Factory.provider_class_registered?("openai")   # => true
Provider::Factory.provider_class_registered?("ollama")   # => true

# List registered providers
Provider::Factory.registered_provider_classes  # => ["openai", "ollama"]

# Check Provider::Register status
Provider::Register.defaults_registered?  # => true

Extending with New Providers

To add a new provider:

1. Create a new class inheriting from Provider::Base:

class Provider::Anthropic < Provider::Base
  def complete(request : ProviderRequest) : ProviderResponse
    # Implementation
  end

  def stream(request : ProviderRequest) : Channel(ProviderStreamEvent)
    # Implementation
  end

  def complete_async(request : ProviderRequest) : Channel(ProviderResponse)
    # Implementation
  end
end

2. Register with the factory:

Provider::Factory.register_provider_class("anthropic", Provider::Anthropic)

3. Add to auto-registration in src/provider/register.cr:

Provider::Factory.register_provider_class("anthropic", Provider::Anthropic)

Troubleshooting

Provider Not Registered

Error: Provider class 'xyz' is not registered

Solution: Ensure the provider is registered before use:

Provider::Register.register_defaults
# or
Provider::Factory.register_provider_class("xyz", Provider::XYZ)

Connection Refused

Error: Connection refused (Ollama)

Solution: Ensure Ollama is running:

ollama serve

Invalid API Key

Error: 401 Unauthorized

Solution: Check your API key configuration:

export OPENAI_API_KEY="sk-your-key"

Wrong Base URL

Error: 404 Not Found

Solution: Verify the base URL includes /v1:

• ✅ http://localhost:11434/v1
• ❌ http://localhost:11434

API Reference

ProviderRequest

record ProviderRequest,
  model : String,
  messages : Array(ProviderMessage),
  tools : Array(ProviderTool) | Nil = nil,
  temperature : Float64 | Nil = nil,
  max_tokens : Int32 | Nil = nil,
  stream : Bool = false

ProviderResponse

record ProviderResponse,
  id : String,
  model : String,
  content : String,
  tool_calls : Array(ProviderToolCall) | Nil = nil,
  finish_reason : String | Nil = nil,
  usage : ProviderUsage | Nil = nil

ProviderStreamEvent

record ProviderStreamEvent,
  event : ProviderEvent,
  data : String | Nil = nil,
  tool_call : ProviderToolCall | Nil = nil,
  error : String | Nil = nil

ProviderEvent Enum

enum ProviderEvent
  Data
  ToolCall
  Done
  Error
end

Summary

The OpenAI and Ollama providers are now fully integrated into the VSKI Agent framework:

✅ Auto-registered on startup ✅ Full API compatibility ✅ Streaming support ✅ Tool calling support ✅ Custom base URLs for local deployments ✅ Thread-safe operations ✅ Comprehensive error handling ✅ Provider caching for performance

The providers work seamlessly with the configuration system and can be easily extended for additional AI services.