Build with Docker Compose for Agents - SambaNova Integration Guide

Docker Compose for Agents lets you define, deploy, and manage AI agent architectures using the same familiar YAML syntax as Docker Compose. It supports AI models, tools, and supporting services in a single configuration file. This guide runs an example stack in which a coordinating agent hosted on SambaCloud works alongside a second agent backed by an open model that Compose serves locally.

Prerequisites

Before you begin, ensure you have:

A SambaCloud account with an active API key
Docker Desktop 4.43.0+ or Docker Engine installed
Docker Compose v2.38 or later, on every platform. The example uses the models top-level element, which older versions of Compose do not recognize
A laptop with GPU support for running open models locally (alternatively, use Docker Offload if you don’t have a GPU)

Confirm your Compose version before you start:

docker compose version

Platform-specific requirements

For Linux (Docker Engine) or Windows (Docker Desktop): Ensure Docker Model Runner requirements are met, specifically that GPU support is enabled and necessary drivers are installed
For Linux (Docker Engine): Install Docker Compose version 2.38.1 or later

Setup

Clone the Repository

Clone the example repository and change into the project directory:

git clone https://github.com/sambanova/integrations.git
cd integrations/docker-compose-for-agents

This clones the default branch, so the example can change between runs. The agent image pins its dependencies in agents/requirements.txt (google-adk==1.5.0 and litellm==1.73.6) and the local model is pinned to the ai/qwen3:8B-Q4_0 tag, so a rebuild reproduces the same stack.To hold a fixed starting point, record the commit you cloned and return to it later:

git rev-parse HEAD          # note this value
git checkout <commit>       # return to it after a later pull

Verify the configuration parses before you start anything:

docker compose config --quiet

It exits 0 and prints nothing when the file is valid. On a Compose older than v2.38 the models element is unrecognized and this is where you find out, rather than midway through a 4.8 GB pull.

Configure Environment Variables

Create a .env file in your project directory with the following configuration:

SAMBANOVA_API_KEY=your_api_key_here
SAMBANOVA_BASE_URL=https://api.sambanova.ai/v1
SAMBANOVA_CHAT_MODEL=gemma-4-31B-it

SAMBANOVA_BASE_URL must end in /v1. The agents call SambaNova through an OpenAI-compatible client, which expects the versioned path.Set SAMBANOVA_CHAT_MODEL to match the flow you plan to run, because no current SambaCloud model covers both:

Flow you want	Model to set	Why
Upload an image and have it described	`gemma-4-31B-it`	The only current model that accepts image input
Hand off from the SambaNova agent to the Developer agent	`Meta-Llama-3.3-70B-Instruct`	Handoff needs tool calling, which `gemma-4-31B-it` does not support

See SambaCloud models for the full list and for the Preview designation that applies to gemma-4-31B-it.

The agent handoff in the example workflow is a tool call, so it only fires on a model listed in Function calling. gemma-4-31B-it is not one of them. If you set it and then ask for the Developer agent, the request is answered by the SambaNova agent instead of being routed, with no error to tell you the handoff was dropped.

Reference the .env.example file in the repository for additional configuration options. Note that it currently sets a model SambaCloud has retired, so use a value from the table above rather than copying it verbatim.

Define Agent Prompts

All agent prompts are defined in the compose.yml file. Customize agent behaviors, roles, and capabilities by modifying the prompts in this configuration file to match your use case.The file has two parts. The models top-level element declares the open model that Compose serves locally, and the service-level models block tells Compose which environment variables to inject so the container can reach it:

services:
  main-agent:
    build:
      context: ./agents
      dockerfile: Dockerfile
    models:
      qwen3:
        endpoint_var: MODEL_RUNNER_URL
        model_var: MODEL_RUNNER_MODEL
    ports:
      - 8000:8000
    environment:
      PORT: 8000
      SAMBANOVA_BASE_URL: ${SAMBANOVA_BASE_URL}
      SAMBANOVA_CHAT_MODEL: ${SAMBANOVA_CHAT_MODEL}
      SAMBANOVA_API_KEY: ${SAMBANOVA_API_KEY}
      SAMBANOVA_AGENT_NAME: sambanova_agent
      SAMBANOVA_AGENT_INSTRUCTION: |
        You are SambaNova. ...
      DEVELOPER_AGENT_NAME: developer_agent
      DEVELOPER_AGENT_INSTRUCTION: |
        You are developer. ...

models:
  qwen3:
    model: ai/qwen3:8B-Q4_0

The three ${...} values are the ones you set in .env. MODEL_RUNNER_URL and MODEL_RUNNER_MODEL are generated by Compose from the models element, so do not set them yourself.

Only the SambaNova agent runs on SambaCloud. The Developer agent runs on the locally served ai/qwen3:8B-Q4_0 model, which is why the GPU and Docker Model Runner prerequisites apply.

Start the Services

Launch the Docker Compose stack:

docker compose up

The first run pulls the ai/qwen3:8B-Q4_0 model, about 4.8 GB, before the agents start. Allow 10–15 minutes. Later runs reuse the cached model and start in seconds.

If you’ve updated the code or configuration, rebuild the containers:

docker compose up --build

The application is available at http://localhost:8000.To confirm the stack came up, list the running services:

docker compose ps

The main-agent service should be listed with a STATUS of Up and 0.0.0.0:8000->8000/tcp under PORTS. If you see only the header row, nothing is running:

NAME      IMAGE     COMMAND   SERVICE   CREATED   STATUS    PORTS

That means the stack exited during startup rather than failing to start, so the logs still hold the reason. Check docker compose logs and see Troubleshooting.

Usage

The agents can be accessed through both the web interface and API endpoints.

Example workflow

Enable Token Streaming for real-time responses, then follow these steps:

This workflow crosses both model capabilities: step 1 needs image input and step 3 needs tool calling. Run steps 1 and 2 with SAMBANOVA_CHAT_MODEL=gemma-4-31B-it, then switch to a tool-calling model such as Meta-Llama-3.3-70B-Instruct and restart the stack to exercise step 3. Because containers read environment variables at startup, changing .env requires docker compose down followed by docker compose up.

Upload an Image

Upload an image such as an application architecture diagram, system design, or workflow visualization through the web interface.

Interact with the SambaNova Agent

Ask the SambaNova agent to analyze and explain the uploaded content:

Sambanova, describe this app
Explain the info and the stack required to develop this app

Replies from this agent begin with SAMBANOVA_AGENT SPEAKING:, which is set by the prompt in compose.yml and is a quick way to confirm which agent answered.

Request Code Generation

Use the Developer agent to generate code based on the analysis:

Developer, can you create code from this description using Python?

This step is a handoff, so it requires a tool-calling model. You know the handoff succeeded when the reply comes from the Developer agent, which runs on the local ai/qwen3:8B-Q4_0 model rather than on SambaCloud. If the SambaNova agent answers instead, the handoff was dropped; see Troubleshooting.

Multi-agent collaboration

The Docker Compose setup enables multiple specialized agents to work together:

SambaNova Agent: Analyzes images, documents, and provides architectural insights
Developer Agent: Generates code based on specifications and requirements
Custom Agents: Define your own agents with specific roles and capabilities

Architecture

Docker Compose for Agents allows you to define:

AI Models: SambaNova models for inference and reasoning
Agent Services: Specialized agents with distinct roles and prompts
Supporting Services: Databases, message queues, storage, and other infrastructure
Tool Integration: Connect agents to external APIs, databases, and services

All components are orchestrated through a single compose.yml file, making it easy to:

Version control your entire agent architecture
Scale services independently
Deploy consistently across environments
Share configurations with your team

Benefits

Docker Compose for Agents with SambaNova combines containerization best practices with high-performance AI inference to deliver:

Familiar workflow: Use the same Docker Compose syntax you already know
Simplified deployment: Define complex multi-agent systems in a single YAML file
Local development: Test and iterate quickly on your local machine
Production-ready: Deploy to production with the same configuration
Flexible architecture: Easily add, remove, or modify agents and services
High-performance inference: Leverage SambaNova’s optimized models for fast response times

Troubleshooting

Authentication errors from the SambaNova agent

Confirm SAMBANOVA_API_KEY is set in the .env file in your project directory, since Compose reads values from there.
An empty value is not the same as a missing one. Compose substitutes a blank string for an unset variable and starts anyway, so the agents come up without credentials and fail on the first request rather than at startup.
Regenerate the key in the SambaCloud portal.
Containers read environment variables at startup, so after editing .env run docker compose down followed by docker compose up to apply the change.

An error mentions OPENAI_API_KEY or api.openai.com

This points at the Developer agent, not your SambaNova key. That agent talks to the locally served model through an openai-prefixed client whose address comes from MODEL_RUNNER_URL, which Compose generates from the models element.
When that address is empty, the client falls back to OpenAI’s public endpoint, which produces an OpenAI-flavored error. Treat it as a signal that Docker Model Runner is not serving the model.
Confirm your Compose version is v2.38 or later, since older versions ignore the models element and never inject the variable.
Confirm Docker Model Runner is enabled with GPU support.

First run appears to hang

The first docker compose up pulls the ai/qwen3:8B-Q4_0 model, about 4.8 GB, before the agents start. Allow 10–15 minutes.
Follow progress with docker compose logs -f.

GPU or Docker Model Runner errors

The stack runs an open model locally and requires Docker Model Runner with GPU support enabled.
On Linux (Docker Engine) or Windows (Docker Desktop), confirm the Model Runner requirements are met and the necessary drivers are installed.
Without a GPU, use Docker Offload instead.

Port 8000 already in use

Run docker compose ps to check whether an earlier run is still active, then stop it with docker compose down.
Otherwise, change the host port in the ports mapping in compose.yml.

Model not found

Set SAMBANOVA_CHAT_MODEL to a model ID served by SambaCloud. See SambaCloud models for the current list. If you copied the value from .env.example, it names a retired model.
For the image steps, use gemma-4-31B-it. The other current models accept text only and fail once you attach an image, so switching to one of them replaces this error with a less obvious one.
Use the bare model ID without a sambanova/ prefix. The agent adds the provider prefix itself, so a prefixed value resolves to sambanova/sambanova/... and fails.

The Developer agent never takes over

Asking for the Developer agent is a handoff, and the coordinating agent performs it with a tool call. If SAMBANOVA_CHAT_MODEL names a model that does not support tool calling, the call is dropped and the SambaNova agent answers the request itself. Nothing in the logs reports an error.
gemma-4-31B-it is not a tool-calling model. Set SAMBANOVA_CHAT_MODEL to a model listed in Function calling, such as Meta-Llama-3.3-70B-Instruct, then run docker compose down and docker compose up.
That model accepts text only, so exercise the handoff with a text description rather than an uploaded image.

The SAMBANOVA_API_KEY variable is not set warning

The full text is warning msg="The \"SAMBANOVA_API_KEY\" variable is not set. Defaulting to a blank string.", and Compose prints one line per missing variable. Because it defaults each to a blank string and continues, the agents start without credentials instead of failing fast.
The .env file must be in the directory you run docker compose up from, which is integrations/docker-compose-for-agents. Compose does not search parent directories.
Confirm the file is present with ls -a, since .env is hidden from a plain ls.

Additional resources

For comprehensive guidance on building agentic AI applications with Docker Compose, visit the official Docker Compose for Agents documentation.

Overview

Developer tools

Frameworks

Interfaces and low-code

Data and infrastructure

Multimodal

Evaluation and monitoring

Docker Compose for agents integration guide

Prerequisites

Platform-specific requirements

Setup

Usage

Example workflow

Multi-agent collaboration

Architecture

Benefits

Troubleshooting

Additional resources

​Prerequisites

​Platform-specific requirements

​Setup

​Usage

​Example workflow

​Multi-agent collaboration

​Architecture

​Benefits

​Troubleshooting

​Additional resources

Prerequisites

Platform-specific requirements

Setup

Usage

Example workflow

Multi-agent collaboration

Architecture

Benefits

Troubleshooting

Additional resources