Skip to main content
NeMo Guardrails is an open-source library from NVIDIA that runs checks, called rails, on what users send to an LLM and on what the LLM sends back. Its openai engine accepts a base_url, so any OpenAI-compatible endpoint can back both the application model and the rails. This page points NeMo Guardrails at SambaCloud and turns on the built-in self-check input and output rails.

Prerequisites

macOS ships with Python 3.9 and exposes it as python3 rather than python. Check your version with python3 --version. If it is below 3.10, install a supported version with pyenv, uv, or brew install python@3.12 before continuing. nemoguardrails declares support for Python 3.10 up to, but not including, 3.14, so use 3.13 or lower. On Python 3.9, pip installs an older nemoguardrails release that this guide was not tested with.

Installation and setup

1

Create a virtual environment

2

Install NeMo Guardrails

This guide was tested with nemoguardrails 0.24.1 on Python 3.12.
3

Set your SambaCloud API key

Configure the model and rails

NeMo Guardrails reads a configuration folder. Create a config folder with two files.
1

Point the model and rails at SambaCloud

config.yml sets the model that answers your users and turns on the two self-check rails. The api_key_env_var setting names the environment variable that holds your key. It sits at the top level of the model entry, not inside parameters.
config/config.yml
The self-check rails use the main model unless you define a separate model for them, so this one model answers, checks the input, and checks the output.
2

Add the self-check prompts

The self-check rails need a prompt for each check. NeMo Guardrails refuses to load the configuration without them. Each prompt asks the model a yes or no question, and a “yes” blocks the message.
config/prompts.yml
Keep max_tokens: 2048 on both prompts. MiniMax-M3.1-Flash is a reasoning model, and NeMo Guardrails limits each check to 1024 tokens by default. When the model’s reasoning uses that whole budget, the check comes back empty, NeMo Guardrails treats an empty answer as unsafe, and it refuses a harmless request. In testing, the input check used up to 1,299 tokens on a single prompt.

Run a guarded conversation

Save this as guarded_chat.py next to the config folder. It sends one harmless message and one message the input rail should block:
guarded_chat.py
Keep options on every call. NeMo Guardrails sends the stop sequence User: with each application model call, and it cannot be changed in config.yml. MiniMax-M3.1-Flash is a reasoning model that can quote the conversation, including User:, inside its reasoning. When it does, the stop sequence cuts the response off before the answer starts, and the output rail refuses the empty reply with “I’m sorry, I can’t respond to that.” Whether this happens depends on the prompt. Passing options removes the stop sequence, so it cannot happen.
Output (abridged)
To see which rails ran for a request, call rails.explain() after generate:
Output for
For the blocked message, only self_check_input: Yes. appears. The input rail stopped the request before the application model was called.
A guarded request makes up to three model calls: the input check, the answer, and the output check. In one test with MiniMax-M3.1-Flash, a short answer took 2.3 to 7.0 seconds without rails and 4.4 to 10.7 seconds with both rails (6 runs each). Your numbers depend on the model, the prompt, and the length of the answer.

Stream responses

Streaming with output rails turned on is off by default. Enable it under rails.output.streaming in config.yml. The full file then reads:
config/config.yml
Then read the response with stream_async:
stream_chat.py
With output rails, NeMo Guardrails checks each streamed chunk before releasing it, and every check is a model call. With MiniMax-M3.1-Flash as the checker, the first chunk arrived after 5.6 to 7.5 seconds and the full answer after 7.1 to 9.9 seconds (20 runs). Individual runs vary, and the first chunk always waits for a check. If you need low time to first token, skip the output rail for streaming responses.

Troubleshooting

NeMo Guardrails checks the variable named in api_key_env_var when it loads the configuration. Export it in the same shell you run Python from, as shown in Installation and setup.
The key in SAMBANOVA_API_KEY is not valid. Copy a key from the SambaCloud API keys page and export it again.
The model value in config.yml is not a SambaCloud model ID. See SambaCloud models for the current list.
The base_url is missing /v1. Set it to https://api.sambanova.ai/v1.
NeMo Guardrails logged LLM returned empty content with finish_reason='length' for the self_check_input or self_check_output task. The reasoning model used the whole token budget before it wrote a yes or no, and an empty check counts as unsafe. Add max_tokens: 2048 to both prompts in prompts.yml, as shown in Configure the model and rails.If there is no such log line, run rails.explain() and look at the general call. An empty general completion has a different cause, described in the next entry.
NeMo Guardrails sends stop: ["User:"] with every application model call, and config.yml cannot change it. A reasoning model can quote the conversation, including User:, inside its reasoning. When it does, the stop sequence ends the response before the answer starts, and the output rail refuses the empty reply. Whether this happens depends on the prompt.Remove the stop sequence for each request by passing it as an LLM parameter, as guarded_chat.py does:
With the stop sequence removed, the input rail still blocks a harmful request. This was verified on nemoguardrails 0.24.1. To confirm the cause on your setup, run rails.explain() and check that the general completion is empty.
stream_async() cannot run when output rails are configured and streaming is off. Set rails.output.streaming.enabled to True as shown in Stream responses, or call generate_async() instead.

Additional resources