openai engine accepts a base_url, so any OpenAI-compatible endpoint can back both the application model and the rails. This page points NeMo Guardrails at SambaCloud and turns on the built-in self-check input and output rails.
Prerequisites
- A SambaCloud account and API key
- Python 3.10 to 3.13
macOS ships with Python 3.9 and exposes it as
python3 rather than python. Check your version with python3 --version. If it is below 3.10, install a supported version with pyenv, uv, or brew install python@3.12 before continuing. nemoguardrails declares support for Python 3.10 up to, but not including, 3.14, so use 3.13 or lower. On Python 3.9, pip installs an older nemoguardrails release that this guide was not tested with.Installation and setup
1
Create a virtual environment
2
Install NeMo Guardrails
nemoguardrails 0.24.1 on Python 3.12.3
Set your SambaCloud API key
Configure the model and rails
NeMo Guardrails reads a configuration folder. Create aconfig folder with two files.
1
Point the model and rails at SambaCloud
config.yml sets the model that answers your users and turns on the two self-check rails. The api_key_env_var setting names the environment variable that holds your key. It sits at the top level of the model entry, not inside parameters.config/config.yml
main model unless you define a separate model for them, so this one model answers, checks the input, and checks the output.2
Add the self-check prompts
The self-check rails need a prompt for each check. NeMo Guardrails refuses to load the configuration without them. Each prompt asks the model a yes or no question, and a “yes” blocks the message.
config/prompts.yml
Run a guarded conversation
Save this asguarded_chat.py next to the config folder. It sends one harmless message and one message the input rail should block:
guarded_chat.py
Output (abridged)
rails.explain() after generate:
Output for
self_check_input: Yes. appears. The input rail stopped the request before the application model was called.
A guarded request makes up to three model calls: the input check, the answer, and the output check. In one test with
MiniMax-M3.1-Flash, a short answer took 2.3 to 7.0 seconds without rails and 4.4 to 10.7 seconds with both rails (6 runs each). Your numbers depend on the model, the prompt, and the length of the answer.Stream responses
Streaming with output rails turned on is off by default. Enable it underrails.output.streaming in config.yml. The full file then reads:
config/config.yml
stream_async:
stream_chat.py
Troubleshooting
Model API Key environment variable 'SAMBANOVA_API_KEY' not set
Model API Key environment variable 'SAMBANOVA_API_KEY' not set
NeMo Guardrails checks the variable named in
api_key_env_var when it loads the configuration. Export it in the same shell you run Python from, as shown in Installation and setup.[401] Incorrect API key provided
[401] Incorrect API key provided
The key in
SAMBANOVA_API_KEY is not valid. Copy a key from the SambaCloud API keys page and export it again.[404] The model does not exist or you do not have access to it
[404] The model does not exist or you do not have access to it
The
model value in config.yml is not a SambaCloud model ID. See SambaCloud models for the current list.404 page not found
404 page not found
The
base_url is missing /v1. Set it to https://api.sambanova.ai/v1.A harmless request returns "I'm sorry, I can't respond to that."
A harmless request returns "I'm sorry, I can't respond to that."
NeMo Guardrails logged
LLM returned empty content with finish_reason='length' for the self_check_input or self_check_output task. The reasoning model used the whole token budget before it wrote a yes or no, and an empty check counts as unsafe. Add max_tokens: 2048 to both prompts in prompts.yml, as shown in Configure the model and rails.If there is no such log line, run rails.explain() and look at the general call. An empty general completion has a different cause, described in the next entry.The application model returns an empty answer for some prompts
The application model returns an empty answer for some prompts
NeMo Guardrails sends With the stop sequence removed, the input rail still blocks a harmful request. This was verified on
stop: ["User:"] with every application model call, and config.yml cannot change it. A reasoning model can quote the conversation, including User:, inside its reasoning. When it does, the stop sequence ends the response before the answer starts, and the output rail refuses the empty reply. Whether this happens depends on the prompt.Remove the stop sequence for each request by passing it as an LLM parameter, as guarded_chat.py does:nemoguardrails 0.24.1. To confirm the cause on your setup, run rails.explain() and check that the general completion is empty.StreamingNotSupportedError
StreamingNotSupportedError
stream_async() cannot run when output rails are configured and streaming is off. Set rails.output.streaming.enabled to True as shown in Stream responses, or call generate_async() instead.
