Extended Token Generation

SambaStack supports extended token generation via the ignore_eos API parameter. By default, a model stops generating as soon as it produces an EOS (End of Sequence) token – a signal that it has finished its response. With ignore_eos enabled, the model ignores that signal and continues producing tokens until it reaches max_tokens. This is useful for benchmarking tools like AIPerf that require a fixed output length regardless of when the model would naturally stop.

ignore_eos is disabled by default. A SambaStack administrator must enable it explicitly per deployment before users can pass the parameter in API requests. If you pass ignore_eos: true on a deployment where ENABLE_IGNORE_EOS is not set, no error is returned – the parameter is silently ignored and the model behaves as if ignore_eos: false.

Enable ignore_eos for a deployment

To allow users to pass ignore_eos on a deployment, add ENABLE_IGNORE_EOS: "true" under engineConfig.env_vars in your ModelDeployment yaml.

apiVersion: sambanova.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: md-<name>
spec:
  bundle: <bundle_name>
  groups:
  - minReplicas: 1
    name: default
    qosList:
    - free
  owner: no-reply@sambanova.ai
  secretNames:
  - sambanova-artifact-reader
  engineConfig:
    startupTimeout: 7200
    env_vars:
      ENABLE_IGNORE_EOS: "true"

The example above deploys a bundle through spec.bundle. The same engineConfig.env_vars block applies to a single-model deployment that uses an inline spec.models instead. Apply the updated ModelDeployment yaml:

kubectl apply -f myModelDeployment.yaml

SambaWiz users only need to add the engineConfig.env_vars block to the generated YAML on the Model Deployment page – the rest of the spec is generated automatically.

Use ignore_eos in API requests

Once a deployment has ENABLE_IGNORE_EOS set, users can pass ignore_eos: true in their chat completions request body.

ignore_eos is a vLLM extension and is not part of the standard OpenAI API. Both the SambaNova and OpenAI Python clients accept it via extra_body.

from sambanova import SambaNova

client = SambaNova(
    base_url="your-sambanova-base-url",
    api_key="your-sambanova-api-key",
)

response = client.chat.completions.create(
    model="Meta-Llama-3.3-70B-Instruct",
    messages=[
        {"role": "user", "content": "Generate 500 words on the topic of renewable energy."}
    ],
    max_tokens=500,
    extra_body={"ignore_eos": True},
)

print(response.choices[0].message.content)

from openai import OpenAI

client = OpenAI(
    base_url="your-sambanova-base-url",
    api_key="your-sambanova-api-key",
)

response = client.chat.completions.create(
    model="Meta-Llama-3.3-70B-Instruct",
    messages=[
        {"role": "user", "content": "Generate 500 words on the topic of renewable energy."}
    ],
    max_tokens=500,
    extra_body={"ignore_eos": True},
)

print(response.choices[0].message.content)

Behavior

`ignore_eos` value	Effect
`false` (default)	Model stops when it produces an EOS token or reaches `max_tokens`, whichever comes first
`true`	Model ignores EOS tokens and continues generating until `max_tokens` is reached

ignore_eos is intended for benchmarking and testing workloads, not production user traffic. Known concurrency limits in SambaStack 1.2 (expected to be addressed in SambaStack 2.0):

Concurrency 1: Stable.
Concurrency 3: Failures at output lengths of 64K tokens or more.
Concurrency 10+: 100% failure rate at all output sizes.

Getting Started

Service Administration

Hardware Administration

Reference Architecture

Resources

Enable ignore_eos for a deployment

Use ignore_eos in API requests

Behavior

​Enable ignore_eos for a deployment

​Use ignore_eos in API requests

​Behavior

Enable ignore_eos for a deployment

Use ignore_eos in API requests

Behavior