Get started with Model Zoo

SambaNova Model Zoo is a public repository that includes sample model source code, along with example applications and libraries for compiling and running models on SambaNova hardware.

Model source code is available for several popular open source models and illustrates how we’ve customized the model for running on SambaNova hardware. Code examples and example apps enable developers to explore the capabilities of our Dataflow architecture and further customize the models.

You can find documentation in several places:

  • README files in the public Model Zoo GitHub repo External link explain how to set up the container environment, how to compile and run the models, and how to modify model source code and parameters. See the Documentation section in the Model Zoo release notes for a detailed list.

  • Code comments in the model source code and the YAML files in the Model Zoo GitHub repo External link help you understand how SambaNova engineers customized open source models to run efficiently on RDU, and how you can experiment with parameters.

  • In this doc set, you learn about the architecture and workflows, best practices, and troubleshooting.

  • The SambaFlow API Reference External link is also included in this doc set.

Model Zoo components

modelzoo folder hierarchy

Model Zoo consists of these main components:

  • README files at different levels of the hierarchy and files in /docs folders have setup instructions. More generic information, such as Troubleshooting, is included in this documentation set.

  • The code for several open-source models in the sambanova_modelzoo/models folder of the modelzoo repo. The code has been optimized for RDU and is commented to help you understand our model implementation.

  • Example apps for compile/train and compile/text generation. These example apps were fully tested with the models we ship with Model Zoo.

    • Minor modifications to the parameters are needed for different models. See Model Zoo best practices for compile commands for each model.

    • The apps for running on CPU (e.g. cpu_generate_text.py) are included so you can easily compare the CPU and RDU apps.

  • A container (called a DevBox) that includes SambaFlow and all prerequisites. Only SambaNova customers can access this container. Instructions are in the Container Setup document on GitHub External link.

Because Model Zoo demonstrates how to run and fine-tune models on SambaNova hardware, only SambaNova customers can use the Model Zoo examples on SambaNova hardware. If you are interested in access to Devbox and SambaNova Hardware, contact us at help@sambanova.ai.

Before you begin

To get started with Model Zoo, you need these components:

  • A container (called a Devbox) that includes SambaFlow and other prerequisite software. See Container Setup document on GitHub External link.

  • The Model Zoo code and examples, which you can download from the Model Zoo GitHub repo External link

  • Model-specific artifacts, which depend on what you want to do. See the Walkthrough—​Inference and Fine-tuning with Llama2 7B for Chat External link. Artifacts could include:

    • A Hugging Face checkpoint for the model.

    • Data for fine-tuning. When you want to fine tune one of the Model Zoo model with your data, you have to prepare those data. Our example illustrates how to do that with the Ultrachat dataset.

    • Data for generation. For the generation step, you might need prompt data or other data, depending on what you want to achieve.

Prepare your data

When there are problems with an AI model, the input data is often the problem. Having the data well structured and in the correct format is your responsibility.

  • Content: Which data you use also affects your model, but that discussion is outside the scope of this document. The internet has excellent discussions, and your favorite AI chatbot might also have an opinion.

  • Format: Model Zoo expects data in HDF5 (Hierarchical Data Format v5) format, an open-source format that supports large, complex data. Use the Data Preparation scripts in the SambaNova Data Preparation public GitHub repo External link.

The Data Preparation step External link in the Walkthrough has an example.

Compile and run text generation

To run text generation with any of the Model Zoo models, you follow the steps in Task 1 the Walkthrough External link.

  1. Collect all materials.

    1. Download the container (Devbox) from the container repository (SambaNova customers only).

    2. Clone the Model Zoo GitHub repo.

      git clone https://github.com/sambanova/modelzoo.git

  2. Set up the Devbox. As part of Devbox setup, you ensure that you’ll have enough storage available.

  3. Compile the model. The compiler creates a graph that is optimized for the SambaNova hardware. For step-by-step instructions, see the Walkthrough in the Examples README. For some background, see SambaFlow compiler overview.

  4. Start a generation run. With our example, you can specify a prompt in the configuration YAML file or on the command line, and the model generates an answer.

Compile and fine tune your model

To fine-tune your model, follow the steps in the Walkthrough in the Examples README in our the Model Zoo GitHub repo. Here’s an overview.

  1. Collect all prerequisite materials.

    1. Download the container (Devbox) from the container repository (SambaNova customers only).

    2. Clone the GitHub repository.

    3. Download a checkpoint for the model you’ll be working with from Hugging Face. Checkpoints that use BF16 are usually compatible with a model that’s been customized to run on RDU.

    4. Download the data we’re using in our walkthrough (Ultrachat) or use your own data and convert them using our data preparation scripts.

  2. Compile the model. The compiler creates a graph that is optimized for the SambaNova hardware. For step-by-step instructions, see the Walkthrough in the Examples README. For some background, see SambaFlow compiler overview.

  3. Start a training run with the checkpoint and your custom data. See the Walkthrough for details.

Learn more