Create and use CoE models

The Composition of Experts is a system of multiple experts, offered in a single endpoint, that enables the orchestration of these domain-specific experts across various fields. After creating and saving your CoE model, adjust the model share settings to share your CoE model with other users and tenants.

This document describes how to:

Create your own CoE model

SambaStudio allows you to create your own Composition of Experts model by creating a new CoE. By selecting and adding a number of experts to your own composition (CoE), you can tailor your new CoE model for your specific tasks.

The following considerations apply when creating a new CoE model:

  • Only SambaStudio platforms configured with an SN40L hardware generation are capable of creating CoE models.

  • All user roles can create a CoE model.

  • The available experts are dependant on the models downloaded to your SambaStudio Model Hub.

  • CoE model creation requires 8 RDUs. This means that one instance of a CoE model will be deployed on a single 8 RDU node.

Follow the steps below to create a new CoE model.

  1. From the Model Hub, click Create CoE Model.

    Create CoE model
    Figure 1. Create CoE model

  2. In the Create CoE Model box, enter a name for your CoE model into the Name field. Click Continue to proceed.

    Create CoE model box
    Figure 2. Create CoE model box

  3. The Add Experts window will open. The Add Experts window allows you to select and add expert models to your Composition of Experts model.

    1. Use the drop-down filters, Search box, and Expert buttons to refine the list of expert models displayed.

  4. Select the box next to the expert(s) you wish to add to your CoE model.

    1. From the Version column drop-down, you can a version of expert models that have more than one available version. The latest version of the model is selected by default.

      • You can select up to 60 experts when creating your CoE model. Selecting more than 60 experts will cause your CoE model to fail.

      • You should not include Samba-1.1 routers as a selection when creating your own CoE model.

  5. Click Add Experts to proceed.

    Add experts window
    Figure 3. Add Experts window

  6. The Review CoE details box will open. Here you can review and make edits to your CoE Model name or Description.

    1. Click the edit icon to edit the CoE Model name or Description.

    2. Click Confirm & create CoE to create your new CoE model.

      Once you confirm the creation of your CoE model, the name and description cannot be edited.
      Add experts window
      Figure 4. Review CoE details box

View your CoE model card

After you have created your CoE model you can view its model card by selecting it from the Model Hub. From the Model Hub, click your CoE model preview.

Model Hub preview
Figure 5. Example CoE model preview

Your CoE model card will open providing detailed information about that model including:

  • Application denotes the model’s application type of Language.

  • Type displays your CoE model type.

  • Owner denotes the CoE model owner.

  • Overview provides the model description you entered.

  • Experts in this Composition lists the expert models used to create your CoE model.

    • You can adjust the number of rows displayed and page through the list.

    • Hover over each expert to view its description.

  • Model version displays the selected version of the expert model used when creating the CoE.

  • RDU requirements provides:

Example user created CoE model card
Figure 6. Example user created CoE model card

Delete your CoE model

CoE models can be deleted by the model creator, organization administrators (OrgAdmin), and tenant administrators (TenantAdmin).

Follow the steps below to delete your CoE model.

  1. Select your CoE model card from the Model Hub.

  2. Click the trash icon to the right of your CoE model name.

    1. It is not possible to delete a CoE model when it has associated endpoints. A warning statement will appear identifying the associated endpoints that will need to be deleted before the CoE model can be deleted.

    2. You can click the blue endpoint name in the list to open its Endpoint window and view detailed information about it.

    3. Click X in the upper-right corner to close the warning statement.

  3. If your CoE model has no associated endpoints, click Yes from the Delete model box to permanently remove your CoE model from the Model Hub and make it no longer available for use.

Create an endpoint using your CoE model

Once you have a created your CoE model, you can use your model to create and deploy a CoE endpoint for prediction and Playground use.

See Create a CoE endpoint using the GUI to learn how to create and deploy your CoE endpoint.

Interact with your deployed CoE endpoint

Once your CoE endpoint has been deployed, you can interact with it as demonstrated in the example curl request below.

Example curl request
curl -X POST \
-H 'Content-Type: application/json' \
-H 'key: <your-endpoint-key>' \
--data '{"inputs":["{\"conversation_id\":\"conversation-id\",\"messages\":[{\"message_id\":0,\"role\":\"user\",\"content\":\"\"}]}"],"params":{"do_sample":{"type":"bool","value":"true"},"max_tokens_to_generate":{"type":"int","value":"1024"},"process_prompt":{"type":"bool","value":"true"},"repetition_penalty":{"type":"float","value":"1"},"select_expert":{"type":"str","value":"llama-2-7b-chat-hf"},"stop_sequences":{"type":"str","value":""},"temperature":{"type":"float","value":"0.7"},"top_k":{"type":"int","value":"50"},"top_p":{"type":"float","value":"0.95"}}}' '<your-sambastudio-domain>/api/predict/nlp/<project-key>/<endpoint>'

The example above makes a call to the llama-2-7b-chat-hf expert via the property "select_expert":{"type":"str","value":"llama-2-7b-chat-hf"}.

Use your CoE endpoint in the Playground

The Playground provides an in-platform experience for generating predictions using your deployed CoE endpoint.

  • See Use the Playground editor to learn how to use the editor with your Samba-1 endpoint.

  • Select a CoE expert or router describes how to choose a specialized CoE expert to use with your prompt in the Playground.

  • To get optimal responses from a task-specific CoE expert in the Playground, use the corresponding template to format your inputs.