Create and use endpoints

With the SambaStudio platform, you can generate predictions from your models on your data by deploying it to an endpoint.

This document describes how to create, view, and edit endpoints.

Create an endpoint using the GUI

Follow the sequence below to create an endpoint using the GUI to be used for prediction.

Base models do not support inference and cannot be deployed for endpoints. It is recommended to use Base models for training and not inference.

  1. Create a new project or use an existing one.

  2. From a project window, click New endpoint. The Add an endpoint window (Figure 1) will open.

  3. In the Add an endpoint window, enter a name for the endpoint into the Endpoint name field, as shown in Figure 1.

  4. The Share settings drop-down provides options for which tenant to share your endpoint, as shown in Figure 1.

    1. Share with <current-tenant> allows the endpoint to be shared with the current tenant you are using, identified by its name in the drop-down.

    2. Share with all tenants allows the endpoint to be shared across all tenants.

    3. Endpoint will be shared with all users in <current-tenant> identifies that the endpoint will be shared with other users in the tenant you are using.

      If the Endpoint will be shared with all users in <current-tenant> option is displayed, the Share with <current-tenant> and Share with all tenants options described above will not be available. Share with all tenants is an optional feature of SambaStudio. Please contact your administrator or SambaNova representative for more information.

  5. Select the ML App from the ML App drop-down, as shown in Figure 1.

  6. From the Select model drop-down, choose My models, Shared models, SambaNova models, or Select from Model Hub.

    1. My models displays a list of models that you have previously added to the Model Hub.

    2. Shared models displays a list of models that have been shared with the selected active tenant.

    3. SambaNova models displays a list of models provided by SambaNova.

      Add an Endpoint
      Figure 1. Add an endpoint window

    4. Select from Model Hub displays a window with a list of downloaded models that correspond to a selected ML App, as shown in Figure 2, or a list of all the downloaded models if an ML App is not selected. The list can be filtered by selecting options under Field of application, ML APP, Architecture, and Owner. Additionally, you can enter a term or value into the Search field to refine the model list by that input. Choose the model you wish to use and confirm your choice by clicking Use model.

      Model Hub
      Figure 2. Select from Model Hub

  7. Enter a value for the Number of instances to designate how many instances will be deployed, as shown in Figure 3. The Number of instances setting helps the endpoint scale to accommodate a high volume number of requests. You can adjust the value based on the expected volume number of requests and available resources.

  1. Click Add an endpoint, as shown in Figure 3, to queue the endpoint for deployment. Once the endpoint is created, it will take a few minutes for the endpoint to be deployed.

    Add an endpoint
    Figure 3. Number of instances, add an endpoint

    Your endpoint is deployed and will be available to generate predictions when the endpoint status displays Live.

Create an endpoint using the CLI

The example below demonstrates how to create an endpoint using the snapi endpoint create command. You will need to provide the following:

  • A project to assign the endpoint. Create a new project or use an existing one.

  • A new endpoint name.

  • The model to be used for the endpoint.

  • The number of instances to be used for the endpoint.

Example snapi create endpoint command
$ snapi endpoint create \
   --project <project-name> \
   --endpoint_name <endpoint-name> \
   --model-checkpoint <model-name> \
   --instances <number-of-instances>

View an endpoint using the GUI

Open the Endpoint window to view detailed information about the endpoint.

  1. Click Projects from the left menu to navigate to the Projects list.

    Project list
    Figure 4. Project list

  2. Click the Project that contains the endpoint you wish to view. That Project window will open displaying the Endpoints table, which is located beneath the Jobs table. The Endpoints table provides a summary of information about each endpoint, including:

    1. Name displays the name of the endpoint.

    2. Model displays the associated model of the endpoint.

    3. Instances displays the number of instances of the endpoint.

    4. Status displays the current status of the endpoint.

    5. Date of creation displays the date and time the endpoint was created.

    6. Owner displays the owner of the endpoint.

    7. Actions provides additional interactions to the endpoint via a drop-down menu.

      Endpoints table
      Figure 5. Endpoints table

  3. For detailed information about the endpoint, click the endpoint you wish to view from the list. The Endpoint window will open. The Endpoint window displays the following information:

    1. The status of the endpoint is displayed in the top Details bar.

    2. The endpoint name is displayed in the top of the window and in the breadcrumb path at the top of the window.

    3. URL provides the URL of the endpoint to use programmatically to make requests, such as generating predictions. Use the copy icon to temporarily store the URL (copy) to your computer’s clipboard.

    4. ML App displays the ML App that was selected when the endpoint was created.

    5. Model/Checkpoint displays the endpoint’s associated model/checkpoint.

    6. The API key allows you to make authenticated requests of the endpoint URL, such as generating predictions. Use the copy icon to temporarily store the endpoint API key (copy) to your computer’s clipboard.

      The endpoint API key and the platform API key have two distinct implementations.

      1. The endpoint API key is a unique API key generated by the platform for each endpoint.

      2. The platform API key is used as an authorization key for authenticated access to the platform. See SambaStudio resources for information about the platform API key.
      Endpoint window
      Figure 6. Endpoint window

View endpoint info using the CLI

The example below demonstrates how to use the snapi endpoint info command to view the info of an endpoint. You will need to provide the following:

  • The project the endpoint is assigned.

  • The endpoint name.

Example snapi endpoint info command
snapi endpoint info \
    --project <project-name> \
    --endpoint <endpoint-name>

Edit an endpoint using the GUI

The platform provides the ability to edit an existing endpoint.

  1. From an Endpoint window, click Edit, as shown in Figure 7. Or, from the Endpoints table, click Edit from the Actions drop-down, as shown in Figure 8. The Update the endpoint window will open.

    Edit endpoint
    Figure 7. Edit endpoint from the endpoint window
    Edit endpoint
    Figure 8. Edit endpoint from the endpoints table

  2. As shown in Figure 9, from the Update the endpoint window you can:

    1. Enter a new description into the Description field.

    2. Update the tenant to share your endpoint from the Share settings drop-down.

    3. Choose a different model from the Select Model drop-down.

    4. Update the Number of instances.

  3. Click Edit endpoint to complete process and send the endpoint to the queue.

    Update endpoint window
    Figure 9. Update the endpoint

The endpoint’s new description, model, and number of instances will be reflected in the endpoints table.

Edit an endpoint using the CLI

The example below demonstrates how to use the snapi endpoint update command to edit and update an endpoint. You will need to provide the following:

  • The project the endpoint is assigned.

  • The endpoint name.

Similar to the GUI, you can:

  • Change the number of instances by using the --instances option.

  • Change the description by using the --description option.

Example snapi endpoint update command
$  snapi endpoint update \
   --project <project-name> \
   --description <new-description> \
   --endpoint <endpoint-name> \
   --instances <number-of-instances>

Stop and restart an endpoint using the GUI

To free up Reconfigurable Dataflow Unit™ (RDU) resources in the platform, you may wish to stop an endpoint from running.

Stop an endpoint

Do the following to stop an existing endpoint from running.

  1. From an Endpoint window, click Stop as shown in Figure 10. An alert box will open..

    Stop endpoint
    Figure 10. Stop endpoint

  2. The alert box (Figure 11) displays a list of the active models in the endpoint that will stop running.

  3. As shown in Figure 11, click Yes to confirm that you want to stop the endpoint from running. Click Cancel to return the Endpoint window.

    Stop endpoint window
    Figure 11. This endpoint will stop running box

Restart an endpoint

From an Endpoint window, click Restart, as shown in Figure 12. The platform will restart the endpoint and will notify you that the endpoint is restarting.

Edit endpoint
Figure 12. Restart endpoint

The status of a restarted endpoint is displayed in the endpoint’s top Details bar.

Delete an endpoint using the GUI

Do the following to delete an existing endpoint.

  1. From an Endpoint window, click Delete as shown in Figure 13. An alert window box open.

    Stop endpoint
    Figure 13. Delete endpoint

    The alert box (Figure 14) displays a message that the endpoint will be deleted and no longer available. The alert also lists the endpoint that will be deleted.

  2. Click Yes to finalize deleting the endpoint, as shown in Figure 14. Click Cancel to return the Endpoint window.

    Delete alert window
    Figure 14. Delete alert window

Delete an endpoint using the CLI

The example below demonstrates how to use the snapi endpoint delete command to delete an endpoint. You will need to provide the following:

  • The project the endpoint is assigned.

  • The endpoint name.

Example snapi endpoint delete command
$  snapi endpoint delete \
   --project <project-name> \
   --endpoint <endpoint-name> \
Endpoint new-endpoint-snapi successfully marked delete.