Create and use endpoints

With the SambaStudio platform, you can generate predictions from your models on your data by deploying it to an endpoint. This document describes how to create, view, edit, stop, and delete endpoints, as well as manage endpoint API keys.

Create an endpoint

You can create endpoints to be used for prediction using the GUI or the CLI. Follow the instructions described in the corresponding sections to learn how.

Create an endpoint using the GUI

Follow the steps below to create an endpoint using the GUI to be used for prediction.

Base models do not support inference and cannot be deployed for endpoints. It is recommended to use Base models for training and not inference.

Create a new project or use an existing one.
From a project window, click New endpoint. The Add an endpoint window (Figure 1) will open.
In the Add an endpoint window, enter a name for the endpoint into the Endpoint name field, as shown in Figure 1.

The Share settings drop-down provides options for which tenant to share your endpoint, as shown in Figure 1.

Share with <current-tenant> allows the endpoint to be shared with the current tenant you are using, identified by its name in the drop-down.
Share with all tenants allows the endpoint to be shared across all tenants.

Endpoint will be shared with all users in <current-tenant> identifies that the endpoint will be shared with other users in the tenant you are using.

If the Endpoint will be shared with all users in <current-tenant> option is displayed, the Share with <current-tenant> and Share with all tenants options described above will not be available. Share with all tenants is an optional feature of SambaStudio. Please contact your administrator or SambaNova representative for more information.

Select the ML App from the ML App drop-down, as shown in Figure 1.
From the Select model drop-down, choose My models, Shared models, SambaNova models, or Select from Model Hub.
1. My models displays a list of models that you have previously added to the Model Hub.
2. Shared models displays a list of models that have been shared with the selected active tenant.
3. SambaNova models displays a list of models provided by SambaNova.
  
  Figure 1. Add an endpoint window
4. Select from Model Hub displays a window with a list of downloaded models that correspond to a selected ML App, as shown in Figure 2, or a list of all the downloaded models if an ML App is not selected. The list can be filtered by selecting options under Field of application, ML APP, Architecture, and Owner. Additionally, you can enter a term or value into the Search field to refine the model list by that input. Choose the model you wish to use and confirm your choice by clicking Use model.
  
  Figure 2. Select from Model Hub
Enter a value for the Number of instances to designate how many instances will be deployed, as shown in Figure 3. The Number of instances setting helps the endpoint scale to accommodate a high volume number of requests. You can adjust the value based on the expected volume number of requests and available resources.
Click Add an endpoint, as shown in Figure 3, to queue the endpoint for deployment. Once the endpoint is created, it will take a few minutes for the endpoint to be deployed.

Figure 3. Number of instances, add an endpoint

Your endpoint is deployed and will be available to generate predictions when the endpoint status displays Live.

Create an endpoint using the CLI

The example below demonstrates how to create an endpoint using the snapi endpoint create command. You will need to provide the following:

A project to assign the endpoint. Create a new project or use an existing one.
A new endpoint name.
The model to be used for the endpoint.
The number of instances to be used for the endpoint.

Example snapi create endpoint command

$ snapi endpoint create \
   --project <project-name> \
   --endpoint_name <endpoint-name> \
   --model-checkpoint <model-name> \
   --instances <number-of-instances>

View endpoint information

Once your endpoint has been deployed, you can view detailed information about it using the GUI or CLI.

View an endpoint using the GUI

The Endpoint window displays detailed information about your endpoint, as well as providing access to common functions. Follow the steps below to navigate to any Endpoint window. Additionally, you can view a list of Live endpoints from the Dashboard.

Click Projects from the left menu to navigate to the Projects window and view the current projects.
Click the Project that contains the endpoint you wish to view.

Figure 4. Project list

That Project window will open displaying the Endpoints table, which is located beneath the Jobs table. The Endpoints table provides the following information about each endpoint:
1. Name displays the name of the endpoint.
2. Model displays the associated model of the endpoint.
3. Instances displays the number of instances used by the endpoint.
4. Status displays the current status of the endpoint.
5. Date of creation displays the date and time the endpoint was created.
6. Owner displays the owner of the endpoint.
7. Actions provides additional interactions to the endpoint via a drop-down menu.
Click the endpoint you wish to view from the list to open the Endpoint window.

Figure 5. Endpoints table

The Endpoint window displays the following information:
1. The endpoint name is displayed in the top of the window and in the breadcrumb path at the top of the window.
2. The status of the endpoint is displayed in the Details panel top bar.
3. ML App displays the ML App that was selected when the endpoint was created.
4. Model/Checkpoint displays the endpoint’s associated model/checkpoint.
5. Instances displays the number of instances used by the endpoint.
6. Created on displays the date the endpoint was created.
7. Updated on displays the date the endpoint was last updated.
8. Owner displays the owner of the endpoint.
9. URL provides the URL of the endpoint to use programmatically to make requests, such as generating predictions. Use the copy icon to temporarily store the URL (copy) to your computer’s clipboard.
10. The API Keys table displays a list of API keys that can be used to make authenticated requests of the endpoint URL. See Endpoint API keys for information on generating and using API keys. The API Keys table displays the following information:
  1. KeyID displays the endpoint’s API key to be used for authenticated requests. Use the copy icon to temporarily store the API key (copy) to your computer’s clipboard.
  2. Description displays the API key’s unique description so that it can be easily identified.
  3. Created on displays the date the API key was created.
  4. Owner displays the owner of the API key.
  5. Status displays the current status of the API key.
    
    Figure 6. Endpoint window

View endpoint info using the CLI

The example below demonstrates how to use the snapi endpoint info command to view the info of an endpoint. You will need to provide the following:

The project the endpoint is assigned.
The endpoint name or ID.

Example snapi endpoint info command

$ snapi endpoint info \
    --project <project-name> \
    --endpoint <endpoint-name>

Endpoint API keys

An endpoint API key allows you to make authenticated requests of the endpoint URL, such as generating predictions. SambaStudio provides the ability to create multiple API keys for an endpoint. This allows organizations to easily distribute and manage endpoints to different entities and users.

API keys can be added, edited, and revoked using the GUI or the CLI. Follow the instructions described in the corresponding sections to learn how.

The endpoint API key and the platform API key have two distinct implementations.

The endpoint API key is a unique API key generated for each endpoint.
The platform API key is used as an authorization key for authenticated access to the platform. See SambaStudio resources for information about the platform API key.

Add an endpoint API key using the GUI

You can add additional API keys to an endpoint. Please be aware of the following when adding a new API key to an endpoint.

Endpoint API keys can be added by:
- The owner of the endpoint.
- An organization administrator (OrgAdmin) across all tenants.
- A tenant administrator (TenantAdmin) within their assigned tenant.
The platform creates an initial API key for each endpoint with the description of FirstKey.

Follow the steps below to add an API key to an endpoint.

From the API keys section of an endpoint window, click Add new.

Figure 7. API keys section

The Add new API key box will appear.
Enter a description into the Add description field for the endpoint’s new API key.
Click Add to complete the process and add the endpoint’s new API key.

Figure 8. Add new API key
The API keys table will display the new API key and the description you created for your endpoint.

Figure 9. API keys table with new API key

Edit an endpoint API key description using the GUI

An endpoint’s API key existing description can be edited. Please be aware of the following when editing an endpoint API key description.

The description of an endpoint API key can be edited by:
- The owner of the endpoint.
- An organization administrator (OrgAdmin) across all tenants.
- A tenant administrator (TenantAdmin) within their assigned tenant.
The initial API key, and its description of FirstKey, created for an endpoint by the platform, cannot be edited.

Follow the steps below to edit an endpoint API key description.

From the API keys section of an endpoint window, click the three dots in the API key you wish to edit and select Edit from the drop-down.

Figure 10. Edit API key drop-down

The Edit key description box will open.
Enter a new description into the Key description field.
Click Save to complete the operation and update the API key description.

Figure 11. Endpoint edit key description box

Revoke an endpoint API key using the GUI

Endpoint API keys can be revoked to prevent further usage. Please be aware of the following when revoking an endpoint API key.

Endpoint API keys can be revoked by:
- The owner of the endpoint.
- An organization administrator (OrgAdmin) across all tenants.
- A tenant administrator (TenantAdmin) within their assigned tenant.
The initial API key (FirstKey) created for an endpoint by the platform, cannot be revoked.
Once an endpoint API key is revoked it cannot be reactivated.
The API keys table will display the revoked API key with the status Revoked.
All users will lose access to a revoked API key.

Follow the steps below to revoke an endpoint API key.

From the API keys section of an endpoint window, click the three dots in the API key you wish to edit and select Revoke from the drop-down.

Figure 12. Revoke API key drop-down

The confirmation box will open with a statement describing the consequences of revoking the API key.
Click Revoke to confirm that you want to revoke the endpoint API key.

Figure 13. Endpoint revoke key confirmation box

List all endpoint API keys using the CLI

Similar to the endpoint API keys table of the GUI, the SambaStudio command-line interface (CLI) can display a list of API keys for an endpoint.

The example below demonstrates how to list all API keys for an endpoint using the snapi endpoint list-apikeys command. You will need to provide the endpoint name or ID.

Example snapi endpoint list-apikeys command

$ snapi endpoint list-apikeys \
   --endpoint <endpoint-name>

Add an endpoint API key using the CLI

Similar to the GUI, the SambaStudio command-line interface (CLI) can be used to add an API key to an endpoint. Please be aware of the following when adding a new API key to an endpoint.

Endpoint API keys can be added by:
- The owner of the endpoint.
- An organization administrator (OrgAdmin) across all tenants.
- A tenant administrator (TenantAdmin) within their assigned tenant.
The platform creates an initial API key for each endpoint with the description of FirstKey.

The example below demonstrates how to add an API key to an endpoint using the snapi endpoint add-apikey command. You will need to provide the following:

The endpoint name or ID.
A description to identify the new API key is recommended.

Example snapi endpoint add-apikey command

$ snapi endpoint add-apikey \
    --endpoint <endpoint-name> \
    --description <endpoint-description>

Edit an endpoint API key description using the CLI

An endpoint’s API key existing description can be edited using the SambaStudio command-line interface (CLI). Please be aware of the following when editing an endpoint API key description.

The description of an endpoint API key can be edited by:
- The owner of the endpoint.
- An organization administrator (OrgAdmin) across all tenants.
- A tenant administrator (TenantAdmin) within their assigned tenant.
The initial API key, and its description of FirstKey, created for an endpoint by the platform, cannot be edited.

The example below demonstrates how to edit an endpoint’s API key description using the snapi endpoint edit-apikey command. You will need to provide the following:

The endpoint name or ID.
The API key ID.
The existing description of the API key.

Example snapi endpoint edit-apikey command

$ snapi endpoint edit-apikey \
    --endpoint <endpoint-name> \
    --apikey <API-key-ID> \
    --description <new-description>

Revoke an endpoint API key using the CLI

An endpoint’s API key can be revoked using the SambaStudio command-line interface (CLI). Please be aware of the following when revoking an endpoint API key.

Endpoint API keys can be revoked by:
- The owner of the endpoint.
- An organization administrator (OrgAdmin) across all tenants.
- A tenant administrator (TenantAdmin) within their assigned tenant.
The initial API key (FirstKey) created for an endpoint by the platform, cannot be revoked.
Once an endpoint API key is revoked it cannot be reactivated.
The API key info will display the revoked API key with the status Revoked.
All users will lose access to a revoked API key.

The example below demonstrates how to revoke an endpoint’s API key using the snapi endpoint revoke-apikey command. You will need to provide the following:

The endpoint name or ID.
The API key ID.

Example snapi endpoint revoke-apikey command

$ snapi endpoint revoke-apikey \
    --endpoint <endpoint-name> \
    --apikey <API-key>

View endpoint API key info using the CLI

You can view detailed information about the API key of an endpoint using the SambaStudio command-line interface (CLI). This is useful to check the endpoint’s API key status (after revoking an API key) or confirm any changes you might have made to an endpoint API key (such as the description).

The example below demonstrates how to view an endpoint’s API key info using the snapi endpoint info-apikey command. You will need to provide the following:

The endpoint name or ID.
The API key ID.

Example snapi endpoint info-apikey command

$ snapi endpoint info-apikey \
    --endpoint <endpoint-name> \
    --apikey <API-key>

Edit an endpoint

You can edit an existing endpoint using the GUI or the CLI. Follow the instructions described in the corresponding sections to learn how.

Edit an endpoint using the GUI

Follow the steps below to edit an existing endpoint using the GUI.

From an Endpoint window, click Edit, as shown in Figure 14. Or, from the Endpoints table, click Edit from the Actions drop-down, as shown in Figure 15. The Update the endpoint window will open.

Figure 14. Edit an endpoint from the endpoint window

Figure 15. Edit an endpoint from the endpoints table
As shown in Figure 16, from the Update the endpoint window you can:
1. Enter a new description into the Description field.
2. Update the tenant to share your endpoint from the Share settings drop-down.
3. Choose a different model from the Select Model drop-down.
4. Update the Number of instances.
Click Edit endpoint to complete process and send the endpoint to the queue.

Figure 16. Update the endpoint

The endpoint’s new description, model, and number of instances will be reflected in the endpoints table.

Edit an endpoint using the CLI

The example below demonstrates how to use the snapi endpoint update command to edit and update an endpoint. You will need to provide the following:

The project the endpoint is assigned.
The endpoint name or ID.

Similar to the GUI, you can:

Change the number of instances by using the --instances option.
Change the description by using the --description option.

Example snapi endpoint update command

$  snapi endpoint update \
   --project <project-name> \
   --description <new-description> \
   --endpoint <endpoint-name> \
   --instances <number-of-instances>

Stop and restart an endpoint

To free up Reconfigurable Dataflow Unit™ (RDU) resources in the platform, you may wish to stop an endpoint from running.

Stop an endpoint using the GUI

Do the following to stop an existing endpoint from running using the GUI.

From an Endpoint window, click Stop as shown in Figure 17. An alert box will open..

Figure 17. Stop endpoint
The alert box (Figure 18) displays a list of the active models in the endpoint that will stop running.
As shown in Figure 18, click Yes to confirm that you want to stop the endpoint from running. Click Cancel to return the Endpoint window.

Figure 18. This endpoint will stop running box

Restart an endpoint using the GUI

From an Endpoint window, click Restart, as shown in Figure 19. The platform will restart the endpoint and will notify you that the endpoint is restarting.

Figure 19. Restart endpoint

The status of a restarted endpoint is displayed in the endpoint’s top Details bar.

Stop an endpoint using the CLI

The example below demonstrates how to use the snapi endpoint stop command to stop an endpoint from running. You will need to provide the following:

The project the endpoint is assigned.
The endpoint name or ID.

Example snapi endpoint stop command

$  snapi endpoint stop \
   --project <project-name> \
   --endpoint <endpoint-name>

Delete an endpoint

You can delete an existing endpoint using the GUI or the CLI. Follow the instructions described in the corresponding sections to learn how.

Delete an endpoint using the GUI

Do the following to delete an existing endpoint using the GUI.

From an Endpoint window, click Delete as shown in Figure 20. An alert window box open.

Figure 20. Delete endpoint

The alert box (Figure 21) displays a message that the endpoint will be deleted and no longer available. The alert also lists the endpoint that will be deleted.
Click Yes to finalize deleting the endpoint, as shown in Figure 21. Click Cancel to return the Endpoint window.

Figure 21. Delete alert window

Delete an endpoint using the CLI

The example below demonstrates how to use the snapi endpoint delete command to delete an endpoint. You will need to provide the following:

The project the endpoint is assigned.
The endpoint name or ID.

Example snapi endpoint delete command

$  snapi endpoint delete \
   --project <project-name> \
   --endpoint <endpoint-name> \
Endpoint new-endpoint-snapi successfully marked delete.