Batch inference

Batch inference is the process of generating predictions on a batch of observations. Within the platform, you can generate predictions on bulk data by creating a batch inference job.

Create a batch inference job using the GUI

Create a new batch inference using the GUI job by following the steps below.

  1. Create a new project or use an existing one.

  2. From a project window, select the New Job button. The Create a new job window will appear.

  3. Select Batch Inference under Create a new job.

  4. Enter a name for the job into the Job Name field.

  5. Select the ML App from the ML App drop-down.

    The ML App selected will refine the models displayed, by corresponding model type, in the Select model drop-down.

  6. From the Select model drop-down, choose My models, Shared models, SambaNova models, or Select from Model Hub.

    The available models displayed are defined by the previously selected ML App drop-down. If you wish to view models that are not related to the selected ML App, select Clear from the ML App drop-down. Selecting a model with the ML App drop-down cleared, will auto populate the ML App field with the correct and corresponding ML App for the model.

    1. My models displays a list of models that you have previously added to the Model Hub.

    2. Shared models displays a list of models that have been shared with the selected active tenant.

    3. SambaNova models displays a list of models provided by SambaNova.

      Batch inference model
      Figure 1. Batch inference job
    4. Select from Model Hub displays a window with a list of downloaded models that correspond to a selected ML App, or a list of all the downloaded models if an ML App is not selected. The list can be filtered by selecting options under Field of application, ML APP, Architecture, and Owner. Additionally, you can enter a term or value into the Search field to refine the model list by that input. Choose the model you wish to use and confirm your choice by clicking Use model.

      Model Hub
      Figure 2. Select from Model Hub
  7. From the Select dataset drop-down, choose My datasets, SambaNova datasets, or Select from datasets.

    1. My datasets displays a list of datasets that you have added to the platform and can be used for a selected ML App.

    2. SambaNova datasets displays a list of downloaded SambaStudio provided datasets that correspond to a selected ML App.

    3. Select from datasets displays the Dataset Hub window with a detailed list of downloaded datasets that can be used for a selected ML App. The My datasets and SambaNova checkboxes filter the dataset list by their respective group. The ML App drop-down filters the dataset list by the corresponding ML App. Choose the dataset you wish to use and confirm your choice by clicking Use dataset.

      Dataset Hub
      Figure 3. Dataset hub
  8. In the Relative dataset folder/file path field, specify the relative path from storage to the folder that contains the data for batch inference.

  9. Set the inference settings to to optimize your batch inference job for the input data or use the default values. Expand the Inference settings pane by clicking the blue double arrows to set and adjust the settings.

  10. Click Run job to start the batch inference.

Batch inference jobs are long running jobs. The time they take to run to completion is heavily dependent on the task and the number of input samples.

Batch Inference
Figure 4. Create a batch inference job

Create a batch inference job using the CLI

The example below demonstrates how to create a batch inference job using the snapi job create command. You will need to specify the following:

  • A project to assign the job. Create a new project or use an existing one.

  • A name for your new job.

  • batch_predict as the job type. This designates the job to be a batch inference job.

  • The model to be used for the batch inference job.

  • The dataset you wish to use for the batch inference job.

    The dataset must be compatible with the model you choose.

Example batch inference snapi job create command
$  snapi job create \
   --project <project-name> \
   --job <your-new-job-name> \
   --type batch_predict \
   --model-checkpoint <model-name> \
   --dataset <dataset-name>

Quit or delete a batch inference job

Follow the steps below to quit or delete a batch inference job.

Option 1: From the detail window

  1. Navigate to the Batch Inference detail window from Dashboard or Projects.

    1. Click the Quit button to stop the job from running. The confirmation box will open.

    2. Click the Delete button to quit and remove the job from the platform. All job related data and export history will permanently be removed from the platform. The confirmation box will open.

  2. In the confirmation box, click the Yes button to confirm that you want to quit or delete the job.

Quit Job Window
Figure 5. Quit or delete a batch inference job from the job window

Option 2: From the job list

  1. Navigate to the Jobs table from the Dashboard. Alternatively, the Jobs table can be accessed from the job’s associated project window.

  2. Click the three dots under the Actions column to display the drop-down menu and available actions for the selected job.

    1. Click the Delete button to quit and remove the job from the platform. All job related data and export history will permanently be removed from the platform. The confirmation box will open.

    2. Click the Quit button to stop the job from running. The confirmation box will open.

  3. In the confirmation box, click the Yes button to confirm that you want to quit or delete the job.

Quit Job List
Figure 6. Quit or delete a batch inference job from the job list

Access results

After your batch inference job is complete, you can access the results as described below. Results of a completed batch inference job are enclosed in a single file that contains predictions for all input samples.

Download results to your local machine

Follow the steps below to download the results of your batch inference job to your local machine.

  1. Navigate to the Batch Inference detail window from Dashboard or Projects.

  2. Click Download results to download a compressed file of your results. The file will be downloaded to the location configured by your browser.

Downloads are limited to a maximum file size of 2GB. Downloading results larger than 2GB will fail to complete. Use the access from NFS method if the file size exceeds 2GB.

Batch inference detail window
Figure 7. Batch inference detail window

Upload to AWS S3

Follow the steps below to upload results from a batch inference job to a folder in an AWS S3 bucket.

  1. Navigate to the Batch Inference detail window from Dashboard or Projects.

  2. Click the Upload results to AWS button. The Upload to AWS box will open. Provide the following information:

    1. In the Bucket field, input the name of your S3 Bucket.

    2. Input the relative path to the dataset in the S3 bucket into the Folder field. This folder should include the required dataset files for the task (for example, the labels, training, and validation files).

    3. In the Access key ID field, input the unique ID provided by AWS IAM to manage access.

    4. Enter your Secret access key into the field. This allows authentication access for the provided Access Key ID.

    5. Enter the AWS Region that your S3 bucket resides into the Region field.

There is no limit to the number of times results can be uploaded to AWS S3 buckets, but only one upload is allowed at a time.

Upload results to AWS
Figure 8. Upload results to AWS

Access from NFS

You can access the results from a batch inference job directly from the NFS and copy it to a location of your choice. It is recommended to use this method to download results larger than 2GB to your local machine. Follow the steps below to access results from NFS.

  1. Navigate to the Batch Inference detail window from Dashboard or Projects.

  2. Scroll to the Details card of the detail window. The details card provides the path to the results file on the mounted NFS.

Batch inference nfs
Figure 9. Example batch inference NFS results path

Evaluate the job using the GUI

Navigate to the Batch Inference detail window from Dashboard or Projects during the job run (or after its completion) to view job information.

View information

You can view the following information about your batch inference job.

Model

Displays the model name and architecture used for training.

Dataset

Displays the dataset used, including its size.

Details & Inference

Displays the number of RDUs utilized and job submission time stamp. Click More to view a list of the inference settings used during training. Click Less to hide the inference settings list.

Batch inference settings
Figure 10. Expanded Details & Inference Settings
Progress bar

The progress bar displays the state of the job as well as the percentage completed of the batch inference run.

Details

The details card provides the path to the results file on the mounted NFS.

Exports table

Displays a list of when and where your batch inference access results were exported.

Batch exports table
Figure 11. Exports table

View and download logs using the GUI

The Logs section allows you to preview and download logs of your batch inference job. Logs can help you track progress, identify errors, and determine the cause of potential errors.

Logs can be visible in the platform earlier than other data, such as information and job progress.

  1. From the Preview drop-down, select the log file you wish to preview.

    1. The Preview window displays the latest 50 lines of the log.

    2. To view more than 50 lines of the log, use the Download all feature to download the log file.

  2. Click Download all to download a compressed file of your logs. The file will be downloaded to the location configured by your browser.

Logs
Figure 12. Logs

View logs using the CLI

Similar to viewing logs using the GUI, you can use the SambaNova API (snapi) to preview and download logs of your training session.

View the job log file names

The example below demonstrates the snapi job list-logs command. Use this command to view the job log file names of your training job. This is similar to using the Preview drop-down menu in the GUI to view and select your job log file names. You will need to specify the following:

  • The project that contains, or is assigned to, the job you wish to view the job log file names.

  • The name of the job you wish to view the job log file names.

Example snapi job list-logs command
$ snapi job list-logs \
   --project <project-name> \
   --job <job-name>
compile-05a8d29f-aeb2-4ca3-8f1e-7c3b35f8d3a2-5t4v8-runner.log
compile-05a8d29f-aeb2-4ca3-8f1e-7c3b35f8d3a2-5t4v8-model.log
infer-12b3a844-826b-408b-8aae-e05e3149ab32-lgxgq-runner.log
infer-12b3a844-826b-408b-8aae-e05e3149ab32-lgxgq-model.log

Run snapi job list-logs --help to display additional usage and options.

Preview a log file

After you have viewed the log file names for your training job, you can use the snapi job preview-log command to preview the logs corresponding to a selected log file. The example below demonstrates the snapi job preview-log command. You will need to specify the following:

  • The project that contains, or is assigned to, the job you wish to preview the job log file.

  • The name of the job you wish to preview the job log file.

  • The job log file name you wish to preview its logs. This file name is returned by running the snapi job list-logs command, which is described above.

Example snapi job preview-log command
$ snapi job preview-log \
   --project <project-name> \
   --job <job-name> \
   --file infer-12b3a844-826b-408b-8aae-e05e3149ab32-lgxgq-runner.log
2023-06-21 14:24:55  -  INFO  -  621b3e76-34d6-4e19-8751-c38e473805f9  -  Runner starting...

2023-06-21 14:24:55  -  INFO  -  621b3e76-34d6-4e19-8751-c38e473805f9  -  Runner successfully started

2023-06-21 14:24:55  -  INFO  -  621b3e76-34d6-4e19-8751-c38e473805f9  -  Received new infer request

2023-06-21 14:24:55  -  INFO  -  621b3e76-34d6-4e19-8751-c38e473805f9  -  Connecting to modelbox at localhost:50061

2023-06-21 14:25:05  -  INFO  -  621b3e76-34d6-4e19-8751-c38e473805f9  -  Running batch inference pipeline

2023-06-21 14:25:05  -  INFO  -  621b3e76-34d6-4e19-8751-c38e473805f9  -  initializing checkpoint path for modelbox:0

2023-06-21 14:25:29  -  INFO  -  621b3e76-34d6-4e19-8751-c38e473805f9  -  initializing metrics for modelbox:0

2023-06-21 14:25:29  -  INFO  -  621b3e76-34d6-4e19-8751-c38e473805f9  -  Staging the dataset

2023-06-21 14:25:33  -  INFO  -  621b3e76-34d6-4e19-8751-c38e473805f9  -  Running inference on pipeline

2023-06-21 14:25:33  -  INFO  -  621b3e76-34d6-4e19-8751-c38e473805f9  -  Running inference pipeline stage 0

2023-06-21 14:25:33  -  INFO  -  621b3e76-34d6-4e19-8751-c38e473805f9  -  Running inference for modelbox

2023-06-21 14:26:50  -  INFO  -  621b3e76-34d6-4e19-8751-c38e473805f9  -  Saving outputs from /shared-volume/results/modelbox/ for node

2023-06-21 14:26:50  -  INFO  -  621b3e76-34d6-4e19-8751-c38e473805f9  -  Saving results file /shared-volume/results/modelbox/test.csv to test.csv

Run snapi job preview-log --help to display additional usage and options.

Download the logs

Use the snapi download-logs command to download a compressed file of your training job’s logs. The example below demonstrates the snapi download-logs command. You will need to provide the following:

  • The project that contains, or is assigned to, the job you wish to download the compressed log file.

  • The name of the job you wish to download the compressed log file.

Example snapi download-logs command
$ snapi job download-logs \
   --project <project-name>> \
   --job <job-name>
Successfully Downloaded: <job-name> logs

The default destination for the compressed file download is the current directory. To specify a destination directory, use the --dest option. Run snapi job download-logs --help for more information and to display additional usage and options.