Run language example applications

From this tutorial you learn how to run a language application example on a SambaNova system and how to use application parameters.

BERT model overview

BERT (Bidirectional Encoder Representations from Transformers) is a machine learning model based on Transformers that was developed by Google in 2018.

The original BERT implementation has two models:

BERT Base: 12 encoders, 12 bidirectional self-attention heads, 110 million parameters, 768 dimension size
BERT Large: 24 encoders, 16 bidirectional self-attention heads, 340 million parameters, 1024 dimension size

For more information about BERT, including an illustration, see the original paper.

We use the Transformers library from Hugging Face to run BERT models on a DataScale system.

Our scripts include modifications of the original scripts that ensure that the model can run on SambaNova RDU chips.

The commands below are used to run BERT models on a DataScale system.

Prepare your environment

To prepare your environment, you:

Check your SambaFlow installation.
Make a copy of the tutorial files.
Download the data files from the internet.

Check your SambaFlow installation

You must have the sambaflow package installed to run this example and any of the tutorial examples.

To check if the package is installed, run this command:
- For Ubuntu Linux
  $ dpkg -s sambaflow
- For Red Hat Enterprise Linux
  $ rpm -qi sambaflow
Examine the output and verify that the SambaFlow version that you are running matches the documentation you are using.
If you see a message that sambaflow is not installed, contact your system administrator.

Create a copy of SambaFlow tutorials

If you haven’t done so already, create your own copy of all tutorial applications so you can experiment:

Copy the content of /opt/sambaflow/apps to a directory inside your home directory. For example:
```
$ mkdir $HOME/sambaflow-apps
$ cp -r /opt/sambaflow/apps/* $HOME/sambaflow-apps
```

If you copied the contents of opt/sambaflow/ops in an earlier release, make a new copy.

Prepare the dataset

Before you compile and run the model, you have to download the dataset. Follow these steps.

Create a subdirectory for the bert datasets in your home directory. In this example we use $HOME/datasets.
```
$ mkdir -p $HOME/datasets/bert
```
Set the DATADIR environment variable to point to this location.
```
$ export DATADIR=$HOME/datasets/bert
```
Download the datasets, which are part of SQuAD (Stanford Question Answering Dataset). SQuAD is a popular dataset used for training and evaluating question answering models, particularly those that leverage pre-trained language models like BERT.
```
$ wget -P $DATADIR https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json
$ wget -P $DATADIR https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json
```

Compile for training

The following set of command performs some setup and then it compiles the model for training. The arguments to compile fine-tune the output. Many of the arguments are specific to transformers_hook.py.

The output of the training run is a PEF file, transformers_hook.pef, which we pass in when we start a training run in the next step.

$ export OUTDIR=$HOME/app-test
$ export DATADIR=$HOME/data/bert
$ mkdir -p $DATADIR
$ python $HOME/sambaflow-apps/nlp/transformers_on_rdu/transformers_hook.py compile \
  --tokenizer_name bert-large-uncased \
  --model_name_or_path bert-large-uncased \
  --do_eval \
  --do_lower_case \
  --data_dir $DATADIR \
  --max_seq_length 384 \
  --per_device_train_batch_size 32 \
  --per_device_eval_batch_size 32 \
  -b 32 \
  --output_dir=${OUTDIR}/hf_output_squad_compile \
  --overwrite_output_dir \
  --seed 1206287 \
  --task_name squad \
  --module_name squad \
  --mac-human-decision $HOME/sambaflow-apps/nlp/transformers_on_rdu/human_decisions/compiler_configs/faster_compile.json \
  --mac-v2 \
  --cache_dir ${OUTDIR}/squad_cache \
  --pef transformers_hook \
  --output-folder=${OUTDIR}

Initiate a training run

To initiate a training run, we have to call run and pass in parameters including the PEF file, transformers_hook.pef.

$ export OUTDIR=$HOME/app-test
$ export DATADIR=$HOME/data/bert
$ python $HOME/sambaflow-apps/nlp/transformers_on_rdu/transformers_hook.py run \
  --model_name_or_path bert-large-uncased \
  --tokenizer_name bert-large-uncased \
  --do_train \
  --do_eval \
  --do_lower_case \
  --data_dir $DATADIR \
  -p ${OUTDIR}/transformers_hook/transformers_hook.pef \
  --max_seq_length 384 \
  --per_device_train_batch_size 32 \
  --per_device_eval_batch_size 32 \
  -b 32 \
  --output_dir=${OUTDIR}/hf_output_squad_run  \
  --overwrite_output_dir \
  --seed 1206287 \
  --task_name squad \
  --module_name squad \
  --learning_rate 3e-05  \
  --eval_steps 1000 \
  --num_train_epochs 0.2 \
  --cache_dir ${OUTDIR}/squad_cache

Compile for inference

Compiling for inference is separate from compiling for a training run, and we name the PEF file transformers_hook_inf.

$ export OUTDIR=$HOME/app-test
$ export DATADIR=$HOME/data/bert
$ python $HOME/sambaflow-apps/nlp/transformers_on_rdu/transformers_hook.py compile \
  --inference \
  --model_name_or_path bert-large-uncased \
  --tokenizer_name bert-large-uncased \
  --do_eval \
  --do_lower_case \
  --data_dir $DATADIR \
  --max_seq_length 384 \
  --per_device_eval_batch_size 32 \
  -b 32 \
  --output_dir=${OUTDIR}/hf_output_squad_inference_compile \
  --overwrite_output_dir \
  --seed 1206287 \
  --task_name squad \
  --module_name squad \
  --mac-human-decision /opt/sambaflow/apps/nlp/transformers_on_rdu/human_decisions/compiler_configs/faster_compile.json \
  --mac-v2 \
  --cache_dir ${OUTDIR}/squad_cache \
  --pef transformers_hook_inf \
  --output-folder=${OUTDIR}

Run for inference

To run for inference, we specify --inference as an argument to run and specify the PEF file that we compiled for inference (transformers_hook_inf).

Note that after compilation, the mode file becomes available in the checkpoint-500 directory, so we specify that when we run inference.

$ export OUTDIR=$HOME/app-test
$ export DATADIR=$HOME/data/bert
$ python $HOME/sambaflow-apps/nlp/transformers_on_rdu/transformers_hook.py run \
  --inference \
  --model_name_or_path ${OUTDIR}/hf_output_squad_run/checkpoint-500 \
  --do_eval \
  --do_lower_case \
  --data_dir $DATADIR \
  -p ${OUTDIR}/transformers_hook_inf/transformers_hook_inf.pef \
  --max_seq_length 384 \
  --per_device_eval_batch_size 32 \
  -b 32 \
  --output_dir=${OUTDIR}/hf_output_squad_inference \
  --overwrite_output_dir \
  --seed 1206287 \
  --task_name squad \
  --module_name squad \
  --learning_rate 3e-05  \
  --eval_steps 6000 \
  --tokenizer_name bert-large-uncased \
  --per_device_train_batch_size 32 \
  --cache_dir ${OUTDIR}/squad_cache