Use Python virtual environments
Python virtual environments allow you to install and manage Python packages separately from the system-level Python installation. Some of the advantages of using virtual environments include:
Avoiding package conflicts. You can install and manage Python packages independently of each other. This means that you can install different versions of the same package in different virtual environments without causing conflicts or compatibility issues.
Dependency management. Virtual environments make it easy to manage dependencies for your projects. You can create a
requirements.txtfile that lists all the required packages and their versions, making it easier to install the same dependencies on other machines.
Isolation. Virtual environments provide a sandboxed environment. This reduces the risk of accidentally installing or modifying packages that are needed by other applications or users on the same machine.
SambaNova engineers prepared virtual environments for example applications that are shipped with SambaFlow.
The application’s directory includes:
A pre-installed virtual environment that you can use to test the model without installing anything yourself.
requirements.txtfile that you can use if you want to create your own virtual environment and install the packages required by the model.
|Each virtual environment provided by SambaNova includes only the packages that are necessary to run the corresponding example model. In many cases you need additional packages for your application, for example data processing packages.|
To customize the virtual environment provided by SambaNova, follow these steps, discussed in detail in Prepare the virtual environment for the model:
Create your own virtual environment.
Install the packages required by the example model (and specified in
Add other packages that you need.
This doc page explains how to set up and customize a virtual environment using Python
venv. You can instead use another virtual environment tool.
Assume you want to use a SambaNova example model. In addition to the required packages, you want to add other Python packages. Let’s look at the
/opt/sambaflow/apps/image/segmentation example and assume you want to add data processing packages for geophysics datasets.
Find the application’s code directory and list available models. For example, here’s how you look at image processing examples:
$ ls /opt/sambaflow/apps/image
classification object_detection segmentation segmentation_3d
Copy the application code for the
segmentationexample into your home directory (or any other directory where you have write access).
$ mkdir -p $HOME/sambaflow-apps/image $ cp -a /opt/sambaflow/apps/image/segmentation $HOME/sambaflow-apps/image/
Create a directory to hold all your virtual environments (if it doesn’t exist). We recommend creating it in
$HOME/.virtualenvsbecause it’s used by tools like virtualenvwrapper by default.
$ mkdir -p $HOME/.virtualenvs
Create a virtual environment using the
Use a meaningful name for your virtual environment, e.g. the name of the model. Don’t use the generic name
venvto avoid problems later.
$ python -m venv --system-site-packages $HOME/.virtualenvs/segmentation
Activate the virtual environment that you’ve just created.
$ source $HOME/.virtualenvs/segmentation/bin/activate
$ pip install --upgrade pip
Install the required packages using the application’s
$ pip install -r $HOME/sambaflow-apps/image/segmentation/requirements.txt
Verify that you can compile the unmodified application code.
$ cd $HOME/sambaflow-apps/image/segmentation $ python compile.py compile
Install additional packages that you need. For example, you need the
segyiopackage to pre-process geophysics datasets.
$ pip install segyio
Now you are ready to work on your model.
After you are done with your model or when you want to switch to some other tasks, don’t forget to deactivate your virtual environment.