Skip to content

Deployment Workflow

We now explore the deployment workflow which involves transferring compiled bitfiles (zipped artifacts) onto the FPGA card. This involves a stack of three Docker images:

  1. esnet-smartnic-fw: This image is specific to your specific bitfiles (artifacts). You will need to rebuild it each time you modify the bitfiles. The rebuilding process becomes significantly faster after the first time.

  2. smartnic-dpdk-docker: This image is bitfile-independent and only needs to be built once. It is used to interact with the FPGA card. You can tar this image and transfer it to your runtime environment, provided that the host system where it was created matches the CPU architecture of your runtime environment. This Docker image provides the DPDK and pktgen tools.

  3. xilinx-labtools-docker: Similar to the smartnic-dpdk-docker image, this Docker image is bitfile-independent and needs to be constructed only once. It complements the functionality of the FPGA card and is transferable to your runtime environment, given the aforementioned CPU architecture compatibility. This Docker image gives you access to Vivado Lab (no license needed). Vivado Lab is the software you’ll use to load new programs onto your FPGA, so it’s a crucial part of the setup.

All three Docker images (along with the corresponding configuration files) are needed to program an FPGA card using the ESnet framework. This deployment process can be replicated on multiple FPGA cards on different hosts as long as the three containers and the sn-stack/ folder are available on the new host.

Once the deployment is complete, you gain access to a powerful set of tools for your experiments:

  • DPDK (Data Plane Development Kit): DPDK enables you to bypass your OS kernel and interact directly with the FPGA card. This allows for high-performance packet processing and efficient data plane operations.

  • pktgen: pktgen is a DPDK application that facilitates the transmission of packets to and from the FPGA SmartNIC through its 2x100 Gbps ports. It also enables communication with the slice host using the PCIe bus. pktgen is a valuable tool for testing and analyzing network performance.

  • ESnet CLI Tools: These command-line tools provide control over the QDMA queues, access to probe counters for packet statistics, management of the control plane rules for your P4 logic, and remapping of egress ports to suit the requirements of your specific experiment.

Provenance: esnet-smartnic-fw

Setting up the build environment

See the SmartNIC firmware build environment section in the Configuration Workflow.

Create ‘smartnic-dpdk-docker’ Image

Provenance: smartnic-dpdk-docker

  • Clone the repository: 
    git clone https://github.com/esnet/smartnic-dpdk-docker.git
git checkout a52dba351cce4ff27b323339d27179cc334e5651
  • Installing git submodules 
    git submodule update --init --recursive

  • Building the smartnic-dpdk-docker container 

    docker build --pull -t smartnic-dpdk-docker:${USER}-dev .
docker image ls

  • You should see an image called smartnic-dpdk-docker with tag ${USER}-dev.

Alternatively, you can tar this image and transfer it to your runtime environment, provided that the host system where it was created matches the CPU architecture of the environment.

Create ‘xilinx-labtools-docker’ Image

Provenance: xilinx-labtools-docker

  • Clone the repository: 

    git clone https://github.com/esnet/xilinx-labtools-docker.git
cd xilinx-labtools-docker
git checkout 87f55f2beb31fef639c0847ab30dee8ecfcff42e

  • Download the Xilinx Labtools Installer

  • Open a web browser to this page: https://www.xilinx.com/support/download/index.html/content/xilinx/en/downloadNav/vivado-design-tools/2023-1.html
  • Under the Vivado Lab Solutions - 2023.1 section
    • Download Vivado 2023.1: Lab Edition - Linux
    • Save the file as exactly: Xilinx_Vivado_Lab_Lin_2023.1_0507_1903.tar.gz

  • Move the file into the vivado-installer directory in this repo

  • Download the Alveo Smartnic Satellite Controller Update Tool

  • Open a web browser to this page: https://support.xilinx.com/s/article/73654
  • At the bottom of the page under the Files section
    • Download loadsc_v2.3.zip
    • Save the file as exactly loadsc_v2.3.zip

  • Move the file into the sc-fw-downloads directory in this repo

  • Verify that you have the downloaded files all in the right places 

    $ tree
.
├── Dockerfile
├── entrypoint.sh
├── LICENSE.md
├── patches
│   └── vivado-2023.1-postinstall.patch
├── README.md
├── sc-fw-downloads
│   └── loadsc_v2.3.zip   <------------------------------------- put the zip file here
├── sc-fw-extra
├── sources.list.focal
└── vivado-installer
    ├── install_config_lab.2023.1.txt
    └── Xilinx_Vivado_Lab_Lin_2023.1_0507_1903.tar.gz   <------- put the installer here

  • Building the xilinx-labtools container

docker build --pull -t xilinx-labtools-docker:${USER}-dev .
docker image ls
  • You should see an image called xilinx-labtools-docker with tag ${USER}-dev.

Alternatively, you can tar this image and transfer it to your runtime environment, provided that the host system where it was created matches the CPU architecture of the environment.

Building a new firmware image

Provenance: esnet-smartnic-fw

Clone the esnet-smartnic-fw repository

git clone https://github.com/esnet/esnet-smartnic-fw.git
cd esnet-smartnic-fw
git checkout579a23b7f55c289d241ab8ee6dc00dcdb1078f51

Git Submodules

Ensure that all submodules have been pulled.

git submodule init
git submodule update

Install SmartNIC Hardware Build Artifact

The firmware build depends on the result of a smartnic hardware (FPGA) build. This file must be available prior to invoking the firmware build.

This file will be called artifacts.<board>.<app_name>.0.zip and should be placed in the sn-hw directory in your source tree before starting the firmware build.

Set up your .env file for building a new firmware image

The .env file tells the build about its inputs and outputs.

There is an example.env file in top level directory of the esnet-smartnic-fw repo that will provide documentation and examples for the values you need to set.

cd $(git rev-parse --show-toplevel)
cp example.env .env

Since the values in the .env file are used to locate the correct hardware artifact, you will need to (at least) set these values in the .env file to match the exact naming of the .zip file you placed in the previous step: 

SN_HW_BOARD=<board>
SN_HW_APP_NAME=<app_name>
SN_HW_VER=0

Build the firmware

The firmware build creates a docker container with everything needed to interact with your FPGA image. Without any parameters, the newly built firmware container will be named/tagged esnet-smartnic-fw:${USER}-dev and will be available only on the local system. 

cd $(git rev-parse --show-toplevel)
./build.sh

Optionally you can use any alternative name by specifying the full container URI as an optional parameter to the build script like this. Using a fully specified URI here can be useful if you are planning to push the newly built container to a remote docker registry. 

./build.sh wharf.es.net/ht/esnet-smartnic-fw:${USER}-dev

The build script also automatically customizes the sn-stack/.env file to refer exactly to the firmware image that you just built.

The entire sn-stack directory will need to be transferred to the runtime system.

cd $(git rev-parse --show-toplevel)
zip -r artifacts.esnet-smartnic-fw.package.0.zip sn-stack