OPAE Installation Guide¶
How to download the OPAE SDK¶
OPAE SDK releases are available on GitHub. Source code for the OPAE DFL device driver for Linux is also available on GitHub.
Install the Fedora¶
Download the Fedora (x86_64 version) installation file in fedora, and install the Fedora in yourserver. You can choose Fedora Workstation or Fedora server.
Build the kernel and DFL drivers¶
For building the OPAE kernel and kernel driver, the kernel development environment is required. So before you build the kernel, you must install the required packages. Run the following commands:
$ sudo dnf install gcc gcc-c++ make kernel-headers kernel-devel elfutils-libelf-devel ncurses-devel openssl-devel bison flex
Download the OPAE upstream kernel tree from github, for example download from fpga-ofs-dev-5.15-lts branch.
$ git clone https://github.com/OPAE/linux-dfl.git -b fpga-ofs-dev-5.15-lts
Configure the kernel.
$ cd linux-dfl
$ cp /boot/config-`uname -r` .config
$ cat configs/dfl-config >> .config
$ echo 'CONFIG_LOCALVERSION="-dfl"' >> .config
$ echo 'CONFIG_LOCALVERSION_AUTO=y' >> .config
$ sed -i -r 's/CONFIG_SYSTEM_TRUSTED_KEYS=.*/CONFIG_SYSTEM_TRUSTED_KEYS=""/' .config
$ sed -i '/^CONFIG_DEBUG_INFO_BTF/ s/./#&/' .config
$ echo 'CONFIG_DEBUG_ATOMIC_SLEEP=y' >> .config
$ make olddefconfig
Compile and install the new kernel.
$ make -j $(nproc)
$ sudo make modules_install -j $(nproc)
$ sudo make install
Build linux DFL Kernel instructions please also refer to: https://github.com/OPAE/linux-dfl/wiki/Build-the-linux-dfl-kernel
When install finished, reboot your system. When the system login again, verify the kernel version is correct. For example:
[figo@localhost linux-dfl]$ uname -a
Linux localhost.localdomain 5.15.lts-dfl-g73e16386cda0 #6 SMP Mon Jun 13 21:21:31 -04 2022 x86_64 x86_64 x86_64
And also you can check the OPAE dfl drivers have auto-loaded.
[figo@localhost linux-dfl]$ lsmod | grep fpga
ifpga_sec_mgr 20480 1 intel_m10_bmc_secure
fpga_region 20480 3 dfl_fme_region,dfl_fme,dfl
fpga_bridge 24576 4 dfl_fme_region,fpga_region,dfl_fme,dfl_fme_br
fpga_mgr 16384 4 dfl_fme_region,fpga_region,dfl_fme_mgr,dfl_fme
[figo@localhost linux-dfl]$ lsmod | grep dfl
dfl_eth_group 36864 0
dfl_fme_region 20480 0
dfl_emif 16384 0
dfl_n3000_nios 20480 0
dfl_fme_br 16384 0
dfl_fme_mgr 20480 1
dfl_fme 49152 0
dfl_afu 36864 0
dfl_pci 20480 0
dfl 40960 7 dfl_pci,dfl_fme,dfl_fme_br,dfl_eth_group,dfl_n3000_nios,dfl_afu,dfl_emif
fpga_region 20480 3 dfl_fme_region,dfl_fme,dfl
fpga_bridge 24576 4 dfl_fme_region,fpga_region,dfl_fme,dfl_fme_br
fpga_mgr 16384 4 dfl_fme_region,fpga_region,dfl_fme_mgr,dfl_fme
Build the OPAE-SDK¶
Before you build the OPAE SDK, you must install the required packages. Run the following commands:
Rocky Linux 8.5¶
# dnf install -y 'dnf-command(config-manager)'
# dnf config-manager --set-enabled powertools
# dnf install -y epel-release
# dnf check-update
# dnf upgrade -y
# dnf install -y python3 python3-pip python3-devel python3-jsonschema python3-pyyaml python3-pybind11 git gcc gcc-c++ make cmake libuuid-devel json-c-devel hwloc-devel tbb-devel cli11-devel spdlog-devel libedit-devel systemd-devel rpm-build rpmdevtools pybind11-devel yaml-cpp-devel libudev-devel linuxptp
# python3 -m pip install jsonschema virtualenv pyyaml
Fedora¶
# dnf check-update
# dnf upgrade -y
# dnf install -y python3 python3-pip python3-devel python3-jsonschema python3-pyyaml python3-pybind11 git gcc g++ make cmake libuuid-devel json-c-devel hwloc-devel tbb-devel libedit-devel rpm-build rpmdevtools pybind11-devel yaml-cpp-devel libudev-devel cli11-devel spdlog-devel linuxptp
# pip3 install jsonschema virtualenv pyyaml
Ubuntu 20.04¶
# apt-get update
# apt-get upgrade -y
# apt-get install -y python3 python3-pip python3-dev git gcc g++ make cmake uuid-dev libjson-c-dev libhwloc-dev libtbb-dev libedit-dev libudev-dev linuxptp pandoc devscripts debhelper doxygen
# pip3 install jsonschema virtualenv pyyaml pybind11
RHEL 8.2¶
Register and enable Red Hat subscription to install any packages on the system.
# subscription-manager register --proxy=PROXY --username=USER --password=PASSWORD --auto-attach
Set the RHEL version and install packages. Set proxy name and port number
# subscription-manager release --set=8.2 --proxy proxy-name.com:port number
# subscription-manager repos --enable codeready-builder-for-rhel-8-x86_64-rpms
# dnf upgrade -y
# dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
# dnf install -y python3 python3-pip python3-devel gdb vim git gcc gcc-c++ make cmake libuuid-devel rpm-build systemd-devel nmap
# dnf install -y python3-jsonschema json-c-devel tbb-devel rpmdevtools libcap-devel
# dnf check-update || true
# dnf install -y spdlog-devel cli11-devel python3-pyyaml python3-pybind11 hwloc-devel libedit-devel
# python3 -m pip install --user jsonschema virtualenv pudb pyyaml
Create opae-sdk packages¶
Download the OPAE-SDK source code from github. For example, download from Master branch.
$ git clone https://github.com/OPAE/opae-sdk.git
Compile and build the OPAE-SDK RPMs (Fedora, Rocky, RHEL 8.2).
$ cd opae-sdk/packaging/opae/rpm
$ ./create fedora
After a successful compile, there are 3 rpm packages generated (Fedora, Rocky, RHEL8.2). For example:
opae-2.1.0-1.fc34.x86_64.rpm
opae-devel-2.1.0-1.fc34.x86_64.rpm
opae-extra-tools-2.1.0-1.fc34.x86_64.rpm
Compile and build the OPAE-SDK deb packages (Ubuntu 22.04).
$ cd opae-sdk/packaging/opae/deb
$ ./create
After a successful compile, there are 3 deb packages generated (Ubuntu 22.04). For example:
opae_2.1.1-1_amd64.deb
opae-devel_2.1.1-1_amd64.deb
opae-extra-tools_2.1.1-1_amd64.deb
OPAE SDK installation with rpm/deb packages¶
The rpm packages generated in the previous step can be installed using these commands:
$ sudo dnf install ./*.rpm
The deb packages generated in the previous step can be installed using these commands:
$ sudo dpkg -i ./*.deb
When you installed the rpms, you can run fpgainfo command to check the FPGA FME infomation. For example:
[figo@localhost install_guide]$ fpgainfo fme
Board Management Controller, MAX10 NIOS FW version: D.2.1.24
Board Management Controller, MAX10 Build version: D.2.0.7
//****** FME ******//
Object Id : 0xEF00000
PCIe s:b:d.f : 0000:08:00.0
Device Id : 0x0B30
Socket Id : 0x00
Ports Num : 01
Bitstream Id : 0x2300011001030F
Bitstream Version : 0.2.3
Pr Interface Id : f3c99413-5081-4aad-bced-07eb84a6d0bb
Boot Page : user
To uninstall the OPAE rpms, you can use this commands
$ dnf list installed | grep opae
$ sudo dnf remove opae*.x86_64
To uninstall the OPAE deb, you can use this commands
$ dpkg -l | grep opae
$ dpkg -r opae-extra-tools:amd64
$ dpkg -r opae-devel:amd64
$ dpkg -r opae
FPGA Device Access Permissions¶
Access to FPGA accelerators and devices is controlled using file access permissions on the
Intel® FPGA device files, /dev/dfl-fme.* and /dev/dfl-port.*, as well as to the files reachable through /sys/class/fpga_region/.
In order to allow regular (non-root) users to access accelerators, you need to grant them read and write permissions on /dev/dfl-port.* (with * denoting the respective socket, i.e. 0 or 1). E.g.:
$ sudo chmod a+rw /dev/dfl-port.0
Memlock limit¶
Depending on the requirements of your application, you may also want to increase the maximum amount of memory a user process is allowed to lock. The exact way to do this depends on your Linux distribution.
You can check the current memlock limit using
$ ulimit -l
A way to permanently remove the limit for locked memory for a regular user is to add the following lines to your /etc/security/limits.conf:
user1 hard memlock unlimited
user1 soft memlock unlimited
This removes the limit on locked memory for user user1. To remove it for
all users, you can replace user1 with *:
* hard memlock unlimited
* soft memlock unlimited
Note that settings in the /etc/security/limits.conf file don’t apply to services. To increase the locked memory limit for a service you need to modify the application’s systemd service file and add the line:
[Service]
LimitMEMLOCK=infinity
Hugepage Settings¶
Users need to configure system hugepages to reserve 2MB-hugepages or 1GB-hugepages. For example, the ‘hello_fpga’ sample requires several 2MB-hugepages. And the fpgadiag tool requires several 1GB-hugepages.
The command below reserves 20 2M-hugepages:
$ sudo sh -c 'echo 20 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages'
The command below reserves 4 1GB-hugepages:
$ sudo sh -c 'echo 4 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages'
For x86_64 architecture processors, user can use following command to find out avaiable hugepage sizes:
$ grep pse /proc/cpuinfo | uniq
flags : ... pse ...
If this commands returns a non-empty string, 2MB pages are supported.
$ grep pse /proc/cpuinfo | uniq
flags : ... pdpe1gb ...
If this commands returns a non-empty string, 1GB pages are supported.