Open Programmable Accelerator Engine (OPAE) Linux Device Driver Architecture¶
The OPAE FPGA Linux Device Driver provides interfaces for user-space applications to configure, enumerate, open, and access FPGA accelerators on platforms equipped with Intel FPGA solutions. The OPAE FPGA driver also enables system-level management functions such as FPGA reconfiguration and virtualization.
The Linux Operating System treats the FPGA hardware as a PCIe* device. A predefined data structure, Device Feature List (DFL), allows for dynamic feature discovery in an Intel FPGA solution.
The Linux Device Driver implements PCIe Single Root I/O Virtualization (SR-IOV) for the creation of Virtual Functions (VFs). The device driver can release individual accelerators for assignment to virtual machines (VMs).
FPGA Management Engine (FME)¶
The FPGA Management Engine provides error reporting, reconfiguration, performance reporting, and other infrastructure functions. Each FPGA has one FME which is always accessed through the Physical Function (PF). The Intel Xeon® Processor with Integrated FPGA also performs power and thermal management. These functions are not available on the Intel Programmable Acceleration Card (PAC).
User-space applications can acquire exclusive access to the FME using
and release it using
close(). Device access may be managed by standard Linux
interfaces and tools.
If an application terminates without freeing the FME or Port resources, Linux closes all file descriptors owned by the terminating process, freeing those resources.
A Port represents the interface between two components:
The FPGA Interface Manager (FIM) which is part of the static FPGA fabric
The Accelerator Function Unit (AFU) which is the partially reconfigurable region
The Port controls the communication from software to the AFU and makes features such as reset and debug available.
Accelerator Function Unit (AFU)¶
An AFU attaches to a Port. The AFU provides a 256 KB memory mapped I/O (MMIO) region for accelerator-specific control registers.
open()on the Port device to acquire access to an AFU associated with the Port device.
close()on the Port device to release the AFU associated with the Port device.
mmap()on the Port device to map accelerator MMIO regions.
Partial Reconfiguration (PR)¶
Use PR to reconfigure an AFU from a bitstream file. Successful reconfiguration has the following requirement:
You must generate the reconfiguration AFU for the exact FIM. The AFU and FIM are compatible if their interface IDs match. You can verify this match by comparing the interface ID in the bitstream header against the interface ID that is exported by the driver in sysfs.
In all other cases PR fails and may cause system instability.
Platforms that support 512-bit Partial Reconfiguration require binutils >= version 2.25.
Close any software programs accessing the FPGA, including those running in a virtualized host before initiating PR. For virtualized environments, the recommended sequence is as follows:
Unload the driver from the guest
Release the VF from the guest
NOTE: Releasing the VF from the guest while an application on the guest is still accessing its resources may lead to VM instabilities. We recommend closing all applications accessing the VF in the guest before releasing the VF.
Assign the VF to the guest
Load the driver in the guest
To enable accelerator access from applications running on a VM, create a VF for the port using the following process:
Release the Port from the PF using the associated ioctl on the FME device.
Use the following command to enable SR-IOV and VFs. Each VF can own a single Port with an AFU. In the following command, N is the number of Port released from the PF.
echo N > $PCI_DEVICE_PATH/sriov_numvfs
NOTE: The number, 'N', cannot be greater than the number of supported VFs. This can be read from $PCI_DEVICE_PATH/sriov_totalvfs.
Pass the VFs through to VMs using hypervisor interfaces.
Access the AFU on a VF from applications running on the VM using the same driver inside the VM.
`` Creating VFs is only supported for port devices. Consequently, PR and other management functions are only available through the PF.
## Driver Organization ## ### PCIe Module Device Driver ### !## Driver Organization ## ### PCIe Module Device Driver ### ![Driver Organization](Driver_Organization.png "Driver Organization") FPGA devices appear as a PCIe devices. Once enumeration detects a PCIe PF or VF, the Linux OS loads the FPGA PCIe device driver. The device driver performs the following functions: 1. Walks through the Device Feature List in PCIe device base address register (BAR) memory to discover features and their sub-features and creates necessary platform devices. 2. Enables SR-IOV. 3. Introduces the feature device infrastructure, which abstracts operations for sub-features and provides common functions to feature device drivers. ### PCIe Module Device Driver Functions ### The PCIe Module Device Driver performs the following functions: 1. PCIe discovery, device enumeration, and feature discovery. 2. Creates sysfs directories for the device, FME, and Port. 3. Creates the platform driver instances, causing the Linux kernel to load their respective drivers. ### FME Platform Module Device Driver ### The FME Platform Module Device Driver loads automatically after the PCIe driver creates the FME Platform Module. It provides the following features for FPGA management: 1. Power and thermal management, error reporting, performance reporting, and other infrastructure functions. You can access these functions via sysfs interfaces the FME driver provides. 2. Partial Reconfiguration. During PR sub-feature initialization, the FME driver registers the FPGA Manager framework to support PR. When the FME receives the relevant ioctl request from user-space, it invokes the common interface function from the FPGA Manager to reconfigure the AFU using PR. 3. Port management for virtualization (releasing/assigning port device). After a port device is released, you can use the PCIe driver SR-IOV interfaces to create/destroy VFs. For more information, refer to "FPGA Virtualization". ### FME Platform Module Device Driver Functions ### The FME Platform Module Device Driver performs the the following functions: * Creates the FME character device node. * Creates the FME sysfs files and implements the FME sysfs file accessors. * Implements the FME private feature sub-drivers. * FME private feature sub-drivers: * FME Header * Partial Reconfiguration * Global Error * Global Performance ### Port Platform Module Device Driver ### After the PCIe Module Device Driver creates the Port Platform Module device, the FPGA Port and AFU driver are loaded. This module provides an interface for user-space applications to access the individual accelerators, including basic reset control on the Port, AFU MMIO region export, DMA buffer mapping service, and remote debug functions. ### Port Platform Module Device Driver Functions ### The Port Platform Module Device Driver performs the the following functions: * Creates the Port character device node. * Creates the Port sysfs files and implements the Port sysfs file accessors. * Implements the following Port private feature sub-drivers. * Port Header * AFU * Port Error * Signal Tap ### OPAE FPGA Driver Interface ### The user-space interface consists of a sysfs hierarchy and ioctl requests. Most kernel attributes can be accessed/modified via sysfs nodes in this hierarchy. More complex I/O operations are controlled via ioctl requests. The OPAE API implementation, libopae-c, has been designed to use this interface to interact with the OPAE FPGA kernel drivers.