OPAE AFU Simulation Environment (ASE) User Guide

Intended Audience

This document is written for developers who plan to use AFU Simulation Environment (ASE) as a tool to begin development on the Intel® Xeon®-FPGA Platform with an Intel® FPGA IP (a.k.a Blue Bitstream) accelerator before deployment in a real system. The document is intended for both beginners and experienced developers.

To use the ASE Environment, the developer must have access to Accelerated Functional Unit (AFU) source code in RTL format like {System}Verilog, VHDL) or in SystemC, etc. or in a High-level Synthesis (HLS) language, so long as the generated output can be interpreted by industry-common RTL Simulator tools. Application software can be developed to operate this AFU in the same unified environment.

The minimum set of skills required for developing with ASE, is competence in C/C++, synthesizable SystemVerilog development, familiarity with RTL simulators like Synopsys* VCS-MX or Mentor Graphics** ModelSim-SE*/QuestaSim, and preferably some FPGA PAR experience. Additional skills may be necessary and recommended for more advanced work. Teams of developers who specialize in either RTL development or Software development may use ASE in lock-step to develop functional AFUs for Intel® Xeon®-FPGA platform.

Introduction

AFU Simulation Environment (ASE) is the term used to refer to the Intel® Xeon®-FPGA user application development and debug environment. It aims to provide a consistent transaction level hardware interface and software API that allows users to develop production-quality AFU RTL and SW host application that can then be run on the real FPGA system without modifications.

ASE can be used to develop and simulate AFU and software code that will run on tightly-coupled FPGA and the programmable FPGA acceleration cards. The figures shown may refer to one of the two platforms, but the enviroment may be used to simulate either of the two platforms.

Note

The ASE environment is integrated to support one AFU and one application at a time. Multiple slot simulation in a single platform is not supported in ASE.

Overview of Supported platforms

Xeon-FPGA Multi-chip overview

Xeon-FPGA Multi-chip overview

The Intel® Xeon®-FPGA tightly-coupled FPGA platform provides coherent accelerator attachment to an Intel platform via Intel® Ultra Path Interconnect (UPI) and two PCIe channels. In this case the FPGA Blue Bitstream (BBS) realized on the FPGA hosts a proprietary UPI Protocol layer and PCIe fabric in FPGA logic (registers, slices, LUTs, etc. This Blue Bitstream incorporates a Caching Agent (CA) that participates in Intel® Xeon® caching activities.

AFU Simulation Environment (ASE)

AFU Simulation Environment (ASE) overview

AFU Simulation Environment (ASE) Overview

ASE is a unified environment that aims to reduce time-to-market AFU hardware and software development. It is packaged in OPAE software distribution.

ASE exposes two interfaces:

  • Software: Open Programmable Acceleration Engine (OPAE) API realized in C
  • Hardware: Core Cache Interface (CCI-P) specification realized in SystemVerilog

Customers developing AFU RTL and Software application must comply with these interfaces to be able to successfully deploy their IPs on the integraded FPGA or programmable FPGA acceleration cards.

Capabilities of ASE

  • ASE provides a protocol checker that helps identify protocol correctness. ASE can be used to rule-check if the Accelerator Functional Unit (AFU) complies to CCI-P protocol specifications, and whether OPAE API has been used correctly. It provides methods to identify potential issues early before in-system deployment.
  • ASE can help identify certain lock conditions and Configuration/Status Registers (CSR) address mapping and pointer math mistakes.
  • ASE presents a fake memory model to the AFU which keeps tabs on memory requested as accelerator workspaces, immediately flagging illegal memory transactions to locations outside of requested memory spaces. This is a good way to identify address violation bugs in simulation.
  • ASE does not guarantee synthesizability of the AFU design. However ASE and Intel® Quartus® Prime Pro suite can be used in an iterative design flow to generate a successfully compiled Green Bitstream (GBS) from AFU RTL functionally verified in ASE for interoperability within the modeled platform.
  • ASE provides a data hazard checker which can be used to warn users of CCI-P traffic patterns that may cause WAW, RAW, WAR hazards. These transactions may be debugged using the waveform viewer or by using a relevant Memory Protocol Factory (MPF) shim.
  • ASE does not require administrator privileges needed to run, it is completely user-level. ASE may be run on a “plain vanilla” user Linux box with all the required tools installed.

Limitations of ASE

When using ASE in the application development cycle, consider the following:

  • ASE is a transaction level simulator for a single AFU slot and single application in platform with Intel FPGA IP. ASE does model either Intel UPI or PCIe specific packet structures and protocol layers.
  • ASE does not simulate caching activities and is not a cache simulator. It cannot reliably simulate cache collision or capacity issues.
  • ASE is designed to take an actual in-system AFU RTL and its corresponding OPAE software application and verify them for correctness. ASE cannot consume a Green Bitstream (GBS) file.
  • Although ASE models some latency parameters, it cannot model real-time system-specific latency behavior. It is also not intended to be a timing simulation of the design or latency/bandwidth profile of the real system, but good for functionally correct development.
  • ASE does not simulate multi-AFU or multi-socket configurations.

System Requirements

ASE is available under the Open Programmable Acceleration Engine (OPAE) software release. The current OPAE ASE release support both tightly-coupled FPGA and programmable FPGA acceleration cards. ASE does not require an FPGA in the platform, and uses the Green Bitstream (GBS) AFU in source form. Bitstream configurations cannot be simulated in ASE.

ASE is supported only on 64-bit Linux operating systems. ASE requires a 64-bit RTL simulator, either Synopsys* VCS-MX or Mentor Graphics* QuestaSim/ModelSim-SE. The RTL simulator specific requirements are not listed here. Consult with your RTL simulator vendor for Synopsys* or Mentor Graphics* specific requirements.

The following RTL Simulators versions are supported in the current version of ASE:

  • Synopsys* VCS-MX (tested versions)
    • VCS-MX H2013.06-SP1
    • VCS-MX J2014.12-SP3
    • VCS-MX K2015.09-SP1
    • VCS-MX L2016.06
  • Mentor Graphics* Modelsim-SE/Questasim (tested versions)
    • QuestaSim 10.5b
    • ModelSim-SE 10.5a
    • QuestaSim 10.4d

Note

Both Mentor Graphics* Modelsim-SE/Questasim and Synopsys* VCS-MX system requirements are different are varied.

ASE is tested in some common configurations including the support list for Mentor* and Synopsys*. It is impractical to verify ASE functionality on all possible OS configurations. When choosing an operating system configuration, please consult the RTL Simulator supported OS List.

ASE uses Inter-Process Communication (IPC) constructs. Although under most circumstances these constructs operate normally without glitches. The following Linux locations should exist and be writeable. In most Linux distributions, /dev/shm comes pre-mounted as a default option.

The other ASE requirements are as follows:

  • C-Compiler: gcc 4.4 or above
    • Boost Development libraries
    • UUID Development libraries
    • JSON Development libraries
    • Please see the dependencies of the OPAE System library build process
  • Cmake: version 2.8 or above
  • GLIBC: version 2.19 or above
  • Python: This is optional, but is used by ase/scripts/generate_ase_environment.py script. However, the functions of the script can be performed manually. Python >=2.7 is recommended.
  • Intel® Quartus® Pro 17: If you are simulating the provided NLB sample, ASE needs to find the $QUARTUS_HOME/eda/sim_lib/ directory supplied with Intel® Quartus® Pro. The NLB sample RTL uses the Altera gates library.set GLS_SIM to 1 in the ase/Makefile.

ASE provides a bash script called env_check.sh in ase/scripts/ directory. Run this script to determine if you have the required tools installed.

Check the RTL simulator’s product information for supported operating systems, installation notes, and other related information. In general, the RTL simulator must be able to compile SystemVerilog Direct Programming Interface (DPI) constructs, have SystemC support, and be able to compile the standard examples that come with the installation.

Intel does not provide consulting support for installing the RTL simulators from Synopsys* or Mentor Graphics*.

Package Description

ASE implementation of OPAE API may be downloaded either in RPM format (see Download instructions) or as a source.

The source form directory tree looks as follows:

OPAE_BASEDIR
    |-- ase
    |   |-- api
    |   |   `-- src
    |   |-- rtl
    |   |   `-- dcp_emif_model
    |   |-- sample_config
    |   |   `-- mcp_nlb0
    |   |       |-- config
    |   |       `-- rtl
    |   |-- scripts
    |   `-- sw
    |
    |-- cmake
    |-- common
    |   |-- include
    |   |   |-- fpga_app
    |   |   `-- opae
    |   `-- utils
    |-- doc
    |-- libopae
    |-- samples
    |-- tests
    `-- tools

The above tree directory roughly depicts the package structure of the ASE distribution. There are a minimum number of directories that are required to successfully build and run the ASE simulator. These are listed in this section.

  • common: This directory contains the OPAE library definitions, and defines various macros for access to an FPGA in an OPAE context.
  • samples: Contains a sample that will run on Native Loopback (NLB) RTL AFU
  • libopae: Intel platfrom specific implementation of the OPAE API.
  • ase: ASE simulator implementation location contains several subdirectories, namely:
    • rtl: RTL components of ASE. This must be compiled in the RTL simulator for both programmable FPGA acceleration cards and tightly-coupled FPGA mode simulator builds.
      • dcp_emif_model: programmable FPGA acceleration card Local DDR memory model. This must be compiled for all programmable FPGA acceleration card mode simulation model.
    • sw: Software components of ASE. These are required for all simulations of ASE, and are compiled using GNU Compiler Collection (GCC).
    • api/src: This directory contains the OPAE Intel ASE implementation. This is compiled into a library that may be statically or dynamically linked.
    • scripts: Contains several helper scripts. These are described in Helper Scripts in ASE Section.
    • sample_config: Contains sample configurations of different AFUS. These are described in Sample Configurations Section.

Helper Scripts in ASE

Several scripts are supplied in the ASE distribution under the ase/scripts directory. These can be used for initialization, help with setting up tools, and even cleanup of an existing ASE simulation environment.

Set up Simulation tools

An RTL Simulator tool is required for running ASE amongst other System Requirements. To set up the tools required, use ase/scripts/ase_setup.sh as a template script. The script has many empty placeholders that are site and environment specific. Consult your Electronic Design Automation (EDA) tools administrator, and/or the RTL simulator User Guides for help setting up the tools.

# This file must be edited before sourcing it.
$ emacs ase/scripts/ase_setup.sh
.
.
.

# Source tools script
$ source ase/scripts/ase_setup.sh

ASE Environment Check

This script checks the status of the OS distribution, distro and available system libraries. This is a non-exhaustive check, and looks for only the most important dependencies, e.g. GCC version, GLIBC version, etc.

$ ./scripts/env_check.sh

Create Empty ASE Environment

The simulator for ASE execution may be built either in-tree or out-of-tree. Choice of either approach does not affect user experience or simulator capabilities. An out-of-tree build may be useful if the OPAE package has been installed in a read-only location and is shared across the file-system.

For this reason, an ASE environment create/clone script is made available. This script will generate only an empty ASE environment, and does not take any inputs. The ASE environment is created

# Change directory to target date
$ cd <work-dir>

# Run clone script
$ <install-path>/opae/ase/scripts/create_ase_simbuild_env.sh
#################################################################
#                                                               #
#             OPAE Intel(R) Xeon(R) + FPGA Library              #
#               AFU Simulation Environment (ASE)                #
#                                                               #
#################################################################
Script running from: /tmp/opae/ase/scripts/
ASE Sources found... will proceed to create directory
ASE Source directory: /tmp/opae/ase

ASE simulator build environment created
Next steps: This simulation environment must be configured for an AFU
            See ASE Documentation on usage steps



# Empty ASE directory created
$ ls
ase ...
$ tree ase
ase
|-- ase.cfg
|-- ase_regress.sh
`-- Makefile

0 directories, 3 files

$

Note

The ase/scripts/create_ase_simbuild_env.sh will not generate any AFU specific configuration files. These will need to be provided by the user or using the generate_ase_environment.py script.

Generate ASE Environment Script

ASE supplies a Generate ASE Environment helper script that does a brute-force check of supplied AFU RTL directories. The script is imperfect and lists every file ending in an .sv, .vs, .vhd, .v file and directories separated by +.

  • Mandatory Option: The script requires a directory path to RTL AFU.
  • Optional -t: By default the tool option selected is VCS. If Mentor* tools are used, supply the QUESTA option.
  • Optional -p: By default integraded FPGA skxp option is selected. If programmable FPGA acceleration card is used, supply the dcp option.
  • Optional -x: Exclsions for path searches can be supplied using this option.

Along with the AFU configuration files, depending on the RTL tools used, certain tool control scripts are generated. * VCS: In VCS configuration, synopsys_sim.setup and vcs_run.tcl files are created. * QUESTA: In QUESTA configuration, vsim_run.tcl file is created.

The .tcl files are used at simulation run-time.

Details on generated files: * vlog_files.list lists all the Verilog and SystemVerilog files found in the AFU directory path * vhdl_files.list lists all the VHDL files found in the AFU directory path. * ase_sources.mk ties the above two files into DUT_VLOG_SRC_LIST and DUT_VHD_SRC_LIST Makefile variables. * ASE_PLATFORM is set to the platform type selected or to the default type. * Extra VCS or QUESTA options may be set using the SNPS_{VLOGAN,VHDLAN,VCS}_OPT or MENT_{VLOG,VCOM,VSIM}_OPT options in the Makefile. See ASE Makefile Variables for more information.

Absolute paths are used wherever possible. One helpful option to use for portability across users and groups may be to use an environment variables may be used. These can be used directly in the generated files for building and running the simulator.

Note

Users must manually check this file for correctness before using it in the simulation.

$ ./scripts/generate_ase_environment.py  --help
#################################################################
#                                                               #
#             OPAE Intel(R) Xeon(R) + FPGA Library              #
#               AFU Simulation Environment (ASE)                #
#                                                               #
#################################################################
usage: generate_ase_environment.py [-h] [-t {VCS,QUESTA}] [-p {skxp,dcp}]
                               [-x EXCLUDE]
                               dirlist [dirlist ...]

positional arguments:
    dirlist               list of directories to scan

optional arguments:
    -h, --help            show this help message and exit
    -t {VCS,QUESTA}, --tool {VCS,QUESTA}
                          simulator tool to use, default is VCS
    -p {skxp,dcp}, --plat {skxp,dcp}
                          FPGA Platform to simulate
    -x EXCLUDE, --exclude EXCLUDE
                          file name pattern to exclude

Clean ASE Environment

the ASE cleanup script located at scripts/ipc_clean.py may be used for killing zombie simulation processes and temporary files left behind by failed simulation processes or crashes.

$ ./scripts/ipc_clean.py

############################################################
#                                                          #
#                ASE IPC Cleanup script                    #
#                                                          #
############################################################
IPC mounts seem to be readable... will attempt cleaning up IPC constructs by user ' user_foo '
Removing .ase_ready file ...
Type 'y' to clean up all zombie ase_simv processes : y
Going ahead with cleaning up ASE processes opened by  user_foo
$

Sample Configurations

Integraded FPGA NLB Mode0 RTL example

ASE ships a copy of the sampel Native LoopBack (NLB) RTL AFU inside the ase/sample_config/mcp_nlb0 directory. This NLB RTL can be simulated in ASE using the opae/samples/hello_fpga.c example. Specific steps to achieve this is described in Running Sample NLB in tightly-coupled FPGA configuration section.

Overview of ASE Usage

AFU Simulation Environment (ASE) is structured as a server-client simulation environment. Broadly speaking the AFU RTL is compiled into a simulator server process and the application compiled/linked against OPAE ASE library forms the client process. Communications between server and client is done using named pipes. Most of the simulation infrastructure is abstracted in ASE code, and does not require any user modifications.

ASE Server-Client Process Flow

ASE Server-Client Process Flow

  • Server Process:
    • Server process interfaces with 3rd Party RTL Simulator packages (currently Mentor* Modelsim-SE and Synopsys* VCS-MX are supported) via SystemVerilog-DPI library and interface to the simulator software.
    • Communication to client is achieved using Named Pipes. A pipe event monitoring engine is also built into the server process. Communication about control, status and session management is done using the named pipes.
    • The server process also exposes a “fake” physical memory modeling engine that allows the RTL AFU to access “physical” addresses. Virtual to “fake physical” address translations are handled in this block.
    • CCI-P interface is managed in SystemVerilog. All CCI-P events are logged and time stamped.
    • The server also includes a CCI-P Checker block that rule-checks CCI-P transactions originating in the AFU. This allows for development-time live checking of protocol issues, and early warning of CCI-P protocol non-compliance, hazards, race conditions, etc.
    • Lastly. buffer allocation calls are mapped to POSIX Shared Memory (/dev/shm). Information about these buffers are shared back and forth between server-client processes using the named pipes.

Note

The Physical addresses generated in ASE are not realistic and are not replicable in-system.

  • Client Process:
    • Client primarily implements an OPAE interface and a library to ASE’s implementation of platform capabilties like MMIO, Buffer management, session control. Actual features are available based on what platform is configured in the build stage. These are exposed using OPAE API’s functions.
    • A compiled program would compile/link against the ASE implementation of OPAE library. All OPAE calls will routed to ASE instead of the OPAE platform driver.

The two processes (both server and client) are built using two separate build scripts.

  • Client: OPAE library implementations for System and ASE are built using the main cmake script supplied at the base of the distribution. The libary produced will be installed in lib directory (depending on the install method).
  • Server: ASE Server process (containing the ASE Software, Systemverilog engines and the AFU RTL logic code) are compiled using a Makefile located inside the ase directory.

AFU Build Instructions

ASE Application (Client) Build Instructions

If you downloaded a source tarball, you will need this step, else go directly to the Application build step.

Note

For the sake this discussion, we will use the directory /tmp/opae/ as the base directory where OPAE API software distribution is installed. /tmp prefix may point to any system or user directory.

ASE Source directory will point to /tmp/opae/ase/

  • The first step in the Software build is to build the OPAE libraries.
# Change directory to opae directory
$ cd /tmp/opae/

# Check contents
$ ls
ase  cmake  CMakeLists.txt  common  doc  libopae  mybuild  README  samples  tests  tools

# Create an out-of-build directory, and change to it
$ mkdir mybuild
$ cd mybuild

# Configure the software build using cmake (see options)
# $ cmake <Path to CMakeLists.txt>
# Install directory, in this example is <Path  to opae>/myinst/

$ cmake ../ -DBUILD_ASE=YES -DCMAKE_INSTALL_PREFIX=/tmp/opae/myinst/
-- The C compiler identification is GNU 4.8.4
-- The CXX compiler identification is GNU 4.8.4
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
.
.
.
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/opae/mybuild

# Build and install libraries
$ make
$ make install

# Check library install paths
$ cd /tmp/opae/myinst/
$ ls
doc  include  lib

$ ls lib
libopae-c-ase.so  libopae-c.so  libopae-c.so.0  libopae-c.so.0.1.0 ...

The software application must be built for ASE using libopae-c-ase.so using a gcc command.

There are a few ways to build this:

Dynamically linking libopae-c-ase.so
# Change directory to <Path to Directory>/opae/samples/
#   -luuid is required by UUID code
#   -lopae-c-ase is required for linking against ASE library
#
$ gcc -g -o hello_fpga hello_fpga.c -L /tmp/opae/myinst/lib/ -I /tmp/opae/myinst/include/ -luuid -lpthread -lopae-c-ase -std=c99

# When running the application, use
# Make sure the relevant LD_LIBRARY_PATH variable is set
$ ./hello_fpga

The ASE implementation of the OPAE library attempts to functionally mirror system behavior. Hence, the application may also be dynamically linked to the System library, and then overloaded with the LD_PRELOAD environment.

# Change directory to <Path to Directory>/opae/samples/
#   -luuid is required by UUID code
#   -lopae-c is required for linking against ASE library
#
$ gcc -g -o hello_fpga hello_fpga.c -L /tmp/opae/myinst/lib/ -I /tmp/opae/myinst/include/ -luuid -lpthread -lopae-c -std=c99

# When running the application, use
# Make sure the relevant LD_LIBRARY_PATH variable is set
# ASE_WORKDIR environment variable must also be set.
$ LD_PRELOAD=libopae-c-ase.so ./hello_fpga
Compiling the libopae-c-ase.so library with OPAE Software application

A much simpler method to use the OPAE ASE library implementation and compile it to the OPAE software application directly in one command using GCC.

# Compile command
$ gcc -g -o hello_fpga_ase hello_fpga.c /tmp/opae/myinst/lib/libopae-c-ase.so -I /tmp/opae/myinst/include/ -std=c99 -luuid

# When running the application, start the simulator
# Then set LD_LIBRARY_PATH and ASE_WORKDIR
$ ./hello_fpga_ase

ASE Simulator (Server) Build Instructions

ASE uses a platform differentiation key in the simulator Makefile to enable different platform features and produces a simulator configuration based on that.

Users must note the following REQUIRED Build configurations

Configuration Description Default
ASE_PLATFORM This is the platform differentiator must be set in ase_sources. mk to point to the required simulator features. FPGA_PLATF ORM_INTG_XEON` ` and ``FPGA_PLATFOR M_DISCRETE are the only legal values. FPGA_PLA TFORM_INTG _XEON
SIMULATOR This must be set in ase_sources. mk to point to the correct kind of RTL simulator. Currently Synopsys* VCS-MX and Modelsim-SE (Questasim) simulator builds are supported in ASE. Only 64-bit configurations are supported. VCS and QUESTA are the only legal values. VCS
DUT_VLOG_SRC_L IST, DUT_VHDL_SRC_L IST, and DUT_INCDIR These options point to AFU Verilog, VHDL and include path settings required for RTL simulators to build the simulation model correctly None supplied — Users must fill this in.

For more information on other switches see ASE Makefile targets

To generate the AFU specific files for the simulator build, you may use the scripts/generate_ase_environment.py script. This script is imperfect, and will require manual inspection of generated configurations in order to get correct operation.

Note

Any AFU RTL specific define macros or include paths must be edited in ase_sources.mk.

You may use the Generate ASE Environment script to generate the required AFU ase_sources.mk and vlog_files.list.

# change to ASE directory
$ cd /tmp/opae/ase/

# Use the scripts/generate_ase_environment.py
$ ./scripts/generate_ase_environment.py -t VCS -p dcp <Path to sample RTL>
.
.
.
# Edit files as needed.

$ ls
ase_sources.mk  vcs_run.tcl  synopsys_sim.setup  vlog_files.list  ...

The vlog_files.list and ase_sources.mk files may need to be checked manually for correctness. The simulator is now be built with a ‘make’ command.

$ cd <opae>/ase/
$ make
.
.

$ ls work/ase_simv
work/ase_simv

ASE Runtime Instructions

Broadly in the server-client configuration of ASE, the server (ASE simulator) is invoked first. Then the client software application is invoked next along. The run time options are shown here.

$ make sim \
    [SIMULATOR=<VCS|QUESTA>] \
    [ASE_CONFIG=<Path to ase.cfg>] \
    [ASE_SCRIPT=<Path to ase_regress.sh if ASE_MODE=4>]

For a start, we recommend using two terminal windows.

Terminal 1: When make sim is run, When the simulator is invoked, ASE initializes and the AFU, issues a reset and then waits for transactions to come in. The software application must wait until “Ready for Simulation” message is displayed in Terminal 1.

The environment variable ASE_WORKDIR is indicated in Terminal 1.

# Invoke the simulator
$ make sim
SIMULATOR=VCS
CC=gcc
#################################################################
#                                                               #
#             OPAE Intel(R) Xeon(R) + FPGA Library              #
#               AFU Simulation Environment (ASE)                #
#                                                               #
#################################################################
ASE platform set to MCP_SKYLAKE mode
.
.
.
SIM-SV: Transaction Logger started
SIM-SV: Simulator started...
SIM-C : +CONFIG /tmp/opae/ase/ase.cfg file found !
SIM-C : +SCRIPT /tmp/opae/ase/ase_regress.sh file found !
SIM-C : PID of simulator is 41819
SIM-C : Reading /tmp/opae/ase/ase.cfg configuration file
SIM-C : ASE was started in Mode 1 (Server-Client without SIMKILL)
        ASE mode                   ... Server-Client mode without SIMKILL
        Inactivity kill-switch     ... DISABLED
        Reuse simulation seed      ... ENABLED
        ASE Seed                   ... 1234
        ASE Transaction view       ... ENABLED
        User Clock Frequency       ... 312.500000 MHz, T_uclk = 3200 ps
        Amount of physical memory  ... 128 GB

.
.
.
SIM-C : ** ATTENTION : BEFORE running the software application **
    Set env(ASE_WORKDIR) in terminal where application will run (copy-and-paste) =>
    $SHELL   | Run:
    ---------+---------------------------------------------------
    bash/zsh | export ASE_WORKDIR=/tmp/opae/ase/work
    tcsh/csh | setenv ASE_WORKDIR /tmp/opae/ase/work
    For any other $SHELL, consult your Linux administrator

    SIM-C : Ready for simulation...
    SIM-C : Press CTRL-C to close simulator...

Terminal 1 make sim can be closed by issuing a SIGTERM to the relevant ase_simv process or by sending the CTRL-C keystroke.

Terminal 2: First set an environment variable ASE_WORKDIR as seen in Terminal 1. In this example ASE_WORKDIR is set to /tmp/opae/ase/work. Then invoke the software application.

# Set ASE_WORKDIR environment variable
$ export ASE_WORKDIR=/tmp/opae/ase/work/

# Run the application
$ export LD_LIBRARY_PATH=/tmp/opae/myinst/lib/
$ LD_PRELOAD=libopae-c-ase.so ./hello_fpga

Note

When simulation is completed (application has exited), the simulator must be closed so the waveform dump process may get completed. On Terminal 1, issue a CTRL-C command.

<Simulator running>
.
.
.
SIM-C : Ready to run next test
818602500C0Tx AlmFull toggled from 1 to 0

818602500C1Tx AlmFull toggled from 1 to 0

<CTRL-C Key hit>

^CSIM-C : Closing message queue and unlinking...
SIM-C : Unlinking Shared memory regions....
SIM-C : Session code file removed
SIM-C : Removing message queues and buffer handles ...
SIM-C : Cleaning session files...
SIM-C : Simulation generated log files
    Transactions file       | $ASE_WORKDIR/ccip_transactions.tsv
    Workspaces info         | $ASE_WORKDIR/workspace_info.log
    ASE seed                | $ASE_WORKDIR/ase_seed.txt

SIM-C : Tests run     => 1
SIM-C : Sending kill command...
SIM-SV: Simulation kill command received...

Transaction count |       VA      VL0      VH0      VH1 |    MCL-1    MCL-2    MCL-4
========================================================================================
MMIOWrReq           9 |
MMIORdReq           2 |
MMIORdRsp           2 |
UMsgHint            0 |
UMsgData            0 |
RdReq           16384 |        0        0    16384        0 |    16384        0        0
RdResp          16384 |        0        0    16384        0 |
WrReq           16385 |        0        0    16385        0 |    16385        0        0
WrResp          16385 |        0        0    16385        0 |    16385        0        0
WrFence             1 |        0        0        1        0 |
WrFenRsp            1 |        0        0        1        0 |

$finish called from file "/tmp/opae/ase/rtl/ccip_emulator.sv", line 2657.
$finish at simulation time           1514962500
V C S   S i m u l a t i o n   R e p o r t
Time: 1514962500 ps
CPU Time:    142.500 seconds;       Data structure size:   1.1Mb
Tue Jun 20 13:29:13 2017

At the end of the simulation, a few files are generated.

  • Waveform dump: Depending on whether VCS or QUESTA is used, a different waveform file is generated. make wave will open the waveform for the selected tool.
    • inter.vpd: VCS Waveform file
    • vsim.wlf: Mentor/Questa waveform file.
  • $ASE_WORKDIR/ccip_transactions.tsv: CCI-P Events log listing all events observed in CCI-P interface. The timestamps indicate the corresponding time interval in the waveform dump VPD file.
  • $ASE_WORKDIR/workspace_info.log: Information about buffers opened by the simulation.
  • $ASE_WORKDIR/ase_seed.txt: Information about simulation seed.
  • $ASE_WORKDIR/ccip_warnings_and_errors.txt: Information about CCI-P warnings and errors.

Tips for successful ASE Usage

  1. ccip_logger.sv is a module in ASE (located at ase/hw/ccip_logger.sv) and can be used to log CCI-P events if there are interfaces inside the AFU which conform to CCI-P protocol. This is common for designs that use the Intel® FPGA IP Basic Building Blocks (BBB). This can be used to compare transactions and traffic flow through various CCI-P interfaces.

Note

When instantiating ccip_logger in multiple locations, distinct file names must be used in each module instantiation.

ASE CCI-P Logger Reuse

ASE CCI-P Logger Reuse

  1. ASE uses a simulation shutdown mechanism that gracefully closes all mutexes, locks, POSIX structures safely before exiting via $finish statement in SystemVerilog. If your AFU design uses as $error or $finish to identify error conditions while simulating, consider using the handle start_simkill_countdown() instead. start_simkill_countdown() calls $finish after completing the steps for a graceful shutdown.
  2. ASE simulations can be easily scripted. When the simulator becomes ready for use, a lock file is written in the work directory. The lock file name is $ASE_WORKDIR/.ase_ready.
# Start Simulator in background
make sim &

# Wait for simulator readiness
# When .ase_ready is created in the $ASE_WORKDIR, ASE is ready for simulation
while [ ! -f $ASE_WORKDIR/.ase_ready.pid ]
do
    sleep 1
done

# Start application
cd $PATH_TO_APPLICATION

# Export ASE_WORKDIR variable
export ASE_WORKDIR={Path to simulator mentioned in ASE green printout}
./Application
  1. If the ASE simulation is too slow, try one of these methods to speed up the simulation:
  • Switch OFF wave dumps: Check your RTL vendor’s recommendation on switching off wave form dumps. In some cases, it may be possible to generate wave dumps for only specific module hierarchies.
  • In ase.cfg, set ENABLE_CL_VIEW = 0. This will prevent events from being printed on the screen.
  1. The latency model of ASE can be modified to generate random patterns of CCI-P transactions for every run. The latency model settings are located at $ASE_SRCDIR/rtl/platform.vh. The *_LATRANGE constraints are set up as a pair and coded as follows:
`define X_LATRANGE     min, max

The min and max values refer to the minimum and maximum cycle counts after which responses will be returned back to the AFU. During simulation, a transaction is assigned a latency value randomly picked from this (min, max) range. The larger the difference between (min, max), the bigger the standard deviation of latency for a given type of transaction. Transaction latencies for RDLINE, WRLINE and other transactions can be individually selected.

Note

None of these values can be implied to be the actual latencies of different transactions when running on the FPGA platform. These settings must be used for simulation and testing only.

/*
 * Latency model
 * Coded as a Min,Max tuple
 * -------------------------------------------------------
 * RDLINE_LATRANGE : ReadLine turnaround time
 * WRLINE_LATRANGE : WriteLine turnaround time
 * UMSG_LATRANGE   : UMsg latency
 * INTR_LATRANGE   : Interrupt turnaround time
 *
 * LAT_UNDEFINED   : Undefined latency
 *
 */
`define MMIO_LATENCY                15
`define RDLINE_S_LATRANGE          20,118
`define RDLINE_I_LATRANGE          20,118
`define WRLINE_M_LATRANGE          20,118
`define WRLINE_I_LATRANGE          20,118
`define UMSG_START2HINT_LATRANGE   39,41
`define UMSG_HINT2DATA_LATRANGE    41,45
`define UMSG_START2DATA_LATRANGE   82,85
`define INTR_LATRANGE              10,15

`define LAT_UNDEFINED              300

`define RDWR_VL_LATRANGE           20,118
`define RDWR_VH_LATRANGE           240,270

`define ASE_MAX_LATENCY            300
  1. ASE includes a CCI-P protocol checker module (located at $ASE_SRCDIR/rtl/ccip_checker.sv). It is recommended to use this for analyzing CCI-P compliance when simulating designs in ASE. The checker sniffs transactions, conditions and header settings and flags them off as either warnings or errors. Multiple classes of issues can be identified using the hw/ccip_sniffer.sv. All checker warnings and errors are logged into $ASE_WORKDIR/ccip_warnings_and_errors.txt.
ASE Error Example

ASE Error Example

  • Memory errors in transactions: This is flagged off with the highest severity. Simulation stops immediately and the transaction is immediately flagged off. The memory error is logged at $ASE_WORKDIR/ase_memory_error.log. The erroneous transaction can be traced back using $ASE_WORKDIR/ccip_transactions.tsv. The timestamp value can be found in the transaction log itself.
  • Protocol errors marked as “[ERROR]” in $ASE_WORKDIR/ccip_warning_and_errors.txt: These are protocol errors found by ASE are flagged off immediately and the simulation stops on finding the event. The timestamps are noted and can be co-related with timestamps from the transaction log.
  • Protocol warnings marked as “[WARN]” in $ASE_WORKDIR/ccip_warning_and_errors.txt: These are protocol warnings found by ASE and do not cause a simulation stoppage. There are many minor events that may cause this (e.g. X/Z found in the transaction, reset ignorance).

Memory hazards (RAW, WAR and WAW hazards) may also be flagged off by ASE.

ASE Examples

Running Sample NLB in tightly-coupled FPGA configuration

Open Two terminals, We will use Terminal 1 to run the ASE simulator, and Terminal 2 to OPAE software application.

  1. In Terminal 2, build the OPAE software stack as described here

    • The make process also builds the hello_fpga.c example located in /tmp/opae/samples/hello_fpga.c and creates an executable in the build directory. In this case, the location of the built binary is located here:
    /tmp/opae/mybuild/bin/hello_fpga
    • When executing this binary, it dynamically links to the libopae-c-ase.so
    $ ldd /tmp/opae/mybuild/bin/hello_fpga
    linux-vdso.so.1 =>  (0x00007ffe44860000)
    libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f23cac8f000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f23caa71000)
    libopae-c.so.0 => /tmp/opae/mybuild/lib/libopae-c.so.0 (0x00007f23ca830000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f23ca468000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f23cae94000)
    libjson-c.so.2 => /lib/x86_64-linux-gnu/libjson-c.so.2 (0x00007f23ca25d000)

  2. On Terminal 1, open the ASE directory. If you are using an out-of-tree build flow, use steps located here for steps to create an empty ASE project directory.

  • A sample NLB RTL and configuration is located in the <install-prefix>/opae/ase/sample_config/mcp_nlb0/ directory. The RTL is located in the rtl directory. A pre-made configuration is available in the config directory. For this experiment, we will build a configuration from scratch using the scripts/generate_ase_environment.py script.

  • The example shows VCS usage but the steps may be used for QUESTA as well.

  • The example can only be used in the ASE_PLATFORM=FPGA_PLATFORM_INTG_XEON configuration. Using the FPGA_PLATFORM_DISCRETE configuration will cause compilation failures.

    # Change to the 'ase' directory
$ cd <work-path>/ase/

# Generate the ASE path
$ /tmp/opae/ase/scripts/generate_ase_environment.py -t VCS -p skxp /tmp/opae/ase/sample_config/mcp_nlb0/rtl/


# INSTRUCTIONS TO USER
# --------------------
# ase_sources.mk, vlog_files.list, and other simulation settings are generated>
# Read the generated files for content
#
# Add the following line to ase_sources.mk file
# This switch will allow add the following switch
#

$ echo "SNPS_VLOGAN_OPT = +define+VENDOR_ALTERA +define+TOOL_QUARTUS +define+NUM_AFUS=1 +define+NLB400_MODE_0" >> ase_sources.mk

# If QUESTA is being used, set the following switch
# $ echo "MENT_VLOG_OPT = +define+VENDOR_ALTERA +define+TOOL_QUARTUS +define+NUM_AFUS=1 +define+NLB400_MODE_0" >> ase_sources.mk

# Build the simulator
$ make ASE_PLATFORM=FPGA_PLATFORM_INTG_XEON

  1. On Terminal 1, we will invoke the simulator

    $ make sim

<wait until simulator starts and prints "Ready for Simulation">

  2. Switch back to Terminal 2: We will invoke the hello_fpga application.

    # Change directory to build directory 'bin'
$ cd /tmp/opae/mybuild/bin/

# Set LD_LIBRARY_PATH environment variable
#
# Depending on the install method used, this path may be different
#
$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/tmp/opae/myinst/lib

# Set the ASE_WORKDIR environment variable
#
# Copy this environment variable from Terminal 1 simulator run (Green text)
#
$ export ASE_WORKDIR=<work-path>/ase/work/

# Run the application
$ LD_PRELOAD=libopae-c-ase.so ./hello_fpga
    • Wait for Simulation to end, and application to exit.

  3. In Terminal 1, issue a CTRL-C to close the simulator.

    • When exiting, the ASE simulator prints out transaction counts seen during the session.
    • Run make wave to open the waveform viewer.
    • Open work/ccip_transactions.tsv in a text viewer to see the actual transactions. The timestamps may be checked in the wavefor viewer.

Operation Reference

ASE Simulator Makefile Switches

The ASE Makefile template consists of many targets and switches. These are described in this section. While this template is only an example and may work for you, for complicated simulation scenarios, users are advised to build their own compilation script.

Synopsys* VCS-MX 64-bit and Mentor Graphics* Modelsim-SE/Questasim 64-bit are the only simulation products supported in ASE. For a complete listing including the versions supported, see System Requirements section.

ASE Makefile Build flow

The abstract steps to compile ASE Simulator are as follows:

  • Step 1: Compile Software objects of ASE into a library file
    • The Software components located in ase/sw/ directory is first compiled into software library.

Note

SIM_SIDE must be defined as a compilation macro when compiling the ASE simulator objects. This is not the same as the OPAE ASE software library.

  • Step 2: Compile ASE Systemverilog files located in ase/rtl/ directories
    • ASE RTL components based on programmable FPGA acceleration card or tightly-coupled FPGA modes are compiled into the chosen simulation databases.
    • If programmable FPGA acceleration card mode is selected, the EMIF Memory controller model is compiled into the ASE environment.
    • If Altera Gate libraries are required, these models will have to be compiled into the ASE environment.
  • Step 3: Compile AFU components into the ASE environment.
    • Use the RTL simulator software tools to compile the AFU components. VHDL or {System}Verilog components may be compiled using the Synopsys* or Mentor utilities.

ASE Makefile targets

Target Description
all Default build target, attempts a simulator build in Synopsys* VCS-MX
header Print version and preamble
check Environment check
sw_build Build ase/sw/ components into software library. SIM_SIDE switch is used to build the simulator software components
help Print help information
vcs_build Synopsys* VCS-MX template build flow
questa_build Mentor Graphics* Modelsim-SE/Qu estasim template build flow
sim Start the ASE Simulator
wave Open Selected RTL Waveform Viewer
clean Clean build, simulation, and log files
distclean Clean distribution - removes the AFU configuration files as well

ASE Makefile Variables

Makefile target Description Default value
VCS_HOME Synopsys* VCS-MX installation path Tool Installation Specific
MTI_HOME Mentor installation path Tool Installation Specific
QUARTUS_HOME Intel® Quartus® Pro installation path Tool Installation Specific
ASE_PLATFORM Platform selection switch: Either tightly-couple d FPGA or programmable FPGA acceleration card may be selected using FPGA_PLATFOR M_INTG_XEON and FPGA_PLATFOR M_DISCRETE respectively FPGA_PLATFORM_ INTG_XEON
SIMULATOR Simulator Key to environment Allowed values VCS or QUESTA VCS
ASE_SRCDIR ASE Source location Current source path
ASE_WORKDIR Location where ASE runs (usually $ASE_SRCDIR/ work) Environment specific execution path
WORK Simulation library compilation location work
``ASE_DISABLE_LOGGER` ` Switch to disable logger build 0
``ASE_DISABLE_CHECKER `` Switch to disable checker build WARNING: This can have side effects on protocol correctness 0
GLS_SIM Enable gate simulation build 1
GLS_VERILOG_OPT Libraries that enable Altera gate simulation Quartus EDA Simulation library paths
ASE_CONFIG ASE Run-time configuration file (described `here <#ase-ru ntime-configur ation-options> `__) $ASE_SRCDIR/as e.cfg
ASE_SCRIPT ASE Regression script path $ASE_SRCDIR/as e_regress.sh
TIMESCALE Simulator timescale 1ps/1ps
ASEHW_FILE_LIST ASE RTL File list see ase/Makefile
ASE_MEM_FILE_LIST ASE RTL file list for programmable FPGA acceleration card mode memory see ase/Makefile
ASE_INCDIR ASE Include directory paths see ase/Makefile
ASE_TOP ASE top level entity (must not be changed) ase_top
CC_OPT ASE Software Library compiler build options

``-g -m64 -fPIC -D SIM_SIDE=1 -I

$(ASE_SRCDIR)/s

w/ -D SIMULATOR= VCS -D $(ASE_PLA TFORM) -Wall -I $(VCS_HOME)/incl ude/``
ASE_LD_SWITCHES ASE Software Linker switches
``-lrt -lpthread
-lstdc++ -luuid

``
SNPS_VHDLAN_OPT Synopsys* VCS-MX VHDL compile options (extra options may be added into ase_sources. mk) see ase/Makefile
SNPS_VLOGAN_OPT Synopsys* VCS-MX {System}Verilo g compile options (extra options may be added into ase_sources. mk) see ase/Makefile
SNPS_VCS_OPT Synopsys* VCS-MX options for building simulator executable (extra options may be added into ase_sources. mk) see ase/Makefile
SNPS_SIM_OPT Synopsys* VCS-MX Simulation options (extra options may be added into ase_sources. mk) see ase/Makefile
MENT_VCOM_OPT Mentor* Modelsim-SE/Qu estasim VHDL compile options (extra options may be added into ase_sources. mk) see ase/Makefile
MENT_VLOG_OPT Mentor* Modelsim-SE/Qu estasim {System}Verilo g compile options (extra options may be added into ase_sources. mk) see ase/Makefile
MENT_VSIM_OPT Mentor* Modelsim-SE/Qu estasim VHDL simulation options (extra options may be added into ase_sources. mk) see ase/Makefile
DUT_VLOG_SRC_LIST Points to a text file listing AFU {System}Verilo g files (usually vlog_files.l ist) User Generated
DUT_VHDL_SRC_LIST Points to a text file listing AFU VHDL files (usually vhdl_files.l ist User Generated
DUT_INCDIR Lists AFU include directories separated by a ‘+’ (usually in ase_sources. mk) User Generated

ASE Runtime Configuration Options

The ASE configuration file is a text file that can be used to configure simulator behavior. An example configuration script is available at ase/ase.cfg

Switch Name Default Description
ASE_MODE 1 Modes in which ASE can be run1 : Standard Server-Clien t Mode2 : Simulator is closed after ASE_TIMEOU T clocks3 : Software shuts down simulator when client application releases session 4 : Regression mode invoked by script>=5 : Ignored (revert to ``ASE_MODE=1 ``)
ASE_TIMEOUT 50000 (only if ASE_MODE=2) Watchdog timer shuts down simulator after ASE_TIMEOU T clocks of CCI-P interface inactivity.
ASE_NUM_TESTS 500 (only if ASE_MODE=4) Number of tests in regression mode. Failure to set this number correctly, may cause the simulator to exit pre-maturely or to keep waiting for tests to get started.
ENABLE_REUSE_SEED 1 Reuses simulation seed, for the same application, CCI-P transactions are “re-playable ”, i.e., the addresses used can be re-played.Wh en set to 0, obtains a new seed.
ASE_SEED 1234 (only if ENABLE_REUSE_SEED=1) ASE seed setting, this is used only when ENABLE_REU SE_SEED is set to 1, else a different seed is used. At the end of the simulation, a $ASE_WORKD IR/ase_seed. txt file is written with the currently used seed.
ENABLE_CL_VIEW 1 ASE verbosely prints all CCI-P transactions . On long simulation runs, setting this to 0 may speed up simulation.
PHYS_MEMORY_AVAILABLE_ GB 32 Restricts ASE address generation to within this memory range.

Logging Verbosity Control

ASE supports verbosity control of logging messages. In general, three logging levels are supported: * ASE_INFO: Mandatory information messages required to indicate operation * ASE_ERR: Error messages during operation * ASE_MSG: General messages indicating check-points in ASE. These can be silenced by setting the environment variable ASE_LOG to 0. By default, these messages are printed on stdout

ASE_LOG=0 can be used while running the application, something like this:

$ ASE_LOG=0 LD_PRELOAD=libopae-c-ase ./hello_fpga

In the simulator, two types of messgaes are generated: * CCI-P events generated by the AFU: These can result in a large amount of print-out depending on the type of simulation. these can be controlled using ENABLE_CL_VIEW=0 in ase.cfg file. * ASE_LOG=0 environment variable setting will find minimal use since there is only minimal ASE events that may generate these. Warnings and errors may not be suppressed.

Troubleshooting and Error Reference

The following are a non-conclusive list of errors and warnings that one may see in ASE.

Observation Problem Next Steps
Either all transactions are not seen or simulation ends earlier than expected. ASE Simulation inactivity is too short for the application use-case to be successfully simulated in ASE. If using ASE_MODE=2 (Daemon with timeout), in the ase.cfg file, edit setting ASE_TIMEOUT to a higher clock count value or a disable.
ASE simulation build error – compilation, or linking failed GCC version might be too old. Enter the ase directory and check if the following command works $ make sw_build ASE is known to build correctly with GCC 4.4 or higher. Use the ase/scripts/env_check .sh script to identify issues.
Synopsys* VCS-MX dumped stack while compiling or running Possible corruption of compiled objects or problem with incremental compilation.

Clean up ASE environment using $ make clean If this fails, clean the distribution with $ ./distclean.sh

Then

rebuild the simulation.
ERROR: Too many open files Past ASE simulation runs did not close cleanly and may have left behind open IPC instances. Use the script located at $ASE_SRCDIR/scripts/i pc_clean.py. Check if System Requirements has been met. If problem continues to occur, check how to increase resource limits for your Linux distribution.
$ASE_WORKDIR environment variable has not been set up Application could not find a valid simulation session Follow the steps printed when the ASE simulation starts. These instructions are printed in GREEN.
.ase_timestamp cannot be opened at <DIRECTORY> Simulator may not have been started yet. Note that when started, the simulator prints: Ready for Simulation$ASE_W ORKDIR may not set up correctly. Check the ASE_WORKDIR environment variable. ``$ echo $ASE_WORKDIR `` Wait for simulator to print: ``Ready for Simulation` `
ase_sources.mk: No such fi le or directory ASE Environment has not been generated. Generate a AFU RTL listing (in vlog_files.list and ase_sources.mk) configuration. See ase/sample_configs for sample configurations. ase/scripts/generate_ ase_environment.py may be used for this purpose
An ASE instance is probably still running in current directory. An ASE simulation is already running in the $ASE_WORKDIR directory. If the simulation process is unusable or unreachable, use the ase/scripts/ipc_clean .py script to clean up the simulation temporary files. It is recommended to do: $ make clean Then attempt to rebuild the simulator.