Silicom fb2CDg1@AGM39D-2

  • Card information:

    • Vendor: Silicom

    • Name: fb2CDg1@AGM39D-2 (ThunderFjord)

    • Ethernet ports: 2x QSFPDD56

    • PCIe conectors: Edge connector

    • FPGA Card Website

  • FPGA specification:

    • FPGA part number: AGMF039R47A2E2VR0, AGMF039R47A1E2VC

    • Ethernet Hard IP: F-Tile (up to 400G Ethernet)

    • PCIe Hard IP: R-Tile (up to PCIe Gen5 x16)

NDK firmware support

  • Ethernet cores that are supported in the NDK firmware:
  • PCIe cores that are supported in the NDK firmware:

    • R-Tile in the PCIe Module

    • See the <NDK-FPGA_root_directory>/card/silicom/fb2cdg1/config/card_conf.tcl file for supported PCIe configurations.

  • Makefile targets for building the NDK firmware (valid for Minimal app, may vary for other apps):

    • Use make 400g2 command for firmware with 2x400GE (default).

  • Support for booting the NDK firmware using the nfb-boot tool:

    • YES, starting with the nfb-framework version 6.28.5

Note

To build the NDK firmware for this card, you must have the Intel Quartus Prime Pro and PACSign tool installed, including a valid license.

HBM2e memory

The card has two HBM2e stacks, one on each side of the die. They are attached through the hard Network on Chip (NoC), which gives the application 16 AXI4 ports per stack, one port per pseudo-channel.

  • Configuration in card_conf.tcl (or overridden in app_conf.tcl):

    • HBM_PORTS: total number of AXI4 ports, distributed evenly between the stacks (0 to 32).

    • HBM_MODULES: number of HBM stacks used (0, 1 or 2).

    • HBM_DATA_WIDTH: width of the AXI4 data bus, 256 (card default) or 512 bits (NDK-APP-Minimal default).

  • AXI interface of each port:

    • Data width: HBM_DATA_WIDTH.

    • Address: 30 bits are used, i.e. 1 GB per port; each port has its own address space (32 GB in total with 32 ports).

    • Burst: with the 512-bit bus the IP runs in pseudo-BL8 mode, one access transfers 64 bytes (one 512-bit word). With the 256-bit bus the pseudo-BL8 mode is disabled and one access transfers 32 bytes (one 256-bit word).

  • Ordering: the NoC attached HBM does not contain a re-order buffer, see the warning below.

  • Clocks:

    • User (AXI) clock: 400 MHz, generated by the noc_pll IP in the HBM wrapper (HBM_CLK).

    • HBM memory clock: 1400 MHz (2800 MT/s).

  • Reset: each stack has its own HBM_FB2CDG1_RESET controller, which retries the calibration until it succeeds.

  • Theoretical throughput with 32 ports and the 512-bit bus:

    • AXI: 32 x 400 MHz x 512 b = 6553.6 Gbps in each direction.

    • DRAM: 32 x 64 b x 2800 MT/s = 5734.4 Gbps, shared by reads and writes. The AXI bus is wider than the DRAM, so the DRAM is the actual limit.

Note

Every NoC attached port has its own address space, so the port index is not encoded in the AXI address. This is why the card top level uses HBM_PORT_ADDR_HBIT = 30 (1 GB per port) and HBM_BASE_ADDR_OFFSET = 0, unlike the cards with a directly attached HBM stack.

Warning

The NoC attached HBM does not contain a reorder buffer. AXI4 only guarantees that the responses of transactions sharing one AXI ID are returned in the order in which the requests were issued; transactions with different IDs may complete in any order.

For this reason the HBM tester (<NDK-FPGA_root_directory>/comp/mem_tools/debug/hbm_tester) must be run without the --axi-id option on this card. It then issues every transaction with the same AXI ID and expects the read data in order. With --axi-id the responses are reordered, the data integrity test reports false errors on about 80 % of the words, and the throughput does not improve.

An application that accesses the HBM directly must respect the same rule: either reuse a single AXI ID, or be prepared to reorder the responses itself.

Measured HBM performance

Measured with the HBM tester on all 32 ports (NDK-APP-Minimal, 512-bit bus, 400 MHz, one AXI ID for all transactions, September 2026). Percentages are relative to the DRAM limit (5734.4 Gbps).

Test

Addressing

Throughput

Read only

sequential

3689 Gbps (64.3 %)

Read only

random

3217 Gbps (56.1 %)

Write only

sequential

3226 Gbps (56.3 %)

Write only

random

3142 Gbps (54.8 %)

Read and write

sequential

4668 Gbps (81.4 %)

Read and write

random

2248 Gbps (39.2 %)

The read and write rows show the sum of both directions, which share the DRAM bandwidth.

Board Variants

  • BOARD_VARIANT=1 - The board uses FPGA part number AGMF039R47A1E2VC (Production sample).

  • BOARD_VARIANT=0 - The board uses FPGA part number AGMF039R47A2E2VR0 (Engineering sample).

Initial Boot instructions

The card itself should be preloaded with firmware designed by Silicom. This design can be used to load an FPGA image to the BMC flash via PCIe using the Silicom rawcardtool tool.

Execute the following steps to perform the manual initialization:

  1. Obtain PCIe address of the card by the command below. In this example, the PCIe address is 01:00.0 and will be used in the next steps:

    $ lspci | grep "Silicom Denmark Device"
01:00.0 Ethernet controller: Silicom Denmark Device 0001

  2. Ensure the card respond to memory space accesses by command:

    $ setpci -s 01:00.0 COMMAND=0x0002

  3. Flash the fpga_page1_pacsign_user1.bin file to the card using RawCardTool provided by Silicom:

    $ rawcardtool_fjord --device 01:00.0 --flash fpga_page1_pacsign_user1.bin

  4. Set your design as default by command:

    $ rawcardtool_fjord --device 01:00.0 --fpgadefault user1

  5. Boot your design executing:

    $ rawcardtool_fjord --device 01:00.0 --reboot-fpga user1

Note

The fpga_page1_pacsign_user1.bin file will be located in the build folder of your application after building. For NDK-FPGA, the folder is ndk-fpga/apps/minimal/build/fb2cdg1.

Note

If some bars are unmapped, power cycle the server.

If the rawcardtool tool is unavailable, it is possible to use JTAG to load the design (SOF file), and nfb-boot command to load the BMC flash with the FPGA image:

$ nfb-boot -f0 fb2cdg1-minimal-pcie1xgen5x8x8-400g2.nfw