HBM Tester

The HBM_TESTER component validates HBM (High Bandwidth Memory) throughput, latency and data integrity. It stress-tests memory controllers and physical layers using both linear and pseudorandom access patterns.

One tester is instantiated per physical HBM module, each in the clock and reset domain of that module. The MI bus is bridged into the HBM clock domain by the internal MI_ASYNC component, so the tester can be controlled from the application clock domain.

Hardware Configuration

• Clock and Reset

  • One HBM module has a single frequency and reset domain.

  • If the design has several HBM modules, instantiate a dedicated tester for each.

• PORT_ADDR_HBIT – capacity (addressing range) of one port.

  • 28 for 256 MB, 29 for 512 MB, 30 for 1 GB.

• BASE_ADDR_OFFSET – size of one port in bytes, that is the step between the base addresses of two neighbouring ports. It must be a multiple of 2**PORT_ADDR_HBIT. The base address of a port is PORT_ID * BASE_ADDR_OFFSET.

  • Use 0 when every port has its own address space, for example a NoC attached HBM.

  • For a 256 MB port packed into one map, use 0x10000000.

• CNT_WIDTH – width of the test duration counter. The maximum duration is 2**CNT_WIDTH clock cycles.

Principle of Operation

The HBM_TESTER_GEN module acts as a traffic generator and data integrity checker.

Address Generation

• Sequential mode: addresses increment linearly. The stride follows the burst length (BL4 or BL8) so that the generated traffic stays contiguous.

• Pseudorandom mode: a Linear Feedback Shift Register (LFSR) generates the addresses. Useful for worst-case latency, because it stops the HBM controller from consistently hitting open rows.

Data Generation and Integrity

• Write path: controlled by CS_GEN_WR_DEAD. With 0 the word holds an incrementing counter in the first 8 bits and zeros in the rest. With 1 the whole word is the constant 0xDEADCAFE and no counter is written.

• Read path and verification: the tester maintains an internal expected counter and compares it with the incoming RD_DATA(7 downto 0).

• Statistics: STAT_DATA_OK_INC pulses when the read data matches the expected counter, STAT_DATA_ERR_INC pulses on a mismatch.

Note

Valid checking requires a strict “write then read” sequence.

Read after Write

The AXI write and read channels are not ordered against each other. A read issued right after a write to the same address may therefore be served first and return the old data, which is what the coherency test looks for. The effect grows with the latency of the memory path; on a NoC attached HBM the read wins often.

To keep this from firing on its own traffic, the generator only enables the read address once the write response of the previous burst has arrived (s_wr_rsp_pending in HBM_TESTER_GEN). This applies to the R/W switching mode only and costs the full write response latency between each write and its read, so that mode is not a bandwidth measurement. Config bit [7] (-w on the command line) turns the wait off, which lets the test measure how often a read overtakes its write on a given memory path.

Traffic Control

• Write-only / Read-only: continuous streams of a single transaction type.

• R/W switching: with CS_GEN_RW_SWITCH enabled the module toggles between writing a burst and reading an address, simulating bidirectional traffic.

• Burst support: BL8 (64B access via two 256b words) and BL4 (32B access).

Warning

The burst is counted in bus words, so these sizes only hold on a 256b bus. On a 512b bus the same encoding issues 128B and 64B, and a real 32B access cannot be generated at all.

Component port and generics description

ENTITY HBM_TESTER IS

Ports

Port	Type	Mode	Description
=====	CLOCK AND RESET	=====	=====
HBM_CLK	std_logic	in
HBM_RESET	std_logic	in
HBM_INIT_DONE	std_logic	in
=====	COMMON MI32 INTERFACE (MI_CLK)	=====	=====
MI_CLK	std_logic	in
MI_RESET	std_logic	in
MI_DWR	std_logic_vector(31 downto 0)	in
MI_ADDR	std_logic_vector(31 downto 0)	in
MI_BE	std_logic_vector(3 downto 0)	in
MI_RD	std_logic	in
MI_WR	std_logic	in
MI_ARDY	std_logic	out
MI_DRD	std_logic_vector(31 downto 0)	out
MI_DRDY	std_logic	out
=====	USER HBM SIMPLE INTERFACE (HSI) PER HBM PORT (HBM_CLK)	=====	=====
WR_ADDR	slv_array_t(PORTS-1 downto 0)(AXI_ADDR_WIDTH-1 downto 0)	in	WRITE ADDR/DATA SIGNALS
WR_DATA	slv_array_t(PORTS-1 downto 0)(USR_DATA_WIDTH-1 downto 0)	in
WR_DATA_LAST	std_logic_vector(PORTS-1 downto 0)	in
WR_VALID	std_logic_vector(PORTS-1 downto 0)	in
WR_READY	std_logic_vector(PORTS-1 downto 0)	out
WR_RSP_ACK	std_logic_vector(PORTS-1 downto 0)	out	WRITE RESPONSE SIGNALS
WR_RSP_VALID	std_logic_vector(PORTS-1 downto 0)	out
WR_RSP_READY	std_logic_vector(PORTS-1 downto 0)	in
RD_ADDR	slv_array_t(PORTS-1 downto 0)(AXI_ADDR_WIDTH-1 downto 0)	in	READ ADDRESS SIGNALS
RD_ADDR_VALID	std_logic_vector(PORTS-1 downto 0)	in
RD_ADDR_READY	std_logic_vector(PORTS-1 downto 0)	out
RD_DATA	slv_array_t(PORTS-1 downto 0)(USR_DATA_WIDTH-1 downto 0)	out	READ DATA SIGNALS
RD_DATA_LAST	std_logic_vector(PORTS-1 downto 0)	out
RD_DATA_VALID	std_logic_vector(PORTS-1 downto 0)	out
RD_DATA_READY	std_logic_vector(PORTS-1 downto 0)	in
=====	AXI INTERFACE PER HBM PORT (HBM_CLK)	=====	=====
AXI_AWID	slv_array_t(PORTS-1 downto 0)(AXI_ID_WIDTH-1 downto 0)	out	WRITE ADDRESS SIGNALS
AXI_AWADDR	slv_array_t(PORTS-1 downto 0)(AXI_ADDR_WIDTH-1 downto 0)	out
AXI_AWLEN	slv_array_t(PORTS-1 downto 0)(AXI_LEN_WIDTH-1 downto 0)	out
AXI_AWSIZE	slv_array_t(PORTS-1 downto 0)(AXI_SIZE_WIDTH-1 downto 0)	out
AXI_AWBURST	slv_array_t(PORTS-1 downto 0)(AXI_BURST_WIDTH-1 downto 0)	out
AXI_AWPROT	slv_array_t(PORTS-1 downto 0)(3-1 downto 0)	out
AXI_AWQOS	slv_array_t(PORTS-1 downto 0)(4-1 downto 0)	out
AXI_AWUSER	slv_array_t(PORTS-1 downto 0)(1-1 downto 0)	out
AXI_AWVALID	std_logic_vector(PORTS-1 downto 0)	out
AXI_AWREADY	std_logic_vector(PORTS-1 downto 0)	in
AXI_WDATA	slv_array_t(PORTS-1 downto 0)(AXI_DATA_WIDTH-1 downto 0)	out	WRITE DATA SIGNALS
AXI_WSTRB	slv_array_t(PORTS-1 downto 0)(AXI_DATA_WIDTH/8-1 downto 0)	out
AXI_WUSER_DATA	slv_array_t(PORTS-1 downto 0)(AXI_DATA_WIDTH/8-1 downto 0)	out
AXI_WUSER_STRB	slv_array_t(PORTS-1 downto 0)(AXI_DATA_WIDTH/64-1 downto 0)	out
AXI_WLAST	std_logic_vector(PORTS-1 downto 0)	out
AXI_WVALID	std_logic_vector(PORTS-1 downto 0)	out
AXI_WREADY	std_logic_vector(PORTS-1 downto 0)	in
AXI_BID	slv_array_t(PORTS-1 downto 0)(AXI_ID_WIDTH-1 downto 0)	in	WRITE RESPONSE SIGNALS
AXI_BRESP	slv_array_t(PORTS-1 downto 0)(AXI_RESP_WIDTH-1 downto 0)	in
AXI_BVALID	std_logic_vector(PORTS-1 downto 0)	in
AXI_BREADY	std_logic_vector(PORTS-1 downto 0)	out
AXI_ARID	slv_array_t(PORTS-1 downto 0)(AXI_ID_WIDTH-1 downto 0)	out	READ ADDRESS SIGNALS
AXI_ARADDR	slv_array_t(PORTS-1 downto 0)(AXI_ADDR_WIDTH-1 downto 0)	out
AXI_ARLEN	slv_array_t(PORTS-1 downto 0)(AXI_LEN_WIDTH-1 downto 0)	out
AXI_ARSIZE	slv_array_t(PORTS-1 downto 0)(AXI_SIZE_WIDTH-1 downto 0)	out
AXI_ARBURST	slv_array_t(PORTS-1 downto 0)(AXI_BURST_WIDTH-1 downto 0)	out
AXI_ARPROT	slv_array_t(PORTS-1 downto 0)(3-1 downto 0)	out
AXI_ARQOS	slv_array_t(PORTS-1 downto 0)(4-1 downto 0)	out
AXI_ARUSER	slv_array_t(PORTS-1 downto 0)(1-1 downto 0)	out
AXI_ARVALID	std_logic_vector(PORTS-1 downto 0)	out
AXI_ARREADY	std_logic_vector(PORTS-1 downto 0)	in
AXI_RID	slv_array_t(PORTS-1 downto 0)(AXI_ID_WIDTH-1 downto 0)	in	READ DATA SIGNALS
AXI_RDATA	slv_array_t(PORTS-1 downto 0)(AXI_DATA_WIDTH-1 downto 0)	in
AXI_RUSER_DATA	slv_array_t(PORTS-1 downto 0)(AXI_DATA_WIDTH/8-1 downto 0)	in
AXI_RUSER_ERR_DBE	std_logic_vector(PORTS-1 downto 0)	in
AXI_RRESP	slv_array_t(PORTS-1 downto 0)(AXI_RESP_WIDTH-1 downto 0)	in
AXI_RLAST	std_logic_vector(PORTS-1 downto 0)	in
AXI_RVALID	std_logic_vector(PORTS-1 downto 0)	in
AXI_RREADY	std_logic_vector(PORTS-1 downto 0)	out

Control Software

The hbm_tester.py script in the sw directory controls the hardware.

Warning

The script is not self-configuring and has to be told the geometry of your build. It does not read the geometry of the tester from the firmware or from the Device Tree, it assumes 32 ports per tester instance, a 256b data bus and a 450 MHz HBM_CLK. When these defaults do not match the design, pass the correct values with -P, -W and -F, otherwise the script reports wrong numbers without any warning, or accesses the registers of ports that do not exist.

One hbm_tester object is created per Device Tree node, that is per HBM module, and the values given on the command line are used for all of them. A design whose modules do not have the same geometry is not supported by the script.

Tests

Test	What it does
speed	Writes and reads continuously and reports the throughput of each port.
speed-rd	Same, but the generator only reads. The write counter stays at zero, so the read figure is what a port reaches with the whole bus to itself.
speed-wr	Same, but the generator only writes.
latency	Same traffic, but reports the average read and write latency. Use -r for the worst case, it stops the controller from hitting open rows.
integrity	Writes the counter to all addresses, then reads them back and checks. Answers “does what I wrote come back?”.
coherency	Fills the memory with 0xDEADCAFE first, then writes the counter to an address and reads that same address straight back, alternating. The read has to return the counter, a returned 0xFE means it overtook its own write. Answers “does a read right after a write see the new data?”.

Command Line Arguments

Argument	Description
-i, --index	Index of the HBM tester (e.g. 0, 1), or all to run on all modules.
-d, --device	Device index (default: 0 for /dev/nfb0).
-t, --test	Type of test: speed, speed-rd, speed-wr, latency, integrity or coherency.
-r, --random	Enable random addressing (for latency or speed tests).
-w, --no-wait	Do not wait for the write response before reading the same address (coherency test).
-b, --bl8	Use the BL8 burst mode, that is a 64B access (default: BL4, a 32B access).
-I, --axi-id	Give each transaction its own AXI ID, the default is one ID for all of them.
-p, --ports	Number of active ports/channels (default: all).
-P, --tester-ports	Number of ports of one tester instance, that is HBM_PORTS / HBM_MODULES (default: 32).
-W, --data-width	The HBM_DATA_WIDTH of the build, 256 or 512 (default: 256). A wrong value scales every reported speed by the same factor.
-F, --freq	Frequency of HBM_CLK in MHz (default: 450). The counters are converted to seconds with it, so it scales both the speed and the latency results.

Register Map

Offset	Name	Description
0x00	Version	HBM_TESTER version.
0x10	Port Enable	One-hot bitmask to enable/run specific ports.
0x14	Config	Configuration register (see the bitfield below).
0x18	Reset	Writing to this register resets the tester logic.
0x1C	Duration	Test duration (limit defined by CNT_WIDTH).
0x20	Status	Monitoring done / stats ready of each port (one-hot address).
0x200	Stats 0	Base address for counter 0.
0x204	Stats 1	Base address for counter 1.

Configuration Register (0x14) Bitfield

Bits	Function	Description
[0]	Connection	0 = user source, 1 = generator source.
[1]	Addr Mode	0 = sequential, 1 = pseudorandom.
[2]	Dead Data	0 = counter value, 1 = 0xDEADCAFE.
[3]	R/W Switch	Enable automatic R/W toggling.
[5:4]	Run Mode	00 = none, 01 = WR only, 10 = RD only, 11 = RD and WR.
[6]	Burst Mode	Size of one access: 0 = BL4 (32B), 1 = BL8 (64B). The number of bus words the access takes is derived from HBM_DATA_WIDTH. A bus wider than 32B cannot generate BL4 and issues the 64B of BL8 in both modes.
[7]	R/W No Wait	0 = wait for the write response before reading the same address, 1 = read immediately. See Read after Write.
[16]	AXI ID Mode	0 = one ID for all transactions, 1 = a new ID for each one. With one ID, AXI4 guarantees the order of the responses within a channel. It does not guarantee the order between the write and the read channel, see Read after Write. A new ID for each transaction needs a reorder buffer in the memory or its IP, otherwise the responses come back in a different order than the tester expects and the data integrity test reports false errors. The NoC attached HBM on ThunderFjord (fb2cdg1) has no reorder buffer, so leave the bit at 0 there.
[10:8]	Counter 0 Mode	Selects the metric for result counter 0 (see below).
[14:12]	Counter 1 Mode	Selects the metric for result counter 1 (see below).

Counter modes

• 000 – read words count

• 001 – write words count

• 010 – read latency

• 011 – write latency

• 100 – data OK count

• 101 – data error count