PCIe telemetry
- ENTITY PCIE_TELEMETRY_MI IS
The PCIE_TELEMETRY_MI collects performance data of the PCIe infrastructure of the NDK and makes it readable over the MI bus. It observes, for every PCIe endpoint, the four MFB buses around the PTC module and the sources that stop the PCIe transfer.
A BRAKE is one input signal of this module. It is active in every cycle in which one named reason stopped the data flow of the PTC.
What is measured
the PCIe settings negotiated with the host: maximum payload size (MPS), maximum read request size (MRRS), extended tag, 10-bit tag, read completion boundary (RCB) and link up,
the free PCIe tags of the PTC and the free words of its downstream storage FIFO. Each of them is reported twice, as the value now and as the lowest value reached,
per MFB bus: transferred words, transferred MFB items, started transactions and the cycles the bus was stalled by its consumer. The cycles in which each region carried data are counted separately,
the MVB items of the buses between the DMA and the PTC. These buses carry the headers of the transactions on a separate MVB,
the cycles in which a BRAKE signal stopped the transfer, one counter per reason. The reasons are those listed at the PCIE_PTC_BRAKE port. A firmware that does not report a reason leaves that counter at zero,
how long the PCIe tags and the storage FIFO words were short. Each of the two is a histogram of four bands, from exhausted to more than half free.
Every clock domain has its own counter of elapsed cycles, so every share can be computed from one measurement window. The counters of the PCIe clock and the BRAKE signals are measured on that same clock, so they describe the same cycles.
Structure
The module instantiates two PCIE_TELEMETRY_PROBE units per PCIe endpoint, one for the PCIe clock domain and one for the DMA clock domain. PCIE_TELEMETRY_ACC sums what the probes report into the wide counters that software reads.
Reading the counters
Reading all counters takes many MI transactions, so software must first issue the SNAPSHOT command. It copies all counters into a second memory. The counter window then returns that second memory. All values read after one SNAPSHOT therefore belong to the same moment.
Register map (byte offsets, all registers are 32-bit)
0x0000 RO MAGIC = 0x50544C4D 0x0004 RO VERSION [31:16] major, [15:0] minor 0x0008 RO CFG_TOPOLOGY [7:0] PCIe endpoints, [15:8] DMA ports per endpoint, [23:16] probes 0x000C RO CFG_LAYOUT [15:0] counters, [23:16] counter width, [31:24] counters per endpoint 0x0010 RO CFG_PCIE [7:0] channels, [15:8] buses, [23:16] regions of the probe, [31:24] BRAKE signals of the PCIe probe 0x0014 RO CFG_DMA the same for the DMA clock probe 0x0018 RO CFG_DRAIN [15:0] DRAIN_PERIOD in cycles, [23:16] bytes per MFB item 0x001C RO CNT_BASE byte offset of the counter window 0x0020 WO COMMAND bit0 snapshot, bit1 clear counters, bit2 clear flags, bit3 clear the lowest values in PCIE_TAGS and PCIE_STFIFO 0x0024 RO STATUS bit0 busy, bit1 a delta was lost, bit2 a read-out started too early 0x0028 RW READ_SEL bit0 0 = read snapshot copy, 1 = read live copy 0x002C RO CFG_REGIONS regions each bus really has, [7:0] RQ, [15:8] RC, [23:16] UP, [31:24] DOWN. A bus narrower than its probe is padded, its extra regions never carry data. 0x0030 RO CFG_WIDTH [15:0] bytes per region of the RQ bus, [31:16] bytes per region of the RC bus 0x003C RO CFG_WIDTH_DMA [15:0] bytes per region of the UP bus, [31:16] bytes per region of the DOWN bus 0x0034 RO CFG_HIST [7:0] bands per histogram, [15:8] histograms of the PCIe clock probe, [23:16] of the DMA clock probe. The bands are the last BRAKE channels of a probe, in the order of the signals they belong to: the tags first, then the storage FIFO. 0x0038 RO CFG_CAPACITY [15:0] PCIe tags, [31:16] storage FIFO words. The band thresholds are an eighth and a half of these. 0x0040 RO PCIE_STATUS one register per endpoint [2:0] MPS, [5:3] MRRS, [6] extended tag enable, [7] 10-bit tag enable, [8] RCB, [9] link up 0x0080 RO PCIE_TAGS one register per endpoint [15:0] free tags, [31:16] lowest free tags 0x00C0 RO PCIE_STFIFO one register per endpoint [15:0] free FIFO words, [31:16] lowest free words 0x8000 RO counters, two 32-bit words per counter, low word firstCounter numbering is described at PCIE_TELEMETRY_PROBE. Counters of one endpoint are stored together. The counters of its PCIe clock probe come first, those of its DMA clock probe second.
Generics
PortsGeneric
Type
Default
Description
PCIE_ENDPOINTS
natural
1
Number of monitored PCIe endpoints.
DMA_PORTS
natural
1
Number of DMA ports (PTC DMA side streams) per PCIe endpoint.
PCIE_RQ_MFB_REGIONS
natural
2
Number of MFB regions of the RQ bus, that is PTC to PCIe. The RQ and RC buses are measured together with the larger of the two values. Tie the unused SOF and EOF bits of the narrower bus to zero.
PCIE_RC_MFB_REGIONS
natural
2
Number of MFB regions of the RC bus, that is PCIe to PTC.
DMA_UP_MFB_REGIONS
natural
2
Number of MFB regions of the UP bus, that is DMA to PTC.
DMA_DOWN_MFB_REGIONS
natural
2
Number of MFB regions of the DOWN bus, that is PTC to DMA.
PCIE_RQ_MFB_REGION_BYTES
natural
32
Number of bytes in one MFB region of the RQ bus. The value is only reported to software and changes nothing in this module.
PCIE_RC_MFB_REGION_BYTES
natural
32
Number of bytes in one MFB region of the RC bus.
DMA_UP_MFB_REGION_BYTES
natural
32
Number of bytes in one MFB region of the UP bus.
DMA_DOWN_MFB_REGION_BYTES
natural
32
Number of bytes in one MFB region of the DOWN bus.
MFB_ITEM_BYTES
natural
4
Number of bytes in one MFB item. It converts the region sizes into item counts. The EOF_POS fields address items, not bytes.
PCIE_EOF_POS_WIDTH
natural
3
Width of one EOF_POS field of the PCIe side buses. It is the log2 of the item count of the larger region of the two.
DMA_EOF_POS_WIDTH
natural
3
The same for the DMA side buses.
CNT_WIDTH
natural
48
Width of the counters that software reads, allowed range is 33 to 64.
DRAIN_PERIOD
natural
4096
Number of cycles between two read-outs of the probe counters. It must be a power of two. A longer period costs more flip-flops in the fast clock domains, but it leaves more time for the MI clock domain.
TAG_CAPACITY
natural
256
Number of PCIe tags the PTC can use at the same time. It only sets the thresholds of the tag histogram, which are an eighth and a half of it.
STFIFO_CAPACITY
natural
1024
Number of words the PTC storage FIFO holds. It sets the thresholds of the storage FIFO histogram, which are an eighth and a half of it.
DEVICE
string
“AGILEX”
FPGA device
Port
Type
Mode
Description
=====
MI INTERFACE (MI_CLK)
=====
=====
MI_CLK
std_logic
in
MI_RESET
std_logic
in
MI_DWR
std_logic_vector(32-1 downto 0)
in
MI_ADDR
std_logic_vector(32-1 downto 0)
in
MI_BE
std_logic_vector(4-1 downto 0)
in
MI_RD
std_logic
in
MI_WR
std_logic
in
MI_DRD
std_logic_vector(32-1 downto 0)
out
MI_ARDY
std_logic
out
MI_DRDY
std_logic
out
=====
PCIE CLOCK DOMAIN OF EACH ENDPOINT
=====
=====
PCIE_CLK
std_logic_vector(PCIE_ENDPOINTS-1 downto 0)
in
PCIE_RESET
std_logic_vector(PCIE_ENDPOINTS-1 downto 0)
in
PCIE_MFB_SOF
slv_array_t(PCIE_ENDPOINTS-1 downto 0)(2*max(PCIE_RQ_MFB_REGIONS,PCIE_RC_MFB_REGIONS)-1 downto 0)
in
Observed buses, bus 0 = PTC to PCIe (RQ), bus 1 = PCIe to PTC (RC).
PCIE_MFB_EOF
slv_array_t(PCIE_ENDPOINTS-1 downto 0)(2*max(PCIE_RQ_MFB_REGIONS,PCIE_RC_MFB_REGIONS)-1 downto 0)
in
PCIE_MFB_EOF_POS
slv_array_t(PCIE_ENDPOINTS-1 downto 0)(2*max(PCIE_RQ_MFB_REGIONS,PCIE_RC_MFB_REGIONS)*PCIE_EOF_POS_WIDTH-1 downto 0)
in
Item of the region in which the frame ends, valid with PCIE_MFB_EOF.
PCIE_MFB_SRC_RDY
slv_array_t(PCIE_ENDPOINTS-1 downto 0)(2-1 downto 0)
in
PCIE_MFB_DST_RDY
slv_array_t(PCIE_ENDPOINTS-1 downto 0)(2-1 downto 0)
in
PCIE_PTC_BRAKE
slv_array_t(PCIE_ENDPOINTS-1 downto 0)(6-1 downto 0)
in
- BRAKE signals of the PTC, one bit per reason:
bit 0 = the pool of free PCIe tags is too small, bit 1 = the tag FIFO had no tag ready, although the pool is deep enough, bit 2 = no free space in the downstream storage FIFO, bit 3 = no free entry in the Completion Header buffer of the PCIe Hard IP, bit 4 = the MFB is held before the split between the DMA ports, bit 5 = the MVB is held before that same split.
Bits 0 to 3 stop the stream towards PCIe. Bits 4 and 5 stop the stream towards the DMA module. A firmware that does not report a reason leaves its bit at zero. Software reads the number of reasons from CFG_PCIE.
PCIE_STFIFO_FREE
slv_array_t(PCIE_ENDPOINTS-1 downto 0)(16-1 downto 0)
in
Free words of the PTC downstream storage FIFO.
PCIE_MPS
slv_array_t(PCIE_ENDPOINTS-1 downto 0)(3-1 downto 0)
in
Negotiated PCIe configuration.
PCIE_MRRS
slv_array_t(PCIE_ENDPOINTS-1 downto 0)(3-1 downto 0)
in
PCIE_EXT_TAG_EN
std_logic_vector(PCIE_ENDPOINTS-1 downto 0)
in
PCIE_10B_TAG_REQ_EN
std_logic_vector(PCIE_ENDPOINTS-1 downto 0)
in
PCIE_RCB_SIZE
std_logic_vector(PCIE_ENDPOINTS-1 downto 0)
in
PCIE_LINK_UP
std_logic_vector(PCIE_ENDPOINTS-1 downto 0)
in
=====
DMA CLOCK DOMAIN
=====
=====
DMA_CLK
std_logic
in
DMA_RESET
std_logic
in
DMA_MFB_SOF
slv_array_t(PCIE_ENDPOINTS-1 downto 0)(2*DMA_PORTS*max(DMA_UP_MFB_REGIONS,DMA_DOWN_MFB_REGIONS)-1 downto 0)
in
Observed buses, bus 2*p = DMA to PTC (UP), bus 2*p+1 = PTC to DMA (DOWN) of the DMA port p.
DMA_MFB_EOF
slv_array_t(PCIE_ENDPOINTS-1 downto 0)(2*DMA_PORTS*max(DMA_UP_MFB_REGIONS,DMA_DOWN_MFB_REGIONS)-1 downto 0)
in
DMA_MFB_EOF_POS
slv_array_t(PCIE_ENDPOINTS-1 downto 0)(2*DMA_PORTS*max(DMA_UP_MFB_REGIONS,DMA_DOWN_MFB_REGIONS)*DMA_EOF_POS_WIDTH-1 downto 0)
in
Item of the region in which the frame ends, valid with DMA_MFB_EOF.
DMA_MFB_SRC_RDY
slv_array_t(PCIE_ENDPOINTS-1 downto 0)(2*DMA_PORTS-1 downto 0)
in
DMA_MFB_DST_RDY
slv_array_t(PCIE_ENDPOINTS-1 downto 0)(2*DMA_PORTS-1 downto 0)
in
DMA_MVB_VLD
slv_array_t(PCIE_ENDPOINTS-1 downto 0)(2*DMA_PORTS*max(DMA_UP_MFB_REGIONS,DMA_DOWN_MFB_REGIONS)-1 downto 0)
in
The headers of the DMA side are carried on separate MVB buses. Every request has an item here, but only a request with payload also reaches the MFB.
DMA_MVB_SRC_RDY
slv_array_t(PCIE_ENDPOINTS-1 downto 0)(2*DMA_PORTS-1 downto 0)
in
DMA_MVB_DST_RDY
slv_array_t(PCIE_ENDPOINTS-1 downto 0)(2*DMA_PORTS-1 downto 0)
in
PCIE_TAG_FREE
slv_array_t(PCIE_ENDPOINTS-1 downto 0)(11-1 downto 0)
in
Number of free PCIe tags in the PTC, on the PCIe clock. The PTC also provides this value resynchronised to the DMA clock. It is measured here so that it shares the clock with the BRAKE signal that becomes active when the tags run out.
Telemetry probe
- ENTITY PCIE_TELEMETRY_PROBE IS
The PCIE_TELEMETRY_PROBE observes several MFB buses and several BRAKE signals in one fast clock domain. It reports the measured event counts to a slow clock domain. A BRAKE signal is active in every cycle in which one named reason stopped the data flow.
The probe keeps one narrow counter per telemetry channel. Every DRAIN_PERIOD cycles all counters are read out and cleared. Each read value is called a delta. The deltas are sent one by one into an asynchronous FIFO. The deltas are summed into wide counters in the slow clock domain, so this module needs only narrow counters. A counter cannot overflow between two read-outs, so no event is lost.
The path from the observed signals to the counters has three register stages: registered inputs, computed increments and the counters. The read-out path adds two more, the group register and the FIFO input register.
Channel numbering, which also gives the order of the counters in memory:
0– elapsed CLK cycles,1 + b*(6+REGIONS) + 0– words transferred on busb,1 + b*(6+REGIONS) + 1– cycles busbwas stalled by its consumer,1 + b*(6+REGIONS) + 2– transactions (SOF) started on busb,1 + b*(6+REGIONS) + 3– items transferred on the MVB bus that belongs to busb. Where the headers are carried on a separate MVB bus, every transaction has an item here. Only a transaction with payload also appears on the MFB. Buses with no MVB leave this counter at zero.1 + b*(6+REGIONS) + 4– cycles the MVB bus of busbwas stalled by its consumer. The MVB has a handshake of its own, so it can be stalled while the MFB of the same bus runs. Buses with no MVB leave this at zero.1 + b*(6+REGIONS) + 5– MFB items transferred on busb. A region that ends a frame counts the items up to RX_EOF_POS. Every other region of an open frame counts all of its items.1 + b*(6+REGIONS) + 6 + r– cycles regionrof busbcarried data,1 + BUSES*(6+REGIONS) + k– cycles the BRAKE signalkwas active.
PortsGeneric
Type
Default
Description
BUSES
natural
2
Number of observed MFB buses.
REGIONS
natural
2
Number of MFB regions of each observed bus. Buses with fewer regions must have the unused SOF/EOF bits tied to zero.
REGION_ITEMS
i_array_t(BUSES-1 downto 0)
(others => 8)
Number of MFB items in one region of each observed bus. A region must hold a single block, so that a frame always starts at its first item.
EOF_POS_WIDTH
natural
3
Width of one RX_EOF_POS field, the log2 of the largest REGION_ITEMS.
REGION_ITEMS_MAX
natural
8
The largest REGION_ITEMS of any observed bus. It sizes the counters. The port clause needs the value as a plain generic, because it cannot search the REGION_ITEMS array.
BRAKES
natural
2
Number of observed BRAKE signals.
DRAIN_PERIOD
natural
4096
Number of CLK cycles between two read-outs of the counters. It must be a power of two. It also sets the counter width CNT_WIDTH.
DEVICE
string
“AGILEX”
FPGA device
Port
Type
Mode
Description
=====
OBSERVED CLOCK DOMAIN
=====
=====
CLK
std_logic
in
RESET
std_logic
in
RX_SOF
std_logic_vector(BUSES*REGIONS-1 downto 0)
in
Start of frame of each region of each observed bus.
RX_EOF
std_logic_vector(BUSES*REGIONS-1 downto 0)
in
End of frame of each region of each observed bus.
RX_EOF_POS
std_logic_vector(BUSES*REGIONS*EOF_POS_WIDTH-1 downto 0)
in
Item of the region in which the frame ends, valid with RX_EOF. The fields keep the order of RX_EOF. Unused high bits stay at zero.
RX_SRC_RDY
std_logic_vector(BUSES-1 downto 0)
in
RX_DST_RDY
std_logic_vector(BUSES-1 downto 0)
in
RX_BRAKE
std_logic_vector(max(1,BRAKES)-1 downto 0)
in
BRAKE signals. Each bit is active while the reason it reports blocks the data.
RX_MVB_VLD
std_logic_vector(BUSES*REGIONS-1 downto 0)
in
Valid items of the MVB bus that belongs to each observed MFB bus, and its handshake. Tie SRC_RDY low on buses that have no MVB.
RX_MVB_SRC_RDY
std_logic_vector(BUSES-1 downto 0)
in
RX_MVB_DST_RDY
std_logic_vector(BUSES-1 downto 0)
in
DRAIN_TICK
std_logic
out
Active for one CLK cycle at the start of every read-out of the counters.
=====
REPORTING CLOCK DOMAIN
=====
=====
MI_CLK
std_logic
in
MI_RESET
std_logic
in
TX_INDEX
std_logic_vector(log2(1+BUSES*(6+REGIONS)+BRAKES)-1 downto 0)
out
Channel index of the reported delta.
TX_DELTA
std_logic_vector(log2(DRAIN_PERIOD*max(1,REGIONS*REGION_ITEMS_MAX)+1)-1 downto 0)
out
Number of events counted on the channel since the previous read-out.
TX_VLD
std_logic
out
TX_RD
std_logic
in
MI_FLAG_CLR
std_logic
in
Every change of this level clears both flags below.
TX_FIFO_OVF
std_logic
out
Set when the FIFO was full and a delta was lost. Stays set until the level of MI_FLAG_CLR changes.
TX_OVERRUN
std_logic
out
Set when a read-out started before the previous one finished. Stays set until the level of MI_FLAG_CLR changes.
Telemetry counter memory
- ENTITY PCIE_TELEMETRY_ACC IS
The PCIE_TELEMETRY_ACC merges the delta streams of all telemetry probes into one memory of wide free-running counters. The deltas arrive one by one, so a single adder and a single memory are enough for all channels.
The memory holds two copies of the counters. The deltas update the live copy. The SNAPSHOT command writes the snapshot copy. Software reads the snapshot copy, so all values it reads belong to the same moment, even though reading them takes many bus transactions.
Deltas keep arriving while a SNAPSHOT or CLEAR command runs. They wait in the probe FIFOs and are applied when the command is done.
Generics
PortsGeneric
Type
Default
Description
STREAMS
natural
2
Number of delta streams (probes).
STREAM_BASE
i_array_t
(0,16)
First counter index of each stream.
ENTRIES
natural
64
Number of counters. It must be a power of two.
IDX_WIDTH
natural
5
Width of the channel index of a delta.
DELTA_WIDTH
natural
14
Width of the value of a delta.
CNT_WIDTH
natural
48
Width of the counters in the memory.
Port
Type
Mode
Description
CLK
std_logic
in
RESET
std_logic
in
=====
DELTA STREAMS
=====
=====
RX_INDEX
slv_array_t(STREAMS-1 downto 0)(IDX_WIDTH-1 downto 0)
in
RX_DELTA
slv_array_t(STREAMS-1 downto 0)(DELTA_WIDTH-1 downto 0)
in
RX_VLD
std_logic_vector(STREAMS-1 downto 0)
in
RX_RD
std_logic_vector(STREAMS-1 downto 0)
out
=====
COMMANDS
=====
=====
SNAPSHOT
std_logic
in
Copies the live counters into the snapshot copy.
CLEAR
std_logic
in
Zeroes both copies of the counters.
BUSY
std_logic
out
A command is in progress. Deltas are not applied while it runs.
=====
SOFTWARE READ PORT
=====
=====
RD_ADDR
std_logic_vector(log2(ENTRIES)+1-1 downto 0)
in
The highest bit selects the copy, 0 = live, 1 = snapshot.
RD_EN
std_logic
in
RD_DATA
std_logic_vector(CNT_WIDTH-1 downto 0)
out
Software
The component is read out by the nfb-pcie-telemetry tool from the OFM
Python package. The tool takes its whole layout from the configuration registers
of the component. The same binary therefore works with any number of PCIe
endpoints, DMA ports and MFB regions, without being rebuilt.
Installation
The tool needs Python 3.11 or newer and the nfb module from the NDK-SW
repository. Install the OFM package from the python/ofm directory:
python3.11 -m venv my-venv
source my-venv/bin/activate
source <ndk-fpga>/env.sh
cd <ndk-fpga>/python/ofm
pip install .
Use pip install -e . instead while you modify the tool itself. It installs
the package in editable mode.
Usage
Print the traffic measured over one second and exit:
nfb-pcie-telemetry
Options:
-d, --device PATH path to the NFB device (default: the first one)
-i, --index N index of the component in the DeviceTree (default: 0)
-t, --interval SEC length of the measured interval in seconds; 0 prints
the totals accumulated since the last clear
-w, --watch keep measuring and printing until interrupted
-c, --clear zero all counters and exit
-j, --json print the raw values as JSON instead of a table
With a non-zero interval the tool takes two snapshots and prints their
difference, so the report describes only that interval. With --interval 0
it prints one snapshot, which is the traffic since the counters were last
cleared. The counters are wide enough (48 bits by default) that they do not
overflow in any practical measurement.
Example of the output:
PCIe telemetry - 1 endpoint(s), 1 DMA port(s) per endpoint
==========================================================
Measured over 1.00 s
PCIe endpoint 0
------------------------------------------------------------
Link up
Max payload size (MPS) 512 B
Max read request (MRRS) 4096 B
Extended tag (8-bit) on
10-bit tag off
Read completion boundary 64 B
PTC tags free 118 (lowest 3)
PTC storage FIFO free 640 words (lowest 96)
PCIe clock 400.0 MHz, DMA clock 200.0 MHz
Bus Used % Words % MFB stall % MVB stall % Used per region % Gbps Mpps RD % Avg B
------------------ ------ ------- ----------- ----------- ------------------- ------ ------ ----- -----
PTC -> PCIe (RQ) 70.15 89.90 0.01 - 89.9 63.6 63.6 63.5 287.33 58.000 - 619
PCIe -> PTC (RC) 44.67 49.40 0.00 - 44.8 44.7 44.6 44.6 182.99 34.000 - 673
DMA0 -> PTC (UP) 66.45 66.90 22.70 13.90 66.7 66.2 68.04 58.000 25.86 198
PTC -> DMA0 (DOWN) 45.45 48.60 0.80 0.30 45.5 45.4 46.54 34.800 - 167
Why the PCIe transfer stopped (share of PCIe clock cycles)
UP stream DOWN stream
no free PCIe tag 13.60 % MFB stalled towards DMA 8.40 %
PCIe tag not ready 0.40 % MVB stalled towards DMA 0.10 %
no room for completions 0.00 %
no completion header entry 0.00 %
PCIe endpoint not ready 0.01 %
Time spent low of none <=1/8 <=1/2 rest
---------------------- ---- ------ ------- ------- ------
PCIe tags 256 0.30 39.80 7.90 52.00
PTC storage FIFO words 1024 0.00 21.00 27.00 52.00
Reading the report
The header of each endpoint shows the PCIe parameters negotiated with the host (MPS, MRRS, extended and 10-bit tags, read completion boundary). It also shows how many PTC tags and how many words of the PTC storage FIFO are free now, and the lowest value each of them reached in the measured interval. Read that lowest value first when the throughput is lower than expected. The histograms below then say how long the resource stayed that low.
The clock frequencies are not read from a register. They are derived from the cycle counters and the wall-clock length of the interval. They therefore also serve as a sanity check that the measurement really covered the interval it claims to.
Meaning of the columns:
- Used %
Occupancy of the bus, that is the sum of its region counters divided by the elapsed cycles times the regions of that bus. A region counts in a cycle in which the bus moved a word and a frame was open in that region. The whole region counts even when the frame ends in the middle of it, so this is the width the traffic occupied and not the width it filled.
- Words %
Share of clock cycles in which the bus moved a word, that is both SRC_RDY and DST_RDY were high. The denominator is every elapsed cycle, so 100 % says the bus never idled and was never stalled.
- MFB stall %
Share of clock cycles in which SRC_RDY was high and DST_RDY was low. What is left of 100 % after this column and Words % is the time the source had nothing to send.
- MVB stall %
The same for the MVB that carries the headers. The MVB has a handshake of its own, so it can be stalled while the MFB of the same bus runs. Buses without an MVB show a dash.
- Used per region %
The counter of one region divided by the elapsed cycles, region 0 first. Values that decrease towards the higher regions mean short transactions. Such transactions do not fill the whole word.
- Gbps
Rate of the transferred MFB items times the item size. A region that ends a frame contributes only the items up to its EOF_POS, so this is the length the traffic really had. The two sides of the PTC count different things. Between the DMA and the PTC the headers are carried on a separate MVB bus, so the DMA side buses carry payload alone. On the PCIe request bus the TLP header is part of the data stream and is counted in. The difference between the two sides is therefore the header overhead.
- Mpps
Transaction rate, millions of transactions per second. The DMA side buses have an MVB of their own, so their rate comes from the MVB items and holds every request including those that carry no payload. The PCIe side buses have no MVB and their frames are counted instead. The choice follows the bus, never the traffic, so it cannot change between two readings.
- RD %
Share of the requests that carry no payload. On the bus from the DMA module to the PTC these are the reads, and the writes are the remainder. Only that bus carries both kinds of request, so the others show a dash. A completion always has payload. On the PCIe side the header is part of the data stream, where these counters cannot tell a read from a write.
- Avg B
Transferred bytes divided by the frames on the MFB, that is the average length of one frame. Mpps counts requests rather than frames, so these two columns do not multiply out to Gbps.
The block at the bottom says how often the PTC had to stop a stream and why. The UP stream column holds the reasons that stop the read requests on their way to PCIe:
the pool of free PCIe tags was smaller than what that cycle asked for,
the tag FIFO had no tag ready, even though the pool was deep enough,
the space left in the storage FIFO no longer covered the completions of the next group of requests,
the Completion Header buffer of the PCIe Hard IP had no entry left for one more read,
the PCIe endpoint itself stopped taking data.
The DOWN stream column holds the reasons that stop the completions on their way to the DMA module. Both are measured before the split between the DMA ports. One is measured on the MFB that carries the data. The other is measured on the MVB that carries the headers. They become active before the MFB stall % of a DOWN bus does. That column is measured on the far side of the clock crossing, where the asynchronous FIFOs have already taken the burst. A DOWN reason is the usual cause of no room for completions in the other column.
The first two reasons both come from the tag path and are counted apart, because they ask for opposite changes. The first one means the completions do not come back fast enough. It is answered by more tags or by shorter reads. The second one is a property of the tag FIFO, not of the traffic. A released tag counts as free as soon as it is written, but it reaches the read ports of that FIFO only a few cycles later. The second reason therefore says nothing about how many tags are left. It appears even when almost the whole pool is free. Use the histogram below, not this line, to decide whether the tags are running out.
The third one needs care when it is read. The storage FIFO is not full when this reason becomes active, because the space is only reserved ahead. The reason therefore appears well before the histogram reaches its lowest band. The fourth one covers a buffer inside the PCIe Hard IP for which no credit is counted. It stays at zero on endpoint types that manage that buffer themselves. The last one is not a counter of its own. It repeats the stalled cycles of the RQ bus, so that every reason can be compared in one place.
Every reason is its own counter of cycles. It is divided by the elapsed PCIe clock cycles of the same period as the rest of the report. The second line of the header names that period: the measured interval, or everything since the last clear.
The two histograms below them say how long each of the two resources that can
run out was short. The lowest value in the header cannot say this: four free tags
means the same whether it lasted one cycle or half a second.
Each histogram splits the time into four bands by how much of the resource was
free: none at all, up to an eighth, up to a half, and the rest. The bands are
narrowest near zero, because that is where the resource limits the transfer. The
of column gives the capacity the bands are shares of. The band thresholds are
derived from that same number. Each band is a counter of cycles divided by the
elapsed PCIe clock cycles. Time spent in the lowest band is time in which the
PTC was close to a stall. A few percent there is worth reading even when the
average over the whole interval shows no shortage.
Both resources are counted on the PCIe clock. The reasons they cause are observed on that same clock. The share of time in the lowest band and the share of time the matching reason was active therefore describe the same cycles, and the two can be compared.
If the counters could not keep up, the tool prints a LOST DELTAS or a
READ-OUT OVERRUN warning at the top of the report. The numbers must not be
trusted then. The firmware keeps both flags set until software clears them. The
tool clears them at the start of every measured interval, so the warning
describes only the interval that was just printed. With --interval 0 there
is no such start, so the flags describe everything since the last --clear.
Python API
The same data is available to scripts:
from ofm.comp.pcie.telemetry import PcieTelemetry, format_report
telemetry = PcieTelemetry()
data = telemetry.read() # takes a snapshot and returns a dict
print(format_report(data, 1.0)) # or render it the same way the tool does
telemetry.clear() # zero all counters
print(telemetry.status) # busy, lost_deltas, readout_overrun
read() first takes a snapshot in the firmware and only then reads it out, so
the returned values are consistent with each other even though reading them
takes many MI transactions.