NDP RX synchronization
When receiving data (RX DMA direction), the driver first tells the DMA Module, where in the computer’s RAM the data can be stored. After doing so, the DMA Module informs the driver about how much data has been received. The driver passes the data to the user and passes more space to the DMA Module.
Function call map
This section contains a cheatsheet of how functions call each other when using DPDK, Libnfb and NDP driver.
Each of these parts uses different structure to contain a block of memory space for packet.
Between the DMA Module and the NDP driver “descriptors” are used.
Descriptors are optimized for minimal PCI overhead.
The NDP driver and the Libnfb pass these infromation using the “Header Buffer” and “Offset Buffer” (the two buffers share control and thus function as a single buffer).
The buffers and the pointers to these buffers are accessible both to the NDP driver and (through vmap) to the Libnfb.
This is basically the only way these two sides communicate.
Otherwise they run independently in parallel.
The Libnfb comunicates with the user using the “ndp_packet” structure. In DPDK memory blocks are managed using the structure called “Mbuf”.
RX
1 rte_eth_rx_burst (dpdk/lib/librte_ethdev/rte_ethdev.h)
2 -+
3 |
4 V
5 nfb_eth_ndp_rx (dpdk/drivers/net/nfb/nfb_rx.h)
6 - is set as dev->rx_pkt_burst
7 - main RX receive function
8 ==============-+
9 |
10 V
11 (nc_)ndp_(v2_)rx_burst_get (swbase/libnfb/include/netcope/ndp_rx.h)
12 - takes values from headers in the Offset Buffer and copies them to an ndp_packet
13 - shifts rhp
14 ==========-+
15 |
16 V
17 ndp_rx_fill_mbuf (dpdk/drivers/net/nfb/nfb_rx.h)
18 - copies information from ndp_packet to an Mbuf Buffer
19 - copies each Mbuf to output Mbuf array (from here it will be freed by the user)
20 -+
21 |
22 V
23 ndp_rx_fill_desc (dpdk/drivers/net/nfb/nfb_rx.h)
24 - allocates a requested amount of Mbufs in the Mbuf Buffer
25 - fills ndp_packets with pointers to the Mbufs' data
26 -+
27 |
28 V
29 (nc_)ndp_rx_burst_put_desc (swbase/libnfb/include/netcope/ndp_rx.h)
30 - checks the amount of free space in the Offset Buffer
31 - copiest info from ndp_packets to Header and Offset Buffer
32 - shifts sync.swptr
33
34 ndp_channel_rxsync (drivers/kernel/drivers/nfb/ndp/channel.c)
35 -+
36 |
37 V
38 ndp_ctrl_rx_set_swptr (swbase/drivers/kernel/drivers/nfb/ndp/ctrl_ndp.c)
39 - is set as ndp_ctrl_rx_ops.set_swptr
40 - shifts shp
41 - pro SIMPLE mod vytvori deskriptory do deskriptoroveho bufferu
42 - in the *Simple* mode creates new descriptors in the Header and Offset Buffers
43 - in the *User* mode:
44 -+
45 |
46 V
47 ndp_ctrl_user_fill_rx_descs (swbase/drivers/kernel/drivers/nfb/ndp/ctrl_ndp.c)
48 - checks the number of new descritpors in the Header and Offset Buffer
49 - creates new descriptors in the HW descirptor buffer
50 - shifts SDP and propagates it to the HW
51 ndp_ctrl_rx_get_hwptr (swbase/drivers/kernel/drivers/nfb/ndp/ctrl_ndp.c)
52 - is set as ndp_ctrl_rx_ops.get_hwptr
53 - reads hhp
54 - sets sync.hwptr accordingly propagatng it higher
55 - if there are items in the Header and Offset Buffer waiting to be propagated as descriptors it does so
Header and Offset Buffer
To better understand the function of individual pointer in the Header and Offset Buffer, here is a cheatsheet for that as well.
RX
1 u.v2.rhp sync.hwptr sync.swptr
2 | items filled by HW, but | created ready items from | non-valid items
3 V not read by the SW yet V the SW but still empty V (freed by the SW)
4 +==-+============================-+==-+======================+==-+======================+
5 ^ ^
6 | |
7 ctrl->php ctrl->shp
8 <- - - - - - - - - - - >
9 items passed from Libnfb to the driver,
10 but not yet from the driver to the HW