Hydra Logo

ipfixprobe usage

This guide expects ipfixprobe is already installed, see installation step. The following sections describe alternative ways how to run ipfixprobe.

Systemd service (recommended)

The ipfixprobe can be set up to be used as a daemon to continuously process incoming packets from the boot up:

Create your instance.conf configuration, and save it in /etc/ipfixprobe/
All settings are explained in the example file

To start the systemd service, use:

sudo systemctl start ipfixprobe@instance.service

 "instance" serves as Your identifier of ipfixprobe instance, it must be equal in the name of the configuration file and in the systemctl command.

To enable the service at system startup, run:

sudo systemctl enable ipfixprobe@instance.service

Command line — starting manually

The simplest way to use ipfixprobe is to process PCAP file using PCAP plugin (need to be ./configured with --with-pcap)

./ipfixprobe -s cache -i "pcap;file=pcaps/http.pcap" -o "text;m"

The ipfixprobe consists of one input, zero or one output, one storage and zero or more process plugins.

Input plugin

Input plugin defines source of incoming packets. Use -i to specify input plugin.

Pcap reader

Input plugin for reading packets from a pcap file or a network interface

Command line parameters used by Pcap reader plugin:
f or file : Defines path to pcap file.
i or ifc : Defines interface name.
F or filter : Defines filter string.
s or snaplen : Defines snapshot length in bytes (live capture only).
l or list : Print list of available interfaces.

Read the pcap file specified by PATH value. Possible PATH value 'pcaps/bstats.pcap'

./ipfixprobe -i 'pcap;file=PATH;' -s 'cache'

Read packets from interface specified by IFC value. Possible IFC value 'eth0'

./ipfixprobe -i 'pcap;i=IFC;' -s 'cache'

DPDK

Input plugin for reading packets using DPDK interface

Command line parameters used by DPDK plugin:
b or bsize : Size of the MBUF packet buffer. Default: 64.
p or port : DPDK port to be used as an input interface.
m or mem : Size of the memory pool for received packets. Default: 16384.
q or queue : Number of RX queues. Default: 1.
e or eal : DPDK eal.
M or mtu : Input interface MTU. Default: 1518.

Read packets using DPDK input interface and 1 DPDK queue, enable plugins for basic statistics, http and tls, output to IPFIX on a local machine DPDK EAL parameters are passed in `e, eal` parameters DPDK plugin configuration has to be specified in the first input interface. The following `dpdk` interfaces are given without parameters; their configuration is inherited from the first one. Example for the queue of 3 DPDK input plugins (q=3):

./ipfixprobe -i 'dpdk;p=0;q=3;e=-c 0x1 -a <[domain:]bus:devid.func>' -i dpdk -i dpdk -p http -p bstats -p tls -o 'ipfix;h=127.0.0.1'

Same example for the multiport read from ports 0 and 1, note comma separated ports:

./ipfixprobe -i 'dpdk;p=0,1;q=3;e=-c 0x1 -a <[domain:]bus:devid.func>' -i dpdk -i dpdk -p http -p bstats -p tls -o 'ipfix;h=127.0.0.1'

DPDK-ring

DPDK ring input interface for ipfixprobe (secondary DPDK app).

Command line parameters used by DPDK-ring plugin:
b or bsize : Size of the MBUF packet buffer. Default: 64.
r or ring : Name of the ring to read packets from. Need to be specified explicitly thus no default provided.
e or eal : DPDK eal.

Read packets using DPDK input interface as secondary process with shared memory (DPDK rings) - in this case, 4 DPDK rings are used

./ipfixprobe -i 'dpdk-ring;r=rx_ipfixprobe_0;e= --proc-type=secondary' -i 'dpdk-ring;r=rx_ipfixprobe_1' -i 'dpdk-ring;r=rx_ipfixprobe_2' -i 'dpdk-ring;r=rx_ipfixprobe_3' -o 'text'

Raw

Input plugin for reading packets from raw interface

Command line parameters used by Raw plugin:
i or ifc : Defines network interface name.
b or blocks : Defines number of packet blocks.
f or fanout : Enables packet fanout.
p or pkts : Defines number of packets in block.
l or list : Print list of available interfaces.

Read packets from interface specified by IFC value. Possible IFC value 'eth0'

./ipfixprobe -i 'raw;ifc=IFC;' -s 'cache'

Benchmark

Input plugin for various benchmarking purposes.

Command line parameters used by Benchmark plugin:
m or mode : Defines benchmark mode: 1f (1x N-packet flow) or nf (Nx 1-packet flow).
S or seed : Defines string seed for random generator.
d or duration : Defines duration in seconds.
p or count : Defines packet count.
s or size : Defines packet size.
I or id : Defines link identifier number.

Read packets from interface specified with DPDK ports 0 and 1

`./ipfixprobe -i 'dpdk;p=0,1;' -s 'cache'

Output plugin

Output plugin defines how flows are expoted. Use -o to specify output plugin.

Text

Provides human readable output to the terminal or file.

Command line parameters used by Text plugin:
f or file : Defines path to savefile to write output in instead of stdout.
m or mac : Boolean flag. Mac addresses are hidden if set.

Print expoted flows to the terminal without mac adresses

./ipfixprobe -o 'text;mac'-i 'pcap;file=...;' -s 'cache'

Print expoted flows to the FILE

./ipfixprobe -o 'text;f=FILE'-i 'pcap;file=...;' -s 'cache'

IPFIX

Exports data in the IPFIX format

Command line parameters used by IPFIX plugin:
h or host : Defines ip address of remote collector.
p or port : Defines collector port to send data to.
m or mtu : Defines maximum size of ipfix packet payload sent.
u or udp : Boolean flag. UDP is used if set.
n or non-blocking-tcp : Boolean flag. Non-blocking-tcp socket is used if set.
I or id : Defines exporter id.
t or template : Defines template refresh rate in seconds.

Send exported data to the localhost using UDP as an exporter 3.

./ipfixprobe -o 'ipfix;h=127.0.0.1,u,I=3'-i 'pcap;file=...;' -s 'cache'

Send exported data to the localhost:4739 using non-blocking tcp as an exporter 3 with maximal transfer unit set to 2000.

./ipfixprobe -o 'ipfix;h=127.0.0.1,p=4739,n,mtu=2000'-i 'pcap;file=...;' -s 'cache'

UNIREC

Exports data in the UNIREC format

Command line parameters used by UNIREC plugin:
i or ifc : Defines unirec interface to use.
p or plugins : Defines plugin-interface mapping. Plugins can be grouped like '(p1,p2,p3),p4,(p5,p6).
o or odid : Boolean flag.If set exports ODID field.
e or eof : Boolean flag.If set sends eof messag on exit.
I or id : Defines exporter id.
h or help : Prints libtrap help.

Send exported data to the Unix socket 'ipfixprobe'

./ipfixprobe -o 'unirec;i=u:ipfixprobe'-i 'pcap;file=...;' -s 'cache'

Same as previous, but should be used with small pcap files to avoid not sending data

./ipfixprobe -o 'unirec;i=u:ipfixprobe:timeout=WAIT:buffer=off'-i 'pcap;file=...;' -s 'cache'

Save exported data to the data.trapcap

./ipfixprobe -o 'unirec;i=f:data.trapcap'-i 'pcap;file=...;' -s 'cache'

Storage plugin

Storage plugin defines how flows are internally stored. Use -s to specify storage plugin.

Cache

Currently only available plugin. Hash table is used to keep flows. Hash table is divided into rows. Each row is managed as LRU.

Command line parameters used by Cache plugin:
s or size : Defines count of flows that are kept in the cache at once. Cache size is 2s.
l or line : Defines length of the cache line. Line length is 2l.
a or active : Defines active timeout. When there is a flow, that is active for more than -a seconds, its exported.
i or inactive : Defines inactive timeout. When there is a flow, that is inactive for more than -i seconds, its exported.
S or split : Boolean flag. Defines if the bidirectional flow between two nodes is splitted into 2 separate unidirectional flows.
fe/frag-enable, fs/frag-size, ft/frag-timeout : Used to enable completing fragmented packets into one packet. Framentation cache size is fs and timeout to consider fragments belong to same packet is ft.

Store flows using 'cache'

./ipfixprobe -s 'cache' -i 'pcap;file=PATH;'

One-time convertion of PCAP file to CSV

The ipfixprobe can be used to convert given PCAP file to the CSV containing flows from that file in the Unirec format.

Requirements:
  • Docker or Podman
  • bash
  • which, mktemp
 This container performs the following tasks:
  1. Copies a pcap file and processing script into the container
  2. Runs the ipfixprobe tool to export flows
  3. Logs the results in CSV format

Build

The script builds the image automatically, but be sure that Dockerfile is in the same directory.
To build the manually image, navigate to the directory containing the Dockerfile and run:

docker build -t docker_ipfixprobe .

Run

Parameters:
  • process_script.sh Script for processing the pcap file inside the container
  • input_file.pcap Path to the input pcap file
  • output_file.csv Path to the output CSV file

bash ./ipfixprobe_wrapper.sh <process_script.sh> <input_file.pcap> <output_file.csv>

To process a file ../pcaps/mixed.pcap using a processing script process_script.sh and output the results to output.csv, use the following wrapper script:

bash ./ipfixprobe_wrapper.sh ./process_script.sh ../pcaps/mixed.pcap ./output.csv

OpenWrt / Turris

Create and save the configuration file to the /etc/config/ipfixprobe. Example of configuration file:

NEMEA ipfixprobe # Copyright (C) 2022-2023 CESNET # Available options for profiles, 'list' options can be used repeatedly: # list interfaces - list of NIC, e.g., eth0, enp0s1, ... # list plugins - list of plugin names, see 'ipfixprobe -h process' for help # ipfix_host - address of IPFIX collector # ipfix_port - port of IPFIX collector, default: 4739 # ipfix_udp - 1 to export to IPFIX collector via UDP, 0 via TCP # cache_size - size of flow cache as exponent of 2, default: 1024 # cache_line - size of flow cache line as exponent of 2, default: 4 # active_timeout - active timeout in seconds, default: 300 # inactive_timeout - inactive timeout in seconds, default: 30 # link - unsigned integer as identification of link/router # dir - unsigned integer as identification of direction/NIC # split_biflow - 1 to split biflow to uniflow, default: 0 to use biflow # ipfix_mtu - size of max transmission unit (MTU), default: 1452 # # respawn - enable respawn of crashed process # respawn_threshold - timeout in seconds for restarting a service after it closes # respawn_timeout - max time in seconds to wait for a process respawn to complete # respawn_retry - max number of attempts to respawn before giving up, 0 means newer stop trying to respawn # core - size of coredump, '0' - not generate, 'unlimited' - unlimited size # # enabled - 1 to enable start daemon instance for that profile, NOTE: if profile is directly specified for start script # (example: '/etc/init.d/ipfixprobe start wan profileX profileY lan'), this option is ignored config profile 'lan' option enabled '0' list interfaces 'br-lan' list plugins 'basicplus' list plugins 'dns' list plugins 'http' list plugins 'pstats' list plugins 'ovpn' list plugins 'wg' list plugins 'dnssd;txt' list plugins 'ssdp' list plugins 'tls' list plugins 'quic' option ipfix_host '127.0.0.1' option ipfix_port '4739' option ipfix_udp '1' option link '1' option dir '1'

Prepare the init script:

wget https://raw.githubusercontent.com/CESNET/Nemea-OpenWRT/master/net/ipfixprobe/files/init.d/ipfixprobe

sudo mv ipfixprobe /etc/init.d/ipfixprobe

To run the script manually use:

/etc/init.d/ipfixprobe start

To start the service at system startup:

/etc/init.d/ipfixprobe enable