Processing core

Processing core's configuration in processing_core.yml file looks like this:

msg_broker:
  host: localhost
  port: 5672
  virtual_host: /
  username: dp3_user
  password: dp3_password
worker_processes: 2
worker_threads: 16
max_inline_generated_tasks: 0
modules_dir: "../dp3_modules"
enabled_modules:
  - "module_one"
  - "module_two"

Message broker

Message broker section describes connection details to RabbitMQ (or compatible) broker.

Parameter	Data-type	Default value	Description
host	string	localhost	IP address or hostname for connection to broker.
port	int	5672	Listening port of broker.
virtual_host	string	/	Virtual host for connection to broker.
username	string	guest	Username for connection to broker.
password	string	guest	Password for connection to broker.

Worker processes

Number of worker processes. This has to be at least 1.

If changing number of worker processes, the following process must be followed:

1. stop all inputs writing to task queue (e.g. API)
2. when all queues are empty, stop all workers
3. reconfigure queues in RabbitMQ using script found in /scripts/rmq_reconfigure.sh
4. change the settings here and in init scripts for worker processes (e.g. supervisor)
5. reload workers (e.g. using supervisorctl) and start all inputs again

Worker threads

Number of worker threads per process.

This may be higher than number of CPUs, because this is not primarily intended to utilize computational power of multiple CPUs (which Python cannot do well anyway due to the GIL), but to mask long I/O operations (e.g. queries to external services via network).

Inline generated tasks

max_inline_generated_tasks is a non-negative integer that limits how many tasks generated by ingestion hooks may be processed inline for one task received from RabbitMQ. The default is 0, which disables inline processing and sends every generated task through the priority queue.

Only generated tasks with the same complete entity key (entity type and EID) as the received task are eligible. The limit counts generated children, not the received source task, and is shared by the whole chain of children and descendants. Tasks for other entities and same-entity tasks above the limit are sent through the priority queue. An overflow task receives a fresh allowance if it is later consumed from RabbitMQ.

The setting is read when workers start, so changing it requires a worker restart. A small value such as 20 is recommended when first enabling the optimization; restore 0 and restart workers to return to queue-only processing.

Modules directory

Path to directory with plug-in (secondary) modules.

Relative path is evaluated relative to location of this configuration file.

Enabled modules

List of plug-in modules which should be enabled in processing pipeline.

Name of module filename without .py extension must be used!