API¶

DP³'s has HTTP API which you can use to post datapoints and to read data stored in DP³. As the API is made using FastAPI, there is also an interactive documentation available at /docs endpoint.

There are several API endpoints:

GET /: check if API is running (just returns It works! message)
POST /datapoints: insert datapoints into DP³
GET /entity/<entity_type>: list current snapshots of all entities of given type
GET /entity/<entity_type>/<entity_id>: get data of entity with given entity id
GET /entity/<entity_type>/<entity_id>/get/<attr_id>: get attribute value
GET /entity/<entity_type>/<entity_id>/set/<attr_id>: set attribute value
GET /entity/<entity_type>/_/distinct/<attr_id>: get distinct attribute values and their counts based on latest snapshots
DELETE /entity/<entity_type>/<entity_id>: delete entity data for given id
POST /entity/<entity_type>/<entity_id>/ttl: extend TTLs of the specified entity
GET /entities: list entity configuration
GET /control/<action>: send a pre-defined action into execution queue.
GET /telemetry/sources_validity: get information about the validity of the data sources

Index¶

Health check.

Request¶

GET /

Response¶

200 OK:

{ "detail": "It works!" }

Insert datapoints¶

Request¶

POST /datapoints

All data are written to DP³ in the form of datapoints. A datapoint sets a value of a given attribute of given entity.

It is a JSON-encoded object with the set of keys defined in the table below. Presence of some keys depends on the primary type of the attribute (plain/observations/timseries).

Payload to this endpoint is JSON array of datapoints. For example:

[
   { DATAPOINT1 },
   { DATAPOINT2 }
]

Key	Description	Data-type	Required?	Plain	Observations	Timeseries
`type`	Entity type	string	mandatory	✔	✔	✔
`id`	Entity identification	string	mandatory	✔	✔	✔
`attr`	Attribute name	string	mandatory	✔	✔	✔
`v`	The value to set, depends on attr. type and data-type, see below	--	mandatory	✔	✔	✔
`t1`	Start time of the observation interval	string (RFC 3339 format)	mandatory	--	✔	✔
`t2`	End time of the observation interval	string (RFC 3339 format)	optional, default=`t1`	--	✔	✔
`c`	Confidence	float (0.0-1.0)	optional, default=1.0	--	✔	✔
`src`	Identification of the information source	string	optional, default=""	✔	✔	✔

More details depends on the particular type of the attribute.

Examples of datapoints¶

Plain¶

{
  "type": "ip",
  "id": "192.168.0.1",
  "attr": "note",
  "v": "My home router",
  "src": "web_gui"
}

Observations¶

{
  "type": "ip",
  "id": "192.168.0.1",
  "attr": "open_ports",
  "v": [22, 80, 443],
  "t1": "2022-08-01T12:00:00",
  "t2": "2022-08-01T12:10:00",
  "src": "open_ports_module"
}

Timeseries¶

regular:

{
  ...
  "t1": "2022-08-01T12:00:00",
  "t2": "2022-08-01T12:20:00", // assuming time_step = 5 min
  "v": {
    "a": [1, 3, 0, 2]
  }
}

irregular: timestamps must always be present

{
  ...
  "t1": "2022-08-01T12:00:00",
  "t2": "2022-08-01T12:05:00",
  "v": {
    "time": ["2022-08-01T12:00:00", "2022-08-01T12:01:10", "2022-08-01T12:01:15", "2022-08-01T12:03:30"],
    "x": [0.5, 0.8, 1.2, 0.7],
    "y": [-1, 3, 0, 0]
  }
}

irregular_interval:

{
  ...
  "t1": "2022-08-01T12:00:00",
  "t2": "2022-08-01T12:05:00",
  "v": {
    "time_first": ["2022-08-01T12:00:00", "2022-08-01T12:01:10", "2022-08-01T12:01:15", "2022-08-01T12:03:30"],
    "time_last": ["2022-08-01T12:01:00", "2022-08-01T12:01:15", "2022-08-01T12:03:00", "2022-08-01T12:03:40"],
    "x": [0.5, 0.8, 1.2, 0.7],
    "y": [-1, 3, 0, 0]
  }
}

Relations¶

Can be represented using both plain attributes and observations. The difference will be only in time specification. Two examples using observations:

no data - link<mac>: just the eid is sent

{
  "type": "ip",
  "id": "192.168.0.1",
  "attr": "mac_addrs",
  "v": "AA:AA:AA:AA:AA",
  "t1": "2022-08-01T12:00:00",
  "t2": "2022-08-01T12:10:00"
}

with additional data - link<ip, int>: The eid and the data are sent as a dictionary.

{
  "type": "ip",
  "id": "192.168.0.1",
  "attr": "ip_dep",
  "v": {"eid": "192.168.0.2", "data": 22},
  "t1": "2022-08-01T12:00:00",
  "t2": "2022-08-01T12:10:00"
}

Response¶

200 OK:

Success

400 Bad request:

Returns some validation error message, for example:

1 validation error for DataPointObservations_some_field
v -> some_embedded_dict_field
  field required (type=value_error.missing)

List entities¶

List latest snapshots of all ids present in database under entity type.

Contains only latest snapshot.

Uses pagination.

Request¶

GET /entity/<entity_type>

Optional query parameters:

skip: how many entities to skip (default: 0)
limit: how many entities to return (default: 20)

Response¶

{
  "time_created": "2023-07-04T12:10:38.827Z",
  "data": [
    {}
  ]
}

Get Eid data¶

Get data of entity type's eid.

Contains all snapshots and master record. Snapshots are ordered by ascending creation time.

Request¶

GET /entity/<entity_type>/<entity_id>

Optional query parameters:

date_from: date-time string
date_to: date-time string

Response¶

{
  "empty": true,
  "master_record": {},
  "snapshots": [
    {}
  ]
}

Get attr value¶

Get attribute value

Value is either of:

current value: in case of plain attribute
current value and history: in case of observation attribute
history: in case of timeseries attribute

Request¶

GET /entity/<entity_type>/<entity_id>/get/<attr_id>

Optional query parameters:

date_from: date-time string
date_to: date-time string

Response¶

{
  "attr_type": 1,
  "current_value": "string",
  "history": []
}

Set attr value¶

Set current value of attribute

Internally just creates datapoint for specified attribute and value.

This endpoint is meant for editable plain attributes -- for direct user edit on DP3 web UI.

Request¶

POST /entity/<entity_type>/<entity_id>/set/<attr_id>

Required request body:

{
  "value": "string"
}

Response¶

{
  "detail": "OK"
}

Get distinct values¶

Gets distinct attribute values and their counts based on latest snapshots

Useful for displaying <select> enumeration fields.

Works for all plain and observation data types except dict and json.

Request¶

GET /entity/<entity_type>/_/distinct/<attr_id>

Response¶

{
  "value1": 10,
  "value2": 5,
  "value3": 43
}

Delete Eid data¶

Delete master record and snapshots with the specified etype and eid.

Raw datapoints are not deleted, and the entity can be restored by sending new datapoints with the same etype and eid.

Request¶

DELETE /entity/<entity_type>/<entity_id>

Response¶

{
  "detail": "OK"
}

Extend TTLs¶

Extend TTLs of the specified entity.

Raw datapoints are not deleted, and the entity can be restored by sending new datapoints with the same etype and eid.

Request¶

POST /entity/<entity_type>/<entity_id>/ttl

The request body must be a dictionary of TTLs to extend, with string keys to identify the type of TTL. The values must be UTC timestamps, for example:

{
  "user_interaction": "2024-10-01T12:03:00",
  "api_dependency": "2024-10-08T12:00:00"
}

TTLs of the same name will be extended, and you add as many TTL names as you want.

Response¶

{
  "detail": "OK"
}

Entities¶

List entity types

Returns dictionary containing all entity types configured -- their simplified configuration and current state information.

Request¶

GET /entities

Response¶

{
  "<entity_id>": {
    "id": "<entity_id>",
    "name": "<entity_spec.name>",
    "attribs": "<MODEL_SPEC.attribs(e_id)>",
    "eid_estimate_count": "<DB.estimate_count_eids(e_id)>"
  },
  ...
}

Control¶

Execute Action - Sends the given action into execution queue.

You can see the enabled actions in /config/control.yml, available are:

make_snapshots - Makes an out-of-order snapshot of all entities
refresh_on_entity_creation - Re-runs the on_entity_creation callback for selected etype
refresh_module_config - Re-runs the load_config for selected module and will refresh the values derived by the module when configured to do so

You can learn more about the actions in the Actions section of the Control configuration documentation.

Request¶

GET /control/<action>

Response¶

{
  "detail": "OK"
}

Telemetry¶

Returns information about the validity of the data sources, i.e. when the last datapoint was received from each source.

Request¶

GET /telemetry/sources_validity

Response¶

{
  "module1@collector1": "2023-10-03T11:59:58.063000",
  "module2@collector1": "2023-12-06T09:09:37.165000",
  "module3@collector2": "2023-12-08T15:52:55.282000",
  ...
}