Series-based config class

cesnet_tszoo.configs.series_based_config.SeriesBasedConfig

Bases: SeriesBasedHandler, DatasetConfig

This class is used for configuring the SeriesBasedCesnetDataset.

Used to configure the following:

• Train, validation, test, all sets (time period, sizes, features)
• Handling missing values (default values, fillers)
• Handling anomalies (anomaly handlers)
• Data transformation using transformers
• Applying custom handlers (custom handlers)
• Changing order of preprocesses
• Dataloader options (train/val/test/all/init workers, batch size, train loading order)
• Plotting

Important Notes:

• Custom fillers must inherit from the fillers base class.
• Custom anomaly handlers must inherit from the anomaly handlers base class.
• Selected anomaly handler is only used for train set.
• It is recommended to use the transformers base class, though this is not mandatory as long as it meets the required methods.
  • If a transformer is already initialized and partial_fit_initialized_transformers is False, the transformer does not require partial_fit.
  • Otherwise, the transformer must support partial_fit.
  • Transformers must implement transform method.
  • Both partial_fit and transform methods must accept an input of type np.ndarray with shape (times, features).
• Custom handlers must be derived from one of the built-in custom handler classes
• train_ts, val_ts, and test_ts must not contain any overlapping time series IDs.

Source code in cesnet_tszoo\configs\series_based_config.py

class SeriesBasedConfig(SeriesBasedHandler, DatasetConfig):
    """
    This class is used for configuring the [`SeriesBasedCesnetDataset`](reference_series_based_cesnet_dataset.md#cesnet_tszoo.datasets.series_based_cesnet_dataset.SeriesBasedCesnetDataset).

    Used to configure the following:

    - Train, validation, test, all sets (time period, sizes, features)
    - Handling missing values (default values, [`fillers`](reference_fillers.md#cesnet_tszoo.utils.filler.filler))
    - Handling anomalies ([`anomaly handlers`](reference_anomaly_handlers.md#cesnet_tszoo.utils.anomaly_handler.anomaly_handler))
    - Data transformation using [`transformers`](reference_transformers.md#cesnet_tszoo.utils.transformer.transformer)
    - Applying custom handlers ([`custom handlers`](reference_custom_handlers.md#cesnet_tszoo.utils.custom_handler.custom_handler))
    - Changing order of preprocesses
    - Dataloader options (train/val/test/all/init workers, batch size, train loading order)
    - Plotting

    **Important Notes:**

    - Custom fillers must inherit from the [`fillers`](reference_fillers.md#cesnet_tszoo.utils.filler.filler.Filler) base class.
    - Custom anomaly handlers must inherit from the [`anomaly handlers`](reference_anomaly_handlers.md#cesnet_tszoo.utils.anomaly_handler.anomaly_handler.AnomalyHandler) base class.
    - Selected anomaly handler is only used for train set.    
    - It is recommended to use the [`transformers`](reference_transformers.md#cesnet_tszoo.utils.transformer.transformer.Transformer) base class, though this is not mandatory as long as it meets the required methods.
        - If a transformer is already initialized and `partial_fit_initialized_transformers` is `False`, the transformer does not require `partial_fit`.
        - Otherwise, the transformer must support `partial_fit`.
        - Transformers must implement `transform` method.
        - Both `partial_fit` and `transform` methods must accept an input of type `np.ndarray` with shape `(times, features)`.
    - Custom handlers must be derived from one of the built-in [`custom handler`](reference_custom_handlers.md#cesnet_tszoo.utils.custom_handler.custom_handler) classes 
    - `train_ts`, `val_ts`, and `test_ts` must not contain any overlapping time series IDs.

    Attributes:
        used_train_workers: Tracks the number of train workers in use. Helps determine if the train dataloader should be recreated based on worker changes.
        used_val_workers: Tracks the number of validation workers in use. Helps determine if the validation dataloader should be recreated based on worker changes.
        used_test_workers: Tracks the number of test workers in use. Helps determine if the test dataloader should be recreated based on worker changes.
        used_all_workers: Tracks the total number of all workers in use. Helps determine if the all dataloader should be recreated based on worker changes.
        uses_all_ts: Whether all time series set should be used.
        import_identifier: Tracks the name of the config upon import. None if not imported.
        filler_factory: Represents factory used to create passed Filler type.
        anomaly_handler_factory: Represents factory used to create passed Anomaly Handler type.
        transformer_factory: Represents factory used to create passed Transformer type.
        can_fit_fillers: Whether fillers in this config, can be fitted.                
        logger: Logger for displaying information.   
        all_ts: If no specific sets (train/val/test) are provided, all time series IDs are used. When any set is defined, only the time series IDs in defined sets are used.
        train_ts_row_ranges: Initialized when `train_ts` is set. Contains time series IDs in train set with their respective time ID ranges.
        val_ts_row_ranges: Initialized when `val_ts` is set. Contains time series IDs in validation set with their respective time ID ranges.
        test_ts_row_ranges: Initialized when `test_ts` is set. Contains time series IDs in test set with their respective time ID ranges.
        all_ts_row_ranges: Initialized when `all_ts` is set. Contains time series IDs in all set with their respective time ID ranges.
        display_time_period: Used to display the configured value of `time_period`.
        aggregation: The aggregation period used for the data.
        source_type: The source type of the data.
        database_name: Specifies which database this config applies to.
        features_to_take_without_ids: Features to be returned, excluding time or time series IDs.
        indices_of_features_to_take_no_ids: Indices of non-ID features in `features_to_take`.
        ts_id_name: Name of the time series ID, dependent on `source_type`.
        used_singular_train_time_series: Currently used singular train set time series for dataloader.
        used_singular_val_time_series: Currently used singular validation set time series for dataloader.
        used_singular_test_time_series: Currently used singular test set time series for dataloader.
        used_singular_all_time_series: Currently used singular all set time series for dataloader.     
        train_preprocess_order: All preprocesses used for train set. 
        val_preprocess_order: All preprocesses used for val set. 
        test_preprocess_order: All preprocesses used for test set. 
        all_preprocess_order: All preprocesses used for all set.                  
        is_initialized: Flag indicating if the configuration has already been initialized. If true, config initialization will be skipped.  
        version: Version of cesnet-tszoo this config was made in.
        export_update_needed: Whether config was updated to newer version and should be exported.      
        time_period: Defines the time period for returning data from `train/val/test/all`. Can be a range of time IDs, a tuple of datetime objects or a float. Float value is equivalent to percentage of available times from start.
        train_ts: Defines which time series IDs are used in the training set. Can be a list of IDs, or an integer/float to specify a random selection. An `int` specifies the number of random time series, and a `float` specifies the proportion of available time series. 
                  `int` and `float` must be greater than 0, and a float should be smaller or equal to 1.0. Using `int` or `float` guarantees that no time series from other sets will be used.
        val_ts: Defines which time series IDs are used in the validation set. Same as `train_ts` but for the validation set.
        test_ts: Defines which time series IDs are used in the test set. Same as `train_ts` but for the test set.         
        features_to_take: Defines which features are used.
        default_values: Default values for missing data, applied before fillers. Can set one value for all features or specify for each feature.
        train_batch_size: Batch size for the train dataloader. Affects number of returned time series in one batch.
        val_batch_size: Batch size for the validation dataloader. Affects number of returned time series in one batch.
        test_batch_size: Batch size for the test dataloader. Affects number of returned time series in one batch.
        all_batch_size: Batch size for the all dataloader. Affects number of returned time series in one batch.
        preprocess_order: Defines in which order preprocesses are used. Also can add to order a type of `AllSeriesCustomHandler` or `NoFitCustomHandler`.            
        partial_fit_initialized_transformer: If `True`, partial fitting on train set is performed when using initiliazed transformer.
        include_time: If `True`, time data is included in the returned values.
        include_ts_id: If `True`, time series IDs are included in the returned values.
        time_format: Format for the returned time data. When using TimeFormat.DATETIME, time will be returned as separate list along rest of the values.
        train_workers: Number of workers for loading training data. `0` means that the data will be loaded in the main process.
        val_workers: Number of workers for loading validation data. `0` means that the data will be loaded in the main process.
        test_workers: Number of workers for loading test data. `0` means that the data will be loaded in the main process.
        all_workers: Number of workers for loading all data. `0` means that the data will be loaded in the main process.
        init_workers: Number of workers for initial dataset processing during configuration. `0` means that the data will be loaded in the main process.
        nan_threshold: Maximum allowable percentage of missing data. Time series exceeding this threshold are excluded. Time series over the threshold will not be used. Used for `train/val/test/all` separately.
        train_dataloader_order: Defines the order of data returned by the training dataloader.
        random_state: Fixes randomness for reproducibility during configuration and dataset initialization.               
    """

    def __init__(self,
                 time_period: tuple[datetime, datetime] | range | float | Literal["all"],
                 train_ts: list[int] | npt.NDArray[np.int_] | float | int | None = None,
                 val_ts: list[int] | npt.NDArray[np.int_] | float | int | None = None,
                 test_ts: list[int] | npt.NDArray[np.int_] | float | int | None = None,
                 features_to_take: list[str] | Literal["all"] = "all",
                 default_values: list[Number] | npt.NDArray[np.number] | dict[str, Number] | Number | Literal["default"] | None = "default",
                 train_batch_size: int = 32,
                 val_batch_size: int = 64,
                 test_batch_size: int = 128,
                 all_batch_size: int = 128,
                 preprocess_order: list[str, type] = ["filling_gaps", "handling_anomalies", "transforming"],
                 fill_missing_with: type | FillerType | Literal["mean_filler", "forward_filler", "linear_interpolation_filler"] | None = None,
                 transform_with: type | TransformerType | Transformer | Literal["min_max_scaler", "standard_scaler", "max_abs_scaler", "log_transformer", "l2_normalizer"] | None = None,
                 handle_anomalies_with: type | AnomalyHandlerType | Literal["z-score", "interquartile_range"] | None = None,
                 partial_fit_initialized_transformer: bool = False,
                 include_time: bool = True,
                 include_ts_id: bool = True,
                 time_format: TimeFormat | Literal["id_time", "datetime", "unix_time", "shifted_unix_time"] = TimeFormat.ID_TIME,
                 train_workers: int = 4,
                 val_workers: int = 3,
                 test_workers: int = 2,
                 all_workers: int = 4,
                 init_workers: int = 4,
                 nan_threshold: float = 1.0,
                 train_dataloader_order: DataloaderOrder | Literal["random", "sequential"] = DataloaderOrder.SEQUENTIAL,
                 random_state: int | None = None):
        """
        Parameters:
            time_period: Defines the time period for returning data from `train/val/test/all`. Can be a range of time IDs, a tuple of datetime objects or a float. Float value is equivalent to percentage of available times from start.
            train_ts: Defines which time series IDs are used in the training set. Can be a list of IDs, or an integer/float to specify a random selection. An `int` specifies the number of random time series, and a `float` specifies the proportion of available time series. 
                    `int` and `float` must be greater than 0, and a float should be smaller or equal to 1.0. Using `int` or `float` guarantees that no time series from other sets will be used. `Default: None`
            val_ts: Defines which time series IDs are used in the validation set. Same as `train_ts` but for the validation set. `Default: None`
            test_ts: Defines which time series IDs are used in the test set. Same as `train_ts` but for the test set. `Default: None`           
            features_to_take: Defines which features are used. `Default: "all"`
            default_values: Default values for missing data, applied before fillers. Can set one value for all features or specify for each feature. `Default: "default"`
            train_batch_size: Batch size for the train dataloader. Affects number of returned time series in one batch. `Default: 32`
            val_batch_size: Batch size for the validation dataloader. Affects number of returned time series in one batch. `Default: 64`
            test_batch_size: Batch size for the test dataloader. Affects number of returned time series in one batch. `Default: 128`
            all_batch_size: Batch size for the all dataloader. Affects number of returned time series in one batch. `Default: 128`   
            preprocess_order: Defines in which order preprocesses are used. Also can add to order a type of `AllSeriesCustomHandler` or `NoFitCustomHandler`. `Default: ["handling_anomalies", "filling_gaps", "transforming"]`              
            fill_missing_with: Defines how to fill missing values in the dataset. Can pass enum `FillerType` for built-in filler or pass a type of custom filler that must derive from `Filler` base class. `Default: None`
            transform_with: Defines the transformer used to transform the dataset. Can pass enum `TransformerType` for built-in transformer, pass a type of custom transformer or instance of already fitted transformer. `Default: None`
            handle_anomalies_with: Defines the anomaly handler for handling anomalies in the train set. Can pass enum `AnomalyHandlerType` for built-in anomaly handler or a type of custom anomaly handler. `Default: None`
            partial_fit_initialized_transformer: If `True`, partial fitting on train set is performed when using initiliazed transformer. `Default: False`
            include_time: If `True`, time data is included in the returned values. `Default: True`
            include_ts_id: If `True`, time series IDs are included in the returned values. `Default: True`
            time_format: Format for the returned time data. When using TimeFormat.DATETIME, time will be returned as separate list along rest of the values. `Default: TimeFormat.ID_TIME`
            train_workers: Number of workers for loading training data. `0` means that the data will be loaded in the main process. `Default: 4`
            val_workers: Number of workers for loading validation data. `0` means that the data will be loaded in the main process. `Default: 3`
            test_workers: Number of workers for loading test data. `0` means that the data will be loaded in the main process. `Default: 2`
            all_workers: Number of workers for loading all data. `0` means that the data will be loaded in the main process. `Default: 4`
            init_workers: Number of workers for initial dataset processing during configuration. `0` means that the data will be loaded in the main process. `Default: 4`
            nan_threshold: Maximum allowable percentage of missing data. Time series exceeding this threshold are excluded. Time series over the threshold will not be used. Used for `train/val/test/all` separately. `Default: 1.0`
            train_dataloader_order: Defines the order of data returned by the training dataloader. `Default: DataloaderOrder.SEQUENTIAL`
            random_state: Fixes randomness for reproducibility during configuration and dataset initialization. `Default: None`       
        """

        self.time_period: np.ndarray = time_period

        self.display_time_period: Optional[range] = None

        self.logger = logging.getLogger("series_config")

        SeriesBasedHandler.__init__(self, self.logger, True, train_ts, val_ts, test_ts)
        DatasetConfig.__init__(self, features_to_take, default_values, train_batch_size, val_batch_size, test_batch_size, all_batch_size, preprocess_order, fill_missing_with, transform_with, handle_anomalies_with, partial_fit_initialized_transformer, include_time, include_ts_id, time_format,
                               train_workers, val_workers, test_workers, all_workers, init_workers, nan_threshold, False, DatasetType.SERIES_BASED, train_dataloader_order, random_state, False, self.logger)

    def _validate_construction(self) -> None:
        """Performs basic parameter validation to ensure correct configuration. More comprehensive validation, which requires dataset-specific data, is handled in [`_dataset_init`][cesnet_tszoo.configs.series_based_config.SeriesBasedConfig._dataset_init]. """

        DatasetConfig._validate_construction(self)

        if isinstance(self.time_period, (float, int)):
            self.time_period = float(self.time_period)
            assert self.time_period > 0.0, "time_period must be greater than 0"
            assert self.time_period <= 1.0, "time_period must be lower or equal to 1.0"

        self._validate_ts_init()

        self.logger.debug("Series-based configuration validated successfully.")

    def _get_train(self) -> tuple[np.ndarray, np.ndarray] | tuple[None, None]:
        """Returns the indices corresponding to the training set. """
        return self.train_ts, self.time_period

    def _get_val(self) -> tuple[np.ndarray, np.ndarray] | tuple[None, None]:
        """Returns the indices corresponding to the validation set. """
        return self.val_ts, self.time_period

    def _get_test(self) -> tuple[np.ndarray, np.ndarray] | tuple[None, None]:
        """Returns the indices corresponding to the test set. """
        return self.test_ts, self.time_period

    def _get_all(self) -> tuple[np.ndarray, np.ndarray] | tuple[None, None]:
        """Returns the indices corresponding to the all set. """
        return self.all_ts, self.time_period

    def has_train(self) -> bool:
        """Returns whether training set is used. """
        return self.train_ts is not None

    def has_val(self) -> bool:
        """Returns whether validation set is used. """
        return self.val_ts is not None

    def has_test(self) -> bool:
        """Returns whether test set is used. """
        return self.test_ts is not None

    def has_all(self) -> bool:
        """Returns whether all set is used. """
        return self.all_ts is not None

    def _set_time_period(self, all_time_ids: np.ndarray) -> None:
        """Validates and filters the input time period based on the dataset and aggregation. """

        if self.time_period == "all":
            self.time_period = range(len(all_time_ids))
            self.logger.debug("Time period set to 'all'. Using all available time IDs, range: %s", self.time_period)
        elif isinstance(self.time_period, float):
            self.time_period = range(int(self.time_period * len(all_time_ids)))
            self.logger.debug("Time period set with float value. Using range: %s", self.time_period)

        self.time_period, self.display_time_period = TimeBasedHandler._process_time_period(self.time_period, all_time_ids, self.logger, self.time_format)
        self.logger.debug("Processed time_period: %s, display_time_period: %s", self.time_period, self.display_time_period)

    def _set_ts(self, all_ts_ids: np.ndarray, all_ts_row_ranges: np.ndarray, rd: np.random.RandomState) -> None:
        """Validates and filters the input time series IDs based on the `dataset` and `source_type`. Handles random split."""

        self._prepare_and_set_ts_sets(all_ts_ids, all_ts_row_ranges, self.ts_id_name, self.random_state, rd)

    def _get_feature_transformers(self) -> Transformer:
        """Creates transformer with `transformer_factory`. """

        if self.transformer_factory.has_already_initialized:
            if not self.has_train() and self.partial_fit_initialized_transformers:
                self.partial_fit_initialized_transformers = False
                self.logger.warning("partial_fit_initialized_transformers will be ignored because train set is not used.")

            transformers = self.transformer_factory.get_already_initialized_transformers()
            self.logger.debug("Using already initialized transformer %s.", self.transformer_factory.name)
        else:
            if not self.has_train() and not self.transformer_factory.is_empty_factory:
                self.transformer_factory = transformer_factories.get_transformer_factory(None, self.create_transformer_per_time_series, self.partial_fit_initialized_transformers)
                self.logger.warning("No transformer will be used because train set is not used.")

            transformers = self.transformer_factory.create_transformer()
            self.logger.debug("Using transformer %s.", self.transformer_factory.name)

        return transformers

    def _set_no_fit_custom_handler(self, factory: NoFitCustomHandlerFactory):

        train_handlers = np.array([factory.create_handler() for _ in self.train_ts]) if self.has_train() else None
        self.train_preprocess_order.append(PreprocessNote(factory.preprocess_enum_type, False, False, factory.can_apply_to_train and self.has_train(), True, NoFitCustomHandlerHolder(train_handlers)))

        val_handlers = np.array([factory.create_handler() for _ in self.val_ts]) if self.has_val() else None
        self.val_preprocess_order.append(PreprocessNote(factory.preprocess_enum_type, False, False, factory.can_apply_to_val and self.has_val(), True, NoFitCustomHandlerHolder(val_handlers)))

        test_handlers = np.array([factory.create_handler() for _ in self.test_ts]) if self.has_test() else None
        self.test_preprocess_order.append(PreprocessNote(factory.preprocess_enum_type, False, False, factory.can_apply_to_test and self.has_test(), True, NoFitCustomHandlerHolder(test_handlers)))

        all_handlers = np.array([factory.create_handler() for _ in self.all_ts]) if self.has_all() else None
        self.all_preprocess_order.append(PreprocessNote(factory.preprocess_enum_type, False, False, factory.can_apply_to_all and self.has_all(), True, NoFitCustomHandlerHolder(all_handlers)))

    def _get_fillers(self) -> tuple:
        """Creates fillers with `filler_factory`. """

        train_fillers = None
        # Set the fillers for the training set
        if self.has_train():
            train_fillers = np.array([self.filler_factory.create_filler(self.features_to_take_without_ids) for _ in self.train_ts])
            self.logger.debug("Fillers for training set are set.")

        val_fillers = None
        # Set the fillers for the validation set
        if self.has_val():
            val_fillers = np.array([self.filler_factory.create_filler(self.features_to_take_without_ids) for _ in self.val_ts])
            self.logger.debug("Fillers for validation set are set.")

        test_fillers = None
        # Set the fillers for the test set
        if self.has_test():
            test_fillers = np.array([self.filler_factory.create_filler(self.features_to_take_without_ids) for _ in self.test_ts])
            self.logger.debug("Fillers for test set are set.")

        all_fillers = None
        # Set the fillers for the all set
        if self.has_all():
            all_fillers = np.array([self.filler_factory.create_filler(self.features_to_take_without_ids) for _ in self.all_ts])
            self.logger.debug("Fillers for all set are set.")

        self.logger.debug("Using filler %s", self.filler_factory.name)

        return train_fillers, val_fillers, test_fillers, all_fillers

    def _get_anomaly_handlers(self) -> np.ndarray:
        """Creates anomaly handlers with `anomaly_handler_factory`. """

        if not self.has_train() and not self.anomaly_handler_factory.is_empty_factory:
            self.anomaly_handler_factory = anomaly_handler_factories.get_anomaly_handler_factory(None)
            self.logger.warning("No anomaly handler will be used because train set is not used.")

        anomaly_handlers = np.array([])
        if self.has_train():
            anomaly_handlers = np.array([self.anomaly_handler_factory.create_anomaly_handler() for _ in self.train_ts])

        self.logger.debug("Using anomaly handler %s", self.anomaly_handler_factory.name)

        return anomaly_handlers

    def _set_per_series_custom_handler(self, factory: PerSeriesCustomHandlerFactory):
        raise ValueError(f"Cannot use {factory.name} CustomHandler, because PerSeriesCustomHandler is not supported for {self.dataset_type}. Use AllSeriesCustomHandler or NoFitCustomHandler instead. ")

    def _get_summary_filter_time_series(self) -> css_utils.SummaryDiagramStep:
        attributes = [css_utils.StepAttribute("Train time series IDs", get_abbreviated_list_string(self.train_ts)),
                      css_utils.StepAttribute("Val time series IDs", get_abbreviated_list_string(self.val_ts)),
                      css_utils.StepAttribute("Test time series IDs", get_abbreviated_list_string(self.test_ts)),
                      css_utils.StepAttribute("All time series IDs", get_abbreviated_list_string(self.all_ts)),
                      css_utils.StepAttribute("Time periods", self.display_time_period),
                      css_utils.StepAttribute("Nan threshold", self.nan_threshold)]

        return css_utils.SummaryDiagramStep("Filter time series", attributes)

    def _get_summary_loader(self) -> list[css_utils.SummaryDiagramStep]:

        steps = []

        attributes = [css_utils.StepAttribute("Train batch size", self.train_batch_size),
                      css_utils.StepAttribute("Val batch size", self.val_batch_size),
                      css_utils.StepAttribute("Test batch size", self.test_batch_size),
                      css_utils.StepAttribute("All batch size", self.all_batch_size),
                      css_utils.StepAttribute("Train dataloader order", self.train_dataloader_order),]

        steps.append(css_utils.SummaryDiagramStep("Transform into specific format", attributes))

        return steps

    def _validate_finalization(self) -> None:
        """Performs final validation of the configuration. """

        self._validate_ts_overlap()

    def __str__(self) -> str:

        if self.transformer_factory.is_empty_factory:
            transformer_part = f"Transformer type: {self.transformer_factory.name}"
        else:
            transformer_part = f'''Transformer type: {self.transformer_factory.name}
        Are transformers premade: {self.transformer_factory.has_already_initialized}
        Are premade transformers partial_fitted: {self.partial_fit_initialized_transformers}'''

        if self.include_time:
            time_part = f'''Time included: {str(self.include_time)}    
        Time format: {str(self.time_format)}'''
        else:
            time_part = f"Time included: {str(self.include_time)}"

        return f'''
Config Details:
    Used for database: {self.database_name}
    Aggregation: {str(self.aggregation)}
    Source: {str(self.source_type)}

    Time series
        Train time series IDS: {get_abbreviated_list_string(self.train_ts)}
        Val time series IDS: {get_abbreviated_list_string(self.val_ts)}
        Test time series IDS {get_abbreviated_list_string(self.test_ts)}
        All time series IDS {get_abbreviated_list_string(self.all_ts)}
    Time periods
        Time period: {str(self.display_time_period)}
    Features
        Taken features: {str(self.features_to_take_without_ids)}
        Default values: {self.default_values}
        Time series ID included: {str(self.include_ts_id)}
        {time_part}
    Fillers         
        Filler type: {self.filler_factory.name}
    Transformers
        {transformer_part}
    Anomaly handler
        Anomaly handler type (train set): {self.anomaly_handler_factory.name}   
    Batch sizes
        Train batch size: {self.train_batch_size}
        Val batch size: {self.val_batch_size}
        Test batch size: {self.test_batch_size}
        All batch size: {self.all_batch_size}
    Default workers
        Train worker count: {str(self.train_workers)}
        Val worker count: {str(self.val_workers)}
        Test worker count: {str(self.test_workers)}
        All worker count: {str(self.all_workers)}
        Init worker count: {str(self.init_workers)}
    Other
        Preprocess order: {normalize_display_list(self.preprocess_order)}
        Nan threshold: {str(self.nan_threshold)}
        Random state: {self.random_state}
        Train dataloader order: {str(self.train_dataloader_order)}
        Version: {self.version}
                '''

Configuration options

Parameters:

Name	Type	Description	Default
time_period	tuple[datetime, datetime] | range | float | Literal['all']	Defines the time period for returning data from train/val/test/all. Can be a range of time IDs, a tuple of datetime objects or a float. Float value is equivalent to percentage of available times from start.	required
train_ts	list[int] | NDArray[int_] | float | int | None	Defines which time series IDs are used in the training set. Can be a list of IDs, or an integer/float to specify a random selection. An int specifies the number of random time series, and a float specifies the proportion of available time series. int and float must be greater than 0, and a float should be smaller or equal to 1.0. Using int or float guarantees that no time series from other sets will be used. Default: None	None
val_ts	list[int] | NDArray[int_] | float | int | None	Defines which time series IDs are used in the validation set. Same as train_ts but for the validation set. Default: None	None
test_ts	list[int] | NDArray[int_] | float | int | None	Defines which time series IDs are used in the test set. Same as train_ts but for the test set. Default: None	None
features_to_take	list[str] | Literal['all']	Defines which features are used. Default: "all"	'all'
default_values	list[Number] | NDArray[number] | dict[str, Number] | Number | Literal['default'] | None	Default values for missing data, applied before fillers. Can set one value for all features or specify for each feature. Default: "default"	'default'
train_batch_size	int	Batch size for the train dataloader. Affects number of returned time series in one batch. Default: 32	32
val_batch_size	int	Batch size for the validation dataloader. Affects number of returned time series in one batch. Default: 64	64
test_batch_size	int	Batch size for the test dataloader. Affects number of returned time series in one batch. Default: 128	128
all_batch_size	int	Batch size for the all dataloader. Affects number of returned time series in one batch. Default: 128	128
preprocess_order	list[str, type]	Defines in which order preprocesses are used. Also can add to order a type of AllSeriesCustomHandler or NoFitCustomHandler. Default: ["handling_anomalies", "filling_gaps", "transforming"]	['filling_gaps', 'handling_anomalies', 'transforming']
fill_missing_with	type | FillerType | Literal['mean_filler', 'forward_filler', 'linear_interpolation_filler'] | None	Defines how to fill missing values in the dataset. Can pass enum FillerType for built-in filler or pass a type of custom filler that must derive from Filler base class. Default: None	None
transform_with	type | TransformerType | Transformer | Literal['min_max_scaler', 'standard_scaler', 'max_abs_scaler', 'log_transformer', 'l2_normalizer'] | None	Defines the transformer used to transform the dataset. Can pass enum TransformerType for built-in transformer, pass a type of custom transformer or instance of already fitted transformer. Default: None	None
handle_anomalies_with	type | AnomalyHandlerType | Literal['z-score', 'interquartile_range'] | None	Defines the anomaly handler for handling anomalies in the train set. Can pass enum AnomalyHandlerType for built-in anomaly handler or a type of custom anomaly handler. Default: None	None
partial_fit_initialized_transformer	bool	If True, partial fitting on train set is performed when using initiliazed transformer. Default: False	False
include_time	bool	If True, time data is included in the returned values. Default: True	True
include_ts_id	bool	If True, time series IDs are included in the returned values. Default: True	True
time_format	TimeFormat | Literal['id_time', 'datetime', 'unix_time', 'shifted_unix_time']	Format for the returned time data. When using TimeFormat.DATETIME, time will be returned as separate list along rest of the values. Default: TimeFormat.ID_TIME	ID_TIME
train_workers	int	Number of workers for loading training data. 0 means that the data will be loaded in the main process. Default: 4	4
val_workers	int	Number of workers for loading validation data. 0 means that the data will be loaded in the main process. Default: 3	3
test_workers	int	Number of workers for loading test data. 0 means that the data will be loaded in the main process. Default: 2	2
all_workers	int	Number of workers for loading all data. 0 means that the data will be loaded in the main process. Default: 4	4
init_workers	int	Number of workers for initial dataset processing during configuration. 0 means that the data will be loaded in the main process. Default: 4	4
nan_threshold	float	Maximum allowable percentage of missing data. Time series exceeding this threshold are excluded. Time series over the threshold will not be used. Used for train/val/test/all separately. Default: 1.0	1.0
train_dataloader_order	DataloaderOrder | Literal['random', 'sequential']	Defines the order of data returned by the training dataloader. Default: DataloaderOrder.SEQUENTIAL	SEQUENTIAL
random_state	int | None	Fixes randomness for reproducibility during configuration and dataset initialization. Default: None	None

Config attributes

Attributes:

Name	Type	Description
used_train_workers	Optional[int]	Tracks the number of train workers in use. Helps determine if the train dataloader should be recreated based on worker changes.
used_val_workers	Optional[int]	Tracks the number of validation workers in use. Helps determine if the validation dataloader should be recreated based on worker changes.
used_test_workers	Optional[int]	Tracks the number of test workers in use. Helps determine if the test dataloader should be recreated based on worker changes.
used_all_workers	Optional[int]	Tracks the total number of all workers in use. Helps determine if the all dataloader should be recreated based on worker changes.
uses_all_ts	bool	Whether all time series set should be used.
import_identifier	Optional[str]	Tracks the name of the config upon import. None if not imported.
filler_factory	FillerFactory	Represents factory used to create passed Filler type.
anomaly_handler_factory	AnomalyHandlerFactory	Represents factory used to create passed Anomaly Handler type.
transformer_factory	TransformerFactory	Represents factory used to create passed Transformer type.
can_fit_fillers	bool	Whether fillers in this config, can be fitted.
logger		Logger for displaying information.
all_ts	Optional[ndarray]	If no specific sets (train/val/test) are provided, all time series IDs are used. When any set is defined, only the time series IDs in defined sets are used.
train_ts_row_ranges	Optional[ndarray]	Initialized when train_ts is set. Contains time series IDs in train set with their respective time ID ranges.
val_ts_row_ranges	Optional[ndarray]	Initialized when val_ts is set. Contains time series IDs in validation set with their respective time ID ranges.
test_ts_row_ranges	Optional[ndarray]	Initialized when test_ts is set. Contains time series IDs in test set with their respective time ID ranges.
all_ts_row_ranges	Optional[ndarray]	Initialized when all_ts is set. Contains time series IDs in all set with their respective time ID ranges.
display_time_period	Optional[range]	Used to display the configured value of time_period.
aggregation	Optional[AgreggationType]	The aggregation period used for the data.
source_type	Optional[SourceType]	The source type of the data.
database_name	Optional[str]	Specifies which database this config applies to.
features_to_take_without_ids	Optional[ndarray]	Features to be returned, excluding time or time series IDs.
indices_of_features_to_take_no_ids	Optional[ndarray]	Indices of non-ID features in features_to_take.
ts_id_name	Optional[str]	Name of the time series ID, dependent on source_type.
used_singular_train_time_series	Optional[int]	Currently used singular train set time series for dataloader.
used_singular_val_time_series	Optional[int]	Currently used singular validation set time series for dataloader.
used_singular_test_time_series	Optional[int]	Currently used singular test set time series for dataloader.
used_singular_all_time_series	Optional[int]	Currently used singular all set time series for dataloader.
train_preprocess_order	list[PreprocessNote]	All preprocesses used for train set.
val_preprocess_order	list[PreprocessNote]	All preprocesses used for val set.
test_preprocess_order	list[PreprocessNote]	All preprocesses used for test set.
all_preprocess_order	list[PreprocessNote]	All preprocesses used for all set.
is_initialized	bool	Flag indicating if the configuration has already been initialized. If true, config initialization will be skipped.
version	str	Version of cesnet-tszoo this config was made in.
export_update_needed	bool	Whether config was updated to newer version and should be exported.
time_period	ndarray	Defines the time period for returning data from train/val/test/all. Can be a range of time IDs, a tuple of datetime objects or a float. Float value is equivalent to percentage of available times from start.
train_ts	Optional[ndarray]	Defines which time series IDs are used in the training set. Can be a list of IDs, or an integer/float to specify a random selection. An int specifies the number of random time series, and a float specifies the proportion of available time series. int and float must be greater than 0, and a float should be smaller or equal to 1.0. Using int or float guarantees that no time series from other sets will be used.
val_ts	Optional[ndarray]	Defines which time series IDs are used in the validation set. Same as train_ts but for the validation set.
test_ts	Optional[ndarray]	Defines which time series IDs are used in the test set. Same as train_ts but for the test set.
features_to_take	list[str]	Defines which features are used.
default_values	ndarray	Default values for missing data, applied before fillers. Can set one value for all features or specify for each feature.
train_batch_size	int	Batch size for the train dataloader. Affects number of returned time series in one batch.
val_batch_size	int	Batch size for the validation dataloader. Affects number of returned time series in one batch.
test_batch_size	int	Batch size for the test dataloader. Affects number of returned time series in one batch.
all_batch_size	int	Batch size for the all dataloader. Affects number of returned time series in one batch.
preprocess_order	list[PreprocessType]	Defines in which order preprocesses are used. Also can add to order a type of AllSeriesCustomHandler or NoFitCustomHandler.
partial_fit_initialized_transformer	list[PreprocessType]	If True, partial fitting on train set is performed when using initiliazed transformer.
include_time	bool	If True, time data is included in the returned values.
include_ts_id	bool	If True, time series IDs are included in the returned values.
time_format	TimeFormat	Format for the returned time data. When using TimeFormat.DATETIME, time will be returned as separate list along rest of the values.
train_workers	int	Number of workers for loading training data. 0 means that the data will be loaded in the main process.
val_workers	int	Number of workers for loading validation data. 0 means that the data will be loaded in the main process.
test_workers	int	Number of workers for loading test data. 0 means that the data will be loaded in the main process.
all_workers	int	Number of workers for loading all data. 0 means that the data will be loaded in the main process.
init_workers	int	Number of workers for initial dataset processing during configuration. 0 means that the data will be loaded in the main process.
nan_threshold	float	Maximum allowable percentage of missing data. Time series exceeding this threshold are excluded. Time series over the threshold will not be used. Used for train/val/test/all separately.
train_dataloader_order	DataloaderOrder	Defines the order of data returned by the training dataloader.
random_state	Optional[int]	Fixes randomness for reproducibility during configuration and dataset initialization.