getml.data.Container

Container(
    population: Optional[Union[DataFrame, View]] = None,
    peripheral: Optional[
        Dict[str, Union[DataFrame, View]]
    ] = None,
    split: Optional[
        Union[StringColumn, StringColumnView]
    ] = None,
    deep_copy: Optional[bool] = False,
    train: Optional[Union[DataFrame, View]] = None,
    validation: Optional[Union[DataFrame, View]] = None,
    test: Optional[Union[DataFrame, View]] = None,
    **kwargs: Optional[Union[DataFrame, View]]
)

A container holds the actual data in the form of a DataFrame or a View.

The purpose of a container is twofold:

• Assigning concrete data to an abstract DataModel.

• Storing data and allowing you to reproduce previous results.

ATTRIBUTE	DESCRIPTION
population	The population table defines the statistical population of the machine learning problem and contains the target variables.
peripheral	The peripheral tables are joined onto population or other peripheral tables. Note that you can also pass them using add.
split	Contains information on how you want to split population into different Subsets. Also refer to split.
deep_copy	Whether you want to create deep copies or your tables.
train	The population table used in the train Subset. You can either pass population and split or you can pass the subsets separately using train, validation, test and kwargs.
validation	The population table used in the validation Subset. You can either pass population and split or you can pass the subsets separately using train, validation, test and kwargs.
test	The population table used in the test Subset. You can either pass population and split or you can pass the subsets separately using train, validation, test and kwargs.
kwargs	The population table used in Subsets other than the predefined train, validation and test subsets. You can call these subsets anything you want to, and you can access them just like train, validation and test. You can either pass population and split or you can pass the subsets separately using train, validation, test and kwargs. Example # Pass the subset. container = getml.data.Container(my_subset=my_data_frame) # You can access the subset just like train, # validation or test my_pipeline.fit(container.my_subset)

Example

A DataModel only contains abstract data. When we fit a pipeline, we need to assign concrete data.

This example is taken from the loans notebook . Note that in the notebook the high level StarSchema implementation is used. For demonstration purposes we are proceeding now with the low level implementation.

# The abstract data model is constructed
# using the DataModel class. A data model
# does not contain any actual data. It just
# defines the abstract relational structure.
dm = getml.data.DataModel(
    population_train.to_placeholder("population")
)

dm.add(getml.data.to_placeholder(
    meta=meta,
    order=order,
    trans=trans)
)

dm.population.join(
    dm.trans,
    on="account_id",
    time_stamps=("date_loan", "date")
)

dm.population.join(
    dm.order,
    on="account_id",
)

dm.population.join(
    dm.meta,
    on="account_id",
)

# We now have abstract placeholders on something
# called "population", "meta", "order" and "trans".
# But how do we assign concrete data? By using
# a container.
container = getml.data.Container(
    train=population_train,
    test=population_test
)

# meta, order and trans are either
# DataFrames or Views. Their aliases need
# to match the names of the placeholders in the
# data model.
container.add(
    meta=meta,
    order=order,
    trans=trans
)

# Freezing makes the container immutable.
# This is not required, but often a good idea.
container.freeze()

# When we call 'train', the container
# will return the train set and the
# peripheral tables.
my_pipeline.fit(container.train)

# Same for 'test'
my_pipeline.score(container.test)

If you don't already have a train and test set, you can use a function from the split module.

split = getml.data.split.random(
    train=0.8, test=0.2)

container = getml.data.Container(
    population=population_all,
    split=split,
)

# The remaining code is the same as in
# the example above. In particular,
# container.train and container.test
# work just like above.

Containers can also be used for storage and reproducing your results. A recommended pattern is to assign 'baseline roles' to your data frames and then using a View to tweak them:

# Assign baseline roles
data_frame.set_role(["jk"], getml.data.roles.join_key)
data_frame.set_role(["col1", "col2"], getml.data.roles.categorical)
data_frame.set_role(["col3", "col4"], getml.data.roles.numerical)
data_frame.set_role(["col5"], getml.data.roles.target)

# Make the data frame immutable, so in-place operations are
# no longer possible.
data_frame.freeze()

# Save the data frame.
data_frame.save()

# I suspect that col1 leads to overfitting, so I will drop it.
view = data_frame.drop(["col1"])

# Insert the view into a container.
container = getml.data.Container(...)
container.add(some_alias=view)
container.save()

The advantage of using such a pattern is that it enables you to always completely retrace your entire pipeline without creating deep copies of the data frames whenever you have made a small change like the one in our example. Note that the pipeline will record which container you have used.

Source code in getml/data/container.py

def __init__(
    self,
    population: Optional[Union[DataFrame, View]] = None,
    peripheral: Optional[Dict[str, Union[DataFrame, View]]] = None,
    split: Optional[Union[StringColumn, StringColumnView]] = None,
    deep_copy: Optional[bool] = False,
    train: Optional[Union[DataFrame, View]] = None,
    validation: Optional[Union[DataFrame, View]] = None,
    test: Optional[Union[DataFrame, View]] = None,
    **kwargs: Optional[Union[DataFrame, View]],
):
    if population is not None and not isinstance(population, (DataFrame, View)):
        raise TypeError(
            "'population' must be a getml.DataFrame or a getml.data.View, got "
            + type(population).__name__
            + "."
        )

    if peripheral is not None and not _is_typed_dict(
        peripheral, str, [DataFrame, View]
    ):
        raise TypeError(
            "'peripheral' must be a dict "
            + "of getml.DataFrames or getml.data.Views."
        )

    if split is not None and not isinstance(
        split, (StringColumn, StringColumnView)
    ):
        raise TypeError(
            "'split' must be StringColumn or a StringColumnView, got "
            + type(split).__name__
            + "."
        )

    if not isinstance(deep_copy, bool):
        raise TypeError(
            "'deep_copy' must be a bool, got " + type(split).__name__ + "."
        )

    exclusive = (population is not None) ^ (
        len(_make_subsets_from_kwargs(train, validation, test, **kwargs)) != 0
    )

    if not exclusive:
        raise ValueError(
            "'population' and 'train', 'validation', 'test' as well as "
            + "other subsets signified by kwargs are mutually exclusive. "
            + "You have to pass "
            + "either 'population' or some subsets, but you cannot pass both."
        )

    if population is None and split is not None:
        raise ValueError(
            "'split's are used for splitting population DataFrames."
            "Hence, if you supply 'split', you also have to supply "
            "a population."
        )

    if population is not None and split is None:
        logger.warning(
            "You have passed a population table without passing 'split'. "
            "You can access the entire set to pass to your pipeline "
            "using the .full attribute."
        )
        split = from_value("full")

    self._id = _make_id()

    self._population = population
    self._peripheral = peripheral or {}
    self._split = split
    self._deep_copy = deep_copy

    self._subsets = (
        _make_subsets_from_split(population, split)
        if split is not None
        else _make_subsets_from_kwargs(train, validation, test, **kwargs)
    )

    if split is None and not _is_typed_dict(self._subsets, str, [DataFrame, View]):
        raise TypeError(
            "'train', 'validation', 'test' and all other subsets must be either a "
            "getml.DataFrame or a getml.data.View."
        )

    if deep_copy:
        self._population = _deep_copy(self._population, self._id)
        self._peripheral = {
            k: _deep_copy(v, self._id) for (k, v) in self._peripheral.items()
        }
        self._subsets = {
            k: _deep_copy(v, self._id) for (k, v) in self._subsets.items()
        }

    self._last_change = _get_last_change(
        self._population, self._peripheral, self._subsets
    )

    self._frozen_time = None

add

add(*args, **kwargs)

Adds new peripheral data frames or views.

Source code in getml/data/container.py

def add(self, *args, **kwargs):
    """
    Adds new peripheral data frames or views.
    """
    wrong_type = [item for item in args if not isinstance(item, (DataFrame, View))]

    if wrong_type:
        raise TypeError(
            "All unnamed arguments must be getml.DataFrames or getml.data.Views."
        )

    wrong_type = [
        k for (k, v) in kwargs.items() if not isinstance(v, (DataFrame, View))
    ]

    if wrong_type:
        raise TypeError(
            "You must pass getml.DataFrames or getml.data.Views, "
            f"but the following arguments were neither: {wrong_type!r}."
        )

    kwargs = {**{item.name: item for item in args}, **kwargs}

    if self._frozen_time is not None:
        raise ValueError(
            f"You cannot add data frames after the {type(self).__name__} has been frozen."
        )

    if self._deep_copy:
        kwargs = {k: _deep_copy(v, self._id) for (k, v) in kwargs.items()}

    self._peripheral = {**self._peripheral, **kwargs}

    self._last_change = _get_last_change(
        self._population, self._peripheral, self._subsets
    )

freeze

freeze()

Freezes the container, so that changes are no longer possible.

This is required before you can extract data when deep_copy=True. The idea of deep_copy is to ensure that you can always retrace and reproduce your results. That is why the container needs to be immutable before it can be used.

Source code in getml/data/container.py

def freeze(self):
    """
    Freezes the container, so that changes are no longer possible.

    This is required before you can extract data when `deep_copy=True`. The idea of
    `deep_copy` is to ensure that you can always retrace and reproduce your results.
    That is why the container needs to be immutable before it can be
    used.
    """
    self.sync()
    self._frozen_time = datetime.now().strftime("%Y-%m-%d %H:%M:%S")

save

save()

Saves the Container to disk.

Source code in getml/data/container.py

def save(self):
    """
    Saves the Container to disk.
    """

    cmd = dict()
    cmd["type_"] = "DataContainer.save"
    cmd["name_"] = self._id

    cmd["container_"] = self._getml_deserialize()

    comm.send(cmd)

sync

sync()

Synchronizes the last change with the data to avoid warnings that the data has been changed.

This is only a problem when deep_copy=False.

Source code in getml/data/container.py

def sync(self):
    """
    Synchronizes the last change with the data to avoid warnings that the data
    has been changed.

    This is only a problem when `deep_copy=False`.
    """
    if self._frozen_time is not None:
        raise ValueError(f"{type(self).__name__} has already been frozen.")
    self._last_change = _get_last_change(
        self._population, self._peripheral, self._subsets
    )

to_pandas

to_pandas() -> Dict[str, DataFrame]

Returns a Container's contents as a dictionary of pandas.DataFrames. key holds the data frame's name, value the data converted to a pandas.DataFrame.

Source code in getml/data/container.py

def to_pandas(self) -> Dict[str, pd.DataFrame]:
    """
    Returns a `Container`'s contents as a dictionary of `pandas.DataFrame`s.
    `key` holds the data frame's `name`, value the data converted to a `pandas.DataFrame`.
    """
    subsets = (
        {name: df.to_pandas() for name, df in self._subsets.items()}
        if self._subsets
        else {}
    )
    peripherals = (
        {name: df.to_pandas() for name, df in self.peripheral.items()}
        if self.peripheral
        else {}
    )
    if subsets or peripherals:
        return {**subsets, **peripherals}

    raise ValueError("Container is empty.")