Interstate 94 - Multivariate time series prediction¶

In this notebbok, we compare getML's FastProp against well-known feature engineering libraries featuretools and tsfresh.

Summary:

Prediction type: Regression model
Domain: Transportation
Prediction target: Hourly traffic volume
Source data: Multivariate time series, 5 components
Population size: 24096

Background¶

A common approach to feature engineering is to generate attribute-value representations from relational data by applying a fixed set of aggregations to columns of interest and perform a feature selection on the (possibly large) set of generated features afterwards. In academia, this approach is called propositionalization.

getML's FastProp is an implementation of this propositionalization approach that has been optimized for speed and memory efficiency. In this notebook, we want to demonstrate how – well – fast FastProp is. To this end, we will benchmark FastProp against the popular feature engineering libraries featuretools and tsfresh. Both of these libraries use propositionalization approaches for feature engineering.

In this notebook, we predict the hourly traffic volume on I-94 westbound from Minneapolis-St Paul. The analysis is built on top of a dataset provided by the MN Department of Transportation, with some data preparation done by John Hogue. For further details about the data set refer to the full notebook.

Let's get started with the analysis and set-up your session:

In [1]:

Copied!





import datetime
import os
os.environ["PYARROW_IGNORE_TIMEZONE"] = "1"
from pathlib import Path

import sys
import time

import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
import pandas as pd
from IPython.display import Image

%matplotlib inline

import getml

print(f"getML API version: {getml.__version__}\n")

getml.engine.launch(home_directory=Path.home(), allow_remote_ips=True, token='token')
getml.engine.set_project("interstate94")
import datetime
import os
os.environ["PYARROW_IGNORE_TIMEZONE"] = "1"
from pathlib import Path

import sys
import time

import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
import pandas as pd
from IPython.display import Image

%matplotlib inline

import getml

print(f"getML API version: {getml.__version__}\n")

getml.engine.launch(home_directory=Path.home(), allow_remote_ips=True, token='token')
getml.engine.set_project("interstate94")

getML API version: 1.4.0

getML engine is already running.
Loading pipelines... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          

Connected to project 'interstate94'

In [2]:

Copied!

parent = Path(os.getcwd()).parent.as_posix()

if parent not in sys.path:
   sys.path.append(parent) 

from utils import Benchmark, FTTimeSeriesBuilder, TSFreshBuilder
import woodwork as ww
parent = Path(os.getcwd()).parent.as_posix()

if parent not in sys.path:
   sys.path.append(parent) 

from utils import Benchmark, FTTimeSeriesBuilder, TSFreshBuilder
import woodwork as ww

1. Loading data¶

1.1 Download from source¶

We begin by downloading the data from the UC Irvine Machine Learning repository:

In [3]:

Copied!

traffic = getml.datasets.load_interstate94(roles=True, units=True)
traffic = getml.datasets.load_interstate94(roles=True, units=True)

Loading traffic...
 100% |██████████| [elapsed: 00:00, remaining: 00:00]

In [4]:

Copied!

traffic.set_role(traffic.roles.categorical, getml.data.roles.unused_string)
traffic.set_role(traffic.roles.categorical, getml.data.roles.unused_string)

In [5]:

Copied!

traffic
traffic

Out[5]:

name	ds	traffic_volume	holiday	day	month	weekday	hour	year
role	time_stamp	target	unused_string	unused_string	unused_string	unused_string	unused_string	unused_string
unit	time stamp, comparison only			day	month	weekday	hour	year
0	2016-01-01	1513	New Years Day	1	1	4	0	2016
1	2016-01-01 01:00:00	1550	New Years Day	1	1	4	1	2016
2	2016-01-01 02:00:00	993	New Years Day	1	1	4	2	2016
3	2016-01-01 03:00:00	719	New Years Day	1	1	4	3	2016
4	2016-01-01 04:00:00	533	New Years Day	1	1	4	4	2016
	...	...	...	...	...	...	...	...
24091	2018-09-30 19:00:00	3543	No holiday	30	9	6	19	2018
24092	2018-09-30 20:00:00	2781	No holiday	30	9	6	20	2018
24093	2018-09-30 21:00:00	2159	No holiday	30	9	6	21	2018
24094	2018-09-30 22:00:00	1450	No holiday	30	9	6	22	2018
24095	2018-09-30 23:00:00	954	No holiday	30	9	6	23	2018

24096 rows x 8 columns
memory usage: 2.16 MB
name: traffic
type: getml.DataFrame

1.2 Define relational model¶

In [6]:

Copied!

split = getml.data.split.time(traffic, "ds", test=getml.data.time.datetime(2018, 3, 15))
split = getml.data.split.time(traffic, "ds", test=getml.data.time.datetime(2018, 3, 15))

In [7]:

Copied!





time_series = getml.data.TimeSeries(
    population=traffic,
    split=split,
    alias="traffic",
    time_stamps="ds",
    horizon=getml.data.time.hours(1),
    memory=getml.data.time.hours(24),
    lagged_targets=True,
)

time_series
time_series = getml.data.TimeSeries(
    population=traffic,
    split=split,
    alias="traffic",
    time_stamps="ds",
    horizon=getml.data.time.hours(1),
    memory=getml.data.time.hours(24),
    lagged_targets=True,
)

time_series

Out[7]:

data model

diagram

staging

	data frames	staging table
0	traffic	TRAFFIC__STAGING_TABLE_1
1	traffic	TRAFFIC__STAGING_TABLE_2

container

population

	subset	name	rows	type
0	test	traffic	4800	View
1	train	traffic	19296	View

peripheral

	name	rows	type
0	traffic	24096	DataFrame

2. Predictive modeling¶

We loaded the data, defined the roles, units and the abstract data model. Next, we create a getML pipeline for relational learning.

2.1 Propositionalization with getML's FastProp¶

In [8]:

Copied!





seasonal = getml.preprocessors.Seasonal()

fast_prop = getml.feature_learning.FastProp(
    loss_function=getml.feature_learning.loss_functions.SquareLoss,
    num_threads=1,
)
seasonal = getml.preprocessors.Seasonal()

fast_prop = getml.feature_learning.FastProp(
    loss_function=getml.feature_learning.loss_functions.SquareLoss,
    num_threads=1,
)

Build the pipeline

In [9]:

Copied!





pipe_fp_fl = getml.pipeline.Pipeline(
    preprocessors=[seasonal],
    feature_learners=[fast_prop],
    data_model=time_series.data_model,
    tags=["feature learning", "fastprop"],
)

pipe_fp_fl
pipe_fp_fl = getml.pipeline.Pipeline(
    preprocessors=[seasonal],
    feature_learners=[fast_prop],
    data_model=time_series.data_model,
    tags=["feature learning", "fastprop"],
)

pipe_fp_fl

Out[9]:

Pipeline(data_model='traffic',
         feature_learners=['FastProp'],
         feature_selectors=[],
         include_categorical=False,
         loss_function='SquareLoss',
         peripheral=['traffic'],
         predictors=[],
         preprocessors=['Seasonal'],
         share_selected_features=0.5,
         tags=['feature learning', 'fastprop'])

In [10]:

Copied!

pipe_fp_fl.check(time_series.train)
pipe_fp_fl.check(time_series.train)

Checking data model...
Staging... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          
Preprocessing... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          
Checking... 100% |██████████| [elapsed: 00:01, remaining: 00:00]          

OK.

In [11]:

Copied!

benchmark = Benchmark()
benchmark = Benchmark()

In [12]:

Copied!

with benchmark("fastprop"):
    pipe_fp_fl.fit(time_series.train)
    fastprop_train = pipe_fp_fl.transform(time_series.train, df_name="fastprop_train")
with benchmark("fastprop"):
    pipe_fp_fl.fit(time_series.train)
    fastprop_train = pipe_fp_fl.transform(time_series.train, df_name="fastprop_train")

Checking data model...
Staging... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          
Preprocessing... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          

OK.
Staging... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          
Preprocessing... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          
FastProp: Trying 365 features... 100% |██████████| [elapsed: 00:02, remaining: 00:00]          

Trained pipeline.
Time taken: 0h:0m:2.505792

Staging... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          
Preprocessing... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          
FastProp: Building features... 100% |██████████| [elapsed: 00:01, remaining: 00:00]

In [13]:

Copied!

fastprop_test = pipe_fp_fl.transform(time_series.test, df_name="fastprop_test")
fastprop_test = pipe_fp_fl.transform(time_series.test, df_name="fastprop_test")

Staging... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          
Preprocessing... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          
FastProp: Building features... 100% |██████████| [elapsed: 00:00, remaining: 00:00]

In [14]:

Copied!

predictor = getml.predictors.XGBoostRegressor()

pipe_fp_pr = getml.pipeline.Pipeline(
    tags=["prediction", "fastprop"], predictors=[predictor]
)
predictor = getml.predictors.XGBoostRegressor()

pipe_fp_pr = getml.pipeline.Pipeline(
    tags=["prediction", "fastprop"], predictors=[predictor]
)

In [15]:

Copied!

pipe_fp_pr.fit(fastprop_train)
pipe_fp_pr.fit(fastprop_train)

Checking data model...
Staging... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          
Checking... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          

OK.
Staging... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          
XGBoost: Training as predictor... 100% |██████████| [elapsed: 00:05, remaining: 00:00]          

Trained pipeline.
Time taken: 0h:0m:5.173095

Out[15]:

Pipeline(data_model='population',
         feature_learners=[],
         feature_selectors=[],
         include_categorical=False,
         loss_function='SquareLoss',
         peripheral=[],
         predictors=['XGBoostRegressor'],
         preprocessors=[],
         share_selected_features=0.5,
         tags=['prediction', 'fastprop'])

In [16]:

Copied!

pipe_fp_pr.score(fastprop_test)
pipe_fp_pr.score(fastprop_test)

Staging... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          
Preprocessing... 100% |██████████| [elapsed: 00:00, remaining: 00:00]

Out[16]:

	date time	set used	target	mae	rmse	rsquared
0	2024-02-21 16:13:15	fastprop_train	traffic_volume	198.9482	292.2493	0.9779
1	2024-02-21 16:13:15	fastprop_test	traffic_volume	180.4867	261.9389	0.9827

2.2 Propositionalization with featuretools¶

In [17]:

Copied!

traffic_train = time_series.train.population
traffic_test = time_series.test.population
traffic_train = time_series.train.population
traffic_test = time_series.test.population

In [18]:

Copied!

dfs_pandas = {}

for df in [traffic_train, traffic_test, traffic]:
    dfs_pandas[df.name] = df.drop(df.roles.unused).to_pandas()
    dfs_pandas[df.name]["join_key"] = 1
dfs_pandas = {}

for df in [traffic_train, traffic_test, traffic]:
    dfs_pandas[df.name] = df.drop(df.roles.unused).to_pandas()
    dfs_pandas[df.name]["join_key"] = 1

In [19]:

Copied!





ft_builder = FTTimeSeriesBuilder(
    num_features=200,
    horizon=pd.Timedelta(hours=1),
    memory=pd.Timedelta(hours=24),
    column_id="join_key",
    time_stamp="ds",
    target="traffic_volume",
    allow_lagged_targets=True,
)
ft_builder = FTTimeSeriesBuilder(
    num_features=200,
    horizon=pd.Timedelta(hours=1),
    memory=pd.Timedelta(hours=24),
    column_id="join_key",
    time_stamp="ds",
    target="traffic_volume",
    allow_lagged_targets=True,
)

In [20]:

Copied!

with benchmark("featuretools"):
    featuretools_train = ft_builder.fit(dfs_pandas["train"])

featuretools_test = ft_builder.transform(dfs_pandas["test"])
with benchmark("featuretools"):
    featuretools_train = ft_builder.fit(dfs_pandas["train"])

featuretools_test = ft_builder.transform(dfs_pandas["test"])

featuretools: Trying features...
Selecting the best out of 118 features...
Time taken: 0h:4m:46.444548

In [21]:

Copied!





roles = {
    getml.data.roles.join_key: ["join_key"],
    getml.data.roles.target: ["traffic_volume"],
    getml.data.roles.time_stamp: ["ds"],
}

df_featuretools_train = getml.data.DataFrame.from_pandas(
    featuretools_train, name="featuretools_train", roles=roles
)

df_featuretools_test = getml.data.DataFrame.from_pandas(
    featuretools_test, name="featuretools_test", roles=roles
)
roles = {
    getml.data.roles.join_key: ["join_key"],
    getml.data.roles.target: ["traffic_volume"],
    getml.data.roles.time_stamp: ["ds"],
}

df_featuretools_train = getml.data.DataFrame.from_pandas(
    featuretools_train, name="featuretools_train", roles=roles
)

df_featuretools_test = getml.data.DataFrame.from_pandas(
    featuretools_test, name="featuretools_test", roles=roles
)

In [22]:

Copied!





df_featuretools_train.set_role(
    df_featuretools_train.roles.unused, getml.data.roles.numerical
)

df_featuretools_test.set_role(
    df_featuretools_test.roles.unused, getml.data.roles.numerical
)
df_featuretools_train.set_role(
    df_featuretools_train.roles.unused, getml.data.roles.numerical
)

df_featuretools_test.set_role(
    df_featuretools_test.roles.unused, getml.data.roles.numerical
)

In [23]:

Copied!

predictor = getml.predictors.XGBoostRegressor()

pipe_ft_pr = getml.pipeline.Pipeline(
    tags=["prediction", "featuretools"], predictors=[predictor]
)

pipe_ft_pr
predictor = getml.predictors.XGBoostRegressor()

pipe_ft_pr = getml.pipeline.Pipeline(
    tags=["prediction", "featuretools"], predictors=[predictor]
)

pipe_ft_pr

Out[23]:

Pipeline(data_model='population',
         feature_learners=[],
         feature_selectors=[],
         include_categorical=False,
         loss_function='SquareLoss',
         peripheral=[],
         predictors=['XGBoostRegressor'],
         preprocessors=[],
         share_selected_features=0.5,
         tags=['prediction', 'featuretools'])

In [24]:

Copied!

pipe_ft_pr.check(df_featuretools_train)
pipe_ft_pr.check(df_featuretools_train)

Checking data model...
Staging... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          
Checking... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          

OK.

In [25]:

Copied!

pipe_ft_pr.fit(df_featuretools_train)
pipe_ft_pr.fit(df_featuretools_train)

Checking data model...
Staging... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          

OK.
Staging... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          
XGBoost: Training as predictor... 100% |██████████| [elapsed: 00:02, remaining: 00:00]          

Trained pipeline.
Time taken: 0h:0m:2.260849

Out[25]:

Pipeline(data_model='population',
         feature_learners=[],
         feature_selectors=[],
         include_categorical=False,
         loss_function='SquareLoss',
         peripheral=[],
         predictors=['XGBoostRegressor'],
         preprocessors=[],
         share_selected_features=0.5,
         tags=['prediction', 'featuretools'])

In [26]:

Copied!

pipe_ft_pr.score(df_featuretools_test)
pipe_ft_pr.score(df_featuretools_test)

Staging... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          
Preprocessing... 100% |██████████| [elapsed: 00:00, remaining: 00:00]

Out[26]:

	date time	set used	target	mae	rmse	rsquared
0	2024-02-21 16:19:18	featuretools_train	traffic_volume	220.4023	321.1657	0.9734
1	2024-02-21 16:19:18	featuretools_test	traffic_volume	210.1988	317.52	0.9746

2.3 Propositionalization with tsfresh¶

tsfresh failed to run through due to an apparent bug in the tsfresh library and is therefore excluded from this analysis.

3. Comparison¶

In [27]:

Copied!





num_features = dict(
    fastprop=461,
    featuretools=59,
)

runtime_per_feature = [
    benchmark.runtimes["fastprop"] / num_features["fastprop"],
    benchmark.runtimes["featuretools"] / num_features["featuretools"],
]

features_per_second = [1.0 / r.total_seconds() for r in runtime_per_feature]

normalized_runtime_per_feature = [
    r / runtime_per_feature[0] for r in runtime_per_feature
]

comparison = pd.DataFrame(
    dict(
        runtime=[benchmark.runtimes["fastprop"], benchmark.runtimes["featuretools"]],
        num_features=num_features.values(),
        features_per_second=features_per_second,
        normalized_runtime=[
            1,
            benchmark.runtimes["featuretools"] / benchmark.runtimes["fastprop"],
        ],
        normalized_runtime_per_feature=normalized_runtime_per_feature,
        rsquared=[pipe_fp_pr.rsquared, pipe_ft_pr.rsquared],
        rmse=[pipe_fp_pr.rmse, pipe_ft_pr.rmse],
        mae=[pipe_fp_pr.mae, pipe_ft_pr.mae],
    )
)

comparison.index = ["getML: FastProp", "featuretools"]
num_features = dict(
    fastprop=461,
    featuretools=59,
)

runtime_per_feature = [
    benchmark.runtimes["fastprop"] / num_features["fastprop"],
    benchmark.runtimes["featuretools"] / num_features["featuretools"],
]

features_per_second = [1.0 / r.total_seconds() for r in runtime_per_feature]

normalized_runtime_per_feature = [
    r / runtime_per_feature[0] for r in runtime_per_feature
]

comparison = pd.DataFrame(
    dict(
        runtime=[benchmark.runtimes["fastprop"], benchmark.runtimes["featuretools"]],
        num_features=num_features.values(),
        features_per_second=features_per_second,
        normalized_runtime=[
            1,
            benchmark.runtimes["featuretools"] / benchmark.runtimes["fastprop"],
        ],
        normalized_runtime_per_feature=normalized_runtime_per_feature,
        rsquared=[pipe_fp_pr.rsquared, pipe_ft_pr.rsquared],
        rmse=[pipe_fp_pr.rmse, pipe_ft_pr.rmse],
        mae=[pipe_fp_pr.mae, pipe_ft_pr.mae],
    )
)

comparison.index = ["getML: FastProp", "featuretools"]

In [28]:

Copied!

comparison
comparison

Out[28]:

	runtime	num_features	features_per_second	normalized_runtime	normalized_runtime_per_feature	rsquared	rmse	mae
getML: FastProp	0 days 00:00:03.963151	461	116.319646	1.000000	1.000000	0.982678	261.938873	180.486734
featuretools	0 days 00:04:46.446138	59	0.205972	72.277372	564.734093	0.974582	317.519976	210.198793

In [29]:

Copied!

comparison.to_csv("comparisons/interstate94.csv")
comparison.to_csv("comparisons/interstate94.csv")