Interstate 94 - Hourly traffic volume prediction (multivariate time series)¶

In this tutorial, we demonstrate a time series application of getML. We predict the hourly traffic volume on I-94 westbound from Minneapolis-St Paul. We benchmark our results against Facebook's Prophet. getML's relational learning algorithms outperform Prophet's classical time series approach by ~15%.

Summary:

• Prediction type: Regression model
• Domain: Transportation
• Prediction target: Hourly traffic volume
• Source data: Multivariate time series, 5 components
• Population size: 24096

Background¶

The dataset features some particularly interesting characteristics common for time series, which classical models may struggle to deal with appropriately. Such characteristics are:

• High frequency (hourly)
• Dependence on irregular events (holidays)
• Strong and overlapping cycles (daily, weekly)
• Anomalies
• Multiple seasonalities

The analysis is built on top of a dataset provided by the MN Department of Transportation, with some data preparation done by John Hogue.

Analysis¶

Let's get started with the analysis and set-up your session:

1. Loading data¶

1.1 Download from source¶

Downloading the raw data and convert it into a prediction ready format takes time. To get to the getML model building as fast as possible, we prepared the data for you and excluded the code from this notebook. It is made available in the example notebook featuring the full analysis. We only include data after 2016 and introduced a fixed train/test split at 80% of the available data.

1.2 Prepare data for getML¶

The getml.datasets.load_interstate94 method took care of the entire data preparation:

• Downloads csv's from our servers into python
• Converts csv's to getML DataFrames
• Sets roles & units to columns inside getML DataFrames

Data visualization

The first week of the original traffic time series is plotted below.

Traffic: population table

To allow the algorithm to capture seasonal information, we include time components (such as the day of the week) as categorical variables. Note that we could have also used getML's Seasonal preprocessor (getml.prepreprocessors.Seasonal()), but in this case the information was already included in the dataset.

Train/test split

We use getML's split functionality to retrieve a lazily evaluated split column, that we can supply to the time series api below.

Split columns are columns of mere strings that can be used to subset the data by forming bolean conditions over them:

1.3 Define relational model¶

To start with relational learning, we need to specify the data model. We manually replicate the appropriate time series structure by setting time series related join conditions (horizon, memory and allow_lagged_targets). We use the high-level time series api for this.

Under the hood, the time series api abstracts away a self cross join of the population table (traffic) that allows getML's feature learning algorithms to learn patterns from past observations.

2.Predictive modeling¶

We loaded the data, defined the roles, units and the abstract data model. Next, we create a getML pipeline for relational learning.

2.1 getML Pipeline¶

Set-up of feature learners, selectors & predictor

Build the pipeline

2.2 Model training¶

2.3 Model evaluation¶

2.4 Studying features¶

Feature correlations

Correlations of the calculated features with the target

Feature importances

Visualizing the learned features

We can also transpile the features as SQL code. Here, we show the most important feature.

To showcase getML's ability to handle categorical data, we now look for features that contain information from the holiday column:

As you can see, getML features which incorporate information about holidays have a rather low importance. This is not that surprising given the fact that most information about holidays is fully reproducible from the extracted calendarial information that is already present. In other words: for the algorithm, it doesn't matter if the traffic is lower on every 4th of July of a given year or if there is a corresponding holiday named 'Independence Day'. Here is the SQL transpilation of the most important feature relying on information about holdidays anyway:

Plot predictions & traffic volume vs. time

We now plot the predictions against the observed values of the target for the first 7 days of the testing set. You can see that the predictions closely follows the original series. RelMT was able to identify certain patterns in the series, including:

• Day and night separation
• The daily commuting peeks (on weekdays)
• The decline on weekends

2.5 Features¶

The most important feature looks as follows:

2.6 Productionization¶

It is possible to productionize the pipeline by transpiling the features into production-ready SQL code. Please also refer to getML's sqlite3 module.

While the feature is less smooth, it is really close to what we got in the 1-step case. This is another indication for the presence of strong time-related patterns in the data.

3. Benchmarks against Prophet¶

By design, Prophet isn't capable of delivering the 1-step ahead predictions we did with getML. In order to retrieve a benchmark in the 1-step case nonetheless, we mimic 1-step ahead predictions through cross validating the model on a rolling origin. This clearly gives Prophet an advantage as all information up to the origin is incorporated when fitting the model and a new fit is calculated for every 1-step ahead forecast. Prophet's performance thus has to be viewed as an upper bound. Further, as noted above, we thought it would be interesting to let Multirel and Relboost figure out time based patterns by itself if we provide only deterministic components. So, in a second step, we benchmark this case against Prophet. For both tools, we use very simple models with all hyperparameters set to their default values.

Here we fit the model. We fit a new Prophet model for every hour. This is a computationally expensive operation even for our test subsample of 30 days. For the 720 hours (= 720 models) it may take up to 6 hours. Therefore we read the predictions from disc. If you want to re-estimate the Prophet predictions, you can rename the csv files on disc.

Score the model.

getML is able to outperform Prophet's 1-step ahead predictions by about 14 percentage points in terms of predictive accuracy. This is a substantial margin. But we have to state that it may be an unfair comparison because we use Prophet for an application it wasn't designed for. To deal with this critique we also calculate h-step predictions below.

Next, we visually compare the 1-step ahead Prophet with 1-step ahead getML predictions.

Note that with getML we calculate predictions for the full sample (instead of just 30 days). This plays as another advantage for Prophet.

We inspect the predictions of both models over the course of a week. Here, we plot the getML and Prophet predictions against actual values (data).

What can we take from this plot? The strong time-related patterns carry on to the testing set and the xgboost is able to incorporate this information to deliver highly accurate 1-step ahead predictions. Notice that Prophets additive components model results in negative predictions for the weekends' lows at night. We can also see an anomaly on March, 24 that neither model is able to predict (feel free to play with the window to verify that its an anomaly).

Now we benchmark the performance of h-step ahead forecasts. Remember, in the models, only deterministic features are incorporated. Here, we fit the h-step ahead forecast with Prophet. This is how Prophet is meant to be applied. We allow for multiplicative seasonalities to at least partially remedy the problem with negative predictions discussed above.

A typical Prophet components graph.

Seasonalities are now applied multiplicatively, but the combination of the totally bottoming out daily cycle and the negative additive trend will result in negative predictions nonetheless.

Score the Prophet model.

4. Conclusion¶

Benchmarks against Prophet

By design, Prophet isn't capable of delivering the 1-step ahead predictions we did with getML. To retrieve a benchmark in the 1-step case nonetheless, we mimic 1-step ahead predictions through cross-validating the model on a rolling origin. This gives Prophet an advantage as all information up to the origin is incorporated when fitting the model and a new fit is calculated for every 1-step ahead forecast.

Results

We have benchmarked getML against Facebook’s Prophet library on a univariate time series with strong seasonal components. Prophet is made for exactly these sort of data sets, so you would expect this to be a home run for Prophet. The opposite is true - getML’s relational learning algorithms outperform Prophet's 1-step ahead predictions by ~15 percentage points: