Air Pollution - Why feature learning is better than simple propositionalization¶

Background¶

Many data scientists and AutoML tools use propositionalization methods for feature engineering. These propositionalization methods usually work as follows:

• Generate a large number of hard-coded features,
• Use feature selection to pick a percentage of these features.

By contrast, getML contains approaches for feature learning: Feature learning adapts machine learning approaches such as decision trees or gradient boosting to the problem of extracting features from relational data and time series.

In this notebook, we will benchmark getML against featuretools and tsfresh. Both of these libaries use propositionalization approaches for feature engineering.

As our example dataset, we use a publicly available dataset on air pollution in Beijing, China: Beijing PM2.5 Data. The data set has been originally used in the following study:

Liang, X., Zou, T., Guo, B., Li, S., Zhang, H., Zhang, S., Huang, H. and Chen, S. X. (2015). Assessing Beijing's PM2.5 pollution: severity, weather impact, APEC and winter heating. Proceedings of the Royal Society A, 471, 20150257.

We find that getML significantly outperforms featuretools and tsfresh in terms of predictive accuracy ( see Discussion ). Our findings indicate that getML's feature learning algorithms are better at adapting to data sets and are also more scalable due to their lower memory requirement.

Analysis¶

1. Loading data
2. Predictive modeling
3. Discussion
4. Conclusion

We start the analysis with the setup of our session.

1. Loading data¶

1.1 Download from source¶

Downloading the raw data from the UCI Machine Learning Repository into a prediction ready format takes time. To get to the getML model building as fast as possible, we prepared the data for you and excluded the code from this notebook.

2. Predictive modeling¶

2.1 Pipeline 1: Complex features, 7 days¶

First, we spilt our data. We introduce a simple, time-based split and use all data until 2013-12-31 for training and everything starting from 2014-01-01 for testing.

For our first experiment, we will learn complex features and allow a memory of up to seven days. That means at every given point in time, the algorithm is allowed to back seven days into the past.

It is good practice to always check your data model first, even though check(...) is also called by fit(...). That enables us to make last-minute changes.

We now fit our data on the training set and evaluate our findings, both, in-sample and out-of-sample.

2.2 Pipeline 2: Complex features, 1 day¶

2.3 Pipeline 3: Simple features, 7 days¶

For our third experiment, we will learn simple features and allow a memory of up to seven days.

2.4 Pipeline 4: Simple features, 1 day¶

For our fourth experiment, we will learn simple features and allow a memory of up to one day.

2.5 Using featuretools¶

To make things a bit easier, we have written high-level wrappers around featuretools and tsfresh which we placed in a separate module (utils).

tsfresh and featuretools require the time series to have ids. Since there is only a single time series, that series has the same id.

2.6 Using tsfresh¶

Next, we construct features with tsfresh. tsfresh is based on pandas and rely\ies on explicit copies for meny operations. This leads to an excessive memory consumption that renders tsfresh nearly unusable for real-world scenarios. Remeber, this is a relatively small data set.

To limit the memory consumption, we undertake the following steps:

• We limit ourselves to a memory of 1 day from any point in time. This is necessary, because tsfresh duplicates records for every time stamp. That means that looking back 7 days instead of one day, the memory consumption would be seven times as high.
• We extract only tsfresh's MinimalFCParameters and IndexBasedFCParameters (the latter is a superset of TimeBasedFCParameters).

In order to make sure that tsfresh's features can be compared to getML's features, we also do the following:

• We apply tsfresh's built-in feature selection algorithm.
• Of the remaining features, we only keep the 40 features most correlated with the target (in terms of the absolute value of the correlation).
• We add the original columns as additional features.

tsfresh does not contain built-in machine learning algorithms. In order to ensure a fair comparison, we use the exact same machine learning algorithm we have also used for getML: An XGBoost regressor with all hyperparameters set to their default values.

In order to do so, we load the tsfresh features into the getML engine.

As usual, we need to set roles:

In this case, our pipeline is very simple. It only consists of a single XGBoostRegressor.

2.7 Studying features¶

This is a typical RelMT feature, where the aggregation (SUM in this case) is applied conditionally – the conditions are learned by RelMT – to a set of linear models, whose weights are, again, learned by RelMT.

2.8 Productionization¶

It is possible to productionize the pipeline by transpiling the features into production-ready SQL code. Please also refer to getML's sqlite3 module.

3. Discussion¶

We have seen that getML outperforms tsfresh by more than 10 percentage points in terms of R-squared. We now want to analyze why that is.

There are two possible hypotheses:

• getML outperforms featuretools and tsfresh, because it using feature learning and is able to produce more complex features
• getML outperforms featuretools and tsfresh, because it makes better use of memory and is able to look back further.

Let's summarize our findings:

The summary table shows that combination of both of our hypotheses explains why getML outperforms featuretools and tsfresh. Complex features do better than simple features with a memory of one day. With a memory of seven days, simple features actually get worse. But when you look back seven days and allow more complex features, you get good results.

This suggests that getML outperforms featuretools and tsfresh, because it can make more efficient use of memory and thus look back further. Because RelMT uses feature learning and can build more complex features it can make better use of the greater look-back window.

4. Conclusion¶

We have compared getML's feature learning algorithms to tsfresh's brute-force feature engineering approaches on a data set related to air pollution in China. We found that getML significantly outperforms featuretools and tsfresh. These results are consistent with the view that feature learning can yield significant improvements over simple propositionalization approaches.

However, there are other datasets on which simple propositionalization performs well. Our suggestion is therefore to think of algorithms like FastProp and RelMT as tools in a toolbox. If a simple tool like FastProp gets the job done, then use that. But when you need more advanced approaches, like RelMT, you should have them at your disposal as well.

You are encouraged to reproduce these results.