TransWikia.com

Automatic caching (validation) system for pipelines?

Data Science Asked by Matteo Giani on April 11, 2021

The vast majority of my DS projects begin with the creation of a simple pipeline to

  • read or convert the original files/db

  • filter, extract and clean some dataset

which has as a result a dataset I can use to compute features and train/validate/test my model(s) in other pipelines.

For efficiency reasons, I cache the result of this dataset locally. That can be in the simplest case, for instance to run a first analysis, a .pkl file containing a pandas dataframe; or it can be data stored in a local database. This data is then typically analyzed in my notebooks.

Now, in the course of a project it can be that either the original data structure or some script used in the pipeline itself changes. Then, the entire pipeline needs to be re-run because the cached data is invalid.

Do you know of a tool that allows you to check on this? Ideally, a notebook extension that warns you if the cached data became invalid.

Add your own answers!

Ask a Question

Get help from others!

© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP