Data Science Asked by Matteo Giani on April 11, 2021
The vast majority of my DS projects begin with the creation of a simple pipeline to
read or convert the original files/db
filter, extract and clean some dataset
which has as a result a dataset I can use to compute features and train/validate/test my model(s) in other pipelines.
For efficiency reasons, I cache the result of this dataset locally. That can be in the simplest case, for instance to run a first analysis, a .pkl file containing a pandas dataframe; or it can be data stored in a local database. This data is then typically analyzed in my notebooks.
Now, in the course of a project it can be that either the original data structure or some script used in the pipeline itself changes. Then, the entire pipeline needs to be re-run because the cached data is invalid.
Do you know of a tool that allows you to check on this? Ideally, a notebook extension that warns you if the cached data became invalid.
Get help from others!
Recent Questions
Recent Answers
© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP