Fully realized, an autodata workflow will break a high-level goal like “I want to predict X” or “I want to know why Y is so high” into a set of declarative steps (e.g., “Summarize the data,” “Build the model”) that require little or no custom code to run, but still allow for introspection and iteration. Autodata tools, when used responsibly, can standardize data workflows, improve the quality of models and reports, and save practitioners time.Īutodata doesn’t replace critical thinking: it just means that in fewer lines of code, a data practitioner can follow a best practices. This new world of autodata tools takes some agency away from practitioners in exchange for repeatability and a reduction in repetitive error-prone work. The community has been automating common procedures including data loading, exploratory data analysis, feature engineering, and model-building. Luckily, the data community has been making a lot of common operations less arcane and more repeatable. It’s no surprise that even after checking off every item of a good data practices checklist, the data practitioner doesn’t fully trust their own work. Worse yet, because of the many dependent steps involved in a data workflow, errors compound. The process for each step involves modifications to hundreds/thousands of lines of copy/pasted code, making it easy to forget to tweak a parameter or default. Data practitioners have to perform tens of steps in order to believe their own analyses and models. Much of the work a data scientist or engineer performs today is rote and error-prone.
0 kommentar(er)
