Data Officer · Paper excerpt
An Improved Data
Exploration Pipeline
I argued that reproducible preparation gives analysts more time to explore their data and communicate useful findings.
Prepare
Import → Tidy → Manage
Explore
Transform · Visualize · Model
Repeat as the questions develop.
Communicate
Explain the finding.
Make preparation reproducible
Use programmatic access and tidy structures to make retrieval and transformation reproducible across collection cycles.
Keep the rules with the data
Use structured storage, explicit relationships and verification rules to support consistent calculations and readable outputs.
Invest in exploration
Choose tools for the problem, assess the process and keep learning so more effort can go into analysis and communication.
The paper describes programmatic retrieval and structured data management already in use, then argues for continued investment in tools, learning and exploration.
Condensed and edited from An Improved Data Exploration Pipeline, sections 2–6. The paper cites R for Data Science for its exploration framework.