From varied inputs to useful outputs
An Improved Data Pipeline
I presented improvements we had made to our data pipeline, explaining how a common data structure supported reproducible preparation, analysis and communication across a wider range of tools.
A shared structure connects the workflow
Excel sheets · Data tables · Reference sources
- 01
Common tidy data
Bring varied inputs into a consistent structure.
- 02
Reproducible preparation
Use R and Python for transformations that can run again.
- 03
Explore and interpret
Analyse with R and Python; refine preparation as the question develops.
Keep several output options:
Excel · PowerPoint · Graphics
Illustrative reconstruction · generic inputs and outputs
Bring inputs together
Different source formats become easier to work with when names, types and structure follow a consistent pattern.
Work across tools
We added R and Python alongside Excel. A common data structure supported reproducible preparation, visualisation and modelling across tools.
Keep useful output options
Excel and PowerPoint remained available for output alongside visualisation tools, all supported by the same structured data.
My talk connected collection, preparation and communication as parts of one workflow.