From varied inputs to useful outputs

An Improved Data Pipeline

I presented improvements we had made to our data pipeline, explaining how a common data structure supported reproducible preparation, analysis and communication across a wider range of tools.

A common structure. More possibilities.Illustrative reconstruction

A shared structure connects the workflow

A shared structure connects the workflowIllustrative reconstruction. Excel sheets, data tables and reference sources converge into a common tidy structure. R and Python support reproducible preparation and analysis, with a return to preparation as questions develop. Excel, PowerPoint and graphical outputs remain available for communication. Refine as the question develops VARIED INPUTS Excel sheetsData tablesReferencesources 01 / STRUCTURE Commontidy data 02 / PREPAREReproducibletransformationsR · Python 03 / ANALYSEExploreand interpretR · Python USEFUL OUTPUTS ExcelPowerPointGraphics

Excel sheets · Data tables · Reference sources

  1. 01

    Common tidy data

    Bring varied inputs into a consistent structure.

  2. 02

    Reproducible preparation

    Use R and Python for transformations that can run again.

  3. 03

    Explore and interpret

    Analyse with R and Python; refine preparation as the question develops.

Keep several output options:
Excel · PowerPoint · Graphics

Illustrative reconstruction · generic inputs and outputs

Bring inputs together

Different source formats become easier to work with when names, types and structure follow a consistent pattern.

Work across tools

We added R and Python alongside Excel. A common data structure supported reproducible preparation, visualisation and modelling across tools.

Keep useful output options

Excel and PowerPoint remained available for output alongside visualisation tools, all supported by the same structured data.

My talk connected collection, preparation and communication as parts of one workflow.