Data engineering
Automating data work
Manual steps replaced by programs that run from start to finish.
ToolsR · Python · SQL
- Research and Policy AnalystAutomation: a direct link to our databaseEI MAR Automation
- Data OfficerAutomatic generation of hierarchical outputs
- Workforce analyst rolesDatabase Programming: SQL Server (T-SQL)
Replacing legacy systems
SAS programs rebuilt in free, open-source tools.
ToolsR · DuckDB · Python
- Research and Policy AnalystEI Data Conversion (SAS to R)
- Data Science ProgrammerThe Migration Project
Proving the results
New code checked against the old system before it is trusted.
ToolsR · DuckDB · ggplot2
- Research and Policy AnalystEI Data Conversion (SAS to R)
- Data Science ProgrammerEnsuring the accuracy of the migration
- Workforce analyst rolesTesting
Speed and large data
Faster database connections, work spread across processor cores, and data too large for memory.
ToolsArrow · DuckDB · ROracle · furrr
- Research and Policy AnalystAzure Infrastructure
- Data Science ProgrammerIncreasing the performance of the connection between R and the Oracle databaseIncreasing the performance of the R code
- Workforce analyst rolesData Analytics
Databases and applications
Code that reads straight from the database, and a web application to manage one.
ToolsSQL · Oracle · Django · SQL Server
- Research and Policy AnalystAutomation: a direct link to our database
- Data OfficerWTO Database Application
- Workforce analyst rolesDatabase Programming: SQL Server (T-SQL)
Modelling and forecasting
Optimization to fill gaps in data, and forecasting methods.
ToolsPython · Pyomo · GLPK
- Data Science ProgrammerSolving non-trivial problems
- Data OfficerPython Programming Showcase
- Workforce analyst rolesPredictive Tools
Platforms and ways of working
Cloud platform
Analysis tools and code repositories set up on the department’s cloud platform.
ToolsAzure ML · Azure DevOps
- Research and Policy AnalystAzure Infrastructure
Working with AI agents
AI agents write and document code under my direction; I review it and run it myself when real data is involved.
ToolsAI agents
- Research and Policy AnalystEI Data Conversion (SAS to R)EI MAR AutomationAI Hackathon: drafting decision records
Version control
Git for the team’s code, with a shared repository colleagues can work from.
ToolsGit
- Research and Policy AnalystGit Workflow
- Data Science ProgrammerGit workflow presentation
Communicating the work
Maps, charts and publication design
Maps, infographics, dashboards and a publication layout.
ToolsR · Python · Power BI · Adobe InDesign
- Data OfficerData manipulation and geospatial mappingInfographicsPublication Design
- Workforce analyst rolesReporting and visualisation
Papers, talks and documentation
Work explained in papers, talks and written documentation, so others can run it and change it.
- Research and Policy AnalystPapers and presentations
- Data Science ProgrammerTalks & Presentations
- Data OfficerTalksPapers and briefings
- Workforce analyst rolesProcess Documentation