Research and Policy Analyst · Papers and presentations
EI Data Process Overview
I presented the new process to a partner team. This two-page summary, written for this site, shows what the work set out to achieve, without the internal detail.
Public summary · two pages, written for this siteWhat we set out to achieve → how we got there
What we set out to achieve
Same results, lower cost, a process the team can own
Before
A data process in SAS
Proprietary software, almost 30 years old. On a special-purpose server. Code that was hard to read and risky to change.
Same results
Every value identical to the output of the SAS programs it was built to replace.
Lower cost, open tools
Free, open-source R and DuckDB on a powerful cloud computer on Azure: no SAS licence.
A process the team can own
Explained page by page, so colleagues can read it, run it and change it.
How we got there
Proven step by step, run with one command
AI agents
Rewrite
AI agents rewrite one step of the SAS code in R and DuckDB.
↓
The test
Prove
Every value is compared with the SAS output.
↓
Me
Decide
I rule on every difference and give my approval.
Then the next step, until every step is proven.
What changed
Every value identical to SAS
As fast as SAS or faster
No licence fees
One command runs the process
About 550 pages of documentation
Written for this site from the EI Data Conversion section. The presentation itself described internal processes and is not published; no internal process detail or data is shown here.