Research and Policy Analyst · Papers and presentations

EI Data Process Overview

I presented the new process to a partner team. This two-page summary, written for this site, shows what the work set out to achieve, without the internal detail.

Public summary · two pages, written for this siteWhat we set out to achieve → how we got there
Same results, lower cost, a process the team can ownBefore: a data process in SAS: Proprietary software, almost 30 years old; On a special-purpose server; Code that was hard to read and risky to change. The goals: Same results: Every value identical to the output of the SAS programs it was built to replace. Lower cost, open tools: Free, open-source R and DuckDB on a powerful cloud computer on Azure: no SAS licence. A process the team can own: Explained page by page, so colleagues can read it, run it and change it. WHAT WE SET OUT TO ACHIEVESame results, lower cost, a process the team can own BEFOREA data processin SAS Proprietary software,almost 30 years oldOn a special-purposeserverCode that was hard to readand risky to changeTHE GOALS 1 Same resultsEvery value identical to the output ofthe SAS programs it was built to replace. 2 Lower cost, open toolsFree, open-source R and DuckDB on a powerfulcloud computer on Azure: no SAS licence. 3 A process the team can ownExplained page by page, so colleaguescan read it, run it and change it. EI Data Process Overview · public summaryThomas Eapen · 1 / 2

What we set out to achieve

Same results, lower cost, a process the team can own

Before

A data process in SAS

Proprietary software, almost 30 years old. On a special-purpose server. Code that was hard to read and risky to change.

Same results

Every value identical to the output of the SAS programs it was built to replace.

Lower cost, open tools

Free, open-source R and DuckDB on a powerful cloud computer on Azure: no SAS licence.

A process the team can own

Explained page by page, so colleagues can read it, run it and change it.

Proven step by step, run with one commandFor every step: Rewrite: AI agents rewrite one step of the SAS code in R and DuckDB. Prove: Every value is compared with the SAS output. Decide: I rule on every difference and give my approval. Then the next step, until every step is proven. What changed: Every value identical to SAS; As fast as SAS or faster; No licence fees; One command runs the process; About 550 pages of documentation. HOW WE GOT THEREProven step by step, run with one command FOR EVERY STEP AI AGENTS RewriteAI agents rewrite onestep of the SAS codein R and DuckDB. THE TEST ProveEvery value iscompared with theSAS output. ME DecideI rule on everydifference and givemy approval.Then the next step, until every step is proven.An AI agentAn automated testMineWHAT CHANGED Every value identical to SASAs fast as SAS or fasterNo licence feesOne command runs the processAbout 550 pages of documentation EI Data Process Overview · public summaryThomas Eapen · 2 / 2

How we got there

Proven step by step, run with one command

AI agents

Rewrite

AI agents rewrite one step of the SAS code in R and DuckDB.

↓

The test

Prove

Every value is compared with the SAS output.

↓

Me

Decide

I rule on every difference and give my approval.

Then the next step, until every step is proven.

What changed

  • Every value identical to SAS
  • As fast as SAS or faster
  • No licence fees
  • One command runs the process
  • About 550 pages of documentation

Written for this site from the EI Data Conversion section. The presentation itself described internal processes and is not published; no internal process detail or data is shown here.