The Digital Benefits Network Showcases Twelve Generative AI Experiments for Benefits Policy at Policy2Code Demo Day
The Policy2Code Prototyping Challenge—organized by the Digital Benefits Network (DBN) at the Beeck Center for Social Impact and Innovation and the Massive Data Institute (MDI), both based at Georgetown University—culminated in the Policy2Code Demo Day on September 17, 2024 during BenCon 2024. Twelve teams presented their experiments, prototypes, and other developments to an audience of technologists, data scientists, civic tech specialists, public benefits advocates, non-profit organizations, private industry consultants, researchers, students, and government officials for feedback, awareness, and evaluation.
Since June 2024, Policy2Code Prototyping Challenge teams tested ways in which generative AI tools—such as Large Language Models (LLMs)—could help make policy implementation more efficient by converting policies into plain language logic models and software code under a Rules as Code (RaC) approach. Teams tested specific technologies and use cases ranging from eligibility screening, caseworker support, and translation to logic and code for a variety of U.S. public benefit programs, including SNAP, Medicaid, SSI/SSDI, LIHEAP, and Veterans benefits.
Key takeaways from the demos include:
- Currently available web-based chatbots, such as OpenAI’s ChatGPT and Google’s Gemini, have high error rates and are likely giving inaccurate information to people seeking benefits today.
- It is possible to improve results by integrating specific policy documents into the pipeline (as Retrieval Augmented Generation (RAGs) or through fine-tuning).
- Results can be further improved by giving a “template” of what to expect.
- Approaches that incorporate human-in-the-loop evaluation and verification will be more successful in the immediate future.
- We have a lot to learn and improve before we can fully automate policy to code.
