That evaluation method is reminiscent of the traditional approach to assessing new surgical techniques — which have sometimes been shown to be worthless or worse, after being subjected to more rigorous testing. Radical mastectomies, for example, were widely used for decades before randomized trials showed that much less extensive and disfiguring surgery followed by radiation was an equally effective treatment.
Of course, conducting randomized trials can pose more challenges for evaluating a new surgery than a new drug. A surgical technique may be harder to standardize enough to be able to test it on a broad population, and blinding the patient to which treatment she receives can be far more difficult.
But such feasibility issues do not apply to new payment methods, which are well-defined and standardized interventions, and where blinding medical providers to the payment rules is not desired.
Still, as in medicine, not all public policies can — or should — have randomized evaluations. One-of-a-kind government projects — like the “Big Dig” in Boston or the Superconducting Super Collider in Texas — have no natural comparison group, randomized or otherwise. In times of crisis, or when policy disagreements are more about ideology than impact, evaluation itself may be ill advised.
But when — as is often the case — there is an opportunity for prospective evaluation and the law requires it, the Innovation Center’s experience underscores the value and feasibility of randomized trials of social policy. They can often be done at the same speed and cost as any prospective study, and can yield more compelling results. Random assignment where the government chooses by lottery who can receive the program may also be the fairest way to allocate an intervention on a limited basis.
Randomized testing may not be the standard for government evaluation yet, but such things take time. For example, the Food and Drug Administration was empowered in 1962 to obtain “substantial evidence” of a new drug’s safety and efficacy, yet it took more than five years for the agency to embrace randomized trials as the appropriate standard.
Now that the Biden administration is, again, properly emphasizing that all federal agencies need to make “evidence-based decisions” based on the highest scientific standards, perhaps truly rigorous testing of social policy will become as commonplace as it is for new vaccines. That would help ensure that government services are delivered as effectively and efficiently as possible.
Amy Finkelstein is the John and Jennie S. MacDonald professor of economics at M.I.T. She co-directs J-PAL North America, a research center at M.I.T. that conducts randomized evaluations.