Caution: Study is no breakthrough


Caution: study is no breakthrough

July 9, 2015 | by Scott Pigg, Principal Researcher, Seventhwave

A recent study questioning the savings achieved by low-income weatherization programs has been generating considerable buzz in the energy efficiency community, especially after the Washington Post, the Wall Street Journal and other mainstream news organizations ran sensationalized articles covering it. The study, issued by researchers from the University of Chicago and the University of California, Berkeley, looked at energy savings for a few weatherization agencies in Michigan, and concluded that savings fall far short of expectations and are not cost effective. The authors have also claimed that the study deserves special attention, because…

“Our study is different. It represents a first-of-its-kind evaluation using a randomized controlled trial, the gold standard for rigorous evaluation…Our application of this approach to residential energy efficiency measures is therefore an important departure from, and improvement upon, previous analyses.”

However, a close read suggests that the study was actually something of an expensive fizzle and, based on what I’ve seen over my 28-year career evaluating similar programs, it sheds almost no new light on savings from weatherization.

The study relied on a technique called randomized encouragement to get households to sign up for weatherization. Despite spending close to half a million dollars on the effort, in the end only about 400 of 30,000 homes included in the study actually received weatherization services.

The low participation rate translates into high uncertainty in the results. The study found average savings of about 20 percent from the program, but the statistical uncertainty interval associated with this estimate is an astonishing ±16 percentage points.* Put another way, one can only confidently conclude from the study that the energy savings from the program are somewhere between 4 and 36 percent. It’s hard to conclude much of anything from such a wide range, let alone level the sweeping indictment of energy efficiency programs implied by the study and its subsequent press coverage.

Far from being an improvement on more traditional approaches to measuring savings from weatherization programs—that typically use later program participants to control for non-program influences on energy use—this seems to be a case of the cure being worse than the disease. In attempting to avoid what I, along with other knowledgeable program evaluators, would contend is a few percentage points of uncertainty from not being able to randomly assign who gets treated and who doesn’t, the authors ended up with a huge margin of uncertainty on their results.

Also troubling is the study’s claim that the measured energy savings fell far short of the average 46 percent projected savings. To those unfamiliar with weatherization, the message is clear: these programs don’t deliver their claimed savings. But what caught my eye is not that the savings fell short, but that the homes being studied had average projected savings that were so high to begin with. There are undoubtedly homes out there that need every measure offered by the program, and can achieve this level of savings. But most households that participate in these programs need only a subset of the improvements that the program offers, and these would not be expected to save nearly as much energy as implied by the reported 46 percent average.

What might have gone wrong with the projected savings in this case? It is well known that computer models have a tendency to over-predict savings, especially in the hands of people who are new to them. Critically, the software used for estimating savings in Michigan was introduced just around the time the study started—and at a time of severe program upheaval due to a massive influx of additional funding under the American Recovery and Reinvestment Act. Those circumstances, and the uncharacteristically high projected savings themselves, make it hard to get very worked up about the savings falling so short of expectations in this case.

Savvy program operators use actual savings gleaned from program evaluation to calibrate future savings projections. Iowa’s low-income program is a good example of such a feedback loop: the most recent evaluation report for the program found 25 percent savings on natural gas, and, notably, that these savings were 99 percent of their projected level.**

The one novel contribution that the study makes is in providing evidence that participants don’t appear to take back any substantial savings on their heating bill by increasing their thermostat settings after weatherization. The study looked at indoor temperatures and thermostat settings before and after weatherization for both treated and untreated households and found little difference. It’s too bad that that little gem has been lost in the hoopla surrounding the more sensational—and less substantiated—claims that the study makes.

In truth, many studies of savings from low-income weatherization programs over the years have found savings in the range of 15 to 25 percent. These are real and substantial reductions in energy consumption. Readers of this particular study should use caution and avoid believing that the study represents a breakthrough in our understanding about savings from these programs.

* for the statistically inclined, I’m reporting the 95% confidence interval here.

** full disclosure: I was involved in the evaluation of this program in the mid 1990s, and helped establish some of the evaluation protocols that are used by it.