Impact evaluations (IEs) of new technologies are critical to improving investment in national and international development goals. Yet many technologies are introduced at times or in places that lack the necessary data to conduct a well identified IE. We present a new method that combines Earth observation (EO) data, advances in machine learning, and survey data so as to allow researchers to conduct IEs when traditional economic data is missing. To demonstrate our approach, we study stress tolerant rice varieties (STRVs) introduced more than a decade ago. Using 20 years of EO data on rice production and flooding, we fail to replicate existing evidence of STRV effectiveness. We validate this failure to replicate with household panel data and through Monte Carlo simulations that demonstrate the sensitivity of past evidence to mismeasurement. Our findings speak to the challenges and promises of using EO data to conduct IEs in data poor settings.