Agricultural productivity growth is vital for economic and food security outcomes which are threatened by climate change. In response, governments and development agencies are encouraging the adoption of ‘climate-smart’ agricultural technologies, such as conservation agriculture (CA). However, there is little rigorous evidence that demonstrates the effect of CA on production or climate resilience, and what evidence exists is hampered by selection bias. Using panel data from Zimbabwe, we test how CA performs during extreme rainfall events - both shortfalls and surpluses. We control for the endogenous adoption decision and find that use of CA in years of average rainfall results in no yield gains, and in some cases yield loses. However, CA is effective in mitigating the negative impacts of deviations in rainfall. We conclude that the lower yields during normal rainfall seasons may be a proximate factor in low uptake of CA. Policy should focus promotion of CA on these climate resilience benefits.