The Economist just posted an article on the state of microfinance in Bangladesh. I’m surprised at how weak it is:
- The article reports that on April 6, the government of Bangladesh arrogated for itself the right to appoint members of the board of the Grameen Bank, while providing almost no interpretive context for this move.
- It describes a new study of the impact of microcredit in Bangladesh by Shahidur Khandker and Hussain Samad as the “biggest” so far, by which it seems to mean the one with the most households in it. Not so. The new study has 1,509–2,322 households depending on how you count (see the paper’s Table 1); as a counterexample, a study in Hyderabad, India, had about 6,850. The new study is however the longest, tracking families for a remarkable 20 years.
- The article describes the positive results of this study as breaking a pattern of research finding “limited or no benefits” of microcredit. Actually, it continues a pattern. Shahid Khandker has authored many papers with similar conclusions: Pitt and Khandker (1998); Pitt and Khandker (2002); Pitt, Khandker, Chowdhury, and Millimet (2003); Khandker (2005); Pitt, Khandker, and Cartwright (2006); Pitt and Khandker (2012); Khandker and Samad (2013); Khandker, Samad, and Ali (2013)… The pattern continued after randomized studies began appearing in 2009 that produced the less-positive findings mentioned.
- More fundamentally, the article evinces no understanding of why those randomized studies are more trustworthy (see below).
- The article appears to commit what Dierdre McCloskey and Stephen Ziliak dub the “standard error of regressions,” which is to confuse statistical significance with real-world significance. Statistical significance, as meant here, is the certainty that the impact of microcredit is not zero. Real-world significance is whether the effect is big enough to matter. “A 10% increase in men’s borrowing raises household spending by 0.04%….Borrowing by women pushes up household spending by one and a half times as much.” Let’s see…because of compounding, seven 10% increases would about suffice to about double borrowing. So doubling female borrowing will lift household spending by \(7 \times 0.0 4\% \times 1.5 = 0.42\%\). To me, that seems small—about a sixth of the impact found in the first study of these families.
Alright, I guess that’s enough beating up on an anonymous journalist. The important question is what to make of the new study. I say: unfortunately, not much.
The issue is standard. Correlation is not causation. The paper makes a strikingly confident statement about how one thing affects another: “microcredit programs have continued to benefit the poor by raising household welfare.” The problem is that in families and villages, everything affects everything. Taking more microloans can make people wealthier or poorer. Being wealthier or poorer can make people take more microloans. The arrows go in circles. Statistics can measure correlations. How do we make the leap to causation?
The burden is on the researcher evincing such certainty to rule out competing explanations such as reverse causation. In this paper, that requires demonstrating and exploiting the equivalent of an experiment. If, for some arbitrary reason, certain families borrowed more than others, a comparison between them could be persuasive. The classic example is some being randomly offered microcredit. But that is not what happened in Bangladesh. Rather, the asserted basis for a “quasi-experiment” in this study is copied from Pitt and Khandker’s first analysis of these families, the one published in 1998. It is that only “landless” families (owning less than half an acre of land) could borrow, at least from the Grameen Bank.
As the launchpad for an experiment that reveals the impact of microcredit, there is a big problem with this splitting. As Jonathan Morduch showed in 1998, the half-acre rule wasn’t followed in practice. Lots of people with well more than half an acre borrowed. Of course, not all did. This leads to the strong possibility that the real basis for the split between borrowers and non-borrowers was differences in entrepreneurial talent (capacity to use credit productively) or demonstrated reliability (capacity to obtain credit). Khandker and Samad’s results may simply be telling us is that people who were more driven and trustworthy did a little better in life 20 years later.
Pitt and Khandker have never truly rebutted Jonathan’s criticism. In fact, Pitt (1999) wrote that the rule dividing borrowers from non-borrowers is “unknown.” Owning more or less than half an acre is asserted to be one factor, but there is no sign of that in the data, and it could easily be confounded by other factors such drive and reliability.
Unfortunately, the Khandker and Samad paper does not even address this issues; it does not shoulder the burden I described. It never defends the assumption that you need to believe in order to buy the paper’s conclusions. It is like asking you to accept a proof in geometry while avoiding scrutiny of the starting axioms about lines and angles.
The statement of the key assumption is at the top of page 15: “The necessary assumption is that the availability of a credit group by gender in a village is uncorrelated with the differenced household errors, ε, conditional on X.” To be fair, few journalists reading that could tell what is missing.