Failed pilots: evaluations of blockchain interventions need transparency

  • Danilo Zak
26 March, 2019

In her book on humanitarian technology, K.L. Jacobsen writes that ‘humanitarian actors deploy new technologies experimentally, and thus expose already vulnerable populations to additional sources of harm.’ Blockchain for development programs, including ID2020’s pilots in Thailand and Indonesia and the World Food Program’s Building Blocks in Pakistan and Jordan, serve as useful archetypes for the ways in which forced migrants are used as experimental subjects when new technologies arrive on the humanitarian scene.

But beyond the inherent ethical issues that arise from a framework that assumes refugees can be test subjects, the very terms of these ‘experiments’ on refugee bodies must be further interrogated. What constitutes failure in so-called proof-of-concept blockchain interventions? What questions are asked about outcomes, and how are they answered? How should blockchain-based pilot projects self-report to maximize long term success and preserve the rights of beneficiaries?

Why Blockchain?

There are many potential applications that make blockchain-based technology so enticing for humanitarian donors and aid agencies. It allows for the construction of auditable, decentralized ledgers, which offers a range of potential humanitarian applications from improving backend efficiency in cash transfers to digital/biometric identification provision. The vision is a decentralized infrastructure that will enable more transparent, more accountable forms of delivering aid, cutting inefficient governments and middlemen out of the picture while retaining data privacy for beneficiaries. These are not just pipe dreams: All over the world, blockchain technology is being used today to support refugees and marginalized groups, albeit almost always in an explicitly exploratory capacity. 

The Risk

Just as interest in blockchain-based aid has exploded among donors and NGOs, many academics have collectively raised three pointed criticisms. Some studies have noted serious ethical concerns about the feasibility of legitimate informed and sustained consent, especially in the contexts of vulnerable populations. Others have raised data privacy and surveillance concerns. Finally, concerning overall utility, Margie Cheesman of the Oxford Internet Institute concluded that ‘working on best practice may involve pointing out when blockchain isn’t really necessary.’ ‘Pilot’ and ‘proof-of-concept’ projects should be focused on measuring outcomes which answer the questions these criticisms raise. If they fail to respond effectively to these issues, programs should be considered failures.

Defining the Meterics: WFP Building Blocks

The World Food Program’s (WFP) Building Blocks intervention is using a blockchain-based system to distribute conditional cash transfers to refugees in camps. The program allows beneficiaries to access their account via an iris scanner, and is currently in operation in Azraq and Zaatari refugee camps in Jordan after an initial pilot in Pakistan. Building Blocks is growing and set to expand to all 500,000 of WFP’s beneficiaries in Jordan and move into programming in two other states. It now stands as a shining example of how a blockchain intervention can grow from a successful pilot to a significant element of an organization’s programming.

However, WFP has been extremely vague about what metrics it has used to gauge the success of Building Blocks. On the initial pilot in Pakistan, WFP reports only that they ‘confirmed basic assumptions about the capabilities of blockchain in authenticating and registering transactions.’ About the first stage of the rollout in Jordan, the only positive outcome measured appeared to reducing backend banking fees for the organisation by 98%. WFP initially stated that the program had no effect on beneficiaries, but recently has made claims that it ‘provides greater security and sensitive data is no longer shared with third parties.’

The reduction in backend fees is significant and nothing to sneer at. The saved money could be rerouted to provide for more people in need. But there is no evidence that WFP is seriously interested in outcomes relating to data privacy, consent, or even utility for beneficiaries. WFP has not reported tangibly on these outcomes for Building Blocks, nor shown any genuine efforts to measure them. The ‘proof-of-concept’ had no defined metric for failure (or success) whatsoever. As a private blockchain-based aid program that combines biometric digital identification with cash transfers, issues of privacy, consent, and utility are fundamental concerns. WFP and Building Blocks have failed to respond adequately in the exploratory phase of their intervention.

ID2020 Thailand and Indonesia Pilots

ID2020 is a public-private partnership, which includes massive corporate players such as Microsoft, that aims to respond to the one billion people in the world who lack state recognized identification. ID2020 has raised millions of dollars and initiated two pilot programs, both running blockchain-based systems. In Thailand, the organization has partnered with the IRC and a nonprofit called iRespond to provide blockchain based iris identification in the Mae La refugee camp. In Indonesia, ID2020 has partnered with a company called Everest to provide secure Liquid Gas Propane (LPG) subsidies on a closed blockchain. The stated outcomes of interest include ‘enhanced financial inclusion,’ ‘modernizing delivery,’ and to ‘reduce financial leakage.’

It is unclear what mechanisms are in place for measuring these outcomes. After visiting the Thai and Indonesian sites in 2018, ID2020 announced they were proud of the projects’ tangible, positive impact, but without ever explaining what this positive impact was or how it was measured. ID2020 also left out outcomes of interest in the initial reporting, as well as any clear discussion about data privacy and consent. These are very new projects, and ID2020 might introduce more rigor and transparency to their pilot interventions in the future. Still, it is concerning that, as in the case with WFP Building Blocks, there does not appear to be any defined metric for ‘failure’ in these explicitly ‘proof-of-concept’ interventions.

IFRC and Red Rose Cash Transfer

The International Federation of the Red Cross and Red Crescent Societies (IFRC) and a humanitarian technology company called RedRose partnered in 2017 to pilot a blockchain-based Cash Transfer Program (CTP) in Kenya. The program was funded by IFRC winning a grant that specified the piloting of a blockchain based CTP pilot. The program enlisted a Red Rose to use blockchain technology to build a data management system that was integrated into the more traditional Safaricom M-pesa mobile money network.

IFRC released a Learning Review of the program analysing its flaws and benefits. It revealed some significant positives of the program, including the provision of ‘an additional layer of security and integrity to CTP’. But it also revealed some serious concerns. On informed consent, the IFRC reports that after the project ‘it remains unclear how informed consent can be obtained from beneficiaries for projects involving complex data structures like blockchain’.

Concerning data privacy, IFRC relied on the European Union General Data Protection Regulation (GDPR) as well as the Handbook on Data Protection in Humanitarian Action to ensure the privacy of the beneficiaries was protected. Yet due to these privacy concerns and lack of buy-in from beneficiaries and third parties, Red Rose was left writing on the closed blockchain for those actors. This meant in practice, the technology provided little more utility than a difficult to alter database.

The IFRC program and subsequent Learning Review can be seen as both a positive and a negative. While the program was far more transparent than WFP and ID2020 about critical concerns surrounding blockchain pilots, its outcomesonly reified those concerns, especially regardinginformed consent and the overall utility of blockchain-based systems. The IFRC program could have also been clearer and more rigorous about how these outcomes were examined. The Learning Review refuses to define the project as a failure, despite it seeming to imply that the use of blockchain was unsuccessful in critical ways. 

Transparency is Key

Blockchain technology offers serious potential in the humanitarian sphere. But ultimately, while Blockchain-for-Development projects are all self-defined as ‘proof-of-concept,’ they lack the transparency, pedagogical rigor, and ethical care to approximate legitimate evaluations of impact. If aid organizations are interested in the sustained success of blockchain-based programming, they must take the experimental nature of these projects seriously, including setting terms for failure.

Experimenting on marginalized populations must bring with it rigorous ethical, qualitative, and quantitative review.  This process begins and ends with transparency. Given the extremely limited information available now, we should view all three pilot interventions discussed above as failures. The next crop of blockchain-for-development pilots can, and should, do better.

Note: This post has been edited to reflect that iRespond is a nonprofit, rather than a corporation, and to clarify the description of Microsoft as “spearheading” ID2020.

Subscribe to the blog

About the author(s)
Danilo Zak