Evidence-Based Blockchain: Findings from a Global Study of Blockchain Projects and Start-up Companies

Evidence-based applications of resources remain one of the greatest challenges faced by governments, businesses, and policymakers. The United States Government Accountability Office (GAO) evaluated ten large programs, which together cost more than $10 billion/year, through randomised control trials – the highest standard of evidence-based practice (EBP) [1]. The evaluation found that nine of them had ‘weak or no positive effects’ on their participants. Many programs were not evaluated at all [2]. In January 2019, U.S. President signed the ‘Foundations for Evidence-based Policy Making Act’ into law [3]. A USAID (US Agency for International Development) study looked at 43 blockchain projects and companies claiming to have solved various problems using distributed ledgers [4]. The study found that almost no company was willing to share their results and MERL (monitoring, evaluation, research and learning) processes [5]. Other observational data revealed that 80–90% of blockchain-based token offering projects failed to deliver on their promises [6], a prediction also made by Vitalik Buterin, the founder of Ethereum blockchain, in 2017 [7]. The concept of evidence-based blockchain (EBB) was first introduced by Naqvi in 2018 [8]. We conducted an evaluation of 517 blockchain firms against PCIO framework of evidence-based practice: Problem – Comparison – Intervention and Outcomes. We define the fundamentals of EBB (Ask, Acquire, Appraise, Apply, Assess), provide a review of the literature on EBB, report findings of our study and propose an Assessment Framework of Evidence Based Blockchain (Figure 12).


Introduction
Evidence-based practice (EBP) is the idea that professional practices should be based on a combination of critical thinking and the best available evidence [9]. However, a study showed that 98% of managers failed to apply best practices when making decisions [10]. In blockchain, research showed that cognitive biases and behavioural heuristics can influence the decision support systems of professionals [11,12].
In the United States, ten large decades-old social programs, which together cost more than $10 billion a year, were subjected to randomised controlled trials, the highest standard of evaluation. The evaluation found that nine of them had 'weak or no positive effects' on their participants. Many programs were not evaluated at all [2,3]. In 2019, President Trump signed the Foundations for Evidence-Based Policy Making Act, making it a law to practice evidence-based policymaking [3]. The book Show me the Evidence [13] [ Figure 3] describes the life story of Barack Obama's fight to ensure that government initiatives are based on robust scientific evidence.

Context and history of EBP
The concept of EBP was first introduced in medicine in 1972 by Archibald Cochrane in his landmark book Effectiveness and Efficiency: Random Reflections on Health Services [14]. Cochrane observed that patients were dying unnecessarily and expressed his concerns over the scarcity of scientific evidence used by the NHS to evaluate the effectiveness of therapies and the use of available resources [15]. In 1991, Gordon Guyatt of McMaster University formally coined the term 'evidence-based medicine' [16].

Timeline
Important timelines of major disciplines embarking on the journey towards evidence-based practice:

Centres advancing EBP
Around the globe, there are now over two dozen 'centres of excellence' advancing evidence-based practices ( Figure 1). The Centre for Evidence-Based Blockchain (CEBB) operates under the auspices of the British Blockchain Association as the world's first centre for distributed ledger technologies advancing evidence-based practices ( Figure 2). There are also numerous books written on the topic of evidence-based practice ( Figure 3).

What is the evidence for EBP?
A study was conducted that examined two groups of senior decision makers -one group was asked to make decisions based on the best available scientific evidence and the other was asked to simply make decisions based on factors such as instincts, organisational policies and personal experience. The results were striking: the group that utilised EBP achieved the desired result 90% of the time, had a 50% reduction in their failure rate and a six-fold increase in the number of correct business decisions. Furthermore, this group exceeded expectations only 40% of the time, compared to the other group that practiced conventional decision making [10].

Why EBB?
EBB attempts to solve five major problems in the blockchain space. The first problem is the inability to clearly define the problem to be solved. Sometimes blockchain is applied to a problem that does not exist or is not significant enough to  There are many examples  from the 2017-18 ICO boom where many projects, while not necessarily scams [26], failed to materialise. Many were considered to be seen as 'blockchain -a solution in search for a problem' also called excessive 'blockchainising' (or solutioneering) [27]. This is a significant problem as it wastes time and resources ( Figure 5).
The second major problem is that we do not examine different sources of evidence and do not always start by searching for the best available scientific evidence. We often rely on superficial Google searches, magazines, expert opinions and blog posts to make judgements about a particular problem, which is often a significant mistake ( Figure 5).
The third problem is inadequate evaluation of the quality of evidence. Often, this is because we have not been trained to provide adequate evaluations or do not think doing so is important. For example, we may go to an event and hear someone talking about their blockchain solution or idea and may not ask the speaker whether this been independently peer reviewed, externally validated or impartially evaluated.
The fourth problem is the lack of application of evidence to improve processes. Interventions and solutions are proposed with no objective scientific evidence to back up their efficacy or effectiveness ( Figure 6).
The final problem is that we often inadequately report the outcomes and results of our experiments, especially when the results are unfavourable. A study by the US Agency for International Development examined 43 blockchain use cases and companies using blockchain that claimed to have solved various problems using distributed ledgers. They found that almost no company was willing to share the data on the results and MERL (monitoring, evaluation, research and learning) processes [5], an observation consistent with our research findings ( Figure 10).

What is EBB?
We define EBB as conscientious, explicit and judicious decision making based on professional expertise and evidence from organisations, stakeholders and scientific research.

How to practice EBB?
EBB is a five-step approach consisting of the '5 As': 1. Formulate a precise question (ASK). 2. Search for the evidence and look for answers to the question (ACQUIRE). 3. Critically appraise the evidence (APPRAISE). 4. Apply the results to your practice (APPLY).

Monitor any changes and evaluate (ASSESS).
These five steps should be followed to evaluate both the problem and the solution.

ASK: Formulate a precise question
Clearly defining the problem is the first step to practicing EBB. One should always ask the following: 'What exactly is the problem here?' 'What is it that we are trying to solve?' 'How exactly are we trying to address this issue?' Instead of asking 'should I use blockchain for my supply chain business?', one should clearly define the type of blockchain, the intervention, the comparison group and the desired outcomes by using the PCIO approach. An example of a more precise question using the PCIO approach is as follows: low-cost international payments for people in Africa, or provide disability funds in Germany, or verify provenance of drugs in Australia or create land registries in Sweden: In each case, there must first be a clear description of the extent and magnitude of the problem, what has been tried to address that particular problem (the conventional legacy systems) and why and how a blockchain-based system would be a better alternative than the existing models.

ACQUIRE: Search for high-quality evidence
While traditional search engines are useful in searching for online content, the vast majority of information from search engines is often unfiltered low-quality blogs, opinion articles, anecdotes and other social media posts. Evidence-based practitioners must ensure that their initial searches include all portals that index high-quality, peer-reviewed research. For scientific peer-reviewed evidence, one could consider the following: Papers and case studies published at arXiv, ResearchGate or SSRN are not necessarily peer reviewed, so it is important to check the sources and platforms where these studies are published to determine if they are subjected to an independent peer review.
It is therefore important to understand the difference between filtered and unfiltered information [59]. Filtered or critically evaluated evidence include critically appraised, peer-reviewed research topics, systematic reviews and critically evaluated individual articles. Unfiltered evidence on the other hand includes non-peer-reviewed case studies, case reports essays, commentaries, blog-posts, magazine articles, opinions, surveys, analyses, company white papers, progress reports, industry or organisation reports, consensus reports and internal audits, stakeholder meetups and consortium presentations/publications.

APPRAISE: Evaluate the quality of evidence
Appraisal is 'a process of carefully and systematically examining research to judge its trustworthiness, its value, and relevance in a particular context' (Burls 2009). Carefully examining the data to establish its validity, applicability and effectiveness is an essential component of the EBP. A high-quality peer review ensures published research is subjected to scrutiny and evaluation by experts in the field, advancing scientific rigour and robustness to the scientific body of evidence.
Once sufficient evidence-based data has been collected, the next step is to apply the evidence to practice. It is important to be mindful of the limitations of the evidence and the inherent bias. As discussed earlier, not all evidence is the same; applying poor-quality, weak evidence to one's practice may result in economic, social and technical harm and waste of resources.
Why is an independent external peer review important? A peer-review process involves an independent, and usually a double-blind (i.e., the author and reviewers do not know each other's identity), review of research to check for accuracy and reliability and verify whether any claims of novelty are consistent and trustworthy. The reviewers ensure that the results and conclusion are consistent with the hypothesis put forward at the start of the paper. Any grandiose claims are also challenged. A review also ensures that a paper follows the correct scientific method and cites appropriate references in support of the claims made in the paper. A review helps advance an emergent consensus among the scientific community and supports the foundations of scientific rigour.

ASSESS (and) PUBLISH results
The final step involves the structured evaluation of evidence to analyse the outputs, outcomes and impact of the EBP. This involves the evaluation of the process itself, the outcome measures and stakeholders' feedback.
It is important that an EBB professional -Writes down the results.
-Presents the results.
-Submits them for peer review, if applicable.
-Publishes them, ideally in an open access journal.
-Evaluates and reports the inputs, outputs, outcomes, impacts and any recorded or otherwise auditable occasion of influence of the research findings.
Reporting outcomes is an integral component of the EBB; it completes the learning loop, provides an opportunity to reflect on the results, sets parameters for future research and encourages the evaluation of practices.
Traditionally, citations have been the cornerstone of measuring attention, impact and scholarly influence. More recently, alternative metrics, also called 'alt-metrics', have become a popular way to gauge impact. Alt-metrics analyse the online activity around research output in sources such as social networks, news outlets, policy documents, conferences and blogs, providing a more robust picture of the attention, influence and reach of published work.
What is a research impact? The London School of Economics defines an impact as recorded or otherwise auditable occasion of influence from research on another individual or organization, demonstrated by references to, citations of or a discussion of the research or the researcher.

Study design and methodology
There are four key constructs that emerge from the principles of Evidence-based Blockchain and these will form the foundations of our study (PCIO questionnaire) • Problem (P) We further categorise each PCIO item into 3 descriptive subsets of questions, making it a total of 12 fundamental questions. These questions will form the foundations of EBB evidence assessment framework [ Figure 11]. For the purpose of our research, we concluded that a firm was evidence based if there was an explicit evidence of demonstration of at least 2 of the 3 criteria.
The problem -A clear description of the problem to be solved is the first step to any blockchain-based solution offering. For the purpose of our research, we looked for explicit description of the following:

Types of evidence
For the purpose of our research, we categorise evidence assessment into two groups:

Filtered evidence
This includes peer-reviewed meta-analysis; systematic review; original research; case studies; and critical reviews published in academic peer-reviewed, open access journals. We also include evidence of presentations at scientific/academic conferences, summits and academic society meetings as filtered evidence. An evidence synthesis underpinning national guidelines, government policy reports, outputs of scientific committee reports, regulations, national benchmarks and frameworks based on an independent evaluation of data are also considered as filtered evidence.
We considered evidence published in academic, peer-reviewed journals as filtered evidence. We scanned this information on the following sites:

Unfiltered evidence
This includes non-peer reviewed essays and research papers on arXiv, ResearchGate and SSRN; commentary; medium or other blog-posts; magazine articles; opinions; surveys; analyses; company white papers, progress reports, industry or organisation reports, consensus reports and internal audits; stakeholder meetups; and consortium presentations, publications (other than academic/scientific conferences).

Sample
The Centre for Evidence-based Blockchain [24]  -Company website -White papers -Yellow and blue papers -Google, Bing and YouTube searches for evidence of official industry talks, pitches and conference presentations by the company/organisation.
We collected data on blockchain companies and start-ups from eight main use cases:

Banking, fintech and payments Digital identity, records and notaries Supply chain and trade finance Health care and life sciences Energy, climate and philanthropy Networking, social impact and media Government, law and public policy Cybersecurity, AI, quantum computing and IoT
The companies that were evaluated included a mix of: a. fundraising-based blockchain projects (security token offerings, initial coin offerings, initial exchange offerings). b. non-fundraising token companies and projects. c. non-token blockchain companies and projects.
We included projects using blockchain or DLTs as a corecomponent of their solution, product or service offering. We excluded the following categories of companies from our research: Companies or projects in pre-launch or betaphase, companies that did not provide evidence of white/yellow/blue papers on their website, or companies not using blockchain or DLT in their product or service offering.

Comparison
We looked for objective evidence of data provided by the firms on existing legacy systems with reference to an externally validated study, public policy report, government documents or industry survey. This evidence included critical reviews of existing solutions to reinforce or highlight a clear need for improvement of existing models. In addition, we examined reviews of past and current interventions that attempted to address the problems at hand, the outcomes of those interventions and statements regarding the clear established need for alternative or DLT-based solutions.

Intervention
We scanned for clear documentation or references to evidence for proposed solution or intervention, and asked the following three questions: 'Our blockchain solution will transform the way data is managed around the globe' (No specific measurable evidence regarding what exactly the transformation will look like and no information on how this will be evaluated based on objective evidence.) 'our blockchain will speed up the transactions and reduce costs for the customers' (No objective evidence for the improvement of speed and cost reductions in terms of data/numbers.) 'We managed to reduce the operational costs by 50%' (No objective evidence provided.) Figure 9: Evidence for the Intervention (Average score %)

Outcomes
Q10: What are the key outcomes of interest?

Conclusion
Our study concluded that almost half of the blockchain firms show no explicit evidence of the problem to be solved. Approximately one-third fail to cite a comparison and intervention analysis, and less than 2% demonstrate evidence of outcomes backed by filtered (critically appraised, peer reviewed) information. (Figure 11).  Figure 11: Summary of Results

Limitations
Our search for evidence was limited to the platforms described in the methodology section. However, it is possible that other research documents or pieces of evidence would have been available on search engines other than Google or Bing. That being said, we focused our search on the two widely used platforms. Similarly, our search for scientific/academic evidence was limited to the academic search engines and portals cited in the methodology section. We analysed the research evidence and other data in open access (CC-BY) journals and publications only. Some of the research evidence published in closed subscription journals could not be fully evaluated.
We were only able to collect and comment on the data provided by companies on their websites and in their white/yellow papers. It is possible that a project may have received a review from a third party that referenced the project in question.
We collected the data over the six-month period between January to June 2020. Some projects might release a new version of their platform or research about their project at a later date, which we could not comment on at the time of writing this paper.

Discussion
Why are most practitioners not using EBB?
The reasons are multifactorial: • Most practitioners do not know about it or how to practice it. • They may think it is 'too academic'. • They may oversimplify or overcomplicate issues.
• Their own beliefs and cognitive biases may prevent them from adopting it. • Their organisational culture may be incompatible with it.
What are the risks of not following EBB?
• It is unethical not to do so.
• Waste or poor allocation of resources.
• Practitioners may adopt poor benchmarks and frameworks. • Practitioners may engage in ineffective policymaking.

Recommendations
We propose an 'Evidence Assessment Framework' ( Figure  12) for all distributed ledger technology projects. This should be undertaken for all existing and new blockchain solutions before they are deployed in real-world settings. We make a case for a 'Chief Evidence Officer' for all organisations where blockchain is being deployed, to ensure that blockchain products, services and solutions are built on best available scientific evidence; this will ensure efficacy, efficiency, impact and effectiveness.
We recommended that the governments, organisations and enterprises looking to invest in blockchain projects ensure that uses of blockchain are based on scientific evidence. For every £100 spent on blockchain and distributed ledgers, we propose that at least £2 should be dedicated to making sure the other £98 actually works.
Policymakers, c-suite executives, investors and senior decision makers in blockchain should be equipped with the fundamental skills of EBB. We must ensure that they have the tools and strategies to critically evaluate both their problems and their proposed solutions. Any investments of time and resources into blockchain projects must be preceded by critical appraisal of the strengths and weaknesses of their project and its potential long-term impact. At the Centre for Evidence-Based Blockchain, we will continue to play our part in advancing the best standards in blockchain.