How to estimate the scientific consensus or not about Hydroxychloroquine?

The difficulty here is that these sorts of questions are hard to understand for non-scientists and the results difficult to interpret for non-scientists.

This will be long, and I will likely miss things while writing this, so anyone please feel free to weigh in and/or edit if you feel the need.

First some background about treatments and how we (scientists and medics) know they work or don't work:

In general for a drug to be assessed for its efficacy it needs to undergo something known as a clinical trial. These are studies that involve proper experimental design and have at least one testable hypothesis with results that can be interpreted statistically. Proper experimental design for these sorts of things is difficult and aims to ensure that any there are no biases in the data that may invalidate the results!

First off the trial should be controlled in some manner. Control in this context means a comparison group who receive something other than the treatment in question. This might be that the patients receive a different drug (of known efficacy or not) or a placebo. You can even compare different doses of a known efficacious drug (say you know it works a dose X, but you want to see if it works at 10-times that dose). You can also compare things like surgery with no surgery - sometimes by doing mock surgery as a placebo.

The treatment also has to be tested in a randomized manner (assigned to some participants on a random basis), you also need to make sure that neither the participants nor the people giving the drugs know which people are receiving the placebo and which the real drug(s) until after the results are in and have been fully analysed. This is a process known as blinding. There are quite a few different sorts of blinding. The one I described above is called double-blinding.

If you put all these things together you have what is known as a controlled randomized double-blinded study. These are often considered the most effective at determining the efficacy of treatments. Not all trials will meet these conditions - for example it would be very hard to double-blind a surgical trial as it would be hard to blind a surgeon as to whether they have done real surgery or not.

Next, clinical trials should also ideally involve lots of random participants from random populations so as to not introduce bias(es) (for example: don't sample just a few patients from your local hospital when you live in a high-wealth area and assume that your results are able to be generalized to the whole population). This means you should sample young, old, rich, poor, body types etc). There are a few exceptions to this rule - perhaps you want to examine the effect of a particular treatment on obesity (so you need obese people to start with) or perhaps a particular age-group, but that can be part of your experimental design so is less of a problem, but you still need to sample randomly within that group

However,in addition to all of the above, there are a few more conditions that should be met:

The next rule is that you need to sample large numbers of people - often in the several thousands. Smaller sample sizes may have larger (relative) biases (e.g. maybe, just by accident, 50% of the 60 patients in your trial were from a single family with a genetic mutation who all caught the bacterial infection you are studying at a family gathering - which could bias your results towards how that genetic mutation affects your treatment). Another issue is that small samples may not have enough statistical power to show minor effects. Proper experimental design will have determined the statistical power needed and how large the minimum sample size should be before the study is conducted. Large sample sizes also allow you to look at different groups within the study during analysis - perhaps your drug isn't effective in the population as a whole, but it is in those over age 70, or those with kidney disease or a missing left leg (very unlikely - but you wouldn't see this last unless you had a large enough sample size!).

A clinical trial should also be registered with a clinical trial register (e.g. The USA one) with open methodolgy and aims etc. outlined. As I write this, there are 5874 clinical trials on COVID-19 registered on the Clinicaltrials.gov website from all sorts of places around the world and looking at things from known treatments (e.g. remdesivir) to wearable smart devices.

Another requirement is that the results are reported in the literature (we are getting closer to your question here...). If they are a well designed study that meets all the above criteria and show a result of interest to the medical community, then it should have no problem with peer-review (where experts working in the field examine all aspects of the trial before the results get published) and will likely get published in a well established and respected scientific or medical journal.

Thanks to @BenBolker for the following idea and paper:

To show you just how hard all of this is, there's a review paper in the British Medical Journal (BMJ, one of the top-ranked medical journals) with the title Drug treatments for covid-19: living systematic review and network meta-analysis. Link to article: here. What this paper aims to do is to look at many of the drug treatments for SARS-CoV-2 infection and aggregate data from all the published trials from the start of the pandemic through to February 2021. The abstract provides a summary of the data starting out with inclusion:

196 trials enrolling 76 767 patients were included; 111 (56.6%) trials and 35 098 (45.72%) patients are new from the previous iteration; 113 (57.7%) trials evaluating treatments with at least 100 patients or 20 events met the threshold for inclusion in the analyses.

This means out of the ~200 studies, just over half met the criteria of significance (for this analysis) - they excluded vaccination, nutrition, antibody-based therapies, supportive therapy and traditional therapies that included more than 1 active molecule.

As further evidence of how hard it is: Figure 1 from the BMJ paper shows the triage process for the articles they looked at. It shows that they started out scanning 31,752 articles on COVID-19 and trimmed that down to 632 randomized trials, which were then examined closer and found that 343 of these weren't actually randomized or were excluded for other methodological problems, leaving 289 trials. These were then pruned to 206 for a bunch of reasons (duplicates, pre-print (so not verified), corrections, retractions etc.) leaving 196. A further 83 were excluded for not meeting criteria on size, lack of data, no outcomes reported or same-drug comparison (not controlled properly). Leaving 113 in total.

So if you take the 632 trials and are left with 113... that's only about 17%: less than 1 in 5 get it right!

Now, this is where it gets murky! What is a "good" journal? Well, that's a tale for another day, but there are many "scientific" journals that are neither scientific nor good. Many of them are listed on Beall's list, which is a list of publishers and journals known or suspected of being "pay for display" - essentially pay them money and they will publish your paper without doing the proper peer-review process, which science uses to determine if a study meets scientific standards and rejects those that don't.

Unfortunately, proper peer-review takes time - the journal gets a paper submitted to them. An editor reads and then rejects or sends out for peer-review. The reviewers then read the paper, write a report suggesting problems, fixes, questions etc - these then get communicated to the editor, who either rejects the paper or sends it back to the authors for amendments (may be needs more experiments, more statistics etc). The authors fix and then re-submit to the editor, who again reads and sees if the review has been addressed. The paper may then be sent out to the reviewers again, rejected (if concerns not addressed) or published. Generally this takes between 3 and 12 months.

With the current pandemic there has been a problem with this system of peer-review. Because of the urgency of the pandemic and the strain on resources, people have been submitting papers to journals for peer-review and posting them on "pre-print" servers (not peer-reviewed), where they are read and used, based on the information presented. Many journals also suspended peer-review because of the need to get information, especially about therapeutics, out to their audiences as quickly as possible. This resulted in some of the checks-and-balances that normally go on in the scientific field to be suspended and papers published that would not normally have passed peer-review.

We are now finally at the point where I can discuss Hydroxychloroquine (HCQ):

Hydroxychloroquine has been around for a long time and touted as a potential therapeutic for a number of diseases. It works well for rheumatoid arthritis, systemic lupus erythrematosis and as an anti-malaria drug. However, it has never been shown to be an antiviral in any form outside of the laboratory.

Initially reports of HCQ as a treatment for COVID came out in May 2020 (e.g. this British Medical Journal paper (an excellent journal). These were small trials which were some times randomized and generally non-blinded (AKA open-label), and showed variable effects. 6 out of 8 papers mentioned in the FDA revocation of emergency use for HCQ letter (see Table 1 on page 5. PDF!) show no effect, while 2 do, with the largest in terms of numbers of people (the BMJ one linked above) showing no effect. The ones that do show an effect are very small sample size

Based on the evidence presented in some of these studies and (I think, can't find publications) some laboratory data indicating that HCQ might be effective against SARS-CoV-2, the FDA issued an emergency use authorization (I also don't know if there was some political interference in this). At the same time the University of Oxford (UK) started recruiting for a large clinical trial of HCQ. This trial is known as the RECOVERY trial and is not just for HCQ, but a bunch of other drugs too. Currently they have recruited nearly 40,000 patients for their trials across multiple countries.

The results from the RECOVERY trial into HCQ were released in June 2020 (this is very very quick for a clinical trial, but then massive numbers of patients) with 11,000 patients tested in a randomized, controlled, double-blinded manner. The results were then published in New England Journal of Medicine (another excellent journal) and showed no effect of HCQ on any of the metrics recorded. Since then other trials, mostly smaller ones (~1000 patients) have been run by a bunch of people, none of which (to my knowledge) have found any effect of HCQ on SARS-CoV-2 infection.

Taken together, but heavily weighted towards the results of the RECOVERY trial, the data indicate that HCQ is not effective at all against SARS-CoV-2.