Greg Craven in What’s the worst that could happen? explains how he decides which sources he depends on. I am offering my own explanation after sending someone a list of some of the sources I depend on. He said, yes, he relied on the same sources, and then forwarded a report from a group of a very different kind. So, it’s an interesting question. My criteria differ from Craven’s, and yours differ from both of ours. How do you decide which sources you trust, and why?
First, some definitions:
• Sources are the original reports of data, analysis, or meta-analysis (combining data from multiple sources, as in the uber-reports from groups like IPCC)
• Peer-review is the process of review by elites in the same field (hence, “peers” of the researcher or theorist submitting the report.) In the science or policy community, peer-review is the first step toward publication or use of the data. These communities are usually careful to avoid relying on or citing data that has not been subjected to peer review.
• Channels pass on information. I aspire to be a channel, try to avoid being a source, i.e., adding mistakes. The NY Times is another example of a channel.
Besides peer-reviewed and non-peer-reviewed work, there is a category called gray literature, described in RealClimate’s IPCC errors: facts and spin. This category includes major organizations such as International Energy Agency, World Bank, United Nations Environmental Programme, government statistics offices, and more. Groups like World Wildlife Fund are also included, but their information needs to be even more carefully checked. Generally, it is more useful to cite the original source WWF depends on and avoid citing WWF.
So my criteria for passing along information on scientific and policy topics: Is it
• peer-reviewed, e.g., published in peer-reviewed journals
• respected by the scientific or policy expert community, as shown by its inclusion in major reports such as National Academy of Sciences of Intergovernmental Panel on Climate Change
• accepted over time, surviving or correcting any criticism that may arise (e.g., no disagreement appears in Science or Nature magazines).
I’m pretty conservative, and rarely use information/analysis in my presentations before it reaches major report status.
Everything I’ve described depends on writing rather than personal statements. People can be respected or not in their field of expertise, but few speak for the science community. Election to head American Association for the Advancement of Science is one of the ways scientists communicate that the person is trusted to report scientific understanding, including nuance.
I have heard a number of people and organizations willing to accept the reports of IPCC on science and impacts, but ignore IPCC’s policy analysis. If this describes you, perhaps you can explain how you choose your sources. Whatever your answers, please share!
Addendum The reason I use this method is because when I started looking into energy and environmental issues, I discovered I had misunderstandings both in facts and interpretation. Scientists and policy makers make mistake—those errors on Himalayan glaciers made it into my presentation. But the number of errors has declined markedly in recent years.
I tend to use gray literature a lot if I want to check something quickly, since those groups tend to have easy to find and use web sites (for instance, I use the CDC and WHO web sites a lot), and look for peer reviewed literature in addition if I want to check something out in more detail. MedLine is really handy for looking for medical literature, and you can get to it through MedScape (where I have a free login); I think there may also be a MedLine search at the CDC.
Of course, sometimes it takes knowing a little even to know what the good peer reviewed sources are. I have an idea of that for medicine (NEJM, JAMA, Lancet) and for psychology; for some fields I might want to check with someone I know in the field as to what the reputable sources actually are.
Things that are really important to remember: 1) Scientific understanding is built up through a series of studies; any single study (even one in a reputable, peer-reviewed publication) may be weak. For instance, Lancet published a study that raised alarms about vaccinations and autism; that study has not been replicated by repeated studies done since, and in fact has since been found flawed and repudiated by Lancet. But the scare stories still circulate. 2) The fact that a popular paper or magazine reports that a study was done means nothing until you find out who did the study, what the methodology was, and whether it was published anywhere peer reviewed. Often some marketing group for a company selling a product does a study for publicity, not necessarily with any sampling method you can rely on, and then gets the press release widely picked up, if the results sound sexy enough. Even good studies may not be correctly represented in the popular press, so it’s good to go to the source.
Issues of figuring out which sources are reliable also come up for non-scientific information. For example, when reporting on legislation, what sources can be counted on to analyze it properly? Sometimes I’ve followed links and checked bills directly, but, truth be told, I’m not wonky enough to do that terribly often. So I look for how bills are summarized in sources like the Congressional Quarterly, and, for things like budget and deficit impact, for what the Congressional Budget Office has to say. I find FCNL a useful source on what proposed legislation actually says on issues I care about; on the other hand, much though I love FCNL and share their goals, I’d look for other sources if it were a matter of scientific fact (no criticism of FCNL as such, just that I think if the dispute is a scientific one I should, as far as I can, look more directly for people who understand the science).
In some cases (e.g., figuring out which information was real and which was false when the protests hit in Iran and Twitter was alive with reports of highly variable quality), it can be very hard to know which sources are solid. But at least when it comes to science, I know that there are processes that help sort out which sources are actually good.
[…] My friend Karen Street and I have been discussing, in the wake of an earlier post of hers, the Friends’ Testimony on Integrity. Now she has a new post, discussing another side to the testimony: Which Sources Do We Trust, and Why? […]