Wikipedia talk:Tiers of reliability

Good start, but...

Some thoughts:

Good point that academic journals publish both peer reviewed and non-peer reviewed stuff. People too often assumed that if it has a doi and can be formatted into a cite journal template it is good. No. Here's another consideration. As far as I know, book reviews, for example, even when published in peer-reviewed journals, are not peer-reviewed themselves.

The problem is that there are many low-quality 'peer-reviewed academic journals or books. See Predatory publishing. Even outside predatory ones, there are journals that are very dubious - remember Glaukopis?

Also, you forget about academic conferences. As well as PhD thesis (and masters).

For journalists, 'BBC, The New York Times, The Economist, Foreign Affairs' are good but what about Fox News?

Tertiary sources. General encyclopedias are one thing but what about specialized academic ones like Routledge Encyclopedia of Philosophy or The J. R. R. Tolkien Encyclopedia? --_{Piotr Konieczny aka Prokonsul Piotrus| reply here} 02:02, 14 March 2021 (UTC)

@Piotrus: thanks for the feedback. I made some edits to the page to incorporate your suggestions.

• Predatory is already listed at the bottom. While some journals are certainly more reputable than others, I don't think WP:RS guideline distinguishes between top-notch and second-tier journals: either it's reliable or it's not. Maybe for weight purposes or in the case of conflicts among journals. One hesitation I have is that I'm not sure how anyone is supposed to determine which journals are the best, and which are second tier, and which are third, etc. There are problems with impact factors, and probably all other rankings. That said, I'd love to have some guidance about how to evaluate the reliability of a journal (other than the usual method: see if other reliable sources cite it, and how often)... I'm just not sure what to write.
• No not Fox News. Not CNN, either. I added some other examples of what I think are top rated international journalism: and "top rated" is according to, e.g. Pulitzers won, or ranking by Columbia Journalism Review, etc. I think there are enough examples there of top American journalism, and I'd love to add some non-US examples, but I'm not sure which or how to rank them. I was thinking about adding something like "national paper of record", e.g. Le Monde, Bund, Time of India, I'm not sure if it's Times in the UK still or not... I don't know enough about int'l media to choose examples. But I do know Fox News is not one of them. :-)
• Should academic conference papers be put under non-peer reviewed academic publications (Tier 2), or self-published papers (Tier 4)?

I added book reviews, separate from literature reviews; specialist encyclopedias separate from general; and theses, all good points. Levivich ^harass/_hound 05:45, 14 March 2021 (UTC)

Levivich, There is no perfect system for ranking academic journals. Impact factors are one, Citation indices are another. Neither are perfect - relatively new journals with high IF may not be in the respected citation indices yet, for example. And journals in said respected citation index can range from pretty impactful to relatively minor (rarely cited) too.

Academic conferences are also a weird beast. First, they are much more relevant in some fields than in others (ex. very important in computer sciences, less so in sociology). Generally papers submitted to a conference are subject to a peer review (but not always), and I think it is less rigorous (but not always...). So sometimes they are Tier 1, sometimes Tier 2. Not Tier 4, but that reminds me about pre-prints and post-prints like arXiv, which are often cited, without citing editors often realizing that they are citing works which are often self-published (still not accepted by a journal). Might as well throw in a diagram: File:Preprint postprint published.svg.

One more food for thought comment. You've now added the academic encyclopedias to Tier 1. Well, I've written two or three articles for such outlets. And here's the rub: they are not peer reviewed (just like 'academic' book reviews). Again, there may be exceptions, but based on personal experiences I'd suggest moving such works to Tier 2. If in a specific case we can show that an encyclopedia article or book review were subject to a peer review, such a work can be upgraded, but I'd say that generally they are not peer reviewed (except, again, by a single editor for the journal/volume). _{Piotr Konieczny aka Prokonsul Piotrus| reply here} 06:41, 14 March 2021 (UTC)

@Piotrus: Originally I had a 9-tier system, but after discussing it with others, I was persuaded that fewer tiers are better: trying to slice it up into too many individual ranks brings a lot of problems, e.g. trying to rank a peer reviewed conference paper vs. a peer-reviewed article in an unknown journal vs. a single book review in a top journal vs. a single-author book by Routledge vs. a specialist encyclopedia by Oxford ... on and on we could go, but the truth is, we really can't make statements like, "all specialist encyclopedias are better than conference papers but not as good as peer-reviewed articles in indexed journals". I don't think we can divide it that finely.

So the rationale is that Tier 1 is best, and it's scholarship. Tier 2 is not scholarship, but it's top-quality sources that are not scholarship. Tier 3 is OK not-scholarship. Tier 4 is not OK. Whether it's a single-author academic book, or a multi-author, single-editor specialist encyclopedia, or a journal article that's peer reviewed by a full anonymous panel, or a literature review... it's all scholarship. Whereas a book review or a news article, even if it appears in an academic publication, is not scholarship, so it's Tier 2. A news article in The New York Times, The Economist, or Nature, are all of roughly the same reliability: somewhere in between scholarship and Fox News (which is Tier 3).

So by that logic, I'd keep specialist encyclopedias in Tier 1 (although I re-arranged the order of the subsections to put "peer reviewed publications" first, before "academic books"; even though I don't intend the subsections to be rankings within the tiers, maybe it's clearer this way), and I added "peer reviewed conference papers" to Tier 1, and "non-peer-reviewed conference papers" to Tier 2.

Also, going back to your earlier point about Fox News, I copyedited the section headers so it's "top rated" journalism (tier 2) vs. "popular press" (tier 3). I'm still not sure if those are the best ways to phrase it. Levivich ^harass/_hound 15:43, 14 March 2021 (UTC)

Levivich, No ranking is perfect, but I think a non-scholarly source can be sometimes strictly better than scholarly. A well-researched book from a popular press (or article in a respectable newspaper) may be more reliable than an academic hackpiece published in something like Glaukopis or another, niche, impact-less entity that calls itself an academic, peer-reviewed journal. See also Sokal squared... --_{Piotr Konieczny aka Prokonsul Piotrus| reply here} 02:26, 15 March 2021 (UTC)

I agree, sometimes being the key word, though. Such sources are "the exceptions that prove the rule". Levivich ^harass/_hound 16:45, 16 March 2021 (UTC)

"Good" is "acceptable", tiers need rules, Rule of Three. You have dog? Is cat now! InedibleHulk (talk) 03:32, 17 May 2021 (UTC)

Sometimes, bad is acceptable, too; even preferable. Levivich ^harass/_hound 05:13, 17 May 2021 (UTC)

Aye, I remumble. InedibleHulk (talk) 05:16, 17 May 2021 (UTC)

News sources

Hi Levivich, thank you for this project: this is going to be a very helpful guide. I have a few comments regarding your classification of news sources:

• One element that I think is missing from this draft is the high credit we give to the reporting of perennial sources considered "generally reliable" by the community. In articles about current event, a source article by the New York Times, the Washington Post or the Financial Times is really as good as it gets.
• I think the distinction you draw between "Top-rated international journalism" and "Popular press" is difficult to apply, i.e. rated by whom? The most relevant rating to Wikipedia is the WP:RSP rating: and if you are inclined to agree with that, then I think we can avoid introducing new and unclear concepts and re-use the ones that are already in the policies ("generally reliable"/"marginally reliable"/"unreliable"). In my understanding, editors should use their best judgement regarding unclassified sources, and try to asses a particular source in light of this classification.
• I think your essay should underline more strongly the distinction between factual reporting and editorial content (WP:RSEDITORIAL): editorial content is a lower tier than factual reporting.

Finally, I also suggest that you add "publications by companies about themselves" in Tier 4: I often see this kind of source in business articles. JBchrch (talk) 10:55, 16 March 2021 (UTC)

Hi JBchrch, thanks for your comments! I added op/ed and corporate websites.

I disagree with you a bit about WP:RSP, though, and actually one of the reasons I started this is to clarify what I think of as the "Three Myths of RSP":

1. That RSP is a source rating system: it's not, it's something different: it's a summary of discussions about sources. It's not an exhaustive list of all sources, and it's not meant to rate them. "Green/yellow/red" doesn't mean "best/adequate/worst", it means "consensus it's acceptable/no consensus/consensus it's not acceptable". The colors don't refer to the quality of the source; the colors refer to the outcome of discussions, which is a subtle but important difference. Yellow means "undecided", not "average quality". A source where the discussions end in "no consensus" (yellow) is not necessarily worse than a source where the discussions end in consensus that it's acceptable (green), nor necessarily better than a source where discussions end in consensus that it's "not acceptable" (red). A yellow source might be just as good as a green source, or just as bad as a red source: or more likely, a yellow source can be just as good as a green source for some content, and just as bad as a yellow source for other content. Further info on this is at WP:RSPUSE. One of the reasons for this essay is to provide a ranking system (Tier 1-4) to clarify that green/yellow/red is not a ranking system. In other words, one of the points of this essay is to get editors to start talking about "Tier 1", "Tier 2", and "Tier 3" sources, rather than "green," "yellow", and "red" sources.
2. That green-at-RSP sources are as good as it gets: they're not. In fact, the "as good as it gets" sources are not listed at RSP at all, because they're so good, nobody ever discusses or disputes them. That's why top journals like Nature, Science, and The Lancet aren't listed at RSP. See WP:RSPMISSING for more on this. I agree with you that BBC, NYT, and FT (all green), are as good as it gets for journalism, but part of the point of this essay is to highlight that there are sources that are much better, and that will never be discussed at RSN or listed at RSP. "Green at RSP" just means "acceptable", i.e., Tier 3 on this rating system; green at RSP doesn't mean "best".
3. That all green-at-RSP sources are equivalent to each other: they're not. BuzzFeed News, People, Rolling Stone, Vox, and many others are listed as green, but they are not in the same ballpark as BBC, NYT, or FT. Part of the reason for this essay is specifically to provide ratings that allow us to subdivide green-at-RSP sources and recognize that some are better than others. This is really the whole point of separating out "top" journalism (Tier 2) from "regular" journalism (Tier 3).

That said, you're right that "rated by whom" is an issue in determining which journalism is "top rated" (or "best" or however phrased), and which journalism is just "regular" or ordinary. I'm not sure that we can formulate a definitive test, or compile a definitive list. Ultimately, I think we can agree on some categories (Pulitzer-prize-winning newspapers with overseas bureaus and top global circulation would be in Tier 2; local television news stations would be Tier 3), but when it comes to specific sources, I think that's a content dispute that will have to be decided on a case-by-case basis. I intend the examples listed in this essay to just be examples; I don't intend for this to be any kind of compilation of which news outlet is Tier 2 and which is Tier 3, etc. In other words, this essay sets out the rating system, it doesn't rate specific sources (other than to use obvious well-known examples as examples). Levivich ^harass/_hound 16:44, 16 March 2021 (UTC)

Thank you very much for this though-provoking answer, Levivich. I have to admit that I had not considered these issues, which you lay out very clearly (may consider a separate essay? 😁). I also admit that I spent more time than I expected thinking about your arguments over the last days and, in the last analysis, they seem correct.

The only thing that I am not entirely convinced of, though, — in a spirit of friendly academic disagreement — is your Tier 3. I still think that popular press is not the best language and could lead to misunderstandings. I think that you could make a better use of the immense amount of work put into WP:RSP to construct this category. So I would propose—instead of popular press— "Other generally reliable news sources". That would allow you to "import" the legacy of WP:RSP while still driving home your point that some reliable sources are better than others. JBchrch (talk) 23:09, 18 March 2021 (UTC)

Good point about "popular press"; I made the change you suggested. Also, I changed "top-rated" to "top-quality", because of the "rated by whom?" question, although I'm still not sure if "top quality" is the best way to describe it. Thanks! Levivich ^harass/_hound 23:47, 18 March 2021 (UTC)

Alternatives from the top of my head: authoritative, prominent, reputable, longstanding, established, (highly-)respected... and the thesaurus entries thereof.--JBchrch (talk) 00:31, 19 March 2021 (UTC)

I changed it to "highly-reputable" which I think is at least an improvement over "top quality" ("reputable" alone might imply that the Tier 3 sources are not reputable). Other ideas I had are "leading", "premier", "prestigious", but those sound like subjective accolades rather than an objective assessments. Levivich ^harass/_hound 02:02, 19 March 2021 (UTC)

Variety among content areas

I think that this fails to account for tiers of reliability for huge swaths of content areas. For instance top-tier high quality film critique often will not appear in something that can be labeled "Non-peer-reviewed academic publications". Relatedly take my own field of books - something like Library Journal is a high quality reliable source but it doesn't plug into any of the listed categories. Alternatively look at sports. We have things like statistical databases, a concept not included at all, and also something like ESPN which maybe could be considered "Highly-reputable international journalism" but other equally reliable sources would fail on account of not being international. If the kinds of sources we want our best editors to use in certain areas are labeled as "acceptable" I think that undermines this (nobel) endeavor more generally. Best, Barkeep49 (talk) 14:53, 13 May 2021 (UTC)

Damnit Barkeep49, those are some really good points I hadn't thought of before. Thank you for raising them! You're right, this scheme might work for science, history, or politics, but not for arts, sports or pop culture topics. Hmm, you've given me a lot to think about! Levivich ^harass/_hound 15:53, 13 May 2021 (UTC)

I vote that we don't really even need to include sports and pop culture. They're way more trouble than they're worth. —valereee (talk) 21:41, 13 May 2021 (UTC)

I mean it's reasonable to say the scope is science, history, and politics for which I think this largely does hold. Best, Barkeep49 (talk) 21:47, 13 May 2021 (UTC)

@Barkeep49: After thinking about this more, I think I disagree that the system doesn't account for large swaths of content areas. I just added some examples for the topic "Aretha Franklin" in each Tier. I'm confident I could do the same for sports, films, food, even things like pro wrestling, trains... it's all been covered by academia, hasn't it? Do the examples, like the Aretha Franklin ones, persuade you at all? Granted, not every subject will be covered in every tier: not every singer is Aretha Franklin. But that strikes me as OK: for some topics, particularly newer topics and less-notable-but-still-notable topics, there might only be Tier 3 or Tier 2 coverage; editors should use the best available. But my point would be: when we write about Aretha Franklin, we shouldn't use E! Online or even the New York Times, when there are higher-tier sources available.

Library Journal seems like a typical trade publication, which I have in Tier 3, "acceptable". Do you think Library Journal is as reliable as, say, the BBC? If both publications wrote an article about the same thing--say, libraries in Gaza--which would be more reliable? Levivich ^harass/_hound 05:29, 16 May 2021 (UTC)

So I'm going to say that your examples on Franklin illustrate the limitations of this for certain areas. I'll stay with-in music. Lordi, who wrote the tier 1 source, is not among the most reliable/expert source available on Franklin. Now it's possible that Solomon is a top tier source, I don't know enough about Franklin to say who is but I'm guessing whatever those sources are would fall in tier 2. I think we can see the dynamic I'm concerned about play out at The 1975 (2019 song) which was promoted to FA on May 14. For that FA there would be no sources from tier 1 and that's not surprising. And it's not just the fact that the song is 2 years old. Aftermath (Rolling Stones album), promoted to FA in early March, is 45 years old and has two tier 1 sources that I see. However, those two sources are cited 2 times out of 178 total citations (not counting some repeated citations). Instead at both of these articles the sources that are best, that is that are going to help editors produce the highest quality articles in terms of both scope and reliability, are in Tier 2. Which is what I would expect and what I was suggesting was an issue.

Also because it was a throwaway comment perhaps you didn't think about it, but I think statistical databases need their own call-out in Tier 4. Best, Barkeep49 (talk) 15:39, 17 May 2021 (UTC)

I see what you mean. I wasn't aiming for best in terms of both scope and reliability, just reliability. I wasn't thinking so much about best for an article, as best for a specific fact or statement; particularly where sources disagree, which should be weighed more heavily. So I changed the section headers from "acceptable/good/best" to "reliable/more reliable/most reliable", which hopefully will clarify the meaning.

I think The 1975 and Aftermath articles actually support this scheme. For example, in The 1975 article, most Tier 3 sources are cited once, whereas the Tier 2 sources, though fewer in number, are (most of them) cited multiple times, the most-cited being The Guardian. In the Aftermath article, Christgau (Harvard), Smith (Oxford), and Perone (Praeger), all Tier 1, are not only cited but discussed in the prose of the article, alongside Tier 2 sources in wiki-notable works or by wiki-notable authors. Though there are Tier 3 sources out there, they're not given the same weight as the Tier 1 and 2. That's an example of what I'm trying to convey: when writing an article about an album, include reviews by Tier 1 and Tier 2 sources, but not so much Tier 3. Unless Tier 3 is all you've got. Does that make sense?

As for stats websites, aren't some stats websites the absolute authority on those particular stats? I'm thinking e.g. NFL.com and baseball-reference.com. Other stats websites are user generated and not reliable. Personally, I think all statistics are primary sources and should be listed under Tier 4/primary. What do you think? Levivich ^harass/_hound 05:04, 18 May 2021 (UTC)

Yes I agree that stats are a kind of primary source but need their own Tier 4 mention just because of how prevalent they are. As for the changes, i do think the "good, better, best" model of reliability is helpful. I need to think about the larger points; if this works I see it being used in complementary/similar ways to WP:RSP and so if it is wrong in how its written it could have a widespread effect in how things actually get resolved onwiki. Best, Barkeep49 (talk) 19:20, 18 May 2021 (UTC)

I was thinking it eventually could be an explanatory supplement to WP:RS (and not a listing of specific sources like RSP, although RSP may decide to label entries by Tier, maybe), but after a lot of incubation and testing against "real world" (well, on-wiki) examples. I hope this develops into something enough editors would agree with that it could be moved from user space to projectspace essay. But I don't think we're there yet. Levivich ^harass/_hound 20:05, 18 May 2021 (UTC)

I think a column at WP:RSP for tier would be really helpful. —valereee (talk) 21:35, 18 May 2021 (UTC)

Levivich: so circling back after further thought, I think your analysis shows what I'm concerned about. In some subject areas, Tier 1 & Tier 2 are really just one giant tier in terms of how we should think about and use them. Your point that we quote the Tier 1 sources is a good one, but we also quote high quality Tier 2 sources in similar ways. If we say they're two tiers we're creating a distinction that doesn't (shouldn't?) exist in some areas. Best, Barkeep49 (talk) 18:31, 24 May 2021 (UTC)

Personally, I think the distinction between Tiers 1 and 2 is crucial but we could turn the 4 tiers into 3 tiers by collapsing Tiers 1 and 2 into a single Tier 1. I originally started this with something stupid like 9 tiers and moving to fewer tiers seems to have moved this further towards consensus. Perhaps the number of the counting shall be 3. Levivich ^harass/_hound 19:07, 24 May 2021 (UTC)

Sports journalism is a tricky area, as it can often present factual info within a promotional context. This is commonly from a hometown perspective, but can also just be a general promotion of the sport viewpoint, since that's what fans are interested in. For instance, sports writers will often find flamboyant ways to describe events and sportsfigures, to draw more attention and increase excitement. So it's more nuanced than establishing reliability and neutrality for a publication: even within one article, the suitability of the text to serve as a citation can vary from one sentence to another and so must be evaluated on a case-by-case basis. isaacl (talk) 19:41, 18 May 2021 (UTC)

Non-peer-reviewed conference papers

Aren't these basically equivalent to preprints? ProcrastinatingReader (talk) 16:29, 26 June 2021 (UTC)

On that topic I'll just note this essay which really made me think about PR and its tier of reliability. Best, Barkeep49 (talk) 18:00, 26 June 2021 (UTC)

I don't know the answer, but I've always considered a paper selected for presentation at a conference to be a step up from a paper uploaded to a website. Maybe that's naive of me? Levivich 01:51, 28 June 2021 (UTC)

See Inter-universal Teichmüller theory. It was described at several conferences but its proof is still not generally accepted. Basically equivalent to a preprint at this point. ProcrastinatingReader (talk) 12:46, 28 June 2021 (UTC)

Moving to projectspace

Levivich Any interest in moving this to projectspace as an explanatory supplement to WP:BESTSOURCES? I think it could be quite a useful resource, especially for newer editors. ProcrastinatingReader (talk) 03:20, 9 January 2022 (UTC)

@ProcrastinatingReader: Sure!  Done Probably needs some finishing touches before fully going live. Levivich 06:06, 9 January 2022 (UTC)

Grey literature

Anyone have an opinion about which tier grey literature belongs in? I think it's Tier 2 or 3, but I'm not sure which, and I'm wondering if it depends on the specific publication or work; i.e., some grey literature is in Tier 2 and others are in Tier 3. Levivich (talk) 17:09, 28 September 2022 (UTC)

The Problem with Tier 1

According to tier 1, sources like The Armenian Massacres in Ottoman Turkey should be considered reliable just because it was published by University of Utah Press, despite the book being overwhelmingly criticized for promoting genocide denial and academic dishonesty. It was likely only published because of bribery. So it is wrong to assume that everything published by an academic publisher is automatically reliable. Dallavid (talk) 23:29, 26 January 2023 (UTC)