User:Duae Quartunciae/QualityReview

There are heaps of problems with this. It contradicts itself in some places, and the dynamics of reviewer disputes are not adequately considered. I'm letting it be as a base for ideas I'm turning over in my mind, and for the moment I am just continuing to engage in wikipedia and reflect on what I see happening, according to my own limited judgment. —Duae Quartunciae (talk · cont) 07:34, 4 August 2007 (UTC)

Quality control for Wikipedia

When I first signed up here about six weeks ago, I described myself as a "long time and very satisfied reader of Wikipedia". Now that I have had an active part in maintaining this resource, I have become much more skeptical. After a bit of thought, I have reached some tentative conclusions.

Wikipedia is not a trustworthy source

It ought to be obvious, and I've always given lip service to this principle. It can be a very useful jumping off point for chasing down information on a topic. It is has a lot of content on various unconventional and fringe ideas that might not be considered worth of inclusion in other sources.

However, it should not be trusted. The specific claims it makes, even when backed up by references and apparently credible sources, are not reliable and serious errors of fact are common.

No quality control is a strength and a weakness

The major feature of wikipedia that distinguishes it from other encyclopedic references is that there is no quality control. Any user can add any content at any time. This is both the great advantage of wikipedia, and its great disadvantage.

There are, to be sure, an army of editors beavering away in the background to find and fix vandalism, or material that fails to live up to the guidelines. The guidelines and policies, of No Original Research, Verifiabilty, and Neutral Point of View are well chosen, but there's no mechanism to prevent people from ignoring them. All we have is the tireless work of many volunteers to fix up cases where the guidelines have not been followed.

When you read an article on Wikipedia, in effect you are simply hoping that improper additions have been found and fixed. That hope is often justified, but there's no real way to tell apart from careful checking on your own behalf.

The advantage of this lack of control is that new content appears quickly and easily, and is drawn from a massive body of users. An enormous amount of useful content appears here from people who just drop in and make a spontaneous contribution. These additions appear immediately for other readers.

The disadvantage, of course, is that new content is sometimes not in line with good practice or wikipedia guidelines.

Wikipedia is especially vulnerable to dedicated individuals determined to push various strange and counter factual perspectives, and see this as an avenue to get their views out to a large audience under a banner of presumed respectability that they could never obtain on their own. There is a process of attrition for editors. It is very common for bad cases of such Point of View pushing to be identified by other editors, and the changes monitored and fixed. There are many little edit wars going on all the time, and often the fringe contributors give up and leave. There is a never-ending supply of new ones to take their place.

Conversely, there is a problem of attrition for expert editors who are best able to fix problems. The difficulty is that a presumption of neutrality means that a genuine expert can never formally rely upon their expertise, and may be frequently called up themselves for violating neutrality in fixing the nonsense from fringe scholarship. First rate contributions can get removed or edited by the comparatively clueless. Expert editors often just give up, and leave wikipedia to those who can be bothered keeping up the maintenance indefinitely.

What should the role of Wikipedia be?

A basic philosophical position of Wikipedia is that formal quality control is not really needed, and we can rely on the workings of a community contributing in good faith to gradually sort out the problems, leading to a final result that is of a high quality, comparable to more established encyclopedias.

Of course, this is a never-ending process; there is no final result. There is always a struggle between viewpoints going on, and the lack of quality control means that even rank nonsense always continues to be a part of that struggle.

So what is it going to be? Is there a benefit to introducing more control, to give a more trustworthy end result? The cost is a loss in the speed and efficiency with which new content can be obtained and included. Or should wikipedia continue to maintain the open structure that allows everyone to work on an equal footing for content? The cost is a lack of assurance of the quality of new content.

A proposal: Approved revisions.

I'm sure someone has thought of this already. The Wikipedia already has in place the capacity to look over the history of articles, and find older versions. Nothing is lost, except in some special cases not available to most editors.

It should be possible to add to a page the capacity to examine designated versions. Here is a way it might work.

The capacity to determine what is or or not a designated version is limited to a special class of contributors, with a formally recognized status as reviewers. Whether a user has reviewer status or not is completely independent of whether they have administrator status or not.

In any page, there is a capacity to look at various designated revisions. I can imagine it working as follows, and I have no idea how easy or hard it would be to implement. The mechanisms are intended to be conservative, so that any dispute between reviewers is biased against approval.

The role of reviewers is to check the official policies of No Original Research, Verifiability, and Neutral Point of View.

Every revision will have associated with it a number of slots that can be filled only by reviewers, and then only if the slot is empty. A user can also remove themselves from a slot. It requires a high level of super-user administrative power to remove names of other users from these slots, to deal with a rogue reviewer or a reviewer who has left the project. There is no history maintained, and no scope for additional comments; reviewers are expected to discuss any addition of their name to a slot in the article discussion page, except for a check on an undisputed page. There is a timestamp on each slot, however. There are also published guidelines for reviewers as to when they should put their name in a slot. The possible slots are:

• Check. This is an indication that a reviewer thinks the revision has a reasonable level of quality.
• Uncheck. This is an indication that a reviewer thinks a revision may not have a reasonable level of quality.
• Approve. The reviewer is declaring that the revision has passed a formal review process, as described in guidelines.
• Unapprove. The reviewer is overriding a declaration of "approve".

A revision can have a status of "approved", "checked", "undisputed" or "disputed". The algorithm for classifying a revision is as follows:

• A revision is approved if the "approve" slot is filled and the "unapprove" slot is empty.
• Otherwise, if the "uncheck" slot is filled, then a revision is disputed.
• Otherwise, if the "check" slot is filled with a more recent timestamp than every "uncheck" slot in the prior history, then the revision is checked.
• Otherwise, if the immediately previous revision is "disputed", then this revision is also disputed.
• In all other cases, the revision is simply undisputed.

The reader of wikipedia is able to examine one of several designated revisions, which are identified from the various review slots.

• Latest. This is simply the most recent revision, regardless of any review. As at present, all edits apply to the the latest revision, and all edits establish a new latest revision, immediately. (View this to get the same as we have now.)
• Undisputed. This is the most recent "undisputed" revision. (Recommended default for general readers. This still gives immediate updates, unless a reviewer has explicitly flagged an unresolved problem.)
• Checked. This is the most recent "checked" revision. (A bit safer; but a bit out of date. Forces you to wait for reviewers.)
• Approved. This is the most recent "approved" revision. This indicates a level of formal assurance that the article was in excellent shape at this point. The article is very likely to have been improved since then, but if you are especially distrustful you may want this revision for safety's sake. Note that "approve" is a comment on content and accuracy, not on presentation.

The guidelines for reviewers are as follows:

• A reviewer can mark a problem edit as "uncheck". This effectively means that the "checked" revision available to readers becomes some earlier revision. This should be used with caution, because the timestamp overrides all check marks in place on subsequent revisions. Any reviewer can thus backdate the current checked revision to any point in its prior history. Any uncheck indicator must come with reasons given in the article discussion page; the reasons should relate to one of the three official guidelines.

• A reviewer can mark a revision with "check", which effectively overrides an "uncheck" indication on all preceding revisions. In general, a reviewer only needs to mention this in the discussion pages if checked revision was previously a disputed revision. The reviewer should explain how the problems have now been solved.

• The "approved" indication is only to be added when there has been a formal review of the article, and a vote in which the registered reviewers voting for approval outnumber those against by more than two to one. At least three reviewers must have voted. At their discretion, reviewers may request expert comment from a WikiProject or other such group, but only the registered reviewers actually get to vote. In a highly technical article, a responsible reviewer will usually choose to be guided by the experts, and so they have the onus of sorting out who is making good sense in arguments on technical details. Usually, a reviewer will be able to make a good guess when poorly founded technical input is provided by fringe POV. The vote is informal, and conducted in the discussion page. Anyone wanting to add the "approved" indication must place a comment listing names of registered reviewers who have voted for and against, and then wait for 48 hours before adding the "approve" slot. This vote is based on accuracy and content. Good presentation, as considered for feature articles, is not required for this form of content approval. The basic requirements are a good fit to the official guidelines.

• "Unapproved" slots are only to be used as a special case to deal with a rogue reviewer or an error in placing the "approved" mark, or a late rush of votes against approval.

General editors can add a tag to the article, requesting help from a reviewer to mark it as checked, or unchecked.

Reviewers are not experts

One feature of this proposal is that the reviewers are not expected to be experts. They can certainly choose to take expert opinion into account; but it is up to them to try and figure out who the experts might be!

Fundamentally, in my proposal the reviewers are expected to check against the official policies, of No Original Research, Verifiabilty, and Neutral Point of View. There are also a few other associated guidelines. The reviewers are there to check for verifiability, not truth. As such, obtaining a reviewer status should be comparatively easy. The major qualifications would several months experience, involvement in a range of topics over that time, and a demonstrated capacity for cool.

The point of having designated reviewers is to get around the problem of having having random people making poor edits, without losing the advantage of having random people making great edits.

The way the system works, most pages would simply be "undisputed", with new edits showing up as immediately undisputed. Those pages that attract controversy will tend to pick up some "uncheck" marks, and then by default general readers get an automatic reversion to periodically checked revisions.

It would be possible to give an additional clause to the revision classification algorithm, to say that if there is ANY disputed revision over the last 24 hours, then new revisions are regarded as "disputed" unless they have an explicit "check" marker (which would be picked up in the algorithm as it stands in any case). But I don't think this is really necessary. Reviewers will tend to watch pages that have problems identified in the past, and will put up the uncheck marks pretty quickly if needed. I think. So in general, new users can still expect to see their edits take effect immediately, or perhaps within 24 hours if the additional clause is used. Unless, of course, new users choose to contribute to an already disputed page.

Discussion here!

Hi Duae! You are right about this having been proposed before... however this level of detail is not the norm. I proposed something similar when I first arrived. In fact, if you have read the latest signpost, you will have seen that Citizendium has employed a very similar system for article approval. Two problems with this proposal: It is not a lightweight system, and people will say that it violates our "anyone can edit....right now!" principle. However, since so much thought has gone into this proposal it is likely to last longer than the typical "Block IP editing cuz threr vandals!!!".

I personally like the proposal and would support merging this kind of organization with our FA and GA processes. Wikipedia has changed significantly since its inception(references weren't the norm for quite a while), and I think your proposal is a step in the right direction. But let's see what others think.--Cronholm¹⁴⁴ 07:43, 6 August 2007 (UTC)

Also see Wikipedia:Flagged revisions—Cronholm¹⁴⁴ 05:45, 21 August 2007 (UTC)

Brilliant! That is the kind of thing I am struggling towards. Thanks for the pointer. —Duae Quartunciae (talk · cont) 07:04, 21 August 2007 (UTC)