Wikipedia talk:Research recruitment

Efficiency

Hi everyone, I read the draft and I want to open a discussion on two points for now. Would the RFC method of inviting people to review the project work faster and more efficient than for example, RCOM? --Michael Tsikerdekis (talk) 08:39, 14 July 2012 (UTC)

This is a good question. I don't think this process will be as efficient as the RCOM process -- at least initially. The problem I foresee is getting a productive conversation about a project started and completed. Sometimes the hardest part about building consensus is just getting anyone to show up.

However, this RFC model will decentralize the process of review, which I think is worth the decrease in efficiency. In order to make sure that this decrease in efficiency doesn't effectively stop research from happening, I think that two aspects of this proposal are important:

1. The researcher is responsible for making sure consensus happens.

   The current RCOM model relies on an RCOM member to push the conversation. If the RCOM member is slow or unavailable, the researcher's project can be delayed. Hopefully, by putting onus on the researcher, researchers will be able to carry their own weight.

2. The researcher will have support.

   This isn't actually part of the proposal, but it is an important aspect to making this work. I'm hoping that the current RCOMers will be available to support researchers going through the RFC process. I, for one, plan to help researchers by advising them on how to get discussion moving and how to push toward a consensus.

Really, I think that we'll have to do a lot of re-adjustment after the first couple of proposals run. From the experience of doing this a couple of times, we should be able to discover issues in this pattern and build better supporting documentation. --EpochFail^(talk|work) 15:37, 24 September 2012 (UTC)

I agree with everything here. Empowering the researcher to seek consensus is a really good idea. Posting it on Village Pump may be a good idea for finding members to come and discuss about the project. However, I sincerely doubt that most users have expertise on the aspects of research methods. Most likely they will comment on privacy issues, findings etc which is a good start and definitely it would mean that the community is comfortable with the research practices of the researcher. Michael Tsikerdekis (talk) 15:37, 8 October 2012 (UTC)

I agree that vetting research methods would be an interesting problem. However, there are wiki denizens like you and me around to help review these sort of proposals. I plan to stick around an keep an eye on new proposals. Hopefully, this process would breath a little bit of life into WikiProject Research. We can also broadcast new proposals to wiki-research-l. --EpochFail^(talk|work) 13:47, 9 October 2012 (UTC)

Ethics

Many of the recommendations are part of the broader ethics guidelines set forth by ASA and similar associations. I believe there should be somewhere as a requirement that a researcher officially makes a statement that he or she will follow the ethics code. Most wikipedians probably won't know about the ethics guidelines and so it is a good idea to have some elements clearly written as they are in the current draft. But, in research there should be always an official statement. I don't believe however that there should an ethics board approval because a) individuals coming from academic institutions will already have to obtain it anyway, b) some institutions like mine, have an ethics review board that deals with complains but does not pre-approve research (although expects that people would adhere to the ethics code) and c) individuals with no affiliation will have no way of obtaining one. In any case, the community will evaluate the proposal but an official statement also legally secures Wikipedia just in case. --Michael Tsikerdekis (talk) 08:39, 14 July 2012 (UTC)

I agree that researchers with associated review boards should include such ethics committees and their approval in their proposals, but as you explained, this is difficult to enforce based on the breadth of ethics review types. I think it would be sufficient if Wikipedians, on the other hand, had the necessary tools for performing their own ethical review. Guidelines should make assertions about informed consent, potential harm, privacy and other ethical concerns. --EpochFail^(talk|work) 17:27, 24 September 2012 (UTC)

Next steps

I reviewed the process once more and it is really straight forward. The next thing that comes to my mind is: What is the next step for moving forward with the policy? Could there be a way to test the process and see how it works (e.g., perhaps with one of the research projects that are pending evaluation)? Michael Tsikerdekis (talk) 15:47, 8 October 2012 (UTC)

That's an interesting proposal. Right now, I don't think we have any requests pending, so we'd have to wait for the next one, but I appreciate your drive to get this process in place. We could try an RFC to bring more eyes to the proposal. --EpochFail^(talk|work) 13:53, 9 October 2012 (UTC)

RfC: What do you think of the policy? Should it be tested?

The policy seems to be solid, however input from Wikipedians is ideal for making sure that everything is alright. Please report your opinion on the policy, if it should be tested and if there are any possible points for improvements. Michael Tsikerdekis (talk) 18:34, 23 October 2012 (UTC)

A similar policy had been proposed previously. See Wikipedia:Research, specifically [1]. --EpochFail^(talk|work) 20:44, 23 October 2012 (UTC)

• Wikipedia:FAQ/Research was a better name, and I do not see this as a guideline or policy but as an essay. I would add to that essay the additional details from here. Recruitment sounds more like canvassing, but this article is not about canvassing in that sense, but in asking editors for research help, instead of asking for votes (and in particular, in asking for votes that will be only to provide more votes in one direction - the direction shared by the canvasser). Apteva (talk) 22:29, 26 October 2012 (UTC)
  • Page needs moving - confusing with editor recruitment
  • Not policy per se
  • Needs to match with the WMF processes
  • Needs more input - currently to many RFCs are called to get the required input.

Rich Farmbrough, 07:07, 5 November 2012 (UTC).

I appreciate the feedback.

• I agree that the page could be named more specifically. How about Wikipedia:Research recruiment.
• Could you expand on what you mean by "not policy"? Is the concept not appropriate for a policy or just the current version of the page? Which direction would you like this to be taken?
• What does "match with the WMF processes" mean? I'm an organizing member of the meta:Research:Committee on reviewing proposals for new subject recruitment. This proposal is the future of that process.
• Agreed on the input. Do you have any suggestions?

--EpochFail^(talk|work) 14:39, 5 November 2012 (UTC)

• Yes, Wikipedia:Research recruitment since that's what I thought it was. Frankly, I'm confused. Are you saying that if someone wants three researchers (with or without Wikipedia editing experience??) they know to collaborate on an article (or a series of articles?) they have to go through this process and not just invite people to help with the article or put a notice on a Wikiproject and encourage them to join the project?? Are you trying to avoid meat puppetry, or what? I think these points have to be explicit or people will just ignore the policy. I will. I'm confused. CarolMooreDC 20:18, 19 November 2012 (UTC)

I'm sad to see the confusion because it shows a serious lack of clarity in the current draft. The intention of this proposed policy is to describe the process by which academic researchers may proceed to recruit wikipedians to be participants (aka research subjects) in their research projects. Such projects include surveys and experiments with new collaborative editing software. I appreciate any recommendations you have on making this more clear in the lead of the proposal.

This policy describes something very similar to the process we currently use to vet recruitment requests via the meta:Research:Committee. We are bringing this to the English Wikipedia in order to allow enwiki's editing community to vet researchers' recruitment plans and to afford legitimacy to researchers who are vetted through the process. --EpochFail^(talk|work) 20:29, 19 November 2012 (UTC)

I just made an update to the lead in order to make the purpose of this proposal more explicit. Did it help? --EpochFail^(talk|work) 20:41, 19 November 2012 (UTC)

Thanks for listening. This was getting me a little paranoid :-)

Frankly I think you need to merge the first two sentences of first two paragraphs something like this as the first sentence so people will understand:

Scholarly research of Wikipedia is useful for a number of purposes and this page documents the process that researchers must follow before recruiting Wikipedia editors through mass messaging to ask for their participation in a research study. Then go into further explanations.

Like a lot of people whose attention span gets briefer and briefer, I often don't get past first paragraph. CarolMooreDC 22:44, 19 November 2012 (UTC)

You're right. I re-ordered the lead so it gets to the point immediately. Thanks for the feedback! The proposal is definitely better for it. Please let me know if you have anymore -- or be bold and make changes yourself. :) --EpochFail^(talk|work) 23:08, 19 November 2012 (UTC)

Yeah! Glad I could help. CarolMooreDC 02:46, 20 November 2012 (UTC)

"Editors" versus "Wikipedia community"

This page has used the term "editors" to describe the "Wikimedia community". It has been my experience that many people outside this community do not know what the term "editors" means, and often think it is like an editor of a magazine who has control over the writing staff. The aspect of "editor" to emphasize is that it is a typical community member with no special rank, and so I changed the term "editors" in some places to "Wikipedia community". Perhaps that should be "Wikimedia". Blue Rasberry (talk) 11:14, 10 June 2014 (UTC)

Looks great. Thanks for the edits! --EpochFail (talk • contribs) 13:37, 10 June 2014 (UTC)

What is it that we are trying to control here?

I am a little bothered by the opening sentence "This page documents the process that researchers must follow before asking Wikipedia contributors to participate in research studies such as surveys, interviews and experiments."

WMF does not "own" me as a contributor; it does not decide who can and cannot recruit me for whatever purposes. What WMF does own is its communication channels to me as a contributor and WMF has a right to control what occurs on those channels. Also I think WMF probably should be concerned about both its readers and its contributors being recruited through its channels (as either might be being recruited). I think this distinction should be made, e.g.

"This page documents the process that researchers must follow if they wish to use Wikipedia's (WMF's?) communication channels to recruit people to participate in research studies such as surveys, interviews and experiments. Communication channels include its mailing lists, its Project pages, Talk pages, and User Talk pages [and whatever else I've forgotten]." Kerry (talk) 00:14, 17 July 2014 (UTC)

Strong objection

[rant - tl;dr]

Ugh, another new instruction creep with an anti-research bent to boot.

That was first third of the problem with RCOM in the first place: next to nobody knew (or knows) about it. When we still get many studies about Wikipedia who clearly display the fact that the researchers fail at basic lit review not citing any prior studies, to expect that most would try to (and be able to) find such pages is nothing but an exercise in bureaucratizing the project. The second third of the problem is that all such policies, if implemented, would make research much more difficult; anytime you add some reviewers to the mix, you add the risk of having good project rejected because of reviewers IDONTLIKEIT, and with the new proposal idea of letting complete amateurs be the reviewers... Fortunately, this doesn't fix the third compound problem of RCOM, which is that a) it had no real power to enforce anything it required and b) next to nobody wanted to invest time into doing the work, because it's a waste of time: non-productive work (not contributing to building an encyclopedia) that very, very few people in our community care about., and that adds an unimportant line to one's professional CV. RCOM is dying of inactivity and of being not needed, we should officially retire it instead of trying to clone it on Wikipedia.

[/rant]

Don't get me wrong, at first RCOM was a nice and noble idea. A guideline page for researchers is helpful, I do like the idea of trying to list and categorize ongoing research (https://meta.wikimedia.org/wiki/Research:Projects), it provides some useful links to data, FAQ and such. However, as in many other places on Wikipedia, this turned into an unnecessary instruction creep, which I very strongly oppose .

A while ago I've contributed to https://en.wikipedia.org/wiki/Wikipedia:Ethically_researching_Wikipedia It's a simple page, the gist of which is that any professional scholar who is researching Wikipedia should already be familiar with their professional codes of ethics, which in turn are perfectly sufficient to protect Wikipedia and its volunteers and users from any abuses. It also doesn't require any policing from the community outside normal scope. Any (extremely rare - can anyone even cite one?) disruptive experiments which breach the professional codes of ethics in the first place should result in bans and WMF official complains. Outside that, Wikipedians can deal with survey/interview requests like everyone else - ignore them if they don't like them. No special body to police researchers is needed. No approval body is needed for anything outside WMF grants, which WMF and/or the existing grant structure can handle.

What we need is for someone to review all research-related pages on Wikipedia and meta, merge any similar ones, and that's it. In other words, we need to condolence and organize the sprawl mess that research pages have become, not to add to them.

Regarding this page, I see this has been an open proposal for two years. I don't believe there's a consensus to implement it, nor much interest in discussing it from the general community. I therefore propose we mark this as failed and archive this. --_{Piotr Konieczny aka Prokonsul Piotrus| reply here} 03:55, 29 July 2014 (UTC)

Hey Piotrus. Since you cross-posted to wiki-research-l, I'm cross-posting my response there back to here.

I don't think that it is appropriate that those who benefit from deregulation (e.g. No oversight for running surveys. No formalized community review process.) make the decisions about what is worth regulating. You'll notice that the proposed policy that Poitr calls "instruction creep" basically states that you do three things:

1. Document your research. Specifically, your methods of recruitment, consent process, data storage and publication strategy.
2. Discuss your research -- with Wikipedians to make sure that you won't cause a disruption
3. Proceed as consensus emerges.

We all seem to agree that this is good practice. Where is the rest of the "instruction creep"? Where is the anti-researcher bend?

Poitr, you speculate about potential problems like people just coming to say "IDONTLIKEIT", but I have yet to see that happen in RCOM's process despite the fact that we invite editors from the population being sampled to the conversation. Even if it was true, I think that if some of your potential participants don't like what you are doing, you ought to address their concerns.

I'm all for developing guidelines (note that Ethically researching Wikipedia IS NOT a guideline). I've wrote my fair share of essays to help researchers & Wikipedians find their way around research projects in Wikipedia. E.g. Wikipedia:Research and User:EpochFail/Don't_bite_the_researchers. However, I've watched good research projects fail because researchers didn't have the wikipedian backgrounds that you guys do (Heather and Poitr). See some examples of (IRB approved) studies running into project-halting difficulties: Wikipedia_talk:WikiProject_Research#Examples_of_unmediated_interactions. These examples are what got me to start working on developing a process in the first place.

If you really think that documenting your research and having a discussion about it is too much instruction, then maybe you shouldn't be allowed to contact Wikipedians. If you do think that every research project that does recruitment should be documented and discussed, why not just say so? --EpochFail (talk • contribs) 13:34, 29 July 2014 (UTC)

To editor EpochFail: I support documenting one's research - I am in fact one of the few Wikipedia researchers who does so on meta. Discussing is nice in theory, but it doesn't happen in practice: next to nobody reads the proposals on meta, and I am not aware of another forum where people would post their proposed research. Consensus is of course important, too, but there are limits to its applicability, particularly the more we move away from Wikipedia and into the "real world". As you say, some of us here are experienced Wikipedians, however most Wikipedians researchers are not. While I support increasing the knowledge of best practices, which is why I suggest streamlining, reorganizing and spreading knowledge of our research pages among researchers of Wikipedia (most of which are unaware of their existence) I object to idea that a body that has not gained either general or even our own academic community approval could, using unclear rules and potentially nontransparent decision making process, veto proposals posted by the few scholars who follow good practice and announce their projects, whereas scholars who don't engage with the community won't be affected at all. Your statement above "then maybe you shouldn't be allowed to contact Wikipedians" is the best example of what I am opposing. I don't recognize your authority to have a right to decide whether I, or anyone else, should or shouldn't be able to do research. If research clearly violates Wikipedia rules (vandalizes it or otherwise disrupts it), our basic rules enforced by run-of-the-mill admins are enough. If it doesn't violate our rules, the only thing we should do is to encourage best practices through carrot approaches (offering peer recognition, certificates, access to WMF tools, etc.). Any stick approaches (right to veto a research) are inappropriate. Researchers don't need yours or mine permission to do what they do as long as they respect their professional ethics, and if they don't act ethically, then our rules (banning them) and common sense (writing to their academic departments with complains/publicizing unethical research and shaming them) is enough. --_{Piotr Konieczny aka Prokonsul Piotrus| reply here} 02:10, 30 July 2014 (UTC)

Piotrus, I'm not sure why, but it looks like you're going on another rant about RCOM. This talk page is for discussion of the current proposal. The current proposal has clear rules and a transparent decision-making process. If it passes an RfC and becomes policy, it would then carry community approval as authority.

For your arguments about how we can simply educate wiki researchers, I'd like to point out that most researchers who I have worked with via RCOM were not on wiki-research-l and had not yet published at WikiSym -- so I'm not sure how you plan to advertise your best practices documentation to those kind of researchers. It's my observation that it's this particular hard-to-reach demographic is the most likely group to inadvertently cause a disruption with their work. --EpochFail (talk • contribs) 12:37, 30 July 2014 (UTC)

My rant about RCOM is quite relevant here, as this proposal seems to take the worst idea of RCOM (gstekeeping) and tries to have it implemented here. Hence, I object to it. We don't need policies on subject research outside those in Wikipedia:Ethically_researching_Wikipedia.

Regarding the difficult to reach demographic, I agree with you, and this is a part of why I am opposing this proposal. If passed, it would do nothing to help us reach the group that as you say may be most disruptive (through I don't think that word is appropriate), but instead would make research of more experienced researchers, particularly active Wikipedians, more difficult, as this would be the only who could be reasonably expected to abide by the new requirements. Yet this group doesn't need any heavy-handed gatekeepers or reviewers. We need to focus on reaching out to those unaware of the basics, not make life difficult to those who are doing things correctly in the first place. --_{Piotr Konieczny aka Prokonsul Piotrus| reply here} 07:26, 31 July 2014 (UTC)

Piotr, you don't know what you're talking about. We mostly had new Wiki researchers who did not understand their potential for disruption come through the RCOM process. If you think that documenting your research publicly and engaging in a conversation about it is "heavy handed" "gatekeeping" then I'm not sure what to tell you, but I call it "ethical practice". --EpochFail (talk • contribs) 12:48, 31 July 2014 (UTC)

The one example you pointed to me earlier seemed totally benevolent. Unless you can actually show example of disruptive research that RCOM has stopped, I'll stand by my view that RCOM gatekeeping function is not needed, if not actively damaging. As I have said several times before, I support documentating research publicly and engaging in conversation; it is after all what I do at meta myself. What I call "heavy handed" is the idea that some researchers have the right to tell others what can or cannot be researched - I believe that our current ethics guidelines are sufficient, we have no need for a body to enforce them. Again, I am willing to reconsider if you can point me to proposals for disruptive research that RCOM stopped. But, Aaron, let's backtrack a little, this conversation is not really becoming of either of us. I respect you for a number of activities here, up to and including the Wikipedia:Don't bite the researchers. So how about we just agree to disagree on this issue? I believe both of us have stated our positions clearly enough, and let's face it - nobody else seems to care :> Shall we move on? At least till we get a chance to talk about it at WikiSym, preferably with some beer, and hopefully then we can figure out were we've gone off track. --_{Piotr Konieczny aka Prokonsul Piotrus| reply here} 14:15, 31 July 2014 (UTC)

Wikipedians who felt that this study was disruptive directed the researcher to RCom's process. This definitely not a shining example of RCom's process. I'd have rather seen the study adjusted so that it would have run and been successful. However the study was halted.

I agree though. Continuing to iterate on this conversation will likely not lead us anywhere useful. Let's reconvene in person.  :) --EpochFail (talk • contribs) 14:34, 31 July 2014 (UTC)