Wikipedia talk:RfA Review/Archive/Reflect

Archive Pre-reflect Reflect

Reflection

Where, exactly, is this reflection taking place? Phil Sandifer (talk) 06:14, 5 July 2008 (UTC)

Fair point, and I need to get a page up explaining what this stage is all about. Essentially, I'm currently compiling a report on all responses to the question phase (circa 200), before submitting it for review for depth of content, balance and neutrality. This report will then form the basis for the recommendation section. It's going to be approximately 2 weeks before the report is up for peer review here, as there's a lot of study to complete. Hope this makes sense. Gazimoff ^Write_Read 18:45, 5 July 2008 (UTC)

So this is a solo project, at least in terms of basic report composition? Phil Sandifer (talk) 18:28, 6 July 2008 (UTC)

Ish. Why do you ask? If you could give me some further information or background to your question, I'd probably be able to give you a more complete answer. The structure of the report, however, is based on the Question phase of the review, encompassing all responses provided. Hope this helps. Gazimoff ^Write_Read 18:50, 6 July 2008 (UTC)

Because I had been under the mistaken impression that this was a community-driven review, not one person running a survey. Why, on a wiki, are you avoiding a communal writing process for the report? Phil Sandifer (talk) 19:04, 6 July 2008 (UTC)

It's a good question, and a valid one at that. Essentially, I offered to perform this review, as previous wiki-centric processes had failed to reach any meaningful consensus. Because editors hold a wide range of disparate views, it was felt that a survey-based approach would allow editors to contribute their thoughts and feelings without the process becoming bogged down in debate. Besides, the process of compiling the report is much the same as crafting an article - it's quite common for an article to be researched or compiled offline or in a sandbox, before being moved into the mainspace (or projectspace) in this case, once it's in a fit state for wider examination. The process is community driven - the structure of the review, the questions and the responses have all been from the community. All that's happening is that the report is being prepared for upload in one block, so that the community can then analyse it, discuss it and so on in one contiguous piece. The methodology being used is quite similar to that used for corporations and businesses to analyse internal processes. I hope that clarifies things for you. Gazimoff ^Write_Read 20:17, 6 July 2008 (UTC)

<- I think, if we had a bigger team and could parcel out the responses 10 or 20 per editor, then a group draft would be workable - and maybe it is. My understanding is that the report from Gazimoff will basically collate the responses and find common threads. For example, on the question "How do you view the role of an administrator?", the report will likely say that, of 200 responses,

1. 77 editors view admins as authority figures
2. 56 editors view admins as janitors
3. 34 editors view admins as enforcers of policy

...and so forth. I would also expect that, once we have this list of responses and how widely held each is, that the community (or Gazimoff, or whomever) would provide some analysis, such as "Examples of admins as policy enforcers included [this diff], provided by [[Response page|This editor]]...", where the (admittedly few) diffs would be incorporated. Most of the questions are intended to gauge where RFA and adminship is currently, and I think the specific question I use as an example is intended to give us a target - where is adminship now, and how should that impact how we choose admins? So long as we can edit the report - which we will - I don't have a problem with Gazimoff doing the initiall analysis and tally - though, if he were to provide his reporting format and a block of responses, I'd be happy to pitch in. UltraExactZZ ^Claims ~ Evidence 15:01, 7 July 2008 (UTC)

That's a goods point, and one that I'm keen on sharing the workload on. My analysis is centred on examining the statements from each editor in order to build up an overall picture, and is time consuming. If you can perform a statistical analysis on the feedback, the report can contain both a mathematical and logical analysis of the survey results. It's a great idea, and I'm all for it. Many thanks! Gazimoff ^Write_Read 10:22, 8 July 2008 (UTC)

OK, let me work up a method, then. Anyone else want to pitch in? UltraExactZZ ^Claims ~ Evidence 12:40, 8 July 2008 (UTC)

I'm game, assuming you guys have any use for relatively unskilled !labor. I've also got access to a dumb station to run things on I won't need for the rest of this week. Cheers. --lifebaka (Talk - Contribs) 20:28, 8 July 2008 (UTC)

I've created a category for the unsubmitted responses, Category:Unsubmitted Wikipedian responses to RfA Review, and my first task will be to move the unsubmitted items from the main Category:Wikipedian responses to RfA Review to the unsubmitted category. Then, the responses category will match the list at Wikipedia:RfA Review/Question/Responses, except that it will be in alphabetical order. Then I'll break the list into 20-25 names apiece, and each of us can read through the responses and tally the comments. I don't think we need to go too in depth, as that's what Gazimoff is doing with his logical analysis. This read will be to determine, for example, how many people oppose Self-Noms? I'll plug the tallys into a spreadsheet and graph the results. Maybe I'll do a trial run tonight, just to see if it's feasible, but I don't anticipate many problems. UltraExactZZ ^Claims ~ Evidence 20:51, 8 July 2008 (UTC)

That's great. I'd like to make a judgement call on completed response sheets that were not added to the responses list but are still in the cat, as there is a lot of feedback in some of them. Personally, I'd like to include all apart from the blanks and non-response-related submissions. Also, if possible I'd like some stats on the responders, such as admin/non-admin, time since registered etc as a form of demographic analysis, although I suspect this would be best suited to bot-work. I also have a cat, Category:Processed responses to RfA Review, which I'm using to tag responses once I've processed them. You might want to look at a similar cat to track tally processing. You might also want to define your tally chart before counting, so that we can get an idea of the granularity. Hope this helps, bit of a rush of blood to the head :) Gazimoff ^Write_Read 21:35, 8 July 2008 (UTC)

I agree, except that I would like to leave the chart open until we're done. If I have three different responses for Question 1, and the next survey has a response that fits none of those, then I can just add it to the list; subsequent items that match it can be added to that tally. I think I moved two unsubmitted responses, mainly to populate the category - but, I agree, only blanks should go there. UltraExactZZ ^Claims ~ Evidence 21:49, 8 July 2008 (UTC)

Fair enough, works for me. If I come across any blanks myself, I'll shift them into your cat as well. Many thanks, Gazimoff ^Write_Read 22:06, 8 July 2008 (UTC)

Opening this up

After some discussion, I'm opening this work up. i was wrong to think I could complete it by myself, and I need help piecing it all together. currently I have handwritten notes on some 60 submissions, which I'm using to create the article structure before filling them out with information from the remainder. Problem is, at this rate it's going to be about 3 months before I manage to process the remaining three quarters of the submissions. That's why I need help in finishing this article.

All submitted responses should be in Category:Wikipedian responses to RfA Review. Processed responses should also be in Category:Processed responses to RfA Review. Blank or meaningless forms should be in Category:Unsubmitted Wikipedian responses to RfA Review. What needs to happen is for each response that has not been processed, responses to each section need to be analysed to see if they add anything to the report at WP:RREV/R, or help to clarify meanings. If they include refs to examples, add those refs to the reflect article. Once that's done, tag it as processed and move on to the next one. Once all are processed, we can develop conclusions and summary bullet points. One of the things that UltraExactZZ was looking at was adding tally information to give an idea of the level of responses for each section of the report. If you'd like to help out in this area as well, that would be great.

Many thanks for any help and support you can provide in completing this report. Gazimoff ^Write_Read 00:46, 18 July 2008 (UTC)

I'll be throwing some time at this, to see what we can extract from it and some pointers as to where we are headed, and we will be pleased to assist in whatever way I can. Pedro :  Chat  07:42, 18 July 2008 (UTC)

If you'd be interested in helping me process responses and build the report, that would be a huge hlp. It's time consuming work, but hopefully worthwhile. Many thanks! Gazimoff ^Write_Read 08:32, 18 July 2008 (UTC)

I'm working my way through a tally of how many times common responses were given. Once I have fifty or so, I'll post something and group the remaining responses, so others can tally as well. I'm basically adding unique responses to the list as I go, but those are getting fewer and farther between as I proceed. One thing I can say with certainty, though - almost everyone agrees that candidates should be able to withdraw at will. UltraExactZZ ^Claims ~ Evidence 15:19, 18 July 2008 (UTC)

Breaking for lunch, I'm up to 36; I expect I'll have 50 by the end of the day. UltraExactZZ ^Claims ~ Evidence 15:28, 18 July 2008 (UTC)

Tally Assistance

OK, there are two phases of the Reflect process. One is the Qualitative analysis, which Gazimoff and others are working on - what was said in the responses? The other is the Quantitative analysis, which I am working on - how many people said X, Y, or Z? I stress that this doesn't dimish the responses into sheer numbers, or a popular vote - rather, it's intended to highlight numerical trends in the responses to help find common threads, and to determine how common those threads were.

Here's where I am. I've taken the first 55 or so responses (alphabetically) and tallyed the comments. The different comments I found are broken out by question at User:Ultraexactzz/RREV Tallys. What I'd like to do, if this format works, is to set up a page for each set of remaining responses, 5 at a time, and get others to assist in tallying the responses. If there's a comment that doesn't fit a current field, no problem - a new row can easily be added. Once we have them all, I'll combine them and make up nice pretty graphs of the results.

Is this a process that can work? Or is there a better way I'm missing? Thanks in advance for the assistance, UltraExactZZ ^Claims ~ Evidence 18:14, 24 July 2008 (UTC)

It looks reasonable enough to me. No problems here. Gazimoff_{(mentor/review)} 18:24, 24 July 2008 (UTC)

 Done I have posted the first 100 responses in ranges of 5 each at Wikipedia:RfA Review/Reflect/Statistical Analysis, and will complete the list later today. It should be pretty straightforward - and thanks to everyone who helps out! UltraExactZZ ^Claims ~ Evidence 14:54, 25 July 2008 (UTC)

I'll take a stab at some of the "groups of 5". Can statistics really be fun? I'll report back.  :) Livitup (talk) 15:52, 25 July 2008 (UTC)

Dunno' what's goin' on with this right now (nothing since the 25th, looks like), but I'll be doin' at least one a day 'til they're all done. Cheers. lifebaka (talk - contribs) 04:07, 31 July 2008 (UTC)

Life intervened, I'm afraid - but I've now posted the remaining 15 lists, so all responses are accounted for. I'll work a list later today; with enough help, we should bang this out in no time. UltraExactZZ ^Claims ~ Evidence 12:43, 31 July 2008 (UTC)

I did one of the ranges. Wow, was it tedious. I mean fun. It was fun. Join in, everyone! Useight (talk) 18:46, 31 July 2008 (UTC)

m While the plural of Majorly might be Majorlys, I'm pretty sure that the plural of tally is tallies. SHEFFIELDSTEEL^TALK 21:23, 31 July 2008 (UTC)

Thanks again for the assistance, we're making good progress. One note, though - if you find a questionnaire that's blank, just take the "1" out of the "Total Responses" line at the top of the page, and leave a note at the bottom or in your edit summary; I'll remove it from the total, and from the category for responses. That'll help when we do totals, such as finding out how many responses favored self-noms vs the total, or what have you. Thanks again for the assistance! UltraExactZZ ^Claims ~ Evidence 14:59, 4 August 2008 (UTC)

Livitup's doing the last one, then we are  Done with analysis. UltraExactZZ still has three to record, though. Cheers, everyone, and great work. lifebaka++ 13:11, 13 August 2008 (UTC)

Nice job, guys. That was some tedious and time consuming work. Well done. Useight (talk) 14:46, 13 August 2008 (UTC)

We're finished!

And... I'm spent. Turns out statistics can, indeed, be fun. :) Livitup (talk) 13:33, 13 August 2008 (UTC)

Thanks to everyone who helped out - we now have statistics on all 209 responses. I'm going to see if there's a way to get this spreadsheet (which includes over 440 distinct responses and statements) into some sort of format to distribute, if only to ensure that this review is open and public. Maybe I can post the CSV as code, perhaps? We'll figure it out, but for now I'll be posting some of the highlights. Again, thanks to everyone for helping! UltraExactZZ ^Claims ~ Evidence 15:33, 13 August 2008 (UTC)

Range 30

On range 30, Tony1 chose to make comments outside of the questions. I'm not quite sure how to handle this, though my first blush idea would be to look at what he said and just put it in the question-blocks where it fits. Just trying to get a little advice first, though. Cheers, everyone, and thanks. lifebaka++ 14:42, 12 August 2008 (UTC)

Range 28 had someone do the same, and I just plugged in the answers where they fit. I've also been doing that for the last question, if they offer a suggestion that I know I've seen in a previous question - statements about NOTNOW falling under voting, for example.UltraExactZZ ^Claims ~ Evidence 14:55, 12 August 2008 (UTC)

Gotcha'. I'll go do it, then. Cheers. lifebaka++ 14:57, 12 August 2008 (UTC)

Tinkering with Graphs

Quick and dirty, graph number one.

OK, so now that we have our numbers, I'm starting to see what graphs would be worth including. Are there any comparisons you see that might be interesting? I have a few in mind already. I'd also like to know if it's worthwhile to put graphs in the Reflect page itself, or if an appendix (or Graph gallery) would be better. Thoughts? UltraExactZZ ^Claims ~ Evidence 16:48, 13 August 2008 (UTC)

I think the 3D on some of the graphs isn't necessary, but it's not a very big issue. So far I like what you've been doing. A pie graph for C4 between "limited canvassing OK"/"link from userpage OK", "unlimited canvassing OK", "no canvassing OK", and "bot generated list OK" might be useful. On the bar graphs, noting total responses to each of the questions would be useful for interpreting the data; this could be included in the footnote, though. Just some thoughts. Cheers. lifebaka++ 18:25, 13 August 2008 (UTC)

Side note, I can't tell the color scheme for 1% and 30% they are too close to each other. Also, how do we end up with 126%?---Balloonman ^PoppaBalloon 21:32, 13 August 2008 (UTC)

Ehm, I believe it's number of users, not percent of users. So, 126 is a reasonable number out of 209. A bit low, if anything. Cheers. lifebaka++ 21:37, 13 August 2008 (UTC)

Participation at RfA

Yeah, that's total responses, which I should indicate somewhere. I'm mainly trying to show which statements popped up more than others, but there are a few snags. For example, we had a few people who said that "RfAs don't get enough attention" AND "Limited Canvassing is OK, if neutral", so when we graph canvassing responses, that person is counted twice - they held both sentiments. So percentages are dicey, and will be for most questions. The exceptions will be the next to last two, "Ever vote in an RfA?" and "Ever a candidate?", where we have a sharp divide between yes and no. For those, I'll add the responses where the editor didn't offer an answer as a third choice, "No response", and give a percentage of the total. I think I already did one already for "Ever voted?". UltraExactZZ ^Claims ~ Evidence 01:52, 14 August 2008 (UTC)

For my own preference, I'd avoid 3D graphs as they can introduce a perspective distortion in the representation. Other than that, it's some fantastic work and given me a bit of a kick up the ass to get my side finished :) Gazimoff 10:19, 14 August 2008 (UTC)

OK, we have stats for every section, as well as some prose to explain what the numbers are. I'd like to get some input on what else to add to the report. Are there RfA statistics that would be worthwhile? I think we're closing in on the next phase, which should be exciting. UltraExactZZ ^Claims ~ Evidence 15:03, 20 August 2008 (UTC)

I was thinking that there's a bot that parses a list of active admins (currently around the 1000 mark). I'd like if possible to get some recent trend graphs to show how the number of total admins and number of active admins vs total registered user count. What do you think? Gazimoff 15:07, 20 August 2008 (UTC)

Total admin counts are listed at User:NoSeptember/admincount, which would be fun to graph. Still looking for the active list, though - but I agree, we should include it as background. UltraExactZZ ^Claims ~ Evidence 15:20, 20 August 2008 (UTC)

Admin statistics, Inactive and Active

OK, so I've got the numbers for active admins, based on trolling the update history of the master User:Rick Bot updated list at WP:LA. Near as I can tell, automatic updates started on 20 August (!), so we have a year of data. I'll run the inactive numbers and compare the trend for the past year, unless someone has stats for more than a year ago... maybe? UltraExactZZ ^Claims ~ Evidence 12:21, 21 August 2008 (UTC)

I dont have one myself. But I think it'd be great to see that graph. You might also want to look at User:E/AdminStats for further background information, particularly the part on total admin actions. Hope it's of interest to you. Gazimoff 12:29, 21 August 2008 (UTC)

I'll have a look. I've got the graph for the past year, though - I'll upload it momentarily. It looks like adminship has trended up (from 1307 last august to 1584 now), while active admins have stayed relatively flat. Let me see what I can come up with. UltraExactZZ ^Claims ~ Evidence 13:02, 21 August 2008 (UTC)

Here's what I have for the past year:

• 
  Total Admins
• 
  Proportion of Active to Inactive
• 
  Active vs Total, Trendlines

There are also some charts at User:NoSeptember/admin graphs, including Image:En-admin-growth.png, which is exactly what I need - but it doesn't have April - July 2007, which I'm missing. If I can track those numbers down, or extrapolate them, we'll have a 6 year trend to review. UltraExactZZ ^Claims ~ Evidence 14:04, 21 August 2008 (UTC)

That's some fantastic work. So essentially, for the previous twelve months, although the total admin count has increased the number of active admins has flattened? I think in order to round this work off, I need to take a copy of the data table that lists the admin actions undertaken by each admin and do some analysis there for another graph in order to examine the admin workload breakdown. I'll let you know once I have more on that. Great work so far though! Gazimoff 14:46, 21 August 2008 (UTC)

It is good work, but if I can impose here just for a small, yet important point (given that I haven't contributed before here), the labelling as given in the description for the thord graph does not correspond to what is shown. The graph shows numbers of people, and the description states "This chart uses data from the List of Administrators and other sources to show the proportion of active administrators to total administrators." which uses "proportions". It may seem a small point, yet it is highly important in the field of interpretation of data, of which others have stated elswhere I have some expert knowledge.

The two (the graph and its description) need to be brought into agreement: Either (a) change the description to use "numbers" instead of "proportions", or (b) recast the graph so that it plots the proportion of active admins against time, there being no need to have the proportion of admins in this case, as it would always be at 100% or 1.0 (depending on whether one uses percentages or fractional depictions of proportions), or (c) change the wording in this graph to use "numbers" and plot a small additional graph which directly plots the proportion.

It depends on whether one wants to interpret the figures as indicating that "the total number of admins has risen, whilst the number of active admins has flattened", or whether "the proportion of active admins has decreased"

If one were really intent on exploring this, and it may be too much to realistically do, one might consider doing some log-linear modelling to assess any statistical significance to such data, though the information would have to be re-jigged to do that (issues to do with satisfying various crietria to make the applied log-linear models valid, etc)  DDStretch  (talk) 15:01, 21 August 2008 (UTC)

Yep, you're completely right about the third chart and I'd add to it that not starting the axis at zero makes it even harder to judge. Mind you, the second one covers the proportion of active to inactive easily enough, while the first one shows total trends easily enough. I'm not sure if any log-lin analysis would help here - the dataset doesn't feel as if it would show anything from it and I'm not sure if it would highlight anything that would be of use. I'm happy to listen to suggestions though. Gazimoff 16:05, 21 August 2008 (UTC)

This was the chart I was thinking of. The data behind it (you can read it off the chart) shows that roughly 80 admins are responsible for 50% of admin actions (deletes, blocks, protects, granting rollback/acc, etc). Slicing it a different way, roughly two thirds of the admins have performed 10% of the total admin actions, while one third are responsible for 90% of the total number of actions. I think this chart shows that not only do we need more admins, but we need more active admins willing to perform janitorial duties. Gazimoff 15:50, 21 August 2008 (UTC)

I've updated the third graph, above, with a clearer description that substitutes numbers for proportions - and remind me never to upload images when I'm in a hurry, as that's what I get for copy-pasting. My intent was to illustrate the trends in both total and active admins, and didn't think that starting at 600 would cause a problem (as neither figure dipped below 800 during the sample period). I can correct that bit, as well, but I concur with Gazimoff that the first two graphs accomplish our purposes well enough. UltraExactZZ ^Claims ~ Evidence 16:43, 21 August 2008 (UTC)

"I think this chart shows that not only do we need more admins, but we need more active admins willing to perform janitorial duties."

— Gazimoff

I disagree. This chart shows that there is a wide variation in the degree of admin-type activity between different admins. It does not demonstrate a need for more admins, or a need for more active admins. Axl (talk) 15:29, 28 August 2008 (UTC)

I think it does demonstrate that the most active administrators over the history of the project have been really, really active. If the top 10 most active admins, say, were a group or acted as a bloc at rfa or elsewhere, it'd certainly fuel concerns over clique-ishness (as were raised under question 1). I'd be curious to see a breakdown of how that chart would look for, say, the past year, rather than the history of the project. A trendline on that would be fascinating. On point, we had two responses that said that there were too many inactive admins, so I don't know that we can tie a recommendation to it... let's think about it. UltraExactZZ ^Claims ~ Evidence 12:21, 29 August 2008 (UTC)

Axl, I think I disagree. The problem is that by having a small nexus of highly active admins, you do two things. Firstly, you place a high level of reliance on those admins for keeping backlogs low. This can lead to problems when a highly active admin leaves the project, as backlogs start to creep up again. Secondly, by placing a large amount of work on a small team of admins you begin to increase the levels of stress and risk of burnout. In order to reduce these risks, it makes sense to recruit more admins who are prepared to carry out admin tasks and reduce the burden placed on any one individual. Lucklily being a voluntary organisation, we are only limited by those who wish to take on the role and who the community feels are suitable. Hope this helps, Gazimoff 12:44, 29 August 2008 (UTC)

"If the top 10 most active admins, say, were a group or acted as a bloc at rfa or elsewhere, it'd certainly fuel concerns over clique-ishness."

— UltraExactZZ

That's a big "If". These data don't demonstrate that. Regarding "too many inactive admins", which Gazimoff also alludes to, the relevant question is: Are backlogs increasing due to inadequate admin-type activity? Axl (talk) 15:37, 29 August 2008 (UTC)

Agreed; I was using a hypothetical to tie the discussion into the survey, I see no evidence from the source to support that. So long as the growth of admins, active and total, is consistent with the growth of the userbase, the article count, and the daily volume of edits - in effect, with the growth of the project - then I'm not concerned about the number of inactive admins. I do want to know what leads them to be inactive, and whether some of the inactive admins still edit - remember, inactive is counted as an admin who has performed no administrative tasks in 30 days. Is it an aspect of the culture surrounding adminship? Or perhaps it is natural that more admins would be inactive over time. UltraExactZZ ^Claims ~ Evidence 17:42, 29 August 2008 (UTC)

"By placing a large amount of work on a small team of admins you begin to increase the levels of stress and risk of burnout."

— Gazimoff

Admins, like all editors, choose what they want to contribute. No individual has a defined minimum or maximum remit. Is there evidence to show that the most active admins have a higher rate of burnout? I expect that the most active admins have a specific personality type that drives them to make more contributions. I am not convinced that increasing the number of admins (or increasing the number of "active" admins) will actually reduce the workload of the most active, or reduce their risk of burnout. Axl (talk) 15:46, 29 August 2008 (UTC)

Stress and burnout can happen in voluntary organisations as well as in the workplace. But the consequences are the same as a highly active editor who decides to move on to other projects, or who loses interest or enthusiasm, or who has a life changing event that requires them to contribute less. My point was around the risk the project faces from having a small number of highly active editors and to raise questions from it. Do we feel that the risk has a low impact or low probability, or do we feel that it has a high impact or probability and requires reasonable mitigation. It's only prudent to ensure that you manage risk effectively, which includes analysing workload distribution and levelling in order to mitigate or lower overall risk to the project. The examples were to try and illustrate the examples that contribute to this risk. Admittedly, I should have qualified my statement somewhat, but the question on overall risk level and what constitutes an acceptable level remains. Many thanks, Gazimoff 21:53, 29 August 2008 (UTC)

Okay, I see what you mean. However Wikipedia is no ordinary voluntary organization. As I said above, individuals determine their own workloads. Are you (or someone else) able to find out who (say) the top ten active admins are, and ask them specific questions on their talk pages? Axl (talk) 08:16, 30 August 2008 (UTC)