The Vector rollouts were contentious. I suspect this was not necessary: points of contention can be localized to the most controversial changes; changes can be rolled out starting with clear consensus improvements; and widely-requested features can be positioned as crowning achievements of getting the fundamentals in order.
The challenge edit
A month into the rollout of V22 to our 1B+ readers, it still has bugs and internal inconsistencies – including in its defining features, how usage is monitored and feedback responded to. It also has a range of uncontroversial feature requests dating to last year, to complete intended features or fix temporary workarounds, that there hasn't been time to implement.
Other projects that could face similar challenges:
- the Event Center (sizeable team, elaborate design, step function move to entire new namespace; no active demand),
- the Blazegraph failure playbook (6+ months + long consults planning for a disruptive change; in the 1.5y since, no move to migrate to avoid it)
Retcon timeline (V) edit
Ground rules for the retcon: no changing the core vision: Readers should have a clean, responsive interface w/ shorter content lines, less clutter, and fewer lines; a better search and TOC experience that doesn't require lots of scrolling; and compact, clearly-sorted menus that allow interlanguage links to be more visible.
Described laboriously here, which may make it seem like a slog. But all but a handful of these points could be automated and done routinely: an N-stage rollout could be scheduled regular releases following roughly this sequence [whatever is done gets pushed out each fortnight], surveys and clerks/grafters and a focus on a terse refactored summary of feedback could be an established norm, quick opt-in/out toggles and A/B testing could be standard and exposed to community for faster smoke testing. A public barnraising to clean up + document rough edges could be the final point on a standard checklist.
- Past work is embraced. Timeless aimed for an 80/20 solution to this, starting w/ clean presentation and responsiveness. Its overall goal was : More responsive skin; excellent options for focused Reading and Editing experiences; Support for a dark mode. (A dark mode remains the most-requested skin feature, since the original Vector.) The planned work is placed in the context of Timeless experiments; Isarra and others who worked on Timeless are formally asked to advise on the Vector update; a similarly compact vision for Vector++ is outlined.
- Advice and delegation is welcomed. Beta testers + advisors are invited to participate in regular short surveys, with summary data that's made public, as each stage rolls out. Some are invited to be skin grafters, helping make the transition seamless and maintain a related checklist.
- Ongoing vision is welcomed. Past skin wishes are brought to the fore and loosely prioritized. Those that can be moved forward by this work, or bug reports obviated by it, are highlighted.
- Set up a dashboard to track skin use and any stats relevant to the rollout. Should be visible to anyone interested, if not fully public
- R1: Cleanup and less-obvious changes. Stylesheet refactoring, logo and header updates, and the new language-links menu are planned for the first release.
- Menu sections get restyled and get a "Hide / Show" option to get a baseline on how many people use those. Keep existing language links in the sidebar, with a note that they are moving to the top.
- Add all new strings (looking ahead to the full update) to start translation + discussions of pending docs updates.
- [Timeline + process updates]:
- Set a tempo around the update process is set, including review and documentation updates and smoke tests (are major pages affected?).
- Define a single stream for contributing or report issues. Make this a space for refactoring and compressing feedback, more than for individual comments and replies: a condensed list of observed issues and responses. Delegate the refactoring and summarization.
- Quick follow-on release: refine which links are in and out of the header (like Login), refining the menu style (background color, line spacing, font size), tuning how the language-switcher works and renders. Bug reports help add use cases to test suites.
- With a new cleaner framework, a roadmap for an official darkmode is laid out: a bonus at the end of the initial update. People who have worked on the various darkmode hacks are invited to join the advisory group.
- R2: Sticky header update. Still minimal change in reading experience, but a nice quality of life improvement for everyone.
- Include search improvements and new keyboard shortcuts. Include new icons
- Expand outermost margins slightly to match the target "wide mode" view
- Leave languages collapsed by default but in the same place (still pointing people to the ULS).
- Introduce the idea of reducing central content width: offer highly-visible opt-in width toggle, with a tour of the feature, and see who chooses a narrower view.
- [Process updates]:
- Complement opt-in with survey + user testing about both wider margin and narrower body.
- Quick follow-on release: ensure it works on all pages and namespaces, doesn't wrap, and small satisfying features like keeping the logo in the header are included. Confirm usage expectations.
- R3: TOC update, expanded sidebar. This is a significant change, affecting all pages.
- Focus on enabling user choice to help determine the right default setting.
- Include prominent sidebar link / banner link for users to opt-in and test the new skin. Connect to single-stream feedback
- Leave the existing TOC in pages, but add one in the sidebar, below languages.
- Start with the same format as the old TOC (in line height, font size, no extra stats below each item, one level of nested indent).
- Explore whether to allow TOC to also / still appear in articles; how to handle related magicwords and long section titles. Identify potential new magicwords related to the TOC
- Implement basic persistent prefs for logged-out readers (for width and menu-collapse status)
- Explore user prefs for different sidebar widths.
- [Process updates]:
- Public discussion of whether this is ready for full release
- Quick follow-on: Classify active pages where the TOC move will be complicated: those with 100+ headers, &c. Leave notes on talk pages of the most active pages affected.
- R4: Body width + other default params update for logged-in users. This is a mre significant change, affecting all pages,
- Set narrower body width, remove the TOC in the body, collapse the tools sub menu.
- Include prominent sidebar link for opting out / switching back to old skin. Make this a seamless one-click process, in both directions.
- Show prominent sidebar link for reader to opt-in and test the new skin. Connect to same single-stream feedback.
- Again highlight the highly visible width toggle, now default to narrow for new accounts, with a tour of the feature.
- Borrow solutions from Minerva and Timeless and other projects that have had to solve narrowness
- Remove sidebar languages, leave the message in its place.
- Discuss persistent prefs: sidebar status, TOC location, page width, light or dark mode.
- [Process updates]:
- Survey a cross-section of those who switch either direction, and those who choose wider views.
- Complement with survey + user testing, as this is the most visible + controversial change.
- Prepare for a crunch week of responses and bug reports. Invite testing of pages that may be edge cases.
- Test moving tools to r.h. sidebar, discuss other uses (for citations, for future comments)
- Run a smoke test of pages with wide layouts that will be affected by a new default (tables, templates, multi-column pages, project pages, &c). Work w/ grafters to summarize downstream changes that need to be made to make this smooth
- Update dark mode timeline.
- R5: Update default parameters for all readers.
- Schedule a barnraising with the community of editors, around the switch.
- Roll out the same update for readers as has been in use by logged-in users, no differences.
- Include prominent sidebar link for readers to give feedback on the new skin; and for opting out / switching back.
- Discuss whether some defaults should be different for readers. Anything that might be different should be settable as a session pref while logged out.
- [Process updates]:
- Hold the barnraising; review how it's gone.
- Survey a cross-section of those who switch.
- Invite new wishlist items now that the full skin has been rolled out, fit the best into the timeline
- Overview / status page
- Include a snapshot of the dashboard of use + feedback
- Summarize how this fits into ongoing and future goals and timelines
- Thanks to those who participated, acknowledge those with remaining concerns and summarize these and answers to questions about them (how they will be monitored for impact, eventually resolved, or why they were declined + whether/how those interested can make progress solving them)
Research and statistics edit
Post-deploy analysis
- Use of the width toggle: Snapshot stats taken on two days (in some places mixed w/ one another despite being measured a few days apart). Method left unspecific; unanswered questions include
- How did toggle-usage change once it was made persistent for anons?
- Do stats include sessions where screens were too small to see the toggle?
- How many users realize the toggle exists? (measured at all? any experiments w/ different toggle-sizes?)
- How did toggle-usage change after the popup message appeared?
- Use of width toggle after showing everyone a popup
- Lack of control: popup message shown once to everyone regardless of their current setting; after that, new popup only reshown to those when they toggle into wide view (so future stats on the # of people who use that to re-limit width after expanding it will not have a counterveiling stream of people choosing to re-expand width).
- A month later: no visible data, no standard place or timeline to see / contribute to analysis
- Use of modal to set width w/ more context: ticket stalled
- Abandoned? no standard place to see status or expectation
- Use of "switch to old look" link:
- This was never made one-click; but 30 seconds and some fussing. No stats on the # of link clicks compared to the # of skin switches.
- No control: no "switch to new look" link in old skin (despite two tickets requesting it)
- Open ticket
- No data on very active editors (despite historical data including that; despite request)
- Early survey methods were poor on two occasions. During RFC some comments suggested they would redo / update an incomplete survey post deploy. None started as of 5/2023.
- Research should have method input from a research genie; many on staff, and many of the world's best in the community (at wikiconfs)
- Persistent noise in data
- no controls. antipathy to controls?
- no removal of hourly, weekly, seasonal, systemic trends
- confirmation bias : desired outcome always known by researcher
- reporting bias : negative outcomes not reported, obscured, minimized
- moving targets: bins, metrics, visualizations change often for anything not put on a dashboard or in a long-lived table. dashes tend to be great, when maintained
- term redefinition: 'active' editors, # of edits --> characters changed, &c