|
Content Discovery Experiments
|
Between July and December 2024, the Web team conducted experiments checking how to make it easier for readers to discover information on the wikis. This was part of our annual plan focused on content discovery for readers.
The team ran four experiments. This page outlines the details on each experiment, as well as our findings and timeline.
The results of the experiments will be used to identify which features we will build for the future, based on how useful they are proven to be in helping readers find useful and interesting information across the wikis.
Updates
[edit]Background
[edit]As new generations of readers come to our sites, we want to make it easy for them to learn. This includes making it easier to find the information they need across articles and pages as well as making it easier to discover new information based on their needs or interests. To do this, we hope to leverage advances in technology and build out new capabilities within our platform that will allow us to make discovery and browsing easier than it was in the past. This goal led the team to focus on more experimental work to begin with, as a way of being able to test out more features quickly. We started with the following hypothesis:
We designed and qualitatively evaluated three proofs of concept focused on building curated, personalized, and community-driven browsing and learning experiences. This allowed us to estimate the potential for increased reader retention (experiment 1: providing recommended content in search and article contexts, experiment 2: summarizing and simplifying article content, experiment 3: making multitasking easier on wikis)
This hypothesis focuses on identifying ideas for features or projects that would make it easier to browse and learn across the wikis on the desktop and mobile websites. Work here included identifying and building proofs of concepts for each idea and providing the results of initial tests on the idea.
The goal is to find ideas that we think would work well at a larger scale and commit to building them and providing them to the wikis.
This work begins discovery into areas that we have not yet been working on in the past, such as summarizing or remixing content. Working across communities to ensure proper editing and moderation workflows for these new content types will be crucial.
Experiment 1: Display article recommendations in more prominent locations, search suggestions
[edit]The search suggestions experiments proposed the development of a feature which provides suggestions to readers when they open the search bar of a page. This feature was proposed as a tool that could make browsing easer for readers by providing a quicker way to explore topics similar to the ones the reader is currently on. This set of experiments explored reader interest in the search feature, and the feature's effects on clickthrough and session length. After all experiment hypotheses were supported, the feature was selected for a full rollout across wikis.
Experiment 2: Improve article recommendations APIs
[edit]We wanted to compare various existing and proposed APIs recommending content independent of the user interface. This work allowed us to test different recommendation methods against one another to determine which ones are the most useful for readers.
Experiment hypothesis
[edit]If we compare readers preferences for suggested article APIs, we will learn which API readers find most useful.
Experimental setup
[edit]A quicksurvey was displayed to a small percentage of readers on English and Spanish Wikipedias, inviting them to take a survey. The survey presented recommendations from three different APIs. Users were be asked to submit which recommendations were most useful and interesting to them.
Experiment 3: Display article recommendations in more prominent locations, article pages
[edit]
We wanted to explore displaying recommendations in other parts of the page. This experiment focused on displaying recommendations alongside article content on desktop.
Experimental setup
[edit]The experiment code was loaded in the Wikipedia Recommendations browser extension. Users of the browser extension saw the recommendations immediately.
Experiment hypothesis
[edit]If we display recommendations alongside article pages, people will be interested in reading the recommended pages, as shown through a clickthrough rate similar to, or higher than, existing recommendation entry points (Related pages, Experiment 1).
Results
[edit]- On English wikipedia (enwiki) desktop, the overall CTR for article recommendations is 0.54%
- On Spanish wikipedia (eswiki) desktop, the overall CTR for article recommendations is 0.28%
We observed that the CTR for sidebar article recommendations is higher than that for related pages on desktop (listed at the bottom of articles) for both English Wikipedia and Spanish Wikipedia
- English Wikipedia (enwiki): 0.41% ->0.54% , increased 0.13 pp
- Spanish Wikipedia (eswiki): 0.13% -> 0.28%, increased 0.15 pp
Hypothesis was successful:
? Readers will want to click on recommendations on the right of the article (measured by comparing the CTR of recommendations in search with other recommendation locations - specifically related pages on desktop).
Next steps
[edit]Since the hypothesis was successful, we recommend to proceed with an A/B test of the feature on the desktop website.
Experiment 4: Article Summaries
[edit]Research
[edit]- Wikimania 2024, "Written by AI" How do editors and machines collaborate to create content. At Wikimania 2024, Wikimedians discussed ways that AI/machine-generated remixing of our existing content can be used to make Wikipedia more accessible and easier to learn from. The goal for the new features is to help a new generation of readers discover and learn from reliable, encyclopedic information on Wikipedia. The session focused on ways to give control and oversight to editors and communities over the structure and content of these new concept and feature ideas.
- Usability study on Simple Summaries. We ran an unmoderated Userlytics study on a prototype of our Simple Summary feature with 8 participants. The prototype included a summary of the introduction of the English Wikipedia article on Dopamine. We analyzed videos and screen recordings of the sessions. Participants found simple summaries easy to use, useful, and had an appropriate level of trust in the machine-generated summary. Responses were much more positive than expected. The main issue to be addressed before release is ambiguity between the Simple Summary and the main article content while the simple summary accordion is closed.
Timeline
[edit]| Estimated time | Planned activity |
|---|---|
| |
| Building experiments | |
| |
| Running experiment 4 |