Wikimedia Foundation, the U.S-based non-profit organization behind the largest online encyclopedia Wikipedia and many of its sisters, is quietly building its own advanced search feature. As the first phase of undertaking the attempt, it received a granted sum of $250,000 by the John S. and James L. Knight Foundation, to research how people access and interact with Wikimedia content.
To initiate its goal, Wikimedia Discovery, previously referred as Knowledge Engine, will help Wikimedia in exploring new ways to make the search and discovery of high-quality, trustworthy information on Wikipedia and more accessible and open.
The grant is to be used "To advance new models for finding information by supporting stage one development of the Knowledge Engine by Wikipedia," reads the Knight Foundation's grant letter to the Wikimedia Foundation. And according to a post on its website, it "will support an investigation of search and browsing on Wikipedia and other Wikimedia projects, with the goal of improving how people explore and acquire information."
Here, the aim of Wikimedia Discovery and its Discovery team is to better encapsulate the scope and goals of the project.
The grant that was approved by the Foundation's Board in December 2015 wasn't disclosed at that time to the public. Wikimedia's initial reluctance to detail the restricted grant, was a factor in the departure of community-elected WMF board member James Heilman.
In the grant document from the Knight Foundation for media innovation, Wikipedia owner Wikimedia describes:
Today, commercial search engines dominate search-engine use of the Internet, and they're employing proprietary technologies to consolidate channels of access to the Internet’s knowledge and information.
The Transparent Search Feature, Not A Search Engine
As one of the few internet services which have the most pages, Wikipedia is relying on search engines and its own search feature to point people to the things they seek. Wikimedia and its team of engineers are trying to create better links between all of the assets it owns so it can become a truly searchable open knowledge hub.
To make this happens, Wikimedia Discovery emphasizes into six areas:
- Publication curation mechanisms for quality.
- Transparency, telling users exactly how the information originated.
- Open data access to metadata, giving users the exact data source of the information.
- Protected user privacy, with their searching protected by strict privacy controls.
- No advertising, which assures the free flow of information and a complete separation from commercial interests.
- Internalization, which emphasizes community building and the sharing of information.
Wikimedia Foundation claims the Knowledge Engine to be the internet's first transparent search engine and the first one originated by the Wikimedia Foundation. But there has been a slight confusion where people sees it as building a 'search engine' when it is actually 'advancing its search feature'.
The rumor of the first concepts were revealed from investigations by Andreas Kolbe, a board member of Wikipedia's Signpost. Wikimedia and founder Jimmy Wales denied the project, calling the report as "a lie". Wikimedia tries to make things clear by saying:
"Through this project, the Wikimedia Foundation will test ways to make relevant information more accessible and investigate transparent methods for collecting, connecting and retrieving this information consistent with the values of Wikipedia and the open Web," the Foundation says.
Samantha Lien, a Communications Associate and spokesperson at Wikimedia has made further correction. In an email she sent to us at Eyerys, she said that:
Wikipedia has no plans to compete with Google or any other companies for that matter.
With the funding, the Wikimedia Foundation has started its six months of deep research, testing and prototyping on user search habits and other practices they do on the Wikipedia encyclopedia. The research, tests and prototyping results will be shared and discussed publicly.
The initial discovery stage process includes an exploration of prototypes of future versions of Wikipedia.org which are "open channels" rather than an encyclopedia. It analyzes the query-to-content path, and embedding the Knowledge Engine "via carriers and Original Equipment Manufacturers".
A budget submitted by the Wikimedia Foundation and included in the grant announcement gave a total of $2,445,873, which will be divided among 14 of its staffs (eight engineers including programmers, two data analysts and four team leaders including a director and a vice president), hardware, and associated costs that include and not limited to travel and medical expenses.
The $250,000 grant is just the first stage funding for the $2.5 million project.
Exploring New Ways From An Old Idea
Search is nothing new to Wikimedia Foundation since it originally relied on a "homegrown" search engine written by Wikimedia volunteers and based off Apache Lucene (later replaced with Elasticsearch and the CirrusSearch extension integrated with MediaWiki sites). Founder Jimmy Wales himself was once focused on creating a search engine when in 2006 he made a passing comment regarding the possibility of a wiki-based internet search.
The result was Wikia Search by Wikia, a for-profit wiki-hosting company he founded in late 2004 with and Angela Beesley.
Wikia Search was an open-source search engine intended to challenge Google and introduce transparency and public dialogue. Wales stated that Google's random tests and its closed algorithm were different from the open, community-oriented crowdsourcing attempts of Wikia Search.
While Google and other search engines approach a given query by matching keywords, subjects and the popularity of pages, Wikia Search showcased short articles written by people on topics.
While Wikia did enhanced it with additional features, the project didn't last long. The development and its service were brought to a halt on May 14th, 2009.
Wales who abandoned the project, said that it had "not been enjoying the kind of success that we had hoped." But later during that time, he also said about his interest about search, saying that "I will return again and again in my career to search, either as an investor, a contributor, a donor, or a cheerleader."
The chance then came when Wikipedia and its sisters (Wiktionary, Wikiquote, Wikibooks, Wikisource, Wikimedia Commons, Wikispecies, Wikinews, Wikiversity, Wikidata, Wikivoyage, Wikimedia Incubator, and Meta-Wiki) were having more than 35 million articles across nearly 300 languages.
"As the quantity, quality, and diversity of this content has grown, so too has the challenge of helping people find the most relevant information," said Wikimedia in a blog post.
But Wikimedia isn't Wikia, and it's not building a "search engine". Instead, it's creating an "advanced search" that focuses on improving Wikimedia experience on areas such as relevance, user experience, multi-language support, multi-projects search, and incorporating new data sources.
Further reading: As Wikipedia Turns 15, The Online Encyclopedia Is Still Facing Its Evergrowing Pain