Google Search has been synonymous with the internet. It's been there to provide answers to almost anything, in less than a second upon hitting the enter button.
However. there are times that it's actually Google's autocomplete suggestions that aim users to the intended questions. This eases the process of using Google Search.
And this time, Google is improving that feature using its 'MUM', a machine learning model that stands for 'Multitask Unified Model.
With it, Google hopes that its search engine can answer users query by processing more than just one question at a time.
At the company’s Search On event, Google' MUM can literally turn the search engine into something like a mind reader.
This is because MUM can pick up information from formats beyond text, to also include pictures, audio and video.
Google's MUM can also transfer that knowledge across the 75 languages and counting, that it’s AI has been trained on.
In an example, Google said on Twitter that users can take a picture of a broken bicycle part and do a Google search with a query: "how do you fix this?"
With MUM, Google can correctly identify what's that exactly in the photo, and understand the exact part that needs to be fixed.
In this example, is its derailleur.
Google then returns answers by showing a list of videos, forum results, and other articles on the web the search engine has crawled and indexed.
"By combining images and text into a single query, we’re making it easier to search visually and express your questions in more natural ways," Google said.
Users can do this by pointing the Google Lens-enabled camera toward the thing they want, and combine that with a query.
Another example involves capturing an image of a style, like a shirt, and ask users about other pieces of clothing, like socks, with the same design.
Next, is an example of Google in using the machine learning model to recognize moments inside a video, and quickly identify the topics related to the scene.
For example users watching a video on a specific animal not mentioned in the video title or description, Google can still pick up on the species and return related links.
Google also wants to improve its search capabilities by redesigning the way it extracts context from images.
The next thing, is the 'Things to know' section, where Google can surface description of a product, and suggests a place where users can buy it, for example.
Google can also populate the section with links, like how to use the item, and other information on certain subtopics associated with the query.
In this particular case, Google’s AI can expand users' search query into even more questions, as if dragging users down a rabbit hole. However, the goal of this feature, is to make users click/tap less, to get more information about what they want.
And pair that with Google's knowledge of its billions of users, the company can tailor users' search results pages with only the things they are interested in, like for example, based on their previous search history.
Google 's Senior Vice President Prabhakar Raghavan who oversees Search alongside Assistant, ads, and other products, said that "search is not a solved problem."
While improvements in the AI can certainly make Google Search a lot more competent, in the end, at the bigger picture, it's Google's way to gather more context about users' queries.
The goal of the improved AI, is less to do with wrangling the web and more to do with adding context to what they find there.
“It’s a cycle that I think will keep escalating,” Raghavan said.
“More technology leads to more user affordance, leads to better expressivity for the user, and will demand more of us, technically.”
More complicated, multimodal searches by combining text and images demand "an entirely different level of contextualization that we the provider have to have, and so it helps us tremendously to have as much context as we can,” explained Raghavan.
Google Search with its inputs and outputs have all become almost unimaginably complex.
Google can expand that capacity using algorithms and language models. The company can do that, thanks to the company having indexed a massive corpus of data from its original mission of indexing the web.
But this thing here, is a monumental task that can go beyond what Google has ever did.
"If you gave me all the machines in the world, I could solve fairly big instances,” said Raghavan. But for this particular ability, it is perhaps unsolvable by just throwing more computers at it. Instead, Google needs to come up with new approaches.
And MUM here, is making use of Google's massive resources of data as its base.
"If you gave me all the machines there were, I’m still bounded by human curiosity and cognition."
MUM made its debut in May. Similar to BERT, the AI is built on a transformer architecture, but is reportedly 1,000 times more powerful and capable of multitasking to connect information for users in new ways.
In its first public application, MUM identified 800 variations of COVID-19 vaccine names across 50 languages in a matter of seconds. At that time, it did not show off the technology’s multimodal capabilities.