While Facebook has allowed users to manually attach tags and locations on photos to make it simpler for others to find them, the social giant has revealed that it's developing its Artificial Intelligence (AI) to better identify photos on its platform.
Facebook has long been able to recognize the faces of people that are inside photos, but it hasn't been good in understanding what's actually is inside them. But things are changing as the developments of AI at Facebook is trying to improve user experience in two ways:
First, Facebook is making it possible for users to search for photos based on what's inside them, rather than just by the dates they were taken, the tags or location. Second, it will improve Facebook's alt text feature which describes photos aloud for visually impaired users.
To enable such feat, Facebook has analyzed billions of photos using deep-learning the company called FBLearner Flow. The technology analyzes images shared on the platform each day, giving it plenty of data to feed on.
And using a variety of signals, Facebook then ordered them in ranked results.
Then using Lumos, a computer vision engine that is built on top of FBLearner Flow, Facebook built it to understand the contents inside photos.
The neural network was trained to recognize only specific pieces of information, such as scenes, animals, places, clothes and attractions. Because those contents in photos can then be searchable, users searching for someone wearing a "red hat" will pull up images of people wearing red hat. Text or tags that indicate the presence of a red hat aren't required.
Lumos was first used by Facebook to improve the platform's accessibility for visually-impaired users by allowing images to be described by their contents. But the technology can be improvised and has other uses, and this is what Facebook was developing.
With Lumos, Facebook has made things easier for users to find photos based on what is inside them.
Facebook took the technology even further to have a basic understanding of behavior. For example, searching for "people dancing," "dog running," and more.
Joaquin Quiñonero Candela, director of Facebook's Applied Machine Learning team, said that that being able to recognize specific actions, like running or jumping, requires a deeper neural network. But these networks are harder to train. The deeper the network, the harder it becomes for error signals.
In addition to being able to identify contents inside photos, Candela stated that Facebook has a object-recognition prototype that works similarly but can also be applied to video.
Previously, Facebook has said that it was working on something to improve its photo-recognition technology. By launching the feature, Facebook is joining some others in the competition, like Google and Apple, that a;ready have the feature for users to search photos by their content.