As Facebook continues to reign on the top of the food chain of social media, it's obvious that it needs to support as many languages as possible.
But the thing is that, the way it rolls out features has been quite complicated due to the fact that Facebook has supported more than 100 languages. So when creating text boxes where users can type on them aren't difficult, but Artificial Intelligence (AI) that drives many parts of Facebook, poses a growing challenge.
Facebook needs to ensure than its systems and AI can understand what users are wanting.
As a solution, the company’s Applied Machine Learning team has been working on a technology called multilingual embeddings, which according to the company, could significantly improve the speed at which its natural language processing technology is able to operate across the many languages it supports.
In early tests, Facebook an increased speed to up to 20 to 30 times faster, if compared to previous methods.
According to Facebook, translating new languages can take as long as building a new application. Using the technology, Facebook uses language vectors to group words with the same meaning together.
This word embedding essentially creates vectors that allows text classifiers to approach languages in a more context-driven way. By highlighting the interrelatedness of words, the system should be able to derive shared meanings to better understand the intention.
Previously Facebook essentially translated foreign languages to English and then running English classifiers on them, but this has been a rough solution due to translation errors.
By mapping multiple languages onto similar word vectors, Facebook’s method "can train on one or more languages, and learn a classifier that works on languages you never saw in training."
With more than 100 languages it supports, it's a pretty intensive labor to gather all the training data for the classifiers. But with the technology, Facebook is hoping to work towards a more scalable approach, which should create a more efficient translation.
Not just reducing latency, the technology can also help Facebook's future features to reach more people quickly, ensuring consistency across services it offers across the globe.
“From the multilingual understanding perspective, I want everybody to use all the features that are deployed by Facebook in their own language,” said Facebook's head of translation Necip Fazil Ayan. "This should not be limited to a particular language, but we want to move to a world where all features are available everywhere, and can be used by everybody.”
Initially, the company's Applied Machine Learning team works on translating idioms and other cultural nuances that might not cross the language barrier easily.
Here, Facebook has made success in embedding German, French, and Spanish, and it’s working to bring the same embedding to the other 100 or so languages available to Facebook users.
Previously, the company’s Applied Machine Learning team that consists of 20 engineers, has developed the technology to detect content-policy violation, surface M suggestions in Messenger, as well as powering the Recommendations feature.