The social giant Facebook is big. With more than 1.5 billion users and counting, the company wants to make sure that Facebook runs perfectly on every device, every time and everywhere.
In line with its conquest to connect the whole world, the company's Oregon data center has a Mobile Device Lab which uses real devices to monitor the social media's performance, real-time.
Scaling to be as large as Facebook is not an easy task. But the company that was born out of a dorm room by founder and CEO Mark Zuckerberg, gets to what it is today with many things involved with tons of research and development.
And to connect the whole world, there are unique problems that the company should encounter, and it is its aim to bring the same Facebook experience to everyone, no matter where they are.
Facebook's production engineer Antoine Reversat explained that the company is not just thinking about its existing 1.5 billion users, who are increasingly logging in via their mobile devices. It's also thinking about the billions of people around the planet coming online for the first time.
So from its website, mobile website to apps (Facebook app, Messenger, and Instagram), Facebook wants its character to be prominent and distinguishable, if it's still isn't that is. And the Mobile Device Lab is there to known how Facebook work on various devices at anytime of the day.
The Mobile Devices Lab takes form of a room that is essentially an extension of the Gondola. It's a place for Facebook to keep all devices on since putting them all at its Menlo Park headquarter wasn't going to work.
The room supporting several metal racks to hold a wide range of devices - from flagships to bargain handsets. Each rack can hold up 32 devices and has its own Wi-Fi access point for the devices. Those racks, which either have eight Mac Minis to run iOS devices, or four OCP Leopard serves to run Android devices. They racks are all custom built and designed to function in an electromagnetic isolation chamber. Each Mac connects to four Apple devices, while each OCP server runs eight Androids.
"Our thinking was to get the emerging markets and get the regularly used phones here," said Reversat, explaining how his team running the mobile lab chooses which phones to test.
Phones on the racks are put in a slightly slanted manner so the camera can record what is happening on-screen. This allows the team to get a visual understanding of how an updated app is impacting a phone.
The entire room can hold 240 devices active at a given time. But in the days where there are tens of thousands of different mobile devices on the market, Facebook wants to run smoothly on all of them. So not stopping there, Facebook wants at least 2,000 devices up and running.
And moving on from Oregon, Facebook also started to roll them out to other of its data centers.
What the Mobile Device Lab does is practically simple: run Facebook and keep it running all the time. But Facebook wants to know how buildings can affect connection to certain devices, how Wi-Fi signals and how to power things along.
And that include how phones react on tests, apps install and reinstall. Facebook also tests previous versions of Facebook, Messenger and Instagram. The company is specifically checking for any regression in performance or power.
So the testings are fairly continuous and detailed process.
As for some of the tests, Facebook puts Wi-Fi limited to the devices. According to the company, Wi-Fi signal from one rack would interfere with the signal from the next. so the phones have their own rack outfitted with foam and copper tape as insulation on the inside so they can function as an electronic isolation chamber. The physical wireless access point issues are also concerned, and they're made to only four-foot distance from the devices (Facebook said that this should provide "sufficient attenuation of the signal").
Facebook is one of the internet companies that constantly updating itself. From releasing new standout features to minor bug fixes. The core of Facebook changes about everyday with small tweaks and fixes constantly updating. Its apps are updated bimonthly.
In order to manage all those, Facebook uses a package manager and a configuration manager on its server. Called the "Chef", it helps the team at Facebook in assuring that configurations are properly configured on each server, and data to the correct services. It handles many things from notifications to configuration changes.
What the Chef does, is to make sure that uniformity is present across all of its severs, and to also help the team to determine any critical issues. Handling about anything on Facebook's server, the Chef can also report if anything goes wrong. But when the Chef says nothing in particular but there are performance issues, the team knows that there are complex problems that need to be solved.
With the many user usage possible, there are many possibilities how people access Facebook. With the many devices available on the market, Facebook wants to work with every combination of operating system and network connection. Given that the social network is constantly pushing out new code, the company is working hard to meet both the demand with the supply.
The Chef here is to keep the phones in a consistent state through tests.
Facebook is a titan on the web, and it doesn't want anyone to say that Facebook is "acting weird", or maybe worse, "Facebook doesn't work well on my phone." So the Mobile Device Lab is essentially a place where Facebook can do on-device testing withing arm's reach.
Looking forward, Antoine Reversat said the team wants to double the density of the racks, enabling each one to hold 64 phones. This will rise some new challenges. For instance, how to maintain the four-foot air gap between phones and their access points to ensure signal efficiency, and how to hold more larger phones on the rack without freeing up new space. The team also wants to see the Chef to support iOS devices better.
Eventually, Facebook wants to open source both the hardware design of its device testing rack, and also the Chef recipes it uses to control the phones.
The 147,000 square foot Prineville, Oregon facility itself is one of the most energy-efficient data center in the industry, according to Facebook. And its presence provides the social media network with more headroom for its fast-growing server and storage infrastructure.
"After a rigorous review process of sites across the West Coast, Facebook concluded that Prineville offered the best package of resources - including a suitable climate for environmental cooling, renewable power resources, available land, talented regional workforce and supportive business environment," said Tom Furlong, Director of Site Operations for Facebook.
"We have come a long way from our roots in a Harvard dorm room, when Facebook was only available at some colleges and run on a single server," said Jonathan Heiliger, Facebook's VP of technical operations, in a blog post. "When Facebook first began with a small group of people using it and no photos or videos to display, the entire service could run on a single server."
Now with 1.5 billion and counting, almost every active users are posting new images and/or videos everyday. This forces Facebook to continually invest in its infrastructure.
"...We must constantly scale our technical infrastructure to meet the demand and deliver you a fast, reliable experience," said Heiliger. "An important step along the way is to build a custom data center so that we can design it to meet our unique needs."