Twitter has announced that it is open sourcing the work it has done to improve MySQL in its production systems. The microblogging and social networking platform is a major user of MySQL.
Twitter has open-sourced some of the numerous tweaks it has made to MySQL, the database technology that the microblogging site uses to store most of the data generated by its users.
The announcement was made by the Twitter developers Jeremy Cole and Davi Arnaut on the Twitter engineering blog on behalf of the Twitter DBA and DB development team.
MySQL is the persistent storage technology behind most Twitter data: the timeline, user data, interest graph and the tweets themselves. Due to the mass amount of data, Twitter push MySQL a lot further than most companies. MySQL, as an open source software, gave the ability to Twitter for changing it to suit its needs. Twitter have decided to open source its MySQL work on GitHub under the BSD New license. Some of the work done includes:
- Add additional status variables, particularly from the internals of MyISAM.
- Optimize memory allocation on large NUMA systems: Allocate MyISAM's buffer pool fully on startup, fail fast if memory is not available.
- Reduce unnecessary work through improved server-side statement timeout support.
- Export and restore MyISAM buffer pool in using a safe and lightweight method.
- Optimize MySQL for SSD-based machines, including page-flushing behavior and reduction in writes to disk to improve lifespan.
The initial publication and announcement was not meant to be groundbreaking - Twitter is setting up some groundwork to be able to collaborate publicly. It comes with an initial set of changes Twitter made to support the production workloads.
Twitter's modifications have been published on GitHub under a New BSD Licence and documented; the change history is also available. Twitter is looking forward to share the work with upstream and other downstream MySQL vendors, with a goal to improve the MySQL community.
Twitter plans to share more about its MySQL usage on Thursday, April 12th, 2012, at the Percona Live MySQL Conference and Expo being held in Santa Clara, California. The company plans to discuss Gizzard -- the sharing and replication framework built on top of MySQL.