Facebook, Google, LinkedIn, Twitter Team Up On WebScaleSQL To Tackle MySQL Challenges


WebScaleSQLBanner650Facebook, Google, LinkedIn, and Twitter face similar challenges in working with open-source database MySQL, and Thursday, the four companies announced that they joined forces to form WebScaleSQL, aimed at sharing a common set of changes to the upstream MySQL branch via open source.

Highlights follow from the announcement of WebScaleSQL in a post on the Facebook engineering blog by Software Engineer Steaphan Greene:

Together, we’re working to share a common base of code changes to the upstream MySQL branch that we can all use and that will be made available via open source. This collaboration will expand on existing efforts by the MySQL community, and we will continue to track the upstream branch that is the latest, production-ready release (currently MySQL 5.6).

Our goal in launching WebScaleSQL is to enable the scale-oriented members of the MySQL community to work more closely together in order to prioritize the aspects that are most important to us. We aim to create a more integrated system of knowledge-sharing to help companies leverage the great features already found in MySQL 5.6, while building and adding more features that are specific to deployments in large scale environments. In the past few months, engineers from all four companies have contributed code and provided feedback to each other to develop a new, more unified, and more collaborative branch of MySQL.

We want WebScaleSQL to be able to collaborate effectively and to move fast. To that end, we have set up a system for collaborating, reviewing code, and reporting bugs. For example, to introduce a code change, a WebScaleSQL engineer can propose a change. Then a WebScaleSQL engineer from another company will review the code and provide feedback. If both engineers agree that the change makes sense and is functional, it will be pushed into the WebScaleSQL branch for everyone to use. Beyond this, each organization may further customize WebScaleSQL to suit its own needs, just as we all do today.

After these initial accomplishments, we’ve started work on a number of other improvements to upstream MySQL. A few activities that Facebook’s WebScaleSQL team is currently working on:

  • Contributing an asynchronous MySQL client (links here and here) which means that while querying MySQL, we don’t have to wait to connect, send, or retrieve. This non-blocking client is currently being code-reviewed by the other WebScaleSQL teams, after being used in production at Facebook for many months.
  • Preparing to move Facebook’s production-tested versions of table, user, and compression statistics into WebScaleSQL.
  • Preparing to push the remaining components of Facebook’s current production-tested version of compression that were not already included in MySQL 5.6 into WebScaleSQL.
  • Adding the Logical Read-Ahead mechanism that we have proven in production to achieve large, quantifiable speed improvements (up to 10x) to full table scans, such as nightly logical backups.

We will keep all our WebScaleSQL work open, to create a useful branch for others within the MySQL community who are focused on scale deployments. We’ll continue to follow the most up-to-date upstream version of MySQL. As long as the MySQL community releases continue, we are committed to remaining a branch — and not a fork — of MySQL.

We’re excited to expand our existing work on WebScaleSQL, and we think that this collaboration represents an opportunity for the scale-oriented members of the MySQL community to work together in a more efficient and transparent way that will benefit us all.