The cross-zone replication daemon is responsible for asynchronously replicating all block puts from one zone to the other. Key-value stores are fashionable and offer high performance, but databases are highly reliable and provide an expressive data model which has allowed us to easily expand our schema and functionality over time. We might build a more sophisticated system eventually, but we’re happy with this for now. We already had thousands of database nodes in service across the Dropbox stack, so this allowed us to leverage the operational competency we’ve built up around managing MySQL at scale. We’d originally planned on building a dedicated key-value store for this purpose but MySQL turned out to be more than capable. The Block Index is a giant sharded MySQL cluster, fronted by an RPC service layer, plus a lot of tooling for database operations and reliability. ( Our real schema is a little more complicated than this to support things like deletes, cross-zone replication, etc. Note: Internally we just call the system “MP” so that we don’t have to feel silly saying the word “Magic” all the time. Feel free to add feedback in the comments below we’ll do our best to delve further in future posts. Magic Pocket is based on a rather simple set of core protocols, but it’s also a big, complicated system, so we’ll necessarily need to gloss over some details. These files are split up into blocks, replicated for durability, and distributed across our infrastructure in multiple geographic regions. Magic Pocket is the system we use to store the file content. We’ve received a lot of positive feedback since announcing Magic Pocket, our in-house multi-exabyte storage system. We’re going to follow that announcement with a series of technical blog posts that offer a look behind the scenes at interesting aspects of the system, including our protection mechanisms, operational tooling, and innovations on the boundary between hardware and software. But first, we’ll need some context: in this post, we’ll give a high level architectural overview of Magic Pocket and the criteria it was designed to meet.Īs we explained in our introductory post, Dropbox stores two kinds of data: file content and metadata about files and users.

Dropbox support call number