As we’ve been extending the API for our site monitoring and diagnostic tool, Where’s it Up, one of my key concerns was the single point of failure our Washington server represented. Luckily, MongoDB was explicitly designed to make this sort of thing easy.
For Where’s it Up, we’re planning on launching with two main servers in different data centres, one in Washington, the other in San Antonio. We’ve considered a few different approaches on the web side, and have decided initially to mark our Washington server as primary on our load balancer, and only fall back on San Antonio as secondary should Washington fail.
Having set up multi-server configurations for databases before, I initially configured both systems in a Master - Slave relationship. Marginally more reading revealed that what I really wanted was a replica set. Within a replica set, an election will be called when the primary fails, and a secondary will be nominated as the new primary. The client library (in our case PHP) should handle this automatically, with only a brief interruption.
To effectively handle nominations, we’d need a third voter; I decided to place it in Montréal. This server would serve as an additional vote and as our backup slave. That voter is required because the systems seek a majority of servers to vote during an election; without a majority, they're unsure if the other machines are actually down, or simply on a segregated network (and possibly nominating their own master). With only two servers, the lone remaining server would be unable to self-nominate after the first crashed.
I also initially configured a fourth voting only server, an arbiter, in Washington after reading through the MongoDB documentation. Luckily the sage Sean Coates explained why this was a bad idea, and I removed it promptly.
Our configuration now stands at: wondernetwork:PRIMARY> rs.config(){ "_id" : "wondernetwork", "version" : 49449, "members" : [ { "_id" : 0, "host" : "washington.mongo:12345", "priority" : 10 }, { "_id" : 1, "host" : "montreal.mongo:12345", "priority" : 0 }, { "_id" : 2, "host" : "sanantonio.mongo:12345", "priority" : 5 } ] }
Under this configuration, Washington should win any elections (with its higher priority) it’s present for; failing that, San Antonio should win. Montréal will never become primary due to its priority of zero. I could have simply used priorities of 0, 1, and 2, but the old days' programming with line numbers have taught me to leave gaps for future expansion.
I’ve configured several different RDBMS’s to include a Master - Slave relationship. I’ve found MongoDB to be at least an order of magnitude easier to work with in this regard, even my first time through.
