I'm amazed at how much WonderProxy has grown. I think we signed up our first client with only 6 servers; we're on 6 continents now, with 89 proxies, and 5 private machines dedicated to things like command & control or monitoring. Despite that, the company hasn't really grown; we've hired Allison to help out with copywriting & billing (which was a fantastic move on our part), but that's about it.

Over the past few weeks Will and I have mostly turned our attention inward, improving the way we monitor and configure our network. This was kicked off when we racked our first co-located box here in Toronto (with its additional layer of physical security):

With the additional redundancy it's currently providing, we're better configuring and monitoring puppet, which watches the rest of our servers. We're testing out a new setup for our web and database systems that provides better isolation and makes them easier to update. Despite feeling anxious about doing work that isn't specifically customer focused, I'm glad we're doing this as I'm expecting it to pay dividends in time by freeing us from some routine tasks.

While growing the network, Will and I have been relentless about reducing the things that require our time and effort but which don't improve things from our customers' point of view. Every minute we spend treading water is a minute we're not striving forward. Will led the charge customizing and automating our server configuration process, which had added complexity because each VPS we buy is a bit different, ranging from a bare bones Debian install, to an "everything including the kitchen sink" install that has little more than a run-level stopping it from launching gnome. From there, we've worked to recognize routine tasks that steal our time but do little to move us forward, and then automated them. Lots of little things have come together to help free us to focus on improving our products or writing new ones.

Things like:

  • Disabling customer accounts automatically on a charge back
  • Making it easier to send a user a free trial
  • Ensuring our messages get delivered into the inbox, not the spam folder, the first time.
  • Tweaking our billing code to recognize most of the common ways renewals come in to correctly attach them to the right account.
  • Handing billing (both accounts receivable and payable) to Allison, to free technical time for technical tasks (she's also much better at it than I was).
  • Documenting irregular, but not-yet-automated tasks so they can be completed quickly, with a minimum of research
  • Distributing our authentication system to remove a single point of failure and ease diagnostics when there's a problem
  • Automatically populating VPN accounts with all available servers

Negating a task (through automation, documentation, or otherwise) that takes just 5 minutes a day earns us back 2 hours of development time per month. It can be hard to justify spending an hour to "fix" something that only takes you 5 minutes, but with a return on investment of only two weeks, it would be foolish not to.

I'm also very picky when it comes to who mail goes to. I work relentlessly to reduce the number of recipients to any automated message. I no longer receive any messages from our hosting providers: they're directed to Will and Allison. Will deals with technical issues, and Allison pays the bills. Will doesn't get my MySQL slow query report, nor messages from our PayPal IPN script: they don't concern him. Every email you unsubscribe from is a gift of time to your future self.

We have a few triggers to discover things we need to work on:

  • Noting common support questions, and either adding documentation or tweaking our applications to avoid them. Populating VPN accounts was a good example of this: our VPN service is built on the same codebase as our Proxy stuff, so when it launched, users needed to add the servers to their account. This generated many support requests that were easily resolved by a few lines of code.
  • Regarding everything we've done several times with suspicion. I've added a user's IP to our white list a few times now: is it time to build an API so they can handle this themselves?
  • Regarding anything we've forgotten to do with suspicion: if it needs to be done that regularly, why does it still need a human?

I'm convinced that if we hadn't been working so hard to automate or remove such day-to-day tasks, we'd never have had time to write Natural Load Testing, Where's it Up, Where's it Fast, or our Global Ping Statistics.


Hi, I’m Paul Reinheimer, a developer working on the web.

I co-founded WonderProxy which provides access to over 200 proxies around the world to enable testing of geoip sensitive applications. We've since expanded to offer more granular tooling through Where's it Up

My hobbies are cycling, photography, travel, and engaging Allison Moore in intelligent discourse. I frequently write about PHP and other related technologies.

Search