Where’s it Up?

WonderProxy is proud to present Where’s it Up?, a new tool to help system administrators determine whether or not their site is up from around the world. The tool accepts a URL, and allows you to select global locations. It then attempts to connect to the given server and issue a HEAD request from the global locations you selected, and reports the results.

Key Features

Servers around the world

Local DNS resolution on each server
Reports IP and timing information
Follows a reasonable number of redirects

Key Technology

Curl

PHP with pecl_http

Memcached

Gearman

Supervisord

How it Works

Building a reasonably robust application was trivial, thanks to being able to leverage the great technology built by others. When a request is received, an intermediate script does some basic checking, then passes off a number of jobs to Gearman (one per location requested), the job ID is handed back to the user and results will be displayed via ajax. In the background Supervisord keeps five workers running all the time, they basically spin waiting for new work. When a job arrives they first check to see if there’s a recently cached result in memcached. Failing that an SSH tunnel is used to connect to the requested server, the request is issued with curl, and the results parsed. This result information is pushed into memcached.

On the client side, requests are made back to the server including the job id, and already obtained results. New results are handed off as they become available.

The system also uses memcached to throttle incoming requests from a given IP based on the example given by Simon Willison. In this case I’m actually using Swatch Beat time to manage the count; the edge case over midnight is quite easy.

Gearman is helpful in this case for three reasons: it provides great separation between the client requesting the work, and the workers actually completing it. It also trivially allows for multiple workers to be used in the resolution of requests. Finally, it provides a very effective work queue; when the site is under heavy load it will just take a bit longer to resolve requests. A small side benefit is the ability to pull down all the workers, update the worker script, then restart them, all without affecting the jobs in queue (except for the momentary delay).

Memcached is helpful as it’s a quick place to store the values that handles expiry for me automagically. In this particular case, as there’s a single server involved, something like APC would provider a thinner solution, however I have a lot of experience building apps with Memcached as a storage engine, and not much using APC.

Supervisord is helpful as it keeps the workers running, starts them up after they crash (hasn’t happened yet) and on machine boot. The dæmonize your PHP post by Sean Coates was incredibly helpful in getting this going.

SSH is helpful as it allows us to display accurate timing information from that machine to the requested location. We could use our own proxies directly, but then we're dealing with the time to connect from Washington -> Sydney -> Your website. That extra bit of math in the middle is only moderately tricky, but more importantly: inconsistent. The tunnel lets us execute the commands directly on the remote server.

Why we built it

We've used tools in the past that allow us to quickly check if a site is up or down. They're nifty, but these days many sites are using anycast DNS to publish different IPs globally, with multiple servers and multiple data centers. Simple tools simply aren't able to adequately express whether or not a site is up, or down. By leveraging the network of servers we already had, building a more advanced and complete tool was easy.

XHGui Improvements
I merged my Highcharts branch into master today, including a bunch of improvements to the GUI for XHProf including:

Using Highcharts for graphing URLs over time, which allows for multiple axis per graph

Adding a pie chart to the individual run page to show where time was spent on page load

Ability to merge various calls for display within the pie chart (e.g. mysql_* into mysql)

Switched to the javascript tablesorter for the single run results

Filter to allow you to view results only from a specific server or domain

Highcharts is a commercial library in Javascript for graphing, it's really easy to use and offers more features than the Google libraries used previously. I've purchased a license for the application so that anyone using this gui, regardless of the type of application they're profiling, can do so. The Highcharts integration was done by my colleague Graham Slater.

I really didn't consider the branch "done", the diff page still needs the pie charts, and to be switched over to the new table format. However, the license file received some serious updates to accurately represent the state of the code base and I considered making that file accurate to be a high priority.

Screenshots

As always, the code is available at GitHub

The accountability problem - Concluded

Last week I blogged about the accountability problem: how to handle tracking who made what change, and providing the ability to revert it.
Thanks to some great comments, and offline discussions, I’m seeing a few options, and have changed my mind on my desired approach. There's also some great technical reading on the subject: Slowly changing dimension

Option 1: Store history in the table

This was the approach I considered initially. Storing each revision of a record in the table, with an active bit set to indicate which version is active. More details in the original post. This approach is simple: it doesn’t require more tables, queries are still pretty simple, and with a combination of timestamps it’s easy to track down associated records that were changed.

The downsides are pretty clear, and somewhat damaging. The number of rows in the most commonly hit tables will grow much faster than the table overall, which is sub-optimal. It can also complicate issues like reporting and the like. In the case of highly normalized designs this design will likely break down, as you’ll need to match revisions across multiple tables on every query.
classes

id version active session title start_date end_date timestamp login_id

1 1 0 Spring Dancing 101 May 1, 2009 May 28, 2009 1274467708 23

1 2 1 Spring Dancing 101 May 1, 2010 May 28, 2010 1274467709 23

2 1 1 Summer Dancing 102 June 1, 2010 July 15, 2010 1274467710 27

3 1 1 Fall Dancing 101 September 1, 2010 September 28, 2010 1274467808 35

I would have used this approach in a smaller project. The system I’m replacing has been running for ten years, I have to plan for this lasting that long with only minimal maintenance. I’d rather not watch those row counts become 3-5x what they should be.

Option 2: Store histories in an external table

Rather than storing the histories in the regular table (say classes) creating a duplicate table (classes_history) and storing the revisions there. This helps trim the size of your most heavily used tables, and keeps things relatively simple.

While trimming the row count, it does double the table count, and still fails to handle complicated objects, or normalized table structures where a single datum requires multiple tables.
classes

id session title start_date end_date

1 Spring Dancing 101 May 1, 2010 May 28, 2010

2 Summer Dancing 102 June 1, 2010 July 15, 2010

3 Fall Dancing 101 September 1, 2010 September 28, 2010

classes_history

id version active session title start_date end_date timestamp login_id

1 1 0 Spring Dancing 101 May 1, 2009 May 28, 2009 1274467708 23

I’ll be using this method for my upcoming project.

Option 3: Store serialized information

Rather than storing rows when information is modified, the entire encapsulating object is serialized and stored. This can be done either with one table per object type, or in one large history table.

The advantage here is that each revision stands alone and complete. Composite records are stored complete, and in a single location. This makes obtaining an accurate history relatively easy. On the downside, since information is serialized it’s hard to locate specific pieces of information via sql. This approach seems perfect for complex objects where a single composite record spans many tables, but a little overkill for this specific case.
classes

id version active session title start_date end_date timestamp login_id

1 2 1 Spring Dancing 101 May 1, 2010 May 28, 2010 1274467709 23

2 1 1 Summer Dancing 102 June 1, 2010 July 15, 2010 1274467710 27

3 1 1 Fall Dancing 101 September 1, 2010 September 28, 2010 1274467808 35

history

id version table data

1 1 classes a:8:{s:2:"id";i:1;s:7:"version";i:1;s:7:"session";s:6:"spring";s:5:"title";s:11:"Dancing 101";s:10:"start_date";s:11:"May 1, 2009";s:8:"end_date";s:12:"May 28, 2009";s:9:"timestamp";s:10:"1274467708";s:8:"login_id";i:23;}

I'll consider this for larger projects with complex objects.

Other points to ponder:

How long will you store histories?

Forever (∞)

Some fixed period of time (30 days)

Application specific (e.g. storing change to a class for three months after it ends)

Storage engine for history

MyISAM

Archive

Backup strategies

History tables are all appends

Thanks to everyone who commented on the last post.

The accountability problem

I'm working on a volunteer side project for a local dance studio, the website will handle the standard sort of things: creating classes, registering students, matching leads to follows, etc.

We would like to have some level of accountability when various records (user, class, etc) are modified. That way if problems arise later, we can see who did what, and when. If needs be, even roll the change back.

This must be a common problem, how have you solved it?

My (proposed) solution is something like this: Take an ordinary table like class:

id session title start_date end_date

1 Spring Dancing 101 May 1, 2010 May 28, 2010

2 Summer Dancing 102 June 1, 2010 July 15, 2010

3 Fall Dancing 101 September 1, 2010 September 28, 2010

And add a few columns to for change management

id version active session title start_date end_date timestamp login_id

1 1 0 Spring Dancing 101 May 1, 2009 May 28, 2009 1274467708 23

1 2 1 Spring Dancing 101 May 1, 2010 May 28, 2010 1274467709 23

2 1 1 Summer Dancing 102 June 1, 2010 July 15, 2010 1274467710 27

3 1 1 Fall Dancing 101 September 1, 2010 September 28, 2010 1274467808 35

The list of classes is obtained by selecting where the active bit is set. Administrators will be able to look at the list of changes and see who made them, and when. I’m tracking login id rather than simply user id so information like IP, and timestamp can be associated with the login, rather than simply who. In the case of reverting a change, the desired version will be duplicated into a new row, and have the active bit set. In reality active will always be the highest revision number (and latest date) but the duplication makes queries easier.

I think there’s a lot of power in this approach, but it is a bit complicated. Thoughts?

A GUI for XHProf

Facebook was kind enough to open source the XHProf extension last year, but it flew under my radar until I saw a presentation including it earlier this year when they showed off HipHop. XHProf provides profiling information about your application, while being lightweight enough to run on a production server (against a percentage of requests). Once we got it installed we ran into a few limitations with the existing GUI. With the help of Graham Slater (one of our front end designers) I hacked on the Facebook code to come up with our own flavor. Let me stress the word “hack." I was given some really tough deadlines for this project so functionality was literally hacked into the existing codebase.

Key Features:

Nice interface

MySQL backend for data storage

Stores key values such as Peak Memory Usage, Wall Time, and CPU Ticks in the database directly for easy lookup.

Stores Get, Post, and Cookie data for easy comparison (so you can determine why a run hit that execution path)

Display includes easy lookup for worst runs, most recent, etc.

Support for profiling a percentage of requests, or on demand

Support for not including a link to profile requests on certain files (like xml documents, images, etc).

Uses Google Data Visualization API to graph runs over time

Concept of “Similar” urls, to handle ?story=23 and ?story=24 having different URLs, but similar to identical code execution paths

Screenshots:

Download:

Please download the source code from GitHub

Warnings:

Seriously, we hacked this up! Please let me know if you encounter any bugs, there's probably a bunch.

Memory usage in PHP

A colleague called me over today for some help with a memory usage issue in his PHP script. The script was performing the basic (but critical) task of importing data, pulling it in from MySQL in large chunks, then exporting it elsewhere. He was receiving the wonderful "Fatal error: Allowed memory size of XXXX bytes exhausted (tried to allocate YY bytes)" error message.

The code was following a basic flow:
10 Get stuff from the database (large array of arrays)
20 Iterate over it, doing stuff
30 Goto 10

I fixed the memory usage exceeded problem with an unset(), at the end of a loop.

Take a look at this sample program: loopOverStuff(); function loopOverStuff() { $var = null; for($i = 0; $i < 10; $i++) { $var = getData(); //Do stuff } } function getData() { $a = str_repeat("This string is exactly 40 characters long", 20000); return $a; }

The important thing to realize is that PHP will end up needing around twice as much memory as getData() takes. The problem is the line $var = getData(). The first time it is called $var is incredibly small, it's clobbered and the return value of getData()is assigned to it. The second time through the loop $var still holds the value from the previous iteration, so while getData() is executing you're maintaining the original data (in $var), and a whole new set (being built in getData()).

Fixing this is incredibly easy: function loopOverStuff() { $var = null; for($i = 0; $i < 10; $i++) { $var = stealMemories(); //Do Stuff unset($var); } }
This way we avoid the duplication in memory of those values on that line. To see this happen in more detail take a look at this sample script with ouptut: memory-usage-example.php.

This isn't critical, except when it is. Once loopOverStuff() completes, and the function ends. The memory is released back to the rest of PHP automatically. You'll only run into problems where Other Stuff + (2 * memory needed in loop) > Memory Limit. There are better architectures available to avoid the issue entirely (like not storing everything in the array, just to iterate over it later) but they're an issue for a different post.

For a very simple base case demonstration of the issue take a look at the simple example.

Search without the Database
I gave my second talk at #ConFoo today, this one was on Searching without the Database. This talk was based on a situation at work where I replaced a MySQL search solution with a Sphinx + Memcached solution for higher performance. If you're interested, here's the slides: Search without the DB - ConFoo 2010.pdf.

If you attended my talk, you can rate it on the Joind.in Page

ConFoo - PHP in the Enterprise
I just finished my PHP in the Enterprise talk at ConFoo, Slides are available here: PHP in the Enterprise - ConFoo March 2010

A Customer found a Bug - Redux
A little over a week ago I posted about a customer discovering a bug on the WonderProxy website, and how I handled the situation. I wanted to speak to that once more, in part as a follow up based on feedback I received, and in part about why I didn’t panic that a customer had found a rather critical problem in my website.

First, the follow up.
The overwhelming opinion is that I did okay in handling the issue, but I could have done better. Namely, I should have emailed the customer as soon as I confirmed the issue. That way the customer would be in the loop throughout the entire process. The fact that resolving the issue only took a few minutes was a mitigating factor, but really I failed to keep the customer in the loop, and that earlier notification could have allowed them to better plan use of my service.

In the future, I’ll behave differently. I have no intention of setting up auto-responders, but I can, and will, do better in terms of keeping customers in the loop.

Second, not Panicking.
Customer Service is a big topic, there’s a tonne of research out there on customer satisfaction, their interaction with customer service, etc. The body of the research shows that customers who have problems that are resolved to their satisfaction are on the whole happier than customers who had no problems at all. Keeping that in mind, I didn’t panic. This was an opportunity to have a very happy customer, rather than a crisis.

The research I read was admittedly based on hotels (with an interesting anecdote about discovering a customer who over something like 207 nights of stay had not yet paid for one, as every time they complained about something and the night was comp’d). But the data resounds well with me. We’ve all got a plethora of customer service horror stories, and precious few great ones. As a result, those great customer service experiences are carried with us, and often shared with friends and colleagues.

While I don’t intend to go as far as to introduce bugs into the system simply so I can fix them in a fantastic manner, I will embrace every opportunity I get to show a customer great service.

X-Host-IP
When you discover a problem on you website, and you run multiple web servers, one of the first questions you ask is "Which server am I on" (often followed immediately by: "Does it happen on all of them").

This presented an unfortunate problem for WonderProxy customers, as when they connect to a server through their proxy, they are no longer able to determine which webserver provided a resource. That information is hidden because their connections take place through the proxy.

This is why we're happy to announce the introduction of a new HTTP header being served through all 18 of our proxies: X-Host-IP. This header is injected by our proxy software when handing data back to the client. This header includes the IP address of the machine that provided the resource, allowing clients to obtain this information through a variety of methods like Live HTTP Headers, Firebug, or Dragonfly.

This is the first of two steps we’re taking to better inform, and empower our clients when it comes to servers. We’re planning to roll out a new feature allowing customers to inject a hosts file[1] onto the proxy server of their choice to allow server selection through a web interface. This should be completed in the next few weeks.

[1] A hosts file (/etc/hosts on most linux machines, and \system32\drivers\etc\hosts.txt on Windows boxes) allows users to hard code hostname and IP combinations, often used for testing and development.

A Customer found a Bug
So this morning I woke up to an email (well, several, but only one is relevant) from a WonderProxy customer. They had discovered a bug in the website. A rather serious bug mind you, as it stopped them from adding servers to their account, and thus prevented them from actually using the service.

Here’s how I approached the problem:

Reproduce the issue
I logged into the website as the customer and attempted to reproduce the issue, no problems. I tried again on the development version of the website and reproduced it there as well.

Estimate bug fix
I took a look at the code and estimated that I could resolve the bug in under ten minutes. The code base for the WonderProxy website is rather small, and bugs there can’t affect customer usage (unless they break the DB) so I wasn’t too concerned about accidentally taking something down. The short time frame for release also let me fix the bug before contacting the customer.

Fix the bug
Found the bug, fixed it. Tested it out in development, checked other accounts to look for unrelated breaks. Finally I rolled it out from development to production and checked again.

Contact Customer
Replying to the customer I indicated that problem was indeed on our end (the customer wasn’t using the site wrong or anything), and that it had been fixed. Additionally I had updated the billing period to reflect the fact that the account hadn’t been usable up until the fix was applied. As a token of appreciation I upgraded the account slightly.

Overall, I found the process rather pain free. When I look at it from a customer’s prospective, I also feel like it was a reasonable problem resolution.

I’d like to delve deeper into how this problem made it onto my production website, but that’s a separate post. For now I’m really just looking at the customer service side, there was a total of 67 minutes between the initial contact (on the part of the customer) and my reply with the fix in place.

Thoughts? How did I do?

WonderProxy - Server Status
A few weeks ago I launched the WonderProxy website, one of the first decisions I made when working on it was the network status bar shown on the left side. That's a nearly live showing of the state of all the servers in the network (as of when the page loads).

What I Did
The box with Nagios running on it runs real tests against all the proxies in the network on a regular basis. I do mean real tests: it uses curl to make a request against a script on a different web server, which checks the headers to ensure that the correct headers are being used. This checks: the proxy is running, the proxy is listening on the correct port, the proxy is able to authenticate users, the proxy is able to retrieve remote documents and return them. Every minute the Nagios box collates the data for all the servers and creates a .json document. That document is then pushed to the web serving machine, ready for consumption by obliging clients.

Note, I've said "what I did", in reality the JavaScript ninjary was done by Christine, and the curl ninjary was done by my obliging sys-admin Will

Why I Did It
Presently, I deal with upwards of twenty hosts between professional and personal projects, that's a lot. I've probably dealt with another ten that I’ve terminated relationships over my professional career. All in all, I’m sick of the hoops that they make me jump through to find out what's going wrong.
Here's a typical outage for me:
0 minutes - Nagios notifies me that something is wrong
+2 - I've logged into nagios, and attempted to log into the affected system for more information, confirming the outage
+9 - I'm on the website for the host of the affected system, finding absolutely no helpful information.
+10 - I've started trying to figure out what my login credentials are for the affected system's hosting provider.
+11 - I've logged in, found no more information
+12 - File a ticket with the host
...
+35 - Nagios informs me that the service is back up
+90 - Host replies with some generic "something went wrong on our end, we fixed it, we may credit your account with $0.12 later" message.

I argue this is patently absurd. Some companies put on a front about being better about this sort of thing, but then an outage occurs and the one girl/guy with access to the twitter account is on vacation, or asleep, or at the movies so nothing happens.

When my box goes down its either my fault, in which case I need to start working my rear end off to bring it back up, or the host is experiencing a real outage, in which case I should just sit on my hands and wait. In that typical listing I've indicated above I don’t know which situation I’m in until 90 minutes on, though I can start poking at logs 35 minutes in.

What I Hope This Means

If you're a WonderProxy customer running tests and something breaks, you can log onto the WonderProxy website and know immediately if the proxy is broken, or if you should be looking harder at your own infrastructure.

Thoughts? Does your host do better?

Holiday Swap
This year I took place in the Book Blogger Holiday Swap which was pretty awesome. My giftee is way more on the ball than I am (also: a cheater, opening things christmas eve!) and blogged already.

I was lucky enough to receive a great book, some chocolate, and a book mark!

As you can see, I've already ripped into the chocolate! I think I'll start into the book after I finish re-reading one of my William Gibson books. I think it will be great for my sporadic reading during my daily commutes!

Thanks Carrie K

Functional URLs - More than Developer Vanity
I read David Sklar's post to the PHP Advent calendar this morning, and as the title rather clearly points out, I disagree with his conclusion.

Before I go any further I'd like to clarify something in the title, namely the word "functional". I'm using this meaning from dictionary.reference.com:

having or serving a utilitarian purpose; capable of serving the purpose for which it was designed: functional architecture; a chair that is functional as well as decorative.

If you define a URL's purpose as simply being an instruction to a browser or other application, then sure a URL such as this one qualifies:

http://www.realtor.ca/map.aspx?&vs=VEResidential&beds=0-0&baths=0-0&minp=0&maxp=0&area=Montreal%2c+Quebec&trt=2#acr:false;ac:false;baths:0-0;beds:0-0;fp:false;gar:false;pmin:175000;pmax:200000;rmin:0;rmax:0;openh:false;
pool:false;stories:0-0;buildingstyle:;buildingtypeid:;viewtypeid:;waterfront:false;forsale:true;forrent:false;orderBy:A;sortBy:1;
LisStartDate:;mapZ:12;page:1;mapC:45.49792488715173, -73.56531143188478;curView:0;

That said, the only purpose of a URL isn't to give an instruction to a browser or other application. It's an indicator to the user, telling them what the page is about, and one they use quite a bit[1]. It's an address that does still need to be typed on occasion, or at the very least copy/pasted into a variety of mediums. It should be a destructible resource that users can use to navigate the site and obtain what they're looking for.

Compare:

store.example.com/plv2/product.php?refid=4AF82BC932

and

store.example.com/toys/dolls/barbie

In the first case you're stuck, there's no options to you for navigating the site beyond the on screen options. In the latter case you can move from looking at Barbie, to looking at all dolls, to looking at all toys, to looking at the entire product line of the store rather easily. I will concede that not all users will think to navigate in that manner, but some will, and they have the option.

In conclusion: URLs are more than some text an end user clicks on, and having good ones has more worth than letting the developer look cool in front of their peers: It's an actively used resource. Having clear, functional URLs is a worthwhile goal for any web property.

[1] Jakob Nielsen's Alertbox, March 21, 1999: 2007 update at the bottom.

p.s. I may receive a beating shortly for the rather ugly URL this blog post possess while standing on a soap box shouting for better URLs. Software I build myself does include more functional addressing, compare wonderproxy.com/servers/Miami/ and wonderproxy.com/servers/ (I'm working on the case sensitivity thing).

noreply@ Needs to Die
It's 2009, there's absolutely no reason to make it hard for your customers to talk to you. The technology exists now, and has for a while, to import whatever mail you receive into whatever ticketing system you're using. Sure, internally use whatever system you're happy with, there's lots of inefficiencies in trying to handle any volume of support through traditional mail clients. Your clients on the other hand, aren't wizards at whatever system you're using, but they do know how to use their email client.

When you send a client an email, monitor the address and import replies.

Yes, there's challenges, watching vacation auto-responders ("Hi, I'm on vacation! I'll come back on the 20th") and ticketing systems (“Thank’s for emailing Widget-Co, Your email will be answered soonish!”) get into a reply fight isn't pretty. Handling character sets, HTML formatted mail, etc. all present challenges. But these are manageable problems.

The only reason not to import email into your ticketing system is if you don't actually want to talk to your customers.