How we organize our websites

nospam@example.com (Paul Reinheimer) — Wed, 15 May 2013 13:19:57 +0000

We recently migrated Where’s it Up to our fancy new hardware, it took a bit more effort than planned (some pains surrounding our use of MongoDB) but I’m incredibly happy with how things have ended up. As mentioned earlier we’ve purchased our own hardware, and have racked it with Peer 1 here in Toronto. We’ve installed a hypervisor, and are running different VMs for critical services: MySQL, Mongo, Web Production, Web Development, etc.

Our websites sit under /var/www, so Where’s it Up resides at /var/www/wheresitup.com/. Under that directory we have /noweb/apache/ which contains both wheresitup.com and dev.wheresitup.com, configuration files for apache. The entire /var/www/wheresitup.com directory tree resides nicely in our version control system. We hand off key configuration options to our websites through the use of Apache’s SetEnv, things like SetEnv mysql_host dev.mysql, these apache configuration options represent the only difference between the two code bases.

I’ve written or maintained code that implied the state (Dev/Production/Stage) based on the Host, directory, or other factors in the past. I much prefer grabbing an explicit constant. It feels cleaner, I don’t have to read up on which variables could have been manipulated by an attacker, and I can ask the exact question I want answered: Is this dev, rather than “is the url the one that means this is dev”.

This allows us to match our Development and Production virtual machines very closely, the only difference between the two is which apache configuration file is sym-linked under /etc/apache2/conf/sites-enabled. Clearly WebDev links to the dev.wheresitup.com file, and WebProd links to wheresitup.com. We actually cloned one machine to produce the other.

Keeping the configuration files so close also makes a lot of sense to me. If I’m adding a new constant on Dev, the immediate presence of Prod reminds me that I’ll need to add it there as well. Storing the entire site: PHP code, supporting apache configuration, etc, all in once place makes it easy to avoid forgetting anything (which is easy when it's a different file on a different server). The only exception is SSL certificates. We currently host a number of our projects with GitHub, and trust them as we might, we’re not willing to hand those to anyone else.

Buying a Zoo (server)

nospam@example.com (Paul Reinheimer) — Mon, 15 Apr 2013 12:25:33 +0000

Buying physical hardware was a new step for WonderProxy, it’s hard to say that we rushed into it, this being our third year, but it sort of feels that way. We’re operating around 100 servers around the world right now, but all of them are either virtual servers, or dedicated machines we’re renting from providers. Having a UPS guy drop off a rather large box one day was a big change.

Everything has worked out well, but there was a few steps that could have gone more smoothly, this post is half note to myself on what to do better next time, and half for you.

Buying Hardware

As far as I can tell the Dell & HP websites have been largely designed to be horrible, in hopes of routing you to a sales person. I tried to fight this, but it was pointless. Phone someone at one of those companies and save yourself several hours.

Watch for extras on the quote, your sales person will likely work your specifications, then insert the most expensive options around it, things like 24/7 hardware replacement with a 4hr SLA, fancy cable management systems, etc.

Your data centre will have power requirements. Your phone rep may be able to help you there, Dell’s UPS website is also capable of turning your server specifications into amperage.

Remote management cards are helpful, but you’ll either need to set up pass-through on the NIC (if your card supports it) or have multiple drops to reach it.

Check each component for compatibility with your operating system if it doesn’t ship installed. We’re using Debian, and had a mild panic attack before we found drivers for our Raid controller in Debian testing.

Hosting

Your hosting provider will sell you bandwidth as a 95th percentile. That means they’ll sample how much bandwidth you’re using on regular increments (say every 15 minutes), sort those results biggest to smallest, delete the top 5% then charge you the next one. Unless you’re buying a lot of bandwidth you’ll probably end up paying more here than you would on a dedicated box by the GB.

Hosting space comes in either U increments (1U, 2U, 4U, etc) or rack portions (full rack, half rack, quarter rack, eighth rack (octal). If you’re buying directly from a provider you’re likely going to need to over-buy if you’re only racking one server.

Providers also care about power usage they will likely tell you something like 8 AMPS. You’ll need to spec your server out appropriately.

The number of network cables and power ports inside your unit will also matter, there’s no point in having a redundant power supply if you’re only going to be able to plug one in.

You will need to plan your move in date, your provider may need a lot of paper work signed and then a few more days before this happens. Talk to your sales rep about dates, and SLA for setting up new space. It may be as long as a week between getting your paper work in order and being able to move in.

Find out how your server will be mounted, there appears to be both round and square holes. As we learned when we were four, you need to match the right peg to the right hole. If you’re renting a very small fraction of space (like an octal) you may not have any mounting brackets at all, instead just letting things rest on the sheet metal between clients.

Your sales guy may not manage things once you’ve signed up, you may be handed off to the network team. Try to keep track of who knows what, if you’re racking your server and have a problem sales guy can’t help (and probably doesn’t answer the phone after hours).

Visiting the Data Centre

You’ll need ID, your network guy should be able to describe the requirements

Depending on how many servers you’re bringing in, you may be able to use the front door, or the loading dock.

Ours had a nice man-trap on the way in, first door needed to close before the second would open

It will be loud in the server room, ear plugs would be prudent

A flash light might help see things, there’s decent lighting but you’ll likely have stuff on top and below you

There should be a monitor, keyboard, and mouse on a trolley somewhere for configuring things

There may not be wifi, or even 3g inside

Pre-configuring your IP details stuff would be prudent, there will not be DHCP

That in hand, hopefully your server buying and racking experience will go smoothly.

Where's it Up - Visualized

nospam@example.com (Paul Reinheimer) — Fri, 12 Apr 2013 23:04:20 +0000

Andrew Quarton developed a nifty little visualization built using the Where’s it UP API called GeoPing. Go take a look then come back.

Our technology stack for the API includes supervisor to run workers, and gearman to manage our job queue. We’re normally running 25 workers to manage the queue. Work tends to come in chunks, and that number of workers has been able to keep the queue minimal or at zero.

Since it’s such an nifty tool, it made the front page of Hacker News today, which led to a few problems on our end. The number of jobs launched for each person hitting the GeoPing tool was rather high, enough to fill all the current workers. When many people started hitting the GeoPing tool in rapid succession the queue built and built. At one point Gearman reported 13,000 jobs in the queue.

Noticing this I quickly changed the number of desired workers in supervisor from 25 to 100, than used /etc/init.d/supervisord restart to apply the changes. That didn’t seem to affect the queue, so I tried 250 workers, used restart to apply the changes once more, and watched. Then I noticed something the restart option wasn’t launching the extra workers I wanted. Running /etc/init.d/supervisord stop, then start did. Then the queue finally started to recover. I kept an eye on the queue with a quick and dirty shell command from stack overflow.
(echo status ; sleep 0.1) | netcat 127.0.0.1 4730 -w 1
From our side, I think a few things went wrong:

We didn’t have tooling in place to warn us when the queue broached reasonable limits

We hadn’t documented the proper way to increase workers (stop/start not restart)

Our graphing system seems to have a hard coded max value, hiding valuable data

Having either of those first two items in place would have allowed us to respond to the issue much more quickly.

We're working on them :)

Yanking Hard Drives

nospam@example.com (Paul Reinheimer) — Sun, 07 Apr 2013 21:35:39 +0000

I invested a portion of my day heading down to our data centre, and yanking a drive out of our newly racked server, and watching to see what would happen. The answer was: not much. The system kept merrily computing along, but it also failed to warn us that it had lost a drive. Some configuration tweaks later, I repeated the process, and my mailbox was quickly filled. Success.

I did this for two reasons: I wanted assurance that our raid controller had been properly configured, and that things would continue operating normally if we lost a drive. I also wanted to ensure that should we lose a drive we'd be notified, redundancy is worth little if actions aren't taken to correct problems that cause it to be lost. Today confirmed the first, and revealed problems in the second. Huge Success.

If you're going to perform the same task, a few pieces of advice:

Do it before there's critical production software running on it, at least the first time

Your raid controller will need to rebuild the drive after it's been pulled. This will take a while, and kills your redundancy (or a portion thereof) while it's happening. If you need to repeat the test pull the same drive each time to avoid real problems.

Work to ensure that problem notifications leave the affected system as quickly as possible. If the only copy of an alert sits on the box that's having issues, you may lose it as well.

Working hard to stop treading water

nospam@example.com (Paul Reinheimer) — Sun, 24 Mar 2013 14:13:13 +0000

I'm amazed at how much WonderProxy has grown. I think we signed up our first client with only 6 servers; we're on 6 continents now, with 89 proxies, and 5 private machines dedicated to things like command & control or monitoring. Despite that, the company hasn't really grown; we've hired Allison to help out with copywriting & billing (which was a fantastic move on our part), but that's about it.

Over the past few weeks Will and I have mostly turned our attention inward, improving the way we monitor and configure our network. This was kicked off when we racked our first co-located box here in Toronto (with its additional layer of physical security):

With the additional redundancy it's currently providing, we're better configuring and monitoring puppet, which watches the rest of our servers. We're testing out a new setup for our web and database systems that provides better isolation and makes them easier to update. Despite feeling anxious about doing work that isn't specifically customer focused, I'm glad we're doing this as I'm expecting it to pay dividends in time by freeing us from some routine tasks.

While growing the network, Will and I have been relentless about reducing the things that require our time and effort but which don't improve things from our customers' point of view. Every minute we spend treading water is a minute we're not striving forward. Will led the charge customizing and automating our server configuration process, which had added complexity because each VPS we buy is a bit different, ranging from a bare bones Debian install, to an "everything including the kitchen sink" install that has little more than a run-level stopping it from launching gnome. From there, we've worked to recognize routine tasks that steal our time but do little to move us forward, and then automated them. Lots of little things have come together to help free us to focus on improving our products or writing new ones.

Things like:

Disabling customer accounts automatically on a charge back

Making it easier to send a user a free trial

Ensuring our messages get delivered into the inbox, not the spam folder, the first time.

Tweaking our billing code to recognize most of the common ways renewals come in to correctly attach them to the right account.

Handing billing (both accounts receivable and payable) to Allison, to free technical time for technical tasks (she's also much better at it than I was).

Documenting irregular, but not-yet-automated tasks so they can be completed quickly, with a minimum of research

Distributing our authentication system to remove a single point of failure and ease diagnostics when there's a problem

Automatically populating VPN accounts with all available servers

Negating a task (through automation, documentation, or otherwise) that takes just 5 minutes a day earns us back 2 hours of development time per month. It can be hard to justify spending an hour to "fix" something that only takes you 5 minutes, but with a return on investment of only two weeks, it would be foolish not to.

I'm also very picky when it comes to who mail goes to. I work relentlessly to reduce the number of recipients to any automated message. I no longer receive any messages from our hosting providers: they're directed to Will and Allison. Will deals with technical issues, and Allison pays the bills. Will doesn't get my MySQL slow query report, nor messages from our PayPal IPN script: they don't concern him. Every email you unsubscribe from is a gift of time to your future self.

We have a few triggers to discover things we need to work on:

Noting common support questions, and either adding documentation or tweaking our applications to avoid them. Populating VPN accounts was a good example of this: our VPN service is built on the same codebase as our Proxy stuff, so when it launched, users needed to add the servers to their account. This generated many support requests that were easily resolved by a few lines of code.

Regarding everything we've done several times with suspicion. I've added a user's IP to our white list a few times now: is it time to build an API so they can handle this themselves?

Regarding anything we've forgotten to do with suspicion: if it needs to be done that regularly, why does it still need a human?

I'm convinced that if we hadn't been working so hard to automate or remove such day-to-day tasks, we'd never have had time to write Natural Load Testing, Where's it Up, Where's it Fast, or our Global Ping Statistics.

Power vs Complexity

nospam@example.com (Paul Reinheimer) — Thu, 14 Feb 2013 00:56:53 +0000

I ended up taking several months off development of Natural Load Testing. I needed to wait while my partner improved the software we actually use to apply the load (it’s ready now), then I was caught up in other projects like the Where’s it Up API, and Where’s it Fast. Coming back to it was fantastically revealing: it was very hard to use.

Looking back, I can see why we made many of the decisions that I now identify as making it hard to use. Most of them were about offering power or flexibility to our end users: I think creation of ‘tests’ and ‘test suites’ is a great example.

Users create tests, then put those tests into test suites: this allows for complex and powerful interactions where virtual users register for a site, then perform one of several different actions, before closing out. A sample load test might be Surf -> Register -> [Search to product OR Browse to product -> Add to shopping card -> Checkout. We can handle that, and I’m proud that we can handle that.

Unfortunately, it means that users getting started who may only want to load test “Register” have to jump through four additional complicated steps before getting the answers they want.

In order to provide the Power that “Power Users” will want, I’ve added tremendous complexity to the product that we present to every user. Since every user will start off as a “New User” I’ve burdened everyone with a complicated interface all in the name of power.

The Challenge

I value the power we’ve built into the product, and I know the few users who have managed the hurdles to get there value it as well. I don’t want to remove it, but I need to do something to allow users who don’t require that power to have an easier time load testing their sites. I’m currently using a few different techniques:

More contextual internal links
Allow users who have missed a step to easily jump there to complete it. For example users on the Test Creation page who are missing requests are linked to the Domain Authorization page so they can add the domain they’re missing.

Options that allow parts of the process to be skipped
The general work flow is Create Tests -> Create Test Suite -> Group Tests into Test Suite. Allow users to create a Test, and a Test Suite containing only that Test in one step, rather than four.

More Ajax
By allowing users to correct omitted steps on the page they’re on, the system can avoid breaking their flow, or forcing them to restart.

It’s a work in progress (clearly) but I’m already finding the site easier to use… though, that may have been the problem in the first place. :)

My New Goal

Provide everyone with the power to perform complex actions, while burdening them with as little complexity as possible.

Fixing (some) Phone Menus

nospam@example.com (Paul Reinheimer) — Tue, 12 Feb 2013 02:11:00 +0000

Calling a phone number on a website, and being asked to punch in numbers or say key words to navigate arcane menus to the person or department you need is silly. If you're already on a high bandwidth system that excels in information display, how is it sensible to be transferred to a low bandwidth system before making decisions?

The average person speaks at 120-150 words per minute, and reads at 250-300 words per minute. So even before any other consideration the user will be able to move through information twice as fast on the website as over the phone. But it’s much much larger than that. With the written word it’s incredibly easy to skim, or skip entire sections you know to be irrelevant, on the phone you’re forced to hear all options that precede the one you’re interested in. It’s also possible to jump between options as you evaluate them against your needs.

Consider:
↓ ↓ ↓
I could navigate that in seconds with great accuracy, and have the ability to jump around while I look for the best choice. I'd actually buy a different phone number for every possible end point (rather than requiring a separate #[code], but even with this it's a vast improvement. I think I’ve only ever seen PayPal do something similar with their web authentication codes for calling in.

The Mystery of the Double Stacked Graph

nospam@example.com (Paul Reinheimer) — Mon, 11 Feb 2013 14:55:00 +0000

There’s been an issue on Natural Load Testing (our fantastic web application load testing tool) for a while now, we’ve been referring to it as the Double Stacked Graph. Normally the graphs should look like this, kindly sharing the load testing results with the end user, hopefully being both easy to start with and powerful as usage grows.

Unfortunately, what we’ve been seeing occasionally is the dreaded Double Stack: Here you can see that there is two stacked red bars on top of each other. They represent the exact same space in time, but the data is being added to the graph twice for some reason.

Initially I expected that this was an issue with me handling the updates to the graph with ajax poorly. We had an issue with Where’s it Up for a while where results were duplicated. This was the result of asking the server for “new” results every second by passing the server the list of results already obtained. If a request took more than a second to complete, the following request (initiated while the first was outstanding) would be initiated with the same list of previously obtained results, so the server would obligingly return the same “new” results to both requests.

It turned out however that this wasn’t the problem at all, having learned from the Where’s it Up issue I’d coded those ajax requests more carefully. It turns out that the issue had to do with how I was communicating with MySQL. Each time the graph updates it asks for results with a timestamp newer than the newest data it has. The exact query is something like this:
$query = "SELECT `ts` as `timestamp`, `active_requests`, `requests_initiated`, `median_response`, `running_workers`, `bytes_transferred` FROM `test_results_stats` WHERE `job_id` = '$job_id' AND `ts` > '$minTime' LIMIT 2";
The issue here turns out to be the fact that I'm quoting $minTime. Take a look at these results: mysql> CREATE TABLE `test_decimal` ( -> `job_id` varchar(32) NOT NULL, -> `ts` decimal(13,3) NOT NULL -> ) ENGINE=InnoDB DEFAULT CHARSET=latin1; Query OK, 0 rows affected (0.01 sec) mysql> INSERT INTO `test_decimal` VALUES -> ('9_0209_511673437c4dd', 1360425827.491), -> ('9_0209_511673437c4dd', 1360425828.491), -> ('9_0209_511673437c4dd', 1360425829.491), -> ('9_0209_511673437c4dd', 1360425830.492); Query OK, 4 rows affected (0.00 sec) Records: 4 Duplicates: 0 Warnings: 0 mysql> SELECT * from `test_decimal` WHERE `ts` > '1360425827.491' LIMIT 2; +----------------------+----------------+ | job_id | ts | +----------------------+----------------+ | 9_0209_511673437c4dd | 1360425827.491 | | 9_0209_511673437c4dd | 1360425828.491 | +----------------------+----------------+ 2 rows in set (0.00 sec) mysql> SELECT * from `test_decimal` WHERE `ts` > 1360425827.491 LIMIT 2; +----------------------+----------------+ | job_id | ts | +----------------------+----------------+ | 9_0209_511673437c4dd | 1360425828.491 | | 9_0209_511673437c4dd | 1360425829.491 | +----------------------+----------------+ 2 rows in set (0.00 sec)

While using the greater than operator, I’m obtaining results that are equal if the term I’m comparing against is quoted. Without the quotes it performs as expected. This confused me greatly, I’ve been told countless times that I can quote my variables in MySQL to no ill effect, in fact I probably should if I’m not using paramaterized queries for increased security.

The crux of the issue will come down to MySQL’s Type Conversion rules. It must be converting the two values to float to handle the quoted comparison, though I may be at a loss to describe why they’re converted differently. By not quoting the decimal value I allow the comparison to occur between decimal values, and obtain the desired result.

Update

A few people have commented (either here or elsewhere) about MySQL versions. I just re-confirmed the test with the newest MySQL Debian would give me: mysql> SELECT * from `test_decimal` WHERE `ts` > '1360425827.491' LIMIT 2; +----------------------+----------------+ | job_id | ts | +----------------------+----------------+ | 9_0209_511673437c4dd | 1360425827.491 | | 9_0209_511673437c4dd | 1360425828.491 | +----------------------+----------------+ 2 rows in set (0.00 sec) mysql> SHOW VARIABLES LIKE "%version%"; +-------------------------+-----------------------+ | Variable_name | Value | +-------------------------+-----------------------+ | protocol_version | 10 | | version | 5.1.66-0+squeeze1-log | | version_comment | (Debian) | | version_compile_machine | i486 | | version_compile_os | debian-linux-gnu | +-------------------------+-----------------------+ 5 rows in set (0.00 sec)

Or our Percona server (issue resolved): mysql> SELECT * from `test_decimal` WHERE `ts` > '1360425827.491' LIMIT 2; +----------------------+----------------+ | job_id | ts | +----------------------+----------------+ | 9_0209_511673437c4dd | 1360425828.491 | | 9_0209_511673437c4dd | 1360425829.491 | +----------------------+----------------+ 2 rows in set (0.00 sec) mysql> SHOW VARIABLES LIKE "%version%"; +-------------------------+------------------------------------+ | Variable_name | Value | +-------------------------+------------------------------------+ | innodb_version | 1.1.8-rel29.4 | | protocol_version | 10 | | slave_type_conversions | | | version | 5.5.29-29.4-log | | version_comment | Percona Server (GPL), Release 29.4 | | version_compile_machine | x86_64 | | version_compile_os | Linux | +-------------------------+------------------------------------+ 7 rows in set (0.00 sec)

Assessing a site

nospam@example.com (Paul Reinheimer) — Mon, 28 Jan 2013 22:58:15 +0000

I used to use the design of a site as a quick measure for how mature the site was. For a variety of reasons: improving tutorials, better tools, resources like bootstrap, this is no longer the case. I’m pretty happy with the design of this site, and Allison (not a designer) did it in just a few hours.
These are things I note while gauging site maturity:

Can I reset my password?

Am I forwarded to the requested page after being bounced to a log in page?

Are emails sent in multipart/mime format with an effective HTML and plain-text version?

Is the URL system consistent and meaningful?

Are there clear links to the privacy policy and terms of service?

None of these elements tend to be priorities while a site is being developed, but they pull at developers to be fixed over time. The more of these things a site has, the longer I tend to believe it’s been around.
What do you look at?

Giving up on Movie Theatres

nospam@example.com (Paul Reinheimer) — Mon, 21 Jan 2013 15:23:19 +0000

I saw The Hobbit an unexpected Journey last night, it was pretty good. I liked some of the changes from the books a bit more than others, but I think that overall the additional will help cement Tolkien’s universe in the minds of many.

What I hated was the overall movie going experience. It was a relatively full theatre, full of people paying full price for a 3D movie, currently $14.25/person before concessions and taxes. So clearly the room had to contain a few people whose social lives couldn’t sustain 3 hours without txting. There was also a man sitting in the third row who I think was completing his own screenplay on one of the largest & brightest screen cell phones I’ve seen. I heard a few people ask him to put it away, including someone who got up walked down four rows, and tapped him on the shoulder. This seemed to constrain him to only keeping brief notes for the rest of the film, but he very much wasn’t done.

It confuses me that we’re a society so inconsiderate that people do that, but simultaneously so civil that no one started a fight.
— Paul Reinheimer (@preinheimer) January 21, 2013

The pivotal moment for me was when a theatre employee walked in, past the guy on his phone (the brightest thing in the room by far, including the movie screen) sign the check in form on the far wall, then back past the phone user once more, without a word, and out of the room.

Perhaps demographics have pushed so much that the dollar value of guests who want to user their cell phones out weighs the dollar value of guests who find that disruptive; theatres are best serving their stock holders by not stopping them. Fair enough, but I’m not going to give them my money while they gladly accept and embrace people who disrupt what I’ve paid to enjoy so much. For the price of two tickets I can get the Blu-Ray in a few months, turn the lights & my phone off, and enjoy the film uninterrupted.

Also: fancy “Digital Picture” we paid to enjoy froze twice for several seconds during the film, but that was by far a smaller disruption.

My Broken Button

nospam@example.com (Paul Reinheimer) — Sat, 05 Jan 2013 04:23:22 +0000

I’m one developer on a visualization project using d3.js, which is pretty fantastic. I’m working on the control panel at present, which displays a series of buttons to the end user to allow them to filter out certain metrics in the data set. The buttons do a bit more than that, they’ve also got a nifty horizontal bar in them that indicates what percentage of that data is presently included. So if we were visualizing fast food data, and you filtered out Hamburgers the progress bar for McDonalds might drop down to 40%.

We have a bit of hierarchical data that lends itself well to nested drop downs. When the user hovers over a button for a while, I’ll display the drop down to let them explore the data in greater detail. Things were working well in Chrome, but my :hover pseudo-event never fired to display the drop down in Firefox. I spent a great deal of time trying to figure out why. My buttons seemed quite simple. <button> <div class=“drophead”>Food</div> <ul> <li>Hamburger</li> <li>Fries</li> </ul> <div class=“barchart”></div> </button>

My usual fallback when I can’t figure something out on the web, especially if there’s inconsistencies involved, is to validate. Validation revealed that my divs and uls were not valid children of the button element. Chrome managed to make this work, but Firefox had issues handling pseudo-events for these invalid children, so my drop downs didn’t work. Even though it worked in Chrome now, there was no guarantee that my invalid syntax would continue to function long term. I swapped the buttons out for divs, and asked a co-worker to help with some refactoring as divs don't allow value attributes as buttons do.

W3C Validator saves the day again.

XHGui on MongoDB

nospam@example.com (Paul Reinheimer) — Tue, 25 Dec 2012 21:37:38 +0000

This has been a long time coming. Sean Coates, Joel Perras, and I made an attempt a few years ago, pounding out a good start over a weekend holed up in Sean’s house. Unfortunately, once we left the momentum was gone, and it’s just gathered dust since. More recently, at True North PHP, my love for the tool was rekindled, and an off-hand remark to Mark Story led to some vague agreements about doing something eventually. Fast forward to December 2nd, and we decided to release something on Christmas.

So we are.

Mark & I are pleased as punch to introduce XHGui on MongoDB. Our goal was to get as close to the original feature set of the tool I worked on a few years ago (which leveraged the starting point provided by Facebook) and then to release what we had. What we’ve got now works; there’s still a good distance to go, but we think it’s far enough that we can ask for help form the community at large.

Why NoSQL?

I feel the data we’re storing lends itself well to the loose document format provided by NoSQL solutions. While it’s possible to normalize the information provided (in fact xhprof.io does just this), that’s not an approach that appeals to me. I’ve also had several colleagues comment that the only reason they had MySQL installed on a server was for this tool. My hope is that by allowing more nuanced querying and graphing, users will be able to better explore the data available.

Why MongoDB?

Tool familiarity. I’d love to see some effort to make it work with Couch or other solutions.

Features:

Collects XHProf data and stores it in MongoDB

Presents most recent calls on the home page

Simplified URL aggregation pages

Detailed run page

Warning

XHProf (the php extension) only records a single layer of call depth when recording performance metrics. Consider two functions A & B each call getData(), and getData() invokes mysql_query() in turn. If mysql_query() takes one second to return one time, and 1ms to return the other, the tool isn’t able to attach those values directly to the grandparent. XHProf only knows that mysql_query() was invoked by getData(). In practice this doesn’t tend to matter.

XHProf only provides inclusive times; exclusive times can be roughly calculated (with the caveat above in mind). I haven’t tested that function thoroughly, it could be filling everyone with lies.

We used Twig. It seemed like a good idea, then I saw the call graphs and memory usage.

It does not have any sort of access control mechanism built in, we’re trusting you to handle this on your own.

The “Custom View” page currently doesn’t graph

We haven’t included a tool to manage actually recording the views. This is coming soon.

The Future

Mark and I both have interest in the tool. I’ll be using it on Where’s it Up, and other WonderNetwork sites. Our goal is to be incredibly responsive when people have suggestions, pull requests, etc. To that end we’re in the market for someone with ~two hours a week to help us out in terms of managing pull requests, cleaning up code and helping keep the ball rolling. WonderNetwork will be providing a bit of cash, so this is a paid position.

Apple TV Stutters and Freezes

nospam@example.com (Paul Reinheimer) — Fri, 30 Nov 2012 23:53:10 +0000

I was having a problem with my first generation Apple TV: videos were stuttering while being played. The issue seemed more severe when I was watching HD content, but I was otherwise unable to correlate “bad” experiences with better ones.

I managed to resolve the issue by downloading all of the content on my Apple TV to my computer, then formatting the device, and re-loading it with selected content. I theorize the issue was that the hard drive on the Apple TV was so full it was unable to buffer as much content as it wanted, so any network variation between it and the source of content caused it to stutter. Formatting the device, then re-loading it while leaving plenty of space free has given it the buffering space it wants.

Fun Stuff at WonderProxy

nospam@example.com (Paul Reinheimer) — Thu, 08 Nov 2012 01:15:40 +0000

I gave an unconference talk this weekend at the True North PHP conference in Mississauga. Chris encouraged me to speak so I decided to do a quick talk on WonderProxy.

My friend Morgan gave me advice several years ago: The best talks are the ones that only you can give. By this he means that anyone can give a talk on "Security" or "Performance", running through the numbers, but only you can give a talk on "Security at $company", or "Solving performance problems at $company", and those talks with the personal edge tend to be better. The more I reflect on the talks I've given, the wiser his words seem.

In the talk I ran through why I started WonderProxy, how the service grew, the problems we ran into along the way, and how we've developed other products on top of our global network of servers. I tried to mix in a few lessons we've learned that might apply to others, but by far the most appreciated parts were the funny anecdotes about the problems along the way.

It was great to speak in Ontario once more (I haven't since php|works left), and I'm looking forward to giving a better prepared and filled out version at ConFoo next year.

How PayPal pushes us away

nospam@example.com (Paul Reinheimer) — Fri, 02 Nov 2012 00:49:09 +0000

There’s a lot of things I don’t like about PayPal: Their UI is horrible, search is weak and slow, and they seem to like randomly freezing accounts. Up until Stripe came to Canada, however it was one of the only choices for easy credit card acceptance online.
Will and I actually talked about selling WonderProxy to a US corporation we’d start, just to get more payment processing options.

One of the things that annoys me most is their handling of fraud. When someone accuses you of something, you’re given an incredibly short list of options to resolve the dispute. The options are heavily weighted towards providing a FedEx or UPS tracking number, or refunding the money. These are in fact your options even when you describe your account and payment options as “Services” rather than “Goods” (something PayPal asks you). As providers of an online service, I find this lack of personalization bothersome.

An average fraud case goes something like this:

Get notified that someone bought something

Get notified that there’s a dispute

Provide some details regarding the person who used our service

Lose dispute

Lose the amount of the sale

Pay a bonus penalty

Occasionally both of the first two steps occur before I wake up in the morning.

Resolving a dispute today, the options I was presented really bugged me:

Examining my options:

I can provide proof that the item was shipped through an approved shipper to the address on the Transaction Details page.
Clearly not applicable, we don’t ship things. I considered sending registered mail postcards once, but I think PayPal only covers your cost of the item, not the actual sale price.
I can provide evidence other than proof of shipment.
Not applicable here, the person didn’t actually use the service.
I can provide proof that I refunded payment for this transaction.
No freaking clue how this could apply. Maybe if the person PayPal’d me money, and I sent it back Western Union?
I will accept liability for this transaction.
My only option.
The choice “I will accept liability for this transaction” makes me feel like I’m telling PayPal that this is my fault, that I’m taking responsibility for the good people at PayPal processing a transaction that was clearly fraudulent. I’ve tarnished their good name through my failure.

This couldn’t be further from the truth: They’ve failed me.

Fraud Reduction is in fact one of the features they’re advertising (and we're paying for, over $3k in fees since January 2011). Apart from that, the entire fraud checking process is a black box to us. We’re not passed a fraud-score with any transaction: either we get the money or we don’t. We’re not even passed granular details about how the transaction occurred: Did the CVV test pass? what about zip code matching? user’s country (as determined by IP) vs credit card country? etc. We get none of that. Nor can we tweak our fraud protection options, I can stop people from paying us with eChecks, that’s about it.

We’ve contacted PayPal in the past when we suspected a transaction was fraudulent, the only option we were given was to pre-emptively refund it. There was no way for us to recommend that they flag that transaction for review, or perhaps examine other transactions on the same account/card further. We reach out to customers making suspicious transactions now, and just refund them if we don’t hear back. This is far from the optimal solution, and I feel it robs their fraud team of useful information, but it’s all we’ve got.

As it stands, WonderNetwork will be using Stripe for all new products. As our payment processing system matures with our new products we’ll be migrating it over to WonderProxy as well. I’m just tired of being treated like a criminal by PayPal for using their system in the manner it was designed.