Some aspects of Natural Load Testing's site have never really performed that well. The site was under incredibly rapid development so I didn't want to invest too much time in making performance tweaks on code or even architecture that might change in the next week.
No joke, we've re-written our load tester at least three times. Two complete re-writes in the same language, then we switched languages to get better threading performance
I think we hit some sort of limit in the past week or two, and things got much worse. Pages that used to be snappy jumped up to taking 10 seconds to load. Clearly something needed to be done.
When our load tester runs, we generate a run id, here's an example: 21_20111116_4ec4802b9e7b8_000016. That's the user id of whomever launched the test, the date, a unique identifier, and the id of the spawned worker. I wanted something unique, and easy to track if anything went wrong, and that certainly fit the bill.
Unfortunately it also left me executing plenty of queries along the lines of: SELECT min(`test_results`.`timestamp`)AS `min`, max(`test_results`.`timestamp`)AS `max`, sum(`test_results`.`response_time`) / 1000 AS `diff`, count(`test_results`.`run_id`)AS `completed_requests`, `test_results`.`run_id`, sum(`response_size`) as `total_transferred`, max(`test_results`.`response_time`) AS `longest` FROM `test_results` WHERE `test_results`.`run_id` LIKE('21_20111116_4ec4802b9e7b8_%') GROUP BY `test_results`.`run_id` ORDER BY `test_results`.`run_id` ASC one key problem there is the WHERE `test_results`.`run_id` LIKE('21_20111116_4ec4802b9e7b8_%'). Even indexed, that query has a hard time. I tried a few optimization techniques with different index types or other options, but the query was still hurting.
The solution was to break up that run_id into its components: the run id, and the worker id. This wasn't quite as pain free as it could have been (user IDs with different lengths forced me to go through this in several iterations) but I'm quite happy with the results. MySQL is performing a HASH index on the new run_id column, which seems perfect for this situation (there's no web interface to do anything other than an exact lookup on that column).
Results
Before: 10 rows in set (11.49 sec)
After: 10 rows in set (0.00 sec)
Sometimes you need to look further than simple indexes and query optimizations to solve problems.
The thing I hate most about cycling: Other cyclists.
Cars treat cyclists like crap, and as an aggregate we deserve it. Cyclists routinely run red lights weaving through pedestrians and cars as they cross the street, run stop signs cutting off cars that have begun to cross, ride the wrong way down one way streets, take to sidewalks when the road is slow, cut in front of turning vehicles while continuing forwards, etc. etc.
I wish more cyclists behaved, so we could start expecting cars to treat us fairly.
As fine patrons of the Internet we’re bombarded daily with brief, one-sided snippets of news at a terrifying rate. News aggregation sites like Fark, Hacker News, and Reddit can further exacerbate the issue with even more sensational (or funny) headlines, and calls to action. Stories can “go viral” long before the other party knows what’s happening, let alone mount a convincing defence. Secondary updates or retractions rarely receive the prominence of the original article.
What I’d like to see is an easy way to ask to be “brought into the loop” on future updates to a story. When someone feels outraged, or surprised by an article they’d be able click a button, and future updates (possibly weeks or months later) would be pushed to them via email or standard aggregation systems. Ideally, this would be a third party service, provided like the social buttons we see on sites now (Like, +1, Tweet, Stumble, Trip, Fall, etc.), which would curate the links between stories. They’d understand how many sites re-hash the same wire stories in the present, and be able to link those as related stories to what was issued in the past. This way when we’re “Looped” in we’d be able to see the full series of articles in chronological order, and updates could effectively root out long standing (and now unwarranted) misconceptions.
As an example: I read an article on Saturday where a girl alleges that she was forced to give up her facebook & email password to school administrators while basically being interrogated at school. A lawsuit is at play, so it’s unlikely I’ll hear the full story until it’s played out in a court case. At present, I’m astounded and disappointed in the school, though it’s entirely possible in the end I’ll simply feel deceived by a sixth-grader. At this point, it’s unlikely I’ll ever hear the end of this story; boring updates to yesterday’s sensationalism rarely receive comparable coverage, even in today’s 24 hour news cycle, and I feel that does a disservice to everyone.
With this, I think we’d have a populace informed not only by sensational allegations, but grounded by an eventual truth.
We spent several months on Natural Load Testing before launch, we also spent a considerable portion of the cash reserve WonderProxy had built up on smart people to help us along. We felt… pretty awesome. We started expanding our beta a bit further, and the feedback started rolling in. Key Feedback like: “It doesn’t work”, and “Something is broken” as well as silent feedback, where users got to a certain point in the process then stopped using the service entirely. That was… not so awesome.
The problem users ran into was quite understandable: A lack of ajax on our load testing page required users to manually refresh the page for results to appear. A quick glance at my wall calendar confirms that it’s no longer 2006, and this is an entirely reasonable position for our users to take.
I sat down, and started resolving the feedback as quickly as I could. My list of changes currently looks something like this:
- ****Ajax updates on Run and Calibration pages (“it doesn’t work”)**** DONE
- Better navigation, you have to go home after every step IN PROGRESS
- Delete things, Suites, etc. DONE
- *Graph results in real time*
- Estimated Time to Completion for test running page
- Configure threshold for reporting rows (currently hardcoded at 1000ms+) DONE
- Newer tests at the top of the run page DONE
- Plot tests to compare runs
- Fancy “load server” feature
- Deep Copy Suites DONE
- Change Suite Domain DONE
- Better identify the calibration runs on test suites DONE
- Check Time Elapsed column ms vs seconds DONE
- Name Runs DONE
- Add min width on response time
- allow editing test suites
- Calibration run can’t actually compare to anything DONE
- left align suite name (and make this a link) DONE
- make start time not a link DONE
- make editing run title only happen when you click the pencil
- zebra stripes for result tables! DONE
- Timings aren’t being saved in configure DONE
- Rename Tests (not just suites)
- Change domains on a Test level (not just test suite)
My original plan was to fix a few critical items that led to users thinking things were broken, then work on the beta invite list. As I finished each item, I looked at the list (which originally only contained a few items) and decided I needed to finish a few more before I could invite users.
It took me a few weeks to discover I was in an endless cycle. Users will always have feedback on how your site could be better. If I kept going like this, a year from now I’d have an incredibly polished site, with two users.
So, I broke the cycle. I just sent out another batch of invites, and I feel great.
but not that great about the typo in the From field of that email 
