Impressed

nospam@example.com (Paul Reinheimer) — Thu, 03 May 2012 21:34:44 -0400

I've used countless password/license key reminder forms, probably hundreds. They're generally horrible, clearly tacked on at the last second by an intern.

The one I used to request that Macromates resend my license key for Textmate was phenomenal. After confirming that it had resent both my license keys, it provide me with links to the relevant sections of their mail log, to help me understand if there was any problems with delivery.

It was an incredibly helpful example of true craftsmanship.
I'm impressed. Kudos.

What Busy Means

nospam@example.com (Paul Reinheimer) — Sun, 29 Apr 2012 20:19:54 -0400

Last week I asked a barista at Starbucks when they’re most and least busy. She quickly shared a few times with me; I thanked her and sat down. Thinking about it, the times she gave me differed radically from my own experiences. Pondering further I realized that we were using the word “busy” differently: Busy for me meant that the tables were full, and I wasn’t going to be able to come in and work, busy for her meant customers or transactions per minute.

There’s a lesson here, and one I want to learn from. It’s incredibly easy to talk past someone, and make assumptions about understanding.

Lytro

nospam@example.com (Paul Reinheimer) — Sun, 08 Apr 2012 20:47:00 -0400

I love photography, I own a bunch of cameras, and have managed to lose a few along the way. When the Lytro was announced a while back I was quite interested, and with a reasonable price point, I pre-ordered. It's arrived, and I've had a few chances to play with it. Including last night while playing A ticket to Ride

Sample Images

Cards & Trains (click for larger image)

As you've noticed, the two images are focused at different depths. The cards are in focus in the first image, while the trains on the board are focused in the second. The interesting point is that I only took one picture. I selected which point to focus on after the image was taken, either in camera (via a touch screen), or via their software. You can even change the focal point yourself in that image via their website. I find the effect works really well in the short to medium range, switching between two distant objects doesn't affect much, as we'd expect with such a small lens.

Pros:

It's nice to be able to play around with images after the fact
It's a fun form factor for a camera, easy to fit in a bag
It's light
It's cheap in camera terms

Cons:

I'm not impressed with the Colour Performance, even with lots of light

The lens cap is useless, it sticks on magnetically, but falls off with even a glancing touch

Exported image quality isn't great

Lytro software isn't that powerful, I need to export to jpg before I can do any real work

LCD Screen has rather low resolution. Apple went Retina, Lytro went the other direction.

Comparison Shots

I tried to take basically the same picture of flowers with my iPhone 4, Lytro, and Nikon D80 (which sells for around $400 on eBay).

iPhone photo was taken on an iPhone 4, pulled off the camera with iPhoto, cropped and resized down to 1024x1024 in Photoshop, then exported using 81% jpeg. Lytro was imported using their software, the focal point was set to the flower, and it was exported at the only available resolution. D80 Photo was pulled off the card with Lightroom, cropped and white balanced in Lightroom, then exported.

Conclusion

Overall, I'd say it's a fun gimmick, very much a "point and shoot" camera. It's a fun thing to throw in the bag to play with, but I generally know where my focal point is when I'm shooting. I can't think of a time that I wanted to change it after the fact. I have been in situations where I wish I could have focused better after the fact, but generally there I'm dealing with items that are far enough away that the Lytro wouldn't matter. They're pretty cheap (by photography standards), and the technology will only get better.

Improving Query Performance

nospam@example.com (Paul Reinheimer) — Fri, 23 Mar 2012 21:29:41 -0400

Some aspects of Natural Load Testing's site have never really performed that well. The site was under incredibly rapid development so I didn't want to invest too much time in making performance tweaks on code or even architecture that might change in the next week.

No joke, we've re-written our load tester at least three times. Two complete re-writes in the same language, then we switched languages to get better threading performance

I think we hit some sort of limit in the past week or two, and things got much worse. Pages that used to be snappy jumped up to taking 10 seconds to load. Clearly something needed to be done.

When our load tester runs, we generate a run id, here's an example: 21_20111116_4ec4802b9e7b8_000016. That's the user id of whomever launched the test, the date, a unique identifier, and the id of the spawned worker. I wanted something unique, and easy to track if anything went wrong, and that certainly fit the bill.

Unfortunately it also left me executing plenty of queries along the lines of: SELECT min(`test_results`.`timestamp`)AS `min`, max(`test_results`.`timestamp`)AS `max`, sum(`test_results`.`response_time`) / 1000 AS `diff`, count(`test_results`.`run_id`)AS `completed_requests`, `test_results`.`run_id`, sum(`response_size`) as `total_transferred`, max(`test_results`.`response_time`) AS `longest` FROM `test_results` WHERE `test_results`.`run_id` LIKE('21_20111116_4ec4802b9e7b8_%') GROUP BY `test_results`.`run_id` ORDER BY `test_results`.`run_id` ASC one key problem there is the WHERE `test_results`.`run_id` LIKE('21_20111116_4ec4802b9e7b8_%'). Even indexed, that query has a hard time. I tried a few optimization techniques with different index types or other options, but the query was still hurting.

The solution was to break up that run_id into its components: the run id, and the worker id. This wasn't quite as pain free as it could have been (user IDs with different lengths forced me to go through this in several iterations) but I'm quite happy with the results. MySQL is performing a HASH index on the new run_id column, which seems perfect for this situation (there's no web interface to do anything other than an exact lookup on that column).

Results

Before: 10 rows in set (11.49 sec)
After: 10 rows in set (0.00 sec)
Sometimes you need to look further than simple indexes and query optimizations to solve problems.

Wish: Cyclists who follow the rules

nospam@example.com (Paul Reinheimer) — Wed, 21 Mar 2012 20:35:42 -0400
I love cycling, I managed to log 1400km on my bike in 2010 and I’d like to beat that this summer.

The thing I hate most about cycling: Other cyclists.

Cars treat cyclists like crap, and as an aggregate we deserve it. Cyclists routinely run red lights weaving through pedestrians and cars as they cross the street, run stop signs cutting off cars that have begun to cross, ride the wrong way down one way streets, take to sidewalks when the road is slow, cut in front of turning vehicles while continuing forwards, etc. etc.

I wish more cyclists behaved, so we could start expecting cars to treat us fairly.

Loop Me

nospam@example.com (Paul Reinheimer) — Wed, 14 Mar 2012 13:21:00 -0400

As fine patrons of the Internet we’re bombarded daily with brief, one-sided snippets of news at a terrifying rate. News aggregation sites like Fark, Hacker News, and Reddit can further exacerbate the issue with even more sensational (or funny) headlines, and calls to action. Stories can “go viral” long before the other party knows what’s happening, let alone mount a convincing defence. Secondary updates or retractions rarely receive the prominence of the original article.

What I’d like to see is an easy way to ask to be “brought into the loop” on future updates to a story. When someone feels outraged, or surprised by an article they’d be able click a button, and future updates (possibly weeks or months later) would be pushed to them via email or standard aggregation systems. Ideally, this would be a third party service, provided like the social buttons we see on sites now (Like, +1, Tweet, Stumble, Trip, Fall, etc.), which would curate the links between stories. They’d understand how many sites re-hash the same wire stories in the present, and be able to link those as related stories to what was issued in the past. This way when we’re “Looped” in we’d be able to see the full series of articles in chronological order, and updates could effectively root out long standing (and now unwarranted) misconceptions.

As an example: I read an article on Saturday where a girl alleges that she was forced to give up her facebook & email password to school administrators while basically being interrogated at school. A lawsuit is at play, so it’s unlikely I’ll hear the full story until it’s played out in a court case. At present, I’m astounded and disappointed in the school, though it’s entirely possible in the end I’ll simply feel deceived by a sixth-grader. At this point, it’s unlikely I’ll ever hear the end of this story; boring updates to yesterday’s sensationalism rarely receive comparable coverage, even in today’s 24 hour news cycle, and I feel that does a disservice to everyone.

With this, I think we’d have a populace informed not only by sensational allegations, but grounded by an eventual truth.

User Feedback, the endless cycle

nospam@example.com (Paul Reinheimer) — Tue, 13 Mar 2012 22:35:10 -0400

We spent several months on Natural Load Testing before launch, we also spent a considerable portion of the cash reserve WonderProxy had built up on smart people to help us along. We felt… pretty awesome. We started expanding our beta a bit further, and the feedback started rolling in. Key Feedback like: “It doesn’t work”, and “Something is broken” as well as silent feedback, where users got to a certain point in the process then stopped using the service entirely. That was… not so awesome.

The problem users ran into was quite understandable: A lack of ajax on our load testing page required users to manually refresh the page for results to appear. A quick glance at my wall calendar confirms that it’s no longer 2006, and this is an entirely reasonable position for our users to take.

I sat down, and started resolving the feedback as quickly as I could. My list of changes currently looks something like this:

Ajax updates on Run and Calibration pages (“it doesn’t work”) DONE

Better navigation, you have to go home after every step IN PROGRESS

Delete things, Suites, etc. DONE

Graph results in real time

Estimated Time to Completion for test running page

Configure threshold for reporting rows (currently hardcoded at 1000ms+) DONE

Newer tests at the top of the run page DONE

Plot tests to compare runs

Fancy “load server” feature

Deep Copy Suites DONE

Change Suite Domain DONE

Better identify the calibration runs on test suites DONE

Check Time Elapsed column ms vs seconds DONE

Name Runs DONE

Add min width on response time

allow editing test suites

Calibration run can’t actually compare to anything DONE

left align suite name (and make this a link) DONE

make start time not a link DONE

make editing run title only happen when you click the pencil

zebra stripes for result tables! DONE

Timings aren’t being saved in configure DONE

Rename Tests (not just suites)

Change domains on a Test level (not just test suite)

My original plan was to fix a few critical items that led to users thinking things were broken, then work on the beta invite list. As I finished each item, I looked at the list (which originally only contained a few items) and decided I needed to finish a few more before I could invite users.

It took me a few weeks to discover I was in an endless cycle. Users will always have feedback on how your site could be better. If I kept going like this, a year from now I’d have an incredibly polished site, with two users.

So, I broke the cycle. I just sent out another batch of invites, and I feel great.
but not that great about the typo in the From field of that email

No Google Ads on Ping Data

nospam@example.com (Paul Reinheimer) — Mon, 06 Feb 2012 13:03:00 -0500
While Will and I were working on the Global Ping Data pages, I briefly considered putting Google Ads on those pages, to try and get some revenue from the traffic we're expecting. I quickly shot that down.

I think ads make sense for sites that generate or syndicate content as their only purpose. Sites like Ars Technica or Slashdot or even my friend Chris Shiflett's blog all fit that bill. That, however, is not what we're trying to do with this data. Our goal instead is to raise product and brand awareness for WonderProxy, Where's It Up? and Natural Load Testing. Adding advertisements to these pages would only distract from our goals, so while there may be some short term revenue gain, I think they'd end up costing us more in the long term.

Cookies don't replace Sessions

nospam@example.com (Paul Reinheimer) — Mon, 23 Jan 2012 12:48:00 -0500

I’ve seen several instances where people have demonstrated the ease with which encrypted cookies can replace sessions within PHP. Michael Nitschinger wrote a piece recently demonstrating the switch with Lithium, while CodeIgniter does this by default (optionally encrypting). The problem is that while replacing sessions with cookies works, it introduces a few risks not present with native session support, and these risks tend to be under documented.

Encryption is often viewed as a panacea for security problems, you sprinkle a little encryption dust around, and your problems dissolve. Unfortunately, while properly implemented encryption solves the problem of other people reading your encrypted data quite well, it doesn’t do much else. It doesn’t (on its own) tell you if other people have duplicated your data, manipulated the data in question, or handed you an old copy of properly encrypted data.

Consider an attacker who manages to sit in between Amazon and one of their warehouses. Amazon (in this theoretical example) encrypts all order instructions well, then transmits them to the warehouse. The warehouse decrypts the instruction, fills the order, and ships. If the attacker wanted free stuff they could place an order during a low traffic period, and wait for Amazon to send an encrypted message and make a copy. A few minutes later they could re-send that encrypted message to the warehouse, never having even tried to read it. With luck, they’ll soon receive two of whatever they just ordered! This is an instance of the replay attack, and that vulnerability is the one this post will examine in detail.

Exploiting cookie based sessions

Executing a replay attack against a cookie based session is easy, legitimately obtain some desired state on the system in question, and make a copy of your cookies. Should something go wrong, simply restore your cookies to the previous state. To demonstrate this, I’ve created a sample gambling application, you can get the source from github, or Play Now! (thanks to Orchestra.io, it's a free account so give it a moment)

In the game you start with $1000, then have the ability to make arbitrary wagers. Make a few bets to get a feel for the system, then make a copy of your cookie values. Bet again. If you lose money, simply restore your earlier cookies to get that money back. That’s it. This attack works because the system is willing to accept any successfully decrypted message as a valid, and current, session.

This attack doesn’t exist under traditional session storage. The user’s cookie doesn’t change from one request to the next as their money goes up and down. If a system is regenerating the session ID when the user changes privilege levels, the old session is deleted and no longer available when invoked properly: session_regenerate_id(TRUE);

A system using cookie based session storage, while using a CAPTCHA to defeat bots is doomed to fail, the attacker could solve the CAPTCHA once, then re-use that same cookie a million times. Encryption makes it difficult for an attacker to read the message, additional systems must be bolted on to defend against this sort of attack. In the stateless world of HTTP and the web, this is exceedingly difficult without some server-side datastore.

While this was clearly a trivial example, the problem is not. If you’re looking at using cookie based sessions it’s one of the things you need to be aware of.

If you’re interested in learning more about cryptography, including a great rundown of this problem and many others, I’d highly recommend Practical Cryptography

Colours of Light

nospam@example.com (Paul Reinheimer) — Sat, 21 Jan 2012 22:41:17 -0500

Take a look at this picture I took in the mall, in particular the Reitmans, Stokes, and Carleton Cards stores (left to right accordingly):

The stores look very different, mostly due to the colour of light they've used. In the Carleton Cards, there's a yellow light, adding a bit of warmth to a store built around relationships and expressing feelings. Stokes (a kitchen store) uses a very clean, white light, appropriate for clean kitchens, and pushing that meme. Reitmans (a clothing store) uses a rose coloured light which improves the look of skin for most people (especially us pasty Caucasians in the middle of winter).

Just another way stores are marketing to you that's easy to miss.

An end to if statements presented by Computers

nospam@example.com (Paul Reinheimer) — Wed, 18 Jan 2012 10:08:35 -0500
I get quite a bit of email, and deal with quite a few automated systems, banks, credit card companies, the cable company, etc. I’m tired of automated systems forcing me to ask myself a question to figure out what’s happening.

Here’s the first email I found with the word “if” in it:

We wanted to let you know that your order has shipped. If you ordered multiple items, you may receive separate shipments with no additional shipping charges.

My problem is that the email contains an “if”. There’s no reason for it. The computer that sent the email either does have access, or should have access, to my order. It knows that I only ordered a single product, so there’s no point in wasting my time with any other information.

My bank does something even more confusing:

A fee of $1.50 will be charged in the currency of the account for each cheque viewed. The fee will be debited from your account by the next business day. You may view a cheque as many times as you wish during your current EasyWeb session.
For personal chequing accounts the View Cheque service is free for customers who havePaperless or Online Only record keeping.

I have no idea what kind of account type I have. I push “Checking” to get money out of the ATM, but that doesn’t seem to be what it’s asking for. I opened this account when I was 16 years old, at a branch and bank that no longer exists. Do I have “Paperless” or “Online Only” record keeping? I’m not sure. Not knowing may cost me $1.50, but the computer running the bank most certainly knows what options I have on my account. That’s how it will know to charge me if I’ve got the wrong kind of account. Yet, I’m faced with the question.

Seeing these issues reminds me of an article I read back in 2004: Ten most persistent design bugs in particular “let you save me some time.”

In both these cases the developer decided to let me save him or her some time. They didn’t complicate their page or email template with an if structure, and instead presented one to me. Every Single person who interacts with that page, or gets that email is now forced to determine which case applies to them, all an effort to save some work on the part of one developer/designer some time ago.

It’s a stupid waste, and I wish it would stop.

Persistent Person Tags

nospam@example.com (Paul Reinheimer) — Wed, 11 Jan 2012 23:16:56 -0500

I wish it was easy for me to persistently and privately tag an individual on the Internet with a keyword, then hopefully a short justification. When reading an interesting blog post or tweet you could tag them as interesting or understands cryptography or the like. Then the next time you ran into a post, comment, tweet, conference schedule, whatever from that individual that tag would be visible.

It could in theory live almost entirely in the browser, but I think it would work best with some sort of centralized backend to facilitate cross device consistency.

It doesn’t need to be perfect, nor does it need to pierce any sort of attempt at anonymity. Think about gravatar, a great service that plugs your picture into blog comments through your email address. It’s neither perfect nor completely pervasive but it’s still helpful.

To be clear this isn’t something I want to add onto Facebook or Twitter. I’m not friends with these people, I don’t need to hear when they’re drunk, dating, or farming pixels. Nor do I want to read every inane comment or checkin they push to Twitter. I’d simply like a little icon to appear the next time I encounter them on any medium letting me know they’re brilliant, or a racist bigot so I can react appropriately…

Apart from the (hopefully) apparent utility, I also think this helps bring us back something we’ve lost. One hundred years ago if someone said something interesting to you they were standing in front of you (or at least within earshot) and you’d likely have an easy time remembering that individual to credit them later, or simply pay extra attention that individual attempted to share a remark. By the same token if someone said something idiotic and offensive you’d have an easy time remembering them, and simply walk to the other side of the bar when they mount their soap box. We’re exposed to opinions from nearly faceless individuals at a terrifying rate these days, and along the way lost the ability to appropriately credit the things we’ve seen, this wish would make serious progress to fix that.

The Danger of Hooks

nospam@example.com (Paul Reinheimer) — Tue, 10 Jan 2012 09:27:01 -0500
I fell in love with Hooks in frameworks recently; the honeymoon period was tragically short.
First, the love story:

I ran into hooks rather simultaneously with two very different frameworks: Code Igniter and Lithium. In both cases I was using a rather nifty hook to handle ensuring that users were properly authenticated and authorized before accessing a page. I think we can all agree having to add some code to every single method is foolhardy: if (!isset($_SESSION['user_level']) OR ($_SESSION['user_level'] != 'admin')) { header('Location: ' . APP_ROOT_URI); exit; }

Hooks provide a great way to solve the issue. Within Lithium you can leverage the filters mechanism to handle authentication quite easily, in fact it's an example case in their tutorials. The beautiful thing about it in my mind is the simplicity of things at the controller level: I simply list the publicly accessible methods in a property (public $publicActions = array('login');), and everything else is assumed to only be accessible to logged in users. Fantastic! If I mess up when adding a new method, it defaults to closed, which is exactly the result I'm looking for.

Over with Code Igniter things are really quite similar. The post_controller_constructor hook can be used to invoke a specific class and controller. That class can return true, redirect a user, whatever as appropriate. Since it's invoked after the controller is instantiated, similar configuration options can be made available.

The honeymoon

I ripped redundant, error prone, easy to forget, and fundamentally stupid checks out of all of the controllers where I'd added them. These new systems were much easier to maintain, required a lot less code, and I didn't need to add 10 lines of unrelated bloat into each controller. Life was grand. Days where I wrote negative lines of code felt glorious.

The big fight

One day, while messing around, I accidentally turned off the hook configuration within Code Igniter (actually I clobbered a file, and restored the wrong one). Then, things came crashing down in a horrible cacophony of... actually they didn't. Everything kept working: that was the problem. The entirety of my security system was turned off because one file was wrong, and things kept working. Sure, specific calls that referenced the current user's username broke, but there was more than enough left vulnerable for me to get a big chill.

Counselling
Revisiting the hooks system, I was shocked by the tremendous lack of depth in my defense: one mis-configured file and security was off. Even worse, nothing broke. There was no evidence that the security was disabled unless you went probing, which is horrible. The only thing worse than a safe that won't lock, is one that looks like it's locked but pulls open to a slight touch.
Conciliation
The easy thing to reach for with both Lithium, and Code Igniter is construct(). A single, unified location to ensure that the authorization tests have been executed. Unfortunately, in both cases construct() is called long before the authorization hooks are run. More specialized solutions are required.
With Lithium

The invoke() method is invoked after the authorization filter, so it's a great candidate for double checking. public function invoke($request, $dispatchParams, array $options = array()) { if (!defined('AUTH_CHECKED')) { throw new DispatchException('Authorization filter not run.'); } return parent::__invoke($request, $dispatchParams, $options); }

With Code Igniter

The _remap() function is called (when available) to allow you to remap incoming requests to a different method. Since it's invoked universally, the check can go there. public function _remap($method) { if(defined('AUTH_CHECKED')) { $this->$method(); }else { exit('ACL Configuration Error'); } }

Both of these cases provided me with the depth I was looking for. I'm no longer entirely dependent on one configuration option or file for my security to function. Should it fail, I've got a secondary check in place; this example of defence in depth allows me to be comfortable with the hooks security system once more.

Final thoughts

Through researching this, and exploring several code bases to which I have access, I've noticed two distinct strategies for managing method level access. In Lithium, each class indicates which methods should be accessible to which security groups using a public property; by contrast, in the other strategy, public methods were explicitly laid out inside the single authorization function. The former has the advantage of allowing you to quickly and easily manage security while you add functions to a given controller; the latter solves the issue of your authorization rules being scattered across each and every controller by centralizing them in one easy to locate file.

Updates!

Some suggestions have cropped up:

Unit Testing
While I'll agree that great, properly implemented, unit testing would have caught this it still doesn't leave me feeling comfortable. First, I've seen a lot of unit testing code, and much of it wouldn't have caught this. Either because the code specifically tested the authentication, and authorization code on its own, not combined with the actual controllers. Or because it used its own configuration file during testing, missing the fact that something had been removed from production. Second, I'm not sure that simply exposing the issue with unit testing would make me comfortable, I definitely would have caught it faster. Possibly even as soon as I'd committed the code. But I'd still like something running along side the code constantly to make sure the checks are in place.
Auth Specific Class
As suggested by Chris Morrell (in this series of tweets) I could be using a dedicated authorization class. While this does solve the configuration problem by calling it explicitly, I'm losing granularity or making a call with every action. I'll lose granularity if I decide to put one initial call in my constructor (or other early globally called method) since it affects the entire class. Ending up making a call with every action was exactly what I was trying to avoid in the first place. That said, the use of getIdentity() vs requireIdentity() is quite slick, I like how simultaneously explicit and transparent a normal action becomes.

DVD Ripper, that Works

nospam@example.com (Paul Reinheimer) — Wed, 04 Jan 2012 14:38:51 -0500
This is the first in a continuing series of things I wish. Posted on Wednesdays

"This doesn't sound hard"

All I want is a DVD ripper that can do what a CD ripper did over a decade ago. I'd like to insert a DVD into my computer, have it look up the meta data on some sort of modern CDDB equivalent, then rip the disk with appropriate meta data for iTunes. It will need to understand the difference between Television and Movies, as well as how to rip the right audio track, and what chapters are associated with what episode. This really doesn't seem that hard. Yet, I don't have one. Everything I've seen requires me to figure out those associations, or handles meta data poorly.

RipIt: Is a functional ripping tool, it however doesn't seem to understand TV disks. I put in Battlestar Galactica Season 1, Disk 2. It ripped it into a single file, that turned out to be something like the middle of episode 3 to the end of episode 4.

Handbrake: Another functional ripping tool, it requires me to figure out which chapters are associated with which episodes on a disc. I don't want to worry about this, put a disc in, wait, take disk out, repeat.

I'm on a mac, though if something truly perfect came out for Linux or Windows I'd rip from a VM.

Updating Theme Please Stand By

nospam@example.com (Paul Reinheimer) — Tue, 31 May 2011 21:13:26 -0400

I'm updating my theme at present, this may take a moment.

I'm still not happy, but it's at least legible now, I'm going to try blogging again as a constant kick in my own ass to fix it further