I’ve tremendously enjoyed every Brooklyn Beta conference I’ve gone to, but this was easily the best yet. The “Make Something You Love” slogan has been around since the beginning, but this year’s choice of speakers really seemed to push that consistently, even when Ben Pierrat spoke about how he felt he failed his startup.

When Aaron Draplin took the stage, I wasn’t really sure what to expect; I was mildly concerned that the orange hat, beard, and demeanour were some sort of act. But he turned out to be one of the more authentic, down to earth, and genuine people I’ve heard speak. If this wasn’t an election year, I might refer to him as a solid “salt of the earth” type of guy. Not bad for someone who designed the recovery.gov logo, and was friends with John Hughes. To me, his talk was all about doing good work you enjoy, and being a good neighbour. He talked about doing jobs for friends, or ‘farmer john’, and how those pieces of work were important to him, even if the clients were small.

I think that there’s a great message for developers and designers in there not to forget clients we may perceive as “little” as our skills and reputations grow. Some of the work Aaron treasured most were those small clients, where his designs had the greatest impact. I was sad when I discovered I’d missed the chance to buy something from his temporary store. 

I knew Alex Payne back when he worked at twitter, and to be honest I’m more than a little jealous that as a Canadian I don’t get access to [Bank] Simple. The vision for Simple was that banking doesn’t have to suck: the technology required to help people save has existed for a long time, but the banks and regulatory hurdles involved in helping people solve those problems are tremendous. He mentioned that part way through development they stopped and moved all the developers into a single location to make things easier. Having done some work on strictly regulated systems, I very much sympathize with that decision. 

Alex was a much calmer speaker than Aaron, but I really enjoyed hearing the story of him working on something he cared about. As much blame as we put on the banks for the recent economic turmoil, if we’d better managed our finances things wouldn’t have been as bad. The interface to our money that Simple provides, such as easily answering questions like “can I afford this” can help move us in the right direction.

I’ll be honest, I was absolutely amazed when Seth Godin took the stage. While the speakers at Brooklyn Beta have always been a very high caliber, they’ve generally been people who are famous within Design, Development, or Computing circles. As a New York Times best selling author, Seth Godin is properly famous well beyond those groups. He wove a story talking about how capitalism is dying, and the next revolution will be art (whilst weaving in his views on tribes). 

His mention of how our education system was really designed with the industrial view of the world struck me squarely. It really reminded me of Sir Ken Robinson’s TED talk (and two books: Out of Our Minds: Learning to be Creative and The Element: How Finding Your Passion Changes Everything) which also centred on Art. Sir Robinson’s viewpoint was more centred on how we’ve no idea what kind of problems we’ll need to solve in the future, so creativity should be the thing we cherish the most in schools (not test taking). Seth also pushed us further than simply making something we loved, he wants us to make something that matters. 

Ben Pierrat’s story was a bit different. I’ve seen many speakers stand up and talk about their success, or at least their work so far. It was simultaneously refreshing and challenging to hear Ben talk about how he felt he’d failed his startup Svpply. It was a very introspective talk, examining the reasons he didn’t want to step down as CEO as his company grew, and how he felt that his leadership ultimately failed his vision. While getting bought out by eBay is by no means a failure, it wasn’t the success they were looking for.

Ben’s talk was difficult for me as an entrepreneur. I’ve been leading WonderProxy and WonderNetwork for a few years now, and we’ve kept all the real decisions internal. Ben’s talk has me wondering if we should have brought someone else in to lead us, accepted sales help, etc. 

Kyle Neath is a director at GitHub, and gave me a lot to think about. He talked about how Github handles hiring, and how they manage their distributed teams. All employees are welcome to work from home, and they’ve developed and improved a series of tools to better support their distributed workforce. The tools and such (as well as the paid hours they’re encouraged to put into them) really seem to have captured the spirit of a “fun workplace” without having everyone in one place. 

I think for me Kyle’s biggest point was between the lines: Remote employees can work at scale. As WonderProxy has grown, and as I’ve encountered many a rough edge for remote employees at $DayJob (where things continue to get better), I’ve become concerned that remote just didn’t work past a few people. Kyle’s convinced me otherwise. 

I’ve blogged about small challenges while working a few times: working where the work gets done, and listing everything you’ve actually done.

TeuxDeux is a fantastic accompaniment to both!

I think I love it for three reasons:

• Easy as paper
• It’s everywhere
• It shows what I’ve done

Easy as Paper

To add an item you click in the text box under the day you’d like to do it, and hit enter. If you’ve got another item for that day you just continue typing, hitting enter after each item. When you’ve completed a task you click on it once and it’s struck through. Items you don’t complete one day are carried forward to the next.

It’s Everywhere

With a fantastic and easy to use website and iPhone app, I don’t need to remember to bring anything with me; I’m already carrying something around that will let me use it.

Shows me what I’ve done

A lot of the bug tracking tools I’ve used hide things that are complete, which can be a boon in a busy interface. In TeuxDeux’s simple interface that’s not required. Seeing my list of crossed out items grow throughout the day can be a small motivator to keep going.

Easy as Paper revisited

My real issue with a lot of bug tracking systems (and I’ve used many: Assembla, Jira, GitHub, Bugzilla, etc.) is the amount of time you spend simply entering a ticket. It’s one of the things I really loved about FogBugz, you could enter ticket items as quickly as you can with TeuxDeux. While you may require the ability to enter more detailed information in some bugs, for a To Do application, simplicity and speed reign supreme.

also: it's free. Go sign up

I was initially a little unclear about how to connect my Lithium application to my MongoDB replica set while specifying multiple serves, I’ve settled with: Connections::add('default', array( 'type' => 'MongoDb', 'host' => 'localhost:12345,washington.mongo:12345', 'replicaSet' => true, 'database' => 'wiu' ));

I’m listing multiple servers that form my replica set in the host field as a comma separated list. While you technically only need to list one (and the driver should discover the rest), it’s a good idea in case the one you have listed goes down.

Well, secretly, that’s not actually how I’m connecting to MongoDB. I’m also telling Lithium that it should set a read preference of RP_NEAREST. This option isn’t available yet in lithium, but I’ve got an open pull request to fix that. I’m actually connecting to MongoDB like this: Connections::add('default', array( 'type' => 'MongoDb', 'host' => 'localhost:12345,washington.mongo:12345', 'replicaSet' => true, 'database' => 'wiu', 'readPreference' => Mongo::RP_NEAREST ));

Note: Specifying your read preference (and still having everything work) currently requires an un-released version of the MongoDB driver. You either need this fix, so you can build from source. Or presumably the fix will be included in 1.3.0 BETA 3.

As we’ve been extending the API for our site monitoring and diagnostic tool, Where’s it Up, one of my key concerns was the single point of failure our Washington server represented. Luckily, MongoDB was explicitly designed to make this sort of thing easy.

For Where’s it Up, we’re planning on launching with two main servers in different data centres, one in Washington, the other in San Antonio. We’ve considered a few different approaches on the web side, and have decided initially to mark our Washington server as primary on our load balancer, and only fall back on San Antonio as secondary should Washington fail.

Having set up multi-server configurations for databases before, I initially configured both systems in a Master - Slave relationship. Marginally more reading revealed that what I really wanted was a replica set. Within a replica set, an election will be called when the primary fails, and a secondary will be nominated as the new primary. The client library (in our case PHP) should handle this automatically, with only a brief interruption.

To effectively handle nominations, we’d need a third voter; I decided to place it in Montréal. This server would serve as an additional vote and as our backup slave. That voter is required because the systems seek a majority of servers to vote during an election; without a majority, they're unsure if the other machines are actually down, or simply on a segregated network (and possibly nominating their own master). With only two servers, the lone remaining server would be unable to self-nominate after the first crashed.

I also initially configured a fourth voting only server, an arbiter, in Washington after reading through the MongoDB documentation. Luckily the sage Sean Coates explained why this was a bad idea, and I removed it promptly.

Our configuration now stands at: wondernetwork:PRIMARY> rs.config(){ "_id" : "wondernetwork", "version" : 49449, "members" : [ { "_id" : 0, "host" : "washington.mongo:12345", "priority" : 10 }, { "_id" : 1, "host" : "montreal.mongo:12345", "priority" : 0 }, { "_id" : 2, "host" : "sanantonio.mongo:12345", "priority" : 5 } ] }

Under this configuration, Washington should win any elections (with its higher priority) it’s present for; failing that, San Antonio should win. Montréal will never become primary due to its priority of zero. I could have simply used priorities of 0, 1, and 2, but the old days' programming with line numbers have taught me to leave gaps for future expansion.

I’ve configured several different RDBMS’s to include a Master - Slave relationship. I’ve found MongoDB to be at least an order of magnitude easier to work with in this regard, even my first time through.

If you’re running this code in Lithium, it’s not doing what you think it’s doing:

This is a really easy line of code to end up writing, especially if you’ve been reading the MongoDB docs a bunch, or are just porting some code into the application. That’s because if you were accessing the MongoDB driver directly you would want:

or in MongoDB at the command line:

They all seem very similar, but the first one is actually returning all the results from MongoDB to PHP, lithium is then obligingly counting the results, returning that count, and then throwing away all the data. If your data set is tiny, the difference here will be negligible. However with large result sets, or simply results that contain large documents, the difference could be catastrophic.

What you’re looking for is:

Last night I started working on a script to help customers integrate the Where’s it Up API with their existing monitoring infrastructure. I started with Nagios since that’s what we’re using for WonderNetwork (I also learned we’re monitoring ~1200 services). Shortly after I started coding I talked with Will since he’s our systems administrator, and the target market for the Nagios integration script. Over the course of a few discussions during the evening, the scope of the script changed drastically (and surprisingly, it actually shrunk). The way I wanted to write the script didn’t match at all with how my target users would want to use it.

My Plan: Configure Nagios to hit the script every minute. The script will fail its first ever check (since calling the API and waiting for the result takes time), but it will cache that result. Subsequent calls will see the cached information. Once data is a minute from expiring it will silently make the call again, so that new data will be presented on the next call. When things do actually fail, make further calls to help diagnose the issue. Those details will be available to the administrator via a log, or email.

Will’s Complaint: I don’t like it when monitoring tools lie to me. If Nagios claims it ran a check at 12:43am that passed, that service better be functioning properly. Nagios is willing to wait 5 minutes (by default) for a result to return, so just make the script wait for results before returning.

So, that’s what I did. I also had rather grand plans involving configuring the script for every check, then having Nagios pass in the check title to invoke that check. Will (thankfully) managed to convince me that this was folly, and the script now accepts relevant parameters via the command line.
Lesson One: Target your efforts to your customer’s wants.

While testing the script I also noticed that our WonderProxy site was occasionally failing my tests. Further investigation revealed that the service was timing out after waiting 5 seconds with no response. We initially blamed Apache as being slow to acquire the entropy required to generate the SSL connection… this was not at all the case. As it turns out, some of the data that’s cached for the page was regenerating too often, and taking longer than 5 seconds to do so. This problem has likely existed for a long time, undetected. Users simply received incredibly poor performance when they were unlucky enough to be the first user after the cache expired.
Lesson Two: Subtle problems can exist undetected for ages, undetected unless you look for them.

Finally, while building the script I noticed a few issues with how our API was handling invalid requests. This occurred while I was unknowingly passing invalid data to the system, which obligingly blundered on, while not actually doing anything, making looking up the results of that request incredibly difficult. I didn’t find these issues while writing my initial test code & curl examples, as they were all valid. These silent failures allowed me to re-visit portions of the API code to error out, and do so in a verbose manner.
Lesson Three: Eating your own dogfood is an incredibly fast way to find problems.

Two years ago at Brooklyn Beta I announced Where’s it Up. It had a horrible green and blue interface that my current design partner still jokes about. Apart from the design update well over a year ago we haven’t really given it any love. It’s not a product that generates any revenue for us, so it was hard to justify while WonderProxy does, and we had a clear route to revenue with Natural Load Testing.

That’s changed.

I was recently inspired to resolve a few of the issues I had with Where’s it Up, and along the way created a backend that was much better positioned to work with an API front end. So I created that front end.

Where’s it Up has always been an easy tool for techies to check in on a site they care about. It hasn’t however been in a position to help them keep an eye on it over time. That’s where our API comes in. Make an API call requesting information, then make a second API a few moments later and start retrieving results. We feel that this system fits in well with how we and our partners monitor our systems. Rather than adding a completely disparate tool - that integrates poorly or not at all with our existing infrastructure - we’re able to integrate these checks into our existing Nagios reporting system.

Where’s it Up API offers HTTP, DNS, and Traceroute checks from around the globe, free for at least one month. We’re adding Ping shortly, and the service will continue to expand as our network does.

I’ve had a few long standing promises to reveal in more detail how the game worked, was scored, etc. Since I’d rather not reach the one year anniversary of those promises, I thought the eve of my flight down to New York for the 2012 edition of the conference would be a good time to post.

One thing to get out of the way, there is no 2012 edition of the game, at least not by my design. Sorry!

Building the Cards

As each individual registered for their cards (and my thinks go to the tireless Kitt Hodson for her help in prodding people to sign up) the system asked them to authenticate with Twitter. This gave me a few basic pieces of information: their twitter handle, a picture, and some way to determine if they were in fact attending the conference. If the individual had a moment longer they were asked to score themselves along four criteria: Developer, Designer, Leadership, Presenter, and provide a brief bio (initially copied from their Twitter bio).

The twitter integration made things mostly painless: by only asking for read-only access to their basic twitter info I feel like we avoided a lot of possible issues. I also took great pains to allow people to opt-out of the entire thing, the conference is so welcoming I didn’t want anything associated with it to seem overbearing.

The card design itself was done by jonny.gotham, while the Brooklyn skyline photo was taken by mia takahara.

Once everyone’s details were collected, and quite a few higher-resolution photos than Twitter provided (thanks everyone!), it was time to actually assemble the cards. The various bits of information were applied to a PDF using the TCPDF library. It drew the red blocks, placed the text and photos on top, then finally spaced the cards appropriately (with full bleed and cut lines) for the printer. The kind and skilled people at Rubiks Montréal handled the printing. I went with a local printing house, rather than someone in New York to allow frequent two-way communication, and avoid having to ship proofs back and forth. The cards were printed on a nice stock of paper, laminated then cut. I’m incredibly happy with how those turned out.

Scoring

By far, the most contentious issue. Sean Coates helped me a lot with running numbers, and coming up with a system that seemed reasonable. Overall I wanted to give an advantage to people who put some effort into understanding the rules, without guaranteeing the prize to someone who just sat at their laptop all night crunching the numbers. So rather than share the scoring formula we trickled out a few rules via the twitter stream.

Reading into the tweets a few rules should be clear:

• Teams must consist of 4 people, a snack, and a location
• Developers and Designers should be of approximately equal skill
• Only one leader will work at full strength
• Only one presenter will work at full strength
• Don’t submit a team with two identical cards

Those rules stemmed from my own impressions on what makes a good team, and were refined considerably by my conversations with Sean. The “perfect team” would have very strong development and design teams, lead by a great leader, whose final product is presented by a great presenter. Clearly if you simply refer to everything as “strong” or “great” you’re going to end up with a good product :).

When it came down to numbers, the food (Cookies, Beer, Bagels, or Pizza) was handled as simple addition. Each card added to the team’s starting score in one of the four categories. From there, the team’s total Development and Design skills were totalled. Then we checked to ensure that neither skill (development or design) was more than 33% ahead of the other, and applied a penalty if it was. Neither strong Design, nor Development can truly hide a large deficit in the other. From there, we found the strongest leader on the team, and added to their skill half of the rest of the teams’ skill in leadership. For presentation we were a bit less giving: your final score was equal to your strongest presenter plus a third of the remainder of the team. Then we considered your place card: each place card advantaged a certain category by adding 15% to its present score.

To determine Leadership’s effect on a team we compared the final Leadership score to a simple chart we had created. A worthless leader (skill 1 after all bonuses and additions) would add absolutely nothing to your total design + development score, while a fantastic leader could provide a 5x multiplier to it.

The presentation skill was handled similarly. We took the final presentation value, divided it by 10, then subtracted .2, and multiplied the result by the total thus far. So a great presenter (score 10) with only a bit of help from her team (+5 = 15) would result in the current sub-total being multiplied by (15/10) - 0.2 = 1.3. This team’s product will “look” 30% better than it actually is as a result of the great presentation. Note that if you had a good presenter (score 8) with no help from his team, or bonuses you’d end up with a lower final score! Presentation matters.

There was one final bonus to apply: in today’s world of rock star designers and developers having big name talent on your team can help. We applied a bonus for people who were present in more than one of the Top 5 Teams, so if J. Smith was in two of the top five, both of those teams received a small bonus for having J. Smith the rockstar.

The actual calculations were handled by a small script, facilitated by the card ID at the bottom of each card.

tl;drInitial score = sum(developer) + sum(designer) * [leadership modifier] * [presentation modifier]

Thoughts

Overall the game was a tonne of fun to set up and administer. It was fantastic to see people walking around trading cards with each other. I hope this game provided some fun, ice breakers, and a cool memento from the conference! I was inspired by the card game that was ran at Webstock 2011.