Where’s it Up?

nospam@example.com (Paul Reinheimer) — Wed, 21 Jul 2010 12:04:35 -0400

WonderProxy is proud to present Where’s it Up?, a new tool to help system administrators determine whether or not their site is up from around the world. The tool accepts a URL, and allows you to select global locations. It then attempts to connect to the given server and issue a HEAD request from the global locations you selected, and reports the results.

Key Features

Servers around the world

Local DNS resolution on each server
Reports IP and timing information
Follows a reasonable number of redirects

Key Technology

Curl

PHP with pecl_http

Memcached

Gearman

Supervisord

How it Works

Building a reasonably robust application was trivial, thanks to being able to leverage the great technology built by others. When a request is received, an intermediate script does some basic checking, then passes off a number of jobs to Gearman (one per location requested), the job ID is handed back to the user and results will be displayed via ajax. In the background Supervisord keeps five workers running all the time, they basically spin waiting for new work. When a job arrives they first check to see if there’s a recently cached result in memcached. Failing that an SSH tunnel is used to connect to the requested server, the request is issued with curl, and the results parsed. This result information is pushed into memcached.

On the client side, requests are made back to the server including the job id, and already obtained results. New results are handed off as they become available.

The system also uses memcached to throttle incoming requests from a given IP based on the example given by Simon Willison. In this case I’m actually using Swatch Beat time to manage the count; the edge case over midnight is quite easy.

Gearman is helpful in this case for three reasons: it provides great separation between the client requesting the work, and the workers actually completing it. It also trivially allows for multiple workers to be used in the resolution of requests. Finally, it provides a very effective work queue; when the site is under heavy load it will just take a bit longer to resolve requests. A small side benefit is the ability to pull down all the workers, update the worker script, then restart them, all without affecting the jobs in queue (except for the momentary delay).

Memcached is helpful as it’s a quick place to store the values that handles expiry for me automagically. In this particular case, as there’s a single server involved, something like APC would provider a thinner solution, however I have a lot of experience building apps with Memcached as a storage engine, and not much using APC.

Supervisord is helpful as it keeps the workers running, starts them up after they crash (hasn’t happened yet) and on machine boot. The dæmonize your PHP post by Sean Coates was incredibly helpful in getting this going.

SSH is helpful as it allows us to display accurate timing information from that machine to the requested location. We could use our own proxies directly, but then we're dealing with the time to connect from Washington -> Sydney -> Your website. That extra bit of math in the middle is only moderately tricky, but more importantly: inconsistent. The tunnel lets us execute the commands directly on the remote server.

Why we built it

We've used tools in the past that allow us to quickly check if a site is up or down. They're nifty, but these days many sites are using anycast DNS to publish different IPs globally, with multiple servers and multiple data centers. Simple tools simply aren't able to adequately express whether or not a site is up, or down. By leveraging the network of servers we already had, building a more advanced and complete tool was easy.

XHGui Improvements

nospam@example.com (Paul Reinheimer) — Mon, 14 Jun 2010 11:15:54 -0400
I merged my Highcharts branch into master today, including a bunch of improvements to the GUI for XHProf including:

Using Highcharts for graphing URLs over time, which allows for multiple axis per graph

Adding a pie chart to the individual run page to show where time was spent on page load

Ability to merge various calls for display within the pie chart (e.g. mysql_* into mysql)

Switched to the javascript tablesorter for the single run results

Filter to allow you to view results only from a specific server or domain

Highcharts is a commercial library in Javascript for graphing, it's really easy to use and offers more features than the Google libraries used previously. I've purchased a license for the application so that anyone using this gui, regardless of the type of application they're profiling, can do so. The Highcharts integration was done by my colleague Graham Slater.

I really didn't consider the branch "done", the diff page still needs the pie charts, and to be switched over to the new table format. However, the license file received some serious updates to accurately represent the state of the code base and I considered making that file accurate to be a high priority.

Screenshots

As always, the code is available at GitHub

The accountability problem - Concluded

nospam@example.com (Paul Reinheimer) — Tue, 25 May 2010 13:43:32 -0400

Last week I blogged about the accountability problem: how to handle tracking who made what change, and providing the ability to revert it.
Thanks to some great comments, and offline discussions, I’m seeing a few options, and have changed my mind on my desired approach. There's also some great technical reading on the subject: Slowly changing dimension

Option 1: Store history in the table

This was the approach I considered initially. Storing each revision of a record in the table, with an active bit set to indicate which version is active. More details in the original post. This approach is simple: it doesn’t require more tables, queries are still pretty simple, and with a combination of timestamps it’s easy to track down associated records that were changed.

The downsides are pretty clear, and somewhat damaging. The number of rows in the most commonly hit tables will grow much faster than the table overall, which is sub-optimal. It can also complicate issues like reporting and the like. In the case of highly normalized designs this design will likely break down, as you’ll need to match revisions across multiple tables on every query.
classes

id version active session title start_date end_date timestamp login_id

1 1 0 Spring Dancing 101 May 1, 2009 May 28, 2009 1274467708 23

1 2 1 Spring Dancing 101 May 1, 2010 May 28, 2010 1274467709 23

2 1 1 Summer Dancing 102 June 1, 2010 July 15, 2010 1274467710 27

3 1 1 Fall Dancing 101 September 1, 2010 September 28, 2010 1274467808 35

I would have used this approach in a smaller project. The system I’m replacing has been running for ten years, I have to plan for this lasting that long with only minimal maintenance. I’d rather not watch those row counts become 3-5x what they should be.

Option 2: Store histories in an external table

Rather than storing the histories in the regular table (say classes) creating a duplicate table (classes_history) and storing the revisions there. This helps trim the size of your most heavily used tables, and keeps things relatively simple.

While trimming the row count, it does double the table count, and still fails to handle complicated objects, or normalized table structures where a single datum requires multiple tables.
classes

id session title start_date end_date

1 Spring Dancing 101 May 1, 2010 May 28, 2010

2 Summer Dancing 102 June 1, 2010 July 15, 2010

3 Fall Dancing 101 September 1, 2010 September 28, 2010

classes_history

id version active session title start_date end_date timestamp login_id

1 1 0 Spring Dancing 101 May 1, 2009 May 28, 2009 1274467708 23

I’ll be using this method for my upcoming project.

Option 3: Store serialized information

Rather than storing rows when information is modified, the entire encapsulating object is serialized and stored. This can be done either with one table per object type, or in one large history table.

The advantage here is that each revision stands alone and complete. Composite records are stored complete, and in a single location. This makes obtaining an accurate history relatively easy. On the downside, since information is serialized it’s hard to locate specific pieces of information via sql. This approach seems perfect for complex objects where a single composite record spans many tables, but a little overkill for this specific case.
classes

id version active session title start_date end_date timestamp login_id

1 2 1 Spring Dancing 101 May 1, 2010 May 28, 2010 1274467709 23

2 1 1 Summer Dancing 102 June 1, 2010 July 15, 2010 1274467710 27

3 1 1 Fall Dancing 101 September 1, 2010 September 28, 2010 1274467808 35

history

id version table data

1 1 classes a:8:{s:2:"id";i:1;s:7:"version";i:1;s:7:"session";s:6:"spring";s:5:"title";s:11:"Dancing 101";s:10:"start_date";s:11:"May 1, 2009";s:8:"end_date";s:12:"May 28, 2009";s:9:"timestamp";s:10:"1274467708";s:8:"login_id";i:23;}

I'll consider this for larger projects with complex objects.

Other points to ponder:

How long will you store histories?

Forever (∞)

Some fixed period of time (30 days)

Application specific (e.g. storing change to a class for three months after it ends)

Storage engine for history

MyISAM

Archive

Backup strategies

History tables are all appends

Thanks to everyone who commented on the last post.

The accountability problem

nospam@example.com (Paul Reinheimer) — Fri, 21 May 2010 13:39:43 -0400

I'm working on a volunteer side project for a local dance studio, the website will handle the standard sort of things: creating classes, registering students, matching leads to follows, etc.

We would like to have some level of accountability when various records (user, class, etc) are modified. That way if problems arise later, we can see who did what, and when. If needs be, even roll the change back.

This must be a common problem, how have you solved it?

My (proposed) solution is something like this: Take an ordinary table like class:

id session title start_date end_date

1 Spring Dancing 101 May 1, 2010 May 28, 2010

2 Summer Dancing 102 June 1, 2010 July 15, 2010

3 Fall Dancing 101 September 1, 2010 September 28, 2010

And add a few columns to for change management

id version active session title start_date end_date timestamp login_id

1 1 0 Spring Dancing 101 May 1, 2009 May 28, 2009 1274467708 23

1 2 1 Spring Dancing 101 May 1, 2010 May 28, 2010 1274467709 23

2 1 1 Summer Dancing 102 June 1, 2010 July 15, 2010 1274467710 27

3 1 1 Fall Dancing 101 September 1, 2010 September 28, 2010 1274467808 35

The list of classes is obtained by selecting where the active bit is set. Administrators will be able to look at the list of changes and see who made them, and when. I’m tracking login id rather than simply user id so information like IP, and timestamp can be associated with the login, rather than simply who. In the case of reverting a change, the desired version will be duplicated into a new row, and have the active bit set. In reality active will always be the highest revision number (and latest date) but the duplication makes queries easier.

I think there’s a lot of power in this approach, but it is a bit complicated. Thoughts?

A GUI for XHProf

nospam@example.com (Paul Reinheimer) — Mon, 26 Apr 2010 14:54:58 -0400

Facebook was kind enough to open source the XHProf extension last year, but it flew under my radar until I saw a presentation including it earlier this year when they showed off HipHop. XHProf provides profiling information about your application, while being lightweight enough to run on a production server (against a percentage of requests). Once we got it installed we ran into a few limitations with the existing GUI. With the help of Graham Slater (one of our front end designers) I hacked on the Facebook code to come up with our own flavor. Let me stress the word “hack." I was given some really tough deadlines for this project so functionality was literally hacked into the existing codebase.

Key Features:

Nice interface

MySQL backend for data storage

Stores key values such as Peak Memory Usage, Wall Time, and CPU Ticks in the database directly for easy lookup.

Stores Get, Post, and Cookie data for easy comparison (so you can determine why a run hit that execution path)

Display includes easy lookup for worst runs, most recent, etc.

Support for profiling a percentage of requests, or on demand

Support for not including a link to profile requests on certain files (like xml documents, images, etc).

Uses Google Data Visualization API to graph runs over time

Concept of “Similar” urls, to handle ?story=23 and ?story=24 having different URLs, but similar to identical code execution paths

Screenshots:

Download:

Please download the source code from GitHub

Warnings:

Seriously, we hacked this up! Please let me know if you encounter any bugs, there's probably a bunch.

Memory usage in PHP

nospam@example.com (Paul Reinheimer) — Fri, 19 Mar 2010 15:38:54 -0400

A colleague called me over today for some help with a memory usage issue in his PHP script. The script was performing the basic (but critical) task of importing data, pulling it in from MySQL in large chunks, then exporting it elsewhere. He was receiving the wonderful "Fatal error: Allowed memory size of XXXX bytes exhausted (tried to allocate YY bytes)" error message.

The code was following a basic flow:
10 Get stuff from the database (large array of arrays)
20 Iterate over it, doing stuff
30 Goto 10

I fixed the memory usage exceeded problem with an unset(), at the end of a loop.

Take a look at this sample program: loopOverStuff(); function loopOverStuff() { $var = null; for($i = 0; $i < 10; $i++) { $var = getData(); //Do stuff } } function getData() { $a = str_repeat("This string is exactly 40 characters long", 20000); return $a; }

The important thing to realize is that PHP will end up needing around twice as much memory as getData() takes. The problem is the line $var = getData(). The first time it is called $var is incredibly small, it's clobbered and the return value of getData()is assigned to it. The second time through the loop $var still holds the value from the previous iteration, so while getData() is executing you're maintaining the original data (in $var), and a whole new set (being built in getData()).

Fixing this is incredibly easy: function loopOverStuff() { $var = null; for($i = 0; $i < 10; $i++) { $var = stealMemories(); //Do Stuff unset($var); } }
This way we avoid the duplication in memory of those values on that line. To see this happen in more detail take a look at this sample script with ouptut: memory-usage-example.php.

This isn't critical, except when it is. Once loopOverStuff() completes, and the function ends. The memory is released back to the rest of PHP automatically. You'll only run into problems where Other Stuff + (2 * memory needed in loop) > Memory Limit. There are better architectures available to avoid the issue entirely (like not storing everything in the array, just to iterate over it later) but they're an issue for a different post.

For a very simple base case demonstration of the issue take a look at the simple example.

Search without the Database

nospam@example.com (Paul Reinheimer) — Fri, 12 Mar 2010 17:27:35 -0500
I gave my second talk at #ConFoo today, this one was on Searching without the Database. This talk was based on a situation at work where I replaced a MySQL search solution with a Sphinx + Memcached solution for higher performance. If you're interested, here's the slides: Search without the DB - ConFoo 2010.pdf.

If you attended my talk, you can rate it on the Joind.in Page

ConFoo - PHP in the Enterprise

nospam@example.com (Paul Reinheimer) — Wed, 10 Mar 2010 16:24:43 -0500
I just finished my PHP in the Enterprise talk at ConFoo, Slides are available here: PHP in the Enterprise - ConFoo March 2010

A Customer found a Bug

nospam@example.com (Paul Reinheimer) — Mon, 11 Jan 2010 21:20:41 -0500
So this morning I woke up to an email (well, several, but only one is relevant) from a WonderProxy customer. They had discovered a bug in the website. A rather serious bug mind you, as it stopped them from adding servers to their account, and thus prevented them from actually using the service.

Here’s how I approached the problem:

Reproduce the issue
I logged into the website as the customer and attempted to reproduce the issue, no problems. I tried again on the development version of the website and reproduced it there as well.

Estimate bug fix
I took a look at the code and estimated that I could resolve the bug in under ten minutes. The code base for the WonderProxy website is rather small, and bugs there can’t affect customer usage (unless they break the DB) so I wasn’t too concerned about accidentally taking something down. The short time frame for release also let me fix the bug before contacting the customer.

Fix the bug
Found the bug, fixed it. Tested it out in development, checked other accounts to look for unrelated breaks. Finally I rolled it out from development to production and checked again.

Contact Customer
Replying to the customer I indicated that problem was indeed on our end (the customer wasn’t using the site wrong or anything), and that it had been fixed. Additionally I had updated the billing period to reflect the fact that the account hadn’t been usable up until the fix was applied. As a token of appreciation I upgraded the account slightly.

Overall, I found the process rather pain free. When I look at it from a customer’s prospective, I also feel like it was a reasonable problem resolution.

I’d like to delve deeper into how this problem made it onto my production website, but that’s a separate post. For now I’m really just looking at the customer service side, there was a total of 67 minutes between the initial contact (on the part of the customer) and my reply with the fix in place.

Thoughts? How did I do?

.htaccess to httpd.conf

nospam@example.com (Paul Reinheimer) — Mon, 12 Oct 2009 11:16:38 -0400
In the past week I've dealt with some servers that were seeing some rather heavy load, with more traffic on the way. I noticed that the servers had AllowOverride All in their httpd.conf files, which immediately struck me as a problem. While .htaccess files can be quite handy during development (AllowOverride instructs Apache to process those .htaccess files), they're also quite horrible in production due to the performance hits involved in using them. Since the files allow configuration to be changed on the fly, they need to be read on every request. This requires a lot of unnecessary stat calls with each and every request.

I approached the development team requesting permission to turn the option off, in order to squeeze some extra performance out of each server. Rather than the happy smiles and agreement I expected, I was instead met with nervousness and uncertainty. Since the front end team uses them constantly, there was concerns that the change over from .htaccess to httpd.conf would consume a lot of time, and would be very difficult to maintain moving forward.

In order to allay their fears I created a .htaccess to httpd.conf script with the help of my friend Rich Bowen. It's designed to be executed from the command line from a sufficiently permissioned user from within the document root. It recursively scans all sub directories looking for .htaccess files, reads them in, and spits it all back out. There's a bit of intelligence involved in order to present the contents of .htaccess files with the least depth first (this way overriding works correctly), and warnings are generated if the RewiteBase instruction is encountered as it may require extra tweaking. Other than that I generally just pipe the output to htaccess.conf, then include that file from my basic configuration file. By re-running this script (and, of course, gracefully restating the server after checking to make sure all the conf files parse) as part of my normal sync process I get all the flexibility of .htaccess files, without the performance hit.

The script requires PHP 5, probably 5.1+ would be safest.

No warranties, test extensively. In terms of field testing (against my rather limited set of .htaccess files) it's been running on a site +10M uniques/day, using roughly 40 .htaccess files. I'd love to hear from you if you end up using it.

Source:.htaccess to httpd.conf converter, and again with thanks to Rich Bowen.

Mobile Content Security

nospam@example.com (Paul Reinheimer) — Wed, 09 Sep 2009 11:25:00 -0400

Securing content without proxying through PHP is a pretty common, and easily solved problem. You get Lighttpd (or any other webserver) to check some sort of a hash when it serves content. In the hash you include the users IP address, a timestamp, and probably the path to the file. This solution works remarkably well for the average desktop user. Sure there's outliers behind non-sticky load sharing proxies, but most people confidently ignore that portion of their user base.

Mobile users present a more serious problem. Their IP changes more frequently, under a large number of circumstances. As users move from tower to tower, or as their carrier changes moods their IP changes, generally within the same subnet. That last statement there provides almost a tease as to a possible solution: only hash a portion of the IP address to allow for greater flexibility.

Problems however, go a bit deeper.

As devices get smarter, we're now able to do nifty things like surf the web at home (on our mobile device) using our WiFi connection. As we walk outside and down the street the device seamlessly fails over to the 3G, Edge, or whatever connection and our browsing session continues. The browser being blissfully unaware of the change can do nothing to help us. During a test over the weekend, I saw tens of thousands of instances of switching within the same subnet, and a few thousand instances of changes beyond. This is a real problem for a mobile site.

In order to solve the problem I've had to balance a few needs: our need to provide even a modicum of protection for our content, the needs of our users to have links work when they view a page. To balance these I've added some checks to our software that determine if a user's IP changes, when those changes are detected the secure links generated omit the IP address but use a far more aggressive timeout. Hopefully balancing our need to protect content, with our desire to serve it to expectant customers.

Have you solved this problem differently?

Top 5 Reasons Windows shops should consider WIMP over LAMP

nospam@example.com (Paul Reinheimer) — Tue, 14 Jul 2009 13:07:00 -0400
Disclaimer, I'm no Windows Fanboy: I use a Mac at home, and at work deploy code to over a hundred LAMP servers on three continents (using a Windows desktop).

I'm always looking for better ways to get my job done, save my employer money, and expand my skill set. So whenever I get a chance I'm looking at the competition of the tools we're currently using to see if there's tricks to learn, or reasons to switch. I've been keeping an eye on IIS on various Windows servers for a while, more recently I've been looking at SyBase and SQL Anywhere, as well as CouchDB for data storage (but more on those two later).

So: Windows, WIMP, and why I think you should be giving it a serious look if you're currently a Windows shop.

1. Use the expertise you already have in-house
If you're currently running Windows you already know how to: install patches, restart machines, manage security and ACLs, connect and administer, get help, etc. When I say you I mean you as an organization, not an individual.

When I was younger, and slightly more full of stupid than I am now. I made probably the stupidest decision of my professional life (thus far). I was in a Windows shop running Windows servers for everything, with quite a few applications running through Lotus Notes. I was tasked with building a new Intranet. Rather than re-training myself (one person, and an intern at that) to use the monstrosity that was Lotus Notes R5, I somehow managed to convince my boss that we should put a Linux machine in the server room. The project for the most part worked: I built a decent (by my then current standards) Intranet system for the company.

I guarantee the system didn't survive six months after my departure.

Why? Because maintaining it required everyone else (many more than one) in IT to learn what I was good at. They had to re-train themselves to maintain a single system in a network of many. It simply wasn't worth the effort.

2. Avoid duplication of Servers
When you introduce a new stack, you're inevitably introducing a new server. That's an extra server to maintain, backup, power, upgrade, manage, etc. If you're doing a small roll out there's inevitably a machine in the room already that's not seeing a lot of load, just throw it on there. If it's an entirely different stack your current solutions for a lot of the problems aren't going to work (your backup scripts and applications for Windows won't auto-magically migrate over to Linux).

If you read my first story, you can see how many additional problems I caused there. Backing up the machine was going to take extra effort. It was another slot in the rack that couldn't be used for anything else (despite ridiculously low load).

3. Avoid duplication of Staff
When you're running multiple stacks for important or critical applications, you need staff to support them. If you've got Windows servers and Linux servers you've got staff supporting them both. Multiple sys-admins, multiple people on-call, multiple sets of vacations to manage.

At my current employer, this is our current situation, as we have linux servers for the web, and Windows for everything else. This means we have two sets of system administrators, one for the Windows stuff, the other for Linux. We've got multiple pagers so one person from each team can be on call 24x7. I'm not suggesting this is bad per se, but it is an additional cost.

4. Integrate tightly with current technologies
When you're consistently running systems of a similar stack, things just play nicely with each other. Reporting applications, backup, information sharing, etc. It all just works. You don't need a different username password combination for your development server as you do for your machine logon, and another one still to restart a service. If your security policies permit (and it seems wise) you can centralize this.

At work we have passwords, many of them. We mount dev systems over semi-reliable samba shares, with unique logins and directory structures not entirely tied to our general login accounts. This causes both groups of sysadmins to come together for every new hire to create the appropriate directories and entries in various configuration files.

5. There's features out there not available on LAMP
Many of us have become overly complacent in the thought of Microsoft as a follower in the web arena, products like Internet Explorer aren't doing much to shake this. IIS 7 is more than a follower, in a lot of realms I feel it has advantages over the technologies I'm used to using. In particular I might call out their XML based configuration system, and per-directory configuration without the performance hit. In terms of configuration XML has a lot of advantages over .ini, in terms of parsing, grouping of elements, parent child relationships, etc. They're also easy to manipulate programatically with predictable results which allows Microsoft and others to create great user interfaces for their maintenance and upkeep (i've never found a graphical apache configuration tool I didn't hate). For per-directory configuration think .htaccess files, only with more power (like virtual hosts), and without the performance hit. Most production sites turn off .htaccess support (AllowOverride None) because otherwise you've got a slew of needless stat calls on every request, with IIS 7 you leave them on, the server simply subscribes to the files so changes are pushed into the application by the operating system, rather than being pulled on each request. You've essentially got drag and drop-able applications (since the entire app, configuration, data (SQLite), and all can reside on disk, in one folder) with no performance problems. Streaming media is also very easy on IIS servers, throw in the media services plugin and you get a wealth of congfiguration options, a process that's still difficult to set up efficently on Apache.

At work we end up leaving .htaccess support on a lot, mostly for some of our more frequently (daily) changing sites. By changing daily not only do I mean it, I also mean changes beyond simple database updates, drastic changes to code and presentation happen regularly. Rather than lean on our hosting provider for frequent restarts we've ended up leaving .htaccess support enabled. To manage the performance hit these pages are relegated to their own machines, in separate clusters from the rest of our production sites. Managing system configuration with all the various virtual hosts has become so complex that only a few staff members are trusted with the task.

In conclusion, and to be perfectly clear. I'm not suggesting that WIMP is the right solution for you. I'm merely encouraging that you pull your head out of the sand and consider your options. If you're already running a substantial Windows server room, there may be more reasons to stay, than to switch.

Storing Sessions in Memcache (how everything behaves)

nospam@example.com (Paul Reinheimer) — Fri, 10 Jul 2009 12:35:44 -0400

I’ve been using memcached to store session data for the past while, but we ran into a few problems at work that led me to dive in a bit deeper and see how PHP, Sessions, and memcached play along.
Short Version
When you start up a session a new entry is created in memcache, the expiry is the same as session.gc_maxlifetime. Session data is retrieved each time a session is started (either with session_start(), or if you have session.auto_start enabled, etc).
Long Version:
When session_start() is called for a new user several things happen very quickly:

A session ID is generated for the end user, this is seen in the HTTP response headers.
PHP Contacts memcached and attempts to retrieve any information in that particular session ID get c446fc5309c3b8d3aa6e90343dddab29
Memcached returns no data for that entry, as it doesn’t exist yet.
Your page executes, setting data in the session
PHP pushes data into memcached: set c446fc5309c3b8d3aa6e90343dddab29 0 1440 32 name |s:6”asdasd;bar|s:3:”foo”;

On every subsequent page where a session is used, a very similar sequence of events is executed. When the session is started PHP contacts memcached and asks for the session data. When the page is complete it pushes the data now in the session to memcached. That last step is kind of critical, because it’s what’s controlling the expiry of the data within memcached. Take a quick look at the data being set once more.
set c446fc5309c3b8d3aa6e90343dddab29 0 1440 32 name |s:6”asdasd;bar|s:3:”foo”;
Data is being set, the key is your session id, 0 is a set of flags that the client can use to mark the data (often used to mark data that’s been compressed). The 1440 represents the expiry time for the data, 1440 seconds from now (24 minutes). That value was taken directly from session.gc_maxlifetime . It’s important to keep in mind that what memcached guarantees is that the data will not be available after that expiry runs out, but it promises nothing about how long it will be available. If your memcached instance crashes, or runs out of memory you’re going to start losing data a lot sooner. Finally we have the data in a somewhat truncated version of PHP’s serialize.

The fact that the data is sent to memcached after every request, even if you haven’t changed what’s stored in the session, is rather important. It’s what makes your session last for session.gc_maxlifetime after the user last did something, rather than simply that amount plus the last time you requested a document.

You can’t maintain code you can’t easily understand

nospam@example.com (Paul Reinheimer) — Thu, 25 Jun 2009 16:24:02 -0400
I just helped a co-worker with a problem in a sorting mechanism inside Javascript. Here’s the snippet that caught my eye:
var ret = (a.l > b.l) ? -1 : ((a.n.toLowerCase() < b.n.toLowerCase()) ? -1 : 1);
I looked at the code, and stated: I don’t like nested ternaries, it’s too easy to mess up or miss something. My co-worker explained that the alternative was too long and drawn out, this is how it’s always done. I pondered the code for a moment, and asked what happened when a.l < b.l, as the case seems to be missing. That was the bug.
The people I’m working with are all smart, great developers. We all miss things, tragically we’re not perfect. When you miss things, you need to be able to find them, writing overly complex code (I would consider nested ternaries an example of overly complex code) makes it really easy to find them.
I’m slowly working on improving our internal coding standards document with additional rules, at the time being I’m considering:

Don’t use GOTO

Don’t use nested ternaries

Document the heck out of variable variables ($$variable), or variable function calls ($this->$method)

Write commit messages

Any other rules you follow?

Note, while I am strongly discouraging the use of GOTO, I do not think adding it to PHP was a mistake. There are good use cases for the language construct, I just feel it’s an easy thing to fall back on when the real solution is to build your code in a more intelligent manner.

Memcached Constants - Lame Code

nospam@example.com (Paul Reinheimer) — Thu, 18 Jun 2009 14:40:31 -0400
Please find below an incredibly lame piece of code to print out the values of all the memcached constants defined on their doc page. I was running into a few errors, and getResultCode() was obligingly returning the value of the constant, rather than the constant itself. I had to look up what that value meant. For some reason I couldn't reflect the extension to get the constants and values thereof, so with some lame scraping you get the following. Code under the jump, answers (as of our version of various libraries) here:

Memcached::OPT_COMPRESSION - -1001
Memcached::OPT_SERIALIZER - -1003
Memcached::SERIALIZER_PHP - 1
Memcached::SERIALIZER_IGBINARY - 2
Memcached::SERIALIZER_JSON - 3
Memcached::OPT_PREFIX_KEY - -1002
Memcached::OPT_HASH - 2
Memcached::HASH_DEFAULT - 0
Memcached::HASH_MD5 - 1
Memcached::HASH_CRC - 2
Memcached::HASH_FNV1_64 - 3
Memcached::HASH_FNV1A_64 - 4
Memcached::HASH_FNV1_32 - 5
Memcached::HASH_FNV1A_32 - 6
Memcached::HASH_HSIEH - 7
Memcached::HASH_MURMUR - 8
Memcached::OPT_DISTRIBUTION - 9
Memcached::DISTRIBUTION_MODULA - 0
Memcached::DISTRIBUTION_CONSISTENT - 1
Memcached::OPT_LIBKETAMA_COMPATIBLE - 16
Memcached::OPT_BUFFER_WRITES - 10
Memcached::OPT_BINARY_PROTOCOL - 18
Memcached::OPT_NO_BLOCK - 0
Memcached::OPT_TCP_NODELAY - 1
Memcached::OPT_SOCKET_SEND_SIZE - 4
Memcached::OPT_SOCKET_RECV_SIZE - 5
Memcached::OPT_CONNECT_TIMEOUT - 14
Memcached::OPT_RETRY_TIMEOUT - 15
Memcached::OPT_SEND_TIMEOUT - 19
Memcached::OPT_RECV_TIMEOUT - 15
Memcached::OPT_POLL_TIMEOUT - 8
Memcached::OPT_CACHE_LOOKUPS - 6
Memcached::OPT_SERVER_FAILURE_LIMIT - 21
Memcached::HAVE_IGBINARY - #&UNDEFINED;#
Memcached::HAVE_JSON - #&UNDEFINED;#
Memcached::GET_PRESERVE_ORDER - 1
Memcached::RES_SUCCESS - 0
Memcached::RES_FAILURE - 1
Memcached::RES_HOST_LOOKUP_FAILURE - 2
Memcached::RES_UNKNOWN_READ_FAILURE - 7
Memcached::RES_PROTOCOL_ERROR - 8
Memcached::RES_CLIENT_ERROR - 9
Memcached::RES_SERVER_ERROR - 10
Memcached::RES_WRITE_FAILURE - 5
Memcached::RES_DATA_EXISTS - 12
Memcached::RES_NOTSTORED - 14
Memcached::RES_NOTFOUND - 16
Memcached::RES_PARTIAL_READ - 18
Memcached::RES_SOME_ERRORS - 19
Memcached::RES_NO_SERVERS - 20
Memcached::RES_END - 21
Memcached::RES_ERRNO - 26
Memcached::RES_BUFFERED - 32
Memcached::RES_TIMEOUT - 31
Memcached::RES_BAD_KEY_PROVIDED - 33
Memcached::RES_CONNECTION_SOCKET_CREATE_FAILURE - 11
Memcached::RES_PAYLOAD_FAILURE - -1001

Continue reading "Memcached Constants - Lame Code"