Paul's Überfeed 2.0

[last.fm] Derek and Clive – Non-Stop Dancer / My Mum Song

Fri, 06 Nov 2009 11:44:18 -0800

http://www.last.fm/music/Derek+and+Clive

[last.fm] Derek and Clive – Alfie Noakes

Fri, 06 Nov 2009 11:40:54 -0800

http://www.last.fm/music/Derek+and+Clive

[last.fm] Marillion – Assassing

Fri, 06 Nov 2009 11:33:50 -0800

http://www.last.fm/music/Marillion

[last.fm] Flight of the Conchords – Foux du Fafa

Fri, 06 Nov 2009 11:27:00 -0800

http://www.last.fm/music/Flight+of+the+Conchords

[last.fm] Flight of the Conchords – Foux du Fafa

Fri, 06 Nov 2009 11:27:00 -0800

http://www.last.fm/music/Flight+of+the+Conchords

[last.fm] Mansun – Getting Your Way

Fri, 06 Nov 2009 01:53:25 -0800

http://www.last.fm/music/Mansun

[last.fm] Mansun – My Idea of Fun

Fri, 06 Nov 2009 01:48:55 -0800

http://www.last.fm/music/Mansun

[last.fm] Mansun – Skin Up Pin Up

Fri, 06 Nov 2009 01:44:53 -0800

http://www.last.fm/music/Mansun

[last.fm] The Smiths – The Headmaster Ritual

Fri, 06 Nov 2009 01:36:06 -0800

http://www.last.fm/music/The+Smiths

[last.fm] The Smiths – The Headmaster Ritual

Fri, 06 Nov 2009 01:36:06 -0800

http://www.last.fm/music/The+Smiths

[highscalability] High Scalability - High Scalability - Paper: No Relation: The Mixed Blessings of Non-Relational Databases

Tue, 03 Nov 2009 00:03:07 -0800

[70cities] Notes on making iTunes play nicely with an Airport Disk and Airtunes

Paul — Mon, 26 Oct 2009 16:00:00 -0700

OK, so six months later, along comes another post. This time it's practical, and a result of the experiences I had recently in trying to downsize my CD collection and shift all of my music onto a shared NAS drive. As it happens, our network at home is already based around an Airport Extreme sase station that supports the attachment and sharing of USB disks as network drives - so I purchased a cheap-arse Western Digital 1TB USB external drive, formatted it as HFS+ (non-journalled), shoved it into the back of the AEBS and away I went..

Setup: iTunes 8.2, Macbook 2nd Gen, OS X Tiger 10.4.11, Airport Extreme, Airport Express with Airtunes.

1. Telling iTunes what to do

Changing the location of the iTunes Library in the preferences and then using 'Consolidate Library..' is by far the easiest option for physically moving all your songs. See Apple's official guide here. All your metadata is preserved and it all Just Works. One pain - you have to copy over the network, so ideally, connect your Mac to the base station using a network cable rather than using wireless for this move as it'll be much, much quicker.

An alternative, do-it-yourself approach that gets you more control over where your music lives is manually editing the paths in the iTunes Music Library.xml file using a text editor. You still have to delete the actual library database and re-import the XML file into iTunes. This way you can connect the eventual NAS drive to your Mac's USB port, copy all the music across, connect it to the AEBS and hammer iTunes into doing the right thing by changing the paths in the XML file. However, for novices or people without much time on their hands, I wouldn't recommend this approach. YMMV.

2. Don't put everything on the NAS

Keeping your album art & iTunes databases on your Mac's local disk seems to be the smart option regardless of how you migrate your music (the Consolidate Library method works this way). You can't shove the whole of iTunes' working files on the NAS and expect that to be speedy - or, even worse, assume that you can share a single library between two instances of iTunes in this way. This is because iTunes continually updates its database files as it's working.

3. Some things are just going to take longer

Wireless just isn't as good as wired for most things. With the convenience of sending data around wirelessly comes certain trade-offs. iTunes may feel less snappy when navigating your library and jumping between songs, as it waits for the wireless disk to catch up. This will annoy you for a week or two, but then the great adaptive nature of the human brain kicks in and you'll just get used to it.

4. Check your wireless network

The performance of your wireless network is by far the biggest factor in making iTunes not suck donkey's balls with this setup. Connected via wired Ethernet, you should find day-to-day performance (including CD ripping) pretty much on par with having the library on local disk, with maybe a second or two of lag occasionally.

Over Airport, though, it's a different story. If your Airport network is not performing well - something you might not usually realise if all you ever do is access and download files from the Internet, as your Internet connection will always be much slower than the speed your wireless network ought to be capable of.

When iTunes attempts to play a file, it will buffer a certain amount of the file into memory before it starts playing it, so achieving low network latency is essential for this process to appear snappy enough to not be annoying - nothing's going to irritate you more than iTunes beachballing every time it moves from track to track.

Things to check:

Your wifi mode - 802.11b/g/n ?
Interference from nearby networks - use iStumbler or a similar wireless network tool to see if certain channels are crowded with networks, and move your network to a less busy channel.
Overall signal strength - is your AEBS positioned optimally? Are all wireless clients getting a good signal? Use the statistics function of Airport Utility (hidden away in the 'Advanced' tab of 'Manual Setup') to see how your Airport network is performing and be prepared to move the base station(s) around to achieve better speeds/less interference.

5. Airtunes bumpiness

Solving point 3 should also solve any AirTunes performance issues you might be having. AirTunes works by streaming a losslessly-encoded version of the music stream to the Airport Express base station, which in itself should not tax a modern 802.11g (or better) network but a weak signal can increase drop-outs and stuttering. By moving the AEBS a little higher and the Express a little nearer to get the best signal-to-noise ratio, I've had it working pretty seamlessly.

An edge case to be aware of is if you have any music encoded in any non-iTunes native codecs - that is, any other than MP3, AIFF or AAC - it cannot be sent over AirTunes. Even through iTunes can play Ogg Vorbis, etc via Quicktime - it can't send files encoded in these formats over AirTunes, which is annoying when files such as these show up in playlists. Even more annoying is that if it's streaming to a set of remote speakers and encounters such a track, iTunes doesn't just skip it - it'll play it out loud on the local computer, whilst the remote speakers go silent!

Anyway, I hope that's of use to some people struggling to get this sort of setup working. It's still a little way off being the perfect wireless-music-anywhere nirvana I was gunning for, but it does feel like I've freed up my listening a bit.

[70cities.tumblr] San Francisco 1958 on Vimeo (via Vimeo)

Sat, 24 Oct 2009 04:14:46 -0700

San Francisco 1958 on Vimeo (via Vimeo)

[flickr] Temporary

nobody@flickr.com (digitaltrickery) — Tue, 06 Oct 2009 05:12:21 -0700

digitaltrickery posted a photo:

A luxury of bus stops in Mortlake. This isn't confusing at all.

[70cities.tumblr] CBRD - Histories - From War to Worboys

Wed, 30 Sep 2009 00:27:44 -0700

CBRD - Histories - From War to Worboys

[70cities.tumblr] Information overload | Technology | The Guardian

Thu, 24 Sep 2009 00:08:42 -0700

Information overload | Technology | The Guardian

[70cities.tumblr] A hundred dance moves per minute: Where Is Beatles Band

Thu, 10 Sep 2009 02:58:39 -0700

A hundred dance moves per minute: Where Is Beatles Band

[flickr] iPlayer in Neon

nobody@flickr.com (digitaltrickery) — Wed, 09 Sep 2009 02:28:23 -0700

digitaltrickery posted a photo:

[70cities.tumblr] A road map to success

Tue, 08 Sep 2009 03:05:36 -0700

A road map to success

[70cities.tumblr] The Nutella maki at Sushi or Death

Sun, 06 Sep 2009 03:45:37 -0700

The Nutella maki at Sushi or Death:

Yum!

[70cities.tumblr] The many ways in which early Twitter has been like early...

Sat, 05 Sep 2009 06:44:30 -0700

The many ways in which early Twitter has been like early blogging (via Meg Pickard)

[70cities.tumblr] BBC iPlayer - Record Producers: The Extended Cut: Sir George Martin

Sat, 05 Sep 2009 06:40:54 -0700

BBC iPlayer - Record Producers: The Extended Cut: Sir George Martin

[flickr] At the Japanese Tea Garden

nobody@flickr.com (digitaltrickery) — Sat, 05 Sep 2009 04:09:32 -0700

digitaltrickery posted a photo:

I took loads more photos here, but they all looked crap. You just can't capture the depth and textures. It was all very beautiful and calming.

[flickr] Lunchtime Hoedown in Yerba Buena

nobody@flickr.com (digitaltrickery) — Sat, 05 Sep 2009 04:08:08 -0700

digitaltrickery posted a photo:

Nice idea - gee up the natives during their lunch hour with a bit of fiddle.

[flickr] Fog over the Bay Bridge from Sausalito

nobody@flickr.com (digitaltrickery) — Sat, 05 Sep 2009 04:06:53 -0700

digitaltrickery posted a photo:

[flickr] Wine in Sausalito

nobody@flickr.com (digitaltrickery) — Sat, 05 Sep 2009 04:06:18 -0700

digitaltrickery posted a photo:

It was pretty rank wine - but the occasion and spot was wonderful.

[flickr] Richardson Bay at Sausalito

nobody@flickr.com (digitaltrickery) — Sat, 05 Sep 2009 04:05:26 -0700

digitaltrickery posted a photo:

[flickr] P1020026

nobody@flickr.com (digitaltrickery) — Sat, 05 Sep 2009 04:04:20 -0700

digitaltrickery posted a photo:

[flickr] Eat Toast!

nobody@flickr.com (digitaltrickery) — Sat, 05 Sep 2009 04:02:11 -0700

digitaltrickery posted a photo:

[flickr] P1020009

nobody@flickr.com (digitaltrickery) — Sat, 05 Sep 2009 04:01:12 -0700

digitaltrickery posted a photo:

[flickr] Tiling

nobody@flickr.com (digitaltrickery) — Sat, 05 Sep 2009 04:00:18 -0700

digitaltrickery posted a photo:

[flickr] Me and Iced Coffee

nobody@flickr.com (digitaltrickery) — Sat, 05 Sep 2009 03:59:05 -0700

digitaltrickery posted a photo:

Very welcome after a day wandering in the sunshine.

[flickr] Up

nobody@flickr.com (digitaltrickery) — Sat, 05 Sep 2009 03:58:08 -0700

digitaltrickery posted a photo:

Not quite a vanishing point.

[flickr] Muni tracks

nobody@flickr.com (digitaltrickery) — Sat, 05 Sep 2009 03:55:57 -0700

digitaltrickery posted a photo:

At the top of Chattanooga.

[flickr] Balmy Alley

nobody@flickr.com (digitaltrickery) — Sat, 05 Sep 2009 03:54:49 -0700

digitaltrickery posted a photo:

[flickr] Balmy Alley

nobody@flickr.com (digitaltrickery) — Sat, 05 Sep 2009 03:53:34 -0700

digitaltrickery posted a photo:

[flickr] Cactus in Balmy Alley

nobody@flickr.com (digitaltrickery) — Sat, 05 Sep 2009 03:52:11 -0700

digitaltrickery posted a photo:

[flickr] Balmy Alley

nobody@flickr.com (digitaltrickery) — Sat, 05 Sep 2009 03:50:57 -0700

digitaltrickery posted a photo:

[flickr] Vegan?

nobody@flickr.com (digitaltrickery) — Sat, 05 Sep 2009 03:49:34 -0700

digitaltrickery posted a photo:

[flickr] Balmy Alley

nobody@flickr.com (digitaltrickery) — Sat, 05 Sep 2009 03:48:20 -0700

digitaltrickery posted a photo:

Fantastic murals - there's a lot in Mission, but these are the biggest.

[wilsondan.co] How 31 Year Olds Consume Media : Dan Wilson's Blog

Wed, 29 Jul 2009 00:26:07 -0700

[moshcam] Sia live at the Metro Theatre

Sun, 26 Jul 2009 07:13:23 -0700

Full-length gig video..

[folksy] Folksy :: Buy "'Bad Music' brown men's/unisex t-shirt"

Sun, 26 Jul 2009 02:25:57 -0700

[dbmsmusings.blogspot] DBMS Musings: Announcing release of HadoopDB (longer version)

Wed, 22 Jul 2009 00:25:39 -0700

[agiledata] Agile Enterprise Architecture

Tue, 21 Jul 2009 12:46:19 -0700

[70cities.tumblr] Scary Go Round Shop :: random Tshirt

Fri, 10 Jul 2009 00:15:16 -0700

Scary Go Round Shop :: random Tshirt:

Whilst I have a growing dislike for ironic t-shirts, this is in no way ironic, and I want it.

[70cities.tumblr] Warming Up | Richard Herring.com

Thu, 09 Jul 2009 00:34:19 -0700

Warming Up | Richard Herring.com:

Richard Herring gets his iPhone nicked. I can’t say I have much sympathy - anyone who walks around with anything valuable on display in Shepherd’s Bush is proffering it to any passing chancer, of which there are many.

[70cities.tumblr] BBC iPlayer - James May at the Edge of Space

Sat, 04 Jul 2009 13:01:51 -0700

BBC iPlayer - James May at the Edge of Space:

Astonishing.

[70cities.tumblr] Trailer Park 09 , Aberdeen - a set on Flickr

Wed, 27 May 2009 11:17:47 -0700

Trailer Park 09 , Aberdeen - a set on Flickr

[70cities.tumblr] Julian Glover on the future of railway branch lines

Mon, 13 Apr 2009 01:12:54 -0700

Julian Glover on the future of railway branch lines

[70cities.tumblr] The Official George W. Bush Presidential Librarium

Sun, 05 Apr 2009 04:43:03 -0700

The Official George W. Bush Presidential Librarium

[mondrian.pentaho] Pentaho Commercial Open Source Business Intelligence:

Tue, 20 Jan 2009 06:53:56 -0800

[reductivelabs] Big Picture - puppet - Trac

Wed, 14 Jan 2009 03:39:11 -0800

[infoq] InfoQ: Scalability Best Practices: Lessons from eBay

Thu, 08 Jan 2009 03:49:27 -0800

[lifescaler] Database sharding unraveled - part I : Lifescaler

Thu, 08 Jan 2009 03:33:46 -0800

[70cities] Start

Paul — Fri, 02 Jan 2009 09:34:00 -0800

Given how infrequently I bother posting here, I thought there'd at least be some amusement value at posting at the very beginning of the year. True to form, I'm already off by one day.

Anyway, happy 2009. Being thousands of miles away from home gives a strong feeling of dislocation, not just physically but psychologically. I lost track of the weekdays almost immediately when we arrived, irrelevant as they are when you've got nothing to do. And the fact it's a brand new year hasn't really registered yet. 2008 was difficult in many ways, so I'm not exactly sad to see the back of it and have the chance to use the division of a new year starting to attempt to try and sort out various things.

As if by chance, today marks the tenth anniversary of me starting my first professional web development job. In 1998/9, jobs that had anything to do with the internet were both rare and fuzzily defined at best - so when I landed a job as a 'Webmaster' at a forward-thinking advertising agency in Newcastle, 6 months after leaving university and kicking around in temp jobs, I was pretty excited. In retrospect, I was very lucky, in the right place at exactly the right time. To think I'd end up where I am now, doing what I'm doing, would have been certifiable insanity.

So, whilst I don't exactly yearn for the days of administering hundreds of sites being served from a tired Powermac running System 7.6 on the end of a dial-back ISDN line, and building them out of handcranked HTML, Lasso and Filemaker Pro (shudder), I'm grateful for the in-at-the-deep-end head start I got on things in my 18 months there.

We don't arrive back in the UK until Tuesday, so I've still got plenty of time to think about what happens this year. There are vague plans, and whole areas where I just haven't done the thinking yet. Meanwhile.. ten years. Blimey.

[70cities] Far, far away

Paul — Sat, 20 Dec 2008 16:07:00 -0800

The last few days were the hardest.

Never mind the months of late nights, missed deadlines, malfunctioning servers, storage constraints, crap data, uncertain roadmap, pressure, bad decisions (mostly my own), new surprises, misbehaving databases, etc etc. I work hard, I can work hard, I do work hard and I work on the kind of project that requires hard thinking, experimentation and ingenuity. So I can do all that - but once I knew that our long-since-planned break was around the corner, it got so much harder.

A week or so before we left, I half-joked to everyone that the moment we arrived, I'd get ill. It was inevitable, I said - indeed, it happened last time we came to San Francisco, like clockwork. The 11 hour plane ride - the longest stretch of time I'd spent doing nothing in several months - is the cue for my body to shut down all the defences, stop fending off all the bugs and actually get ill.

I finished up, made sure everyone knew vaguely what they had to do in my absence, worried about minor things that hadn't quite been done but were still niggling, unsure as to whether everything we'd achieved in the run up to Christmas was enough. Couldn't stop reading emails I'd already read twice, for nuances I might have missed. Then I was told in no uncertain terms to get the fuck out the office - in a nice way.

Home, pack, pizza. Sleep - fitfully. Wake, early. Rush around, ensure cat is OK, worry about cat, cab arrives. No time left.

M4, Heathrow, check in. Lounge - yay! Breakfast. Departure gate - upgrade! In your face! A good omen and a surprise which made the holiday feel that little bit extra special.

11 hours later, arrive. Tired, but excited. SFO, a greeting with antlers, cab to SoMa. Already, I can feel gunk building up at the back of my throat. Possibly just the air pressure or a reaction to something? No. It's a cold. The cold I've been putting off for months and months through adrenaline and exhaustion.

48 hours later - better. Jetlag's hit us like a hammer but staying up as late as we could and sleeping as late as we can has seen most of that off, hopefully. So nice to be back here again. Our gracious hosts are wonderful and we've rented, by accident, the world's most enormous car to dart around in. Haven't felt like drinking much due to the tiredness but I think I'll remedy that tonight. Mini-break in Sonoma hopefully sorted. Miles away - literally- from the pushing and pulling of our normal lives.

We needed this holiday and it's finally happened. Now - time to think. What's next?

[70cities] Beautiful URLs

Paul — Sat, 13 Dec 2008 05:22:00 -0800

http://service.o2.co.uk/IQ/SRVS/CGI-BIN/WEBCGI.EXE/,/?St=413,E=0000000002116867044,K=3744,Sxi=2,question=ref(User):str(Mobile),CASE=obj(12891)

Makes the heart sing, doesn't it?

[70cities] Bright Lights

Paul — Sat, 11 Oct 2008 03:20:00 -0700

I had yesterday off work (mostly anyway) and so spent most of the day alternating between boring-but-necessary chores and catching up on shows I'd bookmarked to watch/listen to. Bliss. In the evening, I headed out for a beer - ironically with people from work, although none of whom I actually work with, so that's OK. It's a big company, that can happen and I rather like it.

On the way to the pub I get suddenly redirected by the invitation to impromptu birthday drinks in town. I'm game, so I head to Covent Garden rather than Shepherd's Bush. It's not a hard choice to make.

I end up in a trendy basement lounge/bar arrangement - it's happy hour, and half full of local suits and fashionistas taking advantage of the half-price cocktails. The music's just one long anonymous boom-chk-boom affair, accurately blared out at the level at which it's impossible to have a reasonable conversation with anyone. I know a fair few people there who I've not seen in ages, and with whom I'd appreciate the chance to catch up with, but there's no chance of that. I sup a mojito, which took half an hour to procure and about three minutes to drink.

T, who's 8 months pregnant, shows up with her SO, who I've not met before. She introduces me as 'the smartest guy I know'. I offer a rebuttal, offering that I'm perhaps the tallest guy she knows. The half-joke gets lost in the din.

Hotpanted girls - I don't know if there's a name for them, or what their job titles could possibly be - float around and proffer test tubes full of God-knows-what. It's too loud to find out, and anyway my carefree vodka jelly days are very much over.

H and L show up, and we attempt to catch up. It's nice to see everyone but it's just not happening. Having made sure I've seen everyone for a hello, I slip quietly out and head home.

I'm hungry, so I scoff the tuna salad in the fridge, and am subsequently taken over by the spirit of Peter Rabbit. Conked out on the sofa in a rum/lettuce coma, the cat sits on my head. Zzzzz.

[70cities] Iterative

Paul — Sun, 05 Oct 2008 03:20:00 -0700

Well, a seven month hiatus - not bad going, really. 2008 has been the most hideously busy, stupidly packed year yet, and it's not over.

Things I've been up to:

Designed and built a highly-parallelisable log processing engine, based on Map/Reduce principles
Sweated blood in the process
Been to York for a stag weekend, and ended up spending two nights in a dreadful 80's-themed nightclub
Bought a new suit that doesn't make me look or feel like Tony Blair
Went to Newcastle and wrote off our car
Went to Scarborough for a wedding
Bought a new car (shiny!)
Spent a fantastic week in the North York Moors
Stayed in a posh apartment complex in Leeds
Struggled with being afflicted with recurring chronic sinusitus
Drank a lot of tea/coffee
Bought an iPhone
Obtained new eyesight prescription and face furniture, which allows me to drive at night at last
Waved sister-in-law and boyf off to live in San Francisco

So, not much really.

[70cities.tumblr] Disused tunnels database

Sun, 28 Sep 2008 08:08:53 -0700

Disused tunnels database

[70cities.tumblr] WebAIM: Blog - History of the browser user-agent string

Thu, 11 Sep 2008 01:04:55 -0700

WebAIM: Blog - History of the browser user-agent string:

Almost no-one understands what a mess these things are and why they’re useless. Arrrrgh!

[70cities.tumblr] Old OS Maps with Google Maps overlay

Fri, 22 Aug 2008 00:53:16 -0700

Old OS Maps with Google Maps overlay

[70cities.tumblr] Gateshead ‘Get Carter’ Car Park Demolition Project

Thu, 22 May 2008 00:45:47 -0700

Gateshead ‘Get Carter’ Car Park Demolition Project

[70cities] TwitterIgnore

Paul — Sun, 20 Apr 2008 11:58:00 -0700

I've been playing with Greasemonkey a bit lately as I wanted to refresh my Javascript/DOM foo. I needed something to build, so I searched in my big red plastic bucket of irritations and knocked up a user script that lets you hide Twitter tweets from a definable list of users without actually unfollowing them - so you can selectively ignore people for periods of time without them being any the wiser - thus avoiding any potential awkwardness. Social networks are delicate things - they're made up of humans, after all.

(This was inspired during SXSWi, when half my Twitter follow list went and I didn't, and the endless noise about it got quite wearing - but I didn't want to unfollow anyone as my Twitter universe is generally quite lovely and interesting).

http://userscripts.org/scripts/show/25365

[70cities.tumblr] 20 Year Flashback: Oink! Book 1988

Wed, 16 Apr 2008 00:56:21 -0700

20 Year Flashback: Oink! Book 1988

[70cities.tumblr] Norman (via qwidge)

Sun, 13 Apr 2008 15:59:26 -0700

Norman (via qwidge)

[70cities] Mux it up

Paul — Wed, 26 Mar 2008 05:01:00 -0700

In common with everyone else with an internet connection this week, I heart Muxtape a lot. Simple, no-nonsense, works. Here's mine, although I've no idea how often I'll ever update it. Needs RSS to track changes, perhaps? Mmm. Then I could plumb it into the uberfeed and whatnot.

In other news, since VD Day passed and I shut down all the Amazon EC2 stuff, I've been mostly working, going to the Isle of Wight, drinking beer, eating nice food, hacking around with a potentially-quite-useful Twitter hack using Greasemonkey and not having my hair cut...

[70cities] Be My Anti-Valentine - a technical perspective

Paul — Sat, 16 Feb 2008 03:30:00 -0800

After many hours of toil, Meg and myself opened the doors to this year's By My Anti-Valentine site a couple of weeks ago. Redesigned, revamped, relaunched with the hope that it won't fall over under the strain of the traffic it generates on Valentine's Day itself, as it has always done previously.

Given that my involvement in the project is one of technical architect, developer and sysadmin (Meg is the ideas, UX and design brains of the effort), it's probably worth me explaining a bit about the technology that drives the site and the problems that we came across last year and have tried to solve (or at least mitigate) this year.

By 2007, Meg had been running the VD cards site seasonally for six years, driven by a variety of successive flat-file systems and ultimately a generic PHP-based freeware card sending application (Sendcard, I think) that just about did the job but frequently caused her server to go belly-up once the traffic to the site reached its inevitable peak on the 14th February.

The problem was that the "choose card-send email to recipient" workflow was completely synchronous and linear within the application, so the emails were dispatched to the outgoing email server within the same web-spawned PHP Apache process. Naturally, once a few hundred people at a time were hitting the 'Send Card' button, the host's SMTP service would run out available connections and stop responding. Also, PHP 4's rather, ahem, unpredictable DB connection handling under load would generally explode along similar lines. Cue lots of server errors and general mayhem as the whole thing fell apart.

So, when I stepped in last year and volunteered to make the problems go away with a bespoke web app written in Perl to suit Meg's desired functionality, I made a few sensible design decisions. Over the years I've had to deal with my fair share of scaling problems and integration issues, so I built in a few tricks of the trade.

As it turned out, the site worked nicely for the few weeks before the 14th of February.. until it didn't. As it turned out, the traffic the site got was so enourmous that even moving it to its own dedicated, beefy server (kudos to Pair, Meg's hosting people, for being extraordinarily quick to do this) couldn't make it cope. Even extensive caching couldn't save the server from falling over.

So for 2008, whilst reconsidering our options, I decided to give Amazon EC2 - their Xen virtualisation cloud - a whirl. I'd heard plenty about it and had tinkered with virtualisation at work, but never gone the whole hog, and the cynic in me didn't buy the spiel that spinning up working application instances was as easy as a web service call.. but yes, by gum, it works a treat. Initial setup is a bit fiddly, but tools like the EC2 Firefox UI and S3Fox are a godsend when you're trying to do stuff quickly.

The site runs on a group of replicated server instances running a heavily customised version of Ubuntu 7.10 Gutsy, all identical with the exception of one (which I call the 'master', though not a Doctor Who homage) that also runs MySQL and the card-sending backend app. All the other instances connect to the master database when creating a new card or retrieving one for display. This is a pretty naïve architecture, but it works as the percentage of card senders against the overall site traffic is pretty small (less than 7% of visitors choose to send a card). Therefore MySQL isn't doing very much at all, and I didn't need to spin up a separate MySQL server instance to cope, which reduced the overall cost and was one less server instance to manage.
The rest of the application stack is fairly standard - Apache 2, FastCGI, Perl.

So what happened on the day itself? Well, we had a *tiny* bit of a wobble at around 13:00 GMT, when the east coast of America woke up (55% of users come from the USA), but I coped by spinning up another server instance and offloading some of the remaining site furniture onto Amazon S3 with some 302 redirects - which made a huge difference. Once the traffic started to drop off on the 15th, I was able to decommission three instances and balance the load across the remaining two, which coped admirably.

Paul's Top N Web Scalability Tips

Never maintain state on your web server. The moment you have to keep a user on a specific application instance for their state to be persisted, you lose all hope of ever scaling the sharp end horizontally. Keep state in your database (or on a shared filesystem if you're a masochist).

Decouple the web interface from backend systems. You don't send emails straight from a web-spawned process. You just don't. In fact, you shouldn't connect to anything other than a database server. Email servers are strange, unpredictable beasts and they generally need to be handled with care. Don't expose one to a process pipeline that you can't explicitly control (namely, a public web server!).

So, when a card instance is 'created' by a user on the VD site, an email object is created and stashed into a database table that effectively acts as a queue. This gets polled by a cron job at regular intervals and the emails get despatched in series. This means the email server is only ever under a constant load, which can be managed depending on what sort of performance you need.

Cache everything you can cache. If a page never changes, save it out to a cache directory on your filesystem and have the web server serve that instead of building a dynamic page on every hit. Thankfully the VD site only really needs to invoke the application when someone sends a card or retrieves one from a link they've been sent, so I took advantage of that and had easy cache-building methods that I could call to generate flat HTML pages. ('Proper' setups have caching loadbalancers like ZXTMs in front of their servers that can do this transparently, but of course I don't have that luxury and was trying to KISS). Which all leads me on to...

mod_rewrite is your friend. I can't rave about mod_rewrite enough. It's easy to get scared by URL rewriting if you're not comfortable with regexes and the general HTTP request handling mechanism in Apache, but it's actually incredibly simple to use with a bit of messing around and most modern web frameworks rely on some form of rewrite engine. The ability to abstract your URL scheme from the physical filesystem brings endless possibilities.

The VD site uses it not only to map its tidy URLs to the application endpoint (what us old-schoolers still call a CGI script), but to rewrite certain URLs to serve flat files from the cache directory but keeping the external URL the same, so turning it off has no effect on the end user. Also, with the flick of a hash character in the global .htaccess file, I can have the all the bandwidth-heavy images served from a CDN - Amazon S3 in this case - rather than the app servers.

Always use INSERT DELAYED when you don't need the row back - Class::DBI, the Perl ORM engine the VD site uses, isn't very good at this, so I had to roll my own methods here, but it was worthwhile - you save a few vital queries when you're stashing your state away and trying to get back to completing the user's request as quickly as possible.

In retrospect I wouldn't have used Class::DBI in the first place, but that's another story..

Phew! That's it over for another year, then. There's a whole bunch of other stuff I could go into (the joys of dealing with Spamcop..), but that's the story thus far. I'm chuffed at having got through a Valentine's Day without having to watch a single server helplessly grind to a halt under the weight of the traffic burst that a site like BMAV inevitably generates, and incredibly impressed that Amazon EC2 makes virtualisation accessible and affordable for projects like this. EC2 FTW!

[70cities] Sad news

Paul — Sun, 20 Jan 2008 15:06:00 -0800

Curry Hell man dies

Tragic news. I had the honour of working closely with Mr Latif back in the day when I was a jack-of-all-trades web monkey, and maintained the Rupali and Curry Hell websites. I can honestly say I've never met a man more dedicated to the twin pursuits of selling incredibly hot curry to pissed-up Geordies, and honest-to-goodness self-publicity.

A man so generous, he thought nothing of ordering me a free cab back to the office when a lunch-hour visit turned into a three-hour curry/booze fest - whereupon I fell asleep in the sofa in reception and got woken up by the marketing director of the coincidentally recently-deceased Pennine Windows, who was coming in expecting to update his web site.

He'll be sorely missed. Particularly by anyone who had a lamb phaal at his restaurant in recent days.

RIP Mr Latif. Off to his very own Curry Heaven, festooned with party hats and Cobra beer.

[70cities] Gamed

Paul — Sun, 13 Jan 2008 02:02:00 -0800

I barely ever play computer games. Consoles, whathaveyou - I've just never had much of an interest in any of it. Back in the 80's, when I was growing up and spent hours slaving over a hot ZX Spectrum, I was far more interested in learning how to write stuff and make it do interesting things than spending hours killing aliens or whathaveyou.

The games I liked then, and still do to this day, are the really simple platform/puzzle type ones. Lunar Jetman, Spellbound, Jet Set Willy, Technician Ted, the Dizzy games, Starquake. My later Atari ST, primarily a games machine much like the Amiga, spend most of its life playing simple platformers. When Driller came out and pioneered the immersive 3D worlds that everyone's used to now, I got bored pretty quickly. Later, when I had a PC, stuff like Rise of the Triads, and Sin gave way to pretty brain-dead stuff like Carmageddon. And even then only briefly. I get bored with games so easily.

In fact, in those rare times I find myself in a gamey mood, I don't reach for my non-existent PS2 or Wii and fire up World of Timewasting or whatever - I use a Spectrum emulator and play Starquake, as I have been for over 20 years (I've only finished it twice - I find the gameplay fun, not the end goal). I just don't get games, really. I just don't care.

Maybe it's the same reason I barely read any fiction or watch any films? Just writing that down makes me sound like some uncultured idiot, I know - but the truth is I get enough from current affairs, technology, comedy and music to keep me occupied for a lifetime and I sort of marvel at those who find the time to glue themselves to an Xbox 360 for hours on end or watch tons of films/boxsets. I can only conclude that I got short-changed somewhere along the line and everyone else in the world gets a thirty-hour day.

[70cities] HNY

Paul — Tue, 01 Jan 2008 15:45:00 -0800

Happy new year! Just when I was finally getting into the swing of 2007, the damn thing goes and ends.

Highlights of 2007 included:

Going to San Francisco and discovering Northern California
Doing some interesting stuff at work after a period of uncertainty
Snooker: seeing the Masters and the World Championships in Sheffield
Gadgets: new Macbook in Feb, and iPod Touch in November
Turning thirty (am fine with it now, in fact I really quite like it)
Spending lots of time with M and P and other nice people
Listening to the Beatles a lot

Lowlights included:

Project getting canned
Subsequent project floundering and getting canned
Not seeing my friends from the North nearly enough
Various worries, stresses and strains I couldn't do anything/much about
Being afflicted with weird sinusy problems as well as having a really crap hayfever year

2008? Changes afoot... hopefully.

[70cities]

Paul — Sun, 23 Dec 2007 15:20:00 -0800

Finally, now I've reached the correct side of the Christmas holidays, I can relax a bit. The last few weeks have been intense really took it out of me. The physical manifestations of this have been interesting - big bags under my eyes, not having a haircut for four months (big mistake - I looked scruffy, but not the 'good' scruffy I was half-aiming for - just 'bloody awful' scruffy), etc.

Anyway, the new version of the thing I work on launched and it's all gone very well. Now to rest up for a bit and do very little for a bit, except eat cheese, drink wine and generally expand. Marvellous.

[70cities] It's a test.

Paul — Thu, 20 Dec 2007 06:57:00 -0800

Test.

[70cities] Dear Santa...

Paul — Sun, 16 Dec 2007 15:38:00 -0800

I want..

To not think about work for a few days on the trot
Much less inadvertent exposure to Russell Brand/Winehouse/Jo Brand/Spice 'Girls'
Less worry
Beatles remasters
To spend less time on buses
Last.fm to work with my iPod Touch without having to jailbreak the thing
Spend more time hanging out with M and P
To see my friends more
More sleep
A haircut
More coffee
More music
A decent suit that doesn't make me look like a Suit
A greater variety of lunch options
To learn the banjo
A house
World peace

Kthxbai!

[70cities] Ding dong

Paul — Mon, 10 Dec 2007 00:41:00 -0800

Despite being one third of the way into December, it's only just dawned on me that it's Christmas very soon. In, like, 15 days. All the music channels on telly are rotating their crappy Christmas songs playlists - which, as they've got so much airtime to fill, are at least throwing up some amusingly bad, previously forgotten festive tunes. Paul McCartney and the fucking Frog Chorus, anyone? I saw it yesterday afternoon for the first time in 25 years and remembered exactly why it's never included on cheapo Chrimbo compilation CDs - even in a genre so pant-wettingly mawkish and feeble as christmas singles, it has a level of awfulness that stands proudly alone. Shudder. Next to this, Cliff Richard's seasonal efforts are verging on a punk ethic.

Anyway, Christmas. Shit! I've barely done a thing about it, what with one thing and another. Must knuckle down and get organising. Unfortunately, this week is already stacked full of Christmas parties and a slew of work to finish off. Who in the name of God decided to put our works Xmas party on the night before a big deployment? (Maybe it's the other way around - even so, someone needs a poke in the eye with a sharp stick).

[70cities] I really suck at blogging

Paul — Wed, 05 Dec 2007 13:29:00 -0800

Let's face it, it's just not happening, is it? I still might turn this into a link/video/mp3log though. I do plenty else around the web; I just don't do much of the whole writing thing. Too many people do it better than I.

Of various things of disinterest that've been happening lately, I bought a new iPod the other day. One of those touchy ones in fact, and it's very nice. Twice the space of me old Nano, and video playback - all very nice, but those features pale into comparison compared to the multitouch interface. I'm no feverent Apple fanboy, but my word, it's a thing of beauty to use. Coincidentally, I got a new mobile recently too (a Nokia 6500) with a view to using that as a bit of a mobile video playing doodah - but despite it having the technical chops to do so, the interface is so clunky and awkward that I completely gave up after about ten minutes. Now I can catch up with the new series of the Boosh on the bus home from work. Marvellous. The future's arrived!

However, despite what Steve Jobs and other Californians might think, ubiquitous open wifi is still rather a rarity. Even in my office, where there's dozens of sniffable networks, only one is unencrypted and even then it won't give me an IP address, most likely being MAC address-restricted. And I spend half of my life on buses anyway. So for that, my phone still trumps (and actually has a fairly decent browser to be getting on with).

That's about it, though. Other than that; work, sleep. Lie around. Repeat.

[equ] Harry S Price rip-offs - equ.in/ox

Wed, 21 Nov 2007 15:25:23 -0800

[web.archive] THE RUTLES -The True Story

Sun, 30 Sep 2007 01:30:37 -0700

[70cities]

Paul — Sat, 29 Sep 2007 03:32:00 -0700

I've been on holiday!

Yes. Hence my silence recently. There's been a much-needed break in the wilderness, away from all things technological and busy. It's been wonderful. Now I sit in a plush hotel in Leeds, soaking up the free wi-fi as we made the sensible decision to break the journey back to London in two and meet up with some old friends - which was wonderful. I'm full of eggs benedict, good coffee and Frosties. Now it's time to hit the road and get back home.

I'm so behind on everything it's pointless trying to catch up on it all. I even gave up halfway through reading all my usual webcomics; somehow, daily content doesn't work so well when trying to read a lot of it in one go.. maybe I need to look at one of those webcomic reading apps I hear people talking about.

[newyorker] A Reporter at Large: The Interpreter: Reporting & Essays: The New Yorker

Sat, 15 Sep 2007 04:43:18 -0700

[70cities] Applause?

Paul — Sat, 15 Sep 2007 04:50:00 -0700

From an email I just received inviting me to apply for tickets to bunch of new TV shows:

"***COMING SOON*** THE KATIE PRICE & PETER ANDRE CHAT SHOW"

Arrrrrrgh! The death of reason is finally upon us. Make it stop!

[70cities]

Paul — Thu, 13 Sep 2007 02:10:00 -0700

So the Adblock saga that's been rumbling along for a while is finally reaching a peak, with people cottoning on to the fact that it's potentially (in theory) damaging to the very revenue streams that make the Web 2.0 world so tasty, supple and pliant. So much so that you've even got mechanisms for sites to block Adblock-enabled Firefox browsers.

This is a bit nuts, no? To me it smacks of classic internet kneejerkery. Where's the evidence that all these bottom lines are being eroded? There isn't any - this is all supposition and FUD at the moment.

Adblock, and browser plugins like it, came about because web advertising had become so intrusive, disruptive and unruly that the hacker community came up with a solution. You very rarely hear anyone bitching about Google Adsense ads, as they're both textual and usefully contextual, and anyone who wants to argue that 200Kb Flash movies choking my bandwidth and sprouting both alongside, above and directly over the content users are trying to access is in any way a good idea needs their head thoroughly scrutinised. The very existence of adblocking tools should be sending a very clear message to web advertising companies - we don't want to be beaten over the head with your shit. Plus, do you really want to be engaging in a incremental bunfight with the hacker community, either side progressively defeating the other's measures? Is it really a good use of everyone's time?

[70cities] Ouch

Paul — Tue, 11 Sep 2007 16:16:00 -0700

Being stung on the lip by a wasp fucking hurts. I know this, because it happened to me this evening.

Thank $deity for the Internets though, because hundreds of unverified old-wives' remedies were merely a judicious Google away. It's now much better, thanks - lemon juice really is the business for just about every random problem or malady that can curse your life.

Owwwww.

I'm going to try and make more of an effort to post here, but I don't think one of those '10 ways to get more stuff done NOW!' blogs is the way to go (not least because I'm fundamentally very lazy) but I'm a senior software engineer and I ought really to have more to say about what I've encountered over the years working on the internets than I have in the past.

Then again, there's no shortage of self-appointed pundits willing to spout grindingly obvious platitudes about software and software engineering - why should I add myself to their ranks? Can I add anything of worth? Only time will tell.

[70cities] What happened?

Paul — Sun, 05 Aug 2007 01:47:00 -0700

Wow, I fail at blogging.

I've been busy. New jobs have that effect.

Meanwhile, here's some Neil Innes:

::http://www.monkeesmixography.com/::THE MONKEES MIXOGRAPHY

Sun, 24 Jun 2007 16:10:43 -0700

::http://map.kitjunkie.org/::OSGB to Google Maps Converter : Kitjunkie Maps°

Sun, 24 Jun 2007 12:33:51 -0700

[70cities] Snot party

Paul — Sun, 27 May 2007 04:31:00 -0700

Someone was really rude to me this week. I was slightly taken aback at it, as they're a long-time acquaintance of mine and there really wasn't any need for it, as I was being super-friendly to begin with. It's annoyed me for several days now, though. I'm not tolerant of this sort of thing, especially when it's an affectation (as I suspect it to be). Grr.

Hay fever is back, yet again, with a vengeance. I've been seeped in warm, runny snot for the last few days and it's really depressing me. I've been afflicted with it since the age of six - and now I'm 30, I still haven't 'grown out of it' like all the doctors over the years have said I would. And yes, I've tried every pill/spray on the planet - and they don't work. I am seemingly condemned to ride out all the summer months of my life like this - sneezing, coughing, itching, sitting around in a dazed trance with my head slightly tilted back. Urgh.

Still - bank holiday, and a nice wet one at that. Perhaps I'll just pull myself together and ignore the interesting pulling and pushing going on in my sinuses and get out and do stuff. Maybe I'll mess around with Flash (a pertinent technology, as it happens). Maybe this, and maybe that.

[70cities] Apple/Idlers

Paul — Tue, 22 May 2007 00:40:00 -0700

The recent outrage over Apple's Macbook displays being only 6-bit instead of 8-bit doesn't bother me in the slightest. My Macbook display is still great, light years away from what my iBook had, and still better than most regular laptop screens I see. So what if it's dithering? If you're going to do proper print design work on a laptop you'll be using an external monitor anyway - or you're some kind of idiot.

As someone who has, by and large, sat around in offices for the last eight years, The mystery of the daytime idle has often gripped me. I don't know if the picture in SF is different from anywhere else, but London is constantly teeming with people seemingly doing bugger all, all day. Of course, given that I work in Shepherd's Bush a good 50% of them are tramps, propped up against benches or sitting on steps, sinking cans of Tennent's Extra and shouting at each other. Come to think of it, San Francisco has a notable tramp population, but the author didn't seem to approach any..

[70cities] Blumenwiese neben autobahn..

Paul — Sun, 13 May 2007 09:06:00 -0700

Crazy week. Loads happening. Can't talk about any of it. Argh!

Had a great weekend so far, though. Friday might, M and I went to see top indie electro-rocksters Maxïmo Park play at the Shepherd's Bush Empire. It's been ages since I've been to a non-club sort of gig, and they were incredibly good live - note-perfect, and incredibly tight. There's also something very compelling about Paul Smith's lyrics for me, given that I grew up more or less where he did and can vividly recognise a lot of the references, places and experiences he alludes to - "standing by the Monument just waiting for the rain". It's been a long while since we had a true Geordie voice in the hit parade - no, Jimmy Nail doesn't count (and probably can't, either).

After that, it was up early on Saturday to hotfoot it (via train) to Brighton for Anna's 30th birthday party. Had enourmous fun seeing the extended family and meeting nice new people whilst munching quality sausages, along with a selection of disgustingly nice French cheeses. Then, in the evening we rushed back to London to catch the Eurovision 'Song' Contest - completely laughable, unfortunately, due to all the block voting going on. Even Sir Terence of Wogan seemed utterly crushed by the complete absence of any real notion of a song contest. I can't see it continuing in its current format for much longer..

Today, hay fever has rendered me helpless (again), despite it being cold, rainy and dull outside. Sniff. Can't do much but sit up straight and swallow snot. Urgh.