YWebDevs

Virtualised Development Environments on a Mac

Tim — Sun, 06 Sep 2009 06:23:37 -0700

Recently, I bought myself a brand new MacBook Pro, which gave me the perfect opportunity to clean up my development environment. Since I’ve started doing all of my development on virtual machines, I began thinking about my development workflow: In theory, I should be able to model the perfect server environment virtually.

I wasn’t sure what environment I wanted to end up with, but I had a good idea what basic virtual machines I wanted as a starting layout.

The Proposed Environment

A basic, well structured development environment should include:

Development Server — Where all development takes place. This will probably have more detailed logs, development dependencies, and a high likelihood of being broken at any given moment in time. Bugs that occur here are likely to be frequent.
Staging Server — A duplicate of the production server that allows testing of build scripts, packages etc. before they are pushed to production. Bugs that occur here are likely to be less frequent and flagged by specific testing.
Production Server — The live environment. By the time the code reaches this server, it should have gone through development testing and staging testing. This means bugs should be at a bare minimum.

If bugs arise on either the staging server or the production server, the fixes should be made on the development server and pushed to staging and then production. No code changes should happen on staging or production.

We could, of course, add more stages between development and live (a quality assurance (QA) server, for instance), and we could even branch the environment to allow for a continuous integration (CI) server (the product of an automated build, and including automated testing; see Wikipedia’s Continuous Integration page for more information).

In my specific case, the environment should be structured thus:

Development Server — The development virtual machine I’ve been using for the past few months. This VM shares key folders with Mac OS X so that I get the pleasure of using Mac based tools for development (see: TextMate).
Staging Server — A duplicate of the production server (or as close to as possible) in VM format. This should be the target for my build scripts, and where I run integration tests.
Production Server — My live web server. Only functioning, tested code should ever be pushed here.

So, rather than developing with just the one virtual machine, it’s clear I should be moving forward with two virtual machines (at least) — one as the development server, and one as the staging server.

Linked Clones

Once I’ve built a good master virtual machine as a source — and set it all up as a basic server, including all the dependencies and packages I want — I could create duplicates for each server. The drawback to that method is the considerable amount of disk space the independent VMs would take up.

The second option would be to branch snapshots on the master, setting up one branch as the development server, and another as the staging server. However, this would mean that I would have no way of running the two servers simultaneously, which would be a terrible trudge when attempting to test a build.

The best solution would be to build linked clones. A linked clone is a copy of a virtual machine that shares virtual disks with the parent virtual machine in an ongoing manner. This conserves disk space, and allows multiple virtual machines to use the same software installation. It also means we’ll be able to run the two VMs simultaneously. However, VMWare Fusion (the Mac version of VMWare) doesn’t actually allow this functionality through the interface; so it has to be done manually.

When you take a snapshot of a VM, the original virtual disk becomes read-only and a new Copy-On-Write (COW) disk is created. To create a linked clone manually, we’re going to set up a virtual machine, then create new ones using that as the base disk. We then snapshot the new VMs so they never try to modify the base disk.

Note: I pretty much followed the “HOWTO: Manual Linked Cloning in VMWare Fusion” article on the VMWare Communites site. I’ve recreated the steps below to allow illustration of my particular implementation, and to hopefully save you flicking backwards and forwards between articles.

Building the environment

The first thing we need to do is create a master VM. This will be our base disk, and the root of all evil… Uh… I mean… The root of all awesome.

Create the master VM

In my particular instance I’m creating web servers, so I’m going to build an Ubuntu VM and base the clones on that. To do this, I’d recommend following Brad Wright’s excellent Ubuntu VM build tutorial, right up to the point where you’ve finished installing VMWare Tools — we don’t want to start sharing folders on our base build because we’ll do that on the development server VM only.

The next step is to set up all the common packages, dependencies, configurations, and applications on your base VM so that you have the minimum to set up on your clones (and so they’re as close to one another as possible at the outset; after all, that’s part of the reason we’re doing this).

In my case, I set up the following:

Apache
PHP
MySQL
Python
Django
CouchDB
Twisted

I also made sure I’d configured them all as I wanted them — including default site layout for Apache — and included interdependencies like MySQL-Python.

Once this is complete, we’ve got a suitable master VM for our linked clones.

Finally, make sure you back up the .vmwarevm package for your master VM. We’re going to be editing the package contents, and deleting a bunch of stuff; plus if anything gets hosed in the future, you’ll be able to restore the master and start over. I’d recommend creating an archive of the .vmware package and storing it somewhere safe.

Prepare the master VM for cloning

The only thing we want to keep from the master is the virtual disk, since the rest of the VM is inconsequential to our linked clones. So, once you’ve definitely made a back-up of the full VM, let’s change directory to where our VMs are stored, copy out the .vmdk files (virtual machine disk), and delete the master bundle:

$ cd /Users/timbo/vms
$ ls
master.vmwarevm
$ mkdir master
$ ls
master master.vmwarevm
$ mv master.vmwarevm/*.vmdk master/
$ rm -f master.vmwarevm
$ ls master
master-s001.vmdk master-s002.vmdk master-s003.vmdk master-s004.vmdk master-s005.vmdk master-s006.vmdk master.vmdk

Next we need to update the the virtual disk to reflect our changed location and to remove it’s UUID (Universally Unique Identifier). To do that, we need to edit the root .vmdk file:

$ cd master
$ mate master.vmdk

I’m using TextMate as my editor of choice, but you could easily use vi, vim, nano, BBEdit etc.

You should see a file that looks something like this:

# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=96100eeb
parentCID=ffffffff
createType="twoGbMaxExtentSparse" # Extent description
RW 4192256 SPARSE "master-s001.vmdk"
RW 4192256 SPARSE "master-s002.vmdk"
RW 4192256 SPARSE "master-s003.vmdk"
RW 4192256 SPARSE "master-s004.vmdk"
RW 4192256 SPARSE "master-s005.vmdk"
RW 10240 SPARSE "master-s006.vmdk" # The Disk Data Base
#DDB ddb.uuid = "60 00 C2 99 ab ce aa 92-d3 3e 55 a9 30 e7 c6 dd"
ddb.toolsVersion = "7462"
ddb.adapterType = "lsilogic"
ddb.geometry.sectors = "63"
ddb.geometry.heads = "255"
ddb.geometry.cylinders = "1305"
ddb.virtualHWVersion = "7"

We need to fix the paths of our .vmdk files so that they will work from other locations. Obviously, for the best results, we should probably use absolute paths so that they work from anywhere. I updated mine like this:

# Extent description
RW 4192256 SPARSE "/Users/timbo/vms/master/master-s001.vmdk"
RW 4192256 SPARSE "/Users/timbo/vms/master/master-s002.vmdk"
RW 4192256 SPARSE "/Users/timbo/vms/master/master-s003.vmdk"
RW 4192256 SPARSE "/Users/timbo/vms/master/master-s004.vmdk"
RW 4192256 SPARSE "/Users/timbo/vms/master/master-s005.vmdk"
RW 10240 SPARSE "/Users/timbo/vms/master/master-s006.vmdk"

The next thing you should do is remove the UUID line, either by commenting it with a #, or by deleting it entirely:

ddb.uuid = "60 00 C2 99 ab ce aa 92-d3 3e 55 a9 30 e7 c6 dd"

Once these changes have been made, save the file and exit back to the command line.

The final thing we should do is write protect these files — make sure you update your path appropriately:

chmod a-w /Users/timbo/vms/master/*.vmdk

Our master is now totally prepared for cloning, and can be used repeatedly. In my instance, this means I can now clone it twice; once for the development VM, and once for the staging VM (since it’s no longer a full VM by itself).

Create a clone

To create a clone, we need to create the VM in VMWare Fusion. Once you’re back in the VMWare GUI, do the following:

Push cmd + N, or select File > New from the menu, to create a new virtual machine.
Click “Continue without disk”.
Select “Create a custom virtual machine” and click “Continue”.
Select the appropriate guest OS type (in my case, Linux/Ubuntu) and click “Continue”.
Click “Customize Settings”.
Choose a name and location for your clone VM. I chose to call mine “child” and saved it in my ~/vms directory.
You should now be looking at the settings of your new VM. Set it up with the same basic settings as you did for the master (256MB RAM, printers disabled), but leave the HDD alone because…
Edit the HDD settings and delete the disk device by clicking the “-” when the device is selected in the list.
Go back to your terminal and delete the .vmdk files in the new VM package:
```
$ cd /Users/timbo/vms/child.vmwarevm
$ rm -f *.vmdk
```
Now copy the root .vmdk file from your master (the one we edited the paths in earlier) to your clone package:
```
$ cd /Users/timbo/vms
$ cp master/master.vmdk child.vmwarevm/
```
We need a copy because VMWare will create a .lck file at the same level as this file. With no copy, all the clones would be attempting to use the same .lck file simultaneously, which would prevent them running in parallel.
Back in the Hard Disk settings pane, press the + button to add a new HDD device.
Select “Choose existing disk…” in the “File name” drop-down, and choose the copied metadata file (the one in child.vmwarevm, not the one in the master folder) and uncheck the checkbox — we’re happy with the file where it is, we don’t want to copy or move it since we’ve already copied it manually (VMWare copies all the .vmdk files over, and that wouldn’t save us any disk space at all).
Finally, create a snapshot of your new VM. This tells Fusion not to try to write to the original disk. I named mine “Base” with the comment “Don’t delete!”. Even if you run without the snapshot, because of the read-only permissions on the .vmdk files you shouldn’t be able to change the original virtual disk, but the clone virtual machine (i.e. not the master) will not be happy.
Run the clone.
Since this is a clone of your master, log in using the same user(s).
Run the interface configurator:
```
$ ifconfig
```
If the eth0 interface is showing, skip ahead to step 19, otherwise…
To fix this problem, we just need to truncate the udev persistent network rules file:
```
$ cat /dev/null > /etc/udev/rules.d/70-persistent-net.rules
```
Reboot the VM to regenerate the rules:
```
$ sudo reboot
```
Now wait for your VM to reboot and log back in again.
Run the interface configurator:
```
$ ifconfig
```
Make a note of the IP address of the eth0 interface, alongside inet addr: since this is the new address for your clone.
Finally, update the hostname of your clone VM by editing and saving the /etc/hostname file:
```
$ sudo vim /etc/hostname
```
Restart the hostname service:
```
$ sudo /etc/init.d/hostname.sh start
```
Use your clone VM as normal.

To create another clone VM for your staging environment, simply repeat the steps above (in the “Create a clone” sub-section).

Playing nice

You might like to run your VMs in “Headless Mode” (see “A Power User’s Guide to VMWare Fusion“), which will prevent you having to open VMWare every time you want to run your VMs. You can activate this option in VMWare itself by running the following command at the terminal of your Mac:

$ defaults write com.vmware.fusion fluxCapacitor -bool YES

Then you’ll be able to select “Enter Headless” under the View menu of your VM in VMWare. Once the VM is running in Headless mode, you can quit VMWare and the VM itself won’t shut down (until your reboot your Mac).

If you do reboot your Mac for any reason, you can boot up the VM using the following command:

$ /Library/Application\ Support/VMware\ Fusion/vmrun start /Users/timbo/vms/child.vmwarevm/child.vmx nogui

If you’re going to do lots of VM stuff from the command line, I’d recommend updating your $PATH variable to include the VMWare Fusion directory by adding the following line to your .bash_profile:

export PATH="$PATH:/Library/Application Support/VMware Fusion"

Now you’ll be able to run vmrun without the directory prefix:

$ vmrun start /Users/timbo/vms/child.vmwarevm/child.vmx nogui

Shared folders

Now that we have a fully functioning development VM, you can continue to set up shared folders between the Ubuntu VM and Mac OS.

I’m not going to walk through this again; it’s covered (including a common read/write issue that can occur) in Brad Wright’s excellent Ubuntu VM build tutorial which I previously mentioned.

Once your shared folders are set up and functioning correctly, you can do all your development with your favourite Mac-based tools, but running it on our Ubuntu virtualised development server.

Incidentally, you can also add new shared folders using vmrun:

$ vmrun addSharedFolder /Users/timbo/vms/child.vmwarevm/child.vmx myfolder /Users/timbo/projects/myfolder

I love vmrun.

Summary

I now have the environment I set out to achieve; and with the aid of Git, and my own build scripts, I can easily develop, test, and release code in much the same way I would with a live server environment. What’s more, I have infinitely more control over my environments than I ever had with MAMP, or MacPorts installed instances of Apache, PHP, MySQL etc.

I guess it’s safe to say that there’s really no going back from here.

CouchDB’s RESTful API

Tim — Sat, 01 Aug 2009 03:06:43 -0700

I’ve recently been playing with Apache CouchDB; a distributed, fault-tolerant and schema-free document-oriented database. CouchDB supplies a RESTful API for the manipulation of data within it. This is a great demonstration of REST at work, so I thought I’d illustrate its usage in a quick post.

Let’s get the technical details out of the way first: CouchDB makes use of HTTP verbs — GET, PUT, POST, and DELETE — and handles requests and responses in JSON only. This means it is entirely possible to utilise CouchDB using JavaScript alone. For the purpose of these examples, however, I’ll be using cURL, which is easier to follow along with via the command line.

My CouchDB server is running on my dev Ubuntu VM, on the default port of 5984. I’ve also set up a “couchhost” alias to the VM in my hosts file, to make my requests more readable.

Creating databases

The first thing we should do is create a database within our CouchDB server. To do that, we use the PUT verb:

$ curl -X PUT http://couchhost:5984/fu
{"ok":true}

In the command above, we specified the database name in the resource URI — which is the correct usage of PUT — and received a JSON object response. As you can see, that JSON object contains a single field; the boolean “ok”, with a value of true.

We can obtain a list of all the databases on our CouchDB server, by sending it a GET request to the _all_dbs resource:

$ curl -X GET http://couchhost:5984/_all_dbs
["fu"]

Here we receive an array response containing all our database ids. In this case, we only have our “fu” database.

Let’s try adding another:

$ curl -X PUT http://couchhost:5984/bar
{"ok":true}

Since we’ve received an ok response, let’s list all the databases again:

$ curl -X GET http://couchhost:5984/_all_dbs
["fu","bar"]

Now we can see multiple values in the array response.

We can obtain information about a specific database by sending a GET request to its URI:

$ curl -X GET http://couchhost:5984/fu
{"db_name":"fu","doc_count":0,"doc_del_count":0,"update_seq":0,"compact_running":false,"disk_size":4096}

Here we can see, amongst other things, the number of documents in the database — none — and how much diskspace the database is taking up — 4k.

CouchDB documents

A CouchDB document is simply a JSON object with some associated metadata. This is an example document (lifted from the CouchDB documentation):

{ "_id":"discussion_tables", "_rev":"D1C946B7", "Sunrise":true, "Sunset":false, "FullHours":[1,2,3,4,5,6,7,8,9,10], "Activities":[ { "Name":"Football", "Duration":2, "DurationUnit":"Hours" }, { "Name":"Breakfast", "Duration":40, "DurationUnit":"Minutes", "Attendees":[ "Jan", "Damien", "Laura", "Gwendolyn", "Roseanna" ] } ]
}

As you can see, a CouchDB document can handle multiple types of data, including arrays and nested objects. By default the structure is flat; in this particular case, the “Activities” attribute is structure imposed by the user.

It’s also worth noting that any top-level fields with a name that starts with an underscore prefix are reserved.

Adding documents

So let’s define a document for one of our databases:

{ "FirstName":"Isabella", "FamilyName":"Huegdon", "Age":2, "Likes":[ "noise", "animals", "chocolate", "kicking Milly", "throwing tantrums" ]
}

This is a fairly straight forward document that includes several different types of data. To add it to our database, let’s use an HTTP POST request:

$ curl -X POST http://couchhost:5984/fu \
-H 'Content-Type: application/json' \
-d '{"FirstName":"Isabella","FamilyName":"Huegdon","Age":2,"Likes":["noise","animals","chocolate","kicking Milly","throwing tantrums"]}'
{"ok":true,"id":"30d53965f5aec18391c258faa66921c1","rev":"2348720638"}

Because we’ve used POST, CouchDB has automagically assigned our document an identifier. You can see this in the id attribute of the response object. But what if we want to specify our own id? Simple; we use HTTP PUT instead.

Let’s define another document:

{ "FirstName":"Amélie", "FamilyName":"Huegdon", "Age":2, "Likes":[ "noise", "animals", "chocolate", "hitting Izzy", "running and screaming" ]
}

And now let’s add that document using HTTP PUT:

$ curl -X PUT http://couchhost:5984/fu/milly-huegdon \
-H 'Content-Type: application/json' \
-d '{"FirstName":"Amélie","FamilyName":"Huegdon","Age":2,"Likes":["noise","animals","chocolate","hitting Izzy","running and screaming"]}'
{"ok":true,"id":"milly-huegdon","rev":"3414255320"}

This time, we can see our specified id in the id attribute of the response object.

Now let’s have a look what documents we’ve got in the database, using the _all_docs resource, beneath our database:

$ curl -X GET http://couchhost:5984/fu/_all_docs
{ "total_rows":2, "offset":0, "rows":[ { "id":"30d53965f5aec18391c258faa66921c1", "key":"30d53965f5aec18391c258faa66921c1", "value":{"rev":"2348720638"} }, { "id":"milly-huegdon", "key":"milly-huegdon", "value":{"rev":"3414255320"} } ]
}

Note: I’ve styled the response here for ease of reading; your cURL output will lack indenting but will contain the same information.

Here we can see, from the total_rows attribute, that there are 2 documents in our database; and we can see the metadata for those documents in the rows attribute array.

To obtain the actual document, we just need to send an HTTP GET request to the document’s URI:

$ curl -X GET http://couchhost:5984/fu/milly-huegdon
{ "_id":"milly-huegdon", "_rev":"3414255320", "FirstName":"Am\u00e9lie", "FamilyName":"Huegdon", "Age":2, "Likes":[ "noise", "animals", "chocolate", "hitting Izzy", "running and screaming" ]
}

Once again, I’ve styled the response somewhat for readability. In this instance, you can also see that CouchDB can handle unicode characters (since JSON should always be encoded as UTF-8) from the FirstName attribute.

Deleting items

Deletion of documents and databases from our CouchDB server couldn’t be simpler. All we need to do is send an HTTP DELETE request.

To remove the bar database we created earlier, we send the following request:

$ curl -X DELETE http://couchhost:5984/bar
{"ok":true}

Summary

You are now able to create and delete items from your CouchDB database. This is all you’re going to need if you just want to use CouchDB as a persistent document store; however, if you want to query the data in your databases, you’re going to need to make use of CouchDB Views. I’ll cover those in a later blog post.

I’m going to continue to play with CouchDB over the next week or so and will continue to blog about it (with the hope of using the blog posts as the basis for a TechTalk at Yahoo! Europe — slides possibly to follow on SlideShare). For now, you should be able to see that CouchDB is a wonderfully easy thing to use, thanks to its entirely RESTful API and its use of JSON as a request and response format.

CouchDB on an Ubuntu development VM

Tim — Wed, 29 Jul 2009 09:41:41 -0700

I’ve recently set-up CouchDB on the Ubuntu 8.10 (Intrepid Ibex) virtual machine I use for development. This was a relatively pain-free experience and I thought I’d outline what I did here for the benefit of others (and my future self).

Firstly, I decided to install using the aptitude package, rather than from source. If you’re interested in building the most recent version, I’d recommend taking a look at Isofarro’s handy “Installing CouchDB on JeOS 8.04″ post.

To begin, install the package:

$ sudo apt-get install couchdb

Once the package has downloaded and installed, you should check the status to see if it’s running correctly:

$ sudo /etc/init.d/couchdb status

If you see the following output, then CouchDB is running successfully:

Apache CouchDB is running as process 32651. Time to relax.

If you don’t see that output, try starting CouchDB manually. I needed to do this:

$ sudo /etc/init.d/couchdb start

And then check the status again:

$ sudo /etc/init.d/couchdb status
Apache CouchDB is running as process 32651. Time to relax.

If you’re still not getting the correct status, you’ll need to refer to the CouchDB documentation.

Next, you should check that you can access CouchDB over HTTP on the default port of 5984. To do that, I used curl:

$ curl http://localhost:5984/

Which should return the following response:

{"couchdb":"Welcome","version":"0.8.0-incubating"}

The final step, to make sure you can access CouchDB externally to your dev VM, is to make sure the CouchDB configuration isn’t bound to the loopback address (127.0.0.1) — which it is by default. To do that, you’ll need to edit the couch.ini file:

$ sudo vim /etc/couchdb/couch.ini

And comment out the following line:

BindAddress=127.0.0.1

You can also change the port that CouchDB is running on in this configuration file if you so desire.

Finally, you’ll need to restart CouchDB:

$ sudo /etc/init.d/couchdb stop
$ sudo /etc/init.d/couchdb start

Now you should be able to log into Futon, the browser-based CouchDB admin system, on the following address:

http://mydevbox:5984/_utils/

Huzzah!

Unit Testing: The Basics

Tim — Sun, 03 May 2009 04:36:52 -0700

As a web development manager, it’s part of my job to be involved in defining best practices for my team. This means defining a set of standard practices that will benefit the whole team throughout the entire development process. A good example of one such best practice would be code testing.

Testing, for web developers, is usually a fairly disorganised affair. Most of the time, we test “on the job”, i.e., we code something and then we run it; if it breaks, we fix it. In fact, often, the thought of actually structuring our testing process can seem somewhat of an over complication. This is certainly not the case.

The benefits of a structured testing process are legion, as any software engineer will attest. At the very least, it provides confidence in our code — something which can only be of incredible benefit when code is shared across a wider team. At its best, the tests we write can influence the way we develop the code being tested; and save effort, time, and money.

In this post I’m going to look at “unit testing”, and how it can be of benefit to web developers in general.

What is unit testing?

Unit testing is the process of validating and verifying the individual “units” of code in a system. A unit is the smallest testable part of a system or application. In the majority of cases within web development, these units are usually individual functions or methods of our object-oriented back-end and front-end code.

At this point, it’s probably worth noting that unit testing is really only applicable to functional code, such as PHP, JavaScript, Python, Perl, Ruby etc., because it breaks down into units nicely. Testing mark-up and CSS is a slightly different process and one that I may look at in a future blog post.

A note on language

The concept of unit testing is entirely language agnostic, therefore the examples in the following tutorials can be ported to PHP, Perl, Ruby, JavaScript, or any other functional or object-oriented language you are using.

I’ll use Python as the language for all examples in this post as it’s my current weapon of choice (and the interactive interpreter is very handy for trying out ideas); if you’ve never seen Python syntax, don’t worry, it’s easily understandable (if you want more information, take a look at the Python tutorial).

Writing a simple test

To begin with, let’s imagine we have a function for calculating the area of a rectangle:

def get_area_of_rect( width, height ): return width * height

Let’s paste this function into the interactive interpreter, so that we can play with it:

>>> def get_area_of_rect( width, height ):
...  return width * height
...
>>>

Now if we call it, we’ll receive the area of our rectangle like so:

>>> get_area_of_rect( 2, 3 )
6
>>>

Now if we wanted to test this function, we could call it with a some defined parameters and test the output against our expectations:

def test_get_area_of_rect(): width = 2 height = 3 expected_result = 6 result = get_area_of_rect( width, height ) if result == expected_result: print "Test passed: Received expected result!"

Now if we run our test, we should see the following:

>>> test_get_area_of_rect()
Test passed: Received expected result!
>>>

Brilliant; our code passed the test because the expected result was the same as the actual result — however, that’s actually not very useful to us. It’s far more important that our tests can trap failing code than code that works.

Imagine our get_area_of_rect function had been updated as follows:

def get_area_of_rect( width, height ): return width / height

Anyone with rudimentary maths skills can tell that this will no longer return the correct area of a rectangle. In fact, if we rerun our test, we’ll get no output:

>>> test_get_area_of_rect()
>>>

A fat lot of good that is. If our code is failing for some reason, we want to know about it. After all, that’s the whole reason for testing it.

Let’s update the if statement in our test_get_area_of_rect function:

if not result == expected_result: raise AssertionError
else: print "Test passed: Received expected result!"

In fact, we could probably dump the else clause since it’s of little real importance except for logging purposes:

if not result == expected_result: raise AssertionError

That’s nicer, but Python provides us a shortcut to this if not … raise AssertionError syntax with the assert statement:

assert result == expected_result

With that in mind, the test_get_area_of_rect function should now look like this:

def test_get_area_of_rect(): width = 2 height = 3 expected_result = 6 result = get_area_of_rect( width, height ) assert result == expected_result

If we run our test again, we should receive more constructive output, considering our get_area_of_rect function is still dividing:

>>> test_get_area_of_rect()
Traceback (most recent call last): File " ", line 1, in  File " ", line 8, in test_get_area_of_rect
AssertionError
>>>

We now have a decent automated test, but we’re only checking a single expected result. If, for instance, we passed in a width of 1 and a height of 1, both multiplication and division would return the same result: 1. This means that we should pass multiple sets of width and height parameters and test against their expected results. Since these parameters will be constant, let’s define them as tuples and loop through them, testing each in turn:

def test_get_area_of_rect(): known_values = ( ( 0, 0, 0 ), ( 1, 1, 1 ), ( 2, 2, 4 ), ( 2, 3, 6 ), ( 3, 0, 0 ) ) for width, height, expected_result in known_values: result = get_area_of_rect( width, height ) assert result == expected_result

As you can see, we’re now passing several sets of parameters. Each set of parameters is valid, and as such, we should expect a successful outcome from our get_area_of_rect() call.

Now if we run our test, we should get something like the following:

>>> test_get_area_of_rect()
Traceback (most recent call last): File " ", line 1, in  File " ", line 8, in test_get_area_of_rect File " ", line 2, in get_area_of_rect
ZeroDivisionError: integer division or modulo by zero
>>>

Ah ha! By passing multiple sets of parameters we’ve uncovered more significant information regarding the AssertionError that was raised earlier; we have a division by zero error on line 2 of get_area_of_rect. This has supplied us with all the information we need to fix our error, so let’s take a look at that function again:

def get_area_of_rect( width, height ): # Line 1 return width / height # Line 2

Change the division back to multiplication, like so:

def get_area_of_rect( width, height ): # Line 1 return width * height # Line 2

And then rerun our test:

>>> test_get_area_of_rect()
>>>

We get no output at all because the test passed successfully. As the old proverb goes, “no news is good news”.

Now if we add to our original code in the future, we’ll still be able to run this testing code to see what has been affected. What’s more, we can add to it by writing more tests, should the complexity grow. In fact, creating an extensive suite of tests is generally the aim of unit testing, but to do so, it’s a good idea to utilise a testing framework, which I’ll look at in the next post of this series.

Summary

Writing tests for code enables more efficient debugging, faster bug fixing, and an overall confidence in the quality of your code. Furthermore, the rest of your team (or whomsoever you’re sharing said code with) will equally have confidence thanks to the tests included.

In brief, here’s what we’ve learnt about unit testing:

It’s most important that your tests catch failure, not success.
Test the smallest amount of code, or “unit”, possible each time.
Test as many different parameters, inputs, and outputs as possible.
Run the tests often.

The next post

In the next post, I’ll be expanding on unit testing by looking at test frameworks, test architecture, code coverage, and other more advanced techniques that can improve your testing environment.

Housekeeping

Tim — Mon, 13 Apr 2009 03:14:55 -0700

I’ve just completed some quick housekeeping on the site — mostly on my feed.

I noticed last night that Google had acquired Feedburner, and that I needed to port my feed over to my Google account. At the same time, I swapped the feed to use Atom instead of RSS. Having completed that, I decided to address a problem with my feed that’s been bugging me for some time:

My feed has always published only an excerpt of each post, instead of the full post. This is because I use the “more” break functionality in Wordpress for the introduction snippets in the bespoke home page theme files, instead of writing an entirely separate excerpt. On previous versions of Wordpress, this also limited the RSS feed regardless of whether you had specifically chosen the “display full article” option in the syndication settings. I’ve had a couple of complaints about this, since most developers seem to prefer reading articles in their feed readers — and so do I, for that matter.

Last time I needed to fix this problem, I had a hell of a time trying to find the offending code in the horrific Wordpress PHP. Thankfully, since then, the bug has been fixed so all I needed to do was update Wordpress. Having completed that update, my feed should now be working as intended originally.

The next step, it seems, will be getting off Wordpress to something home-grown.

A better OS X development environment

Tim — Sun, 12 Apr 2009 14:41:44 -0700

When I first began developing using a Mac I made use of the bundled versions of Apache and PHP that were included in OS X Tiger. This meant I only needed to install MySQL to be able to develop fairly complex websites.

However, as my requirements evolved, I discovered that recompiling PHP to include support for features such as HTML Tidy was going to be a pain in the proverbials. So, following advice from friends, I opted to install another version of Apache, PHP5, and MySQL using MacPorts package management. This required minimal configuration and allowed me to easily activate or deactivate features on install — and it also allowed me to add to my environment as I learnt new skills; including Python and Django.

Recently, however, this became an issue again, as I attempted to install mod_wsgi in the MacPorts Apache. No matter how hard I tried, I just couldn’t get it working with more complex Python scripts. At this point, I sought more help from my friends and, after a couple of attempts to rectify the problem, was suggested the movement of my entire development environment to a dedicated virtual machine.

So, thanks to Danny Amey and Brad Wright, I’m now running a proper Ubuntu server as my development environment and have just as much control over it as I do on my live web server. In fact, the two are pretty much identical; and, best of all, I have access to all the files on my dev server using VMWare shared folders, which means I can use all my favourite OS X tools to create and edit files. I really can’t advocate this method enough.

For more information on creating such an environment, I recommend reading Brad’s post, “development virtual machines on OS X using VMWare and Ubuntu“.

Web Servers and Email: Postfix on Ubuntu

Tim — Sat, 11 Apr 2009 09:16:39 -0700

Setting up web servers is an oft-covered subject in the web development blogosphere. However, when I needed to set up a shared web and email server, I discovered that such documentation was somewhat more dispersed. With that in mind, and following the development of an email newsletter at work (in which I demonstrated several email testing techniques that were of interest to some of the other developers), I began drafting a blog post that would hopefully bring some of that information together. Sadly, I never finished.

Due to an ever more stressful workplace and the arrival of another child in the household, I took a somewhat forced break from blogging. This was entirely unintended and I’ve decided to get back on the wagon again by finishing the aforementioned post.

So here it is; the finished article on setting up Postfix on a shared mail and web server. Please remember that I’m technically a mail server numpty and probably won’t have all the answers to questions should you have them. However, feel free to leave them in the comments and hopefully someone out there (or someone I can poke for comment elsewhere) can help you out.

A bit of background

I made the jump from supported, pre-built hosting to entirely self-managed hosting last year. As a result, I soon discovered the need to handle email received by the various hosted domains on my server.

One solution to this problem would be to use Google Apps for Business. Too easy, drill sergeant; and also $50 more than I was willing to spend a year. Another solution would be to install my own email server and manage it myself. Despite this idea giving me The Fear™, it also meant I’d be learning something new — a buzz I find difficult to ignore at the best of times.

Given those options, I’m sure it’s not too hard to figure out which route I took, and the following blog post is a small amount of what I learnt in the process.

Hostname

Often, when building a web server, the server hostname is overlooked as unimportant. This is generally because the DNS system provides all the functionality needed when serving files over HTTP. However, when the server is required to handle email, it is important to set a hostname because it is used by the various email protocols as a source and destination identifier; ergo a key component in the identification of spam.

Postfix will let you send and receive mail just fine between accounts on a machine with a fantasy name. However, you cannot use a fantasy hostname in your email address when sending email via the Internet, because no-one would be able to reply to your mail. In fact, more and more sites refuse mail addresses with non-existent domain names.

Further to this, you may want to specify a fully-qualified domain name hostname. A hostname is considered to be a fully qualified domain name (FQDN) if all the labels up to and including the top-level domain name (TLD) are specified. The hostname “mail.nerfabulousdesigns.co.uk” terminates with the top-level domain “co.uk” and is thus fully-qualified. Depending on the system, an unqualified hostname such as “server1″ or “test-server” may be combined with default domain names in order to determine the fully qualified domain name. However, in most cases, a web server or mail server won’t be running in that sort of environment.

With all that in mind, it’s best to use a domain name you can access the DNS records for, as you’ll want to forward email to it using the MX record. An accepted standard, when working with mail, is to use “mail.mydomain.com”, however, in this case, we’re sharing email and web serving duties on the same box. A better solution would be to come up with any name that you can use to uniquely identify the server in your DNS settings. For the purpose of my examples, I’m going to work with the hostname “spartacus.nerfabulousdesigns.co.uk”.

DNS for your hostname

Before I start wading into the installing and configuring the server, it’s important to get the DNS set up in your DNS manager. So, if the IP address of our server is 123.456.789.123 we’ll want the following DNS records:

A Records:
spartacus.nerfabulousdesigns.co.uk -> 123.456.789.123
mail.nerfabulousdesigns.co.uk -> 123.456.789.123 MX Record:
mail.nerfabulousdesigns.co.uk

Any other domains you want to set up for email only need the following rules:

A Records:
mail.yourdomain.com -> 123.456.789.123 MX Record:
mail.yourdomain.com

Once you’ve saved those changes, and you’re awaiting DNS propagation, it’s time to get back to our server.

Server settings for your hostname

To find out the current hostname of your server (which is defined during installation), type the following:

$ hostname

The hostname command will output the current hostname if you run it without any parameters. If you want to see the fully qualified hostname, type the following:

$ hostname -f

To change your hostname temporarily, you can simply enter the new hostname as the first parameter of the hostname command. So, assuming I want to identify my server as “spartacus”, I’d enter the following:

$ hostname spartacus.nerfabulousdesigns.co.uk

Now we can check our change:

$ hostname
spartacus.nerfabulousdesigns.co.uk

Huzzah!

Unfortunately, that change will be reset should the machine be rebooted for any reason.

Not-huzzah!

To make the change permanent, you’ll need to edit the single-line /etc/hostname file:

$ sudo vim /etc/hostname

Update it to contain the following:

spartacus.nerfabulousdesigns.co.uk

Now you can trigger your change by restarting the hostname daemon:

$ /etc/init.d/hostname.sh start

However, you don’t need to do that just yet since you’ll probably want to restart your server later on in the process.

The hosts file

Next we’ll need to update the hosts file on the server so that it recognises the new hostname as a referer to the loopback IP 127.0.0.1. To do that, edit the hosts file with the following command:

$ sudo vi /etc/hosts

And then update it so that it contains a line similar to this:

127.0.0.1 spartacus.nerfabulousdesigns.co.uk

Once this file has been saved, we can do that reboot we were going to do earlier:

$ sudo reboot

Once your server is back up, the hostname change is complete. Your server will now use the loopback address when hostname references are used, instead of wasting valuable time checking DNS records.

Install Postfix

Now that the server is primed for email, it’s time to install postfix — and while we’re at it, let’s install mailx, an improved version of the mail email user agent program, and let’s also confirm we have telnet installed as we’re going to use it for testing later on:

$ sudo aptitude install postfix mailx telnet

As Postfix installs, you’ll be asked a number of configuration questions:

For “general type of mail configuration”, choose:

Internet Site

You will then be asked the fully qualified domain name of your server, which in our case is:

spartacus.nerfabulousdesigns.co.uk

Once you’ve done that, the installation is complete. Your web server is now capable of sending email; and to prove it, let’s just test it from the command line.

Testing

Let’s send an email to a real email address (remember to replace somebody@somewhere.com with the real email address of your choice):

$ mail somebody@somewhere.com

The mail program will now ask you for a subject. Once you press return, mail will now be accepting the body of your email (albeit without a prompt). To tell the program that you’ve reached the end of your email body, you’ll need to place a single full-stop character, “.”, on a line by itself (since the return character is perfectly valid in the body of your mail). Finally, the mail program will then ask you for any CC addresses.

Here’s an example of the complete screen output:

$ mail somebody@somewhere.com
Subject: Testing my all new email server!
Red leader, this is mail server. Are you receiving, over?
.
Cc:
$

You won’t receive any confirmation that the email has been sent — you should just return to the command prompt — however, as long as everything was set up correctly, you should have received the email.

The next step is to configure our mail server so that it knows what to do with received email.

Configuration

Configuration of a mail server is where all the pain and torment begins; but in this case, I only want to set up mail forwarding — which takes all the pain and torment away and leaves us with a fairly straight forward, unburdened mail server that’s using very little disk space. I could usually do this at the DNS level, but I chose not to in case I decided to work with real mail boxes in the future — and also because I was still relishing the learning experience. If you want to set up something different, I’m afraid you’re going to have to look elsewhere for a tutorial because I haven’t the time (or the patience) to go through the multitude of different set-ups that are possible.

The first step in setting up mail forwarding in Postfix is to create a virtual hash. To do that, we need to create the following file:

$ sudo vim /etc/postfix/virtual

And inside that file, we need to define some values. Here’s how we’d set up nerfabulousdesigns.co.uk as a mail forwarding domain:

john@nerfabulousdesigns.co.uk john.smith@yahoo.com
joe@nerfabulousdesigns.co.uk joe.bloggs@yahoo.com # For a catch-all, do the following
@nerfabulousdesigns.co.uk example@yahoo.com

Once you’ve saved that file, run the following command to create the postmap:

$ postmap /etc/postfix/virtual

Next you’ll also need to update this file:

$ sudo vim /etc/postfix/main.cf

Add your domain to virtual_alias_domains and your virtual hash to virtual_alias_maps like so:

virtual_alias_domains = nerfabulousdesigns.co.uk ...other hosted domains...
virtual_alias_maps = hash:/etc/postfix/virtual

Finally, you’ll need to do the following to restart Postfix:

$ postfix reload

Any emails to the domains or addresses you added to the virtual hash should now be forwarded as expected.

Telnet

Finally, let’s test the email server is working using Telnet. This is a handy little technique for checking everything is as expected, since it will also display responses at every stage of the mail sending process.

You can telnet from any machine that has it installed — and since we installed it on our server earlier, we can even telnet the server to itself, which might be useful when trying to track down problems.

To begin with, let’s Telnet to our server over the default SMTP port 25:

$ telnet mail.nerfabulousdesigns.co.uk 25

Which should trigger a response similar to this:

Trying 209.20.86.249...
Connected to mail.nerfabulousdesigns.co.uk.
Escape character is '^]'.
220 spartacus.nerfabulousdesigns.co.uk ESMTP Postfix (Ubuntu)

If the connection was a success, you should receive a 220 SMTP reply code. For more information, here’s a handy reference on SMTP reply codes.

The next thing we need to do, in the essence of being polite, is greet the mail server. We can do that one of two ways. We can either use the SMTP command, HELO:

HELO my.local.domain
250 spartacus.nerfabulousdesigns.co.uk

Or we can use the more recent EHLO command, which will return more detailed information about the server:

EHLO my.local.domain
250-spartacus.nerfabulousdesigns.co.uk
â€¦ More detailed info. hereâ€¦

Great, so now that we’ve identified ourselves, we’re reading to start defining the envelope for our message. First, we need to tell the mail server who the sender is:

MAIL FROM: 
250 2.1.0 Ok

Second, we need to tell the mail server who we’d like the email delivered to:

RCPT TO:joe@nerfabulousdesigns.co.uk
250 2.1.5 Ok

That’s it; the envelope is now complete. We can now move onto filling our message with data. To do that, we simply use the command DATA, which will then allow us to enter the subject and main content for our message:

DATA
354 End data with . 
Subject: Test Email
This is a test email.
.
250 2.0.0 Ok: queued as 40F438D023

Our message is complete, and has been queued by the mail server for redistribution to the recipient. We’re done here, so let’s quite out of the telnet session:

QUIT
221 2.0.0 Bye
Connection closed by foreign host.
$

And we’re now back at our original command prompt. If you check the mailbox of your recipient, you should have successfully received the email.

Summary

Despite the fact that I could probably have handled email in more efficient ways, I thoroughly enjoyed learning about Postfix and the black magic therein. I’m sure that some of my peers will have plenty to correct me on, and I sincerely hope they do; as previously stated, this was intended as a learning experience, and learning is what I have done.

Hopefully, some — if not all — the techniques used in this article will be of some use to you. Certainly, they have aided me in my understanding of core internet technologies.

Please feel free to comment, question, or to point out flaws with my methods. I hope the discussion is fruitful.

Practical Django Projects and Django 1.0

Tim — Thu, 25 Sep 2008 02:41:46 -0700

I recently purchased a copy of “Practical Django Projects“, by James Bennett, with the intention of diving straight in and learning Django. I’d prepared for this daring feat of code-ninjutsu [yes, that's the correct spelling] with a crash course in Python via “Dive Into Python“, by Mark Pilgrim.

This appears to have been the correct choice, as I’ve already acquired a confident grasp of the concepts and techniques of Django in less than a few days of playing with it (granted; with the odd reference look-up on http://www.djangobook.com/ and http://docs.djangoproject.com/en/dev/).

However, last night it all went a bit wrong when I upgraded my local version of Django from 0.96 to 1.0…

Most of my issues were due to the fact that some of the code in the examples, included in Practical Django Projects, was now out of date. However, I had expected this (thanks to warnings from Cyril Doussin and Steve Webster) and went about fixing the problems using the informative Porting Guide on the Django documentation site.

Following my extensive fixes, I was still experiencing issues when decoupling the Admin class from its relevant model. In most cases, this was a relatively easy process, but in one particular situation, where I had used edit_inline and needed to update the method to use the new InlineModelAdmin objects, I had a slightly confusing issue: I needed to add the InlineModelAdmin object to one of the django.contrib applications â€” FlatPage â€” to get the SearchKeyword application to work.

After a short period googling, I found an interesting, well-written blog post by Jason Broyles, “Add InlineModelAdmin to a django.contrib app“. Unsurprisingly, it seemed others had had exactly the same issue.

Jason’s final model.py and admin.py files were as follows:

# models.py from django.db import models
from django.contrib.flatpages.models import FlatPage class SearchKeyword(models.Model): keyword = models.CharField(max_length=50) page = models.ForeignKey(FlatPage) def __unicode__(self): return self.keyword

# admin.py from django.contrib import admin
from cms.search.models import SearchKeyword
from django.contrib.flatpages.models import FlatPage
from django.utils.translation import ugettext_lazy as _ class SearchKeywordInline(admin.StackedInline): model = SearchKeyword extra = 3 max_num = 6 class FlatPageAdmin(admin.ModelAdmin): fieldsets = ( (None, {'fields': ('url', 'title', 'content', 'sites')}), (_('Advanced options'), {'classes': ('collapse',), 'fields': ('enable_comments', 'registration_required', 'template_name')}), ) list_display = ('url', 'title') list_filter = ('sites', 'enable_comments', 'registration_required') search_fields = ('url', 'title') inlines = [SearchKeywordInline,] admin.site.unregister(FlatPage)
admin.site.register(FlatPage, FlatPageAdmin)

Jason followed these examples with the following explanation:

The above admin.py file will make it so the SearchKeyword model is accessible in the FlatPages admin interface. It does this by first creating the SearchKeywordInline class and adding it to a new FlatPage ModelAdmin function. I also didn’t want to loose out on any of the extra options from the default django.contrib.flatpages ModelAdmin so I copied lines 6 and 14-20 from it and added the code to my file. Line 23 then unregisters the default FlatPage ModelAdmin and line 24 registers this new one with my inlines included.

This solution fixed my problems, however I was concerned that copying and pasting code from django.contrib.flatpages ModelAdmin to admin.py was a massively inelegant solution. With that in mind, I came up with the following solution using inheritance:

# admin.py from django.contrib import admin
from cms.search.models import SearchKeyword
from django.contrib.flatpages.models import FlatPage
from django.contrib.flatpages.admin import FlatPageAdmin
from django.utils.translation import ugettext_lazy as _ class SearchKeyword_Inline(admin.TabularInline): model = SearchKeyword extra = 3 max_num = 6 class FlatPageAdmin(FlatPageAdmin): inlines = [SearchKeyword_Inline] admin.site.unregister(FlatPage)
admin.site.register(FlatPage, FlatPageAdmin)

This worked perfectly.

As a final note, I’d just like to thank Jason for posting his solution; without that much needed information, I probably would have spent a lot more time trying to fix the issue than was really required. It just goes to show that developers that blog are a huge benefit to confused developers everywhere.

The Joy of UNIX

Tim — Tue, 23 Sep 2008 05:33:20 -0700

One of the best things about developing from a Mac is the fact that it is a UNIX-based operating system. This means that my development environment is a far closer match to my production hosting environment (Linux) than it was when I was developing under Windows.

The net result of this shift in paradigm is a rapidly expanding knowledge of the command-set and OS structure. The exposure to UNIX has done me the world of good; particularly in regard to web development and server management. Sadly, the relative shelter I was accustomed to at Rentokil Initial, thanks to the inclusion of immensely talented and helpful server admins within our team, had hindered my development as a web developer, and provoked ridicule at the hands of my fellow Y! EU developers (in the nicest possible way, of course).

With all that in mind, here’s a bunch of stuff I’ve learnt that might be a useful reference resource…

Books

The first book I’d recommend reading for a good grounding in UNIX is “The UNIX Programming Environment“, by Brian W. Kernighan and Rob Pike. Although this is book is over 20 years old, it’s still highly relevant; Kernighan and Pike both worked at Bell Labs during the development of UNIX and contributed greatly to the OS and its school of thought. The book covers a huge expanse of subjects, from the basics through to full program development.

The second book, as is so often the case, is published by O’Reilly. “UNIX in a Nutshell“, by Arnold Robbins, is a great, up-to-date examination of the operating system, including a vast array of information perfect for beginners and experienced users alike.

Both of these books should provide enough knowledge for you to feel safe getting down and dirty with UNIX; so let’s dive in with a look at some useful keyboard and command line shortcuts:

Keyboard and Command Line Shortcuts

The following are common keyboard and command line shortcuts for use in UNIX-based systems. Please note that all UNIX variants differ and some of these shortcuts may not be available to you, or may require different syntax or keystrokes.

Keyboard Shortcuts

To begin with, here’s a list of useful keyboard shortcuts:

CTRL + A â€” Moves the cursor to the beginning of the line.
CTRL + E â€” Moves the cursor to the end of the line.
CTRL + U â€” Erases the complete line.
CTRL + B â€” Moves the cursor backward one character.
CTRL + F â€” Moves the cursor forward one character.
CTRL + H â€” Erase one character. Similar to pressing backspace.
CTRL + W â€” Deletes the last word typed.
CTRL + C â€” Cancels the currently running command.
CTRL + D â€” Logs out of the current session.

Most of those are self-explanatory, but you’ll wonder how you lived without them soon enough.

Command Line Shortcuts

Next, let’s take a look at some useful command line shortcuts, starting with directory navigation:

~ â€” References the current user’s home directory.
. â€” References the current directory.
.. â€” References the parent of the current directory.

Here’s a quick demonstration to hopefully make their use a little clearer (the names have been changed to protect the innocent). Note, this example uses several UNIX commands; cd = change directory, pwd = print working directory, ls = list contents of directory, and cp = copy files:

$ pwd
/Users/timbo
$ cd /var/www/nefarious/
$ pwd
/var/www/nefarious
$ cd ..
$ pwd
/var/www
$ ls
a-website another-website nefarious
$ cd /var/www2
$ pwd
/var/www2
$ ls
$ cp /var/www/* ./
$ ls
a-website another-website nefarious
$ cd ~
$ pwd
/Users/timbo

Now that we can effectively navigate the directory structure, let’s have a look at the incredibly useful history utility and its command line shortcuts:

!! â€” Re-run your last command.
!10 â€” Re-run line number 10 in the history.
!-n â€” Refer to the current command line minus n.
!?str? â€” Refer to the most recent command containing str (string).
!str â€” Re-run the last command ran that started with str (string).
!$ â€” Re-use the parameters from the previous command.

I won’t go into detail on each one of those, but here’s an example of the last one â€” arguably the most useful â€” in action:

$ pwd
/Users/timbo
$ ls projects/django/
cms coltrane davis
$ cd !$
cd projects/django/
$ pwd
/Users/timbo/projects/django

Summary

This is just a quick reference to some useful shortcuts; I could have delved deeper into setting up aliases in your profiles and making your prompt look pretty, but I thought I’d save that for a possible follow up post.

I hope somebody out there finds this useful; and feel free to add your own tips and tricks in the comments.

Back on the horse

Tim — Sun, 10 Aug 2008 04:03:01 -0700

Right then. After almost a years hiatus, it’s about time I got back into this blogging lark. Unfortunately, I have a plethora of subjects I’d like to cover and no idea where to start!

I think I’ll get the personal stuff out of the way first and then I can think about planning the web development stuff…

The biggest news is that I’m going to be a father again, but thankfully this time it’s just the one baby. Further to that, my twin girls are now 14 months old and extremely mobile (in horizontal and vertical directions — we’ve had to rearrange the furniture to discourage climbing).

I’m still working at Yahoo! in London and, despite the odd crisis of morale, am still enjoying working with what is arguably the best corporate web development team in the world. It’s a shame my commute is 2 hours each way otherwise I’d also be enjoying the more social aspects of the team as well (but since most evenings are spent on Xbox Live with my colleagues, I’m not missing too much).

I also found out that I have Coeliac Disease; a harsh name for what is, effectively, an allergy to gluten (found in wheat, barley, oats, and rye). This hasn’t been all that difficult to adapt to since my sister has been diagnosed since she was 3 years old — ;and my dad worked out, shortly after she was diagnosed, that he is too. This basically means that I’ve been used to shopping, cooking, and dining-out with Coeliacs since a very early age. Want to know the worst part of the allergy? I can no longer drink beer. Sweet, sweet beer. I really miss a tasty pint of Guinness.

Finally, I’ve also undertaken a few personal projects: a new rock band, a (possible) book on web development, a cooking blog (no link because it’s not up yet), and various other things.

So that’s the past year in a nutshell. I’m sure there’s lots more I could’ve written but it’s easier to just condense. Now I can get started on the web development related blog posts…

… although I still have no idea where to start!

Decade

Mon, 16 Nov 2009 09:32:05 -0800

Ten years ago today I registered socket7.net. It began hosted on a linux box in my closet for a good 2-3 years before finally landing on Dreamhost. It was the crux of teaching myself more about the web. Through it I expanded my knowledge of HTML, CSS, JavaScript, PHP and MySQL; not to mention everything behind the scenes like Apache, DNS, etc.

One year is an eternity on the internet. Hell, even a few days after something big breaks on the 'net if someone mentions it for the first time they get lambasted for reporting "old news." Ten years, then, is almost unquantifiable by those standards. So much has changed, and so much more will change in the next ten years.

As much as I tend to neglect the site lately, there's no denying that I wouldn't be who I am today without it. Here's to you, socket7.net. May you live another ten years on the web.

IE VPC Images now with WGA

Wed, 09 Sep 2009 12:38:23 -0700

Just when I was on a virtualization roll using VirtualBox for testing IE on my Mac, Microsoft stopped me dead in my tracks. The latest version of Internet Explorer Application Compatibility VPC Images now have the anti-piracy features of Windows Genuine Advantage enabled.

The hardware signature detected when a VPC image first boots in a non-VPC host is different from the signature present when the image was first created, presumably in VPC. This difference triggers WGA's re-activation requirement to make sure you're not a dirty pirate. Of course you're not a dirty pirate, but if you wish to re-activate Windows you're going to need the original Windows Product Key which isn't provided with the VPC images. Microsoft has effectively blocked end users from hosting the VPC images with anything but Virtual PC.

To their credit, Microsoft has done a great thing by providing these VPC images free of charge, and it makes sense they would use their own virtual disk format for distribution. But it doesn't make sense for them to deny non-Windows (thus, non-VPC users) the ability to use the images.

One of the largest complaints I hear from web developers who work in a non-Windows environment is the difficulty of testing in IE. Until recently, VirtualBox was working quite nicely for me in this regard. Microsoft would be doing another great thing by loosening the reigns a bit. Alternatively, they could release Virtual PC for other operating systems.

How to run IE on your Mac with VirtualBox

Tue, 11 Aug 2009 17:02:14 -0700

Recently at work I've switched from Parallels to VirtualBox for my virtualization needs. VirtualBox has come a long way recently, and I'm very pleased with its performance and ease of use. Parallels was starting to become noticeably slow, and was generally not working well for me anymore.

At first, I tried to convert my Parallels (.hdd) images to VM Ware (.vmdk) format, which is natively supported in VirtualBox. Virtual PC (.vhd) format is also natively supported. Since my Parallels images are fully licensed versions of Windows XP, I didn't want them to go to waste. Sadly, the 'converted' images were corrupt and weren't recognized by VirtualBox, and I ended up wasting several hours.

I decided to ignore Parallels completely and simply use the free Internet Explorer Application Compatibility VPC Images from Microsoft. I should have done this in the first place, because setting them up was a breeze with the help of the internet and a few helpful how-to articles I found. I could link to those how-to's here but instead I wanted to post my own condensed version of the steps I took setting everything up. Mainly for myself for future reference.

Step 1

Download and install VirtualBox and Stuffit Expander. You can skip Stuffit Expander if you already have a tool for opening self-extracting .exe files.

Step 2

Download the free Windows XP IE App Compat VPC images that you'll need and extract the .vhd images with Stuffit Expander (or your tool of choice).

Using Finder, navigate to /Library/VirtualBox and create a new folder for each image. Move the extracted .vhd files into these new folders.

Step 3

In Terminal, run the following for each image:

cd /Library/VirtualBox/<image>
VBoxManage internalcommands sethduuid <image>.vhd

This undocumented command will avoid UUID conflicts when creating multiple VMs. If you're only setting up one VM, you can skip this.

Step 4

In VirtualBox, create a new Windows XP VM and follow the prompts. When asked, choose Use existing hard disk. Click the folder icon to open the Virtual Media Manager. Click Add and navigate to a .vhd file you extracted and relocated in step 2. Click Select to choose the new virtual disk. Finally, click Next then Finish.

Step 5

Start your new VM. Close/cancel any Windows popups or dialogs that appear. There will be quite a few.

From the VirtualBox VM menu bar in OS X select Devices > Install Guest Additions... and follow the Windows prompts. Reboot Windows when asked.

Step 6

Now it's time to install the network drivers. Why this isn't covered by the VirtualBox Guest Additions is beyond me.

In Windows go to Start > Run... and run the following:

D:\VBoxWindowsAdditions-x86.exe /extract /D=C:\Drivers

Back to Start > Run... and run:

devmgmt.msc

In Device Manager, find Network adapters > Ethernet controller. Right click it, and select Update Driver....

In the Hardware Update Wizard choose No, not this time and click Next. Select Install from a list or specific location and click Next. Check the option Include this location in the search and paste the following into the Location box:

C:\Drivers\x86\Network\AMD

Alternatively, you can choose Don't search ... and navigate to the location manually. Click Finish and you should have a working network connection.

Step 7

In Device Manager, find Batteries > Unknown device. In my case, there were two. Right click each and select Disable and confirm. These appear to be VirtualPC specific. In any case, they aren't really necessary anyway.

Next find Universal Serial Bus controllers > Universal Serial Bus (USB) Controller. Again, there were two. Disable them as well.

Finally, find Sound, video and game controllers > Multimedia Audio Controller and disable it.

Repeat steps 4-7 to set up additional VMs, and enjoy your new VirtualBox goodness.

Jabber::Bot now on GitHub

Wed, 25 Feb 2009 16:11:33 -0800

I've finally gotten around to tossing Jabber::Bot up on GitHub. There have been several requests in the past few months to add group chat support and other features, and I haven't had time to look into it. I've even had some patches sent via email that I haven't been able to get to (thanks Matt!).

Perhaps the magical powers of GitHub can give Jabber::Bot a new life.

Kick ass wisely.

Thu, 12 Feb 2009 17:47:22 -0800

Looking at the archives here, it's become obvious I've nearly forgotten how to blog since starting at Yahoo! almost a year and a half ago. It's not that I haven't been doing anything interesting, or come across interesting things to talk about — far from it.

I've been doing and seeing some of the most amazing things I've ever done or seen, from a career perspective. But, I have found that doing kick ass things at work has made me less likely to do kick ass things while not at work. And those non-work kick ass things are what I used to like to blog about. Doing kick ass things is a tons of fun, but it takes a lot of effort. One shouldn't do kick ass things all the time, or else one might run out of ass to kick. So I save my ass kicking energy for the asses that most need kicking.

Most of my time since joining Yahoo! has been spent working on SearchMonkey, and jumping between various other kick ass projects. Of course, there have also been a good number of not so kick ass projects. Again, it's important to not run out of ass to kick, or energy to kick ass. On top of that, be sure to kick the right ass, not just any old ass that comes along. Even if an ass appears to be begging you to kick it. Even if it offers you cash. Even if peer pressure makes you feel like you'd be less kick ass if you didn't kick this particular ass. There's not much worse than a wasted or misplaced ass kick. Kick ass wisely.

Anyway, late last year I had decided this here blog could use a redesign, and I could use a swift kick in the ass to get things moving again. But, now I'm not so sure. I haven't blogged because I haven't had anything much to say, and that's fine. Despite the kick ass things going on at work I just haven't found myself thinking, "Hey, I should blog about this!" The day I do have something to say (like today) it'll still be here waiting. In the meantime, for all the little blips of life from "whoa" to "wtf" there's always Twitter.

Links for 2008-12-17 [del.icio.us]

Thu, 18 Dec 2008 00:00:00 -0800

Laconica
Open source Twitter clone.

Links for 2008-10-28 [del.icio.us]

Wed, 29 Oct 2008 00:00:00 -0700

They were all in love with life, drinking from a fountain...

Links for 2008-10-15 [del.icio.us]

Thu, 16 Oct 2008 00:00:00 -0700

Blackbird - Open Source JavaScript Logging Utility

Links for 2008-09-08 [del.icio.us]

Tue, 09 Sep 2008 00:00:00 -0700

Typechart
CSS generator for pretty web typefaces.

Links for 2008-07-08 [del.icio.us]

Wed, 09 Jul 2008 00:00:00 -0700

Opera Web Standards Curriculum

Links for 2008-06-23 [del.icio.us]

Tue, 24 Jun 2008 00:00:00 -0700

Jiffy Firefox Extension

Links for 2008-06-19 [del.icio.us]

Fri, 20 Jun 2008 00:00:00 -0700

Ack in Project TextMate bundle

OR: D.A.R.C. Observatory, 11/21/2009

Sun, 22 Nov 2009 16:49:40 -0800

After much consideration, I decided to head to the DARC observatory on Saturday night. It was a little bit of a gamble since the clear sky clock had forecast below average to average transparency, which usually means that wispy clouds would pass overhead, possibly ruining my chances of detecting the most subtle details on the faint targets I was planning to observe. But since it was the last opportunity to go out and observe during this new moon, I took the chance and hoped for the best.

I arrived after dark, around 6pm, and quickly setup my equipment next to Peter Nastcher and his 24″ Starmaster telescope (always a prime spot) Shortly after, a “show and tell” presentation took place inside the observatory building. Dr. Bob Caton talked about some of the photographical work he’s done recently with his 4″ wide field refractor and 20″ Ritchey-Chretien telescope. Daniel Stefanescu talked about M1, and I briefly showed off a few photographs of the early construction of my new 16″ F/4 telescope.

Around 8:50pm, the sky got dark enough to start observing. As expected, thin clouds were passing overhead. Thankfully, they remained to the north, and never really interfered with my plans to observe Herschel II objects in Cetus. By 10pm, the sky was completely clear, and the transparency was actually very decent. The NELM was 6.6 in Aries. After midnight, the air became slightly drier and the transparency slightly better, bumping the NELM to 6.7. The wind was non-existant. The temperature was hovering between the mid and the low 30s. The humidity was high, but the optics remained mostly dew free for the entire night.

Among the highlights of the night, I saw the horse head nebula (Barnard 33) for the first time, using my 12″ telescope and a UHC filter. I was actually surprised by how easy it was to detect. Using a Panoptic 27 without a filter, I was even able to see it along with Alnitak (Zeta Orionis) and the Flame Nebula (NGC 2024) in the same field of view. What an incredible sight! We also looked at the horse head nebula through Peter’s 24″ scope, and the horse head shape was unmistakable.

The deep red/pink color in M42 was very obvious around the bright blue-green core region and in the two far flung extensions. The E and F stars in the trapezium were easily detected. Finally, Sirius’ companion, affectionately called “the pup”, was seen by several observers.

I would like to thank Dr. Lee Hoglan and Dr. Bob Caton for their continued hospitality. It was a good night, and I’m already looking forward to the next opportunity to observe there. Below is my log for the night. Cheers!

Location: D.A.R.C. Observatory [Elevation 1400ft]
Telescope: Meade Lightbridge 12″ F/5
Eyepieces used:
- Televue Panoptic 27mm (56x – 1.2° TFOV)
- Televue Nagler 16mm type 5 (95x – 52′ TFOV)
- Televue Nagler 9mm type 6 (169x – 29′ TFOV)
- Televue Nagler 7mm type 6 (217x – 22′ TFOV)
- Televue Nagler 5mm type 6 (305x – 16′ TFOV)
(All times are PST)

NGC 151 GX Cet 00h34m34.6s -09°38′56″ 12.3 mag 09:05pm
2.5′x1′ elongated ENE-WSW. Very faint halo. Moderately faint core, round, approximately 30″ in diameter. Stellar nucleus. Moderately bright star just outside the halo at the ENE end.

NGC 217 GX Cet 00h42m05.7s -09°57′55″ 13.5 mag 09:20pm
Moderately faint core about 1′x30″ elongated WNW-ESE. Very faint stellar nucleus. Extremely faint halo roughly 2′ in length elongated WNW-ESE.

NGC 337 GX Cet 01h00m21.9s -07°31′21″ 12.0 mag 09:50pm
2′x1.2′ elongated NW-SE. Moderately faint, very slightly brighter in the center. Subtle hints of mottling detected (?)

NGC 357 GX Cet 01h03m53.7s -06°17′02″ 13.2 mag 10:05pm
Round, 20″ in diameter. Hints of a stellar nucleus (?) Only the core was positevely detected. The halo must be extremely faint! Moderately faint star about 40″ E.

NGC 636 GX Cet 01h39m38.2s -07°27′37″ 12.4 mag 10:15pm
Fairly bright core, round, 20″ in diameter, fairly bright stellar nucleus, very faint halo, gradually dimmer toward the outside. Did not note the extent and position angle of the halo.

NGC 428 GX Cet 01h13m28.0s +01°02′13″ 11.9 mag 10:45pm
3′x2′ elongated NW-SE, moderately faint. Very slightly brighter core, round, 45″ in diameter. Two bright stars (mag 8.6 and 8.7) about 8′ W and 8′ NNE. Interestingly enough, I did not note the presence of two prominent albeit dimmer stars (mag 11.9 and 12.5) located just outside the halo to the S and the NW.

NGC 1045 GX Cet 02h41m00.0s -11°14′01″ 13.5 mag 10:55pm
Fairly faint, round, 30″ in diameter. Moderately faint stellar nucleus.

NGC 991 GX Cet 02h36m04.2s -07°06′32″ 12.9 mag 11:30pm
Very faint uniform glow about 1.5′ in diameter, very slightly brighter core. Moderately bright star (mag 13.4) about 1.5′ S.

NGC 1035 GX Cet 02h40m00.6s -08°05′23″ 12.9 mag 11:45pm
2′x30″ elongated NNW-SSE, moderately faint, fairly uniform. Moderately faint star at the SSE tip.

NGC 1087 GX Cet 02h46m57.7s -00°27′17″ 11.5 mag 12:05am
2′x1′ elongated N-S. Moderately bright, pretty uniform, very slightly brighter core.

NGC 1090 GX Cet 02h47m06.5s -00°12′11″ 12.6 mag 12:10am
2′x1′ elongated E-W, fairly faint, pretty uniform. Slightly brighter core about 20″ in diameter. Pretty faint star (mag 15.2) just 45″ S.

NGC 1032 GX Cet 02h39m56.4s +01°08′21″ 12.6 mag 12:30am
2′x45″ elongated ENE-WSW. Moderately faint core, about 30″ in diameter, slightly elongated ENE-WSW. Three moderately bright stars (mag 13.2, 13.2 and 13.5) to the E, NE and N, about 1.5′ from the core.

NGC 1070 GX Cet 02h43m55.6s +05°00′47″ 12.6 mag 12:40am
Moderately faint core, about 30″ in diameter, very slightly elongated N-S. Very faint halo, about 1.5′ in diameter.

NGC 1073 GX Cet 02h44m13.0s +01°25′13″ 11.6 mag 01:05am
Round, uniform, faint glow, about 4′ in diameter. Extremely slightly brighter core. A DSS image shows that this galaxy is a barred spiral. I did not see any visual evidence of that.

NGC 1514 (Crystal Ball Nebula) PN Tau 04h09m57.0s +30°48′15″ 10.8 mag 01:30am
Moderately faint, roundish nebulosity, about 2′ in diameter, very slightly elongated NNW-SSE, surrounding a bright star (mag 9.4). Appears uniform at first. A UHC filter reveals irregularities in brightness. The nebulosity appears slightly dimmer around the central star, although this may be a visual artifact due to the presence of the bright central star (a DSS image shows that it is indeed a real feature). Hints of mottling. Brighter knots to the SSE and NNW.

NGC 1762 GX Ori 05h04m10.0s +01°35′21″ 13.4 mag 01:40am
Very faint glow, less than 1′ in diameter, possibly slightly elongated N-S. Moderately faint superimposed star just 15″ E.

NGC 1587 GX Tau 04h31m12.8s +00°41′09″ 12.7 mag 01:50am
Round, about 45″ in diameter, moderately faint, gradually brighter toward the center. Forms a nice tight pair with NGC 1588, located just 1′ E.

From Dream To Reality

Sat, 21 Nov 2009 10:40:10 -0800

I have always been a pragmatic person. While designing my telescope using computer assisted design software was a very rewarding experience, I have felt a much greater sense of accomplishment making it happen. This is why I am very proud to show off the mirror box of my future telescope. This is the result of 4 nights of work at the sawdust shop in Sunnyvale. I am very happy with the way it turned out, and look forward to getting more work done in the next few weeks! Note: click on the thumbnails below to see the full size photographs.

OR: Dinosaur Point, 11/14/09

Sun, 15 Nov 2009 19:49:45 -0800

After a short (55 minutes) and pleasant drive, I arrived at Dinosaur Point early (around 4pm) because of the new access rules, and to have plenty of time to setup my equipment. Albert Highe, George Feliz and Peter Natscher arrived shortly after and setup right next to me. Dave Cooper, Jamie Dillon, Bob Jardine and Al Howard were a little further down the parking lot. What a great team of observers! (ok, Al doesn’t really count as an observer, but he’s cool nonetheless)

The wind was non-existant. The temperature dropped from the low 60s to the mid to low 40s, which is not a huge difference. This always helps with the seeing, especially in large newtonians. The transparency was about average until about 9pm, when it got slightly better. At 9:30pm, I did a NELM test in Triangulum and obtained 6.3, which is pretty good for the site.

Since I don’t have a whole lot of Herschel 400 targets left around this time of the year, I decided to observe mostly Herschel II objects, and finished the night with a couple of galaxy groups from Alvin Huey’s “Selected small galaxy groups” observing guide. Among the highlights of the night, we observed NGC 891 and M 33 through Peter’s 24″ Starmaster telescope. The views were absolutely breathtaking! We also looked at the hydrogen filaments inside M 1, which we also detected through George’s 13″ scope, and to a lesser extent in my own 12″ scope.

Overall, I was very pleased with the evening, and it reminded me of why I like Dinosaur Point so much: short easy drive, fairly good skies, setup on a paved parking lot, in bed by 2am after 6+ hours of observing! Below is my log for the night. Cheers!

Location: Dinosaur Point [Elevation 648 ft]
Telescope: Meade Lightbridge 12″ F/5
Eyepieces used:
- Televue Panoptic 27mm (56x – 1.2° TFOV)
- Televue Nagler 16mm type 5 (95x – 52′ TFOV)
- Televue Nagler 9mm type 6 (169x – 29′ TFOV)
- Televue Nagler 7mm type 6 (217x – 22′ TFOV)
- Televue Nagler 5mm type 6 (305x – 16′ TFOV)
(All times are PST)

NGC 7184 GX Aqr 22h03m13.7s -20°45′54″ 11.7 mag 06:20pm
Moderately faint slightly elongated core, stellar nucleus, very faint halo elongated 3×1 ENE-WSW. Very faint superimposed star about 1′ ENE of nucleus. Fairly bright star at the ENE end of the halo.

NGC 7218 GX Aqr 22h10m45.0s -16°36′39″ 12.4 mag 06:40pm
2′x1′ elongated NNE-SSW. Gradually brighter toward a moderately faint core. Moderately bright superimposed stars 1′ E and 1′ NNE. Best seen at 217x.

NGC 7171 GX Aqr 22h01m35.0s -13°13′17″ 13.1 mag 06:50pm
Moderately large, very low surface brightness, only very slightly brighter in the core. Elongated 3:2 NW-SE. Extremely faint superimposed star 1′ E.

NGC 7377 GX Aqr 22h48m20.9s -22°15′31″ 12.1 mag 07:10pm
Small (about 1′ in diameter), round and fairly bright. Stellar nucleus. A small grouping of 5 fairly bright stars is located just SW of this galaxy.

NGC 7392 GX Aqr 22h52m21.9s -20°33′18″ 12.6 mag 07:20pm
2′x1′ elongated WNW-ESE. Stellar nucleus embedded inside a fairly bright small core.

NGC 7600 GX Aqr 23h19m26.0s -07°31′25″ 12.9 mag 07:35pm
1.5′x1′ elongated ENE-WSW. Moderately faint stellar nucleus.

NGC 7156 GX Peg 21h55m04.6s +02°59′32″ 13.3 mag 08:00pm
Small (about 1′ in diameter), round, dim, fairly uniform, only slightly brighter core.

NGC 7177 GX Peg 22h01m10.3s +17°47′23″ 11.9 mag 08:10pm
Round bright core, about 1′ in diameter, surrounded by a faint halo 2.5′x1.5′ elongated E-W. Stellar nucleus. Best seen at 217x.

NGC 7332 GX Peg 22h37m54.1s +23°51′14″ 12.0 mag 08:25pm
Very bright stellar nucleus embedded inside a bright compact core. Moderately faint halo roughly 2.5′x45″ elongated NNW-SSE. Forms a beautiful pair with NGC 7339 at 217x.

NGC 7742 GX Peg 23h44m47.4s +10°49′32″ 12.3 mag 08:50pm
Small (about 1′ in diameter), round, fairly bright, gradually brighter to a stellar nucleus. Mag 12.7 star just 1.5′ ESE.

NGC 23 GX Peg 00h10m25.7s +25°59′00″ 12.8 mag 09:05pm
Small (< 1') round faint halo, fairly bright almost stellar core. A fairly bright star is at the SSW end. Forms a nice pair with dimmer NGC 26.

NGC 604 BN Tri 01h35m11.7s +30°50′17″ mag 09:25pm
This nebula physically belongs to M33. Seen easily in a 12″ scope as a bright round patch about 1′ in size. In Peter Natscher’s 24″ scope, its shape appears slightly irregular.

NGC 598 (M 33) GX Tri 01h34m26.5s +30°42′53″ 6.4 mag 09:25pm
Barely detected naked eye. Easily seen in 9×50 finder scope, including the broad S-shaped spiral structure. Elongated 3:2 NNE-SSW. Large bright round core. Appears amazing in a 12″ scope, with lots of fine details easily visible in the spiral arms. Absolutely dazzling though Peter Natscher’s 24″ scope! Several NGC objects physically belong to M33: NGC 604, NGC 595, NGC 592 and NGC 588. All were seen very easily in a 12″ scope.

NGC 7640 GX And 23h22m36.4s +40°54′19″ 11.6 mag 09:40pm
6′x1.5′ elongated NNW-SSE. Moderately faint halo with a slightly brighter core. Slight impression of mottling. Fairly bright superimposed star just SE of the core. 2 superimposed stars in the halo N of the core.

NGC 206 BN And 00h41m04.3s +40°47′34″ mag 09:45pm
Star cloud that physically belongs to M31. Located about 40′ SW of M31’s core. Roughly 5′x2′ elongated N-S. Fairly faint.

NGC 214 GX And 00h42m01.3s +25°33′29″ 12.9 mag 10:00pm
Fairly small (1.5′x1′), elongated ENE-WSW, moderately faint, pretty uniform, slightly brighter core. Barely detected stellar nucleus.

NGC 1068 (M 77) GX Cet 02h43m13.3s +00°01′54″ 9.7 mag 10:20pm
Roughly circular, moderately faint halo, about 2′ in diameter, very slightly elongated N-S. Bright slightly S-shaped core elongated 3:2 NE-SW containing a very bright stellar nucleus. Some mottling visible (hints of broader spiral arms emanating from the core)

NGC 672 GX Tri 01h48m30.0s +27°29′11″ 11.4 mag 10:50pm
5′x2′ elongated ENE-WSW, pretty uniform, moderately bright. Forms a nice pair with much dimmer galaxy IC 1727 located about 8′ SW.

NGC 925 GX Tri 02h27m54.9s +33°37′36″ 10.6 mag 11:10pm
4′x2′ elongated WNW-ESE. Moderately faint, pretty uniform, slightly brighter core. Many fairly bright field stars enhance the view.

NGC 890 GX Tri 02h22m38.8s +33°18′53″ 12.8 mag 11:15pm
Small (about 1.5′x45″) elongated NE-SW. Fairly bright, gradually brighter to an almost stellar nucleus.

NGC 68 group
Pretty compact. A few field stars distract the observer, making the 3 galaxies in the center harder to spot.
NGC 68
NGC 69 (very faint)
NGC 70
NGC 71
NGC 72
NGC 72A (very faint)
NGC 74 (very faint)

NGC 383 group
Fairly bright members, loose group.
NGC 373 (faint)
NGC 375 (faint)
NGC 379
NGC 380
NGC 382
NGC 383
NGC 384
NGC 385
NGC 386 (faint)
NGC 387 (very faint)
NGC 388

Entering the World of Amateur Telescope Making

Sat, 07 Nov 2009 11:05:45 -0800

For the past few weeks, I’ve been designing my next telescope, an Albert Highe inspired 16″ dobsonian. The decision to build my own telescope stemmed from the fact that I knew exactly what I wanted, and I could not get it from any of the well known US telescope manufacturers. In addition, I wanted to learn new skills, including computer assisted design and woodworking. The image to the left is the latest rendering of the optical tube assembly as I envisioned it (click for a higher resolution image). Below are my original specifications:

The mirror box and rocker box must fit together within 24″ x 24″ x 18″, so that it fits in my car.
The eyepiece height cannot exceed 64″, so that I don’t have to use a step stool or a ladder.
The weight of the heaviest component cannot exceed 40 lbs, so that I can easily lift it.
Overall cost must not exceed $5,000.

I opted for a 16″ F/4 mirror, which should fit my goals. In the next few months, I will be posting photographs as I build and assemble the various parts that make up the telescope. Stay tuned!

OR: Deep Sky Ranch, 10/16/09: Tarentulas and Galaxy Clusters

Sat, 17 Oct 2009 15:44:08 -0700

I had originally planned to go to the DARC observatory on Saturday night. However, due to a dismal weather forecast, I was forced to rethink my plans. The forecast for Friday night was actually looking pretty good, and an opportunity came up to go to Willow Springs. I had never been up there before, so I jumped at the opportunity, packed my gear in a hurry, and headed down to Deep Sky Ranch.

While driving on Little Panoche Rd, something interesting caught my eye: a tarantula was slowly crossing the roadway. On the way back this morning, I pulled over to take this photograph:

Willow Springs is a great place for deep sky visual observing. The sky is fairly dark, and the only really noticeable light dome, emanating from Hollister, is not much of a problem. Here is a summary of last night’s weather conditions:

No wind.
Temperatures in the low 50s.
Average seeing.
Average transparency.
Very high humidity: my table top and paper charts were soaked, and my eyepieces were covered in dew.

Were present Steve Gottlieb, Mark Wagner, Mark Johnston, Rogelio Bernal Andreo, Richard Navarette and of course our gracious host, Kevin Ritschel. Here is a photo of famous DobZilla, Kevin’s 30″ dobsonian telescope:

Since I am almost done with the Herschel 400 list, I had decided to start a new project: observing the galaxy groups featured in one of Alvin Huey’s free downloadable observing guides: Selected Small Galaxy Groups. These galaxy groups are fascinating because they are a real test of the observer’s visual acuity and equipment. Although the brightest members are usually immediately detected, these groups reward patient observers. The more time is spent at the eyepiece, the greater the number of galaxies that become visible! This way, I was able to detect a good number of members after spending about 30 minutes on a given group. I did not take detailed notes of my observations this time around, but I did write down the list of members I was positively able to detect. All observations were done using my Meade Lightbridge 12″ F/5 dobsonian telescope, and using mostly my 9mm Televue Nagler type 6.

NGC 383 Group (also known as the Pisces Group)
NGC 373/NGC 375/NGC 379/NGC 380/NGC 382/NGC 383/NGC 384/NGC 385/NGC 386/NGC 387 (hard!)/NGC 388

NGC 3 Group (Pisces)
NGC 3/NGC 4 (hard!)/NGC 7837/NGC 7838

NGC 507 Group (Pisces)
NGC 494/NGC 495/NGC 496/NGC 499/NGC 501/NGC 503/NGC 504/NGC 507/NGC 508/NGC 515/NGC 517/IC 1682/IC 1687/IC 1689/IC 1690

NGC 80 Group (Andromeda) – Featured in the current edition of S&T’s “Going Deep” column.
NGC 79/NGC 80/NGC 81 (hard!)/NGC 83/MGC 4-2-10 (PGC 1384, and also sometimes incorrectly labeled NGC 84) (hard!)/NGC 85/NGC 86/NGC 90/NGC 93/NGC 94/NGC 96
I also observed this group through Mark Johnston 18″ scope, and confirmed seeing some of the dimmest members.

Perseus Cluster
NGC 1129/NGC 1130/NGC 1131/IC 265/MCG 7-7-3 (PGC 10953)/MCG 7-7-8 (PGC 10980)

Big thanks to Kevin Ritschel for hosting this star party and for sharing stunning views through DobZilla. Also, big thanks to Alvin Huey for putting together the observing guides and making them freely downloadable on his web site.

OR: Henry Coe State Park, 10/10/09

Sun, 11 Oct 2009 20:04:30 -0700

Mark Johnston, Peter Krottje and I met at the overflow parking lot at Henry Coe state park last night. I brought my Meade Lightbridge 12″ (after flocking the inside of the UTA with black velvet ordered from Edmund Optics), while Mark brought both his 18″ Starmaster dobsonian as well as a newly acquired equatorially mounted 190mm Orion Mak-Newt. Peter was using his homemade 10″ ultra light dobsonian.

The sky grew darker as the evening went on thanks to the fog dimming the lights emanating from San Jose, Morgan Hill and Gilroy. By 11pm, we were getting SQM readings of almost 21.2, which is pretty good for Henry Coe. The transparency however was clearly below average, and the low level haze was causing some obvious dimming up to 15 degrees of elevation, especially to the south and the west. The sky was however more than decent between the north east and the south east, which is where I spent most of the night, observing mostly open clusters in Andromeda, Lacerta and Cassiopeia.

Here’s a summary of the weather conditions:

Average seeing.
Mild (but annoying) westerly wind.
Surprisingly cold (I had to really bundle up, and almost needed gloves by 11pm)

Here are some of the highlights of the evening:

18 open clusters from the Herschel 400 list. Most of them were honestly not very interesting, except maybe the ET cluster, and a couple of clusters in Cassiopeia that happened to fit within the same FOV of my Pan 27, showing some differences in structure.
We observed the California nebula with my 50mm finder scope using a 16mm Nagler type 5 and a UHC filter. We got a much better view through Mark’s 190mm Orion Mak-Newt using a wide field eyepiece and an H-beta filter.
We saw the fourth flea around NGC 7331 through Mark’s 18″ scope, and I later caught a glimpse of it in my 12″ scope, although it was coming in and out of view (knowing where to look really helps!)
The GRS was transiting. A darker slightly elongated region was spotted right next to it. Probably some disturbance in the equatorial band.
There was a nice shadow transit by Io on Jupiter.

It was my first night using the Orion self-centering adapter (instead of a standard 1 1/4″ adapter with a tightening ring) and I have to say that I really liked it! I’m sure it will be very handy this winter when switching eyepieces while wearing gloves.

I took off around midnight, and was soundly asleep by 1am. Overall, it was a good night, and I’m glad I went out. I’m looking forward to observing under darker skies next weekend to start on the Herschel II list, since I’m running out of Herschel I objects. Below is my log for the night. Cheers!

Location: Henry Coe state park [Elevation 2600 ft]
Telescope: Meade Lightbridge 12″ F/5
Eyepieces used:
- Televue Panoptic 27mm (56x – 1.2° TFOV)
- Televue Nagler 16mm type 5 (95x – 52′ TFOV)
- Televue Nagler 9mm type 6 (169x – 29′ TFOV)
- Televue Nagler 7mm type 6 (217x – 22′ TFOV)
- Televue Nagler 5mm type 6 (305x – 16′ TFOV)
(All times are PDT)

NGC 7296 OC Lac 22h28m27.3s +52°22′19″ 9.7 mag 08:00pm
About 30 stars, a dozen of which are fairly bright, grouped within an area roughly 4′ in diameter. Relatively inconspicuous, best seen at 217x.

NGC 7243 OC Lac 22h15m32.9s +49°57′09″ 6.7 mag 08:05pm
About 80 stars, 20 of which are pretty bright, within an area 25′ x 15′ elongated NE-SW. The center region, delimited to the west by a nice double star, is very sparsely populated.

NGC 7209 OC Lac 22h05m32.1s +46°32′11″ 7.8 mag 08:10pm
About 80 stars, 40 of which are uniformly pretty bright, within an area 20′ x 15′ elongated NNE-SSW. Moderately bright and slightly orange star near the center. Bright yellow star (HT Lac, mag 6.2) about 15′ N.

NGC 7686 OC And 23h30m37.3s +49°11′31″ 5.6 mag 08:15pm
About 30 moderately faint to moderately bright stars grouped within an area 15′ in diameter centered on a bright yellow-orange star (HD 221246, mag 6.2)

NGC 7789 OC Cas 23h57m56.4s +56°46′02″ 7.5 mag 08:50pm
Very rich and fairly compact. Very large number of moderately faint stars of comparable brightness scattered pretty uniformly within a roughly circular area 15′ in diameter.

NGC 7790 OC Cas 23h58m56.7s +61°16′01″ 7.2 mag 09:00pm
About 20 moderately faint stars within an area roughly 5′ x 3′ elongated E-W. The proximity of numerous star clusters in the vicinity (esp. NGC 7788) makes this cluster interesting to track down.

NGC 129 OC Cas 00h30m36.0s +60°16′34″ 9.8 mag 09:10pm
Located 15′ NNW of a bright star (HD 2626, mag 5.9) This cluster contains a good number of moderately bright to fairly bright stars. It is not very well delimited, especially to the north, making star counts unreliable. A few fairly bright stars form a ring elongated 3:2 NW-SE.

NGC 136 OC Cas 00h32m07.4s +61°34′04″ mag 09:20pm
Maybe a dozen faint stars grouped within an area 1.5′ in diameter. About 6′ NE of a fairly bright yellowish star (SAO 11238, mag 8.4) This cluster is so inconspicuous that a good atlas is a must to confirm that you’re indeed looking at the right spot.

NGC 381 OC Cas 01h08m59.4s +61°38′19″ 9.3 mag 09:30pm
Fairly sparse cluster containing about 30 moderately faint to moderately bright stars within an area roughly 6′ in diameter. Best seen at 169x.

NGC 225 (Sailboat Cluster) OC Cas 00h44m16.8s +61°49′55″ 8.9 mag 09:35pm
About 30 bright stars within an area 12′ x 8′ elongated NE-SW. Requires some imagination to see the outline of a sailboat…

NGC 457 (ET Cluster) OC Cas 01h20m15.4s +58°20′27″ 5.1 mag 09:40pm
Two bright stars (mag 5 and 7, the one to the NE being the brighter), located at the western end of the cluster, about 3′ apart, form the eyes of the ET cluster. A narrow group of stars, elongated NW-SE, form the body, and a few more bright stars form the arms.

NGC 559 OC Cas 01h30m14.5s +63°21′35″ 7.4 mag 10:25pm
About 30 fairly faint stars within an area roughly 5′ in diameter. Best seen at 169x.

NGC 436 OC Cas 01h16m38.2s +58°51′58″ 9.3 mag 10:25pm
Fairly compact cluster containing about 25 moderately bright to moderately faint stars within an area 5′ in diameter. Best seen at 169x.

NGC 637 OC Cas 01h43m49.4s +64°05′29″ 7.3 mag 10:35pm
Small, compact group of about 20 stars, 5 of which are fairly bright, the others are fairly faint, within an area roughly 3′ in diameter elongated NE-SW. Best seen at 217x.

NGC 654 OC Cas 01h44m44.2s +61°56′11″ 8.2 mag 10:40pm
About 40 moderately faint stars within 5′. Best seen at 169x. Located 3′ NNW of a bright yellow-orange star (HD 10494, mag 7.3) Fits with NGC 663 in the same field of view of my Panoptic 27mm eyepiece, showing some nice difference in structure.

NGC 663 OC Cas 01h46m53.1s +61°17′10″ 6.4 mag 10:40pm
About 80 fairly bright stars within an area 15′ x 10′ elongated NW-SE. Fits with NGC 654 in the same field of view of my Panoptic 27mm eyepiece, showing some nice difference in structure.

NGC 659 OC Cas 01h45m07.6s +60°43′29″ 7.2 mag 10:45pm
About 15 moderately faint stars within an area 4′ in diameter. Best seen at 169x. Located about 15′ NE of a trio of bright stars.

NGC 752 OC And 01h58m18.5s +37°50′08″ 6.6 mag 10:50pm
Seen with the naked eye as a fuzzy spot between Tri and And. Best seen using the 50mm finder scope using a 16mm Nagler type 5 eyepiece. In my 12″ scope, using a Panoptic 27mm eyepiece, more than 100 bright stars are visible. Two bright yellowish stars were seen near the south-western boundary (56 And and HD 11727, mag 5.7 and 5.9 respectively)

CalStar 2009

Fri, 25 Sep 2009 11:58:23 -0700

I was only able to attend the first two nights of CalStar, an annual Star Party that takes place at Lake San Antonio in California. The first night (Thu, Sep 17), the transparency was mediocre (when I called it a night, around 2am, Jupiter was surrounded by a bright halo…). Therefore, I used that first night to focus on bright easy objects. The second night (Fri, Sep 18) was much better with an NELM of 6.7 in UMi. Seeing on both nights was about average. However, the high temperatures during the day caused my (thick) mirror to take many hours to cool down, even with the fan on (and I made the mistake of keeping the shroud on…)

Among the highlights, I was able to see Uranus (easy) and M 33 (hard) naked eye, as well as catch the two brightest satellites of Uranus with my scope. Dave Cooper and Peter Natscher also shared some superb views in their scopes (respectively an AP 6″ apo refractor and a 24″ F/3.7 Starmaster dobsonian telescope)

I thoroughly enjoyed the conversations I had during the day with fellow observers. That’s why I’ve come to love star parties. Even if the weather is not so great, it’s always a fantastic opportunity to learn new things.

Below is my log for both nights. Most objects are from the Herschel 400 list. From now on, I will start mixing in Herschel II and Herschel I objects in my observing routine, in addition to some objects from Alvin Huey’s awesome free guides: “Selected Small Galaxy Groups” and “Selected Galaxy Trios” (which I had printed and coil binded at Fedex Kinko’s) Cheers!

Location: Lake San Antonio [Elevation 1082 ft]
Telescope: Meade Lightbridge 12″ F/5
Eyepieces used:
- Televue Panoptic 27mm (56x – 1.2° TFOV)
- Televue Nagler 16mm type 5 (95x – 52′ TFOV)
- Televue Nagler 9mm type 6 (169x – 29′ TFOV)
- Televue Nagler 7mm type 6 (217x – 22′ TFOV)
- Televue Nagler 5mm type 6 (305x – 16′ TFOV)
(All times are PDT)

Thursday, September 17, 2009

NGC 7142 OC Cep 21h45m25.3s +65°49′26″ 10.0 mag 08:40pm
Very inconspicuous group of about 40 moderately faint stars over a milky background. Slightly elongated WNW-ESE. A few brighter stars delimit this cluster on the north and east sides.

NGC 7331 GX Peg 22h37m33.1s +34°28′14″ 10.2 mag 09:00pm
Elongated 3×1 NNW-SSE. Fairly bright core region surrounding a bright stellar nucleus visible at 217x and 305x during brief moments of better seeing. The halo is rather dim compared to the core. It extends about 6′ along its major axis. The western half of the galaxy looks darker, due to the presence of a dark lane. Very faint superimposed star about 2′ SSE of nucleus. NGC 7335, NGC 7337 and NGC 7340 were easily detected. NGC 7336 was not detected. I did not search for nearby NGC 7325 and NGC 7326.

NGC 6210 (Turtle Nebula) PN Her 16h44m54.4s +23°47′05″ 9.7 mag 09:25pm
At 95x, this planetary nebula appears bright, almost stellar, and exhibits an obvious blue color. At 217x and 305x, the color is a bit washed out. At those magnifications, this planetary nebula appears round and very uniform, about 10 to 15″ in diameter, and seems to be surrounded by a slightly larger and rather faint envelope (unless this is scattered light). I did not detect a central star.

NGC 6572 (Blue Racquetball) PN Oph 18h12m35.8s +06°51′28″ 8.0 mag 09:40pm
At 95x, this planetary nebula appears bright, stellar, and exhibits an obvious blue color. At 217x and 305x, the color is a bit washed out. At those magnifications, this planetary nebula appears very uniform, roughly 10″ in diameter, and seems surrounded by a very faint envelope about 3 times the size of the nebula (unless this is scattered light). I did not detect a central star. There is a faint group of a dozen stars 10′ west.

NGC 7217 GX Peg 22h08m20.6s +31°24′39″ 11.1 mag 10:35pm
Bright, round non stellar core surrounded by a larger, relatively bright uniform halo. Faint superimposed star less than 1′ NNE of the core.

NGC 7814 GX Peg 00h03m47.1s +16°12′11″ 11.6 mag 10:50pm
Moderately bright and large. Appears roundish at first, progressively brighter toward its center. At higher magnification and with careful examination, a faint halo extending NW-SE can be detected. The bisecting dust lane obvious on photographs was not seen.

NGC 7448 GX Peg 23h00m34.6s +16°02′10″ 12.1 mag 11:00pm
Pretty small, moderately bright, elongated 2×1 NNW-SSE, faint nucleus seen intermittently at high magnification. Suspected the detached segment that makes this galaxy peculiar to be at the northern tip of the galaxy.

Mayall II (M31-G1) GC And 00h33m20.0s +39°38′04″ 13.7 mag 12:20am
Appears faint and stellar, thus requiring a very detailed chart to find! Forms a very tight triangle with two field stars. In order to confirm the observation, I sketched a few field stars along with the position of the presumed globular. The result is very close to the sketch published by Sue French in the December 2005 issue of Sky & Telescope’s Deep Sky Wonder column. I believe that excellent seeing and high magnification are essential to see the non-stellar nature of this target.

NGC 205 (M 110) GX And 00h40m56.5s +41°44′30″ 8.9 mag 12:30am
Fairly large and bright, elongated 2×1 NW-SE, gradually brighter to a stellar nucleus.

NGC 891 GX And 02h23m12.6s +42°23′30″ 10.9 mag 12:45am
Moderately faint, elongated roughly 6×1 NNE-SSW, about 12′ along its major axis. The dust lane splitting this galaxy in half is clearly visible only through the central bulge. Several fairly bright stars are superimposed. Peter Natscher’s 24″ scope showed the bisecting dust lane much more clearly, along with some mottling.

NGC 7293 (Helix) PN Aqr 22h30m12.6s -20°47′05″ 6.3 mag 12:55am
Large, fairly dim, best seen using a UHC filter at low power. The ring structure, very slightly elongated NW-SE, can easily be seen. The NW and SE portions of the rim are clearly dimmer. A few superimposed stars, including what appears to be the central star, can be seen inside the ring over a faint background.

NGC 7662 (Blue Snowball) PN And 23h26m24.4s +42°35′30″ 8.6 mag 01:30am
At 95x, this planetary nebula appears bright, stellar, and looks slightly bluish. At 217x and 305x, the center region of the nebula appears darker, and an elliptical ring oriented roughly E-W can be detected. What I failed to see is the lumpy texture of the ring, beautifully documented in the October 2009 issue of Astronomy Now’s Drawn To The Universe column.

Friday, September 18, 2009

NGC 6822 (Barnard’s Galaxy) GX Sgr 19h45m32.4s -14°46′43″ 9.4 mag 08:30pm
Large (roughly 15′ x 10′), elongated NNE-SSW, extremely pale, very easy to miss when sweeping the area, which is probably why this galaxy was missed by Herschel and was discovered so late (Barnard, 1881). There are a few superimposed stars throughout. A couple of small knots (probably HII regions) were seen using a UHC filter (although I did not note their position) This galaxy is part of the Local Group of galaxies.

NGC 7027 PN Cyg 21h07m25.2s +42°16′47″ 9.6 mag 08:50pm
Small, bright, exhibiting an obvious blue color, elongated 3:2 NW-SE, rectangular in shape, slightly wider on the NW side. A faint outer envelope, about twice the size of the nebula, was suspected (unless it could be scattered light)

NGC 6503 GX Dra 17h49m20.4s +70°08′46″ 10.8 mag 09:15pm
Fairly bright, moderately large, elongated 3×1 WNW-ESE, fairly uniform with only a slight brightening of the core region. Detected a very weak nucleus or superimposed star.

NGC 7023 BN Cep 21h01m45.3s +68°12′26″ mag 09:40pm
Fairly dim nebulosity, roughly 2′ in diameter, around V380 Cep (mag 7.4). Dim patch around 2 to 3′ south of V380 Cep (photographs confirm the presence of this extension). A UHC filter makes the nebula disappear, confirming its reflective nature.

NGC 7479 GX Peg 23h05m28.0s +12°22′45″ 11.7 mag 10:20pm
Moderately bright, fairly small, elongated 5:2 N-S, weak fairly large core. Moderately bright (mag 12.9) superimposed star at the northern end, slightly dimmer (mag 13.7) star about 1′ SW of the center. Suspected some mottling in the core. This galaxy is actually a barred spiral. The arms should be visible under better conditions and with a slightly larger scope.

NGC 7078 (M 15) GC Peg 21h30m28.0s +12°12′47″ 6.3 mag 10:25pm
Fairly large, very bright, very intense core, resolved throughout. Located about 7′ SW of a bright (mag 7.6) star.

NGC 7099 (M 30) GC Cap 21h40m57.3s -23°07′56″ 6.9 mag 10:35pm
Moderately large and bright, well defined core, well resolved throughout. The SE region looks darker. Quite a few brighter stars are visible in the halo.

NGC 404 (Mirach’s Ghost) GX And 01h10m01.9s +35°46′21″ 11.2 mag 11:00pm
Located about 7′ NW of Mirach (Beta And), a mag 2.0 star exhibiting a beautiful golden color. Moderately bright and small, round, gradually brighter to a fairly bright stellar nucleus best seen at high magnification.

NGC 628 (M 74) GX Psc 01h37m15.2s +15°50′11″ 9.7 mag 11:35pm
Fairly bright and small core embedded inside a large round faint halo showing hints of mottling at higher magnification. Very faint superimposed star within 1′ ESE of the center, slightly brighter one about 1.5′ ENE. Spiral structure better seen in Peter Natscher’s 24″ scope.

NGC 7606 GX Aqr 23h19m37.3s -08°25′42″ 11.7 mag 12:10am
Moderately large and bright, elongated 2×1 NNW-SSE, fairly bright stellar core, uniform halo.

NGC 7727 GX Aqr 23h40m26.2s -12°14′02″ 11.6 mag 12:20am
Fairly small and bright, small round intense core, round uniform halo. NGC 7723 and NGC 7724 nearby.

NGC 7723 GX Aqr 23h39m29.4s -12°54′15″ 11.9 mag 12:25am
Fairly small, moderately faint, moderately bright stellar nucleus surrounded by a fairly uniform halo elongated 3:2 NE-SW. Located about 20′ ENE of a bright yellow double star. NGC 7727 and NGC 7724 nearby.

NGC 253 (Sculptor Galaxy) GX Scl 00h48m04.1s -25°13′53″ 7.9 mag 12:45am
Very easily seen along globular cluster NGC 288 in 9×50 finder scope. Very large and bright, elongated 6×1 NE-SW. The core region is large, bright, and shows some obvious mottling (seen even better in Peter Natscher’s 24″ scope). Numerous fairly bright field stars are superimposed.

NGC 247 GX Cet 00h47m39.8s -20°42′11″ 9.7 mag 12:55am
Pretty large, moderately faint, elongated 6×1 N-S, very weak central condensation. Fairly bright (mag 9.5) star at the southern tip, moderately bright (mag 11.6) superimposed star about 7′ S.

NGC 288 GC Scl 00h53m15.8s -26°31′34″ 8.1 mag 01:00am
Very easily seen along galaxy NGC 253 in 9×50 finder scope. Fairly large and bright, fairly uniform, well resolved throughout.

NGC 613 GX Scl 01h34m47.8s -29°21′46″ 10.7 mag 01:10am
Moderately large and bright, elongated 3×1 NW-SE, fairly bright small round core. Hints of two spiral arms, one starting at the NW end, bending towards the W, and a symmetrical arm starting at the SE end, bending towards the E.

NGC 7793 GX Scl 23h58m22.3s -32°32′00″ 9.7 mag 01:15am
Moderately large, fairly faint, elongated 4:3 E-W, faint almost stellar core.

NGC 524 GX Psc 01h25m20.5s +09°35′35″ 11.4 mag 01:30am
Bright, fairly small. Intense core surrounding an almost stellar nucleus. Fairly bright round halo. Brightest member of the Shakhbazian 40 group.

NGC 1023 GX Per 02h41m02.8s +39°06′23″ 9.6 mag 01:35am
Moderately large, fairly bright, small bright round core, halo elongated 2×1 E-W. Two faint superimposed stars about 1′ W of the center, and one faint superimposed star about 1′ E. The companion NGC 1023A was not seen.

NGC 488 GX Psc 01h22m19.2s +05°18′41″ 11.1 mag 02:00am
Fairly small and bright, bright round core surrounded by a halo very slightly elongated N-S.

NGC 185 GX Cas 00h39m32.8s +48°23′36″ 10.2 mag 02:15am
This is a satellite of M31. Fairly large, pretty uniform, large faint core surrounded by a halo elongated 4:3 NNE-SSE. Located in a crowded region of the milky way. Could not detect the dust patch seen on photographs.

NGC 278 GX Cas 00h52m39.9s +47°36′20″ 11.5 mag 02:30am
Fairly small, pretty bright, fairly bright stellar nucleus, round uniform halo. Located about 3′ S of a bright (mag 8.8) field star.

A Better Implementation Of The Input Prompt Pattern

Mon, 14 Sep 2009 11:43:01 -0700

The Input Prompt pattern consists in prefilling a text field with a prompt as a way of supplying help information for controls whose purpose or format may not be immediately clear. In the browser, this pattern is most often implemented by dynamically modifying the value property of a text field element via the focus and blur event handlers attached to the text field, as shown in this example (live demo):

CSS:

.hint { color: #999;
}

Markup:

<input type="text" id="sbx">

JavaScript (based on YUI 3.0.0):

YUI().use('node', function (Y) { var sbx = Y.get('#sbx'); Y.on('domready', function () { sbx.set('value', 'Search'); sbx.addClass('hint'); Y.on('focus', function () { if (this.get('value') === 'Search') { this.set('value', ''); this.removeClass('hint'); } }, sbx); Y.on('blur', function () { if (this.get('value') === '') { this.set('value', 'Search'); this.addClass('hint'); } }, sbx); });
});

Note: the code is intentionally implemented inside a domready event handler to work around issues related to form field caching.

The main problem with implementing this pattern using the value property is that the default text is used if the form is submitted while the input prompt is showing. Trying to work around this by testing the content of the text field when the form is submitted makes the default text impossible to use as a value. Another side effect of this implementation is that most developers will forget to attach a element to the text field, leading to a confusing experience for screen reader users as they lack the necessary context to understand the purpose of the control.

A better implementation of this pattern consists in using a element and positioning it on top of the text field it is attached to. Here is an example of this implementation (live demo):

CSS:

#container { position: relative;
} #container label { position: absolute; top: 4px; *top: 6px; left: 3px; color: #999; cursor: text;
} #container label.offscreen { left: -9999px;
}

Markup:

<div id="container"> <label for="sbx" class="offscreen">Search</label> <input type="text" id="sbx">
</div>

JavaScript (based on YUI 3.0.0):

YUI().use('node', function (Y) { var sbx = Y.get('#sbx'), lbl = Y.get('#container label'); Y.on('domready', function () { sbx.set('value', ''); lbl.removeClass('offscreen'); Y.on('mousedown', function () { setTimeout(function () { sbx.focus(); }, 0); }, lbl); Y.on('focus', function () { lbl.addClass('offscreen'); }, sbx); Y.on('blur', function () { if (sbx.get('value') === '') { sbx.set('value', ''); lbl.removeClass('offscreen'); } }, sbx); });
});

As always, I am looking forward to reading your comments and answering your questions in the comments section of this blog.

OR: Henry Coe State Park, 8/15/09

Mon, 17 Aug 2009 11:07:17 -0700

A lot of people showed up for some stargazing at the overflow parking lot at Henry Coe state park. The conditions were pretty good overall:

Light winds coming from the north west
Temperature in the high to mid 60s
Good seeing
Average transparency

While waiting for the sky to turn dark, I pointed my scope toward Antares, which was showing a fairly clean split while still high in the southern sky. The dim companion, a white dwarf, was located just to the west of the red giant.

I then spent most of the evening tracking down some bright targets from the Herschel 400 list. Among the highlights, the Saturn nebula (NGC 7009) was showing a lot of details (see description below)

Before wrapping up around 1am, I looked at Jupiter. The view was absolutely incredible, although the wind proved to be a bit of a problem with my light weight scope. The great red spot was more colorful than I can ever remember. Its color used to be a very delicate pink mixed with a little bit of gray a few years ago. But this time, it was definitely orange. A lot of tiny details could be spotted throughout the bands, and even in the polar regions! What a view to behold!

Here are my observations. Overall, it was a good night, and I’m glad I went out. Cheers!

NGC 7128 OC Cyg 21h44m19.3s +53°45′39″ 11.3 mag 09:30p
About 15 moderately bright stars within an area roughly 4′ x 2′ elongated NE-SW. The brightest star, located NE of the cluster, shows a nice orange/reddish color.

NGC 7086 OC Cyg 21h30m49.2s +51°38′39″ 11.6 mag 09:40p
About 50 moderately bright stars within 10′.

NGC 7062 OC Cyg 21h23m50.4s +46°25′18″ 8.3 mag 09:45p
About two dozen moderately bright to moderately faint stars located within 5′. Slightly elongated E-W.

NGC 7006 GC Del 21h01m58.1s +16°13′44″ 10.6 mag 09:50p
Small, faint, small fairly bright core, fairly faint halo, unresolved.

NGC 7009 (Saturn Nebula) PN Aqr 21h04m44.6s -11°19′22″ 8.3 mag 10:30p
At low power (95x), this planetary nebula appears small, fairly bright, elongated 3:2 WNW-ESE, and exhibits a delicate blue/green color. At higher magnification (217x and 305x), the center of the nebula appears very slightly darker. Faint extensions show up on both sides of the oval disk, terminated by a tiny slightly brighter knot. The extension on the ESE side seems to slightly bend toward the NE. Finally, during brief moments of better seeing, a ring elongated 2:1 appears inside the nebula. The central star was not detected.

NGC 6940 OC Vul 20h34m52.3s +28°19′09″ 7.2 mag 10:45p
About 80 stars within an area 25′ x 10′ elongated WNW-ESE. Orange star in the center.

NGC 6885 OC Vul 20h12m27.3s +26°30′35″ 5.7 mag 10:50p
Sparse cluster of roughly 20 fairly bright stars within an area 20′ in diameter, centered around 20 Vul (mag 5.9) Another open cluster, NGC 6882, is embedded inside NGC 6885.

NGC 6882 OC Vul 20h12m13.2s +26°50′35″ 10:50p
About 20 moderately bright stars within an area roughly 10′ in diameter located about 10′ NW of 20 Vul (mag 5.9), and inside another open cluster, NGC 6885.

NGC 6939 OC Cep 20h31m44.1s +60°41′48″ 10.1 mag 11:10p
About 80 moderately bright stars within an area roughly 10′ in diameter. The southern boundary seems delimited by an almost perfect alignment of 5 brighter stars. Slight triangular shape, like an arrow moving west… Forms a superb couple with galaxy NGC 6946. Both objects can be seen within the 1.2 degree field of view of the Panoptic 27mm.

NGC 7160 OC Cep 21h53m59.4s +62°39′00″ 6.4 mag 11:40p
About 20 fairly bright stars scattered within an area roughly 5′ in diameter, centered around two very bright stars (mag 7.0 and 7.9) separated by about 1′.

NGC 7380 OC Cep 22h47m46.7s +58°11′00″ 8.8 mag 12:40a
About 30 moderately bright stars within 15′. Apparently, this cluster is embedded inside a nebula (Sh 2-142) but I did not notice anything, and since I was not aware of it, I did not think of trying a narrow-band filter.

NGC 7510 OC Cep 23h11m30.4s +60°37′22″ 9.3 mag 12:45a
About 20 moderately bright to moderately faint stars within an area 5′ x 2′ elongated WSW-ENE.

NGC 40 PN Cep 00h13m36.3s +72°34′28″ 10.7 mag 01:00a
Moderately large and bright, fairly bright central star, very slightly elongated NNE-SSW. Appears pretty uniform at first. With careful examination, the southern region of the nebula shows a slight darkening. The rim is fairly well defined, and slightly brighter than the disk itself. Superimposed star on the SSW rim.

OR: D.A.R.C. Observatory, 7/25/2009

Mon, 27 Jul 2009 16:11:29 -0700

I went to the D.A.R.C. observatory last Saturday to observe or re-observe a few Herschel 400 summer objects, as well as enjoy some eye candy under a dark sky. I was setup right next to Dennis Beckley and his classic Obsession 18″. Here is a summary of the weather conditions:

Wind died down shortly after sunset
Temperature in the low 70s
Good seeing (6 to 7 on the Pickering scale)
Average transparency (the light dome from Los Banos, which I use as an indication of the moisture content in the atmosphere, was fairly conspicuous)
NELM 6.7 in Lyra
Thin clouds started showing up around 1am from the east

Overall, it was a pretty good night, although not as good as what the D.A.R.C. observatory has proven capable of. Among the highlights of the night, I got to observe Triton for the first time. The Jupiter impact mark was seen very easily, and Jupiter’s satellites were resolved as tiny discs of different diameters. The “ink spot” (Barnard 86 located right next to OC NGC 6520) looked absolutely remarkable. Stephan’s quintet was looking pretty good in my scope, and was easy in Dennis’ 18″ scope. Finally, I really enjoyed looking at V Aql, a fairly bright carbon star of deep orange color with a tinge of red. What is interesting about this star, beyond its striking color, is the presence of planetary nebula NGC 6751 about 30′ SE. It is therefore possible to fit both objects within the same field of view in modest amateur telescopes (8 to 12″), portraying two stages of stellar evolution as the carbon star will eventually become a planetary nebula in a not so distant future.

On Sunday morning, I had an immense satisfaction when I looked at a DSS image of galaxy NGC 6217 (Arp 185), and noticed that my notes (see below) were absolutely right on! I find it absolutely amazing that such wispy details can be detected with such a small aperture under a good dark sky.

I cannot end this report without thanking Dr. Lee Hoglan for letting us observe on his property. Below is my log for the night. Cheers!

NGC 6217 GX UMi 16h32m15.6s +78°10′52″ 11.9 mag 10:15p
Small, faint, elongated 3:2 NW-SE, fairly faint stellar core or super imposed star. Very faint super imposed star about 20″ SE, another one about 2.5′ NW. Hints of two spiral arms, one on the N side going E, and another one on the S side going W. This observation was later beautifully confirmed with a DSS image.

NGC 6834 OC Cyg 19h52m37.2s +29°26′04″ 9.7 mag 10:30p
Seen as a tiny smudge in a 9×50 finder scope. Small grouping of about 30 moderately bright stars. One brighter star in the middle, with a possible slight yellow tinge. Located in a region with relatively few stars (dark nebula?) in an otherwise crowded milky way star field. For some reason, I found it hard to star-hop to…

NGC 6866 OC Cyg 20h04m16.1s +44°11′11″ 9.1 mag 10:40p
About 30 fairly bright stars within a region 15′ in diameter. Detected in 9×50 finder scope.

NGC 6910 OC Cyg 20h23m34.7s +40°48′37″ 7.3 mag 10:45p
Small group of about 15 stars, elongated 2×1 WNW-ESE (dimensions: 10′ x 5′) The two brightest stars have a yellow tinge. Located about 30′ N of Gamma Cyg (Sadr)

NGC 7044 OC Cyg 21h13m32.8s +42°32′06″ 12.0 mag 10:50p
Small compact group of maybe 50 faint stars (hard to get a reliable star count) scattered within a circular region of about 5′ in diameter.

NGC 6522 GC Sgr 18h04m14.3s -30°02′08″ 9.9 mag 11:00p
Fairly small, moderately faint, fairly well defined core, halo appears grainy. There is a moderately bright star located about 30″ E which probably does not belong to this cluster. Forms a beautiful couple with globular cluster NGC 6528 located about 15′ E.

NGC 6528 GC Sgr 18h05m28.3s -30°03′25″ 9.6 mag 11:00p
Fairly small, moderately faint, fairly well defined core, unresolved. Located at the NW edge of B298. Forms a beautiful couple with globular cluster NGC 6522 located about 15′ W. This cluster is slightly dimmer than its neighbor NGC 6522 (some of its light may be absorbed by B298?)

NGC 6818 PN Sgr 19h44m32.6s -14°07′45″ 10.0 mag 11:25p
Tiny fairly bright disk, almost circular (extremely slightly elongated NS) showing a delicate bluish tinge, especially at low power. At higher magnification, the center of the nebula appears very slightly darker. The central star (mag 15.0) was not seen. Located inside a small triangle of moderately bright stars.

NGC 6664 OC Sct 18h37m10.4s -07°48′17″ 8.5 mag 11:45p
About 60 moderately faint stars within 15′. This cluster appears slightly elongated N-S, and is located about 25′ NE of Alpha Sct, which has a fairly distinctive yellow tinge.

NGC 6712 GC Sct 18h53m37.6s -08°41′33″ 8.1 mag 11:50p
Moderately large and bright with a poorly defined core region. Many stars are resolved over a milky background.

NGC 6802 OC Vul 19h31m02.2s +20°16′59″ 11.7 mag 12:50a
Small (5′ x 2′) compact group of about 25 very faint stars, elongated N-S. Located just east of the famous coat hanger asterism (Cr 399)

NGC 6823 OC Vul 19h43m35.6s +23°19′27″ 7.1 mag 12:55a
About 30 fairly bright stars scattered within 6′ around a small group of 4 bright stars and a few fainter ones. Not very conspicuous in a crowded star field. Nearby nebula NGC 6820 was not seen (although I did not really look for it and thus was not using a nebula filter)

NGC 6830 OC Vul 19h51m25.8s +23°07′33″ 8.9 mag 01:00a
Very inconspicuous grouping of about 20 fairly bright stars.

NGC 6934 GC Del 20h34m41.3s +07°26′22″ 8.9 mag 02:00a
Relatively small, fairly bright, intense core. A great number of stars are resolved throughout at 217x.

Test Suites for CSS 2.1, ARIA, and HTML5

Tue, 27 Jan 2009 23:43:59 -0800

Just hours ago Microsoft released an amazing new resource that helps the entire frontend engineering industry. Their Windows Internet Explorer Testing Center contains thousands of test cases covering CSS 2.1, HTML5, and WAI-ARIA.

CSS gets the most coverage with 7005 tests, 3784 of them developed just since IE8’s “beta 2″ a few months ago. IE8 passes all 7005, including, mysteriously, 52 tests that do not pass on any other major browser.

For HTML5, coverage includes 13 cross-document messaging and 30 DOM Storage tests. For WAI – ARIA they submitted new samples to support their previously-submitted ARIA to MSAA roles, events, and mappings.

While a great resource for the standardization movement in general, it also goes a long way to support their stated belief that “IE8 RC1 has the most complete implementation of the CSS 2.1 specification in the industry.” It will be very interesting to see if any of the other browsers care to comment. I’m hoping for a four-way tie.

The Eyeballing Game

Thu, 16 Oct 2008 11:55:15 -0700

Here’s an enjoyable way to spend ten minutes giving your brain some exercise: The Eyeballing Game. The game/exercise asks you to modify a polygon to create a parallelogram and right angle, find the midpoint of a line, bisect an angle, find the center or a triangle and circle, and identify a convergence point.

My average score, the degree on inaccuracy, (as you can see below) was 4.01 (low is better). By best showing was bisecting an angle.

Think you can do better? Give it a shot: http://woodgears.ca/eyeball/

Walking for Farm Animals

Sat, 11 Oct 2008 10:55:53 -0700

This weekend, Aimee and I are taking part in the Walk for Farm Animals 2008 to raise money for Farm Sanctuary, a national non-profit that works to end cruelty to farm and food animals through rescue, education, and advocacy. It’s a great organization that, to me, is way more palatable than more confrontational organizations such as PETA.

We hope you’ll support us by donating $5-10 (or more) via our FirstGiving page (a secure way to send directly to Farm Sanctuary, and they’ll mail you a tax-deductible receipt).

We visited Farm Sanctuary’s California farm in Orland last weekend (they have one on NY, too). We stayed in the farm’s guest cabin and were able to spend a lot of time with all the animals (Aimee discovered I’m something of a turkey whisperer). It was fun to see them in action and we had a chance to volunteer a little by preparing food and feeding many of the animals and brushing the goats.

Aimee posted a set of photos (ad 2 vids) from our time at Farm Sanctuary on Flickr.

Also, if you vote in California, please join us in supporting Prop 2 (sponsored by Farm Sanctuary) with a “Yes” vote in November.

Prop 2 is a modest measure that would allow farm and food animals the ability to stand up, stretch and turn around. Through the reduction of these inhumane caging/crating practices (most commonly used by factory farms) will improve the health and safety of our food, support family farmers, and reduce the environmental degradation caused by these unnecessary practices. The NY Times has endorsed Prop 2 in a thoughtful and straightforward article.

We hope you enjoy our pictures and hope you’ll consider helping us raise money for this important organization.

Thanks and love,
nate & aimee

DJ Z-Trip Mixtape for Obama

Fri, 10 Oct 2008 14:49:33 -0700

DJ Z-Trip (with designer Shepard Fairey) has thrown some fundraisers for the Obama campaign called the “The Party for Change.” A few days ago he made the 54 minute set available as a free mp3 download.

You can read more about it and grab it on his site, or save him a bit of bandwidth and grab it from me (.zip, 85mb). (It’s public domain and he encourages its wide distribution!)

Here’s a great DJ with a great ear and premier turntable skills that pulls music from across genres. This mix is no exception.

I agree with his signoff:

I honestly feel if we make our voices heard, this time WILL be different.

Finances: Avoid Rash Moves

Fri, 10 Oct 2008 10:27:08 -0700

The New York Times has a story up today called Switching to Cash May Feel Safe, but Risks Remain. I highly recommend that you read it.

I’m not a particularly savvy financial person, but “think long term” always struck me as wise. There’s no doubt that it’s pretty hairy out there right now, but I think changing course or being overly emotion is the worst reaction. It’s tempting to be emotional and run for cover, but this article cites research that should give you strength and resolve.

As the title says, switching to cash may fell safe. But:

But if you sell now, you’ll be locking in your losses. And once you’re in cash, there isn’t much upside. In fact, with interest rates low, you’re likely to lose money in cash, because inflation will probably eat up the after-tax returns you earn from a savings or money-market account.

Beyond the low upside of cash, selling low is, obviously, the worst time to sell. It’s bad because you lock in your losses, but the read damage is that you’ll miss the eventual upswing, to devastating effect.

The research they cite is pretty amazing (emphasis mine):

From 1963 to 2004, the index of American stocks he tested gained 10.84 percent annually in a geometric average, which avoided overstating the true performance. For people who missed the 90 biggest-gaining days in that period, however, the annual return fell to just 3.2 percent. Less than 1 percent of the trading days accounted for 96 percent of the market gains.

…

A portfolio belonging to an investor who missed the 10 best days over several decades across all of those markets would end up, on average, with about half the balance of someone who sat tight throughout.

They note that if you need the money in the next five years — if you’re about to retire — then perhaps the loss of the move to cash might make sense. But for me, and for many of you, “today’s price is not your price. Your price is 10 or 20 years from now.”

So, hang on to what you’ve got and don’t make any rash moves.

Accessibility Movers – Henny Swan to Opera from RNIB

Tue, 09 Sep 2008 05:44:50 -0700

I just noticed that ~~Henry~~ Henny “iheni” Swan — Senior Web Accessibility Consultant at Royal National Institute of the Blind for the past six years — is taking a job at Opera Software as a Web Evangelist. In addition to wishing ~~him~~ her well, ~~his~~ her post, Hello Opera provides a few quick tidbits about the Opera roster these days.

Congrats, iheni! I hope to see at a conference soon.

Update: I wrote this post in the middle of the night after working entirely too long and late. In my delirium, I misread Henny as Henry and used incorrect pronouns. Further, in the headline I mistakenly typed Swar instead of Swan (though I got it right in the body and in the permalink).

Sincere apologies for my clumsiness. Thanks to Henny for setting me straight!

Wireframing with Balsamiq Mockups

Tue, 09 Sep 2008 03:17:38 -0700

Thanks to Pras for the pointer to Balsamiq’s Mockups application. I was sketching wireframes quickly within minutes of finding the product.

I believe in low-fidelity sketching at the wireframe stage. Balsamiq makes it easy with its large library of UI control stencils, its auto-complete driven keybroad stencil selection, on-screen snap-to alignment guides, a powerful inspector for precise control when rarely needed, and, more of all, a simplicity that makes it easy to start sketching or tweaking your mockup immediately.

The output is Balsamiq files, PGN or flattened image files, and XML. Because it exports XML it’s possible to use Balsamiq as a programmatic ingredient for downstream engineering systems and tools (such as partially automating the creation of detailed functional specifications, or using it as source for the automated building on the actual interface.

There is a rumor that they’ll be announcing clickable output files shortly, which might allow for the fast creation of clickable wireframes for usability testing (and other) needs.

I haven’t noticed, but it should be possible to customize what’s in the included UI Widget Library to a) take on a different visual skin; b) reflect new or fewer interface widget options.

All and all, I’m pretty intrigued. It seems there’s a market for consumer-friendly ways to design interfaces. Once more people catch on how to much fun we’re having, they’ll want a shot at designing and realizing all the apps they’re dreaming up, too!

I’d love to hear what you think of this approach. Have you tried it? Does it work for your teams”

Large Hadron Collider (LHC) Rap

Tue, 09 Sep 2008 02:35:36 -0700

Here’s a pointer to a scientifically accurate rap about the Large Hadron Collider (LHC).

Story on www.telegraph.co.uk – YouTube permalink.

From the story:

Now a larky but accurate rap song explaining the point of the 17 mile circumference machine [under Switzerland and France], which formally starts up on September 10, has made a star of Kate McAlpine, 23, aka “alpinekat”, who stars with her friends in a YouTube video that has been downloaded more than [was 400,000 times when the article was published, and nearly 1.4mm now.]

I liked watching the video. I’ve heard about the LHC before, but this 4 minutes taught me new facts about both the collide *and* its science. And there’s funny dancing.

Songbird Public Beta (0.7)

Thu, 21 Aug 2008 12:08:06 -0700

Congrats to my buddies (yo Koshi!) over at Songbird for reaching another big milestone: public beta.

Songbird is a media player like iTunes. Except that it’s build on top of the awesome Mozilla Firefox foundation. And like Firefox, it has an extensive array of extensions, themes, and assorted addons. Earlier versions haven’t supplanted iTunes for me, but it’s looking like this version may well do that.

I had some trouble imagining what type of addons would make sense, but in this release we’re beginning to see. An early favorite for me is the ticketing integration:

You can read all about the release on their blog, download it here, and see a screenshot below:

Twitter Faster than Reality

Tue, 29 Jul 2008 18:56:02 -0700

LA shook at 11:42:15 today according to the official record from the U.S. Geological Survey. But according to [a report of] Twitter activity today (by the tweetip site) it happened 43 seconds earlier at 11:41:32 (adjusted for time zone).

(graphic snagged from tweetip site)

That Twitter routinely breaks news fastest is often discussed, notably in the wake of the May quake in China.

Today the AP’s wire posted news of the earthquake 9 minutes after it happened. 9 minutes is fast. Negative :43 is amazing.

(Yeah, yeah. I know. It’s explainable as an accounting error in twitter’s api or tweetip’s processing. But the point remains that twitter is always on the scene.)

Links for 2009-11-27 [del.icio.us]

Sat, 28 Nov 2009 00:00:00 -0800

"Streams of Content, Limited Attention: The Flow of Information through Social Media"
For the longest time, we have focused on sites of information as a destination, of accessing information as a process, of producing information as a task. What happens when all of this changes?

Links for 2009-11-26 [del.icio.us]

Fri, 27 Nov 2009 00:00:00 -0800

On-demand Media: Publishing, aggregation and “The Innovator’s Dilemma”
examine the market data, study the demographics, lifestyles, challenges and habits of potential readers and then bring together editorial, advertising, technology and marketing to create digital products that serve users and advertisers and do so profitably.

Socially linked data

Thu, 26 Nov 2009 02:46:31 -0800

The semantic web folks, including Sir Tim Berners-Lee, have been saying for years that the Internet could become significantly more compelling by cooking more intelligence into the way things link around the network. The movement is getting some legs to it these days, but the solution doesn’t look quite like what the visionaries expected [...]

Positioning real-time web platforms

Thu, 26 Nov 2009 02:48:50 -0800

Like many people, I’ve been thinking a lot about the live nature of the web more and more recently. The startup world has gone mad for it. And though I think Microsoft’s Chief Software Architect Ray Ozzie played down the depth of Microsoft’s commitment to it in his recent interview with Steve Gillmor, [...]

Links for 2009-11-25 [del.icio.us]

Thu, 26 Nov 2009 00:00:00 -0800

apophenia: spectacle at Web2.0 Expo... from my perspective
The problem with a public-facing Twitter stream in events like this is that it FORCES the audience to pay attention the backchannel. So even audience members who want to focus on the content get distracted.
The New Journalist in the Age of Social Media
An audience will watch you fall on a sword. A community will fall on a sword for you.
9 Ways to Visualize Proportions – A Guide | FlowingData
With all the visualization options out there, it can be hard to figure out what graph or chart suits your data best. This is a guide to make your decision easier for one particular type of data: proportions.

Links for 2009-11-24 [del.icio.us]

Wed, 25 Nov 2009 00:00:00 -0800

Vogue Turns To Obama’s Online Fundraiser For Revenue Drive | paidContent
A big part of what Vogue wants to learn is whether there are enough people out there willing to pay for the magazine’s online content. But executives think they’ll learn more than that for its revenue drive, such as how to market its e-commerce and subscription marketing initiatives.

Links for 2009-11-19 [del.icio.us]

Fri, 20 Nov 2009 00:00:00 -0800

Dash to D.C.! Tech Guru Will Head Gov't Incubator, Digitize Democracy | The New York Observer
“The current administration is comprised in great part of digital natives. The government is already using technology to talk to citizens, but we’re going to make technology that helps government listen to them.”

Links for 2009-11-17 [del.icio.us]

Wed, 18 Nov 2009 00:00:00 -0800

APIs In The Late Afternoon
the best way to monetize apis is to send money with the api. Paul Forster, CEO of Indeed: "We tried charging for our API without much success. Then we paid developers to use it and it took off."

Links for 2009-11-16 [del.icio.us]

Tue, 17 Nov 2009 00:00:00 -0800

Advertising in Reverse
Any of us should be able to broadcast, in a secure and selective way that protects our privacies, specified goods we’re shopping for.

The thinking behind the Activate Summit event

Thu, 04 Jun 2009 02:32:26 -0700

The premise that the Internet is changing everything is only more potent now than it was when many people first considered that it might be true. Today we’re seeing how its capabilities have found their way into the hands of those who are actively changing the world. But the key questions haven’t yet been played [...]

The cost of transparency and accountability

Tue, 12 May 2009 03:50:43 -0700

The MP Expenses issue is a very interesting story. There are lots of reasons why it is spreading so aggressively… in hard times people look for someone to blame for what’s wrong in the world people are learning to expect more transparency from their government the availability of the data compells the curious to dive into all [...]

Response to the Open Platform launch

Tue, 24 Mar 2009 06:47:50 -0700

The Open Platform launch earlier this month was one of the more exciting days I’ve had in a long time. We’ve done a good thing at the Guardian, and it seems we’re not alone in thinking that. Here are some of my favorite Twitter posts about the launch (more here): @IanYorston “Guardian Open Platform may be [...]

A few interesting data projects

Mon, 02 Feb 2009 05:39:56 -0800

The contagious data bug must be sweeping through the office, as several very different but very interesting data-driven publishing projects rolled out almost simultaneously. First, infographics editor Paddy Allen explains the global recession through a very elegant interactive piece “Where did all the money go?“. Paddy has quite a collection of brilliant work from his [...]

Building communities from Twitter posts

Sat, 03 Jan 2009 12:54:38 -0800

I spent a little time over the last couple of weeks playing around with some Twitter data. I was noticing how several people, myself included, were sharing the funny things their kids say sometimes: “it is noisy outside. my daughter said, ‘the texans are shooting their guns!’ no, dear. just fireworks“ “Asked my son to hold [...]

Breaking through the attention barrier

Sun, 14 Dec 2008 13:58:11 -0800

For some reason I get a bit annoyed when people write about our information overload limits. This happened the other day when I saw Seth Godin’s piece titled “Warning: The internet is almost full.” He writes: “The decentralized nature of the net means that it will never be physically full. As long as we [...]

Hacking BNP data

Fri, 21 Nov 2008 15:42:47 -0800

Less than a week after trying out some new data mapping concepts at Guardian Hack Day, a big pile of data appeared on the Internet begging to be mapped. Using some of their new skills and a convenient constituency data tool, a small team of innovators got to work to produce some really interesting [...]

Notes from Hack Day at The Guardian

Tue, 18 Nov 2008 10:59:05 -0800

We hosted our first Hack Day last week at The Guardian. Amazing fun. Here’s a 15min highlight reel: We did a lot of the standard stuff that makes Hack Day so interesting, but there were a few innovations to the event format itself that I thought worked really well, too: DabbleDB. Simon Willison setup [...]

Python-YQL: client library for YQL

Tue, 24 Nov 2009 13:52:18 -0800

You’d of have to have been under a rock to not have heard about Yahoo Query Language which provides a “SQL-like” abstraction to Yahoo’s own APIs as well as a growing number of third party APIs.

I’ll be honest; for a while I was quite dismissive of YQL thinking that if I was using those APIs then I’d query them directly rather than use a 3rd party abstraction. But then I saw a recent post from Christian Heilmann that touched on making YQL calls from the server that began to pique my interest. It made me start to think about all the data from the web that could be accessed from just some simple SQL-like queries.

So I decided in order to try out YQL I’d build a dedicated client library. This also provided me with an opportunity to explore OAuth properly as I’d need to make the client work with both Two and ThreeLegged Oauth so it’s also relatively straightforward to query Private tables from a web application.

The result is Python-YQL which you can install from PyPi with:

sudo easy_install yql

The documentation is available at http://python-yql.org/ and branches are available at Launchpad.net or you can get a branch with the following:

bzr branch lp:python-yql

Basic usage is along the following lines:

>>> import yql
>>> y = yql.Public()
>>> query = 'select * from flickr.photos.search where text="panda" limit 3';
>>> y.execute(query)

For details on more advanced usage see the docs at python-yql.org

Patches Welcome!

So go and check it out and kick the proverbial tyres. It’s early days and there’s bound to be some bugs and missing features but this is where you come in. I’ll happily expedite provision of fixes as they come in. Patches/Bug reports/branches are welcome!

Python Oauth2

For oauth integration, I started out using python-oauth until Cyril mentioned that Joe Stump had created a new oauth library based on various forks of the original python-oauth. Python-oauth2 provides an easy to use well-tested library that is a natural progression of python-oauth.

Once I knew about it I couldn’t resist switching python-yql over to using it. At the moment I’m bundling python-oauth2 as I needed to patch in support to to make it work with python versions < 2.6. However hopefully in due course this will cease to be necessary.

Future

There’s lots I can improve upon but for now I’m planning to spend some time using the library in order to work out what can be better. I’m thinking one of the first things I’d like to do is make what’s returned an object rather than just simply returning the data. This will mean that I can provide useful attributes such as the query that was called to return that dataset for example. This is now a reality.

The version in the trunk branch now returns a YQL object which provides a much more awesome API for accessing the data returned by YQL queries. This will be released as 0.3 and at that point I’ll update the docs. Until then here’s a taster:

>>> import yql
>>> y = yql.Public()
>>> res = y.execute("use 'http://yqlblog.net/samples/search.imageweb.xml' as searchimageweb; select title from searchimageweb where query='pizza' limit 3")
>>> print res

>>> res.rows
[{u'title': u'Pizza Hut'}, {u'title': u"Domino's Pizza -
Official Site"}, {u'title': u"Papa John's"}]
>>> res.rows[0]
{u'title': u'Pizza Hut'}
>>> res.rows[1]
{u'title': u"Domino's Pizza - Official Site"}
>>> res.query
u"use 'http://yqlblog.net/samples/search.imageweb.xml' as searchimageweb; select title from searchimageweb where query='pizza' limit 3"
>>> res.uri
u'http://query.yahooapis.com/v1/yql?q=use+%27http%3A%2F%2Fyqlblog.net%2Fsamples%2Fsearch.imageweb.xml%27+as+searchimageweb%3B+select+title+from+searchimageweb+where+query%3D%27pizza%27+limit+3'
>>> res.count
3

Spotify: Linking to a specific time in a track

Wed, 11 Nov 2009 03:56:10 -0800

For a while I’ve wanted to be able to share a link to a track on spotify so that it jumps the right place. An example is when learning covers for the band I play in; or just to point out a great part of the track.

The good new is that this is already possible! I asked a question on Get Satisfaction and a Spotify employee Emil Hesslow answered straight away that this feature already exists. Here’s some examples:

Check out George Benson’s guitar break: Uncle Albert/Admiral Halsey 4:04
A Nice hammond organ break from Greg Rolie: Toussaint L’Overture 0:46

You can also use the spotify protocol:

e.g: spotify:track:39Bi2scq80BWdgnxz2llWT#04:04

The format for the link is just a simple fragment identifier added to the end of the link in the format #mins:secs e.g: #4:04

From using this (I’m running Spotify under wine) I’ve found you need to URLencode this in the “http://open.spotify…” links for it to work so #4:04 becomes %2304%3A04.

Web Application Security – LugRadio Live 2009

Sun, 01 Nov 2009 03:04:32 -0800

These are my slides from the presentation I gave at LugRadio Live 2009 at Wolverhampton. The presentation was a brief tour of some common security issues you might come across developing web applications. I also covered ReDOS which is a lot less well known but an interesting vulnerability.

The notes are available on slideshare.net

View more presentations from Stuart Colville.

~~I’ve had to pull the presentation from slideshare.net temporarily – I’ll re-upload as soon as possible~~ The problem at slideshare has now been resolved.

Ubuntu: Lock Screen and Pause Spotify

Thu, 22 Oct 2009 09:23:00 -0700

When I used a mac I had a shonky little applescript to pause iTunes when I locked the screen.

These days I mostly listen to Spotify (running under wine) on Ubuntu which of course doesn’t have applescript. What I wanted was a similar script so I can hit a keystroke and pause spotify and lock the screen at the same time.

With a few utilities I was able to programatically tell if Spotify is playing and get focus on the Spotify window and send a keystroke to get spotify to pause. But the solution it has to be said is far from perfect.

The need to know if Spotify is actively playing is as a result of there only being a play/pause toggle activated by spacebar. Without testing this locking the screen with Spotify already paused and open would result in it starting when you lock the screen.

You’ll need the following packages installed to get it to work:

sudo apt-get install wmctrl xvkbd

wmctrl is a window manager CLI tool and xvkbd is a vitual keyboard that also has a CLI interface to send keystrokes.

Here’s the script:

#!/usr/bin/env bash # By Stuart Colville
# http://muffinresearch.co.uk/archives/2009/10/22/ubuntu-lock-screen-and-pause-spotify/
# sudo apt-get install wmctrl xvkbd # Check the title in the window list to work out if it's playing
TITLE=$(wmctrl -l | grep -o -e "Spotify.*$") # If it's playing pause it
if [[ "$TITLE" != Spotify ]]; then wmctrl -a "Spotify" && xvkbd -q -delay 100 -text '\ '
fi # Lock the screen
gnome-screensaver-command --lock exit 0

Save it somewhere and make it executable e.g. chmod +x ~/bin/lock_screen.sh.

If you want it to work on a keycombo then just add an item to System -> Preference -> Keyboard Shortcuts

Known Issue: A problem I’ve spotted is that locking the screen causes Spotify to start playing if it’s open and paused. So I need to add a way to detect if Spotify is currently playing without which this script is pretty much useless. ~~Fixed with libinotify-tools~~

The EPIC Hack here is using libinotify to detect if Spotify is playing by watching file accesses to the Storage directory. Can’t help but feeling there’s got to be a better way. It also means that a delay of 5 secs is needed to wait for an event. – There was a better way! This is now done by interogating the window list via wmctrl

Updates and suggestions are welcomed.

Open Redirects and Phishing Vectors

Wed, 30 Sep 2009 15:43:52 -0700

There was an interesting article on the Google Webmaster Central blog back in Jan talking about open redirects being abused by spammers.

One point they didn’t go into too much detail on is that of phishing vectors. If you’re running a site with any kind of user registration and you have a redirect script that allows redirects to any arbitrary urls. Then it’s fairly likely that you’ve straight away made it possible for a 3rd party to phish your registration form.

Let’s see an example of how something like this would work, first a standard redirection script:

Target Site: buyeverythingawesome.com
Attack URL: http://buyeverythingawesome.com/redirect?url=http://buyeverythingawesomee.com/login/
Attacker’s Bogus Domain: buyeverythingawesomee.com

The target site has a script which blindly redirects to anything passed to it.

All the attacker has to do is send a victim an email telling them to login to their account to view the latest offers. The link they send is http://buyeverythingawesome.com/redirect?url=http://buyeverythingawesomee.com/login/

On the bogus domain the attacker will have prepared a copy of the site’s login screen. All they need to do is get the user to login and then redirect the user back to the original site and if possible directly to the failed login page of the original site.

The way to mitigate this is to only allow redirects to your internal site or provide a whitelist of external urls that are allowed to be redirected to.

Open Redirect Login Script

Open Redirects in a login script are even worse in a way as the attacker will send the user to the real site’s login script.

The symptoms of this kind of problem are visible with a setup like so:

Target Site: buyeverythingawesome.com
Attack URL: http://someawesomsite.com/login?after=http//:somewawesomsite.com/loginfailed/
Attacker’s Bogus Domain: somewawesomsite.com

All the attacker needs to do is to get the unwitting victim to visit http://someawesomsite.com/login?after=http://somewawesomsite.com/loginfailed/

The user will be presented with the login screen for the genuine site. Once they login they are redirected to a bogus site with a “Your login has failed message – please try again” simply copied from the real site. The second time the user logs in they are logging-in to a bogus copy of the real site. Once the credentials are stolen the user can be redirected back to the real site where they are successfully logged in (This is because they aren’t redirected until they sucessfully login).

This is particularly nasty because the start and end points of this hack are the genuine site and as a result this kind of attack is much more likely to slip by unnoticed.

As a picture speaks a thousand words see this diagram of the flow: Open Redirect Phishing Vector (.png)

Preventing these kind of attacks

Firstly make absolutely sure you only allow redirects to internal links or whitelisted sites. If you need to redirect to arbitrary sites an interstitial page might be necessary to make it absolutely clear that the user is being redirected to a third party.

You might think that your site is already checking that links redirected are only internal ones. However it’s crucial that you take care to check more than just validating that the url starts with a slash.

Check you’ve covered all eventualities

It’s possible to write a link as //foo.com and the scheme (http/https etc) will be inherited from the base URI . So if you’re on http://baz.com and there’s a link with an href attribute of //google.com this will resolve to http://google.com. It’s therefore important that any validation of redirect URLs covers this case or you could be caught out.

Note: If you’re interested this aspect of how URIs work is covered by examples given in rfc 3986

As an example http://has-open-redirect.com/login/?done=//google.com would redirect to http://google.com after login if vulnerable.

Why is any of this important?

Right now you’re probably thinking that there’s nothing behind the login to your site that a hacker would want.

However, have you stopped to consider that your users might also use the same username and password to login to their amazon account for example?

Lot’s of people will only have one username and password for every site they use (this infosecurity magazine article claims 46% of all UK adults us the same password). If it’s possible to phish accounts on a low profile site – the chances are there will come a point when the attacker will hit the jackpot and get access to every site that user frequents with same username and password.

Update: The GMail team has a nice post detailing some good tips for choosing a good password. Tip number one is don’t use the same password for all sites

Distraxion

Wed, 16 Sep 2009 05:01:28 -0700

Brilliant animation – and nails exactly how I feel about Smooth Jazz. via BoingBoing

Cross Platform Keyring Library for Python

Wed, 26 Aug 2009 03:18:46 -0700

Kang Zhang has been busy on a Google Summer of Code project to create a cross platform keychain library for Python.

The Python keyring lib provides a easy way to access the system keyring service from python. It can be used in any application that needs safe password storage. It supports OSX, KDE, Gnome and Windows’s native password storing services. Besides this, it is shipped with kinds of Python implemented keyring for the left environments.

It’s also been written in a way that makes it possible to create your own Keyring backend if you want to.

As this provides a native Python interface to the OSX keychain I’d certainly recommend looking at this over and above my own noddy OSX keychain wrapper.

Good stuff!

See the site for more info: http://home.python-keyring.org

PHP: Multiple DNS Queries Using fopen

Wed, 12 Aug 2009 11:03:16 -0700

Whilst working on some inherited PHP code that used fopen I noticed an interesting comment in the PHP manual which pointed out that fopen always makes a DNS lookup for every request. Taking the following code as an example:

Using wireshark for capturing and calling that script 3 times I got 3 DNS lookups because fopen doesn’t make use of any DNS lookup caches:

Not only is this a problem for fopen but I also found the same problem with file_get_contents too.

The comment in the manual suggests using gethostbyname which uses the DNS cache. You can then use this to provide the ip address in the arguments to fopen. However as soon as you’re trying to fetch something which uses name-based virtual hosts this approach will fail. This is due to there being several sites on the same server all being served on the same ip address; if you contact the server by ip address it will simply serve you content from the default virtual host which is the conf which happens to be first alphabetically.

A Solution

The cURL library (php5-curl is the package you’ll need on Ubuntu) is the preferred way of fetching content with PHP, mainly because it gives you far greater control over requests.

The other big benefit of using cURL is that it makes use of the DNS cache so we can save a DNS lookup for repetitive calls to the same hostname:

Running that script three times now results in only one DNS request (I manually cleared the DNS cache with sudo /etc/init.d/networking restart first)

Conclusion

If you’re using php to fetch data from the web cURL is a much more powerful solution than relying on fopen or file_get_contents. Not only that if you’re fetching a lot of data from the same hosts frequently your scripts will run faster as a result of only making the minimum DNS requests.

VirtualBox: Error: Cannot register the hard disk

Thu, 23 Jul 2009 03:27:08 -0700

If when cloning VirtualBox VMs you get an error like this:

$ VBoxManage clonehd WinXP-IE7.vdi WinXP-IE8.vdi
VirtualBox Command Line Management Interface Version 2.2.4
(C) 2005-2009 Sun Microsystems, Inc.
All rights reserved. ERROR: Cannot register the hard disk '/home/scol/.VirtualBox/VDI/WinXP-IE7.vdi' with UUID {3858b1b1-c306-4505-8264-235af812f337} because a hard disk '/home/scol/.VirtualBox/VDI/WinXP-IE7.vdi' with UUID {3858b1b1-c306-4505-8264-235af812f337} already exists in the media registry ('/home/scol/.VirtualBox/VirtualBox.xml')
Details: code NS_ERROR_INVALID_ARG (0x80070057), component VirtualBox, interface IVirtualBox, callee nsISupports
Context: "OpenHardDisk(Bstr(szFilenameAbs), AccessMode_ReadWrite, srcDisk.asOutParam())" at line 603 of file VBoxManageDisk.cpp

Talk about a completely ambiguous error message! The solution is to provide an absolute path to the source VDI and target VDIs (leaving the path off the target will put the image in ~/.VirtualBox/HardDisks by default):

$ VBoxManage clonehd $(pwd)/WinXP-IE7.vdi $(pwd)/WinXP-IE8.vdi
VirtualBox Command Line Management Interface Version 2.2.4
(C) 2005-2009 Sun Microsystems, Inc.
All rights reserved. 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Clone hard disk created in format 'VDI'. UUID: b7166421-9743-4808-aedf-3c90aeb7c902

Project Fondue Blog

Tue, 14 Jul 2009 04:16:16 -0700

Last weekend I was in Paris with Cyril and Ed spending some time on Project Fondue Projects. One of the things we got done was putting out the initial iteration of the Project Fondue Blog. This is based on Pocoo’s Zine which we’ve really enjoyed using. The only downside so far is that it’s dependencies are quite cutting edge so if you are sticking to older Debian based distros (in our case Hardy LTS) it’s a case of pulling together the necessary packages from Debian Repos.

I’d certainly recommend subscribing to the rss feed – as there will be some interesting posts coming up.

The difference between a web developer and an engineer

Sun, 29 Nov 2009 20:09:10 -0800

An engineer builds platforms. A web developer creates websites using those platforms.

Symfony 1.3 Web Application Development

Sun, 29 Nov 2009 20:09:10 -0800

Title: Symfony 1.3 Web Application Development (on Amazon.co.uk)
Authors: Tim Bowler, Wojciech Bancer
Publisher: Packt Press
Publish Date: September 2009

Disclaimer: I received a review copy of this book from Packt Publishing in September 2009.

Symfony is a PHP5 framework for rapidly building dynamic web applications. This book is targeted at Symfony 1.3 when it hits production-ready status. At the time of writing the 1.3 version is still in alpha. Unfortunately this book suffers from the uncertainly as final decisions haven't been made. The major one is whether the default ORM will be Propel or Doctrine. The book flip-flops on which one is the default ORM. On the whole the book feels like a rushed job, trying to get to market before Symfony 1.3 hits production ready.

The Symfony project themselves have useful introductions to using Symfony, most particularly their Jobeet tutorial which covers Symfony through the practical development of a Job listing site. This new book by Packt attempts the same approach, building a website for a chain of milkshake restaurants. It is a decent example that covers many of the features of Symfony, and so comparing it to the Jobeet tutorial is a useful exercise.

The intended audience for this book are developers using older versions of Symfony as well as developers new to Symfony. I'm fairly new to Symfony and I've been developing with it for almost a year now.

Overview of the book

The first chapter of Symfony 1.3 Web Application Development covers getting and setting up Symfony. The next chapter sets out the skeleton of our application by defining the database schema, routing, actions and templates. This is followed by a chapter on adding business logic to to model and adding navigation. A chapter on building a simple form to email module takes us through Symfony Forms, and then takes us through converting it into a reusable plugin.

Then we create the administration interface through chapter 5, customising the admin screens as well as locking parts down through authentication. Chapter 6 covers adding Javascript to the admin interface by adding an image uploading widget, a date picker and rich text editing.

Chapter 7 shows off some of Symfony's internationalisation and localisation features. There follows a chapter on adding 3rd party libraries to symfony, and a useful chapter on various caching layers in Symfony. The final chapter talks about deploying your application to a production environment.

Building the skeleton of the application

The chapter on setting up Symfony and creating the skeleton of the application competes directly with the Jobeet tutorial, and unfortunately comes off second best. The approach the book takes to installing Symfony is to grab the current version from the Symfony SVN repository and then create a sandbox using that code. This is different to the preferred Symfony 1.2 method of doing an SVN checkout to the lib/vendor directory of each project.

What I found bizarre was the authors' preference for using the Propel XML schema format for defining the database instead of Symfony's consistent YAML methods. The authors state that they find YAML more difficult to read and with spaces instead of tabs this causes problems with large YAML files such as a database schema. I find this puzzling since all of Symfony's configuration is done with YAML, so it's impossible to avoid - the authors later have to teach the reader how to do YAML anyway, so introducing the XML way is just not efficient.

And introducing the XML vocabulary they don't. There's two pages of database schema in XML format, and they explain just 4 attributes of the entire document. No mention is made of the structure, or the purpose of the elements, let alone the various field types that are allowed. The whole XML way of defining a database schema looks left as an exercise for the reader.

The HTML in the code examples left me horrified. Essentially the page layout template is a set of divs with inline style attributes. Including the 5 item navigation list, each li having the same three lines of inline style properties. I'm astonished that in the 21st century we have such a poor understanding of CSS from people teaching development on the Web. Jobeet beats the pants off this section of the book by embracing the customisability and flexibility of CSS by creating a competition for skinning their application - that's perfect web standards evangelism. This Packt Press book however, shows a lack of modern web development techniques that have been prevalent over at least the last four years.

The attention to detail is lacking. There's no mention of how to configure the web server to serve content from the web directory. Considering that the book heavily uses the custom domain milkshake, leaving out the web server configuration is a disservice to the reader, since they can't really follow along with the book. Again, this seems to be another exercise for the reader with no clear guidance given of what needs doing. A passing mention of aliasing the Symfony assets directory is the closest the book comes to the web server configuration. Jobeet covers this particularly well in day one of their tutorial.

Dealing with business logic

The chapters covering adding in more complicated business logic are better, they feel that they hold together better than the previous chapters. It takes the reader through creating new modules and adding in custom logic, as well as getting hold of request-related information. The routing sections are decently explained. Particularly the explanation of pagination and partials were well written.

The next chapter on creating a newsletter signup script, so taking a form, adding the received data to a database and firing off an email is a good chapter, the plugin section being extremely useful. It introduces the Symfony Forms feature as we build the module from a database schema and generate the form classes before customising them in both the structure of the form and how it is rendered on screen.

As an accessibility practitioner, I'm noticing that there's no mention of explicit HTML labels and grouping by fieldsets, and the error handling on form validation leaves little to be desired. Still, it sounds like that these renderers can be rewritten to produce accessible markup.

Security is covered by the vaguest wave to a CSRF token (Cross Site Request Forgery), and that this hidden input field code snippet helps prevent this. Honestly, that is the coverage of security in its entirety. I understand that this book isn't about teaching security to web developers. The Jobeet tutorial is slightly better in this regard, but not by much.

There's no detail on where or how to use third party libraries involved in email delivery. The code example assumes that the library is already installed at the correct location and ready to be used.

The section of creating a plugin is perhaps the most useful section, taking the reader through the steps of creating a reusable plugin for modules. Perfect for isolating bits of code to use in other projects. The higher level process is easy to understand, but the actual implementation is under a cloud because of the setup concerns of the first chapter. I hope the actual step-by-step process is correct, and that really means this book needs to be properly tech-edited and proof-read by a hands on developer.

Configuring the Administration area

I struggled getting the admin area working through the Jobeet tutorial, so I'm happy to say that the process of customising the administration configuration makes a lot more sense after reading this chapter. Particularly, I found the Jobeet process of locking down access to the adminstration side of the application made even the login page requiring authentication before being accessed (possibly I needed to clear the cache?). This book makes a more determined effort to explain how the authentication system works and how it is configured, so that fills me with more confidence.

The chapter on enhancing forms with Ajax came as a surprise as they added the functionality to the adminstration area instead of a simple form outside. That shows the confidence of the author to pull this off. I hope it works.

Internationalisation

It's another useful chapter, this time on internationalising the application. The chapter takes the reader through extending the database schema to handle translations of each piece of content, and the changes needed to address the model to reflect the selected locale (the authors use the term 'culture' for locale).

For localising the templates and interface text the book steps through using XLIFF (XML Localisation Interchange File Format) which takes a text string as a key and returns a translation that's stores in the XML translation files. There are automated tools to extract existing text from templates to populate the initial files. And then filtering all static text through a special filter function produces the correct translations.

However, after being through an internationalisation hell for the latter half of 2008, I realise the book isn't anywhere near complete in it's internationalisation coverage. There's no mention of handling static text that contains request-variable information, like This is page 6 of 12, nor any mention of Symfony's ordinal and plural grouping of numbers which impressed me in its ability to handle Polish. Nor was there any mention of currency, number or date formatting.

Granted that translating plain text is a major step forward for being able to localise sites, but it's one step in a series of steps. (This is despite the code examples containing mojibake, presumably because the printing process of the book is at fault here)

Extending Symfony, Performance and Deployment

This chapter covers using other PHP5 libraries inside of Symfony. The demonstration of using ez Component's Pie Chart component inside a Symfony application impressed me, and again demonstrates the confidence of the authors. Unfortunately the followup about integrating Zend framework components is two paragraphs and feels completely unfinished.

The performance chapter covers the various cache levels of Symfony. Right from configuring gzip encoding within symfony, to template cache and data caches, and even when to avoid using the ORM when raw performance is necessary. The step by step guide is very useful, as we watch the load times of the page shrink drastically.

The chapter on deployment covers how to disable a Symfony application, enable outage messages, and then recommends using rsync to migrate code from the development environment to production. It's a very quick overview of rsync.

Conclusion

There is a fair amount of useful information in this book. I will be using it's plugins and administration chapters again fairly soon. But this book suffers badly, it really needs a solid technical review and a very sharp editor review. It needs a proper developer test - take a developer who has no knowledge of Symfony and get them to follow along with the book.

One serious issue was the Finishing off the location page section of Chapter 3 is supposed to cover adding in vacancies on the page, which is assumed to be there in Chapter 5's Handling foreign keys using admin generator.

My gripe against the use of XML will be alleviated by either explaining the XML syntax of the database schema, or switch back to YAML. (But it would also help having a typesetting review of the book, the indenting of the code examples - including the YAML code snippets - aren't correct or consistent.)

To be honest, I would wait for the second edition of this book; and only if it's strictly tech-edited, sternly proofread and typeset reviewed, after the first release of Symfony 1.3. It can be quite a good companion book to the Symfony framework, it's just not ready yet. When it is, it will be a useful guide to Symfony 1.3

The Bletchley Park experience

Sun, 29 Nov 2009 20:09:07 -0800

On the 18th July 2009 a rag-tag army of geeks invaded Bletchley Park. Thanks to the genius of Neil Crosby's unorganising Big Geek Day Out. However, the geeks were spotted well in advance by the Bletchley Park organisation, and soundly routed and disarmed and subject to a wonderful show of hospitality and an abundant knowledge.

Britain's best-kept secret

Bletchley Park is one of Britain's best-kept secrets of World War II, it was instrumental in shortening the duration of the War, being positively involved in every major conflict. It was a collection of talent and genius; staffed with mathematicians, engineers, linguists, chess players and demon crossword puzzle solvers. They quietly carried out their duty, careful never to break the seal of secrecy. And even twenty-five years after the end of World War II, the public still had no idea what happened at Bletchley Park. When the official secrecy expired and those records were made public for the first time in the mid 1970s we started to see the marvelous and ingenious work successfully executed by the collaboration of brains at Bletchley Park.

I think Bletchley Park exemplifies the quiet British determination and grit, its quirky inventiveness and eccentricities even at the most desperate of times. It's also a fascinating story of modesty, and so many of the people of Bletchley Park never received the recognition they so richly deserved - even today.

In war we glorify the conquerors - Eisenhower, Montgomery, Rommel, Doernitz - the people who made their names on the field of battle. We worship the political leaders Roosevelt and Churchill. Sometimes we remember the operatives working well behind enemy lines always in danger of being caught every moment of their lives. But for a few geeks sitting in wooden huts safely ensconced hours north of London, they are so easy to overlook, and yet their contribution to the defence of Europe is absolutely massive.

Bletchley Park was the center of Britain's intelligence war against the Axis powers. It principally was devoted to cracking and translating messages between the enemy military units, but it also ran counter-intelligence and special operations.

The main story of Bletchley Park is the story of Enigma and Bombe, and the codebreakers. Enigma, the machine that encrypted German military communications, and Bombe, the British machine that sought to break that encryption.

Cracking Enigma with Social Engineering

The breaking of Enigma wasn't all brute force. The codebreakers had a number of cribs - or techniques to reduce the possibilities - and many of them, as is thematic even in modern day codebreaking are down to social engineering. But in Bletchley Park's case it was reverse social engineering. Understanding habits of Enigma operators or message senders, and exploit it to reduce the possibilities.

One particular brilliant piece of social engineering was finding duplicate messages. If you have the entire encrypted message and you can with reasonably certainty guess the start of the message then we immediately narrow the possibilities of setup because we know how the first few letters are translated. That crib is then handwired into Bombe, and it spouts out possible starting settings that result in the matched combinations of letters.

There were a number of sources of these messages, the first was weather messages sent to U-Boats in the Atlantic, the German navy used standard weather reporting. So if the codebreakers could guess the weather pattern they could be reasonably certain of the decrypted message, so had a menu for the Bombe to process.

Why crack a message you already know? Well, the German military made one gigantic mistake - mainly out of confidence that Enigma was unbreakable. The Enigma start settings were the same for everyone in the same branch of the military, so breaking that one known message meant that they had the Enigma settings for all messages sent by the same branch of the military of that day.

The second piece of social engineering was a small German army unit in the middle of the North African desert. They sent virtually the identical message everyday reporting through their chain of command. Every day the message was "Nothing to report". That's enough of a menu for Bombe to brute force it's way to a solution. Now, what was utterly brilliant was Bletchley Park's request that the British Army in North Africa not send any of their units into the area this German unit was based. And so, the codebreakers virtually guaranteed that everyday they had the same message arriving ready for them to break the current day's settings.

Another source of repetitive messages were commands issues from near the top of the military hierarchy. They often started with the formal "Special Orders/Instructions" (out of habit essentially). So frequently the codebreakers guessed certain messages would start with the German word "Besonder".

I was amazed that social engineering was a critical part of breaking Enigma. Kevin Mitnick infamously demonstrated the power of social engineering in cracking computer systems, but the Bletchley Park codebreakers had already proved how effective it can be.

Chess and Bletchley Park

The 1939 Chess Olympiad in Buenos Aires was interrupted by the outbreak of World War 2. On the English team were Stuart Milner-Barry (later knighted), Conel Hugh O'Donel Alexander (or just Hugh Alexander) and Harry Golombek. All three left immediately and made their way back to Britain. Milner-Barry was the first to be recruited into Bletchley Park, and he in turn recruited both Alexander and Golombek.

So three of the best chess players in England were part of the Bletchley Park intelligence war. Each contributed substantially in their own way.

Milner-Barry headed up Hut 6 under Gordon Welchman, was later the person chosen to hand-deliver that petition to Winston Churchill begging for funding to keep the Bletchley Park operation going, which in turn Churchill became a staunch supporter of Bletchley Park's efforts. (Sir Stuart reprised that role by hand delivering a letter to John Major pleading for the preservation of Bletchley Park in 1992).

Hugh Alexander became the head of Hut 8 (the Enigma code-breaking effort), taking over from Alan Turing, becoming an instrumental part of the code breaking efforts. In Building B they recognise 4 individuals in particular for their efforts during the War. Turing was one of them, so was Gordon Welchman and Alexander was another.

After the war Alexander was recruited into GCHQ, and that basically killed off his chess ambitions. In a radio-match he beat the Russian champion Mikhail Botvinnik (who was regarded as the strongest player in the post-war world, and two years later becoming World Chess Champion). Unfortunately Alexander's commitments to the British intelligence community mean that playing international tournaments in the Soviet Block was far too risky to national security. So the British chess community were deprived of a potential World Championship candidate and potential grandmaster. His chess contributions were limited to local tournaments and writing chess books - including a memorable book about the legendary Boris Spassky - Bobby Fischer World Championship Match.

Alexander showed so much promise in the pre-war years, establishing himself in International tournaments like Nottingham 1936 where he placed respectfully despite having to face 5 World Champions. That promise and intellect was refocused into intelligence work through his experiences in Bletchley Park, so what was robbed from the chess world played a part in keeping Britain safe during the War and the subsequent cold war.

Harry Golombek continued being a chess correspondent, and a well recognised one. His book covering the 1948 World Championship Match Tournament is one of many splendid example of chess journalism.

Something to be proud of

I'm immensely proud of all the people who worked in and with Bletchley Park during the War. Every single one of them. I am still aggrieved however that Britain, or Churchill in particular, gave up its leading role in the computer revolution. I understand the political circumstances of the imminent threat of Communist Europe and Cold War threat provoked Churchill into hiding all evidence of the superiority of British Intelligence Services; but each time I see Silicon Valley herald itself as being the dog's bollocks, I wistfully remember that we handed that advantage over to them on a silver plate. And that is the nature of the British Spirit and stoicism.

I loved our big geek day out in Bletchley Park. I learnt far more than I imagined. Geekwise I'm satiated. But I know I want to go back and learn more. Thank you to the Bletchley Park staff for making us feel very welcome. Thank you for fighting hard to keep the story of Bletchley Park alive - the story of the British geeks and their phenomenal contribution to protecting our freedoms.

Bletchley Park - no movie can do it justice. Not even a blog post can convey just how instrumental they were. And I didn't even mention the working Colossus, the counter-intelligence operations, the pigeons, the D-Day involvement, the unheeded warnings, the Lorenz cipher, the Harrier Jumpjet, the amazing private tour we had, Alan Turing, the Computer History museum, the Churchill museum, the cipher museum... so much.

Installing CouchDB on Jeos 8.04

Sun, 29 Nov 2009 20:09:07 -0800

JEOS (Just Enough Operating System) is a wonderful baseline for creating Linux-based virtual machines for web development. Bradley Wright has written up the best set of instructions for creating a new virtual machine for web development running on JEOS and that is the starting point for building a CouchDB server.

With a bare-bones server up and running it's time to install CouchDB. CouchDB is available as an apt install on Ubuntu 8.10 and above, but it won't be the cutting edge code and so some features may be missing. So I had no choice but to go for the tar.gz install. But first, we can install the package dependencies:

sudo apt-get install automake autoconf libtool subversion-tools help2man
sudo apt-get install build-essential erlang libicu38 libicu-dev
sudo apt-get install libreadline5-dev checkinstall libmozjs-dev wget
sudo apt-get install libcurl4-gnutls-dev

Next we need to get Erlang 5.6 or above. This is another case where newer releases of Ubuntu have this version, but not Ubuntu 8.04. But this time the Apache folks on the CouchDB project have a nifty solution of using the Ubuntu Intrepid repository to grab the updated version of Erlang. (Chris Strom's blog post Yak Shaving is the new Dependency Hell help deal with a number of dependency issues thrown up by configure)

That's all the dependencies sorted. Now onto downloading the most up-to-date CouchDB source code and building it (this follows the approach documented by Craig R Webster) :

wget http://mirror.public-internet.co.uk/ftp/apache/couchdb/0.9.1/apache-couchdb-0.9.1.tar.gz
tar -xzvf apache-couchdb-0.9.1.tar.gz
cd apache-couchdb-0.9.1
./configure
make && sudo make install

At this point we have a working installed version of CouchDB. Craig continues with some additional work, and even gets couchdb installed as a service that can be started and stopped using the bog-standard Ubuntu init.d methods:

sudo adduser couchdb
sudo mkdir -p /usr/local/var/lib/couchdb
sudo chown -R couchdb /usr/local/var/lib/couchdb
sudo mkdir -p /usr/local/var/log/couchdb
sudo chown -R couchdb /usr/local/var/log/couchdb
sudo mkdir -p /usr/local/var/run
sudo chown -R couchdb /usr/local/var/run
sudo cp /usr/local/etc/init.d/couchdb /etc/init.d/
sudo update-rc.d couchdb defaults
sudo /etc/init.d/couchdb start

At this point CouchDB is running as a daemon. And we can test it via HTTP by calling the root URL on localhost:

wget http://localhost:5984/

Wget saves the contents of this url into index.html, and this contains:

{"couchdb":"Welcome","version":"0.9.1"}

And we are done!

MacSpeech Dictate: First Impressions

Wed, 18 Nov 2009 16:00:00 -0800

I suspect that the first thing any developer does when they get their hands on MacSpeech Dictate is to begin writing a review of MacSpeech Dictate in MacSpeech Dictate. I am no exception.

Over the last three months I've develop more pain than usual in my wrists and elbows. It’s not been painful enough to stop me from typing altogether, but certainly enough to make me stop and think about the amount of time I spend typing at the computer.

My dad has been playing with Dragon Naturally Speaking on and off for quite some time now, so speech recognition software has been on my radar for years. In fact, I suspect he personally financed the software’s research and development over the first few years of its life… Of course the software was initially more or less worthless, its output was simply miserable through the first few versions. But my dad continued to upgrade. Year after year he picked up a new version of Dragon, installed it, spent hours training at, and eventually began to actually get things done using the software. Now it’s my turn.

So, first impressions (beyond the fact that it’s simply strange to talk instead of type):

I'm a little surprised at the delay between my speech and its recognition. It shouldn’t be surprising when considering how much work is involved in this sort of thing, but it is. I speak, and about a second and a half later text appears on the screen. As long as I'm not actually paying attention to the screen, this doesn’t bother me at all. At the moment, however, I'm paying quite a bit of attention to the screen, so it’s a little distracting.

The text is generally correct, so that delay is easy enough to ignore. There is a slider in the preferences that offers a balance between speed and accuracy. I’ll play with that later, as I'm interested in the effect it has on the text produced.
I'm very surprised at the out-of-the-box accuracy. Dictate is doing a much better job than I expected it to, especially given my lack of experience with speech recognition software, and the seemingly insubstantial amount of text used to train the program. I read to it for about 10 minutes total and based on that small amount of training, I'm getting something like 90% accuracy. Reading back over the text that Dictate has produced, I certainly see some issues: it’s not perfect by any means. I’ll need to go back and edit a few portions, but overall I'm impressed.
I need to get used to the command structure. Specifically, I need to train myself to pause before and after commands. Dictate will delete the last phrase when I say “scratch that,” and delete the last word when I say “scratch word.” If I speak too fluidly, Dictate doesn’t recognize “scratch that” as a command. When I make a mistake I have to remember to pause a moment before correcting myself. That’ll take some time to get used to.
Dictate forces me to pay much more attention to the words I choose then I usually do. I normally write very fluidly, editing myself as I go. I’ll write half a sentence, jump back to the paragraph before to make a change, delete the sentence I'd started, rewrite the sentence, and start again. That process simply doesn’t work with speech recognition software. It’s much more difficult to jump around in the document, so it’s much more important for me to address a topic more linearly. I have to think before I speak, which is new to me.

Overall, I like this. I think it’s something I can get used to. It’s certainly better on my hands and wrists, and that’s something I very much appreciate at the moment. So far, it’s looking like a worthwhile purchase. Let’s see how I feel about a week or two.

A note on process: I dictated this document via Dictate in a bit more than half an hour, stopping relatively often to figure out exactly what I was doing. I then copied the text out of Dictate’s notepad, pasted into a real editor, and cleaned up the text. I'm not thrilled with workflow, but it’s not bad. Much less typing, which I'm certainly happy about.

My Jekyll Fork

Wed, 11 Nov 2009 16:00:00 -0800

Jekyll is an interesting project, a well-architected throwback to a time before unnecessary dynamism reigned supreme. In contrast to blog engines like Wordpress or Textile, Jekyll doesn’t attempt to do anything other than push raw content through a few simple filters out into the world in the form of static HTML files. Jekyll’s publication philosophy is very much in line with my own, and I appreciate the work that’s gone into it. It’s relatively widely used, and therefore much more stable and well-tested than anything I'd write on my own. Given my recent experience, I want something that will Just Work™, and this looks like it. I finished moving this site to Jekyll yesterday, and I'm quite happy with how it’s working…

It doesn’t fit me perfectly, though. Here, I’ll point to a few features that I think are missing, and a few design decisions that I think are worth reconsideration. Happily, it’s an open source project, so I’ll also be able to point to my fork of the project where I'm busy addressing these shortcomings.

“Generated” Pages: Tags and Archives

The biggest gap I see in Jekyll’s feature set is support for “generated” pages (covered in Issue #16). HTML that Jekyll produces is tied one-to-one with files you create on the filesystem. This has the appeal of simplicity, fails in a number of ways to support two quasi-dynamic things that I consider essential to the kind of sites Jekyll aims to produce: tags, and archives.

Tags are a nice way of grouping content on a site, and surfacing that content to readers in an unobtrusive way. Jekyll, out of the box, does a miserable job of making them available in templates. site.tags gives you a list of all the site’s tags, page.tags gives you the tags for the current page, and that’s it. That’s simply not enough structured data to do anything useful with; I want more. “More”, in my case, meaning two things: a separate page for each tag at /tags/[TAG], listing each article that fits; and a page listing out all the tags on the site (in cloud form, if only because I'm so Web 2.0). The latter is (painfully) possible out of the box, the former is not.

My solution (based heavily on Matt Flores‘ fork) is available in the tag_index branch of my Jekyll fork. The implementation is very low-impact: simply add a tag_detail.html layout to your site’s _layouts directory. Jekyll will auto-generate pages using that layout for each tag on your site, providing page.tag as a variable inside each as they’re rendered. This allows you to dive into site.tags to pull out lists of articles in a very straightforward way. Once rendered into HTML, the result is placed into a directory you specify via a configuration variable (tag_root). This has worked brilliantly for me here on this site.

Archive pages listing out content written during a certain period are another nice way of dividing up posts on a site. Especially useful for sites with more than a few posts, it’s a mechanism for showing users posts that fit together temporally. It’s nice to be able to see all of 2007’s posts, for instance. Or all posts from November of 2008.

Again, I borrowed a bit of code from Matt Flores, brought it up to date with the latest Jekyll tag (0.5.4 at the time I'm writing this), and check it into the archive branch of my fork. Similarly to the tagging system above, archives depend on adding a few extra layouts. archive_yearly.html, archive_monthly.html, and archive_daily.html are supported, and offer page.year, page.month, and page.day, which can be used to reference posts in site.collated_posts. My archive_monthly.html is indicative of how this can work.

Generated pages are written to /[YEAR]/index.html, /[YEAR]/[MONTH]/index.html, and /[YEAR]/[MONTH]/[DAY]/index.html if posts exist over the specified time period.

Filters

Jekyll uses the Liquid templating engine, which isn’t exactly my first choice. It’s a solid engine, as far as it goes, but it’s no Jinja2. Regardless, Jekyll has built in a number of useful filters that can be used to perform operations on text before it’s rendered. textilize() is a good example of this, running text through a Textile parser, then writing the output instead of the original text. It’s great!

Except, of course, for the fact that Textile is hideous. :) I much prefer to write Markdown formatted text (it’s just easier for me to read, really), so I was a bit miffed when I discovered that a Markdown counterpart to textilize() was simply missing.

A more robust system is being discussed (slowly) in Issue #19. I decided not to wait for a perfect solution, and simply added markdownize() in the filters branch of my fork. A trivial, but very necessary change.

Default Configuration Values

I really like the way that Jekyll expects posts to be formatted. Each post lives in it’s own file, and each file begins with a YAML block specifying metadata such as titles, teasers, and layout style. This allows you to configure each post separately, and lends quite a bit of flexibility to the end product.

As Issue #25 points out, it'd be nice if layout in particular could be specified at the site level as a default value. Posts that need different layouts are (generally) few and far between, and a global configuration would make the most common case a bit simpler.

Henrik took a stab at a solution to the problem, which I ran off with and improved upon in the post_defaults branch of my fork. I'm waiting on someone to take a look at this work now, but I'm not holding my breath for it to be merged into the official release.

Problematic Design

Beyond gaps in the feature set, Jekyll does one or two things that I simply disagree with.

Jekyll tightly couples content and layout by assuming that both will exist together in a defined directory structure. Leaving a bit of complexity to the side, a typical Jekyll site contains a _posts subdirectory filled to the brim with lovely raw content, and a _layouts directory filled with Liquid-based HTML templates. The former is exclusively concerned with content, the latter exclusively with layout.

For the same reasons that we eventually started building websites without inline style information, separating the concerns of the site’s semantics from it’s layout and behavior, I don’t believe that these bits belong in the same repository. At a minimum, I'd like to be able to deploy a version of my website’s look and feel without worrying about whether or not I tagged the release before or after adding a post. The one activity has nothing to do with the other, and both ought be able to proceed in parallel. Jekyll’s current implementation encourages mixing the two, which I don’t appreciate. Instead, I prefer to run two distinct repositories: one containing pure content, the other containing site-specific layout and configuration. This feels cleaner to me.

The solution I've hacked together is available in the contentpath branch of my Jekyll fork. I've added a single configuration variable (content_root) that contains an absolute path to the directory containing the site’s content. That directory will be parsed in it’s entirety (e.g. no _posts subdirectory is required). If a _posts directory exists in the usual location ([SITE_ROOT]/_posts/) it will be parsed as well to ensure backwards compatibility.

I don’t expect this change to make it into the main tree, as it’s probably not interesting for Jekyll’s main audience of GitHub Pages users who do in fact very much want to deal with a single repository. Moreover, when dealing with potentially malicious users, it’s not a brilliant idea to give them the ability to generate publicly accessible pages from any readable directory on a system. For my use, however, it’s more or less perfect, and I’ll do my best to keep it rebased on top of the latest Jekyll tags for anyone else who’s of the same mind.

Fallow fields, revisited

Sat, 03 Oct 2009 17:00:00 -0700

I'm currently in the process of gutting my website, and rebuilding it piece by piece. I suspect I'm doing this to distract myself from the fact that I don’t seem to have anything interesting floating around in my head to write about. “Surely it’s the site’s fault; raze it to the ground!”, the large, simple, and shouty part of my brain tells me. So I build anew (this is possibly ironic, but I'm ignoring that).

Happily, the small, quiet, and generally reasonable portion of my brain agrees with the plan, at least insofar as it’s clear that the current system (fallow) was a solid idea but poorly implemented. The system works, and I'm happy I wrote it. It was a good introduction to Ruby and Git, and a good reason to migrate off the almost-as-inefficient-as-wordpress Textpattern. But it’s failing me in a number of ways, the most important being that I literally forgot how to get content onto the site, and it took me 45 minutes of reading through painfully structured Ruby code to figure it out again. That’s the sort of thing that happens when you don’t touch a website for 6 months.

Rather that catalog the failings of the system I'm replacing (for they are legion), I'd like to touch on the carefully considered bits I'm keeping:

URL structure: Posts live at /[year]/[month]/[URLified Title] which seems more or less perfect to me. It’s meaningful, while containing just enough temporal context to make completely outdated information easy to spot. Moreover, it provides a natural /[year]/ and /[year]/[month]/ for yearly and monthly archive pages. Tag pages live under /tags/[tag], which makes sense, and ad hoc pages have ad hoc URLs (/is/, for instance). This strikes me as a clean setup, one which I can’t see any way to improve upon.
Content storage: The site’s content consists entirely of UTF-8 encoded text files on disk. Text files are simple to work with, and have a more or less infinite shelf life. A site like this one simply doesn’t need a database, a single flat text file per piece of content is good enough. Metadata (title, tags, etc.) is contained in a YAML block at the top of each file. It’s a format that is clear, human readable, and easily parsed, and I'm especially pleased to see that the format I'd decided upon for Fallow matches up quite well against more widely used systems like Jekyll.
Static HTML: Dynamic content doesn’t really exist on this site. I write an article, then post it online. That’s the extent of the processing that particular page needs. The server shouldn’t be working to rebuild an article from last week (or last year!) every time it’s requested, that’s simply wasteful. This site, therefore, generates a page once when it’s created, or when a template changes, and then simply serves that cached copy over and over again. Similarly, overview pages (like tag pages, or current archives) are regenerated when a new article is published, then served straight from disk. On a small VPS, I can serve upwards of 300 static requests per second through Nginx with extremely low load. Textpattern would fall over and die at those absurd traffic levels.
Historical redirects: The (miserable) /blog/id/[ID] URL structure I decided upon in 2005 still works for the content I've kept from that period. The (also bad) /archives/[Title]/ structure from 2007-8 works too. The (not so lovely) Tumblr-generated links for content that used to be at blog.mikewest.org will redirect nicely. All these old URLs will continue to generate nice, clean permanent redirects for the foreseeable future: why make the reader jump through hoops created by my lack of foresight?

So, those are the good bits I'd like to keep going as I rebuild. With the understanding that I'm about to make one of those dangerous “forward looking statements” that I never seem to follow through on as cleanly as I'd like, I expect mikewest.org to be running a new Jekyll-based backend sometime in October. With luck, no one will notice a thing but me. With even more luck, I’ll squeeze out a post or two about the bits of Jekyll I'm adjusting, and the places where it’s falling down completely.

Productivity Or My Lack Thereof

Tue, 22 Sep 2009 17:00:00 -0700

I just spent the last half-hour screwing around with my Vim configuration to set up a “more perfect” writing environment. Fullscreen! Eighty columns! Proper line-wrapping! Large, readable font! Markdown syntax highlighting! Spellcheck! Etcetera, etcetera! Hilariously wrongheaded, of course, since I created this environment instead of writing anything. Likewise, I've played around with todo.sh quite a bit in the last week or two (it’s nice), trying to put together a good system for recording my ever-growing list of things I need to take care of. Perhaps there’s a reason that the list is ever-growing.

Building new bash scripts that make recording tasks easier might feel like Doing Important Work, but it isn’t. Not really. This is unfortunately true even when “write new bash scripts to make recording tasks easier” is on my newly created todo list.

This is a long way of saying that Marco Arment is dead on:

The best way to increase your productivity, hack your life, and be minimalist is to stop reading those sites.

Also, hello Internet. It’s been a while.

Mnot's Redbot

Fri, 22 May 2009 17:00:00 -0700

Mark Nottingham has put together a really useful tool that aids in the analysis of the behavior of HTTP resources. Visit http://redbot.org/, type in a web address, and Redbot will report the resource’s cachability based on the returned headers, and provide a helpful list of recommendations for improvement.

Running it on http://mikewest.org/, for instance, flags the fact that I compress the response if the browser tells me that it can accept compressed content, but that I've neglected to send the proper Vary header to let caches know that the response needs to be negotiated, and cached based on the Accept-Encoding request header. This is useful, especially for actual applications for which it might matter.

So useful, in fact, that I'm forking it to submit some patches. Mark’s put the Redbot code up on GitHub (mnot/redbot), and I've started putting together a command line version based on the web version he’s built.

I've mentioned that I love GitHub, right? This is exactly why. :)

Playing with Placemaker

Wed, 20 May 2009 17:00:00 -0700

Yahoo’s latest API is really quite cool: Placemaker takes your unstructured data (e.g. any HTML page, RSS feed, etc), and extracts a nice list of locations that your data refers to. It’s a brilliant tool, and I can think of quite a few ways I'd like to use it in the future. Along with their release of a ton of WhereOnEarth ID codes that allows you to make use of Yahoo’s various geo-services, this is a really good day to play with geocoding unstructured data.

So, let’s play:

Accessing Placemaker is simple: assuming that you've somehow managed to obtain an application id, you simply make an HTTP POST request to Yahoo!’s Placemaker endpoint with a tiny bit of data specifying the nature of the data you’re dealing with, and it’s URL. If you like curl on the command line, this might look like:

curl -d 'inputLanguage=en-US&documentType=text/html&documentURL=http://mikewest.org/&appid=[APPID]' http://wherein.yahooapis.com/v1/document

You’ll get back an XML document (RSS is also available as a response format). Digging into the contents yields:

Boilerplate:

<?xml version="1.0" encoding="utf-8"?>
<contentlocation xmlns:yahoo="http://www.yahooapis.com/v1/base.rng" xmlns="http://wherein.yahooapis.com/v1/schema" xml:lang="de-DE"> <processingTime>0.380778</processingTime> <version> build 090508</version> <documentLength>15906</documentLength>

A (list of) document element(s) containing the extracted locations:
```
 <document>
```

A best guess at the document’s “scope” (the smallest region that “best describes” the document):

 <administrativeScope> <woeId>20071093</woeId> <type>County</type> <name><![CDATA[Munich, Bavaria, DE]]></name> <centroid> <latitude>48.1549</latitude> <longitude>11.5417</longitude> </centroid> </administrativeScope> <geographicScope> <woeId>29388625</woeId> <type>MMA</type> <name><![CDATA[MMA München, Bavaria, DE]]></name> <centroid> <latitude>48.1549</latitude> <longitude>11.5417</longitude> </centroid> </geographicScope>

Latitude/longitude for a bounding box that contains the places mentioned in the document (which makes it trivial to draw an “area of discussion” on a map):

 <extents> <center> <latitude>48.1364</latitude> <longitude>11.5775</longitude> </center> <southWest> <latitude>48.0417</latitude> <longitude>11.3771</longitude> </southWest> <northEast> <latitude>48.2292</latitude> <longitude>11.749</longitude> </northEast> </extents>

Detailed breakdown of the places mentioned in your document, giving you lat/long coordinates identifying the place’s center point. Moreover, it tells you where in your document the place was found (string offsets and XPath expressions, to each their own). This is helpful for those occasions when the text you've entered doesn’t exactly match the place’s name that Yahoo returns (“Munich” vs. “Munich, Bavaria, DE”, in this example) Annotating your documents with this data should be a piece of cake.

 <placeDetails> <place> <woeId>676757</woeId> <type>Town</type> <name><![CDATA[Munich, Bavaria, DE]]></name> <centroid> <latitude>48.1364</latitude> <longitude>11.5775</longitude> </centroid> </place> <matchType>0</matchType> <weight>1</weight> <confidence>7</confidence> </placeDetails> <referenceList> <reference> <woeIds>676757</woeIds> <start>50</start> <end>56</end> <isPlaintextMarker>0</isPlaintextMarker> <text><![CDATA[Munich]]></text> <type>xpathwithcounts</type> <xpath><![CDATA[/html[1]/body[1]/div[2]/div[1]/p[1]]]></xpath> </reference> </referenceList>

And some more boilerplate. :)
```
 </document>
</contentlocation>
```

Full documentation of the Placemaker query parameters and response format are available on YDN. Christian has put together a demo of a basic PHP implementation (though it’s XSSable, and shouldn’t ever be used in production).

In general, this is brilliant stuff. I'm looking forward to playing with it!

Well. That was easy.

Fri, 15 May 2009 17:00:00 -0700

Getting JSLint running inside Spidermonkey was much simpler than I expected it to be:

[14:12] src/jslint $ python
Python 2.6.2 (release26-maint, Apr 19 2009, 01:58:18) [GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import spidermonkey
>>> rt = spidermonkey.Runtime()
>>> cx = rt.new_context()
>>> text = file( './fulljslint.js' ).read()
>>> cx.execute( text )
u'use strict'
>>> cx.execute( text + 'JSLINT("i++", {})' )
False
>>> cx.execute( text + 'JSLINT("i++", {}); JSLINT.report();' )
u'<div id=errors><i>Error:</i><p><i>Implied global:</i> <code>i</code>&nbsp;<i>1</i></p><p>Problem at line 1 character 4: Missing semicolon.</p><p class=evidence>i++</p></div><br><div id=functions><div><i>No new global variables introduced.</i></div></div>'

Yay!

Running `python-spidermonkey` on JeOS

Fri, 15 May 2009 17:00:00 -0700

Paul Davis' python-spidermonkey project looks brilliant.

In a nutshell, he’s building a Python bridge to Mozilla’s SpiderMonkey “JavaScript on C” environment. I'm excited about that, because it means I might be able to put together a headless testing environment without trying to make everything work correctly inside Rhino. Hooray for options!

I ran into a snag or two while getting up and running on JeOS (my dev environment of choice), so I'm documenting the process here.

Install python headers and pkg-config. Forgetting to install pgk-config will give you some exciting errors in the compilation phase later on that make it sound like NSPR failed to install correctly. If you’re on JeOS, it’s more likely the case that you don’t have pkg-config at all:
```
sudo apt-get install python2.6-dev pkg-config
```
Install netscape portable runtime:
```
sudo apt-get install libnspr4-dev
```

Pull down python-spidermonkey

mkdir -p ~/src
cd ~/src
git clone git://github.com/davisp/python-spidermonkey.gitnsp

Build python-spidermonkey

cd ~/src/python-spidermonkey
python setup.py build

Test python-spidermonkey:

Actually, don’t do this. One of the tests fails on JeOS, and it fails in a way that sucks up all your resources and leaves you in an infinite loop. Yay!
Install: Even though the max time test fails (spectacularly), the bridge seems to work pretty well. Install the package anyway, just don’t rely on being able to set a max execution time on your code. :)
```
sudo python setup.py install
```

Now, to see about getting JSLint running in this environment… Fun!

The Value of Measurement

Sat, 09 May 2009 17:00:00 -0700

While I agree fully with many of the conclusions Lukas Mathis draws in an excellent essay on the recent Google/Douglas Bowman split, a few bits deserve further study. Lukas hits the central problem squarely on the head in his footnoted claim that measurement isn’t Google’s problem, rather the lack of a design team capable of balancing those measurements against other concerns. I'd elaborate on this claim, and argue that the kind of testing being discussed matters. In general, engineers understand and can relate well to automated A/B testing, and designers understand and can relate well to more personal usability testing. The two are, however, not the same, don’t provide the same data, and ought not be conflated.

What do A/B tests test?

“41 shades of blue” has become a catchphrase for those who see Google’s approach as flawed; this seemingly mindless insistence on A/B testing as the sole arbiter of a design’s “value” seems to me to be the point Bowman most objected to. I have a good amount of sympathy for that viewpoint.

A/B testing is, as Lukas rightly points out, hugely valuable to any team focused on improving an established system through a series of small, iterative changes. Serving a green button in place of a red button to 1% of your users, and discovering a statistically meaningful shift in usage patterns distinctly tells you something about your audience that no amount of surveying could. Ignoring these user’s claims while carefully observing their behavior can reveal interesting and useful bits of data about the way they interact with the interfaces you've built.

Especially with an audience the size of Google’s, you'd have to be crazy to write off the value of building small changes incrementally, and testing those piecemeal improvements against a subset of your audience. You’ll discover that one small tweak or another has an impact, and feed that back into your design process. Continually testing each iteration of tweaks against your audience will continually improve the quality of your pages and the overall user experience: It’s a virtuous circle of exactly the kind I'd like to encourage.

I absolutely stand behind the core of Lukas' essay: “Design is not held back by data, design is refined and perfected by data.”

The Downside

A/B testing is appealing because it’s obvious. It’s absolutely clear what needs to be done, the results are generally clear in the direction they lead (it’s either “A” or “B”), and if you have the tooling in place to make it easy (Google does), it’s easy to see how it can become a central part of the way you work.

That said, you'd be just as crazy to make every decision dependent on this sort of testing. In order to isolate the effect of a change, as a true engineer-statistician ought demand, broad and sweeping suggestions should be broken down into testable components, and each served across a just-wide-enough swath of audience to return meaningful data while minimizing risk. An insistance on A/B measurement of each change, no matter how small, means that everything you measure will be “trivial” in the way that the most glaring examples of “Googlism” have been: shades of colour, or widths of border. Only trivial changes are testable changes, everything else is fraught with layers of uncertainty: “Was it the new 3px border, or the new position of the button, or the completely new logo that drove traffic?” “Who knows, let’s do more tests!”

Moreover, it’s easy to see how a reliance on A/B testing might lead to a culture of extreme risk-aversion. When you have a ton of data at your fingertips that help guide your decisions, your worst enemy is the black hole that a new untested/untestable/unmeasured change represents. If a change isn’t testable/tested, it’s unknown. Unknown changes might be catastrophic. Catastrophic is bad. Therefore untested/untestable/unmeasured changes are, in the absence of evidence, bad.

I suspect that Bowman isn’t at all exhorting his fellow designers to abstain completely from testing; I'd interpret his remarks as cutting against an over-reliance on A/B testing. Within a primarily A/B framework, I'd consider it impossible to do real design work that reevaluates a product’s decisions from first principles. Where can you begin? What interesting set of changes can you propose when each has to be buildable in a way that allows testing against the current status quo of the product as a baseline?

I believe that Bowman is right to worry about missing “truly great” designs by getting bogged down in continual, iterative improvement of a mediocre design. Getting wrapped up in asymptoticly approaching the best shade of the best button you can possibly create, and backing it all up with testing, might blind you to the fact that the action might really be better represented as a slider. Or a dropdown box. Or that it’s completely wrong and would be better reworked entirely.

Lukas quickly notes one answer in passing. I think it deserves more attention.

Usability Testing and Prototypes

In an interview with Salon in 2008, Leander Kahney said:

Steve Jobs doesn't wake up one morning and there's a vision of an iPhone floating in front of his face. He and his team discovered it through this exhaustive process of building prototype after prototype.

Each of these prototypes functioned in one way or another, and each was put into someone’s hands to play with. This methodology of testing (especially for new designs) is simply not measurable in the same way as determining whether a green link on a pre-existing page is clicked on more or fewer times than the same link in red. A/B testing is not usability testing sweeping new prototypes; it has a different purpose, and gives different data points.

Prototyping new solutions to problems, and putting them in front of real people as quickly as possible, is probably the best way to make sweeping changes while maintaining some semblance of confidence that you’re moving in the right direction with regard to the audience you’re aiming for.

This sort of guerilla testing designs on friends and acquaintances with something like Silverback absolutely ought to be part of the design process, early and often. Getting real feedback from people who use the system you’re trying to improve is critical to designing and implementing an interface that makes the right tradeoffs in the right places.

In short, I'm highly sympathetic to the Nielsenian notion that users can’t and won’t tell us what they want, that we need to examine their clicks in order to measure the impact of the changes we make. However, I think that notion needs to be tempered insofar as never asking a user what she thinks is incredibly dangerous because it never allows you to break out of a certain comfort zone surrounding the product as it stands.

So. Test. Early, often, and persistently. Testing gives you immense amounts of data, and data is invaluable to your designs and implementations. But don’t limit your tests to those best expressed in a spreadsheet. Prototyping, and asking actual users actual questions, should be a part of your creative process.

Finally, to reiterate Lukas' point: you aren’t Google. Even if they’re dangerously-reliant on A/B testing, you almost certainly aren’t A/B testing enough. I know I'm not…

DSL Woes

Sat, 09 May 2009 17:00:00 -0700

My DSL connection is really quite fast; I'm apparently only ~800m away from the nearest DSLAM, and so I should in theory be capable of maxing out the connection. This holds up pretty well in speed tests: I get great downstream, and passable upstream scores when connecting to pretty much anything in Europe.

All told, I should be thrilled with my service, but I'm not, because it only seems to actually work about 80% of the time. Every 10 minutes or so, my DSL modem gets confused about exactly what’s going on and drops the connection, picking it up again about a minute later, and continuing on it’s merry way with a new IP. This wreaks havok with any sort of persistant-connection driven tool, be it SSH or IRC or Adium… drives me a bit nuts, actually. It’s gotten bad enough over the last few weeks that even Elisabeth notices the problem… :)

I've been on the phone with Alice pretty regularly for a few weeks now, and it’s the same every time: I call in, explain the problem to the first-line tech, who checks the logs, asks me what kind of modem I have, and passes me on to the second-line tech, who checks the logs, asks me what kind of modem I have, and runs a line test. The line test is pretty much ok, but they pass it on to the third-line tech anyway. The third-line tech does whatever it is that they do, and calls up a few hours later to tell me that they don’t see any problem, but that they’ll replace something anyway. Last week it was the splitter. The time before that it was something in-between my apartment and the DSLAM. I'm curious to see what it’ll be this time. Maybe the modem?

I'm actually not as frustrated about this as I could be. The customer service has been more or less all that I could ask for (short of actually solving the problem, of course). The reps are friendly, seem relatively knowledgeable, and don’t treat me as though I'm the problem. I'd love for them to actually track down the bugs and get them worked out so that I don’t drop out of chat rooms every few minutes, but I'm willing to give them the benefit of the doubt so far. This is a marked contrast to our previous dealings with T-Online and Telekom in general, with whom I now simply refuse to do business based on the absurdity of the customer experience.

In any event, they’re working on my DSL again, and I'm hoping the days of mikewest_, mikewest__, and mikewest___ will soon be behind me.

Ch-ch-ch-ch-changes

Thu, 27 Aug 2009 20:07:36 -0700

Lot going on this summer. My roommate moved back to CT, meaning that I had to buy a car and get a new apartment. I moved from the YAP team to the YUI team. Also, I’ve been spending more and more of my free time playing around with server-side Javascript, specifically Narwhal and Jack. My github account has been pretty busy.

I won’t say that I’m retiring this blog, exactly, but the posting frequency may be dropping considerably. Instead of 1-2 posts every month or two, expect more like 1-2 posts a year, if that.

In the meantime, I’ve got all sorts of other stuff going on at my tumblr blog, so you can follow me there.

Foo Hack 4.0

Tue, 05 May 2009 00:21:15 -0700

Partly because I haven’t been writing much CSS these days, and partly because it’s been just over a year since the last redesign, I felt like this site needed a face lift.

Fonts

I’ve been getting more and more excited about the impending freedom of the @font-face CSS declaration. The prospect using any true-type or open-type font in a web page without resorting to flash or images is incredible. If you’re viewing this site in Safari 4 or the latest beta releases of Firefox, Opera, and MSIE, you’ll have a much improved experience.

The body text is set in Bodoni SvtyTwo OS, chosen for its beautiful italics (not an oblique, but a true italic), dramatic hinting, and straightforward lines. I chose the OS over the regular Bodoni SvtyTwo because of the more stylish “lower case” numerals: 1234567890.

Update : I decided to go with Hoefler Text instead of Bodoni. Bodoni looks nice, but without the stronger hinting, it was hard to read on some monitors, especially for folks who are crazy enough to browse the web on Windows. Hoefler Text has a lot of what I liked about Bodoni, and was a strong contender, but it’s not free-as-in-speech. However, it is free-as-in-MP3, so I’ll use it until Hoefler & Frere-Jones send me a C&D . I also widened the column a bit to accommodate the wider typeface.

The headings are set in Qlassik Medium, by Dimitri Castrigue. It has just enough fun to make it stand out, but not so much as to be ridiculous.

Code snippets are set in the famous Bitstream Vera Sans Mono. I’d initially planned to set all things monotype in my favorite coding font, Century Schoolbook Mono. Once you get used to coding with serifs, nothing else feels right. Unfortunately, it doesn’t include bold and italic versions. While the OS will “fake” bold and italics with fonts that it has installed, it won’t play ball when asked to manipulate dynamic fonts linked from a CSS file. I put CentSchBook Mono first, so if you have it (which you should!), it’ll work. Otherwise, you’ll get Bitstream.

If you don’t have a browser that understands dynamic fonts, you can grab them from http://foohack.com/tpl/fonts/.

Layout

I really wanted to stretch my CSS abilities a bit, and also apply some of the principles of typographic design that I’d been learning about lately. The single-column is about 66 characters wide in the target font, and feels about “right” for readability.

Despite my best intentions, a lot of feature creep had somehow taken over, prompting the last revamp. In this version, I put back some features that I’d removed, and removed some things I’d left in.

The vertical measure is set at 30px, with a font-size of 20px. While that’s a bit large, I’ve always favored erring on the side of “too big”, and the Bodoni just felt too cramped at 16/24.

I’ve also taken a lesson from the print world, and set all my bullets and other adornments off into the margins, such that the text is kept strictly flush left. The only element that I couldn’t figure out how to do this with is the element when it starts a paragraph. In that case, it should have a text-indent somewhere around -0.5ex, but I couldn’t manage to do that without borking quotes that start somewhere in the middle of the paragraph.

Since all the major browsers support resizing fonts that are set in pixels, and “zooming” is now preferred over simple font resizing, I went ahead and set everything in pixels so that the math would be easier. There’s really no benefit to so-called “fluid” layouts that are set in ex or em measurements. Plus, browsers that support dynamic fonts seem to have a really hard time with ex/em measurements, since the layout is rendered before the font has downloaded. I do hope that gets fixed at some point, but doing layout in pixels was a welcome return to basics.

Supported Browsers

The luxury of a personal site: I didn’t test even once in MSIE, and I don’t plan to.

Also, since a lot of the style is in the typography, anyone with a browser older than 6 months isn’t going to see my beautiful styles. But they’ll see their own default font settings, I suppose, and that’s probably fine.

What do you think?

1234567890

Fri, 13 Feb 2009 15:31:30 -0800

Right now, the Unix Epoch Time is 1234567890.

Is it excessively nerdy that I’ve been looking forward to this for the past few years?

CSS vs Tables: You’re Doing It Wrong

Mon, 02 Feb 2009 23:40:22 -0800

Definitions first.

Table: A specified set of columns and rows with cells that automatically and fairly smartly expand to fit their contents; cells can span rows or columns; the table as a whole can be treated as one block, and cells can contain tables. A table is a metaphor for visually laying out 2-dimensional content.
CSS: cascading style sheets; not floating divs, not any specific markup, but just the concept of the visual display of markup specified by a series of rules that are kept separate from the markup they operate on, generally in an external linked file or a style tag in the head of the document.

Talking Past Each Other

CSS is (for various reasons cited elsewhere ad nauseam), in most cases, the more optimal technological approach. That’s not to say that the existing state of the CSS language is perfect. In fact, even the most vocal CSS advocates regularly assert that there are deep problems with the existing spec and browser support.

Tables are (for various reasons cited elsewhere ad nauseam), in most cases, a more optimal metaphor than any of its rivals. There’s nothing inherently wonderful about the tags themselves. In fact, even the most vocal table advocates regularly assert that the markup is ugly.

Therein lies the crux of the problem. CSS is a great technological approach, and tables are a great metaphor. The question then usually becomes, Should I mark up my content using

tags or

 tags?, which really means, Is the separation of style from content more or less important than using the optimal layout metaphor? The debate goes a little like this:

 tables 

 This layout metaphor is better, so you should use these tags. All you idealists are unrealistic! (Also, if you think markup matters, you believe in fairies. It so doesn’t!) 

 CSS 

 Metaphor shmetaphor! You don’t care about the sanctity of data or code quality. Think of the maintenance! (Also, if you can’t do it with divs, you’re stupid. It’s so easy!) 

 The Right Answer, it would seem, is to use the table metaphor with the CSS technology. In the ideal world, you’d always mark up your content using the tags that would communicate your meaning most clearly to your intended user agents. Then you’d tell user agents how to treat those tags visually. To describe this visual display, you should be able to use the table metaphor, or the float metaphor, or absolute and relative positioning, or anything else, but that should not be done by your markup.
 Getting Real

 In the real world, it’s not so nice. The CSS specification has a set of display values designed to mimic the table tags, but they aren’t well supported, and anyway, it’s an imperfect imitation of a hack that was not really intended to be used the way we all ended up using it.
In other words, CSS is not an ideal example of what CSS aims at, and the table/tr/td tags are not an ideal implementation of the table/grid metaphor. CSS tables, though they are clever and in some cases quite useful, take it from bad to worse.
 The state of the CSS art is not at a point where you can realistically expect to make any significant stylistic changes to your pages without altering the markup. If you need to move the navigation bar from the right to the left, or re-skin the page with dropshadows instead of rounded corners, or convert your gradients from 2-tone to 3-tone, or make your boxes vertically centered instead of top-aligned, you’re probably going to have to change your markup, at least a little.
 In theory, it’s possible. I know a handful of my colleagues will vehemently disagree and point at countless approaches to enable virtually any kind of reskinning using only CSS changes. (I’ve used Leslie Sommer’s CSS Mojo technique to great success in the past; we used it for pretty much everything on Yahoo! Buzz.) But let me tell you, from years of experience building production sites with CSS, most of the time, in reality it just doesn’t work that way.
 And why should it? What kind of crazy lunatic writes all their HTML by hand in text files, anyhow? Clearly you have some kind of back-end engine spitting it out from templates, so you just change the template, and voilà!, the markup is changed everywhere.
 A Very Blue Bike Shed

 Working at Yahoo, I’ve met some webdevs with truly incredible CSS ability, who crank out live code under real deadlines to exacting standards. They use CSS, and not tables, and they Get Shit Done. There are a lot of them, more than I can list here, but Nate Koechley, Matt Sweeney, Hedger Wang, and Leslie Sommer all deserve a special mention. I came to Yahoo knowing CSS pretty well, but I became an expert largely as a result of working in such a highly skilled community of developers.
 Also due to my time at Yahoo, I’ve seen some absolutely crazy debates about markup and coding standards on the devel-frontend mailing list. I mean, you think this little flare-up in the blogosphere is anything? You got no idea, buddy. Seriously. And these are coworkers who (for the most part) really like and respect one another.
 You wanna know a secret? It makes no difference.
 Language is as language does. If everyone uses the 













 tag for markup, guess what happens: 
 is now a semantically meaningless (or at least, semantically vague) tag, and any user agents that might care about navigating tabular data have to deal with that fact. It becomes just another , for all intents and purposes, but a little less flexible, and with a default styling that makes it handy for layout.

 The passion for the rants comes from the fact that it is meaningless! Of course it’s a fertile ground for bikeshedism!

 That’s right: It doesn’t matter.


 Use whatever tags work for you. I don’t care. You’ll still be a good person.

 I can tell you from experience that a deep knowledge of CSS is essential for serious front-end development. (Most so-called DHTML is actually just manipulating CSS with Javascript, and the Javascript is pretty light.) CSS can do some things that table tags can’t, so you ought to learn it. The reverse is also true, so you should know how to use tables, too.

 Write code that you can maintain, that is flexible enough to let you change it without sacrificing the stability of your product. Test it in the user agents that you expect to support. Stop debating the color of this bikeshed.

 If there’s ever a good reason to go back and change your tables to divs, or vice versa, you can do that. Hell, plan for that, because whether you use CSS for layout or not, you’re going to need to touch your markup sometimes. It is much much harder to build a product people want than it is to build a product that works in Browser X.


Will be moving to http://hedgerwow.blogspot.com/ in 3 days.
Mon, 23 Nov 2009 00:07:41 -0800
Dear all.
I am done with DreamHost and will move to http://hedgerwow.blogspot.com/

Maybe I shall get a new domain or keep the existing one.
See you soon  



Preview image inline before uploading.
Thu, 04 Jun 2009 00:12:55 -0700
I hope HTML5 and Google Gear could do this thing better 

http://hedgerwow.appspot.com/image-upload-preview/demo.html 



JOT: The JavaScript Object Template
Tue, 19 May 2009 14:05:56 -0700
http://code.google.com/p/jot-project/wiki/SyntaxAndHowTos  , still beta.
Hope you’d like it.



Found a new JS pattern from MSDN.
Thu, 30 Apr 2009 13:36:26 -0700
One day, I was trying to detect whether an element is an element that has no close tag.

So I did this:
var tagName = el.tagName.toLowerCase();

if( tagName == ‘img’ || tagName == ‘br’ || tagName == ‘input’ || tagName == ‘embed ‘ ){

// This is just a example, not all single tags are tested. These tags are :

// area,base,basefont,br,col,frame,hr,img,input,isindex,link,meta,param,embed.

// Do something here.

}
Well, this is not that efficient to write your program.

So I switch to use a string map, which is rather simple and John Resig uses the same trick in his HTML Parser.
var map = makeMap(’area,base,basefont,br,col,frame,hr,img,input,isindex,link,meta,param,embed’);

if( map[tagName]){

// Do something here.

}
Of course we can use RegExp to test it, just like many others did.
var reg = /^(area|base|basefont|br|col|frame|hr|img|input|isindex|link|meta|param|embed)$/i;

var tagName = ‘BR’;

if( reg.test(tagName)){

// Do something…

}
and maybe you can find more tricks that solve the same problem with low cost, but below is the most interesting one that caught my attention.
if ( tagName != “BUTTON” | “B” | “A” ){

// Do something…

}
Wait, this seems really strange to me and seems not to perform as what I was expecting.

However, since I found this code snippets from MSDN, there must be something magic which makes this working.

Unfortunately, this is simply not working though I almost thought I found a new JS pattern from MSDN and was very excited about it for just about 10 seconds.
It’s fun to learn JS from microsoft.





update:Cross Browser Base64 Encoded Images Embedded in HTML
Thu, 16 Apr 2009 21:57:58 -0700
http://www.hedgerwow.com/360/dhtml/base64-image/demo.php 



Notes from stylesheet-debug.js
Fri, 20 Feb 2009 11:35:18 -0800
Below are the notes from the newly released YUI version 2.7.0 stylesheet.js

and the notes from its source probably explains why web development is too tricky to deal with.
Adapted from tylesheet-debug.js.....
/*
Copyright (c) 2009, Yahoo! Inc. All rights reserved.
Code licensed under the BSD License:
http://developer.yahoo.net/yui/license.txt
version: 2.7.0
*/

* Style node must be added to the head element. Safari does not honor styles
applied to StyleSheet objects on style nodes in the body.
* StyleSheet object is created on the style node when the style node is added to the head element in Firefox 2 (and maybe 3?)
* The cssRules collection is replaced after insertRule/deleteRule calls in
Safari 3.1. Existing Rules are used in the new collection, so the collection
cannot be cached, but the rules can be. * Opera requires that the index be passed with insertRule.
* Same-domain restrictions prevent modifying StyleSheet objects attached to
link elements with remote href (or "about:blank" or "javascript:false") * Same-domain restrictions prevent reading StyleSheet cssRules/rules
collection of link elements with remote href (or "about:blank" or
"javascript:false")
* Same-domain restrictions result in Safari not populating node.sheet property for link elements with remote href (et.al)
* IE names StyleSheet related properties and methods differently (see code)
* IE converts tag names to upper case in the Rule's selectorText * IE converts empty string assignment to complex properties to value settings
for all child properties. E.g. style.background = '' sets non-'' values on
style.backgroundPosition, style.backgroundColor, etc. All else clear style.background and all child properties.
* IE assignment style.filter = '' will result in style.cssText == 'FILTER:'
* All browsers support Rule.style.cssText as a read/write property, leaving only opacity needing to be accounted for.
* Benchmarks of style.property = value vs style.cssText += 'property: value'
indicate cssText is slightly slower for single property assignment. For
multiple property assignment, cssText speed stays relatively the same where style.property speed decreases linearly by the number of properties set.
Exception being Opera 9.27, where style.property is always faster than
style.cssText.
* Opera 9.5b throws a syntax error when assigning cssText with a syntax error. * Opera 9.5 doesn't honor rule.style.cssText = ''. Previous style persists.
You have to remove the rule altogether.
* Stylesheet properties set with !important will trump inline style set on an element or in el.style.property.
* Creating a worker style collection like document.createElement('p').style;
will fail after a time in FF (~5secs of inactivity). Property assignments
will not alter the property or cssText. It may be the generated node is garbage collected and the style collection becomes inert (speculation).
* IE locks up when attempting to add a rule with a selector including at least
characters {[]}~`!@%^&*()+=|? (unescaped) and leading _ or - such as addRule('-foo','{ color: red }') or addRule('._abc','{...}')
* IE's addRule doesn't support comma separated selectors such as
addRule('.foo, .bar','{..}') * IE throws an error on valid values with leading/trailing white space.
* When creating an entire sheet at once, only FF2/3 & Opera allow creating a
style node, setting its innerHTML and appending to head. * When creating an entire sheet at once, Safari requires the style node to be
created with content in innerHTML of another element.
* When creating an entire sheet at once, IE requires the style node content to be set via node.styleSheet.cssText
* When creating an entire sheet at once in IE, styleSheet.cssText can't be
written until node.type = 'text/css'; is performed.
* When creating an entire sheet at once in IE, load-time fork on var styleNode = d.createElement('style'); _method = styleNode.styleSheet ?..
fails (falsey). During run-time, the test for .styleSheet works fine
* Setting complex properties in cssText will SOMETIMES allow child properties to be unset
set unset FF2 FF3 S3.1 IE6 IE7 Op9.27 Op9.5
---------- ----------------- --- --- ---- --- --- ------ -----
border -top NO NO YES YES YES YES YES -top-color NO NO YES YES YES
-color NO NO NO NO NO
background -color NO NO YES YES YES -position NO NO YES YES YES
-position-x NO NO NO NO NO
font line-height YES YES NO NO NO NO YES -style YES YES NO YES YES
-size YES YES NO YES YES
-size-adjust ??? ??? n/a n/a n/a ??? ??? padding -top NO NO YES YES YES
margin -top NO NO YES YES YES
list-style -type YES YES YES YES YES -position YES YES YES YES YES
overflow -x NO NO YES n/a YES ??? - unsetting font-size-adjust has the same effect as unsetting font-size * FireFox and WebKit populate rule.cssText as "SELECTOR { CSSTEXT }", but
Opera and IE do not.
* IE6 and IE7 silently ignore the { and } if passed into addRule('.foo','{
color:#000}',0). IE8 does not and creates an empty rule. * IE6-8 addRule('.foo','',n) throws an error. Must supply *some* cssText
*/






Be careful with getComputedStyle
Thu, 19 Feb 2009 23:07:03 -0800
Sometimes you really need to check whether an element is in the document already before calling some DOM methods.

See http://www.hedgerwow.com/360/bugs/dom-get-computed-style.php 



Imageless Fuzzy Shadows with CSS + HTML
Fri, 05 Dec 2008 17:41:55 -0800
It’s fun to play with CSS over 10 years.

Year 1999: filter:blur();

Year 2009: background:url(data:image/png;base64);

http://www.hedgerwow.com/360/dhtml/css_fuzzy_shadow/demo.php
  



What new Chief Technology Officer of the US need to do?
Sun, 26 Oct 2008 14:16:29 -0700
Banning the use of IE6. 

I do believe it’s just the right time to do the right change to make the internet faster, greener, cheaper, which helps to boost the poor economy.



Choosing a DOCTYPE that is just simple and strict
Thu, 16 Oct 2008 15:53:25 -0700
I did not know this trick until I view the HTML source of



a randomly search result from google



http://www.hedgerwow.com/360/dhtml/short_doc_type.html





Enliven your namespaces
Thu, 01 May 2008 09:59:32 -0700
It’s a common good practice to use namespaces when writing your JavaScript. The reasons why have been stated time and again by much smarter and more eloquent people than myself, so suffice it to say, you should do it.
The typical method for namespacing has been to use object literals such as
var myNS = { myFunc : function () { /* do stuff */ }
};
However, A namespace needn’t be inert¹. It’s just an object to hang stuff off of, so technically it can be anything that you can hang stuff off of. That is, anything that descends from Object, as well as other things like RegExp instances and, more interestingly, functions.
Yay lambda
In JavaScript, functions are first class objects. This allows code such as this:
var foo = { bar : function (m) { alert(m); }
}; var barFunc = foo.bar;
var baz = { bar : foo.bar, goop : barFunc
}; // All of these will now do the same thing
foo.bar('Hello world');
barFunc('Hello world');
baz.bar('Hello world');
baz.goop('Hello world');
foo.bar.call(someRandomObject, 'Hello world');
The only difference between them is the resolution of the this keyword.
Breathe life into your namespace
Let’s use a function as a namespace instead. Let’s say you have an object literal containing a suite of functions, but one function is called far more often than the others. We can use that function as the namespace, then alias the function name from within the namespace for consumers that like to type.
// convenience function for merging objects
var set = function (to,from) { for (var k in from) { if (from.hasOwnProperty(k)) { to[k] = from[k]; } }
}; // create a namespace called node (in practice, choose something better)
var node = function (id) { return document.getElementById(id);
}; // Add functions to the function namespace just as you would to an object literal namespace
set(node, { byId : node, // alias to NS function for completeness /* … and then a few more functions that sit in the node namespace */ _templates : {}, create : function (name,conf,content) { var n = (this._templates[name] || (this._templates[name] = document.createElement(name)); if (n) { n = n.cloneNode(true); set(n,conf); // set things like className or innerHTML if (content) { n.innerHTML = n.innerHTML.replace(/{([\w\-\s]+)}/g,function (x,placeholder) { return (placeholder in content) ? content[placeholder] : ”; }); } } return n; }, remove : function (n) { if (n.parentNode) { n.parentNode.removeChild(n); } }, /* etc etc */
}); // Now you can use the namespace node(x) rather than type node.byId(x)
var n = node(’foo’);
var n1 = node.byId(’foo’); // synonyms // And of course, the rest of the functions work the same
var div = node.create(’div’,{className: ‘bar’, innerHTML: ‘{message}
’}, {message: ‘hello world’});
this()
An interesting side effect of using a function as a namespace is that during the execution of functions under that namespace, this is resolved to the function ref. That means you can do something like
var node = function (id) { return document.getElementById(id);
};
node.remove = function (id) { var n = this(id); if (n && n.parentNode) { n.parentNode.removeChild(n); }
};
The use of this as a function ref in that case would cause a JavaScript error in the case where the node.remove function was not called from the node context such as in
var rm = node.remove;
rm('boom'); // JavaScript error. 'this' evaluates to the global object (window) which isn't a function // or
node.remove.call(somethingOtherThanAFunction,'boom');
But that’s a separate issue. The language has the capacity to be used in this way, and the value of leveraging this tactic needs to be evaluated and contrasted to the the potential breakages.
I’m personally a big fan of function namespaces, but it hadn’t occurred to me until just this morning that this also gained some super powers in the process. I haven’t yet thought of a case for its use (other than to trim code), but boy am I interested to hear suggestions!

Actually, even the object literal namespace isn’t inert. It brings with it all the methods and members that hang off of Object instances.



label + checkbox concerns
Mon, 21 Apr 2008 15:27:04 -0700
You have a label with a nested checkbox input element like so:
foo
When you click in the label area, all browsers except IE6 will fire the onclick event for the label followed by the onclick event for the input. IE6 will only fire the input’s onclick if the label’s for attribute is set and points to the id of the input.
foo
Also, both IE6 and IE7 will incorrectly render the following style:
label { background: #def; padding: 2em 4em; }
input { margin-right: 2em; }
Firefox


IE


And the icing: neither IE6 nor IE7 will fire the label’s onclick event for clicks occurring in the marginal space.




Tyranny? I’m all for it!
Tue, 15 Apr 2008 20:35:45 -0700
I sometimes wonder if the NPR newscasters feel frustrated because they are paid to relate some of the most inane drivel, but in the same tone as genuinely important information.
Pope Benedict XVI arrived in the US today, greeted by president Bush, and  NPR got the pleasure of reporting that they shared many of the same values, such as opposition to tyranny and abortion.
That’s right, amongst their similar values, opposing tyranny was apparently distinct enough to be included in a summary list of two items. In case we might have thought that one or the other was (publicly) in favor of tyranny.
I wonder what else was on the list that didn’t make the cut…
They both like red?

They both fear Jello?

They both think stinky cheese is best left in France?

They both think good stuff is better than bad stuff?


I <3 Firefox 3
Fri, 11 Apr 2008 23:35:58 -0700
I just noticed Firefox 3 (on Mac) is rendering dotted borders rather interestingly.
It appears to start with a square in each corner with a half circle on the clockwise side, then circular dots spaced evenly between that semi-circle and the midway point of the next corner block.


I played with it a bit and came up with this:

Which renders like this on FF3b5 Mac:


Too bad I’m late for Valentines day.


I can has spambot
Sun, 30 Mar 2008 21:06:42 -0700
I read on Wired the I Can Has Cheezburger folks started a new venture involving graphs/charts. Having such a chart lying around, I figured I’d submit it. Apparently that was a big mistake.
I used to receive around 100 (properly filtered) spam messages per day, but the day after the submission, I’m now receiving nearly 2000, many of which have landed in my inbox.
Chewbacca defense?
Given normal circumstances, I probably would have considered this possibility and used my crap email, but it just so happened my keyboard had stopped working, behaving as though I’d changed the key mapping from qwerty to some approximation of wookie.
In my stubborn insistence to do one simple thing before tearing the keyboard apart for deep cleaning (read: likely necessitating the purchase of a new one), I discovered that most of the qwerty right center row were represented by the top number keys and ‘a’ was backspace. Between this and some random cutting and pasting into the email, I was able to eliminate the todo item and get busy extracting 3 years worth of lunch and dinner from my keyboard’s innards.
In my …haste? I overlooked the small detail of from address. I know 2000/day may not seem especially notable given the average email addy’s signal:noise ratio, but I had managed to do a pretty good job of keeping that account in the "light spam" department, even while displaying it here on ls.n (using some horrible, antiquated js obfuscation). It pains me to see the result of one slip.
Dear ICHC crew: bad form.
Spam sucks.


Yahoo! Messenger 10 beta
Tue, 25 Aug 2009 10:53:02 -0700

Yahoo! Messenger 10 beta (windows download) was released yesterday. It is a major milestone for the instant messaging team, and for me in particular because it means the new insider page is now live! This is the project I have been working on for the past few months. It is essentially a mini version of the Yahoo! homepage built with YQL cloud technology, complete with mail, news, weather, search assist, a high-revenue ad position, alternate stylesheets (skins!), a location picker, and there’s even an easter egg or two. Go find them if you can.  
The page is intended to run “inside the IM client” but it works just fine in any modern browser window, too. Check it out here (you will need a Yahoo! id).


XML string to PHP array
Thu, 13 Aug 2009 19:59:59 -0700
One common need when working in PHP is a way to convert an XML document into a serializable array. If you ever tried to serialize() and then unserialize() a SimpleXML or DOMDocument object, you know what I’m talking about.
Assume the following XML snippet:


  

    Brian

    Chris

    Meg

  


There’s a quick and dirty way to do convert such a document to an array, using type casting and the JSON functions to ensure there are no exotic values that would cause problems when unserializing:

Here is the result for our sample XML, eg if we print_r($a):
Array
( [show] => Array ( [@attributes] => Array ( [name] => Family Guy ) [dog] => Brian [kid] => Array ( [0] => Chris [1] => Meg ) )
)
Pretty nifty, eh? But maybe we want to embed some HTML tags or something crazy along those lines. then we need a CDATA node…


  

    Brian

    Chris

    Meg

    Stewie]]>

  


The snippet of XML above would yield the following:
Array
( [show] => Array ( [@attributes] => Array ( [name] => Family Guy ) [dog] => Brian [kid] => Array ( [0] => Chris [1] => Meg [2] => Array ( ) ) )
)
That’s not very useful. We got in trouble because the CDATA node, a SimpleXMLElement, is being cast to an array instead of a string. To handle this case while still keeping the nice @attributes notation, we need a slightly more verbose conversion function. Here is my version, hereby released under a do-whatever-but-dont-sue-me license.
loadXML($xmlstr); return domnode_to_array($doc->documentElement);
}
function domnode_to_array($node) { $output = array(); switch ($node->nodeType) { case XML_CDATA_SECTION_NODE: case XML_TEXT_NODE: $output = trim($node->textContent); break; case XML_ELEMENT_NODE: for ($i=0, $m=$node->childNodes->length; $i<$m; $i++) { $child = $node->childNodes->item($i); $v = domnode_to_array($child); if(isset($child->tagName)) { $t = $child->tagName; if(!isset($output[$t])) { $output[$t] = array(); } $output[$t][] = $v; } elseif($v) { $output = (string) $v; } } if(is_array($output)) { if($node->attributes->length) { $a = array(); foreach($node->attributes as $attrName => $attrNode) { $a[$attrName] = (string) $attrNode->value; } $output['@attributes'] = $a; } foreach ($output as $t => $v) { if(is_array($v) && count($v)==1 && $t!='@attributes') { $output[$t] = $v[0]; } } } break; } return $output;
}
?>
and the result, for our Stewie snippet:
Array
( [show] => Array ( [@attributes] => Array ( [name] => Family Guy ) [dog] => Brian [kid] => Array ( [0] => Chris [1] => Meg [2] => Stewie ) )
)
Victory is mine!  


Upgraded to WordPress 2.8.x
Thu, 11 Jun 2009 18:08:11 -0700
Update: now on 2.8.4 - woooo!!!
 Growing tired of the constant WordPress updating! Sigh… 
 See the changelog for details on what is new in this release. Honestly I didn’t even read it. Instead I chose to fall asleep while watching the jazzy intro video.
 [ video embedded after the jump ]


Roadtrip Redux
Sat, 23 May 2009 06:58:01 -0700
Rejoice, Governator: I will be paying the state tax after all. Job prospects here in Texas are not all that great for Anne. In contrast, she has a number of opportunities in California. So, since it don’t matter where I work from now that I’m officially a telecommuting Yahoo!, we will be heading back west next week, after my uncle and aunt return from their trip to Europe.
This time, we intend to take the northern route, via Oklahoma and Salt Lake City. As usual, stay tuned to the Twitter for updates!


Moving back to Austin
Wed, 01 Apr 2009 15:06:36 -0700

We have decided to move back to Austin, TX. Our appartement lease is up, all the stuff is in boxes, and I’m getting ready for the transition from too much Sushi to too much Mexican food. 
I’ll still be working (remotely) for Yahoo!.
The dot in the middle of the map is Roswell, NM – where we will obviously stop for some dorky tourist trap. For realtime updates during the roadtrip follow the Twitter. 


Re-tweaked h2o theme
Mon, 19 Jan 2009 16:24:24 -0800
The theme used here, h2o, is becoming somewhat of a monstruosity. I really should write a completely new one. But for now, iterating already allows a little refresh, so behold a new hackishly re-tweaked theme!
Because of Twitter, I rarely post anything anymore… so I went from two-posts on the homepage to just one. The header was also changed, hiding the description so that focus is now completely on the latest tweet block and it’s awesomely mind-blowing font-size animation. 
Let me know what you think in the comments!


Upgraded to WordPress 2.7
Sun, 14 Dec 2008 17:55:12 -0800
Wordpress‘ new admin interface is very nice. 
The upgrade process was smooth as usual.
Growing tired of H20. I think I’m going to tweak a theme again. This time I’ll base it on YUI AutoGrid minimal and have support for widgets.
Then again, why mess with what works already ?


Bloodbath Redux
Wed, 10 Dec 2008 16:10:29 -0800
I was not laid off today, but another set of co-workers were, including my immediate counterpart. Bits Blog (NYTimes.com) thinks that the Yahoo Layoffs Today May Not Be Last…

Yahoo began laying off 1,500 workers on Wednesday as part of a plan, announced in October, to slash expenses by $400 million a year. The cost cutting, however, may have to go deeper in the coming year. 
“There could be additional staff reductions next year,” said Brad Williams, a Yahoo spokesman. “It depends on the decisions we make about prioritization, and on things we can’t predict in the economy.” Mr. Williams added: “We are trying to instill a culture of cost-discipline in our business.” [He] also said that the job cuts today will affect most parts of the company, but that Yahoo executives are still evaluating which projects will be de-emphasized or cut altogether. “This is more of a cost reduction rather than business prioritization,” he said about the layoffs. “Business prioritization will continue going forward. We are looking at businesses we may put into maintenance mode.” 
Yahoo’s chief executive, Jerry Yang, penned a farewell note to laid-off employees on the company’s blog. If you want to know how managers are supposed to notify employees whose jobs are being cut, check out Valleywag.

Sad day.


Yahoo! CEO Plans to Step Down
Tue, 18 Nov 2008 15:08:50 -0800
Yahoo! said Monday evening that Jerry Yang, its chief executive, would step down from that role after the company finds a replacement.
Mr. Yang, a co-founder of Yahoo!, assumed control of the company a year and a half ago from Terry Semel, a Hollywood studio boss that he hand-picked for the job. His tenure has been a tumultuous period during which Yahoo! rejected a $47.5 billion takeover offer from Microsoft and failed to cement an advertising partnership with Google.


 Jerry Yang, Yahoo Chief, Plans to Step Down – Bits Blog – NYTimes.com 
Yahoo’s CEO Choices | Kara Swisher | BoomTown | AllThingsD
Yodel Anecdotal – Stepping down (Jerry Yang)



Obama & Scientific Innovation
Wed, 05 Nov 2008 18:09:47 -0800

Yesterday, the American people chose Barack Obama as the country’s 44th president, promising a sea change in US policy that could affect not just the US, but the whole world.
Take a look at what Obama has pledged over the lengthy presidential campaign, to see what his administration will mean for science and technology.
read more | digg story


A most Singular project
Tue, 03 Feb 2009 03:36:07 -0800





NASA Ames Exploration Center

Originally uploaded by Laughing Squid



Over the last 2-3 months, a lot of people have been watching my tweets and listened to my various conversations wondering what exactly I’m up to. I haven’t been able to say much, but can now finally talk about it. Biting lips for this long has been tough, but I can finally say I’m heading up Singularity University – some great coverage from TechCrunch and CNet (among others).
I’ve been fascinated with innovation and growth for many years, and my experience at Yahoo building and running Brickhouse was very foundational. Analyzing thousands of ideas, putting together great teams and working with some of the world’s best engineers to launch cutting edge products was a hell of a privilege. While at Yahoo, I established an important relationship between Brickhouse and NASA, which led to an invitation to the founding meeting of this initiative.
Once there, I was pretty well hooked. The vision that Ray Kurzweil and Peter Diamandis laid out was extremely compelling: Fundamentally, the world is facing some very large challenges and the technologies identified by Ray can lead to scalable solutions to address these global issues. Add to that the interdisciplinary expertise brought by Peter’s experience and it was too much to pass up. They only had to ask once.
The full press release is here.
Great innovation almost always happens when two disparate concepts are brought together. At SU, we’ll be bringing together dozens of experts across several distinct, accelerating technologies and letting them think about some of the grand challenges facing humanity. 
A well-publicized example is 3D organ printing. Scientists are combining 3D printing with stem stell research to ‘print’ human organs. And that is one idea…





Analysis: Blackberry Curve vs. G1 Android
Sun, 18 Jan 2009 18:49:15 -0800







Blackberry Curve vs. G1 Android



As part of a new project, our team was donated some G1 Android phones to play with. Already being a T-Mobile customer, it was a no-brainer to play with it for a couple of days and I got to put the phone through its paces. Here’s what I discovered when comparing it to my Blackberry Curve (readers of this blog are no strangers to my endless attempts to optimize my mobile experience)…
SETUP

=====

Moving over to the G1 was very simple given I’ve been using a lot of Google services. I have Google Sync on my BB, so my contacts and calendar are already online and I use Remember The Milk for task (mis)management, so that was also easy. I was up and running in less than 5 mins, including IMAP setup
THE PRO’S

=========

- the G1 is intuitive and easy to use with an excellent interface

- the keyboard is surprisingly good and not having to hit ’shift’ for periods or numbers is huge – I could actually use this for medium-heavy typing efforts

- I downloaded a few apps and they worked really well. The big draw for me was Skype (worked well) and tethering (which I didn’t finish configuring)

- screen is awesome – very bright

- browser is awesome – very thorough and easy to use

- GPS! Fabulous (easy to update FireEagle)

- youtube on the phone – excellent
CONS

=====

- profile – bigger and heavier than my Curve

- battery life (SUCKS!) – big problem – lasts a half day with heavy use

- sliding open and turning sideways everytime I have to type – slight hassle and requires two hands

- calling someone requires way more clicks (and often, both hands)
SUMMARY

========

The Curve is totally optimized to be a great, great phone/email tool and other stuff (music, video) is secondary. The G1 is great for rich media but clunkier as a phone/email tool.
Overall, I’m sticking with the Blackberry for now as I make a lot of calls and emails and doing that really well with a whole day of battery life makes a big difference for me. But I will continue to play with the G1, and if I can get tethering working, then will carry it along on trips.






Sonal Shah Appointed as Key Obama Adviser
Mon, 10 Nov 2008 02:33:13 -0800

Wow – when it rains it pours. After eight years of Bush drought we finally get a big relief – first Obama runs away with the election and now I just heard that he’s appointed Sonal Shah to his inner circle.
I first met Sonal in 2003 at the awards gala where she was appointed India Abroad’s Person of the Year for founding Indicorps. Then she then came out to California to run a big chunk of Google.org and we were housemates for most of last year, during which we became very good friends. I haven’t seen her much recently as she’s been heavily involved helping with the campaign, so was taken quite unawares with this fabulous news. 
I am, however, not at all surprised. Sonal is an extraordinary choice. She’s dedicated her life to social causes and philanthropy and is one of the wisest, brightest, most energetic people I know.
Certainly if I were President I’d have her in my inner circle  )
Go Sonal!! The full story is here.



Obama Wins – Thank The Lord…
Mon, 10 Nov 2008 01:50:19 -0800







It’s Obama – by a landslide…



Originally uploaded by Nicki’s Knickers™



Readers of this blog (all three of them) will know that I don’t often comment on politics. And though my twitter feed has leaned towards the democrats, I’m more of a libertarian. Given the soaring heights of this occasion, though, I feel I must break my non-blogging policy and put down a few thoughts. 
First, the Republicans:

- how can you call yourselves conservative after the insanity of the spending increases? What happened to fiscal prudence?

- McCain, what in the name of all that is holy possessed you to put Palin on the ticket?

- Dubya, there are better ways of handling unresolved paternal issues than lying, invading one country and bankrupting your own 

- Americans, how could you re-elect Bush in 2004? I mean, really… 
And to the Democrats:

- what have you been doing in the last six years??

- please please don’t go too crazy with the spending

- whatever you do, please fix the healthcare system
And to Obama:

- many people I respect very highly got on planes or spent considerable time helping you out. That said it all for me. They include Sunil Paul, Robert Goldberg, Andrew Rasiej, Sonal Shah and who can forget Oprah leaning on Sam Perry… thrilling.

There were times in the last 2-3 years that I despaired utterly for this country. Now I’m back up to ‘very concerned’.






Maxroam Totally Rocks (a review)
Wed, 18 Jun 2008 23:58:13 -0700
On my second last trip to Ireland, Pat Phelan, one of my favorite bloggers, bounced into the speaker’s dinner, handed me a SIM card and bounced right out again. Pat runs MaxRoam, a disruptive new telecoms play. On my recent trip, I put it through its paces. 
WHAT IT DOES

MaxRoam helps avoid international roaming charges. Simple. 
HOW

They put multiple numbers (all land lines) on the same SIM card. For instance, I have numbers for the U.S, Ireland, UK and Germany. If any of those numbers is called, my phone rings, wherever I am. I pay the local 25 cents (or whatever) charged by the local carrier. It costs 3 euros per month per number and you prepay the service. The SIM costs 25 euros. (Disclosure: because I’m an honorary Cork man, Pat gave me the SIM for free with a bit of credit to play with).
THE PROS

- being landlines, it’s cheap for people to call you

- you can get a new number right from the website and it works instantly (amazing)

- you can forward (for free) any of your numbers to a local landline OR a local mobile and nobody pays. Outside North America, this is monstrously cost-saving as the caller usually pays (a fortune). Here, Maxroam swallows the costs.

- 5 cent SMS worldwide (I didn’t test this)
THE CONS

- you can’t put your current number onto it (but you can forward your number to it)

- forwarding usually works, but occasionally it took 2-3 tries when I used it

- lots of numbers to think about

- you need a totally unlocked phone (I used an old Nokia)

- not the greatest website in the world (but who cares, it works)
OVERALL

After a bit of fiddling to get used to it, I’m hooked. I totally loved the service – can’t imagine traveling without it. Being able to forward a 212 number to a UK mobile and talking for hours for no charge internationally on a mobile was fabulous. I worked out I saved about $185 in my two week trip. The cost for my two weeks of rather frequent calling was about 14 euros. I’m hoping he adds a data service soon.
I think Pat is onto something really huge here. The addressable market for international roaming is monstrous, and creating an MVNO is a very interesting way of attacking it. I wish I could invest (Pat?, nudge wink?). More coverage on VentureBeat and TechCrunch.
Pat, by the way, is also behind Twitterfone.


Europe trip log
Mon, 16 Jun 2008 12:42:44 -0700





Tom HC Bday card



Wow, how bad a blogger am I (ok, I never really had the moniker anyway). But it’s way long since I updated the ole papyrus. I’ve been back from Europe for 2-3 weeks and it’s been non-stop meetings since. Let’s first start by recapping that trip.
First stop was London – a few meetings and some quality time with my god-daughters (little Irish identical twin girls). Next was Dublin for more meetings – I’m on the advisory board of PutPlace, so met with Joe Drumgoole and then lunch with Joe and Niall Larkin. Joe, after paying for lunch, also organized a very nice little meetup and some interesting folks showed up, including Paul Campbell, Marcus MacInnes, Eoghan McCabe and Dennis Deery. Oh yes, and I said hi to Bill de hÓra for 60 secs as he whizzed by… good to see him again, albeit briefly.
Then it was off to Hamburg, where Sarik Weber invited me to give a keynote address at the Facebook Developer Garage. Nice reviews of my talk here and here (if you read German).
Back in London, Tom Hughes-Croucher took me out to lunch and gave me this birthday card (like, a real one), which I haven’t seen for ages. Thanks Tom!
And finally, I met the BT/Osmosoft chaps… Jeremy Ruston, Paul Downey, Phil Whitehouse, Phil Hawksworth and others. Confabb is doing a very interesting collaboration with them that’ll beta at SuperNova.





Igniting Web 2.0 Expo
Tue, 15 Apr 2008 12:35:56 -0700





A proposal for an Ignite talk I submitted has been accepted for next weeks Web 2.0 Expo in San Francisco. It’s going to be about the metaphysics of growth, something I’ve been thinking, reading & writing about for about 15 years.
I touched on this topic very briefly at Brad Templeton’s Nanotechnology Conference which we hosted at the Yahoo HQ.
This’ll be the first time I’ve ever talked about some of this stuff publicly, so the response should be interesting. The basic question is: “How does growth happen”? I’ll be talking about the basics of it and then applying the principles to the rapid growth environment of Silicon Valley.
Now all I have to do is write it… 


Nevada Town Hall Politics (Live-blogging) – Part 3
Wed, 19 Mar 2008 20:46:59 -0700
Onto the Q&A… this should be the fun stuff…
Q: Why is x traffic study not being considered in your proposal

A: The study wasn’t done in time
Q: You’re claiming that after 400 new units the traffic will decrease?

A: [[can't hear answer]]
Crowd is getting raucous… “what are you doing about widening x road”?
The questions are specific to individual situations… lots of side conversations starting at the back of the room… some people starting to leave. Some folks getting annoyed. This is fun stuff!
Tough questions now, and the developer folks are trying to keep the conversation civil. Lots of arguments about projected traffic numbers.
UPDATE:

So I talked to some of the attendees afterwards and they said their complaint was that the developer plans to build way too many condos and time shares in Crystal Bay. The resulting project would increase the population of the town by about 4X, cause increased pressure on traffic, fire and pubic resources and be potentially detrimental to Lake Tahoe. Also their studies were curiously out of date. I have not talked to the developers yet, so this is a slightly one-sided viewpoint, but after hearing the presentation, I’d be extremely nervous if I lived there. Here’s a link to the petition if anyone wants their voice heard…


Nevada Town Hall Politics (Live-blogging) – Part 2
Wed, 19 Mar 2008 20:07:16 -0700
 


Boulder Bay town hall mtg
 
 Originally uploaded by salimismail
 

Second speaker is up… mostly talking about road changes

“… we’re putting in guard rails where necessary…”

“… heh heh.. well, that’s it…”

“… if there’s a wildfire, folks can use our parking lots…”
7.50pm – Third speaker, who’s a “traffic engineer” – now talking through the circulation plan. Claiming the traffic is actually declining in the area. Folks are mostly sporting pursed lips…. especially as he’s now estimating 0.5% traffic growth per year for the next 20 years.





Nevada Town Hall Politics (Live-blogging) – Part 1
Wed, 19 Mar 2008 19:38:01 -0700
Scene-setting: This is my first ever live-blogging effort. I’m in a town hall-type meeting in Boulder Bay, Nevada (on the shore of Lake Tahoe).
The topic is a proposal to tear down the Tahoe Biltmore, a lovely old hotel, and replace it with a 400-unit condo complex. Local residents are not happy, and about 250 of them are in this room listening with some suspicion. Apparently there was an earlier presentation in November which seems not to have gone down very well, especially around traffic congestion.
I’ve never witnessed small-town politics before, so this is new and fun stuff for me.
7.15pm – the developers are nervously starting a powerpoint presentation… sample quotes: “…we know a lot of you are worried about traffic…”

“… this is a work in progress and will continue to evolve… ”

“… tourist-friendly wellness center… ”

“… these projects have a rumor mill… ”
7.25pm – he’s now going through a set of slides diagramming what it’ll look like… lots of acronyms around project approvals
“… we are at the very early stages of this process… it’ll be a long time before anything gets decided… ”
On to the next speaker… 


Unix Permissions Calculator Released
Mon, 02 Mar 2009 06:55:12 -0800
Over at Project Fondue we've just released a new tool - Permissions Calculator. If you always had trouble understanding how Unix file permissions worked then this is the tool for you.
It allows you to specify the permissions you need by ticking checkboxes and it then generates the correct octal code for use with chmod. Alternatively supply it with an octal code and it'll provide a breakdown of what permissions that code corresponds to. The URL of the breakdown is bookmarkable making it easy to share with other people or to cross link from online documentation.
Additionally the tool can generate the symbolic values needed to added permissions to an existing, base permissions set.


Favicon Generator & Editor Released
Mon, 09 Feb 2009 18:18:54 -0800
I've been working on several projects with a couple of former colleagues (Stuart Colville & Cyril Doussin) and we've just released the first of these - a favicon generator. The CSS Sprite Generator tool we released back in 2007 has also had a new lick of paint and we've set up Project Fondue to promote our work so check back often for new tools.
Here's an extract from the about page text from the Favicon Generator which explains exactly what it does.


This tool provides an easy way to convert any GIF, PNG or JPEG to ICO which is supported by all modern web browsers. It also enables you to create favicons from scratch via a handy online editor. Additionally the editor lets you manually tweak generated favicons to ensure the best possible result.




Inbound Link Analyser - a tool for analysing links to your site
Tue, 14 Oct 2008 06:23:39 -0700
Eric Migralia, of Yahoo! YUI fame, has just released a tool, Inbound Link Analyser, for listing and analysing inbound links to any site. It's utilises YUI extensively and, I'm sure, provides a fantastic example of how to use a number of the core YUI components. I strongly recommend a browse around the source. Data for the tool is taken from Yahoo!'s Site Explorer API and del.icio.us.

So far the tool has helped me unearth plenty of inbound links I hadn't discovered through the usual suspects (Google Analytics, Feedburner etc) so given how much I like trawling through this stuff I'm guessing I've got plenty of fun filled browsing ahead of me.
The interface also lists other pages that have been indexed for the domain entered. Clicking on these or any of the inbound links opens separate tabs within the interface for you to analyse those resources further.
Eric's written a short introduction over on his blog (also newly released) so I'd suggest heading over there and checking that out first.
Here's a link to an analysis of my CSS Sprite Generation tool. It provides plenty of data to start exploring the tool with.


Vote for my SXSW 2009 panel proposal
Thu, 14 Aug 2008 08:26:58 -0700
Stuart Coville, Nicole Sullivan, Stoyan Stefanov and myself have put forward a panel proposal for SXSW 2009 - Practical Website Perfomance. We'll be covering a number of frontend techniques aimed at improving site performance, discussing how the way browsers work affects performance and generally discussing what we've learned about developing high traffic websites.
If this interests you please take a moment and head over to the SXSW panel picker to vote for our panel.


Source code for version 2 of CSS Sprite Generator
Sun, 13 Jul 2008 03:41:10 -0700
Stuart and I have just released the source code for the latest version of our CSS Sprite Generator. As mentioned in my previous post the code has undergone a major refactoring and support for Image Magick has been added which greatly improves the quality of the images produced. Additionally there are options for tweaking the number of colours used in the output image, the image quality and for compressing images through OptiPNG.
Download a tar.gz of the source or check out the Bazaar branch.


Version 2 of CSS Sprite Generator released
Mon, 23 Jun 2008 05:42:06 -0700
Stuart and I have just released a new version of our CSS Sprite Generator. We've given it quite a major overhaul and it now uses Image Magick to generate sprite images instead of GD. This produces much higher quality images and fixes all of the related bugs filed. Additionally we've added options to control the number of colours used in the sprite image for PNGs and GIFs and the image quality for JPEGs. Finally we've added an option to pass sprite images through OptiPNG to compress the output when using PNGs. This frequently results in at least a 50% decrease in file size and should save trips to Image Ready to post optimise your images.
Behind the scenes we've completely restructured the code and heavily commented it to make it easier to understand. We'll be updating Launchpad with this latest version of the code in the next few days. Stay tuned.
The addition of these features has meant we're missing a number of translations. If you think you can help us fill in the gaps for your language please drop us a line or visit our translation help page for more information.
As you can imagine CSS Sprite Generator is very CPU intensive and therefore requires a fairly high quality hosting environment. To help support future development and hosting costs we're now selling a couple of small graphical button ads along side the text links we've sold on the site up until now. We've tried to ensure these have been added as unobtrusively as possible so hopefully they won't impact your experience whilst using the tool. If you're interested in advertising we've got a page which gives all the details of the options available.
For the future we're mulling over a number of ideas for new features. One in particular is the addition of a REST based API (returning JSON and serialised PHP). We'd love to hear what you think about this and your thoughts on what else we should add to the tool going forward.


Opera releases Dragonfly for Opera 9.5 beta 2
Tue, 06 May 2008 06:46:11 -0700
Opera has released Dragonfly, their Firebug like developer console, as part of the latest release Opera 9.5 beta 2. It looks pretty promising - at a quick glance I think it compares favourably with the equivalent tool in Safari 3.
The initial version includes a JavaScript Debugger, a DOM Inspector, CSS Inspector, error console and a command line and, although it doesn't currently offer editing capabilities, they'll follow shortly as well as the ability to inspect XHR/HTTP headers.
I used Opera as my main browser for a couple of months at the start of the year but found myself reluctantly returning to Firefox as I was having to run it almost all the time anyway for access to the developer tools. The addition of editing capabilities to Dragonfly will likely see me return to Opera as I found it much snappier and memory usage over time much lower.
More information and download details available from the official Dragonfly web site. Chris Mills has also posted a detailed walk through explaining how to use the tool on the Opera Developer site.


Mobile Browser Concurrency Test
Mon, 28 Apr 2008 07:52:44 -0700
Cloud Four, specialists in mobile development, have set up a research project aimed at finding out more about the capabilities of mobile devices to determine the performance implications of building web sites for the wide variety of devices available. Inspired by desktop browser research, such as that carried out by Yahoo!'s Exceptional Performance Team, they're particularly interested in finding out the following information:
 What is the number of concurrent http connections that the mobile browser supports both per domain and overall?
 Does the mobile browser support gzip or other methods for reducing the size of pages?
 Does the mobile browser support caching if you set the Expires header far into the future?

They plan to publish the information under a creative commons license to allow mobile web developers to benefit from their findings.
You can help them by running their test suite from your mobile phone's web browser.


Problems building MySQLdb on Mac OS X 10.4 (Power PC)
Wed, 16 Apr 2008 20:00:07 -0700
Over the last couple of days I've been trying to build and install MySQLdb on my PowerPC based Mac Mini with very little success. I got the following message followed by a ton of type errors:
Compiling with an SDK that doesn't seem to exist: /Developer/SDKs/MacOSX10.4u.sdk
Please check your Xcode installation

I tried reinstalling Xcode Tools to see if it would solve the problem but with no success. Anyway after searching around it turns out that the installer doesn't add a required SDK package which you can find in Xcode Tools->Packages on the install disk. It's called MacOSX10.4.Universal.pkg.
I (and others) have not had similar problems on Intel based Macs running Tiger or Leopard so I'm guessing it's specific to the PPC install.
With that out of the way - on with Django.


Automatic versioning of CSS, JavaScript and Images
Mon, 07 Apr 2008 08:24:00 -0700
See comments for important information about HTTP caching issues with URLs containing query string parameters.
When including CSS and JavaScript resources in your pages you should version file paths and update these version numbers every time the files change. This is necessary as a visitor's browsers may, depending on settings, continue to cache files even after a change. This can result in a mismatch between your HTML and these external resources which may cause rendering or functionality problems. One will likely encounter similar caching issues with images.
 So if we work on the basis that we should version these resources and that changing version numbers forces the browser to reload those resources then we can actually gain large performance improvements by explicitly instructing the browser to cache them for an extended period of time (say 10 years) thereby limiting the number of times the browser goes back to check for fresh copies.
 This can be achieved fairly easily within the virtual host settings of your Apache conf file. Simply add the following directives within the relevant virtual hosts section substituting /assets/ with the file path corresponding to the location where your CSS, JavaScript and images are stored.
  <Location /assets/>
 ExpiresActive On
 ExpiresDefault "access plus 10 years"
 </Location>
 If you have the mod_deflate module installed then you can gain an additional performance benefit by gzipping CSS and JavaScript resources to reduce the amount transmitted down the wire.
  <Location /assets/>
 ExpiresActive On
 ExpiresDefault "access plus 10 years"
 SetOutputFilter DEFLATE
 </Location>
 These directives can also be added to an appropriately placed .htaccess file but as Stuart explains there are performance implications with this method so it should only be chosen if you don't have access to your Apache conf files (perhaps because you're using shared hosting).
 Applying both these settings covers off a couple of recommendations made by Yahoo!'s YSlow performance testing tool so not only will you end up with a faster site but you'll also improve your YSlow rating by a grade or two.
 Of course manually versioning files is a pain, it's repetitive and as Stuart is always reminding me manually repetitive tasks should be eliminated wherever possible. In the past I wrote a script which automatically merges CSS or JavaScript files and versions the combined output. Here's some simpler code which versions individual files based their last modified date - every time the file is updated it's version number updates accordingly.
  <?php
 class Version {
 private static $aLookup = array();
 public static function Get($sFilename) {
 if (!array_key_exists($sFilename, self::$aLookup)) {
 $sRealPath = realpath($_SERVER['DOCUMENT_ROOT'] . "/$sFilename");
 if (file_exists($sRealPath) && ($iTimestamp = filemtime($sRealPath))) {
 self::$aLookup[$sFilename] = $iTimestamp;
 $sFilename .= "?v=$iTimestamp";
 }
 } else {
 $sFilename .= '?v='.self::$aLookup[$sFilename];
 }
 return $sFilename;
 }
 public static function GetLink($sFilename) {
 return '<link rel="stylesheet" type="text/css" href="'.self::Get($sFilename).'">';
 }
 public static function GetScript($sFilename) {
 return '<script type="text/javascript" src="'.self::Get($sFilename).'"></script>';
 }
 public static function GetImage($sFilename, $iWidth, $iHeight, $sAlt = '') {
 return sprintf('<img src="%s" width="%d" height="%d" alt="%s">', self::Get($sFilename), $iWidth, $iHeight, $sAlt);
 }
 }
 ?>
 Download plain text version (Updated version to fix HTTP caching issues)
 Using it is pretty simple - wrap the required function around the file path you'd normally have supplied to link, script or img tags.
  <link rel="stylesheet" type="text/css" href="<?php echo Version::Get('/css/dark.css'); ?>">
 In fact it's even easier than that - I've also provided some wrapper functions which make outputting link, script and img tags a little bit easier. The example above could be written more simply as follows:
  <?php echo Version::GetLink('/css/dark.css'); ?>



Collide-O-Scope
Fri, 04 Sep 2009 21:43:00 -0700

Here's a fun little vacation project, a canvas-powered Collide-O-Scope.

 
After being absolutely floored by some really impressive new HTML5 demos--see the 9elements Twitter mash-up, for one--I figured it was time to give it a try myself. Here's the result.
Assuming you're running a canvas-aware browser--see compatibility notes below-- you should be seeing several moving sets of randomly-generated circles, all projected with their seven possible translations and reflections. Each set has an X and Y vector; before each move we're running some very crude collision detection and altering those vectors if needed.
While it's running, you can change things around by clicking anywhere inside the canvas. This will remove the oldest set of circles and add a new one where you click.
The fuzzy/translucent effect comes from building up a number of very faint circles, on on top of the other. Depending on the colors, you'll see fuzzy blobs, sharp stained-glass sections, and other interesting stuff when two sets of circles overlap.
At the moment I'm seeing smoothest results under Firefox in OSX, closely followed by Safari, with Chrome in last place. All three--and Opera 10--also run in Windows, but since I'm running Parallels I can't say how they compare to OSX, performance-wise.
Sorry, no love for IE; I did try excanvas, but it failed right away and I'm afraid I'm just not committed enough to caring about IE to investigate. (Hobby/pretty/vacation project, remember?)
Try It Out
If you'd like to try Collide-O-Scope, save this as an HTML file on your local directory and drag it into your browser:
<!DOCTYPE HTML>
<html>
<head>
<title>Collide-O-Scope</title>
<style>
body { margin:0; padding:0; background-color:#666; text-align:center;
}
canvas { margin:50px auto 0; border:2px solid #ffa;
}
</style>
</head>
<body>
<script src="http://kentbrewster.com/collide-o-scope/collide.js">
{"height":400, "width":400}
</script>
</body>
</html>

Height and width go inside the <SCRIPT> tag in curly brackets; for more on the techniques, please see any of my articles about Case-Hardened JavaScript.
Other valid configuration variables may be discovered by viewing the script and looking in the houseKeep function; you can change things like the number of circles on the page, the maximum number of rings per circle, ring thickness, ring opacity, background opacity, and canvas background color.
As always, have fun with this, don't leech the script for production use on your site, and please let me know how it goes.


Fixing Twitter Replies
Wed, 27 May 2009 21:02:00 -0700

Hidden Tweets



A short while ago, something sucked the fun right out of Twitter. All of a sudden, replies from people who I was following to people who were a) not me, or b) not people I was also following, mysteriously quit showing up. Without knowing more about the underlying data structures I'm not certain I understand Twitter's explanation; it seems to me they'd have to go out of their way to find and remove tweets from people I'm following that were in reply to tweets from others that I'm not following, rather than just show me all the tweets from everybody I am following.
Anyway. I figured I'd go with it for a while and see if I noticed any difference ... and ... yeah, I do. Turns out the main way I found new, interesting people to follow was through those half-conversations. I haven't followed anybody new since replies broke, and find myself growing increasingly bored with Twitter.
Enter the Hack
At the top of this page, you'll see an entry box with my Twitter id filled in. If you click the button, you will see a list of tweets from people I'm following. Each of these tweets is the most recent one posted in reply to another Twitter user. If I'm not following that person, I won't see it when I look at my Twitter page. Right, some of these are false positives, for reasons mentioned below. I could clean it up, but I'm probably not going to do it. It's a hack, people. :)
If you'd like to try it out and see some (probably not all) of what you're missing, fill in your Twitter screen name and click Go.
What This Won't Do

Show all your friends. I'm calling the friends/status API five times to get the most current status from the last 500 users you followed; that's enough to show some of what you're missing. (Hey, while I have your ear ... please take a moment to think about the extra carbon you're causing Twitter to release into the atmosphere with all those follows you never actually pay attention to. 500 ought to be enough!)
Show all the tweets you're missing. Sorry, it's only looking at the current status and reporting it as possibly hidden if it's listed as in-reply-to another Twitter user.
Hide the tweets you're not missing. Again, sorry: since I'm only looking at the last 500 people you followed, I have no way of knowing what's in reply to the other 6000. (Again, please ... thin that ridiculous herd of follows, and help save Al3x's sanity.)
Work on your site with a single line of JavaScript. Sorry, this is not a case-hardened badge. (It might make an interesting Greasemonkey plug-in ... but somebody else gets to write it. Please keep in mind that you will quickly use up your 100 API calls in an hour if you bang on it too hard.)

Please Re-Tweet! :)
Twitter is much less useful to me without replies to friends I haven't met yet, so I really do hope they can fix it. Have fun with this, and if it's useful, kindly tweet it up on Twitter. Thanks!

 ";
echo $source;
echo " ";
include_once('/home/kentbrew/public_html/inc/footer.inc'); ?>


Twitter Search Badge
Fri, 08 May 2009 09:46:00 -0700

Twitter Search is turning out to be a very powerful force in real-time search. Instead of relying on a vaguely-defined, possibly-corrupt, almost-always-out-of-date algorhythm owned by somebody else, why not check out what your fellow Internet readers are marking as interesting and/or relevant? Here are three instances of the same search prototype, two with text queries and one with a from: query, showing tweets from a single person.


 


 


 


If you'd like to try the Twitter Seach Badge on your site, include this bit of JavaScript wherever you want it to pop up:
<script src="http://r8ar.com/tsb.js"></script>

The usual batch of configuration variables--height, width, etc--are available; to show a default search, do something like this:
<script src="http://r8ar.com/tsb.js">{"query":"obama"}</script>
<script src="http://r8ar.com/tsb.js">{"query":"from:scobleizer"}</script>

Right, obviously I need to combine this with Twitterati. I'm just wishing the Twitter Search API had a way to come up with the most recent tweets by friends of username. Will probably have to munge something together with Pipes to do that.
Also still to come: Get More and Go Back links at the bottom.
Here's my talk from the May 9th BayJax meetup, tracing the development process of this badge, which I wrote in a single afternoon. An older but much more detailed version of this presentation is online here, under Case-Hardened Web Badges: The Live Version.
 
Start Presentation in New Window
If you'd like me to present this, or something like it, or something completely unlike it (like Ajax security or OAuth or rolling your own API with Pipes) at your conference, please contact me and I'll see what I can do. As always, have fun and please let me know how it goes.


Netflix Catalog API Explorer
Tue, 28 Apr 2009 17:13:00 -0700
Netflix, in case you didn't know, is a marvelous place to rent movies and television series on DVD, and watch an increasing percentage online. Over the last ten years, Netflix has accumulated a treasure trove of information about movies and the people who make them, and has recently opened it all up with the Netflix Catalog API. (Full disclosure: yes, I work there, and yes, it's awesome.)
 Here's a handy tool to help you get your feet wet without having to worry about OAuth.
 Before You Begin
 You'll need a developer account, from developer.netflix.com. Once you're signed up and logged in, the link to Consumer Key--below, to the left of the first text entry box--should take you right to the spot where you can copy your key and secret.
 If you're feeling paranoid, please view source before clicking the Onwards! button. This thing just below that looks like a form really isn't a form, and is not going to submit your information anywhere except the Netflix API. The only reason why I'm using an all-client solution is so you, the developer, can actually try out your very own consumer key and shared secret, without me, the potentially-nefarious third party, ever actually knowing what they are.
 Try Your Queries Right Here
  Consumer Key: 
 Shared Secret: 
 Query: 
 Got your consumer key and shared secret filled in? Click me: 
  Your OAuth-Signed URL
 
 Sample Queries (Click to Try)
  Autocomplete search results for the string "mcdo":
http://api.netflix.com/catalog/titles/autocomplete?term=mcdo
 Find the first Frances in the People section:
http://api.netflix.com/catalog/people?term=Frances&max_results=1
 Zoom in on Frances McDormand:
http://api.netflix.com/catalog/people/61544
 See Franceses 2 through 10:
http://api.netflix.com/catalog/people?term=Frances&start_index=1&max_results=9
 Frances McDormand's filmography, with synopses:
http://api.netflix.com/catalog/people/61544/filmography?expand=synopsis
 Zoom in on Fargo:
http://api.netflix.com/catalog/titles/movies/493387
 Fargo again, with expanded cast and synopsis:
http://api.netflix.com/catalog/titles/movies/493387?expand=cast,synopsis
 
 These are only a few examples; please see the REST API Reference for detailed information.
 Things to Do and Notice
  When your results come back, just about any link that starts with http://api.netflix.com/catalog can be copied and pasted right back into the query blank for further exploration.
 A link to your OAuth-signed URL will magically appear under the iframe containing your results. If you click this it ought to open your results in a new page, so you can see exactly what OAuth did to your request string. This is very, very handy when you're wrestling with some of the more arcane aspects of the API, such as what needs to be URL-encoded.
 Use Netflix's handy expand parameter to avoid extra calls to the API. Any of the link attributes in the main tree should be expandable; details are in the API docs, which I am currently in the process of refactoring.
 To try out version 1.5 of the Netflix API, add v=1.5 to any query. Please note that expand behaves differently in version 1.5.
 
 Don't Forget to Break It!
 No, seriously. This is critical; you need to know what happens when things go wrong. Change your key or secret, or enter an URL that doesn't compute, and make note of the errors you get back.
 Bad Ideas
  Trying to retrieve the entire catalog. (Okay, I admit it: I tried this myself. The resulting bolus of data choked my browser, and I had to reboot and recover some lost stuff that I hadn't saved.) Client apps should never need the entire catalog; if you are thinking about doing this, you are doing it Wrong.
 Using term search to power autocomplete. That's what /catalog/titles/autocomplete is for, folks; it's even been left free of OAuth requirements so people banging on it won't use up your daily quota of API calls.
 Putting this script into production anywhere. Although OAuth calls can be made using nothing but JavaScript, doing so will reveal your consumer key and shared secret on every call, which could lead to trouble if either item is hijacked and used by someone who is not you.
 
 Other Important Resources
  If you'd like to play around with OAuth parameters, see the OAuth Test Page.
 To run signed-in calls to the Netflix API--accessing things like a specific user's queue--you'll want Flixo.
 Although it feels very GOOG-centric, the OAuth Playground has a spot under "Choose Your Scope" where you can enter any OAuth endpoint. Netflix works there.
 
 The JavaScript Source
 For the morbidly curious, here's how I got client-side OAuth to work in eighty lines of JavaScript. It may be worth looking at if you're a throwback like me who refuses to use a library he doesn't understand. The heavy lifting happens at the bottom of the script, from oAuthEscape on down, with much help from Paul Johnston's HMAC-SHA1 routine.

 
 If you'd like to learn more about the various snakes that bit me while I was figuring it out, please see True OAuth Confessions, or Why My Hand-Rolled Calls All Blew Chunks.
 Have fun, please let me know how it goes, and don't forget to read the documentation, at http://developer.netflix.com/. It will be better soon, I promise! :)

 ";
echo $source;
echo " ";
include_once('/home/kentbrew/public_html/inc/footer.inc'); ?>


True OAuth Confessions
Mon, 27 Apr 2009 15:08:00 -0700
I'm settling in at Netflix and starting to work my way through the developer docs. One of the main concerns we're seeing from outside developers is getting that first OAuth call to work. Here, in an effort to reassure my fellow noobs that they're not crazy, is my tale of OAuth woe:
When I first encountered OAuth I bounced off the spec and a couple of libraries, which seem to be documented in a language created specifically to make guys like me give up in disgust.
Backup plan: I cracked into a couple of URL returns with Firebug, thought I understood what was going on, and plunged blindly in. (Famous last words: "It's just a string, right? How hard could it possibly be?")
My real mistake: not just blindly trusting the libraries to do the right thing. Seriously. Unless you've got the same thing wrong with you that I have with me and you absolutely cannot stand the idea of using something you don't fully understand, stop here and just go plug in a library.
So. Here begins my top ten list of horrible mistakes made while trying to reverse-engineer OAuth:

My Timestamp was Stale

Unless it's only a few minutes old, the URL somebody else generated isn't going to work. And forget about trying to copy and paste the one they show in the docs. If you're working in JavaScript, your client's clock needs to be within ten minutes of your API's clock. (WinXP under Parallels seems to lose track of time every once in a while; cue the sound of Nelson Muntz from The Simpsons: "Ha-hah!")
The Parameters in my Signature Base String were Out of Alphabetical Order

Note to self: oauth_signature_method comes BEFORE oauth_timestamp, and AFTER oauth_nonce. Parameters like foo and baz and qux might show up before or afterwards. Or during, depending on how psychotic the API you're trying to hit turns out to be.
I Forgot a Question-Mark between my Request and my OAuth Payload

Here's where you do NOT want an ampersand. You'll go blind trying to spot it.
I Didn't URL-Encode my Signature

Just because it came back in base-64 from my HMAC-SHA1 function didn't make it safe to fire off; there are plus-signs and equals-signs and other unsavory characters in many base-64 strings, and they all needed to be encoded.
I Didn't URL-Encode All Illegal Characters

Right, see, I was initially doing this with JavaScript, and uriEncodeComponent turned out not to fix exclamation points, asterisks, single-quotes, or parentheses.
I URL-Encoded the Ampersands Separating my Method, URL, and Request Parameters

The server needs those raw ampersands, so it can tell where to split the string you sent.
I Didn't Append an Ampersand to my Consumer Secret to Make my Signature Key

Another picky ampersand-related detail: when signing a request without a token secret I STILL needed to add an ampersand to the end of my consumer secret.
I Used the Same Nonce, Over and Over and Over Again

Turns out there is a special circle of Error 401 Hell reserved for people who re-use their nonces; this makes sense, if you think about the sort of replay attacks the spec is designed to prevent. Naturally you would need to READ the spec before thinking about this.
I Generated a Random Nonce, but (you guessed it) Failed to URL-Encode It

No need to be fancy with this; just pick a random 16-digit number. Don't (for instance) use every possible printable character.
And, Finally: I URL-Encoded my Signature Key

Yeah, I really did do this. It took days to unscrew. Not recommended.

Anyone else have something to confess? Let's hear it; it will be good for your soul, and will help our fellow pilgrims not to fall down the same holes we did.


Amazon Wish Lists Are Dreadfully Insecure
Wed, 25 Mar 2009 08:26:00 -0700
Updated 3/27/2009, 10:01PDT: looks like this may have been fixed a few minutes ago. Read on, to learn more about what was happening.
Old friends may remember the How to Tell if a User is Signed In to Service X series, which ended last year around this time. As you can see from the comments in Patching Privacy Leaks, I advised users to sign out of Amazon.com on 17 October 2008, but did not say why.
Six months and multiple warnings later, nothing's been done. So here it is:

If you are signed in to the United States version of Amazon.com and have a wish list, the button should add an item. You'll see an alert with a success or failure message, and then this paragraph will change to tell you what happened and where to go to see it. If you're using Firefox or IE, we will be able to determine your Amazon login status, by watching onError. If all else fails, we will assume after a few seconds of inactivity that something went wrong.
I'm not sure what will happen if you have multiple lists or if you delete your wish list. Taking your wish list private will make it invisible to the bit that shows your name, but will NOT stop the item add from happening.
How It Works
By examining the source of Amazon's Universal Wish List toolbar bookmarklet, we find something suspicious: an HTTP GET that seems to modify data on behalf of the signed-in Amazon user. This is trouble, since Amazon is depending only on browser cookies to verify user identity. Anyone can create an URL, like this:
http://www.amazon.com/gp/wishlist/add/ref=wl_bm-add
?submit=1&operation=add&mode=JS&priceInput=&id=
&imageUrl.0=http%3A%2F%2Fi2.ytimg.com%2Fvi%2FE62DXiL_8Vs%2Fdefault.jpg
&name.0=Raccoon%20Party
&itemComment.0=amazon%20wishlists%20are%20dreadfully%20insecure
&productUrl.0=http%3A%2F%2Fwww.youtube.com%2Fwatch%21v%3eDeQ1DN7n2Eg

... and fire it off on behalf of the signed-in user. Here I'm being polite and requiring the user to click a button, but it would be trivial to list it as the SRC attribute of a SCRIPT or IMG tag.
Data Returned from Amazon
AUWLBook.results('SUCC', 'Wish List',
'http://www.amazon.com/registry/wishlist/JWMG6ATT26YQ/ref=wl_bm-view-list');
This is very helpful: it gives you back a SUCC or FAIL message, the title of the victim's wish list, and an URL pointing back to it, whether or not the victim has market his list as private. If the list is public, determining the victim's name is trivial; running the wish list through YQL and trimming the result with Pipes spits it right out.
I'm not going to go into great detail about how the JavaScript behind the exploit works; it's pretty self-explanatory. Feel free to view source and poke at it yourself, if you like.
Why It's Scary

I can think of lots of evil things to do with this, from the crude (add a bunch of porno links to your victim's wish list and send it to his boss or his wife or his mother) to the obvious (causing any visitor to my site to instantly wish for my book, over and over and over again) to subtle things like gray-hat SEO, in which links quietly stack up in thousands of public wish lists and change search engine results.
Possibilities for clandestine data mining--many people add their birthdays and home locations to their wish lists--abound. Once I know your name, home town, and birthday, I have a fair number of the items most commonly used as security questions for password recovery.
Although I haven't documented it here, there is at least one link on the user's wish list that will save a change to its configuration in response to a GET, and not just add an item. This is very, very bad.

The bottom line: Amazon never told their users that wish lists are actually online presence indicators and can be used invisibly to gather data about them wherever they go.
Lessons Learned

Never use GET to modify data on behalf of the user.
Don't give back live, executable JavaScript wrapped in easily-intercepted callbacks.
Returning HTML to signed-out users and JSON to signed-in users allows curious operators to determine the user's sign-in status by watching window.onerror.
Shipping a toolbar bookmarklet that changes data on behalf of a signed-in user is probably not a great idea; now that it's broken, everybody who's installed it will have to either re-install it or quit using it.
Have a clearly marked path for reporting security holes, listen, and be extremely responsive!

Thank You

Last-minute testers: Norm, Cyn, Ryan.
Those guys who do the thing in that place I used to work, who tried multiple times to get Amazon to fix this. They are heroes; I wish people would listen harder when they speak up.



Bad JavaScript Advice Filter
Thu, 05 Mar 2009 15:55:00 -0800
When somebody comes to me and says "I want to learn HTML," I always ask "Why?"
 Don't get me wrong; unlike writing, I don't believe that anyone who can be discouraged from coding should be. Web programming is a fine thing to know, but it's very hard to learn just for the sake of learning it. Big topic, lots of pitfalls, you get the idea. So if I'm talking to someone who just has a vague idea that it would be cool to know how to do this stuff, I make vague comforting noises and try not to offer advice.
But if the person talking to me pops back with "I want to build a Web site for Specific Reason X," we're off and running. It's much easier to learn the tiny subset of knowledge necessary to get your specific project off the ground than it is to gain general background knowledge enough to do anything you want, at some distant point in the future.
After representing Yahoo! at several University Hack Days and watching superstar Monkey Wrangler Paul Tarjan--fresh out of Stanford, first real job, builds a game-changing search app for Yahoo, quick, eat him now while his bones are still soft--I was fascinated. But I didn't jump straight in with Searchmonkey, because I couldn't think of anything interesting to do with it.
That changed this morning, when I was poking around looking for some embarassingly basic JavaScript advice. (I'm lazy, and ignorant, and figure "why waste space in my brain if the cloud is available to store all that knowledge?")
News flash: there is a crapload of really bad information out there. It's a) old, b) misguided, and c) wrapped in banner ads or sign-in-to-see-the-answer bait-and-switch gotchas. Unfortunately, it's been there a long time, and has many supporting links and other SEO-fu around it, so it tends to sort to the top.
I click into this junk a lot, and it bothers me.
Enter the Monkey.
With a simple set of URL filters, I was able to build bad.js, a Searchmonkey add-on for Yahoo! Search that warns me away from all those lovely canned scripts and talking banner ads from 1997. Bad.js was simple to build, easy to publish, and generally a joy to do; if you want to play, I strongly recommend checking out Searchmonkey, at http://gallery.search.yahoo.com.

Suggestions for other URLs will be gratefully accepted; please leave them on the Searchmonkey app's Comments tab, so I can find them all in one place. (Lazy, remember?)