All Yang's Subscriptions

Facebook Tips: How do I find and "fan" a Page?

Morin Cynthia Oluwole — Mon, 23 Nov 2009 11:11:05 -0800

Type the name of the celebrity, artist, brand or organization in the Search box in the upper-right corner of any page on Facebook. From the results page, select the the Pages filter in the left-side menu to show only Page results. Click the "Become a Fan" link next to the Page name and posts from that Page will begin appearing on your home page.

If you already know the web address, or URL, of the Page you want, simply type it in the address bar of your web browser. For instance, to find Vin Diesel on Facebook, you can navigate directly to www.facebook.com/VinDiesel. From there, select the "Become a Fan" button next to the Page's name.

You also can become a fan of a Page that you discover from your friends' recent connections. For example, if you see a News Feed story about your friend becoming a fan of the iTunes Page, you can connect with it by clicking the "Become a Fan" link within the story or click on the name of the Page to check it out further.

Once you're a fan of a Page, it is listed on your profile and is included in the list of Pages you're a fan of that's viewable to "everyone" unless you have changed your "Search" privacy settings.

Remember that you always have the option to remove yourself as a fan of any Page if you want to stop receiving updates or no longer want to be connected. Go to the Page and click on the link "Remove me from Fans" at the bottom of the left-side menu.

Check out the nearly 1.4 million active Pages on Facebook at www.facebook.com/pages to start connecting and interacting with your favorite celebrities, political leaders, media outlets, non-profit organizations, TV shows, musicians, and much more.

Morin, who works on Facebook's user operations team specializing in Pages, just became a fan of Ibiss Boutique.

The Mass Animation of a Hollywood Short

Yair Landau — Fri, 20 Nov 2009 10:10:12 -0800

Yair Landau is the former president of Sony Pictures Digital and the director of a new animated short opening today called "Live Music." We asked Landau to share the story behind Live Music's creation as the first production from Mass Animation, a worldwide crowdsourcing of animation on Facebook done in partnership with Intel, Dell, Autodesk and Reel FX Entertainment.

When moviegoers sit down in select theaters tonight to see the opening of the new animated feature "Planet 51," they first will be greeted with a 5-minute, computer-animated short that resulted from the collaboration on Facebook of 58,000 people from 101 countries. Set in a musical instrument shop, the short "Live Music " tells the story of Riff, a rock 'n' roll guitar, who falls in love to the wrong song but ends up with Vanessa, the classical violin of his dreams.

What makes this short unique is that it is the first result from an ambitious project called Mass Animation to collaboratively create animation on Facebook. Started in August 2008, Mass Animation enabled animators from around the globe to join the creative process from anywhere. They could animate with tools and models provided through Mass Animation, submit and view them through an application on Facebook and vote on the best animated shots. In the end, 51 animators' shots were selected for inclusion in "Live Music."

As someone who's been involved in moviemaking for 19 years, I was passionate about Mass Animation because I realized it could open the animation process to a wider array of creative people. The premiere of "Live Music" marks a milestone for animated film because it shows that you can have an impact on the creative process no matter where you live in the world. Social media—and specifically Facebook in this project—enabled us to connect with animators we never would have reached before.

As participating animator Cyrus Cords put it, "In the middle of Minnesota opportunities like this don't happen."

The selected animators hailed form 17 different countries—ranging from Australia, Brazil and Russia to India, Kazakhstan and the U.S. —were between the ages of 14 and 48, and were composed of over 20 percent women.

Animator Gregory Naud of Belgium said he found the experience motivating. "It's cool to be part of a big international project," he said. "For those who don't have the chance to work in USA, for example, it was a great opportunity."

I hope you'll be as inspired by the short as we were in creating it through mass collaboration on Facebook. Take your own peak behind-the-scenes of its production by checking out the Mass Animation Facebook Page and watching the video below. Don't forget to download "Live Music" today from iTunes and for your mobile from Myxer to watch the full short.

Yair is stoked about the release of "Live Music" and grateful to all the talented people who made it happen.

Facebook Engineering Bootcamp

Facebook Engineering — Thu, 19 Nov 2009 16:51:53 -0800

Optimizing asset bundling and serving with Rails

kneath — Thu, 19 Nov 2009 12:30:47 -0800

We spend a lot of time optimizing the front end experience at GitHub. With that said, our asset (css, javascript, images) packaging and serving has evolved to be the best setup I've seen out of any web application I've worked on in my life.

Originally, I was going to package what we have up into a plugin, but realized that much of our asset packaging is specific our particular app architecture and choice of deployment strategy. If you haven't read up on our deployment recipe read it now. I cannot stress enough how awesome it is to have 14 second no downtime deploys. In any case, you can find the relevant asset bundling code in this gist

Benefits of our asset bundling

Users never have to wait while the server generates bundle caches, ever. With default Rails bundling, each time you deploy, each request until your server generates the bundle has to wait for the bundle to finish. This makes your site pause for about 30s after each deploy.
We can use slower asset minifiers (such as YUI or Google Closure) without consequence to our users.
Adding new stylesheets or javascripts is as easy as creating the file. No need to worry about including a new file in every layout file.
Because we base our ASSET_ID off our git modified date, we can deploy code updates without forcing users to lose their css/js cache.
We take full advantage of image caching with SSL while eliminating the unauthenticated mixed content warnings some browsers throw.

Our asset bundling is comprised of several different pieces:

A particular css & js file structure
Rails helpers to include css & js bundles in production and the corresponding files in development.
A rake task to bundle and minify css & javascript as well as the accompanying changes to deploy.rb to make it happen on deploy
Tweaks to our Rails environment to use smart ASSET_ID and asset servers

CSS & JS file layout

Our file layout for CSS & JS is detailed in the README for Javascript, but roughly resembles something like this:

public/javascripts
|-- README.md
|-- admin
| |-- date.js
| `-- datePicker.js
|-- common
| |-- application.js
| |-- jquery.facebox.js
| `-- jquery.relatize_date.js
|-- dev
| |-- jquery-1.3.2.js
| `-- jquery-ui-1.5.3.js
|-- gist
| `-- application.js
|-- github
| |-- _plugins
| | |-- jquery.autocomplete.js
| | `-- jquery.truncate.js
| |-- application.js
| |-- blob.js
| |-- commit.js
`-- rogue |-- farbtastic.js |-- iui.js `-- s3_upload.js

I like this layout because:

It allows me to namespace specific files to specific layouts (gist, github.com, iPhone, admin-only layouts, etc) and share files between apps (common).
I can lay out files however I want within each of these namespaces, and reorganize them at will.

Some might say that relying on including everything is bad practice -- but remember that web-based javascript is almost exclusively onDOMReady or later. That means that there is no dependency order problems. If you run into dependency order issues, you're writing javascript wrong.

Rails Helpers

To help with this new bundle strategy, I've created some Rails helpers to replace your standard stylesheet_link_tag and javascript_include_tag. Because of the way we bundle files, it was necessary to use custom helpers. As an added benefit, these helpers are much more robust than the standard Rails helpers.

Here's the code:

Our application.html.erb now looks something like this:

<%= javascript_dev ['jquery-1.3.2', "#{http_protocol}://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js"] %>
<%= javascript_bundle 'common', 'github' %>

This includes jQuery and all javascript files under public/javascripts/common and public/javascripts/github (recursively). Super simple and we probably won't need to change this for a very long time. We just add files to the relevant directories and they get included magically.

For pages that have heavy javascript load, you can still use the regular javascript_include_tag to include these files (we keep them under the public/javascripts/rogue directory).

Bundle rake & deploy tasks

The javascript_bundle and stylesheet_bundle helpers both assume that in production mode, there'll be a corresponding bundle file. Since we are proactively generating these files, you need to create these manually on each deploy.

Throw this into lib/tasks/bundle.rake and the corresponding YUI & Closure jars and then run rake bundle:all to generate your javascript. You can customize this to use the minifying package of your choice.

To make sure this gets run on deploy, you can add this to your deploy.rb:

Tweaks to production.rb

The last step in optimizing your asset bundling for deploys is to tweak your production.rb config file to make asset serving a bit smarter. The relevant bits in our file are:

There's three important things going on here.

First— If you hit a page using SSL, we serve all assets through SSL. If you're on Safari, we send all CSS & images non-ssl since Safari doesn't have a mixed content warning.

It is of note that many people suggest serving CSS & images non-ssl to Firefox. This was good practice when Firefox 2.0 was standard, but now that Firefox 3.0 is standard (and obeys cache-control:public as it should) there is no need for this hack. Firefox does have a mixed content warning (albeit not as prominent as IE), so I choose to use SSL.

Second— We're serving assets out of 4 different servers. This fakes browsers into downloading things faster and is generally good practice.

Third— We're hitting the git repo on the server (note our deployment setup) and getting a sha of the last changes to the public/stylesheets and public/javascripts directory. We use that sha as the ASSET_ID (the bit that gets tacked on after css/js files as ?sha-here).

This means that if we deploy a change that only affects app/application.rb we don't interrupt our user's cache of the javascripts and stylesheets.

Conclusion

What all of this adds up to is that our deploys have almost no frontend consequence unless they intend to (changing css/js). This is huge for a site that does dozens of deploys a day. All browser caches remain the same and there isn't any downtime while we bundle up assets. It also means we're not afraid to deploy changes that may only affect one line of code and some minor feature.

All of this is not to say there isn't room for improvement in our stack. I'm still tracking down some SSL bugs, and always trying to cut down on the total CSS, javascript and image load we deliver on every page.

Multiple file gist improvements

kneath — Wed, 18 Nov 2009 16:11:04 -0800

We've always had the ability to embed multiple file gists:

But today we added the ability to embed specific files in a multi-file gist!

We also added permalinks next to each file name so you can hard link to specific files like so

Enjoy!

Building a Better Photo Uploader

Facebook Engineering — Wed, 18 Nov 2009 20:19:39 -0800

New Privacy Policy Adopted

Michael Richter — Tue, 17 Nov 2009 17:21:11 -0800

On Nov. 5, we wrapped up a week-long notice and comment period for a proposed revision to our privacy policy. This was a continuation of our ongoing effort to run Facebook in an open and transparent way. The goals of the revised policy were to make it more accessible and easier to understand.

We've spent the last week reviewing each and every one of your comments. While a lot of people participated, the total number of people commenting did not reach the threshold of 7,000 that makes a vote necessary according to our Statement of Rights and Responsibilities. Because of this — and the fact that many of the comments were positive — we've decided to adopt the revised policy. We'll be posting it in English, French, Italian, German and Spanish soon.

In the coming weeks and months, we plan to build on the progress we've already made in making the document more accessible by also adding definitions of key terms, screen shots of important pages and informational "learn more" videos. We think these visual resources will make it even easier to understand how privacy works on Facebook.

We received excellent feedback from many of you over the course of the week, and the number of fans for the Facebook Site Governance Page more than doubled. Overall, you supported the proposed changes. For example:

Most of you liked that we removed more complicated technical and legal terms and replaced them with simpler language that everyone can understand.

You preferred the structure of the new document, which provides key points at the beginning and easy links to jump to the different sections.

A few of you asked for a redline version so you could compare the proposed revision to our current policy and see exactly what's changed. We weren't able to provide one this time because the policy was completely rewritten, but we think it's a great idea and are committed to doing it for future proposed changes to our governing documents.

As always, thanks for taking part in this important process. We're glad to have so many interested and engaged people using Facebook. Stay tuned for future updates by becoming a fan of the Facebook Site Governance Page.

Michael, Facebook's deputy general counsel for intellectual property, product and regulatory affairs, is grateful for your feedback.

Xbox Connects with Facebook

Gareth Davis — Tue, 17 Nov 2009 11:54:37 -0800

Games are more fun when played with friends, which is why they are among the most popular activities on Facebook. We're excited that beginning today the social game experience is coming to Xbox Live, so now you can play Xbox games with your Facebook friends.

More and more services are integrating with Facebook to make it easy for you to connect with friends wherever you are — including websites like Digg, devices like the iPhone and desktop applications like iPhoto.

The Xbox Facebook integration allows you to take part in some of the most popular activities on Facebook directly from your living room on your television. With Facebook for Xbox, you can update your status, browse updates from your friends and view photos on the big screen. You also can link your Facebook profile to your Xbox Gamertag to find your Facebook friends and connect with them on Xbox Live. It's never been easier to discover friends you never even knew played Xbox.

Keep in mind that Xbox will respect the privacy you set on Facebook, so you can control who sees your activity. The new Facebook for Xbox can be added to your Xbox 360 through a system update now.

As social gaming continues to grow, we look forward to people playing games with their friends on consoles, mobile phones, the Web and their desktops.

Gareth is on his Xbox viewing recent photos and playing games with his Facebook friends, all from his television.

GitHub Meetup SF: RubyConf Edition

defunkt — Mon, 16 Nov 2009 12:26:25 -0800

In SF for RubyConf? Join us this Thursday the 19th at Zeitgeist (map) starting at 9pm!

Zeitgeist has pitchers and ample outside seating, so bring a jacket. Also: cash only.

Coming from the conference? Take BART to the 16th st Mission stop then walk North on Valencia (map)

See you there!

How to Bullyproof Yourself on Facebook

Richard Allan — Mon, 16 Nov 2009 11:40:51 -0800

In October, we called attention to National Crime Prevention Month in the United States to raise awareness about cyberbullying. This week is National Anti-Bullying Week in the United Kingdom, and we've teamed up with media outlets BBC Radio 1, BBC 1Xtra, and other sites like Bebo, Habbo, MSN, MySpace and YouTube to address this important issue again. While only a small fraction of people will ever experience abusive behavior on Facebook, we're committed to promoting an environment where everyone can connect and share comfortably.

For more information on how to handle bullies, check out the BBC's new Facebook Page and website. You can show your support for the "bullyproof" movement and receive a free badge to post on your Wall or send to a friend.

We want to ensure that people understand how to address cyberbullying if it occurs on Facebook, so here are some useful tips from our Help Center:

Accept Friend Requests from People You Know
In order to prevent harassment from strangers, be careful to accept friend requests only from people you know in real life and remember to report any messages or profiles that look suspicious. Facebook is based on a real-name culture, and fake profiles are regularly disabled when they're reported to us. Only confirmed friends can post to your Wall or contact you via Facebook Chat, so if you're worried that someone will make inappropriate posts or send offensive messages, just ignore that person's friend request.

Use the Block Feature
A block prevents specific people from viewing your profile. When you use this feature, any ties you currently have with them will be broken, and they won't be able to contact you through Facebook. You can block another user by listing his or her name in the "Block People" box at the bottom of your Privacy Settings page.

Report Abusive Behavior
The most efficient way to report abuse is to do it in the same place it occurs on Facebook. For example, if you receive a harassing message in your Inbox from someone who is not your Facebook friend, you can report it by clicking on the "Report" link next to the sender's name as you are reading the message. If you receive an harassing message from a Facebook friend, you should remove that person as a friend and report the message. Reporting a message will automatically add the sender to your Block list. You can also use the "Report/Block person" link that appears at the bottom of the abusive user's profile. Reports are confidential and the user being reported does not know about the report. After a report is submitted, we will investigate the issue and make a determination as to whether or not the content should remain on the site based on our Statement of Rights and Responsibilities.

Customize Your Privacy Settings
Facebook offers granular privacy settings that allow you to customize what information you share and with whom. By customizing the settings on the Privacy Settings page, you can control who views your information, dictate who can find you in searches, and see your Wall, profile and photos, among other things.

Respond to Abusers in the Right Way
Cyberbullies often seek a reaction from the people they harass. When they fail to get one, they often will give up. Rather than respond to a bully via an Inbox message, a Wall post or Facebook Chat, you can delete offensive posts from your Wall or messages from your Inbox and then use the Block or Report functions to resolve the issue. To delete an offensive Wall post, select the "Remove" button that appears in the upper right-hand corner. To delete a message from Inbox, simply click the "Delete" button at the top of the message. Only confirmed friends can post to your Wall or send you a message through Chat. If you are receiving posts and Chat messages you don't like, you should consider removing the sender from your friends list.

Protecting our users is a top priority for Facebook, and we're proud to be a part of this year's Anti-Bullying Week efforts in the UK.

Richard, Facebook's European Director of Public Policy, is a fan of the BBC bullyproof Page on Facebook.

GitHub Meetup London

mojombo — Sun, 15 Nov 2009 17:55:36 -0800

It’s the last stop on our Europe tour and we’re capping things off with an evening at the Royal George near Soho in London at 19:00 on Monday the 16th. PJ, Scott, and I will be in attendance along with (if the rumors are true) a ton of local developers and tech luminaries from the greater London area. This is a meetup you will not want to miss!

View Larger Map

Parents Have Rules to Follow Online, Too

Liz Perle — Thu, 12 Nov 2009 14:30:56 -0800

Liz Perle is the Editor-in-Chief of Common Sense Media, a nonprofit organization that is dedicated to improving the media and entertainment lives of kids and families. The organization believes that parents should have a choice and a voice about the media their kids consume. We've asked Liz to share some common sense tips for parents as they navigate technology and the web with their kids.

Where do kids learn about good online behavior? From each other, perhaps. Maybe from a teacher. But parents have a big role to play in making sure their kids use the incredible technologies at their fingertips in responsible ways. For parents, that means not only talking the talk, but walking the walk.

It's really up to parents to explain the basics of online safety and stress the importance of guarding privacy in a world where something shared with one friend online can quickly spread to a vast audience of many others. Parents should be sure their kids understand that anything and everything that people post online can be altered, copied, pasted and sent around to a gazillion friends of friends in an instant.

It's a parent's job to make sure kids understand that everything they do online leaves a "digital footprint" — the idea that once something is shared online, it's really almost impossible to take it back. Just ask my son about the Halloween photo of him that he thought was so funny in 8th grade. Two years later, it's resurfaced. Let's just say it's not doing much for his dating life.

But what about the parent's side of the equation? Kids learn from their parents, not just from what their parents say but also from what they do. It turns out that parents have as much to do with their kids' online behaviors as their kids do.

At Common Sense Media, we're dedicated to helping kids and families get the most out the powerful digital technologies that run through the center of our lives. Some of this involves helping kids learn responsible behavior, but kids are only half the story. Parents also have some "Rules of the Road" they should be following:

Model good behavior. If you're on your phones at dinner or during family events, why should your kids listen to you when you tell them to turn their phones off?

Pay attention. Know what your kids are doing online. For instance, talk to your kids about whether or not they're comfortable letting you "friend" them on social networks — many will be.

Impart your values. Right and wrong extends to all areas of life, whether it's online or through a mobile device. Cheating, lying, being cruel — they're all non-starters, no matter where you are.

Establish limits. There's really a right time and place for everything. Set guidelines for when it's acceptable to use the phone, download videos and surf the web.

Encourage balance. The Internet opens doors to new worlds. Encourage your kids to explore their own offline world as well, particularly when there is no cell phone or Internet service available.

Make kids accountable. Let your kids know that having access to technology is a privilege. Let's make sure they earn it.

Explain what's at stake. Let them know that what might seem acceptable today can be embarrassing tomorrow.

Do your homework. Get familiar with the websites and services your kids use and the type of content they're downloading. Armed with knowledge, you can find ways to use technology to say "yes" more often.

Don't be techno-phobic. Don't be afraid of technology. Learn to text, send a mobile photo, set up a Facebook profile, upload a video. Or have your kids show you how. It's impossible to guide what you don't understand. Not only that, but think of all the anxiety you can avoid by knowing how things work.

Lighten up, embrace their world and enjoy the possibilities together. No parents want a digital divide in their relationships with their kids. It's up to you to join the fun and help your kids seize the potential.

Liz wants to help parents create positive experiences for their kids with the articles and topics available on Common Sense Media's website.

Stay Tuned: TV on Facebook

Annie Ta — Fri, 06 Nov 2009 12:32:27 -0800

The following is part of our series on different ways you can use Facebook to connect with public figures and organizations around the world. Read the previous blog post in this series here.

From reality television to prime-time dramas, I admit it, I can be a bit television-obsessed. I often find myself spending an evening after a long day's work catching up on my favorite television shows. Unfortunately, a busy schedule can prevent even the most devoted fan from catching key plot lines and dramatic twists. Luckily, I can keep up to date on my favorite shows with my friends on Facebook, so I never miss a beat.

Watching Online

Every September, there are a dozen new television shows that I want to check out, but I only have so much time to tune in each night. One of my favorite websites, Hulu, streams TV shows directly on its Facebook Page using the application Watch Now. This way, I can watch full episodes of the hottest fall shows whenever I want, including "House," one of my favorite medical dramas on Fox. My friends can see how far along I am in the current season of diagnostic mysteries when I share the episode to my Facebook profile.

I also am discovering funny clips that I've missed thanks to my friends. I first heard of the popular joke from NBC's workplace comedy "The Office" about a social networking site called "Book Face" because a colleague shared the Hulu clip on his profile.

Teasers

Sometimes the best part of my day is when one of my favorite shows posts a new teaser for an upcoming episode on their Facebook Page. "Gossip Girl," the CW drama about young Manhattan elites, airs on Monday nights and then posts video and photo previews for upcoming episodes on Tuesdays. The week of waiting to watch the next episode of gossip among Serena, Nate, Chuck and Blair is more bearable when I can see fashions from "Gossip Girl" shared directly on my home page in the meantime.

Wednesdays are equally interesting when the Fox musical comedy "Glee" posts clips from the night's upcoming episode paired with introductions from all of my favorite cast members. Sometimes the show even includes clips of its ever-popular musical numbers like this rendition of "Keep Holding On."

Then there are times when my friends share their favorite TV shows with me on Facebook. For instance, one friend posted that she had seen the future in the "FlashForward Experience." It piqued my curiosity, and when I visited the show's Facebook Page, I was directed to a Facebook Connect-enabled website where I was suddenly immersed in the ABC show's trailer. The site uses the viewer's Facebook photos to create an individualized video after that viewer grants permission . The trailer asks, "Where were you on September 24?" I am definitely looking forward to the answer when I tune into the new science-fiction series.

New Favorites

Although my television schedule is busy with old favorites, I always have room for a new show. Those can be the most surprising. Several of my friends became a fan of the Travel Channel's competitive eating show, "Man v. Food," through an ad they saw on Facebook with the challenge, "5 lb burger, 4 peppers, 25 min." This, I had to see for myself.

I've also seen friends become fans of the new ABC sitcom "Modern Family." The show highlights its "Favorite Modern Moments" in a photo album on its Facebook Page. I laughed out loud at this photo and immediately tuned in through the ABC.com link I found on Facebook.

Annie, an associate on Facebook's communications team, can't wait to see tomorrow's TV line-up on Facebook today.

Get Educated on Facebook

Julie Trescott — Thu, 05 Nov 2009 10:12:39 -0800

As a recent college graduate, I use Facebook to keep up with the latest news from my alma mater, Pomona College, in Claremont, Calif. For instance, I learned about a happy hour for young alumni via a status update, and I saw photos of John Krasinski, who plays "Jim" in "The Office," on his recent visit to campus.

Pomona isn't the only college utilizing Pages. In fact, in addition to colleges and universities, a number of academic groups are now on the site. We also recently launched a Facebook in Education Page on the site as a resource for teachers, professors, administrators, counselors and others who work in academia.

Here are a few other interesting examples of how educators are using Facebook to connect.

Texas A&M University has connected with nearly 150,000 people through their Facebook Page. They share highlights from the latest Aggie win on the football field, exclusive interviews with the university President and admissions information.

Stanford University uses their Facebook Page to host open office hours for people to connect with professors and luminaries. They solicit questions on the Page and post responses in a series of videos. You can see the most recent edition here with Provost John Etchemendy.

High schools from San Jose, Calif. to Sanford, Fla. also are using Facebook to organize student and alumni events, generate support and excitement around upcoming games and keep their local communities informed about school activities.

Svein Arber, a high school teacher in San Francisco, Calif., created a Facebook group to connect with his Advanced Placement (AP) literature class. Over the course of the year, students write posts on the group's discussion board to analyze the narration style in Phillip Roth's "American Pastoral" and debate whether a character in Albert Camus' "The Stranger" is fundamentally good or evil. Svein also uses the Events feature to remind students of due dates for assignments. "It's been very positive," he said. "I'm able to reach my students where they live."

Other teachers are encouraging students to use Facebook to collaborate with peers across the globe. For example, two high school students from the Loudoun Academy of Science in Sterling, Va. are studying the positive qualities of maggots in collaboration with two students in Singapore, more than 9,000 miles away. One student, Joanne Guidry, told The Washington Post, "When we talk on Facebook, you can't tell they are halfway around the world."

There also are Facebook applications like Schools, which help students before they even step foot on campus, during school and after they graduate. Incoming students can contact admissions offices and engage with their peers. Current students can get help with homework from classmates and plan club meetings, while graduates can locate other alumni and organize reunions. You already can find schools like Arizona State University, Michigan State University and Becker College through the Schools application.

The Courses application allows educators and students to view class rosters and manage their schedules, while the Flashcards application keeps students language skills fresh by allowing them to to study spelling, French vocabulary and much more.

Now, that's some A+ work. Be sure to check out the recently launched Facebook in Education Page for even more tips on using the Facebook in the classroom, stories about educators and answers to frequently asked questions about Facebook and academia. We would love for you to share your own experiences, as well.

Julie, a specialist on Facebook's user operations team, is moving to Ireland to put her teaching skills to work in Facebook's Dublin office.

Facebook Tips: How Do I Change My Email Notifications?

Melody Quintana — Wed, 04 Nov 2009 10:40:48 -0800

The following is part of our series, Facebook Tips, which answers some of the most commonly asked questions about using Facebook. While we hope the tips we share on the blog will be informative, remember that we are unable to answer individual questions here. We'll always direct you to the Help Center for additional information about the topics we cover.

You don't have to be logged in to Facebook to get the latest updates from friends. Email notifications allow you to know when people interact with you on the site, whether through Wall posts, comments, tagged photos or friend requests. Similar to the notifications you see in the lower right hand corner on Facebook, email notifications let you know what's happening even when you aren't logged in.

You can manage email notifications by selecting the "Settings" link in the upper right hand corner of any page. From the Notifications tab, you can choose which actions you want to generate emails. For example, if you want to be notified by email each time someone writes on your Wall or receive weekly email digests about your friends' upcoming birthdays, just check the boxes next to these options. If you'd rather not be notified when people poke you or comment on your photos, just leave those boxes unchecked.

To change the email address associated with your account, select the "Settings" tab. You'll then see your current email address and can select "change" to add or delete addresses. Your Facebook email notifications will only be sent to the address you designate as your "Contact Email," so also be sure this information is accurate.

Melody, a specialist in Facebook user operations, has three new email notifications.

Parents Have Rules to Follow Online, Too

Liz Perle — Thu, 12 Nov 2009 14:30:56 -0800

Model good behavior. If you're on your phones at dinner or during family events, why should your kids listen to you when you tell them to turn their phones off?

Pay attention. Know what your kids are doing online. For instance, talk to your kids about whether or not they're comfortable letting you "friend" them on social networks — many will be.

Impart your values. Right and wrong extends to all areas of life, whether it's online or through a mobile device. Cheating, lying, being cruel — they're all non-starters, no matter where you are.

Establish limits. There's really a right time and place for everything. Set guidelines for when it's acceptable to use the phone, download videos and surf the web.

Encourage balance. The Internet opens doors to new worlds. Encourage your kids to explore their own offline world as well, particularly when there is no cell phone or Internet service available.

Make kids accountable. Let your kids know that having access to technology is a privilege. Let's make sure they earn it.

Explain what's at stake. Let them know that what might seem acceptable today can be embarrassing tomorrow.

Do your homework. Get familiar with the websites and services your kids use and the type of content they're downloading. Armed with knowledge, you can find ways to use technology to say "yes" more often.

Don't be techno-phobic. Don't be afraid of technology. Learn to text, send a mobile photo, set up a Facebook profile, upload a video. Or have your kids show you how. It's impossible to guide what you don't understand. Not only that, but think of all the anxiety you can avoid by knowing how things work.

Lighten up, embrace their world and enjoy the possibilities together. No parents want a digital divide in their relationships with their kids. It's up to you to join the fun and help your kids seize the potential.

Liz wants to help parents create positive experiences for their kids with the articles and topics available on Common Sense Media's website.

GitHub Meetup Oslo, Norway

mojombo — Tue, 10 Nov 2009 17:19:44 -0800

Continuing our tour of Europe, PJ, and Scott, and I will be in Oslo on Friday the 13th and we’d like to invite you to join us at Gloria Flames at 20:30 for drinks and good times. We’re rolling through Oslo specifically to meet the locals, so don’t disappoint us! I’ll be wearing a “fork you” tshirt so I’ll be easy to identify. See you there!

If for some reason we need to change venues, check our twitter accounts “mojombo”, “pjhyett”, and “chacon” for updates.

View Larger Map

GitHub Rebase #30

qrush — Tue, 10 Nov 2009 06:23:12 -0800

This is the second of many themed Rebases to come…our first was the book edition. In related news, the column has also been going on for over a year now! If you’ve got ideas for projects to feature or themes, don’t be afraid to message me.

Featured Project

ooc is a statically typed, object oriented language with some functional programming hints that’s growing up right here on GitHub. The compiler is currently written in Java, and it boils down to C99 so it can run anywhere. Grab the zip from the Downloads section or clone away and set it up. The language has some neat features: everyone’s favorite generic types, a clean import/use system that makes Ruby’s require look like yesterday’s news, and the cover keyword that allows for some really neat abstractions over types and clean interfacing with other C code. If you’re a language geek or need a breath of fresh air, check it out.

Notably New Projects

rock is an ooc compiler written in ooc, of course. This has some great examples of quite complex ooc code and aims to replace the Java compiler.

ooc-sqlite3 is an SQLite driver that’s also just getting off the ground, but it’s starting to pick up some steam. Next stop: a web framework!

woot is ooc’s first unit testing system, and is headed towards a bright future since it’ll be used to test ooc itself. This dogfood is tasty!

ooc-glew uses GLEW to open up the power of OpenGL to ooc. There’s also a little Ruby magic baked in to generate the bindings since there’s a lot of functions.

Stirling3D is also for the graphically minded. This project is a graphics/physics engine that also hooks into OpenGL and is starting to render some neat stuff.

ooc-yajl is a set of bindings to yajl, a former featured project that is a lightning fast JSON parsing library. It’s also a neat example of how easy it is to hook into C libraries and how the .use files work.

Recent UI updates to GitHub

kneath — Sun, 08 Nov 2009 17:33:36 -0800

On my first day at GitHub, I asked the crew what they wanted to me to do around here. Their response was something along the lines of "make GitHub sexy" Now that we've been rolling out steady UI updates for about a month, I thought I'd take some time and go over what all I've been working on.

I previously covered the updated dashboard repository lists and gist improvements, but I haven't had a chance to go over the new userbox, profile pages, account pages, and dashboard headers.

The new userbox was rolled out to solve a problem we had -- people with long usernames were breaking the UI, forcing links into the next line. I also took the time to reorganize the navigation and spruce up the look & feel.

The new profile pages were one of the first purely cosmetic updates to the site. You'll notice a new style of page header being rolled out. This header does have some functional benefits -- but they're subtle. On pages that are yours, you'll notice a slight yellow highlight. This isn't terribly useful right now, but as we roll out new features you'll see why I'm going this route.

I was also testing out the waters with this page. GitHub has an extremely particular and vocal audience that doesn't represent the typical web user. I wanted to make sure that the general reaction was positive. Judging from the twitter updates and in-person feedback I've gotten, people seem to like the changes -- so you'll see more of this style around the site in the future.

Along with the new profile pages, we also rolled out updated account settings pages. Previously, you had to edit your profile information in-place on the profile page, but now it's grouped with the rest of your settings.

Previously all of the billing information was crammed into a little grey box in the upper right of the account settings page. With the updated account settings page I decided to create a dedicated billing page. This page shows your plan usage in an easier to access manner and (hopefully) makes it easier to upgrade to paid accounts! :)

The last update we've rolled out are new-style dashboard headers. The reason for these aren't entirely obvious right now -- but in time they will become a lot more useful. I realize these new headers do push down the repository lists by about 60px, but it was a compromise that I decided to make to give us room for the future.

Hopefully you guys are enjoying the evolution of the GitHub interface, I've been having a blast watch the feedback pour in after each push.

GitHub Meetup Poznań, Poland

mojombo — Sat, 07 Nov 2009 09:06:04 -0800

Tom, PJ, and Scott will be hanging out at MOOD Club at 21:30 tonight (Saturday) in Poznań, Poland following the first day of the RuPy Conference. If you’re local or in town for the conference, come by and say hi!

Decertifying "Terrorist" Pilots

schneier — Mon, 23 Nov 2009 12:36:46 -0800

Al Qaeda Secret Code Broken

schneier — Mon, 23 Nov 2009 05:24:28 -0800

Quickpost: SelectMyParent or Playing With the Windows Process Tree

Didier Stevens — Sun, 22 Nov 2009 12:36:53 -0800

I read something very interesting in “Windows via C/C++” today: starting with Windows Vista, CreateProcess can start a program where you specify the parent process! This is something forensic investigators must be aware of when they analyse processes running on a Windows machine.

Normally the parent process of a new process is the process that created the new process (via CreateProcess). But when using STARTUPINFOEX with the right LPPROC_THREAD_ATTRIBUTE_LIST to create a process, you can arbitrarely specify the parent process, provided you have the rights (i.e. it’s your process or you have debug rights).

I developed a small tool to start a program while specifying its parent process: SelectMyParent. Here I use it to start notepad as a child of lsass.exe:

2 remarks about this example:

to make lsass.exe a parent process, you need to use SelectMyParent with admin rights and elevate its rights (Run as administrator)
the notepad process takes over the parent process’ account: NT AUTHORITY\SYSTEM

I don’t know how one can detect that a process’ parent is not the process that created it, because a process has no access to its extended startup info (only to its startup info). And it is the extended startup info that contains the attribute list with the handle to the parent process.

SelectMyParent version 0.0.0.1 is available here.

Quickpost info

Best of Application Security (Friday, Nov. 20)

Jeremiah Grossman — Fri, 20 Nov 2009 10:52:00 -0800

Ten of Application Security industry's coolest, most interesting, important, and entertaining links from the past week -- in no particular order. Regularly released until year end. Then the Best of Application Security 2009 will be selected!

WhiteHat Security is a leading provider of website security services.

Friday Squid Blogging: New Squid Discovered

schneier — Fri, 20 Nov 2009 14:57:09 -0800

Interview with Me

schneier — Fri, 20 Nov 2009 11:21:14 -0800

FailBlog on Security

schneier — Fri, 20 Nov 2009 09:11:19 -0800

Hex-Rays Plugin Contest

Ilfak Guilfanov — Fri, 20 Nov 2009 06:42:45 -0800

We are glad to announce the results of our first plugin contest! For the contest rules, please check this page:

http://www.hex-rays.com/contest.shtml

Or you may directly go to the contest results and check out some cool plugins:

http://www.hex-rays.com/contest2009

It was our first contest, but we are happy with the results and will repeat it in the near future.
Have fun!

Denial-of-Service Attack Against CALEA

schneier — Fri, 20 Nov 2009 04:11:17 -0800

Update: bpmtk with hook-createprocess.dll

Didier Stevens — Thu, 19 Nov 2009 11:32:34 -0800

There are no real changes in this new version of bpmtk, only a new DLL (hook-createprocess.dll) was added. You can use this DLL to protect your Windows machine from getting infected by the current malicious documents found in-the-wild.

You can download bpmtk version 0.1.6.0 here.

Hook-createprocess.dll is a DLL that patches the process into which it is loaded to prevent it from creating new processes. It does this by patching the Import Address Table of kernel32.dll for ntdll.dll to hook API functions NtCreateProcessEx, NtCreateProcess and NtCreateUserProcess.
Calls to these functions are intercepted and not passed on to the original functions. Instead, a code is returned indicating that the operation was blocked. The result is that functions in kernel32 used to create new processes fail (like WinExec) and hence that the patched process can’t create new processes.
This is all it takes to block most shellcode found in malicious documents like PDF malware. Shellcode like this does the following:

Of course, since this protective measure is taken by patching the process, shellcode could undo this patching and bypass our protection. Or it could use the ntdll API and not be hindered by our patch. But actual malware found in-the-wild doesn’t do this (not talking about targeted attacks) and is thus prevented from executing the trojan it just downloaded or extracted from the PDF document.

If you want better protection, you’ll have to use something that works at the level of the kernel, like sandboxing software.

However, this patch comes with some drawbacks, because it also blocks bening new processes. For example, the update function of Adobe Acrobat requires the creation of a new process. To reenable the creation of processes, you have to unload hook-createprocess.dll (unloading removes the hooks). bpmtk has a function to unload DLLs from a process (reject).

There are a couple of trick to load this DLL with the program you want to protect. I’ll describe a generic method in an upcoming post, but now I want to explain it for a specific program.
Programs have a list of DLLs they need for their execution. We will use a PE-file editor to add our hook-createprocess.dll to this list. hook-createprocess.dll exports a dummy function (_Dummy) just so you can add to the imports table of an executable. We will use LordPE to add hook-createprocess.dll with _Dummy to Adobe Reader:

Right-click the Import table:

And don’t forget to save…

A Taxonomy of Social Networking Data

schneier — Thu, 19 Nov 2009 10:51:17 -0800

Stabbing People with Stuff You Can Get Through Airport Security

schneier — Thu, 19 Nov 2009 05:10:41 -0800

How Smart are Islamic Terrorists?

schneier — Wed, 18 Nov 2009 11:45:32 -0800

Quantum Ghost Imaging

schneier — Wed, 18 Nov 2009 04:22:26 -0800

Secret Knock Lock

schneier — Tue, 17 Nov 2009 12:00:25 -0800

vsftpd-2.2.2 released

Chris — Tue, 17 Nov 2009 12:42:00 -0800

Just a quick note that I released vsftpd-2.2.2.
Most significantly, a regression was fixed in the inbuilt listener. Heavily loaded sites could see a session get booted out just after the initial connect. If you saw "500 OOPS: child died", that was probably this.

http://vsftpd.beasts.org/

A Useful Side-Effect of Misplaced Fear

schneier — Tue, 17 Nov 2009 03:58:07 -0800

Page2RSS Monitored Page: Stefan Savage

(author unknown) — Mon, 16 Nov 2009 14:53:25 -0800

Anti-Malware Detection and the Original Trojan Horse

schneier — Mon, 16 Nov 2009 11:09:45 -0800

Public Reactions to Terrorist Threats

schneier — Mon, 16 Nov 2009 04:39:11 -0800

Special LAMBDA meeting – Brent Yorgey

Andy Gill — Mon, 23 Nov 2009 09:11:45 -0800

Brent Yorgey (former editor on HWN, PhD student from the Programming Languages Group at the University of Pennsylvania) will be visiting KU on Tuesday. He will be talking at 1pm.

Title: Random testing and beyond with combinatorial species
Time: 1pm
Place: ITTC (conference room TBD).

Abstract:

How can we perform effective random testing of functions involving recursive data types? What about functions involving recursive data types and constrained (type class) polymorphism? Can we extend our programming languages with mechanisms for declaring and computing with data types whose subparts are not completely ordered (e.g. bags or cycles)? These are just a few of the questions we may be able to answer using the theory of combinatorial species, which deserves to be much better known in the functional programming community. I will focus on test case generation for recursive data types as a case study, using it to build some intuition for combinatorial species and the magic of generating functions. I will then hint at some other applications of combinatorial species and directions for further research.

Brent’s web page: http://www.cis.upenn.edu/~byorgey/

David Kravets on ACTA

Lex Spoon — Mon, 23 Nov 2009 05:47:00 -0800

David Kravets writes in Wired that he is not real happy with ACTA, especially with the way it's being developed:

Dan Glickman, the MPAA’s chairman, informs lawmakers that millions of film-related jobs are in peril because of internet piracy. Simply put, those who don’t back the proposed Anti-Counterfeiting and Trade Agreement don’t support intellectual property rights, he wrote.

“Opponents of ACTA are either indifferent to this situation, or actively hostile toward efforts to improve copyright enforcement worldwide,” Glickman wrote.

That’s an insultingly black-and-white viewpoint. It’s also not an accurate description of the treaty’s critics.

I am not hostile to intellectual property in general, but I oppose ACTA. I simply think we should take it as a constraint that we not give up basic liberties like being able to copy a CD for a friend. I don't think we necessarily have to protect incumbents as technology improves, but in this case copyrights can still be protected and made profitable. For example, we could loosen copyright but tighten rights on public performance.

More fundamentally, this issue should be discussed in public, and with representation by people trying to devise a better way. The ACTA negotiations appear to be between two groups of people: national officials and current copyright holders. I expect the first group is offering police work, the second group taxes, and they'll close the deal once they've agreed on how much of each.

Adding automatic semicolon insertion to a Javascript parser

Luis Diego Fallas — Mon, 23 Nov 2009 04:03:00 -0800

A couple of weeks ago I wrote a blog post about a Javascript parser written using the Newspeak parsing combinators. As mentioned in that post, no semicolon insertion was supported. This post shows how the feature was added.

Automatic semicolon insertion

As detailed in section 7.9 of the ECMA 262 document[PDF], in Javascript you can use newline as statement separator in some scenarios. For example a semicolon is "implicitly inserted" if expression-statements are separated by line terminators:


if (condition) {
 print("A")
 print("B")
}

This code snippet is equivalent to:


if (condition) {
 print("A");
 print("B");
}

Solution

In the a original post about the parser, espin pointed me out to a paper[PDF] by A. Warth that mentions how the semicolon insertion problem was solved in a Javascript parser written in OMeta. The solution presented is this post is based on the one from the paper.

I wanted to isolate the code that performs this function. So in order to add this functionality I created a subclass that overrides the productions that get involved in this process. This way we can have both a parser with and without the feature. Here's the code:


class JSGrammarWithSemicolonInsertion = JSGrammar (
"Parser features that add automatic semicolon insertion"
|
 
 specialStatementTermination = ((( cr | lf ) not & whitespace ) star,
 (semicolon | comment | lf | cr | (peek: $})) ) 
 wrapper: [ :ws :terminator | | t | t:: Token new. t token: $;. t].

 returnStatement = return, (specialStatementTermination | 
 (expression , specialStatementTermination)).

 breakStatement = break, (specialStatementTermination | 
 (identifier , specialStatementTermination)). 

 continueStatement = continue, (specialStatementTermination | 
 (identifier , specialStatementTermination)). 

 whitespaceNoEOL = (( cr | lf ) not & whitespace ) star,
 (((peek: (Character cr)) | (peek: (Character lf))) not) .

 throwStatement = throw, whitespaceNoEOL , expression , specialStatementTermination. 

 expressionStatement = (((function | leftbrace) not) & expression), specialStatementTermination.

 variableStatement = var, variableDeclarationList, specialStatementTermination.
| 
)

The result of parsing the following code:


var x = 0
while (true) {
 x++
 document.write(x)
 if ( x > 10)
 break
 else continue
}

... is presented using the utility created for the previous post:

Code for this post is available here.

David Pollak is happy about using Scala

Lex Spoon — Fri, 20 Nov 2009 17:17:00 -0800

David Pollak is happy about using Scala:

I've been a JVM guy since '96, so finding a language that was as on the JVM was a plus for me. I was looking for a statically typed language with high performance, but with the syntactic economy of Ruby. I bounced around a couple of language listing sites and found Scala. Three years ago, I fell in total love with Scala. That love continues today.

I can't say I blame him.

Dehydra Testsuite Passes on GCC 4.5

tglek — Fri, 20 Nov 2009 16:24:04 -0800

I spent couple of days fixing the remaining test-suite failures on GCC 4.5 trunk for Dehydra. Since the last time I looked into this, GCC went from crashing all over the place to only crashing if I did something bad. It was nice to discover that as a result of switching to 4.5 Dehydra users will get saner .isExplicit behavior and more precise location info.

Treehydra will take more work due to me misunderstanding GTY annotations.

By the way, I am really grateful for all of the people who contributed GCC 4.5 fixes so far. You guys have been a big help in getting Dehydra testsuite to 100% on 4.5. Looks like I will meet my goals to finish De+Treehydra by the end of the year in time for GCC 4.5 release and my “Introducing Dehydra to the Developer World”-type talk at LinuxConf.au.nz 2010.

Startup
I reduced my focus on startup speed at the moment to catch up on Dehydra. I plan to work on reducing xpconnect overhead during startup next, ie more of this bug.

A list is an odd creature, take 2

Philip Wadler — Fri, 20 Nov 2009 03:38:00 -0800

A second entry, from Timothy Abraham.

I have created this animal which represents a list, it's a snake whose tail consists of heads, in the end you'll see that the snake releases air with the empty list in it, this symbolizes the empty list, because the empty list is like air it is something but it is also nothing.

Go Channels

Ian Lance Taylor — Thu, 19 Nov 2009 23:02:02 -0800

Since the first thing I liked about Go is channels, I’ll talk some more about them.

IA channel can transmit a value of a specific type. That is, a channel value of type chan T can transmit any value of type T. This is not much of a restriction, as the type interface {} can be used to match any value at all.

There are two different kinds of channels: buffered and unbuffered. Unbuffered ones are the default. An unbuffered channel provides synchronous transmission of a value from a sender to a receiver. This means that if you try to send a value on the channel, and nothing is receiving from the channel, then you will block until there is a receiver. Similarly, if you try to receive and nothing is sending, you will block until there is a sender (which is pretty obvious). In other words, an unbuffered channel serves as a synchronization point between two goroutines. This is the key point of the Go memory model, which more or less amounts to saying that if you receive a value on a channel, then you will see all the writes made by the sender of the channel.

Buffered channels are more straightforward: if you write a value to the channel, it simply goes into the buffer. If there isn’t room, you want until there is room.

Implementing the above is fairly straightforward. However, there is another aspect to channels in Go: the select statement. The select statement permits a program to wait until some operation on a channel can proceed. It can list an arbitrary number of channels to wait for, and for each channel it can indicate whether it wants to wait to send a value or to receive a value. Implementing this efficiently is at the heart of the goroutine scheduling mechanism. For example, one interesting case is when one goroutine is selecting to write to an unbuffered channel while another goroutine is selecting to read from the channel. The scheduler must get both goroutines to synch up and transmit the value.

In gccgo I currently use a thread for each goroutine, although this will have to change at some point. In order to implement select, each unbuffered channel has a linked list of select statements that are waiting for the channel to be available. When a new select statement comes along such that a send and a receive are linked, and the new select statements picks that operation to run, it has to wake up the other select such that it is forced to use the new channel. This can fail due to race conditions if there is a third select statement involved, in which case the first select has to restart. This was all a huge headache to get working, and I completely rewrote all the select code three times.

Go Channels

Ian Lance Taylor — Thu, 19 Nov 2009 23:02:02 -0800

Since the first thing I liked about Go is channels, I’ll talk some more about them.

Buffered channels are more straightforward: if you write a value to the channel, it simply goes into the buffer. If there isn’t room, you want until there is room.

“Elements of Programming” Study Group

Dave Abrahams — Thu, 19 Nov 2009 13:41:25 -0800

Easy File System Path Manipulation in C#

Sandro Magi — Wed, 18 Nov 2009 19:39:00 -0800

I came across this type-safe module for handling file paths in the Haskell subreddit this week, and thought it look kind of neat. Handling paths as strings, even with System.IO.Path always bugged me.

So I created a close C# equivalent of the Haskell type and added it to the Sasa library, to be available in the upcoming v0.9.3 release:

public struct FsPath : IEnumerable<string>, IEnumerable
{
 public FsPath(IEnumerable<string> parts);
 public FsPath(string path);

 public static FsPath operator /(FsPath path1, FsPath path2);
 public static FsPath operator /(FsPath path, IEnumerable<string> parts);
 public static FsPath operator /(FsPath path, string part);
 public static FsPath operator /(FsPath path, string[] parts);
 public static FsPath operator /(IEnumerable<string> parts, FsPath path);
 public static FsPath operator /(string part, FsPath path);
 public static FsPath operator /(string[] parts, FsPath path);
 public static implicit operator FsPath(string path);

 public FsPath Combine(FsPath file);
 public IEnumerator<string> GetEnumerator();
 public static FsPath Root(string root);
 public override string ToString();
}

The above is just the interface, minus the comments which you can see at the above svn link. This struct tracks path components for you and C#'s operator overloading lets you specify paths declaratively without worrying about combining paths with proper separator characters, etc.:

FsPath root = "/foo/bar";
var baz = root / "blam" / "baz";
var etc = FsPath.Root("/etc/");
var passwd = etc / "passwd";

The library will generate path strings using the platform's directory separator.

One significant departure from the Haskell library is the lack of phantom types used to distinguish the various combinations of relative/absolute and file/directory paths. C# can express these same constraints, but I intentionally left them out for two reasons:

The type safety provided by the file/directory phantom type is rather limited, since it's only an alleged file/directory path; you have to consult the file system to determine whether the alleged type is actually true. The only minor advantage to specifying this in a path type, is as a form of documentation to users of your library that you expect a file path in a particular method, and not a directory path. I could be convinced to add it for this reason.

The relative/absolute phantom type seems a bit useless to me, though the reason may be less obvious to others. IMO, the set of file and directory info classes should not have GetParent() or support ".." directory navigation operations, nor should they support static privilege escalation operations like File.GetAbsolutePath() which ambiently converts a string into an authority on a file/directory, since this inhibits subsystem isolation; without GetParent() or privilege escalation functions, any subsystem you hand a directory object is automatically chroot jailed to operate only in that directory. This has long been known to the capability-security community, and is how they structure all of their IO libraries (see Joe-E and the E programming language). Thus, in a capability-secure file system library, all paths are relative paths interpreted relative to a given directory object. As a result, FsPath also strips all "." and resolve all ".." to within the provided path to supply this isolation.

It goes without saying that .NET does not currently handle files and directories this way, but I plan to handle this eventually as well. FsPath will play a significant role in ensuring that paths cannot escape the chroot jail.

As an interim step along that path, the FsPath interface is a good first step.

[Edit: the reddit thread for this post has some good discussion.]

Three Kinds of Concept Maps

Doug Gregor — Wed, 18 Nov 2009 08:45:37 -0800

Visualizing Empires Decline

Philip Wadler — Wed, 18 Nov 2009 03:24:00 -0800

An elegant animation by Pedro M Cruz. Spotted via Boing Boing.

Ditto

Philip Wadler — Wed, 18 Nov 2009 01:41:00 -0800

A new Internet meme.

Guide :: Heading for Deprecation: `export`, Exception Specification and `register`

(author unknown) — Wed, 18 Nov 2009 02:19:26 -0800

A list is an odd creature

Philip Wadler — Tue, 17 Nov 2009 02:44:00 -0800

I mentioned to my first-year students that a list is an odd creature: it has only a head and a tail, where the tail is itself a creature with only a head and a tail, and so on. I challenged them to produce a picture, and Saulius Lukauskas came up with this:

I have just remembered your request for a picture of a creature, with lots of heads and a tail.

The closest to this I was able to come up is a shark eating another shark eating ... etc.

Let's think of shark's head as an element in the list. This way one could think of ":" as shark teeth 'connecting' to other shark (Cons even look like a mark from a bite!). Then the head of the smaller shark is the head of new list, and the tail of the list is the bigger shark.

If we assume that water is [], then it also works in the bottom case with no elements in the list (no sharks) - only water. And water swallows the shark up to it's head so this also work (shark : []).

Mandelbulb

Philip Wadler — Tue, 17 Nov 2009 02:34:00 -0800

The quest for the 3D-analogue of the Mandelbrot curve, with some remarkable photos. Passed on by my colleague Bob Fisher, who knew of my work with students on interactive application, Mandelbrot Maps. Thanks, Bob!

Extensible, Statically Typed Pratt Parser in C#

Sandro Magi — Mon, 16 Nov 2009 06:41:00 -0800

I just completed a statically typed Pratt-style single-state extensible lexer+parser, otherwise known as a top-down operator precedence parser, for the Sasa library. The implementation is available in the Sasa.Parsing dll, under Sasa.Parsing.Pratt. Two simple arithmetic calculators are available in the unit tests.

This implementation is novel in two ways:

Aside from an alleged implementation in Ada, this is the only statically typed Pratt parser I'm aware of.

Pratt parsers typically require a pre-tokenized input before parsing semantic tokens, but I've eliminated this step by using the symbol definitions to drive a longest-match priority-based lexer.

Symbols by default match themselves, but you can optionally provide a scanning function used to match arbitrary string patterns. The symbol selected for the current position of the input is the symbol that matches the longest substring. If two symbols match equally, then the symbol with higher precedence is selected.

The design was heavily influenced by this Python implementation, though numerous departures were made to restore static typing and integrate lexing. In particular, symbols and semantic tokens are separate, where symbols are used primarily during lexing, and construct the appropriate semantic tokens on match.

Here is a simple arithmetic calculator:

class Calculator : PrattParser<int>
{
 public Calculator()
 {
 Infix("+", 10, Add); Infix("-", 10, Sub);
 Infix("*", 20, Mul); Infix("/", 20, Div);
 InfixR("^", 30, Pow); Postfix("!", 30, Fact);
 Prefix("-", 100, Neg); Prefix("+", 100, Pos);

 // grouping symbols, ie. "(_)"
 Group("(", ")", int.MaxValue);

 // provide a predicate to identify valid literals
 Match("(digit)", char.IsDigit, 1, Int);

 // ignore whitespace
 SkipWhile(char.IsWhiteSpace);
 }

 int Int(string lit) { return int.Parse(lit); }
 int Add(int lhs, int rhs) { return lhs + rhs; }
 int Sub(int lhs, int rhs) { return lhs - rhs; }
 int Mul(int lhs, int rhs) { return lhs * rhs; }
 int Div(int lhs, int rhs) { return lhs / rhs; }
 int Pow(int lhs, int rhs) { return (int)Math.Pow(lhs, rhs); }
 int Neg(int arg) { return -arg; }
 int Pos(int arg) { return arg; }
 int Fact(int arg)
 {
 return arg == 0 || arg == 1 ? 1 : arg * Fact(arg - 1);
 }
}

The above is a simplified version of the calculators used in the unit tests. You can clearly see the definition of left and right-associative infix operators, and postfix operators. The numbers are precedence levels, and the delegates are the semantic actions associated with each token type.

There are pre-defined functions for creating left and right-associative infix, prefix, postfix, and both prefix and infix ternary operators -- prefix ternary is a simple "if _ then _ else _", infix ternary is like C's ternary operation, "_ ? _ : _". You are not constrained to these symbol types however, as you can define arbitrary character sequences as symbols and parse the results at your whim.

Adding variable binding to the above calculator requires a few extensions to support lexically scoped environments, but the parser requires only the following two extensions:

// symbol for identifiers, ie. variable names
Match("(ident)", char.IsLetter, 0,
 name => new Var { Name = name });
// let binding, ie. "let x = 1 in x", is a
// prefix-ternary operator
TernaryPrefix("let", "=", "in", 90, Let);

"let" is not matched as an identifier because the "let" symbol has a higher precedence. Parser definitions with ambiguous parses thus require some care to resolve the ambiguity either via match length or precedence relationships.

There are two popular choices when it comes to parsing: parser generator tools and parser combinator libraries. I've never liked generators personally, as they require a separate language and entirely separate tools to define and process the grammar thus increasing the learning and maintenance curve.

I actually like parser combinators, and there are now good techniques for supporting arbitrary left recursion in grammars (see Packrat parsing in OMeta). However, handling operator precedence often involves a series of complex extensions to the otherwise simple recursive descent, or it requires the parsed results to maintain all operator and grouping information so a post-processing phase can readjust the parse tree to reflect the proper priorities. The post-processing phase typically uses a shunting yard or Pratt-style operator precedence algorithm to resolve precedence, so why not just build the parser itself directly on Pratt parsing?

Unlike parser-combinator libraries, of which I've built a few, Pratt parsers natively support arbitrary operator precedence and are efficient as they require no backtracking. Pratt parsers are also Turing-complete, because the semantic actions can perform arbitrary computations on the parser input. Turing completeness carries both benefits and costs of course, but the single-state nature of Pratt parsers keeps it manageable. The "single state" of the parser is simply the current token matched from the input.

For example, if prefix-ternary operators weren't already available, you could simply define one like so:

// "let" is a prefix symbol on a name
var let = Prefix("let", 90, x =>
{
 // lexer matched "let" prefix operator on x
 // argument, a variable name
 Advance("="); // advance past the "="
 var xValue = Parse(0); // parse the value to bind to x
 Advance("in"); // advance past the "in"
 var body = Parse(0); // parse the let expression body
 // return a Let expression node
 return new Let{ Variable = x, Value = xValue, Body = body};
});
// declare the other symbols involved for the lexer
// but they have no associated semantic action
Symbol("=");
Symbol("in");

The above builds on "Prefix", but you could break this down even further as in Pratt's original paper and define arbitrary symbol behaviour by providing a scanning function, "left denotation", and "null denotation", and so on.

To define a parser, you need only inherit from Sasa.Parsing.Pratt.PrattParser, provide the semantic type T, and override the constructor for your parser to define your symbol table. The only downside is that each parser instance is monolithic and not thread-safe, so it can only parse one input at a time. There is no limit to how many parsers you can create however.

Pratt parsing is simple, compact, and efficient, so I hope it finds more widespread use, particularly now that there is a statically typed implementation for a popular language.

[Edit: the PrattParser will be available in the upcoming v0.9.3 release of the Sasa class library.]

Go

Ian Lance Taylor — Sun, 15 Nov 2009 22:40:46 -0800

I heard about the Go programming language in May 2008 and was immediately interested. I spent a couple of weeks writing a preliminary gcc frontend for Go. I’ve been working on it ever since, with just a few interruptions here and there. It’s a relief that this work is finally out there.

Both the language and the frontend are still explicitly experimental. So I was quite surprised by how much reaction it got. I was expecting just a few people to actually try it. Instead, I’ve basically spent all my time since Tuesday responding to comments on IRC and on the public mailing list.

Anyhow, what attracted me to the language is something I’ve written about in the past, namely multicore programming. Go provides cheap threads built into the language (they’re called goroutines) and it provides cheap communication between the threads, via channels. I won’t describe them in detail in this post, but I think that making these the easy mechanisms for multi-core, as opposed to making locks easy, greatly encourages correctness.

Go has a simple philosophy for multicore, which is “Do not communicate by sharing memory; instead, share memory by communicating.” What this means is that when you want two threads to share some data structure, you should do it by using explicit communication between the threads handing off ownership. You should not do it by having them both modify shared memory. Of course this only works if they have some way to communicate, which is what Go provides via channels. The point is that communication can be used to make it clear who is free to modify the data structure, and strict adherence to this approach eliminates race conditions.

It’s far too early to tell whether Go will catch on, but I certainly hope that even if it doesn’t, the basic ideas in the language will become part of programming’s standard tools for multicore.

There is of course more to the language than multicore, and I plan to write a few more notes about it as a I find time between mailing list replies.

Go

Ian Lance Taylor — Sun, 15 Nov 2009 22:40:46 -0800

There is of course more to the language than multicore, and I plan to write a few more notes about it as a I find time between mailing list replies.

NEPLS submission

Jonathan Edwards — Sun, 15 Nov 2009 16:36:31 -0800

I submitted the following to NEPLS:

Co-action

Co-action is a new form of synchronous reactive programming. From the programmer’s point of view, all actions take place simultaneously, and all interdependencies between actions are resolved instantaneously. Eliminating execution order avoids many of the complexities and pitfalls of imperative programming. What is novel is that co-action also supports an imperative model of state, with a heap of dynamically allocated objects pointing to one another, and assignment through pointer variables. Other forms of declarative programming either eliminate pointers or fall back to explicitly ordering execution. Co-action attempts to combine the best of both declarative and imperative programming.

Declarative languages avoid pointers for good reason: to determine a correct execution order, data flow dependencies must be known, but pointer aliasing makes that undecidable at compile time. Co-action instead finds a correct execution order at run time by monitoring program execution and doing rollbacks to backtrack. The undecidable problem at compile time is converted into a search problem at run time. Co-actions are in a sense the dual of transactions, combining synchronous actions instead of isolating asynchronous actions, and parallelizing instead of serializing.

To better study and communicate these ideas, I have defined a small formal calculus. The correctness criterion of an execution order is coherence: every field is written at most once, and prior to any reads. The completeness theorem states that if a coherent execution order exists, then the co-action execution algorithm will find it. As of this writing I have been unable to prove completeness, and am seeking advice.

Hadoop World: Hadoop + Vertica from Omer Trajman

alex — Mon, 23 Nov 2009 15:00:42 -0800

Today’s Hadoop World talk comes from Omer Trajman at Vertica. He goes into detail about how Hadoop complements Vertica’s column-oriented analytic database. Our thanks to Omer — and stay tuned for more!

Comments on a fabricated press release quote

Curt Monash — Mon, 23 Nov 2009 13:54:03 -0800

My clients at Kickfire put out a press release last week quoting me as saying things I neither said nor believe. The press release is about a “Queen For A Day” kind of contest announced way back in April, in which users were invited to submit stories of their data warehouse problems, with the biggest sob stories winning free Kickfire appliances. The fabricated “quote” reads:

As we went through the contest entries in detail, it was readily apparent that today’s data warehousing solutions are either massively expensive or non-existent,” said Curt Monash, Founder of Monash Research. “Clearly, there is major dual-market opportunity for a product such as the Kickfire appliance that can not only provide an affordable data warehousing solution to small companies; but can also target larger companies that have made an initial investment in high-end solutions, yet still need to add some affordable query processing power in other areas of the organization.”

In reality:

I spent a few minutes reviewing summaries of eight stories selected by Kickfire from the entrants, and emailed comments back to Kickfire about them. I have no further role to play in the contest.
The part of the “quote” that slams Kickfire’s competitors is not reflective of my views.
The “market opportunity” is in line with the positioning I’ve encouraged Kickfire to adopt. A good shorthand for it is the “Sybase IQ market.” In essence I see Kickfire as an interesting Sybase IQ alternative. But Sybase IQ is a formidable competitor, and there are many other competitors as well. This is hardly an untapped market ripe for Kickfire’s plucking.

I’m satisfied that this is all a case of lousy marketing execution – something Kickfire has a history of — rather than deliberate deception. Kickfire has recently turned over its VP of Marketing (twice) and PR resource (at least once). Scott Humphrey, Kickfire’s new outside PR guy, says he was incorrectly told by his predecessor that the press release and quote in question had been approved, and put it out without fact-checking. I believe him. I hope Kickfire CEO Bruce Armstrong will be able to add stronger marketing leadership soon. Bruce seems aware of the need, and is making reasonable marketing strategy decisions himself in the mean time, so there’s some basis for optimism.

And by the way – I don’t let vendors write press release quotes for me anyway. I let them edit in precise product names and so on, but otherwise the words are mine. The last occasion on which I recall bending this policy was inadvertent and over a year ago, when Greenplum emailed something to me — which was genuinely similar to my opinion — while I was on the phone with Aster at a particularly frenzied time, and I didn’t immediately realize the words weren’t my own.

Recursive CTE’s: PostgreSQL

Quassnoi — Mon, 23 Nov 2009 12:00:24 -0800

In the previous article on recursive CTE’s in SQL Server I demonstrated that they are not really set-based.

SQL Server implements the recursive CTE’s syntax, but forbids all operations that do not distribute over UNION ALL, and each recursive step sees only a single record from the previous step.

Now, let’s check the same operations in PostgreSQL 8.4.

To do this, we well write a query that selects only the very first branch of a tree: that is, each item would be the first child of its parent. To do this, we should select the item that would be the first child of the root, the select the first child of that item etc.

This is a set-based operation.

Oracle’s CONNECT BY syntax, despite being set-based, offers some limited set-based capabilities: you can use ORDER SIBLINGS BY clause to define the order in which the siblings are returned. However, this would require some additional work to efficiently return only the first branch.

In a true set-based system, this is much more simple.

Let’s create a sample table:

CREATE TABLE t_recursive ( id INT NOT NULL PRIMARY KEY, parent INT NOT NULL, orderer INT NOT NULL, data VARCHAR(100) NOT NULL ); CREATE INDEX ix_recursive_parent_orderer ON t_recursive (parent, orderer); SELECT SETSEED(0.20091123); INSERT
INTO t_recursive
SELECT s, (s - 1) / 5, FLOOR(RANDOM() * 10000), 'Item ' || s
FROM generate_series(1, 1000000) s;

This table contains 1,000,000 records and implements an adjacency tree hierarchy.

Each item has at most 5 children and a randomly filled column, orderer, which defines its order along its siblings.

Now, let’s try to make a query that would select the first branch, the order being defined by the value provided in orderer.

To do this, we should make the anchor step to return the first child of the root item, and the recursive steps to return the first child of the previously returned item.

To return the first child of a given parent in the orderer order, we can use PostgreSQL’s DISTINCT ON functionality.

This is the same as DISTINCT, but can return the whole row rather than a column DISTINCT is being applied to.

If two rows share the value of the column DISTINCT ON is being applied to, only one of the rows will be returned. Which row will it be is defined by the ORDER BY clause.

This solves the problems like return a single row that holds group-wise maximum.

This query:

SELECT DISTINCT ON (parent) *
FROM t_recursive
ORDER BY parent, orderer

is the same as this one:

SELECT (q.r).*
FROM ( SELECT r, ROW_NUMBER() OVER (PARTITION BY grouper ORDER BY orderer) AS rn FROM t_recursive r ) q
WHERE rn = 1

, but more legible and in some cases more efficient.

To apply this clause to the task we need, we should just use DISTINCT ON (parent) in both anchor and recursive parts:

WITH RECURSIVE rows AS ( SELECT * FROM ( SELECT DISTINCT ON (parent) * FROM t_recursive WHERE parent = 0 ORDER BY parent, orderer ) q UNION ALL SELECT * FROM ( SELECT DISTINCT ON (c.parent) c.* FROM rows r JOIN t_recursive c ON c.parent = r.id ORDER BY c.parent, c.orderer ) q2 )
SELECT *
FROM rows

id	parent	orderer	data
3	0	1686	Item 3
19	3	3370	Item 19
98	19	42	Item 98
492	98	1762	Item 492
2464	492	2295	Item 2464
12322	2464	2050	Item 12322
61614	12322	768	Item 61614
308074	61614	1925	Item 308074
8 rows fetched in 0.0004s (0.0042s)

CTE Scan on rows (cost=3188.19..3207.83 rows=982 width=230) CTE rows -> Recursive Union (cost=0.00..3188.19 rows=982 width=230) -> Unique (cost=0.00..15.46 rows=2 width=23) -> Index Scan using ix_recursive_parent_orderer on t_recursive (cost=0.00..15.45 rows=5 width=23) Index Cond: (parent = 0) -> Unique (cost=313.84..314.33 rows=98 width=23) -> Sort (cost=313.84..314.08 rows=98 width=23) Sort Key: c.parent, c.orderer -> Nested Loop (cost=0.00..310.60 rows=98 width=23) -> WorkTable Scan on rows r (cost=0.00..0.40 rows=20 width=4) -> Index Scan using ix_recursive_parent_orderer on t_recursive c (cost=0.00..15.45 rows=5 width=23) Index Cond: (c.parent = r.id)

This gives us a single branch containing just the records we need: each one being the first child of its parent.

Now, what if we wanted to return a list of records, each being the second child to its parent?

Since each recursive part takes only one record as an input (and returns one record as an output), we can just replace DISTINCT ON (which returns the first child of each group) with a OFFSET 1 LIMIT 1:

WITH RECURSIVE rows AS ( SELECT * FROM ( SELECT * FROM t_recursive WHERE parent = 0 ORDER BY orderer OFFSET 1 LIMIT 1 ) q UNION ALL SELECT * FROM ( SELECT c.* FROM rows r JOIN t_recursive c ON c.parent = r.id ORDER BY c.orderer OFFSET 1 LIMIT 1 ) q2 )
SELECT *
FROM rows

id	parent	orderer	data
1	0	2540	Item 1
6	1	3405	Item 6
33	6	2884	Item 33
166	33	3084	Item 166
833	166	1848	Item 833
4169	833	993	Item 4169
20850	4169	3126	Item 20850
104251	20850	3021	Item 104251
521256	104251	5492	Item 521256
9 rows fetched in 0.0005s (0.0043s)

CTE Scan on rows (cost=1564.44..1564.66 rows=11 width=230) CTE rows -> Recursive Union (cost=3.09..1564.44 rows=11 width=230) -> Limit (cost=3.09..6.18 rows=1 width=23) -> Index Scan using ix_recursive_parent_orderer on t_recursive (cost=0.00..15.45 rows=5 width=23) Index Cond: (parent = 0) -> Limit (cost=155.79..155.79 rows=1 width=23) -> Sort (cost=155.79..155.91 rows=49 width=23) Sort Key: c.orderer -> Nested Loop (cost=0.00..155.30 rows=49 width=23) -> WorkTable Scan on rows r (cost=0.00..0.20 rows=10 width=4) -> Index Scan using ix_recursive_parent_orderer on t_recursive c (cost=0.00..15.45 rows=5 width=23) Index Cond: (c.parent = r.id)

This query returns a whole branch of the items that are second children to their parents.

Both these queries are available in SQL Server 2005: despite the fact they do not distribute over UNION ALL, they can be rewritten using a ROW_NUMBER().

As was shown in the previous article, for some strange reason, SQL Server 2005 does not forbid this clause, but rather implements it in the incorrect way. This, however, allows writing the query we’re after.

Now, let’s make a query that would recursively return the first two children.

This requires a set-based solution, since first two children can come from different parents.

A grandchild that is third to its grandfather won’t count, even if it was a first child to the first child of the grandfather.

A first child to the third child of the grandfather won’t count either, even if it was the first grandchild to its grandfather.

To do this right, on each step the query should accept 2 items, return 2 items and process the items accepted at once.

This is also easy in a true set-based recursive CTE:

WITH RECURSIVE rows AS ( SELECT * FROM ( SELECT * FROM t_recursive r WHERE parent = 0 ORDER BY orderer LIMIT 2 ) q UNION ALL SELECT * FROM ( SELECT c.* FROM rows r JOIN t_recursive c ON c.parent = r.id ORDER BY c.orderer LIMIT 2 ) q )
SELECT *
FROM rows r

id	parent	orderer	data
3	0	1686	Item 3
1	0	2540	Item 1
8	1	2181	Item 8
19	3	3370	Item 19
98	19	42	Item 98
99	19	1351	Item 99
497	99	1245	Item 497
496	99	1255	Item 496
2486	497	205	Item 2486
2484	496	362	Item 2484
12431	2486	29	Item 12431
12423	2484	311	Item 12423
62119	12423	1113	Item 62119
62120	12423	1121	Item 62120
310602	62120	341	Item 310602
310605	62120	468	Item 310605
16 rows fetched in 0.0007s (0.0043s)

CTE Scan on rows r (cost=3122.65..3123.09 rows=22 width=230) CTE rows -> Recursive Union (cost=0.00..3122.65 rows=22 width=230) -> Limit (cost=0.00..6.18 rows=2 width=23) -> Index Scan using ix_recursive_parent_orderer on t_recursive r (cost=0.00..15.45 rows=5 width=23) Index Cond: (parent = 0) -> Limit (cost=311.58..311.58 rows=2 width=23) -> Sort (cost=311.58..311.82 rows=98 width=23) Sort Key: c.orderer -> Nested Loop (cost=0.00..310.60 rows=98 width=23) -> WorkTable Scan on rows r (cost=0.00..0.40 rows=20 width=4) -> Index Scan using ix_recursive_parent_orderer on t_recursive c (cost=0.00..15.45 rows=5 width=23) Index Cond: (c.parent = r.id)

This query takes all children of the previous set and returns the first two of their descendants.

We see that PostgreSQL, unlike SQL Server, implements the recursive CTE’s in truly set-based way.

The recursive part of the query can accept a set on input, return a set on output and do the set-based operations on the set received.

Big Data on Grids or on Clouds?

Pawel Plaszczak — Mon, 23 Nov 2009 07:23:46 -0800

Battling Anti-NOSQL Trolls

Jeff Darcy — Mon, 23 Nov 2009 06:30:00 -0800

Originally from Snail in a Turtleneck, who deserves all the credit for a cool name and a good post.

I have a Twitter feed for the term “nosql” and every day I get tweets like:

“What moron came up with #nosql? you’re all fired!”
…
I hope I meet someone who says this to me someday, though, so I can say: “Boy, what a good point! If only Google and Yahoo and LinkedIn and Twitter and the thousands of other high-traffic websites had listened to you.

I see a lot of this stuff too, though in real life rather than Twitter. There are at least three points I almost always have to make when one of these trolls pops up.

We already know it’s more about ACID than SQL. Not only is such a “critique” unoriginal, but it’s irrelevant as well because it really doesn’t change anything about the relationship between traditional vs. non-traditional databases.
“Not Only SQL” isn’t about saying our flavor should be the only one. It’s about rejecting that attitude from the ACID grognards.
The CAP Theorem and its practical application are real, not just things that only exist in a few caffeine-addled newbies’ heads . The theorem itself was presented by a respected academic in an ACM keynote. Since then it has been the subject of formal proof by other academics, while the value of its application has been empirically demonstrated by everyone from Amazon to Yahoo.

Those three points are usually sufficient to stun the trolls into silence, even if expecting them to grow up and admit their error or apologize for their rudeness is unrealistic. Now, if only I could find a similar deterrent for the “cloud is just grid” trolls.

Boston Big Data Summit keynote outline

Curt Monash — Sun, 22 Nov 2009 22:25:50 -0800

Last month, Bob Zurek asked me to give a talk on “Big Data”, where “big” is anything from a few terabytes on up, then moderate a panel on cloud computing. We agreed that I could talk just from notes, without slides. So, since I have them typed up, I’m posting them below.

The top two points from Q&A probably were:

Big Data and the cloud actually have relatively little to do with each other, a few exceptions notwithstanding, especially if the data is in a shared-nothing DBMS (as opposed to, say, a MapReduce-oriented file cluster). Two principal reasons are:
- Redistributing data from node to node is a little slow, undermining some of the elasticity benefits of the cloud.
- Getting data into the cloud in the first place is a lot slow.
The NoSQL movement is a lot like the Ron Paul campaign — it consists of people who are dissatisfied with the status quo, whose dissatisfaction has a lot to do with insufficient liberty and/or excessive expenditure, and who otherwise don’t have a whole lot in common with each other.

Anyhow, here are my notes for the talk, edited in just a couple of places for readability or linkage.

Quick introduction

Big Data vs. cloud
How big is Big Data?
At the low end of that range, there’s little you can’t do with conventional technology if you have:
- An unlimited budget for hardware
- An unlimited budget for software
- An unlimited budget for people, especially Oracle DBAs

Big Data in OLTP

Hard-core OLTP
- Focus of DBMS technology for a long-time
- Big budgets because each transaction has significant value
- Tough to get users to change technologies
Lighter-weight OLTP
- Classic example = web companies
  - Big ones — retail-oriented ones (eBay, Amazon) partially excepted — rolled their own technology stacks
  - Reluctant to give money to anybody
    - Open source, etc.
- Difficulty finding market
  - Product vs. feature
    - Clustering/HA/DR/whatever
    - Ditto cloud enablement
  - True products haven’t found much traction yet

Analytic Big Data use cases

Kinds of data for analytics
- More of same != big
- More detail and/or new kinds
  - Complete data sets
  - Transactions
  - Call details
  - Tick/trade history
  - Web clickstreams
  - Network event logs
  - Other machine-generated data
  - CAM bottom line
    - Anything human-generated should and will be retained in its entirety
    - Quantities of machine-generated data retained should and will grow roughly in line w/ computing cost reductions (Moore’s Law, etc.)
Analytic uses of Big Data
- Analytics is mainly about three things
  - Problem detection
  - Customer relationship improvement
    - (Those overlap when the customer relationship is bad)
  - Financial statements on steroids
- Main kinds of analytics
  - What BI vendors traditionally sell
    - General reporting and dashboards
    - Ad-hoc query (now driven from those reports and dashboards)
    - Planning (allegedly integrated with BI)
  - Research
    - Ad hoc relational query (worth mentioning twice because it drives so much of the market)
    - Data mining
    - Most web search and web mining
  - Operational/near-real-time
  - Archiving/compliance
- What gets Big?
  - Mainly research and archiving
  - But when reporting or operational get Big, you have really interesting computing problems

Technology issues and trends

Moore’s Law
- CPUs — All about cores, hence parallelism is key
- RAM
- SSDs – hence replace disks
- Sensors – hence generate lots more data
Kryder’s Law
- But rotational speeds up only 12.5X since Eisenhower Administration
- Hence solid-state memory (or RAM) will soon take over
In the mean time, I/O bottlenecks have had to beaten
- Hence sequential scans
- Hence index-light architectures
- Hence columnar
DBMS “overhead”
- Raw license and maintenance fees – software increasing fraction of total
- OLTP vestiges – locking and all that
- DBAs
  - People costs = huge fraction of total
  - Index-lightness addresses
  - So does appliance
- Many people don’t really know how to write SQL
Configuration
- Appliance/tightly-balanced
  - Netezza
  - Teradata earlier
  - Greenplum/Sun
  - Oracle
  - IBM
  - Microsoft/Madison
- Commodity/do what you want
  - Vertica
  - Greenplum now
  - Infobright, Aster and others
  - MapReduce-oriented file systems
- Extreme rigidity is silly
  - Teradata, Oracle have both signaled moving to more modularity
  - Big driver of that = heterogeneous storage
    - Cheap disk
    - Expensive disk
    - Solid-state
    - RAM
  - CPU/storage ratio is even more of a driver

Theoretically defensible ways to segment the market

Latency requirements
- High availability and low latency go together
Query types
- Simultaneous users for same
Database size
Budget

Actual segments right now

Utter ADW/EDW
Data mart
- Size
- Naturally columnar vs. naturally row-based
Operational/frontline
Less dramatic/smaller EDW

Is Sandia National Lab's Red Sky Really Able to Deliver a PUE of 1.035?

James Hamilton — Sun, 22 Nov 2009 08:30:19 -0800

MySQL Performance Blog was down today

peter — Sat, 21 Nov 2009 22:16:07 -0800

MySQL Performance Blog (and percona.com too) were down today because the switch in our rack died completely. It took a while to fix it using secondary switch we had. Provider was not willing to do it as remote hands so I had to drive to the data center to fix it.

We got number of calls and messages from the customers and friends about web site going down so we probably have to invest into getting infrastructure more redundant – currently we were quite cheap and a lot of servers have single network card (so you can’t use trunking to eliminate switch as single point of failure).

The customer case management systems were not affected by this outage.

Entry posted by peter | No comment

Add to: | | | |

The future of the data center is green: Takeaways from WiredRE data center event

Gleb Budman — Fri, 20 Nov 2009 15:50:06 -0800

What do Google providing search, Coca-Cola operating its systems to track inventory, and Backblaze backing up your data have in common? The computers that handle all of this live in data centers. And those data centers use power - lots of it.

In the U.S. alone there are over 20,000 data centers - each of which houses thousands or tens of thousands of servers. Combined, these data centers make up 3% of all U.S. energy consumption (not just electricity) - more than the entire domestic air fleet.

So when I went to an event on Wednesday called:
THE TRUTH ABOUT THE FUTURE OF THE DATA CENTER:
CLOUD, COLOCATION, & DATA CENTER REAL ESTATE
it should be no surprise that the focus was on power, power, power.

And lest you think this is people getting wrapped up in the green movement or just jumping on a marketing trend - let me dissuade you. Datacenters in the U.S. spend $23 billion a year on electricity according to KC Mares of MegaWatt Consulting. In fact, electricity can often cost over 50% of the purchase price of a server over it’s lifetime. Minor improvements can have massive implications not only on global warming but also company bottom lines.

KC provided a fascinating overview of innovations and experiments that operators of data centers and the companies building out large server deployments are pursuing. Some examples:

* VFDs - variable frequency drives to adjust the speed of blower fans that adjust to need rather than spinning at a constant rate.
* Natural cooling - using outside air and fans rather than air-conditioning to keep data centers cool; it turns out most servers are perfectly happy running at temperatures much higher than what data centers attempt to keep them at.
* Shorter cooling regions - having air flow almost directly around a server in the process of cooling it rather than through the entire building; shorter distances mean less air friction and less energy spent moving it around.
* Eliminating UPS systems - getting rid of the backup power systems and assuming servers will go down…and having backup servers or data centers instead.
* Using 480 volts - higher voltage means lower amperage and thus less heat loss and higher efficiency. More of today’s server systems are capable of handling this voltage.
* Higher efficiency power supplies - switching to 90% efficient power supplies on servers rather than using 70% or 80% ones; these are more expensive upfront but can still pay off fairly quickly.

A number of these items pay for themselves in a couple months and then generate savings ongoing from then on. KC has a variety of information on his site and blog.

Why SHMConfig is off by default

(author unknown) — Fri, 20 Nov 2009 08:56:40 -0800

[$] Fedora 12 and unprivileged package installation

jake — Fri, 20 Nov 2009 13:28:15 -0800

Paul McCullagh answers your questions about PBXT

Morgan Tocker — Fri, 20 Nov 2009 11:29:55 -0800

Following on from our earlier announcement, Paul McCullagh has responded with the answers to your questions – as well as a few I gathered from other Percona folks, and attendees of OpenSQL Camp. Thank you Paul!

What’s the “ideal” use case for the PBXT engine, and how does it compare in performance? When would I use PBXT instead of a storage engine like MyISAM, InnoDB or XtraDB?

Unfortunately it is not possible to point to a specific category of applications and say, “PBXT will be better here, so try it”. PBXT is a general purpose transactional storage engine, designed to perform well on a broad range of tasks, much like InnoDB. However, PBXT’s log-based architecture makes performance characteristics different to both MyISAM and InnoDB/XtraDB. Tests show that PBXT’s performance is similar to InnoDB but, depending on your database designed and the application, it can be faster.

PBXT is a community project and, of course, we depend on users trying it out. In the long run, this will determine to what extent we are able to continue to develop and improve the engine. So, despite this rather vague answer, we are hoping that more people try it out, and work with us to improve the engine as necessary. My thanks to all who are already doing this!

I think I remember reports that PBXT (at an early stage) out performed InnoDB with INSERTS and UPDATES (but not SELECTS). That would make PBXT very interesting for non-SELECT-intensive applications (finance, production management etc.) in my opinion. Is this the case, and do you have any recent benchmarks available?

This is no longer necessarily the case. For example a test (http://mysqlha.blogspot.com/2009/03/pbxt-is-fast-no-kidding.html) by Mark Callaghan shows that PBXT can actually out perform InnoDB with SELECTs under circumstances.

The implementation of full-durability has changed the performance characteristics of PBXT from “MyISAM-like” to more InnoDB-like. Originally PBXT was conceived as an engine that would be somewhere between MyISAM and InnoDB in both performance and features. The early version of PBXT was not fully durable (equivalent to innodb_flush_log_at_trx_commit=2).

A major change was completed at the beginning of last year with the implementation of full-durability. In doing this it was important to keep the log-based architecture which was the reason for the high write performance of earlier versions.

Traditional transactional implementations suffer from the problem that a backlog of asynchronous writes accumulate until it swamps the engine. There has been a lot of work on both InnoDB and XtraDB to solve this problem. The key words here are fuzzy and adaptive checkpointing (the former, originally implementation by Heiki for InnoDB, and the latter, an excellent addition to XtraDB).

Both methods improve the management of the asynchronous writes. The idea behind the log-based solution, on the other hand, is to avoid the accumulating a backlog of asynchronous writes, but writing synchronously.

Although write performance is comparable with InnoDB, I am not entirely convinced that PBXT’s implementation of the log-based I/O is optimal at this stage. This is ongoing work for PBXT 1.5.

Morgan notes: As well as Adaptive Checkpointing, Oracle has also been working on Adaptive Flushing for the InnoDB Plugin. The engine being ’swamped’ problem that Paul is referring to is best described visually – see this post for more info.

What were the hard decisions or trade-offs that you had to make when designing PBXT?

If you read the white paper from 2006 (http://primebase.org/download/pbxt_white_paper.pdf) you will notice that the original design was uncompromisingly MVCC-based. Some of this has been changed to make PBXT more InnoDB-like, but other principles have remained.

Pure-MVCC does not do any locking. Read locks are not required because each transaction effectively gets its own snapshot of the database. And write locks are not acquired when updating. Instead, the application can hit a “optimistic lock error” if a record is updated by another user.

Now PBXT does acquire locks for 2 reasons: to support SELECT FOR UPDATE, and to avoid optimistic locking errors. This makes PBXT’s behavior identical to InnoDB in REPEATABLE READ mode.

On the other hand, there are currently no plans to implement InnoDB style “gap locking”. Gap locking effectively involves locking rows that do not exist. This, in turn, means that PBXT transactions are not SERIALIZABLE. A result of this is that statement-based replication is not supported by the engine.

Another hard decision was not to implement clustered indexes which I mentioned in more details later.

How does online backup work in PBXT, and is incremental backup possible?

A recent version of PBXT (1.0.09) supports the MySQL Backup API which was originally implemented in MySQL 6.0. This feature is now scheduled for an upcoming version of MySQL 5.4.

The Backup API makes it possible to pull a consistent snapshot of an entire database even when tables use different engine types. The API does not yet support incremental backup, but this is planned.

Internally this feature is implemented by PBXT using an MVCC-based consistent snapshot.

Does PBXT have a maintenance thread like InnoDB’s main thread?

PBXT has several system threads, that are responsible for various maintenance tasks. The most important of these are the “Writer”, the “Sweeper” and the “Checkpointer”;

The Writer transfers data from the transaction log to the database table files. This task runs constantly and is the main source of asynchronous I/O in the engine.
The Sweeper is a thread unique to the PBXT architecture. It’s job is to clean up after a transaction. Basically it recognizes which record versions are no longer needed and deletes them from the database.
The Checkpointer flushes the data written by the Writer to disk. It is also responsible for writing consistent snapshots of the index files to disk. The frequency of checkpoints, as with other engines like InnoDB, determines the recovery time.

Does PBXT support clustered indexes?

No, currently it does not. This is one of the original design decisions (as raised by a previous question). Two things contributed to this decision:

Supporting clustered indexes would have made the implementation of PBXT more complicated than I would have liked. My goal was to make PBXT as simple as possible, so that it is easy to maintain the code and add features.
I believe clustered indexes are becoming less relevant with the rise of Solid State technology. As random read access times decrease clustering of data will become less and less important.

What is the page size in PBXT, and can it be tuned?

PBXT uses 16K pages for the index data and (approximately) 32K pages for the table data. Both sizes can be set using compile time switches. However, if the index page size is changed, then the indices need to be rebuilt, which can be done by REPAIR TABLE. The table data page size does not require a rebuild because a page of records in the table is just a group of records (not an actual fixed length page).

Are there any differences in the PBXT implementation of MVCC that might surprise experienced InnoDB DBAs? Also – In MVCC does it keep the versions in indexes, and can PBXT use MVCC for index scans?

If you are using InnoDB in REPEATABLE READ mode, then there is essentially no difference in the isolation paradigm between the two engines.

REPEATABLE READ is often preferred over SERIALIZABLE mode because it allows a greater degree of concurrency while still providing the necessary transaction isolation. So I do not consider the lack of serializability as a serious deficit in the engine. And, fortunately MySQL 5.1. supports row-based replication which makes it possible to do replication while using REPEATABLE READ.

PBXT does use MVCC to do index scans. Basically this means that all types of SELECTs can be done without locking.

Morgan notes: Indexes not using MVCC is one of the main differences in the Falcon storage engine.

PBXT supports row-level locking and foreign keys. Does this create any additional locking overhead that we should be aware of?

Firstly, PBXT does not acquire read locks. A normal SELECT does not lock at all. In addition, an UPDATE or DELETE only acquires a temporary row-lock. This lock is released when the row is updated or deleted because the MVCC system can detect that a row has been changed (and is therefore write locked).

This means that PBXT does not normally need to maintain long lists of row-level locks. This is also the case when a foreign key leads to cascading operations which can affect thousands of rows.

The only case you need to be aware of is SELECT FOR UPDATE. This operation acquires and holds a row-level lock for each row returned by the SELECT. These locks are all stored in RAM. The format is quite compact (especially when row IDs are consecutive) but this can become an issue if millions of rows are selected in this manner.

I should also mention that the consequence of this is that SELECT … LOCK IN SHARE MODE is currently not supported.

When I evaluate a storage engine my key acceptance criteria are things like backup, concurrency, ACID compliance and crash recovery. As a storage engine developer, what other criteria do you think I should be adding?

Yes, I think you have mentioned the most important criteria. What I can add to this list are 3 things that make developing a storage engine extremely demanding: performance, stability and data integrity.

Of course, as a DBA or database user these aspects are so basic that they are taken for granted.

But engine developers need to keep performance, stability and data integrity in mind constantly. The problem is, they compete with each other: increasing performance often causes instabilities that then have to be fixed. How to optimize the program without compromising data integrity is a constant question.

Relative to maintaining performance, stability and data integrity, adding features to an engine is easy. So I would say that these are the criteria that concern a developer the most.

MySQL supports a “pluggable storage engine API”, but it seems that not all the storage engine vendors are able to keep all their code at that layer (Infobright had to make major changes to MySQL itself). What war stories can you report on in plugging into MySQL?

Unfortunately the “war” continues. I have already received several e-mails that PBXT does not compile with the recently released MySQL 5.1.41!

Any dot release can lead to this problem, and I think PBXT is fairly moderate with its integration into MySQL.

My main advantage: I have been able to avoid modifying any part of MySQL to make the engine work. This means that PBXT runs with the standard MySQL/MariaDB distribution.

But this has required quite a bit of creative work, in other words, hacks.

One of the main problems has been running into global locks when calling back into MySQL to do things like open a table, create a session structure (THD) or create a .frm file.

One extreme example of this is PBXT recovery. When MySQL calls the engine “init” method on startup it is holding the global LOCK_plugin lock. In init, the engine needs to do recovery. In PBXT’s case this means opening tables (reading a .frm file), which requires creating a THD. The code to create a THD in turn tries to acquire LOCK_plugin!

Unfortunately a thread hangs if it tries to acquire the same mutex twice, so this just does not work!

We went through quite a few iterations (MySQL code was also changing during the development of 5.1) before we came up with the current solution: create a background thread to do recovery asynchronously. So the thread can wait for the LOCK_plugin to be unlocked before it continues.

The affect is that the init function returns quickly, but the first queries that access PBXT tables may hang waiting for recovery to complete.

PBXT seems to have very few configuration parameters. Was this an intentional design decision, and do you see it creating opportunities for you in organizations with less internal IT-expertise?

No, this is not by design.

While I try to only add tuning parameters that are absolutely necessary, PBXT is not specifically designed to be self-tuning, because I believe that is a very hard problem to solve in general.

Tuning parameters are often added to an engine in response to performance problems in particular configurations. This is not necessarily a bad thing because it provides DBA’s with the tools they need.

My goal for PBXT in this regard is twofold:

I hope that most installations can get away with setting a minimum of tuning parameters, in particular, just the cache values.
On the other hand, I aim to provide expert tuning parameters for installations that need to extract maximum performance from the hardware. This work is ongoing.

Morgan notes: There are more in InnoDB/XtraDB now than there were three years ago. This is probably something that emerges over time as we get to understand more about an engine.

Entry posted by Morgan Tocker | No comment

Add to: | | | |

Hadoop World: Hadoop + Clojure from Stuart Sierra and Tim Dysinger

alex — Fri, 20 Nov 2009 08:00:22 -0800

Today’s Hadoop World talk comes from Stuart Sierra at Columbia University and Tim Dysinger at Sonian Networks. Unfortunately Tim couldn’t make the talk, so Stuart covered for him. Stuart goes into detail about how Hadoop can be used with Clojure, a neat and new language implemented on the Java Virtual Machine (JVM). Tim, Stuart, thank you! We’ll be posting more of the Hadoop World talks in the weeks to come.

ACM Symposium on Cloud Computing

James Hamilton — Fri, 20 Nov 2009 06:24:30 -0800

Rare evil MySQL Bug

peter — Thu, 19 Nov 2009 23:21:09 -0800

There is the rare bug which I ran into every so often. Last time I’ve seen it about 3 years ago on MySQL 4.1 and I hoped it is long fixed since… but it looks like it is not. I now get to see MySQL 5.4.2 in the funny state.

When you see bug happening you would see MySQL log flooded with error messages like this:

091119 23:03:34 [ERROR] Error in accept: Resource temporarily unavailable
091119 23:03:34 [ERROR] Error in accept: Resource temporarily unavailable
091119 23:03:34 [ERROR] Error in accept: Resource temporarily unavailable
091119 23:03:34 [ERROR] Error in accept: Resource temporarily unavailable

filling out disk space

Depending on the case you may be able to connect to MySQL through Unix Socket or TCP/IP or neither.
It also looks like there is a correlation between having a lot of tables and such condition.

Previously I was unlucky with seeing this issue in production so we had to restart MySQL quickly currently I have a test MySQL showing some behavior.

Here is what strace tells me:

[percona@test9 msb_5_4_2]$ strace -f -p 19229
Process 19229 attached with 23 threads – interrupt to quit
[pid 19286] rt_sigtimedwait([HUP QUIT ALRM TERM TSTP],
[pid 19285] futex(0×165962ec, FUTEX_WAIT_PRIVATE, 1, NULL
[pid 19284] select(0, NULL, NULL, NULL, {0, 765000}
[pid 19283] select(0, NULL, NULL, NULL, {0, 412000}
[pid 19248] futex(0×1781193c, FUTEX_WAIT_PRIVATE, 1, NULL
[pid 19247] futex(0×178118bc, FUTEX_WAIT_PRIVATE, 1, NULL
[pid 19246] futex(0×1781183c, FUTEX_WAIT_PRIVATE, 1, NULL
[pid 19245] futex(0×178117bc, FUTEX_WAIT_PRIVATE, 1, NULL
[pid 19244] futex(0×1781173c, FUTEX_WAIT_PRIVATE, 1, NULL
[pid 19232] futex(0×1781113c, FUTEX_WAIT_PRIVATE, 165, NULL
[pid 19229] accept(16392,
[pid 19241] futex(0×178115bc, FUTEX_WAIT_PRIVATE, 319, NULL
[pid 19243] futex(0×178116bc, FUTEX_WAIT_PRIVATE, 1, NULL
[pid 19242] futex(0×1781163c, FUTEX_WAIT_PRIVATE, 1, NULL
[pid 19240] futex(0×1781153c, FUTEX_WAIT_PRIVATE, 3, NULL
[pid 19239] futex(0×178114bc, FUTEX_WAIT_PRIVATE, 7, NULL
[pid 19238] futex(0×1781143c, FUTEX_WAIT_PRIVATE, 1, NULL
[pid 19237] futex(0×178113bc, FUTEX_WAIT_PRIVATE, 1, NULL
[pid 19236] futex(0×1781133c, FUTEX_WAIT_PRIVATE, 1, NULL
[pid 19235] futex(0×178112bc, FUTEX_WAIT_PRIVATE, 7, NULL
[pid 19234] futex(0×1781123c, FUTEX_WAIT_PRIVATE, 1, NULL
[pid 19233] futex(0×178111bc, FUTEX_WAIT_PRIVATE, 207, NULL
[pid 19231] futex(0×178110bc, FUTEX_WAIT_PRIVATE, 1, NULL
[pid 19229] <... accept resumed> 0×7fffffac70b0, [18423225245113516048]) = -1 EAGAIN (Resource temporarily unavailable)
[pid 19229] accept(16392, 0×7fffffac70b0, [18423225245113516048]) = -1 EAGAIN (Resource temporarily unavailable)
[pid 19229] accept(16392, 0×7fffffac70b0, [18423225245113516048]) = -1 EAGAIN (Resource temporarily unavailable)
[pid 19229] accept(16392, 0×7fffffac70b0, [18423225245113516048]) = -1 EAGAIN (Resource temporarily unavailable)
[pid 19229] fcntl(16392, F_SETFL, O_RDWR) = 0
[pid 19229] fcntl(16392, F_SETFL, O_RDWR) = 0
[pid 19229] select(16394, [1025 1040 1042 1044
....
8 9709 9714 9716 9717 15368 15369], NULL, NULL, NULL*** buffer overflow detected ***: strace terminated
======= Backtrace: =========
/lib64/libc.so.6(__chk_fail+0×2f)[0x35f36e6aff]
/lib64/libc.so.6[0x35f36e5ad3]
strace[0x408b60]

So as you can see accept gets pretty high socket number – probably because of large innodb_open_files I tested with in this case – which all can be used during recovery.

Note the process gets accept on the socket to fail with EAGAIN (which is not well described in the manual) and later getting call to select call with 16384 sockets. This does not seems to be the healthy number to work with SELECT call and It is quite possible something goes wrong because of it.

If you have any ideas why could be going wrong in this case.

Entry posted by peter | 7 comments

Add to: | | | |

Hadoop World: Protein Alignment from Paul Brown

alex — Thu, 19 Nov 2009 13:00:49 -0800

Today’s Hadoop World talk comes from Paul Brown at Booz Allen. He goes into detail about how Hadoop is used for bioinformatics and protein alignment. Our thanks to Paul — and stay tuned for more!

Sigh.

(author unknown) — Thu, 19 Nov 2009 12:32:10 -0800

10 eBay Secrets for Planet Wide Scaling

Todd Hoff — Tue, 17 Nov 2009 11:27:33 -0800

You Will Be Assimilated

Jeff Darcy — Thu, 19 Nov 2009 06:19:41 -0800

I have to admit that part of my general antipathy toward Java is a form of envy. Don’t get all excited, Java fans; it’s only a small part, and “envy” might not even be the right word. The Java language certainly has its flaws, as others have written, but every language has flaws and I could probably forgive Java’s. The Java programming environment – the vast array of libraries and frameworks and IDEs and other crutches, seems more flawed and less forgiveable. The Java culture, with its at-best-misleading claims of “write once run everywhere” and “as fast as C” and its endless swirl of enthusiasm for patterns and “agile” and every other fad to come along, is worse still. What’s worst of all is the way all of these things get bundled together. Both the dependency structure and the culture seem to preclude picking out and benefiting from the good bits while leaving the bad bits behind. I’ve yet to meet a Java programmer who even seemed to try. To be a Java programmer, it seems, means to dive headlong into the entire Java ocean and leave anything else behind. That kind of enforced isolation from the broader world is a defining characteristic of cults, by the way.

So why do I care, and why do I admit to envy? Because there is some cool stuff that happens in the Java world. For example, I recently came across Structure101. Please, go read what they have to say about architecture extraction and enforcement, and especially about complexity debt. Set aside twenty minutes or so and watch the videos. Their analysis of the problems that occur in large-scale software development rings very true to me, and I think they’ve created an awesome tool to help address those problems. So what’s the problem? If you go to their “versions and pricing” page, you’ll see it’s only for Java. Both the problem and the solution are completely language-independent, but their implementation is Java-specific. I don’t blame Headway for this, or the other vendors (e.g. for static code analysis tools) I’ve seen make the same decision. I’m sure their choice is based on rational analysis of the market. I’d love to be able to use such tools. I can’t, hence the admission of envy. I resent the Java cultists for that, not because Java itself has no value but because of the opportunity cost they’ve imposed on the entire industry. Those grapes aren’t sour. I’m sure they’re very sweet, which is exactly why putting a fence around them is so offensive. Because of the “all or nothing” nature of the Java world, lots of cool things that could benefit the broader programming community – from C to Python to Erlang to Clojure – never will. Theirs is a fundamentally selfish “come play in my sandbox or we won’t play at all” kind of attitude. It’s very similar to the “any kind of database you want as long as its ACID and relational” attitude I’ve also written about lately, and quite antithetical to the UNIX/Linux/open-source way of making tools that do their own jobs well and combine well with other tools to do larger jobs.

Java is the Borg of the computing world.

SQL Server: are the recursive CTE’s really set-based?

Quassnoi — Wed, 18 Nov 2009 12:00:18 -0800

Answering questions asked on the site.

Karl asks:

I’ve read your article about adjacency lists vs. nested sets in Oracle. A nice explanation, thanks.

You write that recursive operations are row-based in Oracle and set-based in SQL Server.

Which approach is better in your opinion?

First, a quick reminder for those who have no idea what’s this all about.

Recursive queries are useful to build the hierarchy trees, traverse graphs, generate arbitrary rowsets etc. Usually this involves joining a set with itself an arbitrary number of times.

A recursive query is usually defined by the anchor part (the initial rowset, or the base of recursion) and the recursive part (some operation over the previous resultset).

This is same as with the plain recursion: take some initial data as a parameter, modify it in some way and then pass the result of the modification as a new parameter to the same function.

Since SQL operates with sets, the input and the output if this function should also be the sets. And here is the main differences between the approaches used by Oracle on one side, and SQL Server, PostgreSQL and some more systems on the other side.

Oracle uses CONNECT BY approach, while SQL Server uses recursive CTE approach.

CONNECT BY

A hierarchical query using a CONNECT BY clause looks like this:

SELECT *
FROM mytable
START WITH start_with_condition
CONNECT BY connect_by_condition (PRIOR mytable)

This query uses two clauses: START WITH defines the anchor part, while CONNECT BY defines the recursive parts.

It’s an analog of this query:

SELECT *
FROM mytable m1
CROSS APPLY ( SELECT m1.* UNION ALL SELECT * FROM mytable m2 CROSS APPLY ( SELECT m2.* UNION ALL SELECT * FROM mytable m3 CROSS APPLY ( SELECT m3.* UNION ALL … ) WHERE connect_by_condition(m2.*) ) WHERE connect_by_condition(m1.*) )
WHERE start_with_condition

On each iteration, exactly one record from the previous table is used. Each innermost query can use the values of only one record of the outermost set.

Here’s an example rowset:

ID	Parent
1	NULL
2	NULL
3	1
4	1
5	2
6	2

and a query over this rowset:

SELECT *
FROM mytable
START WITH parent IS NULL
CONNECT BY parent = PRIOR id

The anchor part consists of two rows:

ID	Parent
1	NULL
2	NULL

, but they are not returned as is. After a row had been returned, the recursive part is being applied to the row:

ID	Parent
1	NULL
Recursive condition applied: `parent = 1`
2	NULL
Recursive condition applied: `parent = 2`

As you can see, the recursive step can only take the value of a single previous record as an input, while producing a resultset as an output.

The recursive steps are applied to each record independently.

CONNECT BY recursion takes a single record as input and returns a set as an output. It is scalar by nature.

Recursive CTE

A recursive CTE looks like this:

WITH rows AS ( SELECT * FROM mytable WHERE anchor_condition UNION ALL SELECT * FROM set_operation(mytable, rows) )
SELECT *
FROM rows

It’s the same as the following:

WITH rows AS ( SELECT * FROM mytable WHERE anchor_condition ), rows2 AS ( SELECT * FROM set_operation(mytable, rows) ), rows3 AS ( SELECT * FROM set_operation(mytable, rows2) ), …
SELECT *
FROM rows
UNION ALL
SELECT *
FROM rows2
UNION ALL
SELECT *
FROM rows3
UNION ALL
…

A set_operation can be any set operation: a JOIN, a UNION, an EXCEPT, whatever. It may involve any number of other tables, functions, in other words everything — as long as it’s result (also a set, of course) has the same number and datatypes of columns as the anchor part does.

Here’s the example on the same sample data:

ID	Parent
1	NULL
2	NULL
3	1
4	1
5	2
6	2

, but with a new query:

WITH rows AS ( SELECT * FROM mytable UNION ALL SELECT mytable.* FROM rows JOIN mytable ON mytable.parent = rows.id )
SELECT *
FROM rows

The anchor part is the same:

ID	Parent
1	NULL
2	NULL

and, unlike the CONNECT BY, it is returned at once. The recursive part is only applied when the previous resultset is known, and it’s applied to the resultset as a whole:

ID	Parent
1	NULL
2	NULL
Recursive part: a `JOIN` with `(1, 2)`

, which yields all records of the original table (though not in the tree order)

CTE recursion takes a set on input and returns a set on output. It is set-based by nature.

Conclusions

From this, two importation conclusions can be drawn.

CONNECT BY approach splits the sets returned by the recursion and applies the recursion to each individual record. This is less powerful, since a record is just a special case of a set, but the results naturally come out in the recursion order.
CTE approach allows the recursion to process the set returned on the previous step as a whole. This is more powerful, since a set is a general case of a record, but it takes some extra effort to sort the resultset in the tree order had the need arose.

Everything that can be done using a CONNECT BY approach can be done with a recursive CTE as well, but not vice versa. That’s why it’s safe to say that recursive CTE approach is more powerful.

However, what’s good in theory, is not always as good in implementation.

SQL Server’s implementation

SQL Server uses recursive CTE approach which is set based and more powerful. But SQL Server places some limitations on what you can use in the recursive part of the CTE. And severe limitations they are. You cannot use aggregates, cannot use OUTER JOINs, cannot use TOP. Here’s the complete list.

What’s the reason behind that?

Let’s see again the analogs of these two approaches:

CONNECT BY:

SELECT current_result.*
FROM previous_result
CROSS APPLY ( SELECT previous_result.* UNION ALL SELECT * FROM mytable WHERE condition(previous_result.*) ) AS current_result

Recursive CTE:

SELECT previous_result.*
FROM previous_result
UNION ALL
SELECT current_result.*
FROM set_operation(previous_result, mytable) AS current_result

The former query can always be rewritten as a latter:

SELECT previous_result.*
FROM previous_result
UNION ALL
SELECT current_result.*
FROM previous_result
JOIN mytable
ON condition(previous_result)

In the CROSS APPLY, each row of the previous is returned, then the scalar condition which returns true or false is checked for each row of mytable and current row of the previous_result. This can be rewritten as a JOIN.

The reverse is not always true.

Under which conditions it is possible for a recursive CTE query to be rewritten as a CROSS APPLY query?

For this to be possible, the set_operation used in the second query should distribute over the UNION ALL operation.

What does it mean?

The distributivity is a well-known concept in algebra.

For instance, the multiplication is distributed over addition, this means that x × (y + z) = (x × y) + (x × z) is always true.

But further generalization, power, does not distribute over multiplication. The following condition: x ^ (y × z) = (x ^ y) × (x ^ z) does not always hold.

In other words, distributivity means that the order of operations is not important. You can add the results of multiplication just as well as multiply the results of addition, the final result will always be the same.

This was arithmetics, but concept of distributivity can be applied to relational algebra just as well. A set operation distributes over UNION ALL if the following condition always holds:

set_operation(A UNION ALL B, C) = set_operation(A, C) UNION ALL set_operation(B, C)

It’s easy to see that if the condition above is true, the two approaches to the recursion become the same.

Indeed, the operation over the UNION ALL is the same as the UNION ALL over the operations, which is the same as the operation applied to each record of the previous_result. And the latter is exactly what CROSS APPLY does: applies an operation to each record of the set and returns the resultsing sets combined together (as if with UNION ALL).

A JOIN distributes over UNION ALL. However, DISTINCT, aggregates, TOP and some other operations do not.

Maximum over a UNION ALL is not the same as the UNION ALL of the maximums. TOP of a UNION ALL is not the same as the UNION ALL of TOPs, etc.

If we look closer into the restrictions put on the recursive operations by SQL Server we will see that placing these restrictions have one ultimate goal: to forbid all operations that do not distribute over UNION ALL.

Aggregates are forbidden, TOP is forbidden, DISTINCT is forbidden. This is a conspiracy.

However, they cannot lie to the world all the time. They cannot conceal every fact. There are two operations they failed to forbid. And now it’s a moment to reveal the truth.

EXCEPT

Consider a sample rowset:

ID
1
2
3

Now, let’s make a recursive query over this rowset:

WITH mytable (id) AS ( SELECT 1 UNION ALL SELECT 2 UNION ALL SELECT 3 ), q AS ( SELECT id FROM mytable UNION ALL SELECT * FROM ( SELECT id FROM othertable EXCEPT SELECT * FROM q ) q2 )
SELECT TOP 20 *
FROM q

What should this query return?

Being rewritten as simple SQL, it yields this:

WITH mytable (id) AS ( SELECT 1 UNION ALL SELECT 2 UNION ALL SELECT 3 ), anchor AS ( SELECT * FROM mytable ), recursive1 AS ( SELECT * FROM mytable EXCEPT SELECT * FROM anchor ), recursive2 AS ( SELECT * FROM mytable EXCEPT SELECT * FROM recursive1 ), recursive3 AS ( SELECT * FROM mytable EXCEPT SELECT * FROM recursive2 ), recursive4 AS ( SELECT * FROM mytable EXCEPT SELECT * FROM recursive3 )
-- , recursive5 AS
-- …
SELECT *, 'anchor' AS setname
FROM anchor
UNION ALL
SELECT *, 'recursive1' AS setname
FROM recursive1
UNION ALL
SELECT *, 'recursive2' AS setname
FROM recursive2
UNION ALL
SELECT *, 'recursive3' AS setname
FROM recursive3
UNION ALL
SELECT *, 'recursive4' AS setname
FROM recursive4
-- UNION ALL
-- …

The setname above is for the sake of clarity only, to distinguish between the sets combined with UNION ALL.

Here’s what this query returns:

id	setname
1	anchor
2	anchor
3	anchor
1	recursive2
2	recursive2
3	recursive2
1	recursive4
2	recursive4
3	recursive4
9 rows fetched in 0.0003s (0.0006s)

We see that the query returns the whole anchor part which is just a SELECT * FROM mytable:

id	setname
1	anchor
2	anchor
3	anchor

The next part, recursive1, applies the EXCEPT operator to the mytable and anchor:

SELECT *
FROM mytable
EXCEPT
SELECT *
FROM anchor

But since mytable and anchor actually match, the result of this operation is an empty rowset. And that’s why we don’t see recursive1 in the resultset above.

The next part, recursive2, applies EXCEPT again:

SELECT *
FROM mytable
EXCEPT
SELECT *
FROM recursive1

This time, the second argument to EXCEPT is empty, and each record from mytable is returned:

id	setname
1	recursive2
2	recursive2
3	recursive2

, etc.

EXCEPT applied to mytable and an empty set yields a copy of mytable. EXCEPT applied to the mytable and a copy of mytable yields an empty set. Therefore, the final resultset will be endless, consisting of alternating empty sets and copies of mytable.

The original recursive query, in theory, should return the same rowset. Let’s check if it’s really so:

WITH mytable (id) AS ( SELECT 1 UNION ALL SELECT 2 UNION ALL SELECT 3 ), othertable (id) AS ( SELECT 4 UNION ALL SELECT 5 ), q AS ( SELECT id FROM mytable UNION ALL SELECT * FROM ( SELECT id FROM mytable EXCEPT SELECT * FROM q ) q2 )
SELECT TOP 20 *
FROM q
OPTION (MAXRECURSION 4)

id
1
2
3
1
2
1
3
1
2
1
3

This differs much from what we expected to see.

Let’s check what happened.

The anchor part returned the first three rows just as it had to:

id
1
2
3

The next set (recursive1) should have been empty. But it has 2 rows instead:

id
1
2

This could only happen it the EXCEPT clause was applied to the last record of the anchor set rather than to the whole anchor:

SELECT *
FROM mytable
EXCEPT
SELECT TOP 1 *
FROM anchor
ORDER BY id DESC

The query above would yield the same results.

In recursive2, the EXCEPT operator, had it been applied to the whole resultset returned by the previous query, would yield a single row 3. However, it returns 1 and 3, which means that EXCEPT clause was again applied only to the last row.

This is not how a properly designed set-based recursive CTE should behave. But this is exactly how CONNECT BY query behaves. It splits the recordset into separate records and applies the recursive part to each record individually.

ROW_NUMBER

Analytical functions without PARTITION BY clause do not distribute over UNION ALL as well.

A UNION ALL of two queries containing the ROW_NUMBER’s is not the same as a ROW_NUMBER applied to the results of a single UNION ALL query. The latter would return a single consecutive counter, while the former would count the records in each query separately.

ROW_NUMBER’s, however, are still allowed in SQL Server 2005.

Consider this adjacency list rowset defining a tree:

ID	Parent
1	NULL
2	NULL
3	1
4	1
5	2
6	2

and make a simple tree query over it:

WITH mytable (id, parent) AS ( SELECT 1, NULL UNION ALL SELECT 2, NULL UNION ALL SELECT 3, 1 UNION ALL SELECT 4, 1 UNION ALL SELECT 5, 2 UNION ALL SELECT 6, 2 ), q AS ( SELECT *, ROW_NUMBER() OVER (ORDER BY id) AS rn FROM mytable WHERE parent IS NULL UNION ALL SELECT mytable.*, ROW_NUMBER() OVER (ORDER BY mytable.id) FROM q JOIN mytable ON mytable.parent = q.id )
SELECT *
FROM q

The tree is only two levels deep, that’s why this query can be rewritten as this:

WITH mytable (id, parent) AS ( SELECT 1, NULL UNION ALL SELECT 2, NULL UNION ALL SELECT 3, 1 UNION ALL SELECT 4, 1 UNION ALL SELECT 5, 2 UNION ALL SELECT 6, 2 ), anchor AS ( SELECT *, ROW_NUMBER() OVER (ORDER BY id) AS rn FROM mytable WHERE parent IS NULL ), recursive1 AS ( SELECT mytable.*, ROW_NUMBER() OVER (ORDER BY mytable.id) AS rn FROM anchor JOIN mytable ON mytable.parent = anchor.id )
SELECT *
FROM anchor
UNION ALL
SELECT *
FROM recursive1

id	parent	rn
1		1
2		2
3	1	1
4	1	2
5	2	3
6	2	4
6 rows fetched in 0.0004s (0.0012s)

The recursive part returns 4 records and calculates a ROW_NUMBER over this set. The values returned by ROW_NUMBER are from 1 to 4, just as they should be.

Now, let’s check the recursive query:

WITH mytable (id, parent) AS ( SELECT 1, NULL UNION ALL SELECT 2, NULL UNION ALL SELECT 3, 1 UNION ALL SELECT 4, 1 UNION ALL SELECT 5, 2 UNION ALL SELECT 6, 2 ), q AS ( SELECT *, ROW_NUMBER() OVER (ORDER BY id) AS rn FROM mytable WHERE parent IS NULL UNION ALL SELECT mytable.*, ROW_NUMBER() OVER (ORDER BY mytable.id) FROM q JOIN mytable ON mytable.parent = q.id )
SELECT *
FROM q

id	parent	rn
1		1
2		2
5	2	1
6	2	2
3	1	1
4	1	2
6 rows fetched in 0.0003s (0.0011s)

See now?

The ROW_NUMBER’s are not consecutive. Instead of a single sequence from 1 to 4, there are two sequences from 1 to 2.

Again, that’s exactly what a CONNECT BY query (or a CROSS APPLY) would return.

Summary

There are two approaches to implementing a recursive query:

One approach is record-based. A recursive part takes a record as a parameter and returns a set. It is then separately applied to each record of the set returned . This approach is implemented by Oracle in CONNECT BY queries.
Another approach is set-based. A recursive part takes a set as a parameter, processes it as a whole and returns a new set. It is then passed as a parameter to the next iteration. This approach can be implemented by recursive CTE’s

The record-based approach is better for building hierarchies because it naturally returns the records in the tree order. However, it’s less powerful in theory.

The set-based approach is more powerful in theory, since it can process a set as a whole and more complex conditions can be applied. Extra effort is required to sort the resulting recorset in the tree order but it is still possible though less efficient.

In theory, recursive CTE’s are more powerful and therefore should be preferred.

SQL Server nominally impelements recursive CTE’s, but in fact they are no more than CONNECT BY queries in disguise. No real set-based operations can be performed over these CTE’s, only the record-based ones.

SQL Server efficiently combines limitations of the record-based approach used by CONNECT BY with the absence of the natural tree ordering implied by the set-based approach used by recursive CTE’s.

That’s why I’d say that Oracle as of 10g implements recursive queries better than SQL Server as of 2005.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

IE8 SmartScreen in action

ieblog — Mon, 23 Nov 2009 13:32:00 -0800

[Review] GGL Review ends today, November 22nd

Hartmut Kaiser — Sun, 22 Nov 2009 15:59:13 -0800

toast version 1.469 (2009-11-21)

(author unknown) — Sat, 21 Nov 2009 14:58:58 -0800

toast version 1.468 (2009-11-21)

(author unknown) — Sat, 21 Nov 2009 14:58:58 -0800

[1.41.0] PDF Build of documentation available.

John Maddock — Fri, 20 Nov 2009 13:24:21 -0800

Announcing the Chromium OS Open Source Project

Sarah Nahm — Thu, 19 Nov 2009 17:49:00 -0800

Today we released Chromium OS, the open source project behind Google Chrome OS. Google Chrome OS is an operating system that is intended for people who spend most of their time on the web. It aims to provide a computing experience that is fast, simple and secure. The Chromium OS project as you'll see it today is comprised of the code that has been developed thus far, our early experiments with the user interface, and detailed design docs for many parts that are under active development.

To learn more about what Google Chrome OS is, watch this short video:

To get a feel for the Google Chrome OS user experience, you can watch the demo from this morning's announcement event.

Posted by Kan Liu, Product Manager

ANNOUNCE: GIMP 2.7.0

Martin Nordholts — Sun, 16 Aug 2009 19:18:39 -0700

ANNOUNCE: GIMP 2.6.7

Sven Neumann — Sat, 15 Aug 2009 21:02:56 -0700

Blender Conference: videos

(author unknown) — Thu, 19 Nov 2009 11:13:00 -0800

showing pop-ups

djcb — Thu, 19 Nov 2009 10:10:00 -0800

Updated: yes, it's %s, not %d Sometimes, it's nice when emacs can warn you when something is happening or should happen. For example, when a new e-mail has arrived, or when there's a meeting in 15 minutes you should attend.

As always, there are different way to do this, but here's what I've been using for while. Various versions of this have been circulating around mailing lists, so I don't know whom to credit with the original idea – anyway, this is the (modified) version that I'm using.

(defun djcb-popup (title msg &optional icon sound) "Show a popup if we're on X, or echo it otherwise; TITLE is the title
of the message, MSG is the context. Optionally, you can provide an ICON and
a sound to be played" (interactive) (when sound (shell-command (concat "mplayer -really-quiet " sound " 2> /dev/null"))) (if (eq window-system 'x) (shell-command (concat "notify-send " (if icon (concat "-i " icon) "") " '" title "' '" msg "'")) ;; text only version  (message (concat title ": " msg))))

A couple of notes:

I'm using notify-send for sending notifications; this assumes you are using that system (it's part of the libnotify-bin package in Debian/Ubuntu). You can of course replace it with whatever is available on your system. Alternatives are zenity or kdialog or xmessage (for old-timers) and their equivalents (?) on Windows, MacOS.
I'm now using mplayer for playing sounds. This is a bit heavy, but at least plays all kinds of audio files. If you only care about .wav-files, you could replace it with e.g. aplay;
as always, please ignore my ego-centric function names :-)

Now, we can use this function by evaluation e.g.

(djcb-popup "Warning" "The end is near" "/usr/share/icons/test.png" "/usr/share/sounds/beep.ogg")

showing pop-ups from `org-mode` appointments

The above popup function is most useful when it's does its work based on some event. To be notified of appointments and the like, there is the emacs appt facility. Here, we set up this appt, and then hook it up with org-mode, so appt can warn us when there's something happening soon…

;; the appointment notification facility
(setq appt-message-warning-time 15 ;; warn 15 min in advance  appt-display-mode-line t ;; show in the modeline
 appt-display-format 'window) ;; use our func
(appt-activate 1) ;; active appt (appointment notification)
(display-time) ;; time display is required for this...
 ;; update appt each time agenda opened (add-hook 'org-finalize-agenda-hook 'org-agenda-to-appt) ;; our little façade-function for djcb-popup
 (defun djcb-appt-display (min-to-app new-time msg) (djcb-popup (format "Appointment in %s minute(s)" min-to-app) msg "/usr/share/icons/gnome/32x32/status/appointment-soon.png" "/usr/share/sounds/ubuntu/stereo/phone-incoming-call.ogg")) (setq appt-disp-window-function (function djcb-appt-display))

Of course, you can freely choose a icon / sound to your liking.

showing pop-ups for new mail

Another event you might want to be warned about is new mail. There is something to be set for not letting yourself be disturbed for new mail, but if you sufficiently filter your mails before they enter your inbox, it can be a good way to periodically bring you back from your deep sl ^H^H thinking. For Wanderlust, I use something like this:

(add-hook 'wl-biff-notify-hook (lambda() (djcb-popup "Wanderlust" "You have new mail!" "/usr/share/icons/gnome/32x32/status/mail-unread.png" "/usr/share/sounds/ubuntu/stereo/phone-incoming-call.ogg")))

Exercise for the reader: adapt this for your chosen mail client.

Boost release 1.41.0 available

Beman Dawes — Wed, 18 Nov 2009 18:26:25 -0800

An Early Look At IE9 for Developers

ieblog — Wed, 18 Nov 2009 09:23:00 -0800

John Bailey: The Quest for Perfection

noreply@blogger.com (John) — Tue, 17 Nov 2009 03:37:58 -0800

Flash Player 10.1 Beta

Mike Melanson — Tue, 17 Nov 2009 00:01:00 -0800

The beta of Flash Player 10.1 is available for download via labs.adobe.com. For this beta release, there is only a 32-bit plugin available for Linux (64-bit users will still need to use the previously released alpha version).

Bugs can be reported and tracked here.

[Review] GGL review extended until November 22nd

Hartmut Kaiser — Sun, 15 Nov 2009 14:50:09 -0800

15 November 2009

(author unknown) — Sat, 14 Nov 2009 04:39:27 -0800

My Favorite IE Add-on: Mouse Gestures by Ralph Hare

ieblog — Fri, 13 Nov 2009 14:45:00 -0800

copying lines without selecting them

djcb — Thu, 12 Nov 2009 11:03:00 -0800

When I'm programming, I often need to copy a line. Normally, this requires me to first select ('mark') the line I want to copy. That does not seem like a big deal, but when I'm in the 'flow' I want to avoid any little obstacle that can slow me down.

So, how can I copy the current line without selection? I found a nice trick by MacChan on EmacsWiki to accomplish this. It also adds ta function to kill (cut) the current line (similar to kill-line (C-k), but kills the whole line, not just from point (cursor) to the end.

The code below simply embellishes the normal functions with the functionality 'if nothing is selected, assume we mean the current line'. The key bindings stay the same (M-w, C-w).

To enable this, put the following in your .emacs:

(defadvice kill-ring-save (before slick-copy activate compile) "When called interactively with no active region, copy a single line instead." (interactive (if mark-active (list (region-beginning) (region-end)) (message "Copied line") (list (line-beginning-position) (line-beginning-position 2))))) (defadvice kill-region (before slick-cut activate compile) "When called interactively with no active region, kill a single line instead." (interactive (if mark-active (list (region-beginning) (region-end)) (list (line-beginning-position) (line-beginning-position 2)))))

It also shows the power of Emacs-Lisp with the defadvice-macro – see the fine documentation. Using defadvice, you can 'decorate' any function with your own modifications. This great power should be used with caution, of course, as to not break other usage that assumes the undecorated versions. In this case, that seem unlikely. And note that the 'advise' only applies when the functions are called interactively.

bookmarks

djcb — Thu, 05 Nov 2009 11:52:00 -0800

Emacs has a very useful system for bookmarks – shortcuts to often-used files. It's also one of those features I only really started using after years of emacs – there seem to be many of such obvious features…

Bookmarks are especially handy if you have long file names, or for examples the special file names for editing root-owned files discussed here before.

To start using bookmarks effectively, there are only a few important key bindings to memorize: C-x r m ('make') will create a new bookmark, defaulting to the current file. Then, you can jump to an existing bookmark with C-x r b ('bookmark') Finally, you can see the list of your bookmarks with C-x r l ('list').

There are a few customizations you can put in your .emacs:

(setq bookmark-default-file "~/.emacs.d/bookmarks" ;; keep my ~/ clean
 bookmark-save-flag 1) ;; autosave each change)

[boost] [Review Results] Boost.Polygon libraryaccepted into boost

Fernando Cacciola — Thu, 05 Nov 2009 15:21:32 -0800

Google Bar Code

Puzzle Solver Wannabe — Tue, 06 Oct 2009 23:15:00 -0700

Google is celebrating the invention of the bar code on its home page.

If you have a bar code scanner on your iPhone or Android phone, scan the bar code and...

If you can figure this out, you may have a future with Google.

Puzzle Solver Wannabe — Mon, 21 Sep 2009 20:17:00 -0700

8 M L D Q 6 T U I
6 T F M L R H A A
N R A 6 Q 8 E F L
D M Q 8 6 I I 2 O 3
2 S 5 J 1 3 J X O J

via TechCrunch's Google Is Searching For Beautiful Minds, But So Far No M.I.T. Students Have Broken Its Code.

Crop Circles. 51.327629, -0.5616088. puzzle?

Puzzle Solver Wannabe — Mon, 14 Sep 2009 21:05:00 -0700

query: crop circles

clue: 51.327629, -0.5616088

Unexplained Phenomenon. Puzzle?

Puzzle Solver Wannabe — Fri, 04 Sep 2009 22:52:00 -0700

query: unexplained phenomenon

Clue:
twitter @google: 1.12.12 25.15.21.18 15 1.18.5 2.5.12.15.14.7 20.15 21.19

Google Engineering Interview Questions

Puzzle Solver Wannabe — Wed, 02 Sep 2009 23:40:00 -0700

Some readers have emailed and pointed out that some of these puzzles are example questions asked at Google engineering interviews. Wei Hwa is a Google software engineer and these puzzles provide a great insight into Google engineering hires. Why didn't I think of this earlier! What a great resource of puzzles from Google. Several users have used the answering format from this blog and got very far into the interview process. Their tips were

use analytical analysis
think out loud when solving the problem
use code to solve problems
learn about Google. Google, Google, Google - use gmail on your resume!
look sharp, pop a mint, be confident, and look good

Check out the archives. Don't forget to read the comments. Lots of user answers.

Not sure if they were hired. Best of luck and hire me when you get in. :)

The aftermath

Johan Jeuring — Wed, 24 Oct 2007 05:53:00 -0700

We are wrapping up our efforts on the ICFP Programming Contest 2007. Most contest-related matters have been finished by now.

If you want to read more about the ICFP Programming contest, you should have a look at our report about it.

You can order a t-shirt with the after picture on the back, and one picture from the arrival sequence on the front from cafepress, or create your own t-shirt using these materials (4.3MB).

If you want to see the presentation we gave about the contest at ICFP 2007, have a look at our video (with an explanation about how to solve the contest problem starting at 18:00, and Endo appearing at around 1:12:00).

The question we got after the contest was:

How much time did you spend creating it?

We have different answers to this question, but probably the one that comes closest to the truth is: as much as we enjoyed creating it.

Endo lives!

Johan Jeuring — Wed, 10 Oct 2007 00:20:00 -0700

At the International Conference on Functional Programming (ICFP) in Freiburg last week, we announced the results of the ICFP Programming Contest 2007.

Just as last year, the first prize was won by Team Smartass from Google research.

The second prize was one by the United Coding Team from the University of Cape Town, South Africa.

The judges' prize was won by Celestial Dire Badger (Jed Davis).

The highest survival chance produced during the contest (by Team Smartass) was over 90%, so Endo survived! Endo actually came with us to Freiburg to say `thank you' to the people present.

Later in the week he was spotted by several ICFP attendants in the city of Freiburg, running around and enjoying himself.

Tickling grass

Alexey — Fri, 17 Aug 2007 06:59:00 -0700

As you all already know, Endo's chance of survival is also affected by other organisms and objects in his environment. One of the organisms that has recently attracted our attention is grass. It turns out that with just a few modifications to Endo's DNA you can make grass grow in completely different places! We have not discovered, however, how to make it grow at the right locations. Take a look at what we tried:

IIPIFFCPICCFPICICFPPICICIPICIIICCCIICIIICCCIIICCCP
IICIPIIICCPIIPIICIIPIPIIICCPIICIIPIICIPIIICCPIICIP
PPCCCFFCCCCCCCCCCCCCCCCCCICIPPCPFCCCFFCCFCCCCCCCCC
CCFFFICIPPCCPCFCFCFFCCCCCFCFFCCCCCCCICIICIIPIFFCPI
CCFPICICFPPICICIPICCIIICICIIICIICCIIICCCPIICIPIIIC
CPIIPIICIIPIPIIICCPIICIIPIICIIPIPIIICCPIICIIPIICIP
IIICCPIICIPPPCCCCFFCCCCCCCCCCCCCCCCCICIPPCPCCFFFCC
FCFFFCCFFFFFFCFFICIPPCCPCFFCFFCCCCFCCCCCFCCCFFFICI
PPCICPCFFFCFFFFFCCCCCCFCCCCCCICIIC

This prefix enables grass repositioning and redefines the function bioMul (see initial conditions help page) as follows:

bioMul x y = bioAdd x y

It seems that the bioMul function influences where grass is going to grow. We are still searching for the right definition of bioMul, and we are confident that when we find it, grass will grow at the right locations. Do you want to help us figure out how to fix bioMul? If you do, you might find it helpful to understand first how bioAdd is encoded in DNA.

Finding palindromes

Johan Jeuring — Thu, 09 Aug 2007 23:25:00 -0700

[Since this blog message has received thousands of hits since I wrote it in 2007, I decided to release the program described in this message. Visit http://www.jeuring.net/Palindromes/to obtain the software.]

A palindrome is a number, verse, word or a phrase that is the same whether you read it backwards or forwards, for example the word `refer'.

When I started as a PhD student in Computer Science in 1988, my professor gave the following problem assignment to me. Construct an algorithm that, when given a string, finds the longest palindrome substring occurring in the string. For example, given the string "yabadabadoo", the algorithm should return the substring "abadaba". The algorithm should not only return the longest palindrome substring, it should also return it fast. The requirement was that the algorithm should only use a number of steps linear in the length of the argument string. This was a difficult problem, which gave me severe headaches in the months to follow. I spent four months on the problem, and solved it. The one comment I got from my professor was "The best thing about this problem and its solution is that they have absolutely no practical relevance."

He was wrong.

Since 1988 I have found that palindromes play an important role in the world around us. Palindromes appear in music, in genomes, in art, architecture and design, mathematics, physics, chemistry, etc.

Particularly interesting is the fact that palindromes occur in the male genome: of the about 20,000,000 characters representing the male-specific region of the Y chromosome, 5,700,000 characters form together eight large palindromes (off by a couple of characters). I don't think there is any proven explanation for the presence of palindromes in the male genome, but I suspect it has something to do with repairing genomes. If you know more about it: please let me know.

As you may or may not have noticed when working on repairing Endo's DNA, the cloud is corrupted. It turns out that the cloud's genome also is a palindrome (off by a couple of characters), and that the corruption can be repaired by taking the mirror image of the palindrome. To do so, you have to find the palindrome. In the case of Endo, this is pretty easy: `duolc' follows cloud immediately in the symbol table. Had this information not been in the symbol table, you would have to find this palindrome in the DNA by means of an algorithm for finding palindromes. Actually, the DNA for the cloud is only 6,500 acids long, and using a naive quadratic-time algorithm for finding palindromes is not a real problem if you have a sufficiently fast machine. A quadratic-time algorithm for determining palindromes in human male DNA isn't going to work: it would take a long time to find the palindromes.

This blog message explains how to find palindrome substrings in linear time. I will develop a program in Haskell for finding exact palindromes. The problem of finding approximate palindromes is left as an exercise to the reader :-) Interestingly, the algorithms for finding palindromes developed by computational biologists generally use suffix trees, and both space and time consumption seem to be worse compared with the algorithm I give in this blog message. But descriptions of the algorithm given here have been published in the eighties (by Galil and others, described in RAM-code) and nineties (by me, described in Squiggol, which is probably even harder to decypher than RAM-code) of the previous century, which is a long time ago of course.

This blog message is a literate Haskell file, which can be interpreted with ghci.

The type of a function for finding palindromes

I want to find the longest palindrome substring in a string. The first attempt at a type for a function longestPalindrome is hence:

longestPalindrome :: String -> String

To determine whether or not a string is a palindrome, I have to compare characters at different positions in a list. It would be helpful if I had random access into the string. For that purpose, I'm going to change the input type of the longestPalindrome function to an array. Furthermore, the longestPalindrome algorithm can also be used to find the longest palindrome in a list of integers, or any other type on which I can define an equality function. So here is the second attempt at a type for a function for finding the longest palindrome:

longestPalindrome :: Eq a => Array Int a -> Array Int a

To find the longest Palindrome in an array, I have to calculate the longest palindrome around each position of the array, where a position in an array is either on an element, or before or after an element. For example, the array corresponding to [1,2,3] has 7 positions: before the 1, on the 1, before the 2, ..., until after the 3. So I want to express the function longestPalindrome in terms of a function longestPalindromes of type

longestPalindromes :: Eq a => Array Int a -> [Int]

where the result list is the list of the lengths of the longest palindrome around each position in the argument array. For example,

?longestPalindromes (arrayList (0,2) [1,2,2])
[0,1,0,1,2,1,0]

I will omit the definition of longestPalindrome in terms of longestpalindromes, and define function longestPalindromes below.

A naive algorithm for finding palindromes

A naive algorithm for finding palindromes calculates all positions of an input array, and for each of those positions, calculates the length of the longest palindrome around that position.



> module Palindromes where
> 
> import Data.Array
>
> longestPalindromesQ :: Eq a => Array Int a -> [Int]
> longestPalindromesQ a = 
> let (afirst,alast) = bounds a
> positions = [0 .. 2*(alast-afirst+1)]
> in map (lengthPalindromeAround a) positions

Function lengthPalindromeAround takes an array and a position, and calculates the length of the longest palindrome around that position.



> lengthPalindromeAround :: Eq a => Array Int a 
> -> Int 
> -> Int
> lengthPalindromeAround a position 
> | even position = 
> extendPalindromeAround (afirst+pos-1) 
> (afirst+pos) 
> | odd position = 
> extendPalindromeAround (afirst+pos-1) 
> (afirst+pos+1) 
> where pos = div position 2
> (afirst,alast) = bounds a
> extendPalindromeAround start end = 
> if start < 0 
> || end > alast-afirst 
> || a!start /= a!end
> then end-start-1
> else extendPalindromeAround (start-1) 
> (end+1)

For each position, this function may take an amount of steps linear in the length of the array, so this is a quadratic-time algorithm.

A linear-time algorithm for finding palindromes

I now describe a linear-time algorithm for finding palindromes. Although the program is only about 15 lines long, it is rather intricate. I guess that you need to experiment a bit with to find out how and why it works.

The algorithm processes the input array from left to right. It maintains the length of the current longest tail palindrome, and a list of lengths of longest palindromes around positions before the center of the current longest tail palindrome, in reverse order. Function longestPalindromes is expressed in terms of function extendTail.



> longestPalindromes :: Eq a => Array Int a -> [Int]
> longestPalindromes a = 
> let (afirst,alast) = bounds a
> in extendTail a afirst 0 []

Function extendTail takes an array as argument, the current position in the array, the length of the current longest tail, and a list of lengths of longest palindromes around positions before the center of the current longest tail palindrome, in reverse order. There are four cases to be considered in function extendTail. If the current position is after the end of the array, we only have to add the final palindromes in the array to the list of longest palindromes, for which we use the function finalCentres. If the current longest tail palindrome extends to the start of the array, we extend the list of lengths of longest palindromes by means of function extendCentres. If the element at the current position in the array equals the element before the longest tail palindrome we extend the longest tail palindrome. If these elements are not equal, we extend the list of length of longest palindromes.



> extendTail a n currentTail centres 
> | n > alast = 
> -- reached the end of the array 
> finalCentres currentTail centres 
> (currentTail:centres)
> | n-currentTail == afirst = 
> -- the current longest tail palindrome 
> -- extends to the start of the array
> extendCentres a n (currentTail:centres) 
> centres currentTail 
> | a!n == a!(n-currentTail-1) = 
> -- the current longest tail palindrome 
> -- can be extended
> extendTail a (n+1) (currentTail+2) centres 
> | otherwise = 
> -- the current longest tail palindrome 
> -- cannot be extended 
> extendCentres a n (currentTail:centres) 
> centres currentTail
> where (afirst,alast) = bounds a

Function extendCentres adds palindromes to the list of lengths of longest palindromes. It takes the array as argument, the current position in the array, the list of palindromes to be extended, and the list of palindromes around centres before the centre of the current longest palindrome tail. It uses the mirror property of palindromes to calculate longest palindromes around centres after the centre of the current longest palindrome tail. If the last centre is on the last element, we call extendTail with a longest tail palindrome of length 1, and the position moved tot he right. If the previous element in the centre list reaches exactly to the end of the last tail palindrome, use the mirror property of palindromes to find the longest tail palindrome. In the other cases, we've found the longest palindrome around a centre, and add that to the list of length of longest palindromes. We proceed by moving the centres one position, and calling extendCentres again.



> extendCentres a n centres tcentres centreDistance
> | centreDistance == 0 = 
> -- the last centre is on the last element: 
> -- try to extend the tail of length 1
> extendTail a (n+1) 1 centres
> | centreDistance-1 == head tcentres = 
> -- the previous element in the centre list 
> -- reaches exactly to the end of the last 
> -- tail palindrome use the mirror property 
> -- of palindromes to find the longest tail 
> -- palindrome
> extendTail a n (head tcentres) centres
> | otherwise = 
> -- move the centres one step
> -- add the length of the longest palindrome 
> -- to the centres
> extendCentres a n (min (head tcentres) 
> (centreDistance-1):centres) 
> (tail tcentres) (centreDistance-1)

Function finalCentres calculates the lengths of the longest palindromes around the centres that come after the centre of the longest tail palindrome of the array. These palindromes are obtained by using the mirror property of palindromes.



> finalCentres 0 tcentres centres = centres
> finalCentres (n+1) tcentres centres = 
> finalCentres n 
> (tail tcentres) 
> (min (head tcentres) n:centres)

At each step in this algorithm, either the longest tail palindrome is extended, and the current position in the array is moved, or the list of lengths of longest palindromes is extended. Since both the array, and the list of lengths of longest palindromes are linear in the length of the array, this is a linear-time algorithm.

The Major Imp Episodes

Jurriaan Hage — Sat, 04 Aug 2007 07:42:00 -0700

Although the Fuun typically want little to do with them, they do suffer from an inferiority complex when it comes to Imps. This is most evident in the large amount of extremely popular spy stories in which the Fuun novice invariably wins against a hard-boiled, sinister Imp character, who either dies (after a heroic attempt by the hero to save his life) or is converted to Fuun philosophy and thereafter serves the functional cause. A few short excerpts of such a story were recovered from Endo's DNA. It is still unclear whether these stories could actually play a role in saving Endo. For your amusement, we have included the very first episode below:

Only just before sleep-groggy roosters reared their scrawny heads, there was a hesitant knock on the steel door of the austere office of semi-colonel Void. "ENTER", rang out from the room, as though spoken in a medium-sized cathedral. Novice Imp entered the cramped room with some trepidation, knowing full well that many a (not so intelligent) agent had become deallocated after nary a slip of the tongue. Lieutenant Ptr, for example, had referred in His presence to "some unwanted side-effects" of a particular investigation. Ptr had promised that it wouldn't happen again, and Void had made sure that indeed it wouldn't.

Somewhat weak at the knees, Imp made for the middle of the room, where he stood at attention. "RELAX", spoke Void, in his usual teletype voice that belied the contents. "I have heard that among our recruits, you are the one adhering closest our revered commandos. We have a sensitive and dangerous mission scheduled and you have been chosen to volunteer..." (call-cc)

How did it go?

Johan Jeuring — Mon, 30 Jul 2007 03:41:00 -0700

The ICFP Contest 2007 ended last Monday. We have seen writeups of several teams that participated in the ICFP Contest. Many of those are interesting and sometimes funny reading. We particularly enjoyed

http://stereotype441.livejournal.com/45150.html

Marco Galotta collected a list of writeups, and posted them on his blog

http://marco-za.blogspot.com/2007/07/icfp-how-we-reached-top-15.html

We don't want to give our writeup now, we'll do that for ICFP in Freiburg, but we do want to briefly share our experiences.

We had a very hectic and pleasant time. I cannot remember I've had such an intense time at work before.

Headquarters was never empty during the 72 hours of the contest. We had a laptop showing several screens via a beamer: last 20 submissions, standings, the icfp mail account, several irc channels devoted to the contest, etc. Not much happened in the middle of the contest, but both the start and end were very hectic. The end was a thriller, I can assure you. But more about that at the ICFP conference in Freiburg.

Everything went fine. We haven't discovered an error in the task description yet. Initially we were worried about the slow start by many teams, but as time passed we were increasingly impressed by the creativity and cleverness of many teams. We have excellent winners. Our machines were unreachable twice for a short period, but that hasn't been a problem. Web statistics show that we processed millions of hits in these three days. All in all we are quite satisfied by how things went.

From now on we will prepare our presentation at Freiburg. By then we will know whether or not Endo survived. In the presentation we will discuss the problem, possible solutions, techniques we used to create the problem, solutions we have seen from the participants, the winners, etc. We will hand out a writeup about the contest in Freiburg.

Thank you

Johan Jeuring — Mon, 23 Jul 2007 04:03:00 -0700

Thank you all so much for participating.

It has been heart-warming to see all participants trying so hard to save the poor Endo. It was a hard struggle, but we are very optimistic Endo will survive. We have sent the best submitted prefix to Arrow, but haven't heard anything since. We hope that this is due to the fact that Arrow is using all its remaining energy to apply the DNA prefix to Endo, and not because something has gone wrong. For now, we can do nothing but wait for a sign of life. Repairing DNA takes time, but we expect that we will know whether or not it has been successful by the time of the ICFP Conference in Freiburg.

Emorphency!

Johan Jeuring — Fri, 20 Jul 2007 00:55:00 -0700

Ok, here's the story, as unbelievable as it sounds. It looks like the message was a call for help, sent by the ship that also appears in the sequence of pictures. The ship seems to be called Arrow and to be a sentient being, or at least intelligent. Apparently the story as told by the picture sequence has actually happened, with the final picture (decrypted by us today as expected) happening three months ago, about when the message was sent to us.

It is the story of Endo, a specimen of the extraterrestial Fuun. Endo was garbage-collected while sleeping in his(?) ship Arrow and unfortunately drifting into a heap of junk. Garbage collectors are apparently common in space and run by a species called the Imps. They like to drop junk on underdeveloped worlds (sorry, this is what Arrow told us). So the ship, together with lots of other stuff, was dropped on Earth. Arrow was damaged in the process, and before it could warn Endo that the atmospheric conditions of Earth are not very suitable for a Fuun, Endo left the ship – only to be hit by a cargo container! Endo is now in serious trouble. According to Arrow, the only chance to save Endo is to adapt him to the conditions that are prevalent on Earth by changing its DNA.

The hitch is that Arrow, being damaged itself, is running low on energy and cannot come up with a good modification itself. It can still perform the actual transformation process, but only if it gains knowledge of a suitable transformation soon. Arrow tells us it only has slightly more than 72 hours left before it will shut down!

During the last days, we have been able to decipher large parts of the original message, found out that it is about DNA and describes how Fuun DNA works. We therefore have some understanding now about the process involved. Unfortunately – and we feel really guilty about it – we have given all of this a low priority due to the upcoming contest, and the one thing we didn't find out until after we actually made contact with Arrow, was that this really is a cry for help, and how urgent it is! If only we had found out earlier, we would have had much more time to save Endo!

Because of that, and also because it is really the only thing we can think of to still give Endo a chance, we will change our plans. We will abandon the original topic we had in mind for the ICFP contest (writing generic programs in order to design boilerplates). Instead, we make it the task of the contest to Morph Endo!

(Of course, we still are not 100% sure if the message is actually authentic, but from what we've seen, there seems to be a lot of structure in the DNA we discovered, and a lot of information provided by Arrow, and we doubt someone would put so much effort into a joke.)

Contact

Johan Jeuring — Tue, 17 Jul 2007 22:24:00 -0700

We seem to have made contact with the author of the message. At least that's what we believe. Communication is still, well, difficult at best.

With the contest starting in a few days and the preparations more or less in place, we can now put a bit more effort into solving this mystery, and I am very curious where it will lead to.

The message

Johan Jeuring — Mon, 16 Jul 2007 01:35:00 -0700

You might remember the 'SPAM' message we received more than 3 months ago (see my message on April 26). For a long time, the only thing we managed to do with it is decrypt pictures, which already takes us a lot of time. We run the decryption process on one machine, and get about one picture per week. It seems there's only one picture left to decrypt, and that should be ready by Friday. The pictures appear to tell a story, but we cannot make much sense of it.

However, it was clear from the beginning that the pictures are only a small part of the message, and the rest remained a big mystery to us ... until last week, when Alexey suddenly had a brilliant idea that turned out to be a major breakthrough. Since then, we have not only deciphered a large part of the message, but we also have an idea about who sent it. It is all very strange, and we still do not know whether it all is a joke (maybe from an ex-student?) or whether it could be real. I'll keep you informed about our progress.

Hello, World!

Johan Jeuring — Fri, 13 Jul 2007 03:51:00 -0700

Feyeneyete loop?

Johan Jeuring — Fri, 06 Jul 2007 07:51:00 -0700

The team

Johan Jeuring — Wed, 04 Jul 2007 06:28:00 -0700

Let me introduce the team. I cannot disclose their contributions in detail, for obvious reasons.

Atze Dijkstra and Doaitse Swierstra. Contributed to the brainstorming sessions in 2005 and 2006.

Eelco Dolstra. I think if someone in the team qualifies for the label hacker, it will be Eelco. Eelco has participated in several contests. He has developed, amongst others, the central component for this years' contest. He is not a religious Haskell hacker (as many of us others are), and that has definitely been a Good Thing.

Chris Eidhof, Maaike Gerritsen, Jeroen Leeuwestein, Eelco Lempsink, Martijn van Steenbergen, Mark Stobbe. Student testers. Contributed to many aspects of the problem after they finished their test run.

Jurriaan Hage. Jurriaan has a broad background, and has contributed to several components of the task. Unfortunately, none of these can be disclosed.

Bastiaan Heeren. Enthusiastic member from the start. Has become the central organiser in the team, and is continuously adding and testing new ideas, implementing tools, pushing students to implement ideas, answering questions, etc. He knows most of what is going on, what still needs to be done, who is doing what, etc.

Stefan Holdermans and Arie Middelkoop. Young, starting, PhD students, who also have to write their first papers, so they cannot spend as much time as some of us others on the contest. I cannot disclose Arie's contributions; Stefan mainly worked on the task description. His perfectionism is exactly what we need for the task!

Johan Jeuring. Johan is the chairman of the organisation committee. He takes care of the `external relations'. Furthermore, he contributed to the problem and the task description. But he spends most of his time on organisational matters.

Andres Löh. One of our two external members. But he did his PhD here, and will become a lecturer very soon, so he cannot really be called external. Has participated in several contests (even won one!). Contributed many things: story, basic constructs, task description, theory development, etc.

Clara Löh. Our other external member. Did most of the artwork.

Alexey Rodriguez. Together with Eelco, I think Alexey is the other candidate for the hacker label. Has been away a couple of months to optimize GHC in GHC-headquarters, but has picked up very quickly again after he returned. His main contributions have to remain a secret for now. They're slightly obscure, but I hope you'll enjoy them anyway.

John van Schie. One of our students, and another hacker, who has implemented the communication components.

What's up?

Johan Jeuring — Thu, 28 Jun 2007 06:47:00 -0700

The process

Johan Jeuring — Mon, 25 Jun 2007 08:13:00 -0700

Ralf Hinze, the general chair of ICFP 2007 asked us already at the end of 2005 if we would like to organise the ICFP Programming Contest 2007. I tried to collect a group of people from the Software Technology group of the Computing Sciences department of Utrecht University that were willing to help. That turned out to be rather easy: the majority of the people I asked were enthusiastic about it. In the first months we had frequent meetings, to discuss the several aspects of organising such a contest. Furthermore, we extended the team with several young colleagues. In particular with people that had participated in a couple of previous contests, which was really helpful.

After two months we stopped meeting to wait for the 2006 contest. We participated in the 2006 contest with four members of the team. As I mentioned in the blog message about the 2006 contest, I think this was a great experience.

We resumed brainstorming at the end of summer 2006. We ended up with three or four problem ideas. Each idea was investigated by a small group of people, and in October we selected one of the ideas, based on the practical requirements for a task we have formulated in a previous blog message.

In the following months we implemented several components for the task, and we tried to solve the several problems we envisaged with our problem idea. In January 2007 we were pretty convinced that our problem idea had no fatal flaws, and we started working out the many details that we had left open.

Since then we have been working on the implementation of the several components, testing, and quality control.

New Years, live-ish with Green Day

Cory — Tue, 24 Nov 2009 00:20:12 -0800

I’ve gone to far too many events this past week. If I don’t put an end to this quick, I may become “normal”…

I took it to the next level tonight, though, by traveling into the future to see Green Day’s New Years Eve concert at LA Live. Well, sort-of. NBC was filming the concert with Carson Daly as the host. They gave us all new years hats and had us all count down from 30 as if the ball was dropping on New Years Eve. Apparently they’ll air this then. TV Magic!

After the count down, Green Day came on with a “Happy fucking New Year!” and rocked the stage for the rest of the night. Their performance was complete with explosions, fireworks, and a good mix between their newer stuff and the older stuff I was listening to on tape when I was eight years old (holy crap) in elementary school. Better yet, a friend I hadn’t seen in ages was down to visit family for the week, so I met her and her brothers there and caught up. Good times!

links for 2009-11-23

sogrady — Mon, 23 Nov 2009 17:02:14 -0800

Peter Thiel Says Don't Piss Off The Robots (Or Bet On A Recovery)

"[An artificially intelligent computer] could be very good, it could be very bad, it could be somewhere in between. Certainly we would hope that it would be friendly to human beings. At the same time, I dont think you'd want to be known as one of the human beings that is against computers and makes a living being against computers. So probably at the margins it would be prudent not to make a name for yourself as a anti- technological human being just as these computers are coming onto the scene."

(tags: peterthiel robots artificialintelligence ai humor)

Extensions: One Step Closer to the Finish Line

Ian Fette — Mon, 23 Nov 2009 11:32:00 -0800

During the last few months, our team has been working hard to support extensions in Google Chrome's beta channel. Today, we are getting one step closer to this goal; developers can now upload their extensions to Google Chrome's extension gallery. We are making the upload flow available early to make sure that developers have the time to publish their extensions ahead of our full launch.

You can find all the info to write an extension in our docs. Once your extension is ready for the gallery, you'll need to upload a zip file of your code and an icon that helps users distinguish your extension. You'll also have the option to submit text, screenshots and/or YouTube videos that describe the functionality of your extension. All types of extensions are welcome in the gallery, provided they comply with our Terms of Service.

For most extensions, the review process is fully automated. The only extensions we'll review manually are those that include an NPAPI component and all content scripts that affect "file://" URLs. For security reasons, developers of these types of extensions will need to provide some additional information before they can post them in the gallery.

Once an extension is uploaded, our gallery takes care of packaging and signing. Updating an extension is also incredibly easy — all a developer needs to do is to upload a new file in the gallery. Finally, to further help developers, in the next few days, we plan to open up the gallery to a small group of trusted testers. They will provide developers with insights and bug reports that will help them polish their extensions ahead of our beta launch.

We can't wait to share all the great extensions that you'll submit with all of Google Chrome's users. In the meantime, we encourage you to submit any bugs you find in the upload process to our Issue Tracker and to ask all relevant questions in our discussion group.

Posted by Lei Zheng, Software Engineer

Node.js is genuinely exciting

simonwillison.net — Mon, 23 Nov 2009 14:58:36 -0800

I agree. I played with it for a couple days and now I can’t shut up about it. That’s basically all I talked about at RubyConf :)

# | Discuss

Can I talk to that William fellow? He was so helpful

oldnewthing — Mon, 23 Nov 2009 07:00:00 -0800

In case you are wondering how Apple’s quaint digital *Animal

n8han — Mon, 23 Nov 2009 09:15:00 -0800

In case you are wondering how Apple’s quaint digital Animal Farm will inevitably advance, Phil Schiller hints of the company sharing their absolute power over Cocoa Touch devices with various salivating governments: “We’ve had a lot of eyes on us. We’ve had inquiries from governments and political leaders asking us what we were doing to protect children from inappropriate content.” It warms the heart.

the left fold: 2009-11-23

(author unknown) — Sun, 22 Nov 2009 23:02:20 -0800

Monster UA string

(author unknown) — Sat, 21 Nov 2009 16:19:00 -0800

Since we're a minority browser, many sites don't support us. This is fine when it's our bug, but sometimes a site is wrong to block us. Our rule is that we put some effort into contacting sites to get them to fix the problem, but in emergencies or where they're unwilling/able, we spoof.

Yahoo mail has three modes:

It just works. As seen on majority browsers.
It gives you a "your browser doesn't work" page with the option to try anyway. This is what it does for Windows Chrome. From my perspective this is the proper behavior until they feel like supporting Chrome, and I appreciate that many people lack the resources to support all browsers so I'm not too upset if it stays like this.
Or it just decides that it can't work. This is what it does for Linux Chrome.

You might ask, how are Windows and Linux Chrome different? And the answer is "they aren't", except for the UA string. (And that we antialias fonts.) So I "fixed" it with a UA string tweak that I find hilarious.

Mozilla/5.0 (Windows; U; Windows Linux x86_64; en-US) AppleWebKit/532.5 (KHTML, like Gecko) Chrome/4.0.250.0 Safari/532.5

I imagine their regex must match "Windows \S+" or something to allow NT, XP, etc.

Arv notes that we're now claiming that we are all of:

Mozilla
WebKit
HTML
Gecko
Safari
Windows
and Linux

ACM censors linking!

christer — Sat, 21 Nov 2009 12:32:06 -0800

OK, WTF, this got me so annoyed that I had to get out of blog posting dormancy. I just went to Ke-Sen Huang’s brilliant page of conference papers on the web. Except, this time, all ACM pages (such as SIGGRAPH 2009, I3D 2009, etc) have been taken down, with the comment “This page has been removed at the request of the ACM Publications Board.”

What the fuck!?

ACM has no rights to ~~request~~ require(*) someone to remove links to pages on the net, and it’s unfortunate that their request was complied with.

According to this page the ACM Publications Board is run by the following nasty little censoring asses:

Ronald F Boisvert (chair) - boisvert@acm.org
Holly E Rushmeier (co-chair) - holly@acm.org
Tamer M. Ozsu (vice-chair) - ozsu@acm.org

I strongly urge you to email them, with a cc to the ACM president Wendy Hall (wh@ecs.soton.ac.uk), to tell them what you really feel about ACM practicing censorship on Ke-Sen Huang’s pages (or anywhere)!

And if you have anything to do with ACM and SIGGRAPH, as people in the games industry tend to do, then I implore you to immediately stop your involvement and furthermore that you boycott their conferences (SIGGRAPH in particular) until they straighten their shit out.

Do not let their censoring of free information stand!

(*) Changed the wording here, because some dip-shit on Reddit can’t understand the message otherwise.

Update 11/23/09 16:04

The latest news is that, while Ke-Sen’s pages have not yet been restored, ACM has rescinded their censoring (through bogus copyright claims) as per the following email:
From: Pat Ryan Date: 2009/11/24 Subject: Web pages with SIGGRAPH Proceedings To: “kesen.huang@gmail.com” Cc: “boisvert@acm.org” , “Rushmeier, Holly” , Bernard Rous , “M. Tamer Özsu” , John White , Prof Wendy Hall

Dear Ke-Sen,

As you are aware, the computer graphics community has expressed dismay and concern about the removal of your web pages. ACM wants to make it possible for you to continue this service that the community clearly values very highly. By this message ACM grants permission for you to repost the pages, with the addition of links to the authoritative versions of the papers in the ACM Digital Library. The author’s home page links may also be included, but should not be links directly to the author’s version of the paper. Please post on the site that the information is being provided with the permission of the ACM. This is the solution you proposed earlier, and it is clear from the community’s comments that it is the right thing to do.

As you know, the concern about your pages was ACM copyright policy with regard to links. As a result of the community discussion, ACM will institute a formal review of this portion of its copyright policy.

Please contact us with any concerns or questions.

Sincerely,

Patricia Ryan
ACM Chief Operating Officer

They are still implying Ke-Sen’s pages were infringing on ACM copyright, which is still a legally suspect claim as (1) Ke-Sen’s pages are first and foremost factual listings which are not subjected to copyright infringements, (2) as Ke-Sen’s pages were sufficiently different from ACM’s so as not to be a derivative copy, and (3) even if they were outright copies of ACM’s table of contents, “fair use” of TOCs have apparently not been tested in a court of law, so ACM has no legal leg to stand on in this issue (and in fact, a ruling against all libraries in the US seems logistically implausible):

http://findarticles.com/p/articles/mi_m0FWE/is_6_8/ai_n6080446/

It will be interesting to see how this truly plays out over the next several weeks, but it seems a battle has been won, though the war is likely to continue for quite some time.

Thanks to everyone who read my post and voiced your concern to the ACM Publication Board. An organization is its members, and there is no stronger means of controlling an organization than through the vote of your voice and your purse!
Similar Posts:

Living in a browser

Mark — Sat, 21 Nov 2009 08:36:16 -0800

(Googler hat off)

I just spent an hour in Starbucks. (My kid is down the street at a Chinese class.) In the past hour, I’ve checked my email, read news, caught up on the HTML5 mailing lists and IRC chatter, paid two bills, balanced my checkbook, and written this post. Without leaving my browser.

So yeah, Google Chrome OS is gonna be a hit.

Bummer about the whole “only runs apps signed by a private key that Google won’t be sharing with anyone” thing. I’m not thrilled about the prospect of working for a DRM company. Google was in the DRM business once before; it ended with giving everyone their money back, twice. And does anyone honestly believe the first Chrome OS machine won’t be jailbroken within a week? Update: I’m getting conflicting reports about whether retail (non-developer) Chrome OS hardware will include a way to run unverified software. I’m locking this discussion thread until I can confirm this important detail.

On the other hand, computer maintenance sucks gargantuan donkey balls, and normal people don’t care about root. If you accost a random person on the street and ask them if they need root on their operating system to install another browser, and they’ll have three questions for you: 1. What’s root? 2. What’s an operating system? 3. What’s a browser?

Still… if I had root on a Linux netbook, the first thing I’d do is install Chromium and then spend 99% of my time in the browser. So I have to think that Chrome OS is a step in the right direction.

This is an open thread, but I won’t answer any questions on behalf of Google.

Go, going forward

(author unknown) — Sat, 21 Nov 2009 08:51:02 -0800

Structured Exception Handling in Assembly Language

rohitab — Wed, 22 Sep 1999 21:12:35 -0700

The difference between assignment and attachment with ATL smart pointers

oldnewthing — Fri, 20 Nov 2009 07:00:00 -0800

The Whigs come to LA

Cory — Fri, 20 Nov 2009 00:28:17 -0800

Just saw The Whigs live at the Troubadour — what a kick-ass show! The owner of my pizza place turned me on to these guys a year or so ago and I’ve been a big fan ever since.

Opening for them was The Dead Trees and The Features. I had heard of neither, but they both put on great sets of their own — I bought their CDs right after the show.

If you’ve never heard The Whigs before, they are giving out two songs from their upcoming CD “In The Dark” for free from their website.

Assorted notes

Phil — Thu, 19 Nov 2009 23:29:00 -0800

Many random notes (some for my own reference), each of which is too short to warrant a full blog post:

Emacs

It had always bothered me, just a little, that C-v and M-v, in addition to scrolling, move the cursor to the bottom or top of the screen. This has the odd effect that C-v M-v is not a no-op. Turns out that setting the variable scroll-preserve-screen-position appropriately can fix this. (via emacs-devel)

Chrome

Have you ever wondered why running Firefox instances get broken when new versions are installed? On GNU/Linux systems the package manager, which runs as root, pays no mind to what non-root users are doing. Some apps, like Firefox, run into major trouble because they'll load additional files after startup that can't be mixed and matched with the previous versions (this isn't really a problem for programs that are essentially just one binary). Chrome acquires immunity to this with its Zygote mode, which basically means do not load anything from disk after startup. A simple idea, but it takes some work to follow through with it.
Among visitors to this blog, Chrome is the second most popular browser, with a 9.5% share! Chrome leads IE (6.6%), Safari (5.9%), and Opera (4.3%), and trails only Firefox (72.6%). You can sort of see what a non-representative sample of web users happens across this blog.

Unix/Ubuntu

Soundconverter (aptitude install soundconverter) is a handy GTK/GNOME program for converting (transcoding) audio between formats. It is easy to use, converts all the music metadata, and takes full advantage of multicore processors.
When using find with xargs, xargs will get tripped up by filenames with spaces (it will treat each space-delimited component as a different argument). You can get around this by changing the delimiter in both find and xargs to \0 instead of space, as follows:

find . -iname '*.tmp' -print0 | xargs -0 rm

You can usually achieve the same effect with find ... -exec but I can never remember how to use it.

HTML, HTML5

Some things I picked up while working on Zeya:

Making a frame-like layout with CSS
HTML5 audio/video APIs as implemented in Firefox. The audio/video APIs provide nearly all the features you'd need for a music player. Just add chrome!
Detecting HTML5 features (from Dive Into HTML5)
One cool feature in HTML5 that you may not have heard of (because it's not quite as earth-shattering as video, canvas, or local storage) is "placeholder text" for text boxes.

(The text goes away when you focus the input and/or type something in it.)

It's a feature that (I assume, at least) people have been using Javascript hacks to implement for a long time.

ReCode / Warnings in Boost

(author unknown) — Thu, 19 Nov 2009 18:58:53 -0800

The Traveling Salesman Problem and Javascript Compression

Ray Cromwell — Thu, 19 Nov 2009 16:39:00 -0800

In my last post, On Reducing the Size of Compressed Javascript, I covered a technique I had been researching on re-ordering Javascript code to enhance the efficiency of the gzip/deflate compressors. The basic approach is to come up with some way of measuring the distance between two functions, and then trying to group functions together by shortest distance.

A key question is, what's a good metric? I knew that the ideal metric would take into account how the LZ77 algorithm works, but for expediency, I went with a reasonable and well known string distance: the levenshtein distance.

GZip vs 7-zip

Another finding from my previous experiments was that 7-zip sports a much superior deflate implementation, it can compress a given input using the same algorithm as gzip with non-trivial size improvements. The problem is, 7-zip is implemented in C++, and I wanted to write a GWT Linker to automatically produce super-compressed Javascript.

In digging around for a Java 7-zip deflate implementation, I ran into this paper on Paul Sladen's page. titled: Compressing File Collections with a TSP-Based Approach. This paper published in 2004 details a similar technique of enhancing TAR compression of a collection of files by re-ordering the TAR to maximize compression. More on that later, but the paper itself has an obscure reference to something called the LZ Distance metric. Tracking down that paper "G. Cormode, M. Paterson, S. Sahinalp, and U. Vishkin. Communication complexity of doc- ument exchange. In Proc. of the ACM–SIAM Symp. on Discrete Algorithms, January 2000." leads us to the definition of LZ Distance.

LZ Distance

The LZ Distance between two strings x and y denoted by LZDIST(x,y) is as follows:

The minimum number of single characters or substrings of y or of the partially built string, which are required to build x from left to right.

As an example, consider x=aabcdefgabc and y=aaaaaaaaaaaaaaaa. We try to parse x in terms of y, and keep a running partially built string called 'p'. Here are the steps:

Prefix 'aa' of x is contained in y, so p = (aa), x = bcdefgabc
No prefix of x found in y or p, so just add 'b' p = (aa)(b), x = cdefgabc
No prefix of x found in y or p ,so just add 'c', p = (aa)(b)(c), x = defgabc
No prefix of x found in y or p ,so just add 'd', p = (aa)(b)(c)(d), x = efgabc
No prefix of x found in y or p ,so just add 'e', p = (aa)(b)(c)(d)(e), x = fgabc
No prefix of x found in y or p ,so just add 'f', p = (aa)(b)(c)(d)(e)(f), x = gabc
No prefix of x found in y or p ,so just add 'g', p = (aa)(b)(c)(d)(e)(f)(g), x = abc
'abc' is found at index '1' in p, so p = (aa)(b)(c)(d)(e)(f)(g)(abc)

Here, the LZDIST(x,y) = 8. But what of LZDIST(y, x)?

Prefix 'aa' of y is contained in x, so p = (aa), y = aaaaaaaaaaaaaa
Prefix 'aa' of y is contained in x, so p = (aa)(aa), y = aaaaaaaaaaaa
Prefix 'aaaa' of y is contained in p, p = (aa)((aa)(aaaa) y = aaaaaaaa
Prefix 'aaaaaaaa' of y is contained in p, so p = (aa)(aa)(aaaa)(aaaaaaaa)

LZDIST(y,x) = 4. Interesting, but what does this buy us?

LZDIST and GZIP

Recall that GZIP's deflate encoder uses an algorithm called LZ77 which looks at the current input tokens, and searches backwards for previous substrings that match the current input string, replacing them with back references. What LZDIST(x,y) gives us, is a measure of how many back references for X can be found in Y, that is, how X can be compressed by references to substrings in Y or in partially built decompression outputs.

Now you can see why this might help improve code re-ordering. If we can order Javascript functions in order to maximize back-references, we might be able to squeeze a little more out of gzip.

Clustering with LZDIST

Remember the earlier paper's title had the acronym TSP? This paper discusses how re-ordering can be viewed as an optimal tour on a graph which means that solving it is as hard as the Traveling Salesman Problem - NP-Complete. Uh oh, that sounds like trouble, but that's only NP Complete for the optimal solution. A good estimate on the true solution might also produce good results, and this paper shows that a greedy tour heuristic produces up to a 10% benefit in compression.

For expediency reasons, I decided not to implement a TSP model at this point, but went with simple pair-based greedy approach detailed in my previous blog post, substituting LZDIST for Levenshtein Distance. I also implemented an LZ-like greedy algorithm where I keep a running string which is the concatenation of all of the functions which have been added to the output, and use LZDIST to find the next closest function to append within a 32k window. This mimics how a gzip compressor might see the input.

Results for GWT Showcase Application

Technique	Size	Ratio	Relative Improvement (vs previous row)


No Compression	196778	100%

gzip -9	64401	32.7%	32.7%

7zip	61500	31.5%	4.6%

gzip -9 + levensthein	61409	31.2%	4.7%

7zip + levenshtein	58536	29.7%	4.7%

7zip + lzdist	57923	29.4%	1.1%

LZMA	51423	26.1%	11.3%

Looking at relative compression, levenshtein vs lzdist, the difference 57923/58536 = 1.1% which seems worthwhile to me. In a large script, this can save 1-2 kilobytes. I threw in LZMA which boasts 12% better compression in this instance than DEFLATE.

Notes on implementation

The LZDIST function is expensive to compute in a naive fashion. It is similar to the longest common substring problem. The brute force version can be on the order of O(n^3), while a dynamic programming version can run in O(n^2). To make this run efficiently and not irritate your users, you need to use a Generalized Suffix Tree approach. Compressors like 7-zip are already using Suffix trees to implement the LZ77 search algorithm, so those of you looking for hints to do an efficient implementation should take a look at the LZMA SDK (both C++ and Java versions are available)

Conclusion

Utilizing clustering compression techniques, LZ distance, and 7-zip, it is possible to reduce the GWT Showcase application from 64401 bytes to 57923 bytes, a savings of about 10.1%.

-Ray

links for 2009-11-19

sogrady — Thu, 19 Nov 2009 17:03:41 -0800

Wolfram|Alpha Blog : Future Directions of Wolfram|Alpha

"But do you know how often we release a new version of Wolfram|Alpha?
It’s once a week. 23 times so far."

(tags: wolframalpha saas updates maintenance upgrades)

ThreadFrac revised

John — Thu, 19 Nov 2009 15:34:00 -0800

Grrr...I went to run my copy of 'ThreadFrac' that was uploaded in a couple of places and it wouldn't run due to side-by-side assembly issues or some such nonsense.

I rebuilt it with an embedded manifest and it should run now.

Let me know if you have a problem; requires Windows with DX9 to run.

The reason I was looking for ThreadFrac is that I want to start messing with the new 3d equation called 'Mandelbulb'.

I'm going to write a real-time multi-threaded raytracer and add it to my ThreadFrac program when I get a chance to work on.

If you haven't seen the 'Mandelbulb' yet, check it out.

Here is the link to the revised executable for ThreadFrac.

What’s After Excel? Big Data and the Future of Spreadsheets

sogrady — Thu, 19 Nov 2009 14:24:52 -0800

For the better part of twenty years, Microsoft Excel has been the most popular spreadsheet application on the planet. In a very real sense, it is the driver of Office revenues, because while office workers will use alternatives to Powerpoint and Word, you can pry Excel from their cold, dead fingers.

So how do you replace it?

You don’t. Many have tried. All have failed. Excel has, at least as measured by marketshare, been chewing up and spitting out competing products for well over a decade.

The best strategy in competing with Office generally, as evidenced by the success of Google Docs, is to reframe the debate. To compete where Office does not. As Sun Tzu tells us, “You can be sure of succeeding in your attacks if you only attack places which are undefended.”

But what territory, precisely, has Excel left undefended? It’s the gold standard for analysts the world over, and it’s actually somewhat frightening how many businesses are run purely on top of it. Excel isn’t online yet, but it will be and even if it doesn’t get there, Google Docs did. So that’s out. Leaving what?

How about big data?

Excel has been used on big data for years, it’s true. But not directly on big data. With a row limit of around 65,000 [Update: Mike Cullina writes in to say the 65K limit was eliminated in the last release, 2007 - the new limit is 1M plus], it certainly can’t be used as a direct window into data warehouses or marts. So instead analysts use it to front end views or other subsets of the original dataset. Which, by the way, has probably been heavily cleansed and normalized.

Want to ask questions of the entire dataset? Or of datasets, pluarl? Terrific. Learn SQL, find and be very nice to a DBA, and beg for the access you need. Also, be prepared to wait days or even weeks for your answer. Working on big data is hard, remember.

Unless you have a new back end, one designed to reduce complicated questions into a set of tasks that can be individually executed on multiple machines. A back end like Hadoop.

The good news about Hadoop, you’ve already heard: it’s very, very good at carving up large workloads when you can supply it with adequate hardware. The bad news, however, is that the front ends for the tool are, to put it charitably, a bit behind. See this slide from Kevin Weil’s presentation at NoSQL East

And that’s the simple interface. Anybody who knows SQL is in good shape, but I think it’s safe to say Hadoop is just a tad less accessible to analyst-types than Excel.

Which is where IBM’s Big Sheets comes in. Called M2 at the time of Rod Smith’s presentation from HadoopWorld, you can think of it as a spreadsheet-like front end (with elements of DabbleDB, ManyEyes, and others) for Hadoop datasets.

Hw09 Enabling Ad Hoc Analytics At Web Scale

View more documents from Cloudera, Inc..

If you can’t see the promise in the slides, just wait for the video from HadoopWorld. Or just trust me: Big Sheets is the real deal, and the shape of things to come. Maybe it’ll be Big Sheets, maybe something totally different: the concept is real.

Should Microsoft be worried? Not in the short term. Excel is near perfectly adapted to its environment, and is in no danger of being replaced by an alternative, whether that’s Big Sheets, Google Docs or OpenOffice.org Spreadsheets. But environments are not static; they have a way of changing, and it certainly appears that we’re in the midst of a change now.

Data hypergrowth has pretty much become a cliche at this point: how many times can you hear “we’re going to produce more data in the next [small time period] that we have in human history” before you stop hearing it? Less apparent, however, has been the impact on individual analysis. Take the Twitter datasets recently released by the Infochimps guys: the monthly base tab separated values spreadsheet is 1.7 GB. The hourly version, it can be assumed, is a bit larger.

Point being: a lot of us are going to be working with large datasets soon, if we’re not already. And while the current toolset of choice – Excel – has its strengths, that doesn’t happy to be one of them. So either it gets there, or someone goes island hopping beyond the most popular spreadsheet product in history and defines the user interface and experience for Big Data.

Adapt or die, as they say.

Disclosure: Both IBM and Microsoft are RedMonk clients; Google is not.

christian — Mon, 16 Nov 2009 21:47:23 -0800

This week we will be confronting a fact that, by definition, haunts the average online dater: no matter how much time you spend polishing your profile, honing your IM banter, and perfecting your message introductions, it’s your picture that matters most.

We’re going to look at how your photos affect both the messages you get and how successful your own outgoing messages are. We all know that beautiful people are more successful daters, but let’s quantify by exactly how much.

To illustrate the exact spectrum of looks we’re talking about here, and to put some human faces on our discussion, I want to introduce a few photos of real OkCupid users. Here are two women near the top our range.

[show men instead]

And here are two rated in the middle.

[show men instead]

As for photos at the bottom of the curve, it didn’t feel right to write someone and say “can I use you to illustrate the concept of ugliness on my blog?” so you’ll just have to extrapolate.

The above featured users have graciously agreed to let me post their pictures, so please don’t make them regret it. Funnily enough, I had to write about a dozen beautiful female users before anyone would even get back to me. Life imitates blog!

Anyhow, I know attractiveness is far from a universal concept, but maybe keep these folks in mind as we go through the data.

. . .

We’ll start with a simple line chart. The information I’ll present in this post is not normalized because, as we’ll see, it’s interesting how men and women evaluate looks differently.

Our chart shows how men have rated women, on a scale from 0 to 5. The curve is symmetric and surprisingly charitable: a woman is as likely to be considered extremely ugly as extremely beautiful, and the majority of women have been rated about “medium.” The chart looks normalized, even though it’s just the unfiltered opinions of our male users.

Given the popular wisdom that Hollywood, the Internet, and Photoshop have created unrealistic expectations of how a woman should look, I found the fairness and, well, realism, of this gray arc kind of heartening.

Now let’s superimpose the distribution of actual messages guys have sent:

When it comes down to actually choosing targets, men choose the modelesque. Someone like roomtodance
2/3 of male messages go to the top 1/3 of women.
above gets nearly 5 times as many messages as a typical woman and 28 times as many messages as a woman at the low end of our curve. Site-wide, two-thirds of male messages go to the best-looking third of women. So basically, guys are fighting each other 2-for-1 for the absolute best-rated females, while plenty of potentially charming, even cute, girls go unwritten.

The medical term for this is male pattern madness.

. . .

The female equivalent of the above chart shows a different bias:

As you can see from the gray line, women rate an incredible 80% of guys as worse-looking than medium. Very harsh. On the other hand, when it comes to actual messaging, women shift their expectations only just slightly ahead of the curve, which is a healthier pattern than guys’ pursuing the all-but-unattainable. But with the basic ratings so out-of-whack, the two curves together suggest some strange possibilities for the female thought process, the most salient of which is that the average-looking woman has convinced herself that the vast majority of males aren’t good enough for her, but she then goes right out and messages them anyway.

Just to illustrate that women are operating on a very different scale, here are just a few of the many, many guys we here in the office think are totally decent-looking, but that women have rated, in their occult way, as significantly less attractive than so-called “medium”:

Females of OkCupid, we site founders say to you: ouch! Paradoxically, it seems it’s women, not men, who have unrealistic standards for the “average” member of the opposite sex.

Finally, I just want to combine the two charts to emphasize how much fuller the inboxes of good-looking people get. I have scaled this graph to show multiples of messages sent to the lowest-rated people. For instance, the most attractive guys get 11× the messages the lowest-rated do. The medium-rated get about 4×.

This graph also dramatically illustrates just how much more important a woman’s looks are than a guy’s.

. . .

Now let’s take a look at how senders’ and recipients’ attractivenesses affect reply rates, not just the number of messages sent.

As you’d expect, more attractive people get more replies. And since they themselves get so many more messages than everyone else, they write back much less frequently. Here’s the graph for female senders, plotted in evenly-spaced “attractiveness groups.”

And here’s the one for male senders.

One interesting thing seems to be going on here: when the best-looking men write the worst-looking women, taste the rainbow,
of self-esteem issues their message success rate takes a big hit. The knee-jerk response would be to somehow chalk it up to hunky spammers, but we very carefully control for that in these articles, and in any event why would better-looking girls be drastically more susceptible to it? It seems to be some kind of self-confidence thing.

As we did before, I’m going to consolidate the line charts to show just how your attractiveness changes how often your messages get responses.

(author unknown) — Mon, 23 Nov 2009 20:10:00 -0800

Extensions: One Step Closer to the Finish Line (Ian Fette/Chromium Blog)

(author unknown) — Mon, 23 Nov 2009 18:25:01 -0800

New Facebook worm - don't click da' button baby! (Nick/AVG Blogs)

(author unknown) — Mon, 23 Nov 2009 17:15:03 -0800

Twitter's Biz Stone says could go IPO route (Georgina Prodhan/Reuters)

(author unknown) — Mon, 23 Nov 2009 13:50:10 -0800

Windows Mobile vs. Android: WinMo Is Better Than You Think (James Kendrick/jkOnTheRun)

(author unknown) — Mon, 23 Nov 2009 13:05:12 -0800

Easy holiday shopping with Google Product Search

A Googler — Mon, 23 Nov 2009 12:13:00 -0800

This holiday season, whether you're shopping for a new sweater for Fido, a Lego set for young builders, or that fancy camera you've been eyeing all year, Google Product Search can help you find what you're looking for at a great price. You can compare products and prices from merchants across the web, from popular retailers like Amazon and Best Buy to places to buy unique gifts like eBay and Etsy.

Check out some of the recent improvements we've made to Google Product Search:

Our new gallery view shows larger, higher-resolution images to help pick the style or model that is right for you. Check out some [digital cameras] or [sweater boots].

Check the Reviews section of our product pages before you buy — we're now including review summaries to help you see what people are saying at a glance.

We've recently integrated video product reviews from YouTube, which appear right on the product pages so you can get an in-depth look at items before you buy.

If you want to see or purchase an item in person, click nearby stores to see a map of nearby store locations for that seller.

Product Search works great on mobile phones, so you can compare prices, read reviews and even find coupons from the Local Business Center while you're shopping. You can also use the nifty barcode scanner on your phone to compare prices quickly and easily.

We hope these new features help you find the perfect gifts for your friends and family (and maybe even something nice for yourself) this holiday season.

Happy shopping!

Posted by Sameer Samat, Product Management Director & Jeff Bartelma, Senior Product Manager

Beyond Bookmark Me! Become a Fan of This App! ... (Eric von Coelln/Inside Social Games)

(author unknown) — Mon, 23 Nov 2009 11:45:09 -0800

Happy Thanksgiving Travels: Google Maps Navigation now available for Android 1.6 (Michael Siliski/Google Mobile Blog)

(author unknown) — Mon, 23 Nov 2009 11:30:00 -0800

Apple Joins AT&T/Verizon Spat With New iPhone Ads (See Them Here!) (John Paczkowski/Digital Daily)

(author unknown) — Mon, 23 Nov 2009 11:05:06 -0800

LinkedIn Platform: Open for Business (Adam Nash/The LinkedIn Blog)

(author unknown) — Mon, 23 Nov 2009 10:05:10 -0800

Displaying the best display ad with Teracent (The Official Google Blog)

(author unknown) — Mon, 23 Nov 2009 10:00:34 -0800

Displaying the best display ad with Teracent

A Googler — Mon, 23 Nov 2009 09:32:00 -0800

Can you spot the difference between these two sample display ads? Of course you can. However, the most important difference is not discernible to the naked eye.

The lower ad was customized and chosen from thousands of different creative elements, automatically and in real-time, by machine-learning algorithms developed by Teracent, a San Mateo, California startup.

We think that this technology has great potential to improve display advertising on the web. That's why we're pleased to announce today that we've entered into a definitive agreement to acquire Teracent. The transaction, which is subject to various closing conditions, is expected to close this quarter.

As you know, we've been busy releasing new features and products to help improve display advertising on the web for everyone. We believe that Teracent's technology fits neatly into these efforts.

Teracent's technology can pick and choose from literally thousands of creative elements of a display ad in real-time — tweaking images, products, messages or colors. These elements can be optimized depending on factors like geographic location, language, the content of the website, the time of day or the past performance of different ads.

This technology can help advertisers get better results from their display ad campaigns. In turn, this enables publishers to make more money from their ad space and delivers web users better ads and more ad-funded web content.

We're looking forward to welcoming the Teracent team to Google and to making this technology available to our display advertising clients — including those who run display ad campaigns on the Google Content Network and our DoubleClick clients.

Posted by Neal Mohan, Vice President, Product Management and Joerg Heilig, Engineering Director

The Many Flaws of Twitter's Retweet Feature

(author unknown) — Mon, 23 Nov 2009 08:24:00 -0800

I've been a Twitter user for almost two years now and I have always been impressed by the emergent behavior that has developed from simply giving people a text box with 140 character limit. The folks at Twitter have also done a good job of noticing some these emergent behaviors and making them formal features of the site. Both hashtags and @replies are examples of emergent community conventions in authoring tweets that are now formal features of the site.

Twitter recently added retweets to this list with Project Retweet. After using this feature for a few days I've found that unlike hashtags and @replies, the way this feature has been integrated into the Twitter experience is deeply flawed. Before I talking about the problems with Project Retweet, I should talk about how the community uses retweeting today.

Retweeting 101: What is it and why do people do it?

Retweeting is akin to the practice of forwarding along interesting blog posts and links to your friends via email. A retweet repeats the content of a person's tweet (sometimes edited for brevity) along with a reference to the user who is being retweeted. Often times people also add some commentary to the retweets. Examples of both styles of retweets are shown below.

Figure 1: Retweet without commentary

Figure 2: Retweet with added comment

Unlike hashtags and @replies, the community conventions aren't as consistent with retweets. Below are two examples of retweets from my home page which use different prefixes and separators from the one above to indicate the item is a retweet and the user's comment respectively.

Figure 3: Different conventions in retweeting

However there are many issues with retweeting not being a formal feature of Twitter. For one, it is often hard for new users to figure out what's going on when they see people posting updates prefixed with strange symbols and abbreviations. Another problem is that users who want to post a retweet now have to deal with the fact that the original tweet may have taken up all or most of the 140 character limit so there may be little room to credit the author let alone add commentary.

Thus I was looking forward to retweeting becoming a formal feature of Twitter so that these problems would be addressed. Unfortunately, while one of these problems was fixed more problems were introduced.

Flaw #1: Need to visit multiple places to see all retweets of your content

Before the introduction of the retweet feature, users could go to http://www.twitter.com/replies to see all posts that reference their name which would include @replies and retweets. The new Twitter features fragments this in an inconsistent manner.

Figure 4: Current Twitter sidebar

Now users have to visit http://www.twitter.com/replies to see people who has retweeted their posts using community conventions (i.e. copy and pasting then prefixing "RT" to a tweet) and then visit http://twitter.com/retweeted_of_mine to see who has retweeted their posts by clicking the Retweet link in the Twitter web user interface. There will be different people in both lists.

Figure 5: Retweets in the Replies/Mentions page

Figure 6: Retweets on the "Your tweets, retweeted" page

It is surprising to me that Twitter didn't at least include posts that start with RT followed by your username in http://twitter.com/retweeted_of_mine as well.

Flaw #2: No way to add commentary on what you are retweeting

As I mentioned earlier, it is fairly common for people to retweet a status update and then add their own commentary. The retweet feature built into Twitter ignores this common usage pattern and provides no option to add your own commentary.

Figure 7: The Retweet prompt

This omission is particularly problematic if you disagree with what you are sharing and want to clarify to your followers that although you find the tweet interesting you aren't endorsing the opinion.

Flaw #3: Retweets don't show up in Twitter apps

One of the other surprising changes is that Twitter retweets have been introduced in a backwards-incompatible manner into the API. This means that retweets created using the Twitter retweet button do not show up in 3rd party applications that use the Twitter API. See below for an example of what I see in Echofon versus the Twitter web experience and notice the missing tweet.

Figure 8: Twitter website showing a retweet

Figure 9: The retweet is missing in Echofon

Again, I find this surprising since it would have been straightforward to keep retweets in the API and exposing them as if they were regular old school retweets prefixed with "RT".

Flaw #4: Pictures of people I don't know in my stream

The last major problem with the Twitter retweet feature is that it breaks user expectation of the stream. Until this feature shipped, users could rest assured that the only content they saw in their stream was content they had explicitly asked for by subscribing to a user. Thus when you see someone in your stream the person's user name and avatar are familiar to you.

With the new retweet feature, the Twitter team has decided to highlight the person being retweeted and treat the person who I've subscribed to that did the retweeting as an afterthought. Not only does this confuse users at first (who is this person showing up in my feed and why?) but it also assumes that the content being retweeted is more important than who did the retweeting. This is an unfortunate assumption since in many cases the person who did the retweeting adds all the context.

Now Playing: Jason Derulo - Whatcha Say

MSNBC.com Taking Over @BreakingNews Twitter Feed; Signs On As BNO News' First Client (Rafat Ali /paidContent.org)

(author unknown) — Mon, 23 Nov 2009 08:40:01 -0800

Another iPhone worm - and this time it's malicious (Chester Wisniewski/Chester Wisniewski's ...)

(author unknown) — Mon, 23 Nov 2009 08:00:14 -0800

AdMob Data Reveals Android's Growth, Device Market Share (Om Malik/GigaOM)

(author unknown) — Mon, 23 Nov 2009 07:35:18 -0800

Volunteers Log Off As Wikipedia Ages (Wall Street Journal)

(author unknown) — Mon, 23 Nov 2009 07:25:03 -0800

Building Scalable Databases: Perspectives on the War on Soft Deletes

(author unknown) — Mon, 23 Nov 2009 06:46:15 -0800

In the past few months I've noticed an increased number of posts questioning practices around deleting and "virtually" deleting data from databases. Since some of the concerns around this practice have to do with the impact of soft deletes on scalability of a database-based application, I thought it would be a good topic for my ongoing series on building scalable databases.

Soft Deletes 101: What is a soft delete and how does it differ from a hard delete?

Soft deleting an item from a database means that the row or entity is marked as deleted but not physically removed from the database. Instead it is hidden from normal users of the system but may be accessible by database or system administrators.

For example, let's consider this sample database of XBox 360 games I own

Name	Category	ESRB	GamespotScore	Company
Call of Duty: Modern Warfare 2	First Person Shooter	Mature	9.0	Infinity Ward
Batman: Arkham Asylum	Fantasy Action Adventure	Teen	9.0	Rocksteady Studios
Gears of War 2	Sci-Fi Shooter	Mature	9.0	Epic Games
Call of Duty 4: Modern Warfare	First Person Shooter	Mature	9.0	Infinity Ward
Soul Calibur IV	3D Fighting	Teen	8.5	Namco

Now consider what happens if I decide that I'm done with Call of Duty 4: Modern Warfare now that I own Call of Duty: Modern Warfare 2. The expected thing to do would then be to remove the entry from my database using a query such as

DELETE FROM games WHERE name='Call of Duty 4: Modern Warfare';

This is what is considered a "hard" delete.

But then what happens if my friends decide to use my list of games to decide which games to get me for Christmas? A friend might not realize I'd previously owned the game and might get it for me again. Thus it might be preferable if instead of deleting items from the database they were removed from consideration as games I currently own but still could be retrieved in special situations. To address this scenario I'd add an IsDeleted column as shown below

Name	Category	ESRB	GamespotScore	Company	IsDeleted
Call of Duty: Modern Warfare 2	First Person Shooter	Mature	9.0	Infinity Ward	False
Batman: Arkham Asylum	Fantasy Action Adventure	Teen	9.0	Rocksteady Studios	False
Gears of War 2	Sci-Fi Shooter	Mature	9.0	Epic Games	False
Call of Duty 4: Modern Warfare	First Person Shooter	Mature	9.0	Infinity Ward	True
Soul Calibur IV	3D Fighting	Teen	8.5	Namco	False

Then for typical uses an application would interact with the following view of the underlying table

CREATE VIEW current_games AS SELECT Name, Category, ESRB, GameSpotScore, Company FROM games WHERE IsDeleted=False;

but when my friends ask me for a list of all of the games I have, I can provide the full list of all the games I've ever owned from the original games table if needed. Now that we understand how one would use soft deletes we can discuss the arguments against this practice.

Rationale for War: The argument against soft deletes

Ayende Rahien makes a cogent argument against soft deletes in his post Avoid Soft Deletes where he writes

One of the annoyances that we have to deal when building enterprise applications is the requirement that no data shall be lost. The usual response to that is to introduce a WasDeleted or an IsActive column in the database and implement deletes as an update that would set that flag.

Simple, easy to understand, quick to implement and explain.

It is also, quite often, wrong.

The problem is that deletion of a row or an entity is rarely a simple event. It effect not only the data in the model, but also the shape of the model. That is why we have foreign keys, to ensure that we don’t end up with Order Lines that don’t have a parent Order. And that is just the simplest of issues.
...
Let us say that we want to delete an order. What should we do? That is a business decision, actually. But it is one that is enforced by the DB itself, keeping the data integrity.

When we are dealing with soft deletes, it is easy to get into situations where we have, for all intents and purposes, corrupt data, because Customer’s LastOrder (which is just a tiny optimization that no one thought about) now points to a soft deleted order.

Ayende is right that adding an IsDeleted flag mean that you can no longer take advantage of database triggers for use when cleaning up database state when a deletion occurs. This sort of cleanup now has to moved up into the application layer.

There is another set of arguments against soft deletes in Richard Dingwall's post entitled The Trouble with Soft Delete where he points out the following problems

Complexity

To prevent mixing active and inactive data in results, all queries must be made aware of the soft delete columns so they can explicitly exclude them. It’s like a tax; a mandatory WHERE clause to ensure you don’t return any deleted rows.

This extra WHERE clause is similar to checking return codes in programming languages that don’t throw exceptions (like C). It’s very simple to do, but if you forget to do it in even one place, bugs can creep in very fast. And it is background noise that detracts away from the real intention of the query.

Performance

At first glance you might think evaluating soft delete columns in every query would have a noticeable impact on performance. However, I’ve found that most RDBMSs are actually pretty good at recognizing soft delete columns (probably because they are so commonly used) and does a good job at optimizing queries that use them. In practice, filtering inactive rows doesn’t cost too much in itself.

Instead, the performance hit comes simply from the volume of data that builds up when you don’t bother clearing old rows. For example, we have a table in a system at work that records an organisations day-to-day tasks: pending, planned, and completed. It has around five million rows in total, but of that, only a very small percentage (2%) are still active and interesting to the application. The rest are all historical; rarely used and kept only to maintain foreign key integrity and for reporting purposes.

Interestingly, the biggest problem we have with this table is not slow read performance but writes. Due to its high use, we index the table heavily to improve query performance. But with the number of rows in the table, it takes so long to update these indexes that the application frequently times out waiting for DML commands to finish.

These arguments seem less valid than Ayende's especially when the alternatives proposed are evaluated. Let's look at the aforementioned problems and the proposed alternatives in turn.

Trading the devil you know for the devil you don't: Thoughts on the alternatives to soft deletes

Richard Dingwall argues that soft deletes add unnecessary complexity to the system since all queries have to be aware of the IsDeleted column(s) in the database. As I mentioned in my initial description of soft deletes this definitely does not have to be the case. The database administrator can create views which the core application logic interacts with (i.e. the current_games table in my example) so that only a small subset of system procedures need to actually know that the soft deleted columns even still exist in the database.

A database becoming so large that data manipulation becomes slow due to having to update indexes is a valid problem. However Richard Dingwall's suggested alternative excerpted below seems to trade one problem for a worse one

The memento pattern

Soft delete only supports undoing deletes, but the memento pattern provides a standard means of handling all undo scenarios your application might require.

It works by taking a snapshot of an item just before a change is made, and putting it aside in a separate store, in case a user wants to restore or rollback later. For example, in a job board application, you might have two tables: one transactional for live jobs, and an undo log that stores snapshots of jobs at previous points in time:

The problem I have with this solution is that if your database is already grinding to a halt simply because you track which items are active/inactive in your database, how much worse would the situation be if you now store every state transition in the database as well? Sounds like you're trading one performance problem for a much worse one.

The real problem seems to be that the database has gotten too big to be operated on in an efficient manner on a single machine. The best way to address this is to partition or shard the database. In fact, you could even choose to store all inactive records on one database server and all active records on another. Those interested in database sharding can take a look at a more detailed discussion on database sharding I wrote earlier this year.

Another alternative proposed by both Ayende Rahien and Richard Dingwall is to delete the data but use database triggers to write to an audit log in the cases where auditing is the primary use case for keeping soft deleted entries in the database. This works in the cases where the only reason for soft deleting entries is for auditing purposes. However there are many real world situations where this is not the case.

One use case for soft deleting is to provide an "undo" feature in an end user application. For example, consider a user synchronizes the contact list on their phone with one in the cloud (e.g. an iPhone or Windows Mobile/Windows Phone connecting to Exchange or an Android phone connecting to Gmail). Imagine that the user now deletes a contact from their phone because they do not have a phone number for the person only to find out that person has also been deleted from their address book in the cloud. At that point, an undo feature is desirable.

Other use cases could be the need to reactivate items that have been removed from the database but with their state intact. For example, when people return to Microsoft who used to work there in the past their seniority for certain perks takes into account their previous stints at the company. Similarly, you can imagine a company restocking an item that they had pulled from their shelves because they have become popular due to some new fad (e.g. Beatles memorabilia is back in style thanks to The Beatles™: Rock Band™).

The bottom line is that an audit log may be a useful replacement for soft deletes in some scenarios but it isn't the answer to every situation where soft deletes are typically used.

Not so fast: The argument against hard deletes

So far we haven't discussed how hard deletes should fit in a world of soft deletes. In some cases, soft deletes eventually lead to hard deletes. In the example of video games I've owned I might decide that if a soft deleted item is several years old or is a game from an outdated console then it might be OK to delete. So I'd create a janitor process that would scan the database periodically to seek out soft deleted entries to permanently delete. In other cases, some content may always be hard deleted since there are no situations where one might consider keeping them around for posterity. An example of the latter is comment or trackback spam on a blog post.

Udi Dahan wrote a rebuttal to Ayende Rahien's post where he question my assertion above that there are situations where one wants to hard delete data from the database in his post Don’t Delete – Just Don’t where he writes

Model the task, not the data

Looking back at the story our friend from marketing told us, his intent is to discontinue the product – not to delete it in any technical sense of the word. As such, we probably should provide a more explicit representation of this task in the user interface than just selecting a row in some grid and clicking the ‘delete’ button (and “Are you sure?” isn’t it).

As we broaden our perspective to more parts of the system, we see this same pattern repeating:

Orders aren’t deleted – they’re cancelled. There may also be fees incurred if the order is canceled too late.

Employees aren’t deleted – they’re fired (or possibly retired). A compensation package often needs to be handled.

Jobs aren’t deleted – they’re filled (or their requisition is revoked).

In all cases, the thing we should focus on is the task the user wishes to perform, rather than on the technical action to be performed on one entity or another. In almost all cases, more than one entity needs to be considered.

Statuses

In all the examples above, what we see is a replacement of the technical action ‘delete’ with a relevant business action. At the entity level, instead of having a (hidden) technical WasDeleted status, we see an explicit business status that users need to be aware of.

I tend to agree with Udi Dahan's recommendation. Instead of a technical flag like IsDeleted, we should model the business process. So my database table of games I owned should really be called games_I_have_owned with the IsDeleted column replaced with something more appropriate such as CurrentlyOwn. This is a much better model of the real-life situation than my initial table and the soft deleted entries are now clearly part of the business process as opposed to being part of some internal system book keeping system.

Advocating that items be never deleted is a tad extreme but I'd actually lean closer to that extreme than most. Unless the data is clearly worthless (e.g. comment spam) or the cost is truly prohibitive (e.g. you're storing large amounts of binary data) then I'd recommend keeping the information around instead of assuming the existence of a DELETE clause in your database is a requirement that you use it.

Now Playing: 50 Cent - Baby By Me (feat. Ne-Yo)

Meet the New AOL Logo: "Aol." (Plus the Press Release) (Peter Kafka/MediaMemo)

(author unknown) — Mon, 23 Nov 2009 07:05:00 -0800

Armstrong On AOL's New Branding And 'Very, Very, Very Inexpensive' Campaign (Staci D. Kramer/paidContent)

(author unknown) — Mon, 23 Nov 2009 06:20:04 -0800

Recovering files using TestDisk

yang — Mon, 23 Nov 2009 18:29:15 -0800

I recently had a hard disk go bad. It had a single NTFS partition where the partition was no longer recognized by Windows or Linux.

I managed to recover (I think) all of my files by reading the data off of the disk using TestDisk. It discovered my partition without issue, and the actual file recovery went relatively smoothly.

I believe the way TestDisk was primarily designed to work was by attempting to actually repair the partition. This should be fine if you’re working on a copy of the raw disk bits (you can probably use dd for Windows for this, or write a program that accesses ), but I’d generally be averse to making any further changes to a bad disk. I chose to recover my data just by copying the files directly out. You can do this as follows:

go to (your disk) > Intel > Analyse > (your dynamic partition)
highlight your partition
press P to list files
press H to hide deleted files
press C to copy the current directory

Yang Zhang — Wed, 18 Nov 2009 11:59:20 -0800

Security

Yang Zhang — Wed, 18 Nov 2009 11:58:30 -0800

Environmental chemicals: feminizing boys?

jxyz — Mon, 23 Nov 2009 20:41:00 -0800

Yes, I'm somewhat behind on blogging--I just finished a big deadline on Saturday.

(A couple of weeks ago) Owen sent me this Slashdot story about an unofficial Danish study suggesting that environmental chemicals are "feminizing boys." From Slashdot:

"Denmark has unveiled official research showing that two-year-old children are at risk from a bewildering array of gender-bending chemicals in such everyday items as waterproof clothes, rubber boots... A picture is emerging of ubiquitous chemical contamination driving down sperm counts and feminizing male children all over the developed world. Research at Rotterdam's Erasmus University found that boys whose mothers were exposed to PCBs and dioxins were more likely to play with dolls and tea sets and dress up in female clothes... '"

Yikes? I didn't realize that chemicals could change culture and socialization in such strong ways*.

* Sarcasm.

Write badly well

jxyz — Mon, 23 Nov 2009 20:35:00 -0800

I love this hilarious blog, how to write badly well, by one of the authors of the book Who Writes This Crap? A quote from the post "Learn about syllepsis, then refuse to stop employing it":

"As he ran a red light, the conversation back in his mind and away from his troubles, he couldn’t help but feel a sense of rising panic and the soft matte finish of his hand-stitched leather steering wheel. Angelica had been absolutely right and his wife for fifteen years, so why was he running scared, these kind of risks and this deadly gauntlet of illicit entanglements?"

Building a Social Data Commons

(author unknown) — Mon, 23 Nov 2009 19:30:21 -0800

Java String Encoding Performance

(author unknown) — Mon, 23 Nov 2009 07:12:31 -0800

new ones

arnab — Sun, 22 Nov 2009 21:38:35 -0800

Information:

(author unknown) — Sun, 22 Nov 2009 10:24:00 -0800

NEW: find only first-authored papers

(author unknown) — Sat, 21 Nov 2009 09:27:00 -0800

Thu, Nov 19, 2009

(author unknown) — Thu, 19 Nov 2009 22:07:00 -0800

math at midnight

Melanie Zhang — Thu, 19 Nov 2009 23:52:00 -0800

hours until my final: 13

pages of systemics notes reviewed tonight: 3
episodes of house watched: 2
stuff learned tonight: not very much
stuff forgot tonight: very much

my thinking goes, the more i study, the more i forget, so i might as well watch some House (at least it's entertaining, though really not all that educational).

i guess it was just a matter of time

(author unknown) — Wed, 18 Nov 2009 22:57:36 -0800

organizing background research

neha — Wed, 18 Nov 2009 07:28:00 -0800

I've been fuddling around with Zotero and a basic .bib file, but I think I found something more awesome: Mendeley. I found it looking for a PDF annotator, and it looks fantastic! I'll update later after I've used it a while. It even syncs with Zotero and can watch a folder for new PDFs.

You know what's hard?

(author unknown) — Mon, 16 Nov 2009 22:46:39 -0800

] [ppt

(author unknown) — Mon, 16 Nov 2009 23:59:00 -0800

JCOP Smartcard Performance

Victor Costan — Mon, 16 Nov 2009 21:22:00 -0800

Abstract
I use NXP JCOP smart-cards for prototyping in my research. I have recently benchmarked the cards using my research code, which contains computational and cryptographic workloads.

I found a couple of surprising results that I want to share, so fellow developers can make informed decisions when choosing their prototyping platforms.

Findings
There is a 1.5-2x speed difference between the different revisions of the same high-end chip, the NXP JCOP41 with 72kb of EEPROM. The V2.2 revision for smart-cards (no longer available, replaced by 2.2.1) has the best performance, and the V2.2.1 revision for the SIM (ID 000) form factor has the worst performance.

There is a significant speed difference between the same revision (V2.2.1) of the same chip (NXP JCOP41, 72kb of EEPROM), in different form factors. The chip for the smart-card form factor is almost as fast as the older V2.2 revision, while the chip for the SIM (ID 000) form factor is significantly slower.

There is a 2-4x speed difference between the same revisions (V2.2, smart-card form factor) of the NXP JCOP31 and the NXP JCOP41 chips.

The 3DES encryption/decryption engine has non-linear performance. The time it takes to decrypt 128 bytes is not very different from the time it takes to decrypt 24 bytes, on the JCOP41 chips. It seems that there is a huge setup cost for the DES engine, which outweighs the actual encryption cost.

There is a 4-8x speed difference between RSA and 3DES encryption on the JCOP41 chips, and a 3x speed difference on the JCOP31 chip. This goes against the conventional wisdom that symmetric encryption is 2 orders of magnitude faster then asymmetric encryption. This is probably due to the time it takes to setup the 3DES engine.

Conclusion
Secure processors in smart-cards have non-obvious performance characteristics. I hope my work saves you from the unpleasant surprises that I had.

Motivation
To the best of my knowledge, there are no easy to get benchmarks on smart-card processors. At least, I couldn't find anything when I searched.

Smart-card retailers disclose vital specifications, like EEPROM size and the cryptographic primitives that are implemented on the chip, but tend to be quiet about speed. The sites I used don't mention anything about the type of processor used, or the frequency of the processor.

For some applications (e.g. prototyping, where I want my unit tests to run quickly), speed is just as critical as the other specifications, and its more important than cost.

Data
The data that I used to reach my conclusions is available below. The benchmarks are described in section 5.1 (page 13) in my paper on a successor to the TPM.

decrypt_3des decrypts 24 bytes of data, while decrypt_3des_long and decrypt_rsa work on 128 bytes of data. 3DES is configured in EDE-CBC mode (112 bits of key material) and uses the ISO-9797 method 2 for padding. RSA decryption uses PCKS#1 padding.

The benchmarks can be reproduced by installing Rubygems, then installing the tem_ruby gem, and issuing the following commands

tem_upload_fw # Uploads my JavaCard applet to the active smart-card.

tem_bench # Runs the benchmarks.

NXP JCOP41 v2.2/72k (no longer available)
time_blank_bound_secpack_3des: 0.20757s
time_blank_bound_secpack_rsa: 0.86173s
time_blank_sec: 0.18017s
time_devchip_decrypt_3des: 0.05803s
time_devchip_decrypt_3des_long: 0.08042s
time_devchip_decrypt_rsa_long: 0.74047s
time_post_buffer: 0.08280s
time_simple_apdu: 0.00515s
time_vm_perf: 0.73887s
time_vm_perf_bound_3des: 0.78137s
time_vm_perf_bound_rsa: 1.43647s

NXP JCOP41 v2.2.1/72k (usasmartcard.com product link)
time_blank_bound_secpack_3des: 0.24155s
time_blank_bound_secpack_rsa: 0.89740s
time_blank_sec: 0.18937s
time_devchip_decrypt_3des: 0.08420s
time_devchip_decrypt_3des_long: 0.10800s
time_devchip_decrypt_rsa_long: 0.76480s
time_post_buffer: 0.08577s
time_simple_apdu: 0.00610s
time_vm_perf: 0.83257s
time_vm_perf_bound_3des: 0.90033s
time_vm_perf_bound_rsa: 1.55637s

NXP JCOP41 v2.2.1/72k USB token (usasmartcard.com product link, probably using this card)
time_blank_bound_secpack_3des: 0.41070s
time_blank_bound_secpack_rsa: 1.23089s
time_blank_sec: 0.34530s
time_devchip_decrypt_3des: 0.19010s
time_devchip_decrypt_3des_long: 0.21410s
time_devchip_decrypt_rsa_long: 1.05600s
time_post_buffer: 0.17213s
time_simple_apdu: 0.01000s
time_vm_perf: 1.11310s
time_vm_perf_bound_3des: 1.19703s
time_vm_perf_bound_rsa: 2.01420s

NXP JCOP31 v2.2 (usasmartcard.com product link)
time_blank_bound_secpack_3des: 0.84673s
time_blank_bound_secpack_rsa: 1.78957s
time_blank_sec: 0.78120s
time_devchip_decrypt_3des: 0.23553s
time_devchip_decrypt_3des_long: 0.50060s
time_devchip_decrypt_rsa_long: 1.54990s
time_post_buffer: 0.88864s
time_simple_apdu: 0.02813s
time_vm_perf: 1.84374s
time_vm_perf_bound_3des: 1.92594s
time_vm_perf_bound_rsa: 2.87900s

Facebook Tips: How do I find and "fan" a Page?

The Mass Animation of a Hollywood Short

Facebook Engineering Bootcamp

Optimizing asset bundling and serving with Rails

Benefits of our asset bundling

CSS & JS file layout

Rails Helpers

Bundle rake & deploy tasks

Tweaks to production.rb

Conclusion

Multiple file gist improvements

Building a Better Photo Uploader

New Privacy Policy Adopted

Xbox Connects with Facebook

GitHub Meetup SF: RubyConf Edition

How to Bullyproof Yourself on Facebook

GitHub Meetup London

Parents Have Rules to Follow Online, Too

Stay Tuned: TV on Facebook

Get Educated on Facebook

Facebook Tips: How Do I Change My Email Notifications?

Parents Have Rules to Follow Online, Too

GitHub Meetup Oslo, Norway

GitHub Rebase #30

Featured Project

Notably New Projects

Recent UI updates to GitHub

GitHub Meetup Poznań, Poland

Decertifying "Terrorist" Pilots

Al Qaeda Secret Code Broken

Quickpost: SelectMyParent or Playing With the Windows Process Tree

Best of Application Security (Friday, Nov. 20)

Friday Squid Blogging: New Squid Discovered

Interview with Me

FailBlog on Security

Hex-Rays Plugin Contest

Denial-of-Service Attack Against CALEA

Update: bpmtk with hook-createprocess.dll

A Taxonomy of Social Networking Data

Stabbing People with Stuff You Can Get Through Airport Security

How Smart are Islamic Terrorists?

Quantum Ghost Imaging

Secret Knock Lock

vsftpd-2.2.2 released

A Useful Side-Effect of Misplaced Fear

Page2RSS Monitored Page: Stefan Savage

Anti-Malware Detection and the Original Trojan Horse

Public Reactions to Terrorist Threats

Special LAMBDA meeting – Brent Yorgey

David Kravets on ACTA

Adding automatic semicolon insertion to a Javascript parser

Automatic semicolon insertion

Solution

David Pollak is happy about using Scala

Dehydra Testsuite Passes on GCC 4.5

A list is an odd creature, take 2

Go Channels

Go Channels

“Elements of Programming” Study Group

Easy File System Path Manipulation in C#

Three Kinds of Concept Maps

Visualizing Empires Decline

Ditto

Guide :: Heading for Deprecation: export, Exception Specification and register

A list is an odd creature

Mandelbulb

Extensible, Statically Typed Pratt Parser in C#

Go

Go

NEPLS submission

Hadoop World: Hadoop + Vertica from Omer Trajman

Comments on a fabricated press release quote

Recursive CTE’s: PostgreSQL

Big Data on Grids or on Clouds?

Battling Anti-NOSQL Trolls

Boston Big Data Summit keynote outline

Is Sandia National Lab's Red Sky Really Able to Deliver a PUE of 1.035?

MySQL Performance Blog was down today

The future of the data center is green: Takeaways from WiredRE data center event

Guide :: Heading for Deprecation: `export`, Exception Specification and `register`

showing pop-ups from `org-mode` appointments