taw's blog

Medieval 2 Total War Concentrated Vanilla 0.08

taw — Thu, 19 Nov 2009 00:08:00 -0800

Time for another version of my Medieval 2 Total War minimod. It doesn't contain that many changes over 0.07.

The biggest change is fixing the "everyone hates you" bug I wrote about a few days ago. That doesn't mean everyone will love you now - but now you have the option of acting chivalrous, getting good reputation, and if you do your alliances will be more likely to survive, your enemies will be more likely to sign peace with you instead of insisting on silly wars until you destroy them and so on. The game still makes everyone hate you if you're too strong, or if you're playing on very hard difficulty, so don't worry about it being too easy.

Rubber swords bug

The second bug fixed is the extremely well known "rubber swords" bug. In Medieval 2 Total War units have between 1 and 3 weapons, usually 2. It's absolutely necessary for missile units, which need some sort of missile weapon, and also some secondary melee weapon in case they have to fight man to man. But many units have two melee weapons - most knights have a lance for charging and sword for close combat.

Like many things this all makes sense in theory. The problem is with pikemen. They have long pikes for fight in formation, and swords for close combat. Unfortunately their logic for weapon choice is pretty much the same as with knights and their lances - instead of trying to use pikes as long as they can, they switch to swords almost immediately after first charge. And with very weak swords, no shields, and weak armor pikemen make for really horrible infantry.

If it was just a matter of them being weak, it wouldn't be that big of a deal - there's plenty of weak units in game. The real problem is that they switch to rubber swords too early, and you don't see any pike action at all! Removing their "rubber swords" completely makes them much stronger, and behave properly.

Now there's a valid counterargument, that is raised on Total War forums every time the rubber sword bug is mentioned - the game was balanced with rubber sword behaviour, Pikemen are extremely cheap, and so shouldn't be expected to be much good.

It has some validity - Pike Militia costs only 150 florins, half as much as Town Militia at 290, so making them into some sort of super-unit would unbalance the game. Recruitment costs are only meaningful in battle mode (that is mostly multiplayer). In campaign mode, cost of an unit is cost of its recruitment, upkeep over multiple turns, and of buildings necessary to recruit the unit.

Compare scenarios of building required buildings (other than basic things like walls and barracks up to city watch and you need anyway), recruiting 10 (for vanilla, or 15 for mod as buildings are 50% more expensive) units from it, keeping them for 5 turns in garrison, and 5 turns in field. Costs per unit would be:

Town Militia: 290 recruitment + 0 garrisoned + 625 in field + 0 only basic buildings = 915
Genoese Crossbow Militia: 570 recruitment + 0 garrisoned + 500 in field + 0 only basic buildings = 1070
Peasants: 110 + 450 garrisoned + 450 in field + 60 mustering hall = 1070
Italian Spear Militia: 460 recruitment + 0 garrisoned + 625 in field + 0 only basic buildings = 1085
Peasant Archers: 220 recruitment + 500 garrisoned + 500 in field + 120 bowyer = 1340
Peasant Crossbowmen: 220 recruitment + 500 garrisoned + 500 in field + 360 practice range = 1580
Pike Militia: 150 recruitment + 0 garrisoned + 625 in field + 960 militia barracks = 1735
Mailed Knights: 680 recruitment + 1250 garrisoned + 1250 in field + 0 only basic buildings = 3180
Lancers: 930 recruitment + 1600 garrisoned + 1600 in field + 2760 all stables = 6890

Recruitment costs are simply a tiny fraction of total costs! Now this depends on which building you'd build anyway (you always need walls; city watches increase public order, shipyards increase trade etc. so you might buy them anyway), how many units you're going to recruit from each such building, how long you're going to keep them garrisoned, and how long in field and so on. These are not definite numbers - but they should give you some idea. It's quite easy to spot the most overpowered and most underpowered unit on the list here - Genoese Crossbow Militia is as cheap as Peasants, but they're ridiculously stronger - it's one of the best missile units in game versus absolutely the worst unit in game!

So anyway, I did some experimenting with pikemen. I've heard about two solutions on Total War forums - either remove their secondary weapons, or make their formations tighter. I made two AIs fight each other - 3 Pike Militia versus 3 Italian Speak Militia. Results were:

Normal Pike Militia - lost, 86% dead vs 37% dead
Tight formation - lost, 83% dead vs 58% dead
No secondary weapons - won, 66% dead vs 80% dead
Tight formation + no secondary weapons - lost, 88% dead vs 88% dead

The only reason tighter formations work is by making it harder for the enemy to get real close, and making pikemen use their pikes longer. Now AI was clearly unbelievably dumb here - happily charging on the pike wall. Manual results were:

Me normal Pike Militia, AI Italian Speak Militia - I won, 65% dead vs 89% dead
Me Pike Militia without secondary weapons, AI Italian Spear Militia - I won, 37% dead vs 85% dead
Me Italian Spear Militia, AI Normal Pike Militia - I won, 1% dead vs 20% dead
Me Italian Spear Militia, AI Pike Militia without secondary weapons - I won, 4% dead vs 13% dead

So properly used, Italian Spear Militia is far stronger than Pike Militia - properly used means flanking, and not blindly charging onto the pike wall. Nothing too complicated. Claims of Pikemen being overpowered without secondary weapons seem greatly overstated.

Scaling down previous changes

I decided to scale down a few changes. Cavalry no longer costs 50% extra - nerfing them was fun, but now that most infantry gets free upkeep in castles, and pikemen actually work, cavalry is decently nerfed anyway.

I reduced extra ammo for missile infantry from +100% to +50%. It doesn't really make that much difference, and I'd rather stay closer to vanilla values unless I have a good reason not to.

Full list of changes

Here's the full list for reference

Fixed standing bug
Fixed rubber swords bug
Castle garrisons free for basic infantry
City garrisons twice as big
King's purse x2
Rebel spawn rates 10x lower (supposedly, I have my doubts if it works)
Missile infantry ammo +50%
Bodyguard unit sizes halved
Building construction times all 1 turn
Mining income x2
Building costs other than mines +50%, mines x3
When defending a settlement, towers are activated from much greater distance
Wall and gate strength x5, but tower strength unchanged
Tower fire rate x2, but only for non-flaming ammo
Spy costs x2
Campaign movement speed +75%

Here's the download.

Dietary consequences

taw — Tue, 17 Nov 2009 08:33:00 -0800

This post is a follow-up to The myth of 5 a day - people never ate that much fruit and veg.

"Because" is not a logical connective of classical logic, it's not possible to construct a truth table for "A because of B" statement. Even when A and B are true, it's really quite difficult to prove that causality flows from B to A - correlations, no matter how strong, don't prove causality.

On the other hand, while proving such statement true is hard, proving it false can be very simple - if either A or B is false, "A because of B" must necessarily be so too. So "crime is on the rise because of violent video games" can easily proven to be false, because the consequence part ("crime is on the rise") is false - therefore collapsing the whole argument before we get anywhere near analyzing causality.

"Guns prevent crime" / "death penalty prevents crime" arguments collapse equally fast - countries with more guns and more executions don't have less crime - so we don't even need to think much about causality here. That doesn't mean that reverse statement "gun control prevents crime" is necessarily true, causality might be missing even if correlation exists. By the way there is actually some pretty good research indicating that violent movies deters violent crime, at least in short term - apparently because violence-loving young males go to theaters to watch violent movies instead of getting into real crimes. It's not too unlikely that it might be the same with violent video games.

Now that we're past the obligatory cat and the obligatory detour, let's get back to diets. There's a widely repeated claim, which basically goes "people are fatter than in the past because their diets have much more fat, especially animal/saturated fat than in the past, and nowhere near as much fruit and vegetables as in the past". The consequence is undoubtedly true - obesity is on the rise pretty much everywhere.

On the other hand, the part about too much (saturated/animal) fat, and not enough fruit and veg is entirely wrong. We actually eat a lot less fat; the fat we eat is increasingly unsaturated vegetable fat; and we eat a lot more fruit and vegetables than in the past.

Take a guess - what are the biggest sources of calories in American diet? Here's the list for 1961 and 2003, in kcal/day. I don't believe 1961's diets were perfect, and I'd love to have data earlier than 1961, but that's as far back as FAO statistics go.

Vegetal products consumption increased a lot - 1871 (65%) to 2708 (72%) kcal/day, while animal products consumption stayed constant in absolute terms, and decreased a lot proportionally - 1010 (35%) to 1045 (28%).

Top vegetal products:

cereals without beer - 627 to 832 (of which wheat - 504 to 603; rice 26 to 94)
sugar and sweeteners - 515 to 657 (of which sugar 453 to 320; HFCS 56 to 331)
vegetable oil - 276 to 606 (of which soybean oil 157 to 492)
alcoholic beverages - 108 to 150 (of which beer 68 to 100)
"fruits" without wine - 79 to 117 (of which oranges 14 to 32)
potatoes - 77 to 100
maize (without HFCS, maize oil etc.) - 59 to 98
"vegetables" - 63 to 77
beans - 32 to 30

It's quite surprising list - foods which can be considered traditional like wheat, potatoes, and beans, are a really small portion of the whole. Modern vegetal products like soybean oil (unsaturated fat), sugar+HFCS (sugar, obviously not fat) dominate the list.

And for animal products:

meat - 335 to 451 (of which poultry 64 to 197; pork 127 to 132; beef 125 to 115)
whole milk - 304 to 199
animal fat products (butter etc.) - 199 to 116
cheese - 48 to 149
eggs - 67 to 55
butter - 65 to 40
fish and seafood - 19 to 28

Total fat and protein consumption in g/day:

Animal fat - 69g to 71g (63% to 46%)
Vegetable fat - 40g to 83g (37% to 63%)
Total fat - 110g to 155g

What increased here was poultry - the approved lean meat; and cheese - the most vegetarian of animal products; what decreased most was whole milk, butter, eggs, and animal fats as whole category.

This is as far away from the conventional story as it gets. Proportionally to their diets, people eat less fat, a lot less of the supposedly "unhealthy" saturated animal fat, less milk and butter (saturated fat), vastly more "healthy" vegetable fat, more lean meat, more vegetables, more fruit. Sugar+HFCS consumption increased a lot in absolute terms, but stayed fairly constant proportionally, even decreasing slightly from 17.8% to 17.5% of all calories.

So it's not that people lack personal responsibility - people adjusted their diets to follow every single point of conventional dieting advice. To keep blaming people's eating habits for obesity epidemic is to blame the victim. You cannot claim that people are fat because they don't follow dietary advice, as they actually do!

The only alternative is that the conventional advice doesn't work. Eating more fruit and veg, switching from animal to vegetable fats, reducing proportion of fats in diet and so on - does not make you slimmer. I won't answer the question if the advice is merely useless or actually harmful in this post (causality in reverse direction) - that would require a lot of research. But we sure know it doesn't work.

I doubt that another round of minor adjustments to the advice (look at all reversals on butter vs margarine), let's say by insisting on more fish oil this time, are likely to magically make it work now, in spite of consistent track record of failure.

I don't have enough data to give you reliable advice, but the most obvious alternative would be eating reasonably-sized portions of the most delicious food you can think of. This will usually be something fairly traditional (even not necessarily from your culture), often but not always with plenty of animal products in it, and extremely rarely with much sugar or vegetable oil. If type of food doesn't matter, and only portion size does, then you can as well enjoy it, instead of torturing yourself with diets.

For the record, author of this post has BMI 23. Never trust overweight people to give you diet advice.

Why everyone hates you in Medieval 2 Total War

taw — Sun, 15 Nov 2009 21:34:00 -0800

Short answer - because of a really stupid bug in data files. If you want it fixed, download corrected file and put it in your C:\Program Files\SEGA\Medieval II Total War\data\ directory or similar. For the long version, read on.

Reputation and standing

First, technicalities. The game keeps track of two important numbers - your reputation (also called global standing), which is global, and standing with a particular faction. Both are numbers between -1 (the worst possible) and +1 (the best possible).

Good behaviour increases your reputation, and your standing; while bad behaviour decreases it - as expected. There are two kinds of deeds - some actions are unconditionally good or bad - they simply add or subtract from your reputation/standing. If you do 5 good deeds and 4 bad deeds of this kind, it's pretty much the same as just doing 1 good deed.

The other kind use what the game calls "normalization". This "normalization" operation has two parameters - target value, and "divisor". It takes your current standing or reputation, and moves it by 1/divisor towards the target.

For example if your current standing is 0.6, then "normalizing towards +1 with divisor 10" will result in standing of 0.6 + (+1 - 0.6)/10, that is 0.64. If your initial standing was -0.8 instead, the result would be -0.8 + (+1 + 0.8)/10, or -0.62. In other words - if your reputation/standing is bad, good deeds improve it a lot, while bad deeds damage it little. And if your reputation/standing is good, good deeds improve it little, but bad deeds damage it a lot. In our example, the good deed had value of +0.04 for the well-reputed faction, but +0.18 for the ill-reputed one.

Normalization to some value with divisor 1 is equivalent to completely changing your reputation/standing - by our equation initial + (target - initial) / 1 = target. Doing both good and bad things of this kind will move your reputation towards neutral values around 0.0, even if one kind of deeds is more frequent than the other. It's all quite sensible in practice.

Obviously we want our relations with different factions to be high - factions that hate us will be more likely to start wars against us, disrespect treaties and so on. Even with enemies we would prefer higher standing, in case we want to sign peace treaty someday.

What changes reputation

Most reputation changes are absolute.

First, your reputation normalizes to 0 with divisor of 200 anyway, which is extremely slow - it takes 140 turns to halve your good or bad reputation. Having an ally normalizes your reputation to +1 with divisor of 400, and being at war (except with rebels obviously) normalizes your reputation to -1 with divisor of 800. These are extremely small divisors, but if you have multiple allies, or multiple enemies they add up.

On top of that, some generic good and bad deeds:

"Taking back" settlement, which you gave by diplomacy (but then do you ever give settlements by diplomacy?): -1.0
Attacking an ally: -0.5
Canceling alliance (also when forced by one of your allies attacking another - tough luck): -0.1
Bribing settlement (it's virtually impossible in Medieval 2 Total War anyway, unlike in Rome Total War where you would be doing it all the time): -0.15
Breaking treaty terms (I'm not sure what counts here): -0.15
Military assistance to your ally in battle: +0.1

Treatment of prisoners is probably the most important source of reputation. If you put your prisoners up for ransom, it doesn't affect your reputation either way - this is the standard expected behaviour in Medieval warfare. What changes it is either releasing or executing prisoners.

Releasing prisoners: +0.01; +0.025 for groups of over 80 soldiers; +0.025 if any family member was released.
Executing prisoners: -0.01; -0.025 for groups of over 80 soldiers; -0.025 if any family member was executed.

I'll get back to this later in more detail.

And here we arrive at the bug. Treatment of civilians is supposed to affect your reputation:

Occupying a settlement peacefully: +0.05
Sacking a settlement: -0.02
Exterminating population: -0.05

But due to a bug the first never happens. Here's the data file:

;------------------------------------------
Trigger 0103_occupy_settlement_increase_global
WhenToTest OccupySettlement

FactionStanding global 0.05
FactionStanding target_faction normalise 1.0 20
FactionStanding target_allies normalise 1.0 40
; FactionStanding target_enemies normalise -1.0 40

;------------------------------------------
;Trigger 0102_city_razed_decrease_global
; WhenToTest CitySacked

FactionStanding global -0.05
FactionStanding target_faction normalise -1.0 10
FactionStanding target_allies normalise -1.0 20
; FactionStanding target_enemies normalise 1.0 20

Semicolon means code is commented out and inactive. The first event describes what happens when settlement is peacefully occupied. The second is old sacking code, probably from Rome Total War times. It's supposed to be commented out, but only trigger was commented out, the effects were not. So the results are equivalent to:

;------------------------------------------
Trigger 0103_occupy_settlement_increase_global
WhenToTest OccupySettlement

FactionStanding target_faction normalise 1.0 20
FactionStanding target_faction normalise -1.0 10
FactionStanding target_allies normalise 1.0 40
FactionStanding target_allies normalise -1.0 20

So not only you don't get the promised reputation bonus, your standing with the target and their allies gets lower, instead of higher - they're actually happier when you sack their settlements than when you occupy them peacefully, what is nonsense! By the way, all websites about Medieval 2 Total War I've seen seem unaware of the bug, even though it's trivial to test.

Just see how different the game is with and without bug. If you take 6 neutral settlements, your reputation should rise to 0.3 (Reliable); with the bug it will stay at 0.0 (Mixed). This has huge effect on how others will treat you!

Standing

Now that we know about your reputation, how about standing? There's extremely long list of rules related to the Pope, crusades and such, which I'll cover later. For normal factions it's much simpler. I'll spare you the numbers, but basically:

Factions with high reputation get liked more, factions with low reputation get liked less. Thresholds are -0.4, -0.1, +0.1, +0.4.
Depending on difficulty, your standing tends to drift towards +1.0 (very easy), 0.0 (medium), -0.5 (hard), and -1.0 fast (very hard).
Factions which are strong get liked less, factions which are weak get liked more.
There's small advantage to having the same religion.
Your allies grow to like you more, and your enemies to like you less. Allies/enemies of your allies/enemies change their standing accordingly, by smaller factors. If this works correctly, you can see two big coalitions developing, and it often happens.

There are miscellaneous actions that increase or decrease your standing. In addition to the obvious acts of war, bribing, sabotage, and spying decreases your standing - spying much more than I thought - to -1.0 with divisor 20, and with divisor 80 for all allies. I'm not sure if it counts only when spy is caught, but it's far higher than I expected. Only 13 acts of spying will move perfect standing all the way to neutral.

What doesn't affect your standing is merchants buying each other out, your princesses marrying (and stealing) their generals, and your priests preaching on their territory. Marrying your princess to their faction heir is worth a lot - +0.2 instantly, and normalization to +1 with divisor 20 every turn.

Adding it all together - the way it's supposed to work is that increased standing due to high reputation should counteract decreased standing due to your size and difficulty level (at least on hard, maybe not so much on very hard). But because you don't get reputation points you deserve for peacefully occupying settlements, there's no balance, and everyone gets to hate you.

The Pope

It's only vaguely related to the bug this post is about, but as I started talking about Medieval 2 Total War standing system, I can as well describe how it works with the pope.

First you get points for building religious building - from 0.02 for small churches and chapels, to 0.2 for huge cathedrals, and additional 0.2 for your first huge cathedral, and additional 0.2 if it's the first huge cathedral in the world - so you can get +0.6 with the pope for just one building, after first getting +0.52 for first cathedral in the world.

On the other hand destroying religious building for money makes the pope hate you, about 5x as much as building them. But as you never really destroy buildings anyway, you probably don't need to bother yourself with it too much.

Crusade get you huge bonuses with the Pope - but only arriving at target region, and taking the target settlement, not just joining. You get bonuses for every general, and every unit, but a lot more if you send your heir (+0.2) or your leader (+0.6) to the crusade.

Something I didn't know about before reading the code is that attacking Orthodox factions while on crusade (usually taking Constantinople on the way to Jerusalem or Antioch) gets pretty high penalties with the Pope. You'll get your papal standing back if you actually conquer your target, but if the crusade is just an excuse for sacking the Byzantine Empire the Pope will not be happy. As you probably already know, attacking non-excommunicated Catholic factions while on Crusade, or Islamic factions while on Jihad is prohibited by the rules.

Recruiting priests and doing papal missions get you points. Getting excommunicated loses you -1.0 points, obviously. Getting your cardinals promoted get you points (0.1 per cardinal), and there are even more points in elections - Pope from your country means +0.8, from allied country means +0.4, enemy country's Pope means -0.6, and voting for the winner pope is +0.2. Surprisingly I don't see any negative points for voting for the wrong candidate, even though in-game text insist on wrong vote being a major transgression.

You also lose points with the Pope for being at war with other Catholics. And look after heretics in your lands - if the Pope has to appoint an Inquisitor, it costs you immediate -0.2.

Now it's going to more or less even out - you will build churches and recruit priests anyway, especially if you're close to Islamic or Orthodox regions - what will score you major Pope points. And you'll lose points for inevitable wars with other Catholics. You're supposed to get extra points for good reputation, but as I said, you won't due to the bug.

What to do with prisoners

There's one more thing I wanted to write about - treatment of prisoners. You naturally know that you shouldn't normally sack or massacre settlements, as these will be your subjects, but why do anything except execution with the prisoners? Let's do some game theory:

Value of reputation gained or lost: A (by marginal view it's supposed to be the same either way, it's not exactly true, but let's keep it this way for simplicity)
Value of money gained from ransom: B
Cost of enemy getting units back: C

So payoffs are:

Released: A-C
Ransom accepted: B-C
Ransom rejected: 0
Executed: -A

It helps to know if the enemy is likely to accept the ransom or not. Usually the answer is clear - due to war they don't have any money, so they cannot pay you. In this case if A>C, you release prisoners, otherwise you ask for ransom and don't get it. A>C is the usual situation, you've just defeated their armies, so they probably weren't that good anyway. The biggest exception I can think of is when you're on your way to take a weakly defended settlement, and these armies would join the defenders and make your work much more difficult when released. Or when it's really late in game, you're at war with everyone, and you don't care about reputation anyway.

How about when you think the ransom will be accepted? First, decision between release and ransom. If A-C>B-C, release straight away. Unless you captured their leader (10000) or heir (5000), reputation will be almost always more valuable than money - mostly because for reputation purposes they're treated just like any other general, but for ransom purposes they're a lot more valuable. Except your enemies are quite unlikely to have that much money, so it's not a common occurrence.

How about executing them? It only makes sense when C>2A, and C>A+B. Reputation cost of execution is twice as high, as now you have to actually execute the prisoners, instead of just request ransom. Your loss due to release of troops is very rarely that high. Even tiny group of soldiers released or executed will make a difference of 0.02 reputation points, about as much as you need to counteract being at war with someone for 16 turns. If you act chivalrously wars improve your reputation instead of damaging them. Paradoxically the worst kind of war for your reputation is a prolonged war in which nothing happens.

This is all a lot less true once the bug is fixed. Due to the bug releasing prisoners is the only way to significantly improve your reputation, and you need that just to stay positive, as you'll lose it due to broken alliances, executions of prisoners, and sackings of settlements that will inevitably happen sometime during the game. Having allies and assisting them militarily just won't be enough - once big major start, getting allies will get really tough, especially if your reputation isn't very high (chicken and egg...), and assisting stupid bots requires way too much of effort.

But without the bug, you should have decent reputation from just straightforward conquest, unless you kill and sack on your way - and value of A gets a lot lower.

To fix or not to fix

With minor bugs the question is clear - definitely fix them. But this bug is so big, and changes the game so much... Not only AI will like you more, AI will like each other more too. And acting chivalrous will be much more important - with buggy M2TW you really had to go out of your way to get decent reputation, but with the fix if you behave you'll reap diplomatic rewards.

Most likely the fix will make the game significantly easier - as long as you act chivalrous it will delay your wars and make your enemies much more likely to agree to cease fire. On the other hand it will make AI factions hate each other less too - so who knows, maybe they'll unite against you? At some point you'll be big enough to overwhelm your reputation-based bonus, so maybe it doesn't matter that much...

I'll try my next game with the fix, and tell you how it went. It's a really long post for a bug that's basically 3 missing semicolons.

Things I read

taw — Thu, 12 Nov 2009 02:34:00 -0800

As some people expressed interest in RSS feeds I follow, here they are.

For every feed I want to include some recent good item (if there's none, then obviously I should just unsubscribe). Standard fair use disclaimer here (some comics are explicitly CC-*, but most aren't; it's all legit fair use, no matter what Rupert Murdoch thinks).

Webcomics

First, I love webcomics, and there's plenty of those in my feed. Most of them are "one self-contained funny story per item" type, which work very well with RSS readers.

Abstruse Goose is sort of xkcd-ish, crudely drawn smart jokes about science and pop culture, less obsessed with girls than xkcd, and without mouse-over tooltips.
oglaf.com calls itself "Comics. Often dirty.", and is exactly that. Most of it is NSFW, so if you need something to read while sipping a morning coffee in the office, and waking up, it's probably not it. This sample is from quite far back, not because more recent ones weren't good, but because it was hard to find one that was SFW. It's also very nicely drawn, unlike most comics here.
Cyanide & Happiness is an extremely well known comics, strangely with multiple people drawing it, all in more or less the same style. I'll use this opportunity to rant about one infuriating thing about their RSS feed is that very often comics get into RSS feed before they get live on their website (it happened like 4 times just last month).
Order of the Stick is something completely different - instead of being "one funny a day", it actually has a multi-year story of a team of adventurers playing Dungeons&Dragons-inspired campaign and making fun of the rules as they go. Well, at first it was "make fun of D&D vaguely connected strips" format, I think it got significantly worse as it tried to get serious, and most of the humour is about OotS characters, not D&D. Here's one really old strip that you'll get even if you don't follow OotS.
Saturday Morning Breakfast Cereal is another one of those good funny comics - it delivers daily, and has fairly mainstream appeal.
xkcd probably got more cult following than any other webcomics ever. In addition to fans, there are blogs of xkcd hate, and even RSS-able Web 2.0-style website Is xkcd shitty today - their answer always being YES, most often unfairly. It can get really cutesey and sad, or mindlessly evil, or philosophical, or anything else. There are many meciocre xkcds, but it reaches pure comics awesomeness level more often than any other comics I know. And don't forget the mouseover tooltips.
Zero Punctuation isn't really a webcomics - it's a weekly sarcastic video game review. It's universally loved as it tends to find faults in games and make fun of them, while most of the so called "reviewers" just rehash press releases. Here's one old review of one particularly good game:

Reading

All right, I sometimes read and not just look at pretty pictures.

538 is probably the only blog about political polling in existence. It has some highlight, such as predicting results of 2008 American elections much better than anyone else, and finding out that one polling firm was a total fraud, and all their results were fabricated.

It contains a mix of detailed analyses of particular elections (not terribly interesting, unless you happen to live there), criticism of bad polling and bad analysis, and probably most interesting of all - big picture view of shifts in public opinion. Unfortunately it's mostly about USA, and rarely writes about Europe and other places.

Less Wrong is a spin-off of Overcoming Bias, or what Overcoming Bias used to be before becoming Robin Hanson's personal blog. It's filled with insightful posts about human rationality, and biases in thinking, and philosophy of thought. It also has scary amount of discussion that's only understandable to insiders. Some examples:

There's plenty more, and I even sometimes post there.

Marginal Revolution is one of the Freakonomics-style "economics made fun, now with more anecdotes" blogs. It publishes insanely often, five posts a day pace is entirely typical. Unlike this blog, which gets more like five posts a month. A lot of it is reposts of interesting things they found somewhere else, what saves you from following too many feeds. Like repost of this bathroom scales that tweets your weight for all world to see as motivating factor, and experiments with placebo coffee.

Actually 90% of Marginal Revolution is reposts with tl;dr summaries, comments, and counter-arguments. It's pretty good this way. Sometimes they get into crazy libertarian mode, especially when discussing health care, but it doesn't happen that often.

Paul Krugman's blog, and New York Times column (which is like blog, just with longer posts). Krugman can get annoyingly dogmatic about whatever the left wing of Democratic Party believes at time, and often gets horribly "wonkish" about tiny details of macroeconomic analysis. Some good posts:

Krugman can be as bad as dogmatic libertarians, but at least it's bad in the opposite direction. It would be highly amusing if Paul Krugman and Tyler Cowen were locked into one room for a week, and forced to Aumann.

Overcoming Bias used to be Eliezer's blog about rationality, then Eliezer moved to Less Wrong, and Overcoming Bias is all Robin Hanson's. Robin has three big ideas: everything people do is about signaling status, modern medicine doesn't work, and prediction markets are universal solution to every problem. Robin has way too many crazy libertarian moment, so if your libertarian-phobia is stronger than mine do not read. Some good recent posts:

Overview of evidence for panspermia
Estimated 41% of "elections" worldwide are fraudulent (mostly poor countries obviously)
Evidence for flu vaccines is dubious

Peak Oil Debunked used to fight peakoiler doomsdayers during their high days of doom. The doom didn't come, peakoiler movement sort of died, so there are few posts these days, but it was really good back then. Most posts are just facts and figures, something peakoilers hate with passion. (my recent post about geoengineering and its followup were in similar style)

Some nice posts:

Schneier on Security is probably the least controversial of blogs on my reading lists - most of them do politics from time to time, but it's hard to be angry at Schneier about anything. Some good posts:

And the last feed I follow - the War Nerd. Unfortunately it hasn't been posting much since The Decider left. He's way better at predicting course of events on the Middle East than the mainstream media - mostly because mainstream media is just rehashing White House press releases instead of doing any independent thinking. Some good pieces, most pretty old:

Plus countless pieces about Iraq and Afghanistan of course.

Enjoy your reading.

Inevitability of geoengineering, part 2

taw — Tue, 10 Nov 2009 02:18:00 -0800

In my previous post, I explained why geoengineering to counteract global warming seems inevitable. To summarize, data suggests that between 2009 and 2050:

World population will increase by about 45%
Poor people are getting a lot richer, average wealth will increase by 200%
This means they will use a lot more energy than now
Neither renewables nor nuclear power seem likely to be able to replace them by 2050, at realistic growth rates
Therefore only plausible source of that energy are fossil fuels, and we already have plenty of examples of countries drastically increasing their fossil fuel use and CO2 emissions extremely quickly.
Reduction in energy use by rich countries is nowhere near big enough to counteract this

What changes around 2050?

World population is expected to more or less flatten out around 2050. We're already adding fewer people each year than during peak growth, so growth is sublinear, let alone expecting it to be exponential. Long term projections need to be taken with a pinch of salt, but it's as safe as it gets to expect much slower growth in second half of 21st century than now.
While world economy will keep growing, average GDP per capita will be more or less where EU levels are now, and energy use past this level doesn't increase that much.
In all likelihood, sources of energy other than fossil fuels will be far better developed than now, so responding to any new energy needs by more renewable energy instead of by more coal power plants, will be much more likely.

Now I don't know exactly when we will have peak CO2 emissions, and at what level - somewhere vaguely around 2050, and at levels twice current seems likely, as decades before it will have much higher pressures to increase fossil fuel use than decades after.

The first problem, is that peak CO2 concentrations will happen long after peak CO2 emissions. Here's a random graph with emissions on the top, and concentrations on the bottom, color-indexed. In all scenarios, including almost immediate halving of emissions, peak is somewhere in 21st century, but CO2 concentrations keep growing long past 2300, peaking hundreds of years from now!!!
Now I don't actually believe CO2 emissions will be falling that slowly - once we figure out how to use renewable energy cheaply, we'll be able to switch almost all our use to it, so peak concentrations will be somewhere during 22nd century.

It gets worse - many effects of CO2 concentrations, like temperature change and ocean acidification, lag it significantly. So peak warming and peak acidification will occur much later than peak concentrations. With 2050 emission peak, and 2150 concentration peak, we might have 2250 climate change effects peak. (last figure completely made up, as it depends on too many details, different effects have different lags, and nobody makes serious estimates that go that far into the future). So even with a lot of optimism about alternatives to fossil fuels, we will have to deal with massive effects of global warming.

This gets us back to geo-engineering. No matter what we do now, short of a genocide and collapse of global economy (world economy grew during the recession, and fell only a few percents in the worst hit countries; this isn't the collapse I'm talking about), our choices are geo-engineering and adapting to higher temperatures.

Disclaimer time

Now disclaimer time. I'm making the common mistake that anthropogenic CO2 emissions are responsible for all global warming. This is not true - added CO2 is only 2/3 of it, the rest being methane, nitrous oxide, CFCs. But there are good news here - their lifetimes in atmosphere are much shorter - while we're talking tens of thousands of years for CO2, it's only 12 years for methane, and about 100 for nitrous oxide and CFCs, so peak concentrations will occur relatively quickly after peak emissions. And in fact, peak concentrations of methane and CO2 seem to have already happened or be happening about now. In longer perspective global warming problem is CO2 problem, with only small methane and nitrous oxide contribution.
The second disclaimer is that there are two kinds of geoengineering - solar radiation management (stratospheric sulphides, cloud seeding, albedo changes and such), and carbon capture and storage.

With solar radiation management we let warming happen, and create equal negative effect.

With CCS we could pull more CO2 from atmosphere than we're emitting. If we went this way peak concentrations might happen much faster after peak emissions than otherwise, and due to rapid fall in concentrations peak warming will be earlier and smaller.

I'm mostly talking about solar radiation management, as we already know how to cool down the planet with current technology and for ridiculously small money; while CCS solutions don't seem to be anywhere near scalable enough soon enough.

Any alternatives like fussion power, nanotech, and new kinds of renewables that people love to talk about won't seriously affect emissions until long after 2050 even in the most optimistic scenarios.

The myth of 5 a day - people never ate that much fruit and veg

taw — Mon, 09 Nov 2009 23:12:00 -0800

NHS (to American readers - it's the British socialized medicine destroying UK, and coming to destroy your country too now) strongly promotes the idea of eating "5 servings of fruit and vegetables" a day, with which I disagree on so many levels.

Here's a quick summary of the alleged benefits, straight from their website:

They're packed with vitamins and minerals.
They can help you to maintain a healthy weight.
They're an excellent source of fibre and antioxidants.
They help reduce the risk of heart disease, stroke and some cancers.
They taste delicious and there's so much variety to choose from.

That's all nonsense.

You are extremely unlikely to have vitamin and mineral deficiency, and if you suspect some (or want to get pregnant, have symptoms of anemia etc.), get real multivitamin pills, or supplements of the right kind. Most fruit and vegetables are fairly unimpressive as for their vitamin contents anyway - for all practical purposes most fruits are just water and sugar, and have been selected this way by farmers, as larger and sweeter fruit sells better, and vitamins don't have taste.
There's absolutely no evidence linking fruit and vegetable consumption to weight - you get stable weight when number of calories taken balances number of calories expended, what's possible on any kind of diet, including a really delicious one based on loads of meat and cheese and chocolate that NHS would not approve of. I'll get back to this point later.
Fibre doesn't do anything - it just passes through your digestive system and goes right into the toilet. "Antioxidants" are one of those terms that make me rage, as they've been stripped from their original scientific meaning, and turned into pure value judgment, and not even of the right value. Your cells know how to deal with oxidative stress already. That's what they're evolved to do over the last 1.7 billion years. They have plenty of ways to do so. And increasing amount of antioxidants will not help your cells. Meta-analysis of all available data shows clearly that eating more of them doesn't do anything for you, and can even be harmful.
There's been no proper long term trials of that kind. There's very few proper long term trials of anything related to diet, and those that are performed invariably show that simple diet advice (eat this, don't eat this) doesn't work - and unless your diet is spectacularly wrong you'll be fine as long as you have adequate exercise, decent portion control, and don't smoke.
As any kid will tell you, fruit and vegetables taste awful - if they were delicious people wouldn't need NHS's bothering about them. Do chocolate and ice cream need NHS campaigns? Obviously not. That's because chocolate and ice cream are tasty, while fruit and vegetables are not, at least not in those ridiculously large amounts.

So it's wrong on all counts. Anyway, that's not what I wanted to post about - I just get distracted easily. Oh, something shiny. Oh, funny cat. I go for all distractions. Now back on track.

What I wanted to talk about was the myth that people used to eat enormous amounts of fruit and vegetables - 5 servings a day or more as NHS advices, and that's it's something modern that they do not do so now. Let's look at FAO data, annual consumption per capita of fruit and vegetables, as far back as it goes (that's unfortunately not that far back). I included some particularly fat countries (USA, Mexico), a particularly healthy country (Japan), random countries from Europe (France, Germany, Poland, UK), and totals for Europe and World. For interest of science I'll say I selected countries and criteria before I had a first look at the data, so it's not a biased selection, but feel free to look at others.

Vegetable consumption, 1961 to 2003, increased in all selected countries:

France - 150kg to 142kg
Germany - 49kg to 90kg
Japan - 95kg to 104kg
Mexico - 25kg to 58kg
Poland - 91kg to 100kg
UK - 60kg to 91kg
USA - 93kg to 123kg
Europe - 90kg to 117kg
World - 63kg to 117kg

Fruit consumption (excluding wine), 1961 to 2003, also increased in all selected countries:

France - 53kg to 95kg
Germany - 79kg to 113kg
Japan - 29kg to 54kg
Mexico - 57kg to 120kg
Poland - 18kg to 47kg
UK - 55kg to 115kg
USA - 76kg to 113kg
Europe - 65kg to 88kg
World - 38kg to 62kg

Fruit and vegetables together, 1961 to 2003:

France - 203kg to 237kg (+17%)
Germany - 128kg to 203kg (+59%)
Japan - 124kg to 158kg (+27%)
Mexico - 82kg to 178kg (+117%)
Poland - 109kg to 147kg (+35%)
UK - 115kg to 205kg (+79%)
USA - 169kg to 236kg (+40%)
Europe - 155kg to 205kg (+32%)
World - 101kg to 179kg (+77%)

So consumption of fruit and veg in the last four decades drastically increased - and so did obesity, heart disease and so on. Increase in UK and Mexico were particularly large - and these are some of the fattest countries in the world, something that was not true in 1961. In increased least in Japan - by far the healthiest of developed countries. NHS was really successful at making Brits eat 5 servings a day - making 63.8% of them overweight in process. So much for fruit and veg consumption helping fight obesity? Here's the list of selected countries ordered by fruit and veg consumption and percentages of overweight people in them:

France - 237kg - 40.1%
USA - 236kg - 74.1%
UK - 205kg - 63.8%
Germany - 203kg - 60.1%
Mexico - 178kg - 68.1%
Japan - 158kg - 22.6%
Poland - 147kg - 47.5%

Is it just me, or do countries that eat more fruit and veg tend to be fatter, not thinner? The answer is probably a lot more like this instead:
And it's not like 1961's fruit and vegetable consumptions were unusually low - they were in fact unusually high by historical standards - normally it was impossible to eat any other than those that grew in your region, in harvest time, what all amounted to very very little compared to what people eat now with globalized industrial agriculture.

The take home message: Ignore NHS; eat chocolate, ice cream, beef, cheese, or whatever else you find delicious. In reasonable amounts.

Inevitability of geoengineering

taw — Wed, 04 Nov 2009 19:49:00 -0800

I want to write a quick post showing some arguments about inevitability of large scale geoengineering which I find extremely convincing. I'm sure I won't convince that many people, most have made up their minds already.

First, look at list of countries by CO2 emissions. In 2006 global emissions were 28.43 billion tons per 6.55 billion people, or 4.34 ton/person. World population is still growing and is estimated to stabilize at levels around 9.5 billion for 2050-2100, which is the time in question.

Now let's see what would happen if everyone got to current European standards of living, and European CO2 emissions, which are 7.84 tons/person. This would result in CO2 emissions of 74.5 billion tons, or 2.6x times current levels. This number is already quite optimistic - it makes a huge assumption that all countries with higher CO2 emissions like US, Japan, Canada, Australia, Russia etc. scale them down quite considerably.

Now a brief discussion is in order on why numbers like that are relevant. First, I assume that there won't be a massive genocide, or nuclear warfare, or pandemic, or anything else that could drastically reduce population levels. The last time there was a significant reduction in population levels was 1340s. None of the world wars, or any other events even stopped population growth - in fact people tend to reproduce a lot more during times of crisis, than during times of peace and prosperity, so it would need to be the biggest war, pandemic, or other kind of failure in history to significantly reduce population levels. This assumption is probably the safest.

The second assumption is that the rest of the world will grow quite quickly. World GDP/capita is currently $9,774 PPP. EU GDP per capita is $30,513 PPP. If world economy grows at per capita rate of 2.8%, then by 2050 world average GDP will reach current EU levels. This coincidentally is almost exactly annual per capita growth rate of the world economy of the last 30 years. Unlike with population, where predictions are quite solid, here there are more reasons why growth could be significantly slower, or significantly faster than predicted. We just had a completely unexpected global recession for a fairly obvious example. And not so long ago we had massive completely unexpected boom in China and then India, so it can go both unexpectedly badly and unexpectedly well. Still, 2.8% is a pretty decent first estimate.

I want to write a bit more on the subject of economy. Many people are sceptical about the idea that the "poor" countries will ever attain levels of economic development seen in rich countries. But the rich-poor divide is illusory. As you can see yourself with Gapminder, the divide is long gone and most countries are in the middle. World average life expectancy today is higher than in most richest countries in 1950s years ago. World average life expectancy in all but one countries is higher than world average 100 years ago - all that in spite of wars, malnutrition, malaria, HIV, and lack of clean water in many of them! Life in countries considered by many to be "permanently poor" is vastly better than it was in Europe of your grandfathers, and rapidly improving. To think it will somehow suddenly stop is to ignore history.

Development can happen extremely rapidly - world's largest CO2 emitter has been China for a few years now, per capita emissions of which increased by factor of x2.2 between 1990-2006. Likewise for India, growth was x1.6 during that time. Even for big countries not typically associated with rapid growth you have high growth - x4 for Vietnam, x2.4 for Thailand, x2.3 for Malaysia, x1.9 for Indonesia, x1.5 for Pakistan, x1.4 for Brazil and Nigeria and so on (and these are all per capita rates, their total emissions increased far more). Developing countries are catching up fast.

Let's look at past performance of world CO2 emissions. Between 1992-2007, world's emissions increased by 38% - mostly driven by rapidly developing countries like China and India. Even developed countries weren't terribly successful - US and Japan had significant increases in emissions, and EU's stayed pretty much even, and nominally decreased mostly due to early-90s' collapse of heavy industry in former communist member countries. Even if developed countries got their act together and implemented reductions that were agreed on in Kyoto Protocol, they would still be swamped by just Chinese increases, resulting in net emission increases.

And the outlook for the next couple of decades doesn't look terribly promising. Intrade markets, thin as they are (and forgive me lack of links, but their website is virtually unlinkable), think it's extremely unlikely that China and India will agree to any CO2 limits anytime soon, and have pretty low opinion on what reductions developed countries will agree to - chance of agreeing to 10% reduction by 2025 seems to be about even, and there's quite a big difference between agreement and implementation (especially due to possibility of accounting shenanigans of land use changes and such), and such reductions would be swamped by developing countries increase. And in any case the 74.5 billion tons number already assumes that developed countries average will get reduced to current EU levels.

If you hope on peak oil to make 74.5 billion tons/year kind of number impossible, not only peak oil doesn't seem to be happening, coal production grows insanely fast, and estimates of "peak coal" talk about mid 22nd century.

Alternatives to 74.5 billion tons a year

So with business-as-usual, we'll be emitting enough CO2 to get the entire planet stir-fried. What would be alternatives?

First, either outright genocide or indirect genocide by limiting economic growth of developing nations - remember that poverty kills hundreds of millions of innocent people and causes vast suffering. Not only is it unlikely to happen, it would be probably the worst scenario of all.

Second, learn to live in a warmer world. This isn't necessarily going to be as bad. It's hard to imagine anyone who would like to have their children die of malaria and diarrhea just to satisfy some rich smug liberals. People who measure costs of limiting CO2 emissions usually focus on limits in States, or EU - but this is nonsense! Big increases don't come from SUVs, but from bringing poor people to more civilized standards of living, and limiting that would be disastrous. Costs of just taking global warming would be much less than the genocide that would ensue to keep poor people in their place. I think this scenario isn't entirely implausible - there were quite a few cases in history where international cooperation completely collapsed, and in externalities cases like CO2 emissions, this is a possible result.

The third solution, and we're getting somewhere now, would be to sever the link between GDP and CO2 emissions. This is surprisingly hard. Yes, it's trivial to move from Hummer to Prius, but the problem is people who cannot afford either yet, but will want some means of transportation eventually. And they will want electricity - coming from coal most likely. The alternatives look pretty bleak - nuclear power, and large-scale hydroelectricity, both provide only 3% of total primary energy use now (or about 15% of electricity each). All fancy types of renewable energy together don't even add up to 1%. A surprisingly large amount of energy is "traditional biomass", or wood and agricultural waste used for heating and cooking (but then, estimates of it seem to vary wildly). Other than that, right now we're entirely dependent on fossil fuels.

A popular idea of high standards of living without corresponding increase in energy use seems pretty unlikely to me. Yes, we might be able to break the correspondence at some point, but energy use per capita in poor countries is so low, it's simply bound to drastically increase.

Much more promising path is energy generation without fossil fuels. Our best chance so far was with nuclear power. Unfortunately we suddenly stopped what looked like extremely rapid exponential growth in late 1980s after Chernobyl disaster - estimated 4000 people will die due to increased radiation exposure (much higher estimates based on linear no-threshold model can be safely ignored, as LNT model is known to be empirically completely wrong).
By ridiculously naive estimations, if this growth continued exponentially as it seemed to have up to Chernobyl, nuclear energy would provide around 20% of total energy production today, or half of all oil could be eliminated. Now this kind of extrapolation is ridiculously naive, but you should give it some thought.

So could we switch to nuclear at ridiculously rapid pace? We would need over 50 times as much nuclear capacity to generate enough energy to cover increased energy use by 2050, what seems ridiculously optimistic by any standards, not to mention nuclear proliferation issues. It looks somewhat less ridiculous when expressed as 10% a year growth (or doubling every 7 years like pre-Chernobyl), but it's just not going to happen.
There aren't that many renewable options. Current global energy consumption is 15 TW, and it will be about 39TW by 2050 by extrapolation. The only large scale renewable energy source we use, hydroelectricity, is nowhere close to our needs, and neither is geothermal really, if we look at what small fraction of it we can practically extract thanks to high local concentrations. We're left with just two then - wind, and solar. We would need to use 5% of all wind energy of Earth to fill the energy gap.

In 2008 wind power had 121GW of installed peak "capacity", and produced 260 TWh of energy (what corresponds to 30GW of actual production). It would require order of 1000x increase in wind power, or 19% a year growth, to make it fill the energy gap. And the actual growth is actually fast enough at 29% in 2008. But there are clear diminishing returns here - best locations will be taken, and massive subsidies cannot really scale to TW levels.

The other alternative is solar power. Just Earth-based solar is so plentiful that using even 0.02% of solar power now, or 0.045% of it in 2050, we can replace all of world's energy use. And it doesn't stop there, with space-based solar we can get numbers many many orders of magnitude larger. It would sort of need a space elevator, but we might get there by 2200.

Anyway, while solar is undoubtedly our long term future, it doesn't look that bright in the short term. There's only 15GW of solar thermal installed, and mere 0.6GW of solar thermal. And these are capacity numbers. We're talking x10,000 increase to cover the energy gap, or 25% a year increase, or x200,000, 35% per year for solar thermal. Now these numbers are actually what we see now, but as I said for wind, these are all based on massive subsidies, and there's no evidence that they're scalable to TW range at all.

By the way all this is based on assumptions that all energy is basically equal, and all greenhouse gas emissions is energy production. Neither of these are true - electricity is relatively easy to replace with renewables, at least if we ignore their intermittent character, but transportation much less so. And even with 100% solar electricity-based economy, there would still be some significant sources of greenhouse gases like methane from agriculture - so whatever we estimate here is bound to be overly optimistic.

With all this said, thanks to better energy efficiency, and widespread use of renewables, it's quite likely that humanity energy use by 2050 will be much lower than baseline number of 74.5 billion tons a year. On the other hand expecting it to be vastly lower than current emissions is just extremely unlikely.

And here comes the final solution - without genocides, without unexpected revolution in either nuclear power or renewables, we will have much more CO2 in atmosphere in 2050 than now. And due to massive lag it has, it will keep warming Earth for a very long time after we finally get our emissions to decrease. So if we cannot agree to live in warmer world, we have to geo-engineer to survive the next century without massive and irreversible environmental change.

Nature did some real life large scale geoengineering testing for us, and it works without major side effects (you probably haven't even noticed the experiment in 1991). Economic estimations put it in ridiculously cheap range - some schemes are estimated to cost just 375 million dollar a year, or just 17,000 Toyota Priuses (seems like a highly relevant unit of measurement) - for comparison replacing every American car with a Prius would cost 5 trillion USD, and it would still do very little towards limiting global CO2 emissions. And stratospheric sulphur isn't the only solution, just the one we have best data on (by the way it's not about polluting more, only about moving less than 1% of existing pollution to higher layers of atmosphere).

So the alternatives are:

Outright genocide
Indirect genocide by forced poverty
Accepting the warming
Increase in nuclear or renewable energy far beyond anything that can be realistically expected in this timeframe
Geoengineering

Guess which one is the least bad?

Now I just need to blog on abortion, and I win blogging controversy bingo.

Why Malthusianism refuses to die

taw — Sun, 01 Nov 2009 07:22:00 -0800

It's astonishing how popular Malthusian catastrophism is, in spite of such spectacular evidence to the contrary. As disaster keeps failing to come, its details are changed, but the core prediction that we're all doomed doesn't budge. In this post I'd like to explore some evidence against Malthusian ideology - as with so much evidence to the contrary, it like creationism long ceased to be anything else than ideological view.

Classical Malthusianism

But first, let's say what I'm criticizing. Malthus' idea was that population growth is exponential, while food production growth is arithmetic, therefore we're all going to fall back into extreme poverty due to overpopulation, until we're poor enough to be dying in sufficient numbers to counteract that.

The second part is, and I'm not making this up - "therefore, we should starve and abuse people who lost their jobs in a recession to lower taxes on the rich". That's a forerunner of the Real Business Cycle theory if there ever was one. The predictable result of being tough on the poor was that the working poor were starved to the point of eating pig fertilizer, beaten, and sexually abused.

But that's not the worst of Malthus-inspired politicians - they decided to starve one million poor people in Ireland. During the alleged "famine", Ireland was a major exporter of food, with its food production even at lowest point estimated to be enough to sustain population twice as high as it had. And there was well known solution to such famines already tried in 1782–83 of temporarily stopping food exports, but those good Malthusians would rather have a bit of a genocide than to stand in ways of profits.

Anyway, I'm not here to criticize consequences of Malthus' ideology, just its factual basis. It's been 211 years since Malthus' infamous essay, how far behind are we in food production?

Oh wait, actually food production looks pretty exponential on this graph, while population growth looks rather linear. It's not even that - according to virtually all projections population growth is slower than linear, and world populations are due to stabilize fairly soon, here's Wikipedia:
Unfortunately I haven't found any useful food production statistics going all the way back to 18th century, but by now it should be very clear that these relationships do not hold.

Yet, Malthusianism refuses to die. The most popular mid 20th century's neo-Malthusian Paul Ehrlich famously wrote in 1968:

The battle to feed all of humanity is over. In the 1970s and 1980s hundreds of millions of people will starve to death in spite of any crash programs embarked upon now. At this late date nothing can prevent a substantial increase in the world death rate...

Or more specifically - "I have yet to meet anyone familiar with the situation who thinks India will be self-sufficient in food by 1971" and "India couldn't possibly feed two hundred million more people by 1980."

That was in the middle of yield increases so spectacular they caused major difficulties to even harvest it all (High yields led to a shortage of various utilities — labor to harvest the crops, bullock carts to haul it to the threshing floor, jute bags, trucks, rail cars, and grain storage facilities. Some local governments were forced to close school buildings temporarily to use them for grain storage. as Wikipedia says). India became net exporter of cereals by 1978.
You'd think that after two centuries of predictive failures people would have given up, but no. The latest neo-Malthusian fad just a few years ago was "peak oil". It sort of fizzled since, but in case you forgot, here's an awesome YouTube video describing peakoilnik ideology while dancing:

"There won't be much food anyway, as oil is used to make fertilizers and pesticides. Up to two thirds of the world population may not make it. Four billion people may not survive."

Because as they say, we're basically "eating oil" as the meme says. It doesn't bother them much that oil is used at no step of fertilizer production - it's all natural gas, and entire 3-5% of world natural gas (1-2% of world energy use) is spent on fertilizer production (not even as energy source, more as convenient hydrogen source easier than electrolysis of water), but alternatives based on coal, and hydroelectricity are well known and used depending on local availability. China already produces most of its fertilizer with coal. The first commercial hydroelectricity-based fertilizer plant was opened in 1905, over a century ago, and in case you're wondering use of electric plowing equipment dates to late 19th century.

Pesticide production is so ridiculously low, by some back of an envelope estimates takes about 0.06% of world's oil, and is trivially replaceable by coal if need be (I know of no serious estimates, as it's too small to bother anyone with a clue).

What a Malthusian world would look like

So why so many people believe in Malthusian-inspired catastrophism? I have a two-fold hypothesis. First, some people just enjoy beliefs that we're all going to die. It seems that virtually every disaster scenario, no matter how plausible or not - attracts countless believers.

But there are people smarter than that who still believe in Malthusianism. Here's the second part of my hypothesis. When people look at history, to see if it matches some theory, they just look at things that would be observable if theory was true, and see if there are present. They don't do proper Bayesian updating.

Here's a simple example - let's say you have a theory which says that clash of civilization causes warfare. Then the naive thing to do would be to look at history, note a few wars between civilizations, and take it as evidence for the theory. It would also be wrong, as far more numerous wars were fought within civilizations, and proper Bayesian thinking must look at both.

So let's take a look at evidence for Malthusian worldview against the null hypothesis that food production is just like any other commodity, and not any more connected to population than production of clothes, bricks, or paper.

If Malthusianism was true, food production per capita would be pretty much constant - because any increase in food production would result in increase in population, and decrease in food production would cause mass starvation, and adjust population accordingly. But if Malthusianism was false, food production per capita would also be pretty much constant, because what would be the point of growing more food if there was nobody to eat it? So there's no evidence either way.

If Malthusianism was true, everyone would be poor. Not only is this ridiculously false, now, historical estimates of life expectancies vary wildly from 18 to 60 for the same time period.

Malthusianism predicts almost all food production would be in whatever generates highest number of calories per hectare. Non-Malthusianism predicts food production would be much more varied - and it is so. Just the fact that we're keeping farm animals, which by their nature only destroy and not produce nutrients, proves Malthusianism wrong. There might be exceptions, but most grazing lands would produce more calories when cultivated, not to mention Malthusian ridiculousness of feeding animals human food as we do.

If Malthusianism was true, any sudden decrease in population, due to wars, famines and such, would result in rapid rebound, as agricultural capacity existed to support much larger populations, proven by its previous existence. If Malthusianism was false, population would have no particular reason to rebound. Here the evidence is very clear - there are numerous example of major population loses which were not recovered for centuries. Two best examples affect entire Europe, which used to have high populations at height of Roman empire, then fell to very low levels and didn't recover until 12-13th centuries, almost a millennium later. Second fall was due to Black Death, recovery from which wasn't complete even two centuries later.

Malthusianism predicts that it would be extremely unlikely for vast amount of spare arable land to exist - yet this was exactly the situation for centuries in early Medieval Europe, with vast tracks of unused lands.

Malthusianism predicts that famines would happen during times of higher population a lot, while almost never happening during time of low population. But even a brief look at Wikipedia's List of famines shows plenty of major famines both during population lows of early middle ages (5th century famines in Western Europe, 750's famine in Spain, 809 famine in Frankish Empire, 963-964 in Ireland, 1005 in England, 1016 all Europe, 1030-1032 France, 1066 England again, 1097 France again). There were supposedly 95 famines on British Isles alone during Middle Ages. In spite of very low population levels, we can see plenty of them, and we'd probably know about more if it wasn't for horrible Medieval record keeping. Notice that during all that time there was plenty of undeveloped arable land!

During second period of low populations after Black Death, there are major famines 1390 in England, 1481-1483 in France, 1504 in Spain, 1518 in Venice and so on. I'm not even going to mention that modern periods of European population explosion had fewer famines than any other time in history.

Non-Malthusian view correctly predicts that famines and population levels are completely uncorrelated. Famines are caused not by the mythical "overpopulation", but commonly by wars, government action (like in Ireland, and 1930s' Ukraine, notice how democratic governments have virtually spotless track record of no famines, in spite of plenty of agricultural disasters), and most of all - simply by major delay between increase in agricultural inputs and outputs. It doesn't matter that there's plenty of unused land around, developing it would take years, and plenty of labour, capital, seed, and farm animal input, before any food was grown on it - all of which being in short supply during famine anyway. By the time food was growing the famine would be long over. To give you some indications of time scales involved, settlers in new developments were freed from any taxation for up to 24 years. During that time net effect on food production would be negative.

Most of all, current overwhelming explosion of wealth is completely and utterly incompatible with Malthusianism. Even the second worst country in the world - Angola - has life expectancies far greater than world had 100 years ago (one tiny country which doesn't has 26.1% HIV infection rates, and is too poor to afford HIV drugs - hardly a Malthusian reason). In spite, or maybe even due to, population explonion, world's average life expectancy is 66.57 now. This is level not reached by even the richest countries before 1930, and by most developed countries well into 1940s an 1950s.

However hard I try, I cannot find any evidence for even very broadly constructed Malthusianism that cannot be explained as well or better by treating food as a normal commodity. So worry not - we're not doomed.

Widespread capital punishment never existed

taw — Sun, 01 Nov 2009 00:13:00 -0700

There's this fancy Western European idea of abolishing death penalty. Wikipedia even has a nice map, with blue countries having abolished death penalty entirely, red still carrying it out, and green/orange having it on the books, but not in practice.
But reality is different than that. Most of the red countries almost never carry out the death penalty, reserving it for extremely unusual cases - in 2008 estimated 5727 people were executed, over 80% of them in mainland China, other than that only Iran and Saudi Arabia doing three-digit numbers. Other big red countries like India, USA, Indonesia, Pakistan, Japan and so on very rarely execute people.

As 59 million people die every year, your chance of getting executed is like 1:10,000, or more like 1:200,000 if you don't like in relatively execution-happy China, Iran, or Saudi Arabia.

Numbers like this are ridiculously uncommon - about 100 people die every year from peanut allergy in just United States. And "accidental suffocation and strangulation in bed", which sounds like a codeword for "BDSM gone awry" causes 327 deaths annually in just USA.

It's not just due to lack of crime - annually, half million homicides take place, and many countries have or had death penalty for lesser things. So if you're a killer, the chance of getting a lethal injection for it are 80:1 (assuming they get the right person), or more like 1500:1 unless you live in one of the three aforementioned countries.

Add to that the usual overconfidence criminals have, and it's clear that fear of death penalty is the least of criminals' concerns. Especially since even in United States you're over three times as likely to die as a result of a shootout with the police than due to legal execution, so we're talking at most 30% extra deaths factor here. The idea that people will stop carrying out crime due to fear of capital punishment is just highly dubious.

History lesson

So is this it? Did the abolitionists win? Not necessarily. Googling indicates that capital punishment for crime was never particularly widespread. Yes, governments killed loads of people - massacres during wars and occupations, killing political opponents, summary executions to quell social unrest, and some such. But legal capital punishment of citizen committing ordinary crimes after proper judicial process? Very unlikely.

Here's some statistics. Stalin killed about 60 million people, and Hitler about 20 million (estimates vary considerably, but let's take those numbers). How many actually received death penalties? Only 40 thousand from Nazis (1 in 500), and 750 thousands from Soviets (1 in 80). And these were mostly political opponents, war and civil war cases and such - not proper criminals.

How about other countries? Czarist Russia averaged less than 60 a year, and remember that many of them were political cases. Between 1770 and 1830 England had allegedly bloodiest criminal code on Earth, with death punishment for "writing a threatening letter", hanging out with gypsies, or "strong evidence of malice" and over 200 other crimes, while still averaging barely above 110 executions annually. Even the infamous Spanish Inquisition barely managed 20 burnings a year (but then, these were mostly religious and political opponents not criminals, so maybe they shouldn't be included here).

It's just really difficult to find any government, that routinely carried out large number of legal executions of criminals during time of peace.

When governments kill

That's not to say governments don't kill. But these killings rarely affect common criminals. Nazis killed people for being of wrong ethnic group. Soviets killed people for being of wrong social class, or just because. The same 18th century United Kingdom which was so cautious with criminals of right ethnicity, killed million people for being poor and Irish (and no, it wasn't any Malthusian bullshit, Ireland was a major food exporter even during this politically caused famine).

Even countries which don't execute criminals at all like Denmark with no executions since 1892, change their mind during and briefly after war time, when 54 people were executed (it's not really a legal capital punishment, as it's all ex post facto). Likewise in Finland, after not executing anyone since 1825, about 550 people were killed during war time (unlike Denmark where it was mostly proper war criminals and people supporting Nazis, here you could get a firing squad for conscientious objection, and without even a real trial).

What does it mean for capital punishment debate?

Other than perhaps in mainland China, Iran, and Saudi Arabia, the debate is purely speculative now. There is virtually no capital punishment anywhere. And other than during wars, occupations, major civil unrests and alikes, it was extremely rare for many centuries. Before that, executions might or might not have been more frequent, but it's hard to talk about proper legal proceedings, so they don't count.

People who support death penalty have a big problem, as there won't be any statistics showing serious effects of widespread application of it during time of peace, as it never happened. There are some bullshit statistics, like those in the original Freakonomics book (very entertaining book, by the way), but due to vast non-random annual variations in number of homicides and executions, we're talking about effects of a rounding error, and social science just cannot do anything useful with them.

So it's mostly about what makes you feel better. If you're a real Christian, then you should follow Jesus and be against death penalty. If you're a fake Christian like Southern Baptist, you should support death penalty because you hate black people more than you love Jesus. And if you're an atheist, do whatever you want, it's not like I can convince you.

Here, I solved it for you.

Medieval 2 Total War Concentrated Vanilla 0.07

taw — Tue, 27 Oct 2009 20:42:00 -0700

Here's another release of my Medieval 2 Total War Concentrated Vanilla mod.

Old releases suffered from the too-much-money problem, just like the pre-crash Goldman Sachs, so I decided to scale it down a bit. Instead I introduced massive free garrisons.

Here's the full list of changes relative to vanilla:

For more dynamic game, all units move 75% faster on strategic map.
One big problem with vanilla was that money didn't matter. No matter if you had a lot or not, there wasn't all that much to spend it on. So now all buildings take 1 turn to build, but are 50% more expensive. This drastically increases importance of money.
To add some strategic richness, mines give 2x the profits, and cost 3x as much (x2, +50% like all other buildings). Some settlements with rich mines like Zagreb or Vienna are worth a lot, maybe even worth starting a war over.
There used to be 50% increase in settlement and merchant trade, and mine profits used to be 3x, but this was a serious overkill. If you manage well, settlements and merchants will give you more money than in vanilla as you can build relevant buildings faster.
There also no longer double population growth penalty/bonus for tax rates - it was an overkill.
To help small nations, King's Purse increased 2x (that is - free money a nation gets unrelated to its settlements - from 1500-2500/turn to 3000-5000/turn).
To reduce annoyance with rebels, rebel and pirate spawn rates reduced 10x. (I'm not actually sure if it reduced them as much as the file says it would, but it feels more or less right now).
Cavalry, especially heavy cavalry was seriously overpowered. Small groups of knights could easily crush infantry armies ten times their budget, so infantry was useless other than free militias to patrol cities, and cheap grunts to operate siege equipment. Because with cavalry-centric warfare missile units are useless, this limited army composition even further - and destroyed rich tactical variety that M2TW could possibly have. So all cavalry is now 50% more expensive to recruit and upkeep. Cavalry is still extremely powerful when used right, except now other kinds of armies are useful too.
Bodyguards were particularly overpowered - in early periods they simply destroyed everything, were numerous, self-healing, essentially free (you get new candidates for adoption if you lose old generals all the time), and as every faction has identical bodyguards they severely reduced tactical richness of the game. So bodyguards units are the half size now (upkeep cost is proportional to unit size, and they're cavalry, so mod's bodyguard unit costs 75% of vanilla's). Old versions of the mod reduced their hitpoints from 2 to 1, but it no longer happens, as this made bodyguards overnerfed in late periods. Unfortunately there no way to get the balance right - as bodyguards stay the same, but other units massively improve, either bodyguards will be too strong early on, or too weak late on. I moved the balance point somewhat towards the weakening.
Vanilla sieges are ridiculously easy, with a single unit of ballistae being able to take down large stone walls. This made sieges was too easy, and as a consequence turned the game into a siegefest. So now gates and walls (but not towers) are 5x stronger. Serious artillery will be needed to take anything but the weakest walls.
Towers now have 2x fire rate for normal ammo, and 1x fire rate for flaming ammo. Increasing flaming ammo rate too much as old versions did made siege equipment like siege towers almost useless. But high normal fire rate should help defenders a lot, even for undermanned settlements that AI leaves. This wouldn't be necessary if AI was competent, but we know well it's not, and leaves valuable settlements defended by one unit of Peasants far too often. This balances AI stupidity considerably, and also frees the player to play more strategically by making defense easier (this is not as essential as in old versions of the mod due to increased free garrisons).
Tower activation range is 8x larger (a bit of reversal to Rome Total War). So you cannot shut down towers just by scaring off defenders with some arrows like in vanilla. Strong walls, and massive towers' firepower together mean sieges will be very difficult. Prepare a lot of artillery, or spies, or massively overwhelming force that can take severe loses before giving up. I'm yet to take a well-manned Fortress in this mod (but then AI often mans its settlements insufficiently, and there are other options like waiting their food supplies out, or provoking a sally).
For consistency with making sieges harder, spies cost 2x to recruit and upkeep. And AI is quite competent at counter-espionage operations, so spy flood doesn't work all that often.
To help defenders in sieges, missile infantry ammo increased 2x. It very rarely affects field battles, as in those they rarely get to expend even half of their ammo.
A big change is doubling free garrison in cities. Village 0, Town 4, Large Town 6, Small City 8, Large City 10, Huge City 12.
The biggest change is introducing free garrison in castles. Motte&Bailey 1, Wooden Castle 2, Castle 4, Fortress 7, Citadel 12. In vanilla castles are either left barely defended, or are massive bleeding money hole - failure both ways.
Just like with cities, castles' basic infantry units can be free garrison, but not cavalry or expensive infantry. Basic infantry is one with morale up to 5, and cost up to 155. So Peasants up to Armored Sergeants, and Peasant Archers up to Genoese Crossbowmen, Venetian Archers, and Hand Gunners can all be free when garrisoned. On the other hand Dismounted Knights of all kinds, and most faction-specific elite units cannot. I guess it might introduce some faction unfairness, but I can live with it.

Overall, what you get is much tactically and strategically richer game, with a lot larger differences between factions, but one that still feels a lot like vanilla Medieval 2 Total War. In the mod defense is easy, so you don't have to mindlessly attack everyone.

The main problem left is the spectacularly stupid AI, about which I cannot really do that much.

I also did some other experiments, that didn't get to the mod:

Making ocean-traveling ships available from the beginning (well, as soon as you can build sufficient port infrastructure) enables fights with natives. But Americas are disconnected from the mainland Europe, and it doesn't really feel like it's part of the same game. Try it once to see what fighting Aztecs is like, then forget they're even part of the game.
Making Mongols (or maybe even Timurids) come earlier. It's a lot more fun to fight Mongols, but then if they come very early, they will totally crush AI on auto-resolve. Unfortunately auto-resolve favours stronger armies too much, so they won't even suffer too much loses, and you'll have to deal with them yourself. One full stack of Pavise Crossbowmen and Feudal Knights led by semi-decent General (or equivalent) can defeat one full Mongol stack without that much trouble in field, and well defended settlement with ballista towers completely destroys them. It's another thing to try once. I haven't fought Timurids yet.
Making Gunpowder available immediately unfortunately counters the effects of very strong walls, and yet good gunpowder infantry and cavalry won't be available until much later. I'm not even sure if gunpowder units are better than plain Pavise Crossbowmen, they don't seem to be from the stats, and I never had an opportunity to play much with them.

Anyway, download the mod here.

Blog posts about previous releases: 0.02, 0.04, 0.06.

Folk Complexity Theory

taw — Wed, 14 Oct 2009 19:03:00 -0700

There's one kind of posts on this blog that get the most hostile reactions. It's not politics (nobody reads those), flaming programming languages, or computer games - the most controversial subject is Big O analysis! Here are relevant posts:

Basically I claim that Big O is useless in most practical situations. And I get flamed a lot. At first I tried to explain to the flamers how they are mistaken, then I basically ignored them after explanations proven to be pointless, but now I see that it's all just big misunderstanding. What I call Big O analysis is a fairly precise construct of computer science. But that's not how most people use the Big O notation - they follow Folk Complexity Theory, which I want to describe here.

The Official Complexity Theory

First, here's a brief repetition of Algorithms and Data Structures 201, or whatever your course explaining this was.

An algorithm is O(f(n)) if in a particular model of computations, if there exist some constants n₀, and c, such that for all input lengths greater than n, computation time is less than c × f(n). A problem is O(f(n)) if in particular model of computations there exists a O(f(n)) algorithm that solves it.

Models of computations are things like different varieties of Random access machine, and different models usually only differ by some logarithmic factors. Sometimes you want to assume that adding two numbers takes constant time, sometimes that it takes time proportional to logarithm of their sum, and details like that. If you disregard logarithmic factors, you don't have to be all that precise about which variety of RAM you're using. That's about it.

In particular, if the problem is only defined for constant-sized input, then by definition it has complexity O(1). There exists some c, such that for all possible sizes, running time of the algorithm is less that c × 1.

So breaking AES-128 is O(1). As is breaking RSA-4096. Breaking RSA-n for arbitrary n isn't, but that's entirely unrelated problem. Breaking AES-n is meaningless concept, as AES isn't defined for arbitrary n.

In particular classes like P, and NP, don't depend much on what reasonable computation model you're using. They're defined on Deterministic Turing Machines for reference (which is polynomial factor slower than RAM), but if something is P on DTM, it's P on RAM and vice versa.

How cryptography handles it

As I got a lot of flames about my claim that breaking RSA-4096 is O(1), let me explain how cryptography handles it. So cryptography doesn't use Big O notation much.

Security is measured in bits - a cryptosystem has k bits of security if the attacker must perform 2^k times as many operations as legitimate user to defeat security of the cipher. For some types of security key size and security are related - for block ciphers security = key size, for hash collisions security = key size/2. For public key cryptography these bit measures can be only approximately specified in terms of key size, and whatever formula we have we're completely uninterested in its asymptotic form, as all feasible key sizes are small, and constant factors are of great interest to us.

The Folk Complexity Theory

And now the main subject of the post - The Folk Complexity Theory. When people say some algorithm is O(f(n)), they actually mean that for "normal" sizes of inputs, on actual computers, it's expected running time is between two "small" constants times f(n) times reasonable number of log(n) factors as long as they don't push the constant too high.

O is used sort of like Θ of the official theory. Folk O(n) algorithm is not considered to be O(n²).
So bubble sort is O(n²), as number of operation tends to be something like 10n². Logarithmic factor for addressing and arithmetic is not considered at all.
Merge sort is O(n×log(n)), as number of operation tends to be something like 10n×log(n). Notice how log factor for number of operations is included here as it's significantly higher than allowed small constants; but log factor for cost of addressing and arithmetic is not, as on normal computers it's constant up to 2³², which is very big n. And only doubles for 2⁶⁴, which is too ridiculously high n to bother. Exactly the same reasoning can be used for both log factors, so this is highly peculiar.
Quick sort is O(n×log(n)), the Folk Complexity Theory always deals with typical case, not worst case.
Breaking AES-128 is O(2ⁿ) because the constant is too big to be allowed. It doesn't matter that there's no n here, and AES is not defined for other key sizes. The problem must be reformulated to have n, no matter if it makes any sense or not. Number of operation like 2¹⁴⁰ is simply "not allowed", while 2¹²×2ⁿ is.
Breaking RSA-4096 is likewise O(2ⁿ), even though there's no n here at all.
Forging signature of n-byte message signed with RSA-4096 is, uhm... O(2ⁿ). Wait, n is already something else... O(2^k) then maybe? The problem is reformulated as RSA-k because large constants are not allowed, and usually there's no point saying O(2^k+n), as n is too small to matter for all practical n.
As an added bonus "NP" usually means Θ(2ⁿ), for real or imagined n. This is spectacularly incorrect (most problems are not decision problems; there can be no Θ bounds even for NP-complete problems until NP != P is proven, and every P problem is NP), but doesn't cause much confusion in practice.

This is the source of all confusion - I was talking about the Official Complexity Theory, while most Redditors interpreted it as Folk Complexity Theory statements, in which they were obviously untrue.

Is Folk Complexity Theory useful?

It's quite amusing that so many people flame me for saying the Official Complexity Theory is overrated, and don't even know it. But I'm not going to blame them - who needs it once they're past exams anyway? In practice Folk Complexity Theory is actually better heuristic than the Official kind. We program real computers. On real input sizes. ceil(log₂(n)) factors shouldn't be ignored, but ceil(log_4294967296(n)) factors which represent cost of arithmetic and addressing on most 32-bit computers hardware totally can and should. And so should all log(log(n)), which nobody ever bother with past exams.

The Folk Complexity Theory has weird status - it's neither mathematics like the Official theory, nor practice like actual benchmarking. It's just a bunch of heuristics dressed as something much more formal than they are. Often even fairly useful heuristics. Just nothing more than that, and not worth the flames.

The Web 2.0 of Metaphors

taw — Wed, 14 Oct 2009 18:30:00 -0700

After hearing that something is Wikipedia of whatever it is about, and getting slightly annoyed by it, I decided, in interest of science, to compile popularity ranking of "the X of Y" metaphors, by Google counting.

I found quite a few false positives, like "the livejournal of Nikki", and "the Linux of your choice", but upholding the glorious scientific tradition of ignoring every methodological issue that cannot be fixed, I completely disregarded them.

"the windows of" - 15,100,000
"the internet of" - 7,790,000
"the apple of" - 6,470,000
"the google of" - 3,160,000
"the youtube of" - 766,000
"the microsoft of" - 397,000
"the imdb of" - 307,000
"the craigslist of" - 303,000
"the livejournal of" - 290,000
"the ebay of" - 277,000
"the amazon of" - 262,000
"the facebook of" - 237,000
"the myspace of" - 235,000
"the linux of" - 197,000
"the wikipedia of" - 180,000
"the yahoo of" - 142,000
"the flickr of" - 124,000
"the ibm of" - 109,000
"the blogspot of" - 108,000
"the web 2.0 of" - 92,900
"the bing of" - 90,900
"the firefox of" - 63,200
"the aol of" - 63,100
"the napster of" - 61,200
"the msn of" - 52,100
"the digg of" - 23,600

It seems the most popular activity is explaining one web service as metaphor of another:

Is Craigslist the "Napster" of the Sex Industry
YouTube Redesign: Becoming the Google of Video?
Is Facebook: the Google of Social Networks
Google Earth Is the AOL of the Geoweb
MySpace is the AOL of Social Media/Web 2.0
YouTube.com — the Flickr of Video

There are also some unusual finds:

The Wikipedia of Slutty Dressing
The Craigslist of Antibiotic Resistance
Is Python the Apple of Programming Languages?

Another interesting find is that while I've been expecting a lot of "the X of sex", only Wikipedia (36,200) and Facebook (29,500) have decent number of matches.

"The X of porn" is a lot more popular, here Google (378,000), YouTube (354,000), and surprisingly Napster (22,200) are taking the lead.

The other most popular Internet activity, spam, got almost no hits.

Efficient go hypothesis

taw — Wed, 14 Oct 2009 10:00:00 -0700

Efficient market hypothesis states that you cannot really beat the market by much, all the prices are about right, and everybody behaves it more or less rational way. Yes, treating it too seriously leads to ketchup economics thinking, but it's often a good starting point - if you postulate that some prices are seriously wrong, you better come up with a story why they're wrong.

But let's forget about economics for a moment. Something very similar to efficient-market hypothesis applies to the game of go. Unlike most other games like chess and football, go is extremely balanced.

Black player, who moves first, pays 6.5 points for the privilege, what makes both players' chances of winning as close to 50:50 as it gets.
If one player is stronger, the weaker player gets as many free stones as necessary to get the chance of winning very close to 50:50.

What's interesting is that during the game, it's impossible to make a move that highly increases your advantage, or "beats the market" - if you could get significantly ahead by playing a stone, then the position was already unbalanced before you've done so due to this possibility. Values of positions consist of complex balance of territory, influence, flexibility, good and bad aji, and many other things - if you ask a computer program like gnugo to estimate balance of the board, you will get widely varying numbers with each moves, each totally ridiculous.

Just like on the market we cannot give correct formula for a price, but we know it's right because people trade at it; in go we don't have a formula for value of a position, but we know it's even if good moves by two players led there. If you postulate imbalance, you better come up with a story telling which move was bad and why.

Go players often perform some complex trades of moves called joseki - standard sequences that lead to more or less even results. Again, why even? Because neither you nor your opponent would purposefully play bad moves to fall behind, and if there are no good moves available it means one of you was already losing earlier.

Joseki example - every single move here and the sequence are about even

This requires significant shift in thinking compared to games where you act to get as far ahead of your opponent. You can play go like an index fund - just try to stay even with each move, and let your opponent make some mistakes and fall behind you.

That doesn't mean there's no room for creativity - sometimes there's only one good move to make, most of the time you have multiple good options with subtly different balances. These tiny variations are enough to win - there is no possibility of draw in go (not counting super-ko and such exceptional situations), so even if you get just 1 point ahead through the entire game, 0.01 point advantage per move, that's just enough. It's kind of like high frequency trades, which earn tiny profits, but with sufficient volume it's worth it.

Most professional games end with just a few points advantage for one side or the other. Or one of the players falls behind just a bit, and then risks it all on a great all-board fight to avoid the few point loss.

So as you can see, go is just like market. Except it never leads to crises costing trillions of dollars.

Practical O(n) problem impossible to solve even for n=1

taw — Sun, 04 Oct 2009 06:24:00 -0700

In computational complexity a fairly popular claim seen in most textbooks equates class P with practically feasible algorithms, and everything not known to be P (but usually within NP, "not known to be P" is an awkward thing we need to say until P!=NP is finally proven, even though we all know it to be true already) with not practically feasible algorithms. While it's often noted that it might be untrue for some cases, with some P algorithms being impractical, and some not known to be P algorithms to be practical, such disclaimers usually consider these cases to be contrived and not worth any further consideration.

Here's a really simple example of a genuinely useful problem in P (linear even) that's not practical at all.

Utterly impractical O(n) problem

Imagine problem of falsifying digital signature for a message in some reasonable digital signature scheme. Inputs are public key of constant length, and message of length n. The algorithm is as follows:

Extract private key from the public key by brute force, or some other algorithm (constant time)
Sign the message (linear time in n required just to read the message, and no more than linear time necessary)

So we have an O(n) algorithm, and because we need to read entire input to sign the message, no faster algorithm is possible. Therefore the problem is Θ(n).

Yet, unless someone breaks the public key system and provides vastly faster method of extracting private key based on the public key, we won't be able to solve it even for n=1.

This problem is of great practical importance - the entire online banking system would fail, and billions of dollars of fraud would happen if someone could practically solve it. So it's not fair to call utterly unfeasible P problems contrieved.

Google Everything and some alternatives

taw — Thu, 24 Sep 2009 06:04:00 -0700

Do you feel a bit uncomfortable using countless web services all operated by a single company? I can feel your pain. Here's a list of things I use (or tried but didn't like), and some alternatives if I know any.

Google Search

It used to be the case that Google Search was vastly better than anything else. Not longer so.
Don't believe me? Try the Blind Search - it searches the same term in Google, Yahoo, and Bing, then you can vote for the winner. Most of the time the results are extremely similar, and the scores are Google - 41%, Bing - 31%, Yahoo - 28%. Google still leads but not overwhelmingly.

While with text searches there's no reason to switch away from Google, Bing Images and especially Bing Video is quite significantly better than Google equivalents. Bing is simply a better search engine for porn than Google, they know it, and even advertise this fact:

I recommend we start using the word "to bing" as "to search Internet for video porn", just like "to google" means "to search Internet for textual websites".

Google Mail

I tried plenty of other webmail providers, and plenty of offline email programs. Even some Intranet email solutions like Zimbra and Exchange. Google Mail is so overwhelmingly better than anything else it's not even funny.

Hotmail, and Yahoo Mail, which I tried as throwaway spam accounts, are just too painful to use. There are no alternatives, it's Google or pain.

Google Maps

Google Maps was quite revolutionary when it first appeared. Unlike with Google Mail, here the competition did catch up quite successfully. Yahoo Maps, Bing Maps, MultiMap, StreetMap (well, that one's UK-only), and probably some others, are all quite usable for directions, and all suck for searching local businesses.

I've heard some drivers having preferences for one or the other, but as a public transport user/occasional cyclist, I'm still waiting for the first to offer decent TFL integration.

YouTube

YouTube annoys me considerably - they don't want to host anything even remotely risky. They limit access to videos by geography, remove those that include some copyrighted music in the background to unrelated funny thing.

On the other hand they have no problem hosting this video of a brutal rape (NSFW).

There are some alternatives, none anywhere near the size of YouTube. Not counting the porn tube of course. I have limited experience with them, so no in-depth comparison here.

Google Chrome

It's nowhere near being the most popular browser. I'm also not sure what's its main selling point, other than tab sandboxing, which Firefox badly needs.

Other than that, Firefox is vastly better. It has countless useful plugins. It has Greasemonkey, and Greasefire. It has Firebug. It has AdBlockPlus. Chrome isn't even close.

Not like I'm complaining, if it can get even one user away from IE, it's a win.

Blogger

Let's face it - Blogger is a pretty crappy blogging platform. But then I haven't heard users of any other being happy about theirs, so perhaps all are equally bad. And effort of moving the blog somewhere else (plus losing the URL) seems too big for me to bother.

In particular the way they handle spam comments is simply atrociously bad. I get a notification of a comment, and then it's about ten clicks to remove it if it's spam, instead of just one or two.

Google Reader

It's basically an online RSS reader. What I hate about most desktop RSS readers is that they tend to push notifications onto you, and don't live inside the most natural place possible - the browser.

With Google Reader Watcher, Firefox tells me how many unread items there are, and which website they're for. Which is exactly as much notification as I need.

I think the only thing left to want is "open every new item in a fresh tab" function, but there's probably some Greasemonkey script for that, or if not it would be pretty simple to write it.

I don't see much need for alternatives.

Google Talk

The main problem with IM programs is that everyone is on a different network, so you need way too many programs to talk with everyone. Adium has most of them. Except Skype. And the most basic functionality only - not webcams, no microphones, not even all the icons.

I don't have particular problems with Google Talk, but I'm going to use this unique opportunity to rant about Skype a bit. OSX version of Skype doesn't seem to have any way of disabling text notifications of contacts becoming available or unavailable. So it constantly spams me with irrelevant garbage. It's not possible to use Skype network from other programs even for IM chats. Many networks try this restriction - and abuse their control over access to other users, but Skype seems to be the only one which did it successfully. Would you be happy if you needed an iPhone to call a person using iPhone, a Nokia phone to call a person using a Nokia phone, and basically 20 different phones to speak with everyone you want to? Me neither, so why are we allowing exactly the same thing in the IM world?

Google News

I don't see Google News as solving any real problem. My main problem with news is that they focus on events of the day, while the important things are usually long ongoing processes, with little need for daily updates. And if you're away for a week or two, there's no easy way to look back at this time and see if anything important happened.

As far as I can tell, it's equally true for all alternatives. If you know a good way to follow important news, tell me about it in comments.

Google Docs

Google Docs is a pretty cool webapp for editing documents together. I have irrational hatred towards WYSIWYGs, especially web WYSIWYGs, so don't ask me about fancy advanced features, but it solves the basic need well enough, and I don't see what could be the big selling point for its alternatives.

Google Code

Google Code is a nice hosting service for Open Source projects. It supports Subversion (obviously), and Mercurial, but not git.

Some of the alternatives are SourceForge (both Subversion, and git, also some other VCSs I don't care about), github (only git, duh). I'm not sure how bad github's limitation of 300MB per open source repository is - my personal code repository's .git is easily bigger than that, but then I freely store binaries and stuff in it (like font files for jrpg).

Overall I don't really see any major advantage for Google Code over it's countless competitors, and it's not even particularly big.

Google Analytics

I'm going to be pretty clueless about it, as Google Analytics is the only one I tried.

Good things - it gives you pretty decent overview of traffic, and is really easy to install.

Bad things - no decent API (screen scrapping possible but quite painful), no access to raw data, I'm not a big fan of their interface, which seems to be focused on all the wrong things.

If I started a website these days, I would definitely try something else, if only to see if it's any better. Switching an existing site would probably not be worth the effort, as all historical data would get lost.

Summary

These are just some of Google services and programs I tried. It's scary in two ways. First, a single company has way too much control over the Internet, and over people's often private data. Google has been relatively well-behaved as far as huge soulless corporations go.

Second, and perhaps more important problem is - what the fuck is the entire competition doing? The only reason people use Google Absolutely Everything is because most of everything else sucks so much. Google didn't succeed just once or twice, it had long stream of spectacular successes where everyone else failed (search, maps), and often even failed to catch up years later (webmail, text ads). Seriously, how difficult is it to innovate a bit online? Difficult enough that a single company does most of it? Hardly anything good on the web ever came from Yahoo, Microsoft, Apple, Sun, and countless other huge Internet companies in ages. The only other major company that seems to be doing anything interesting is Amazon. A damn bookstore. A damn bookstore is doing more web innovation than all the software and web giants put together.

Yes, there are still startups doing the cool stuff, but most of them stop as soon as they get bought. Like del.icio.us, which was revolutionary, but then got bought by Yahoo, and is stagnant ever since.

In search for Just Wars

taw — Thu, 24 Sep 2009 03:51:00 -0700

I know people don't really read my political posts. If I wanted to be read, I'd be writing angry rants about video games, programming, and Harry Potter fan-fiction.

Here's a quick post about politics, namely about just wars, and other violence. Right wing nuts are generally pro-war, no matter who's fighting whom. What's surprising is that even left wing nuts are pro-war. Here's some Chomsky, it doesn't get much left-wing-nuttier than that:

ROBERT B. SILVERS: ... Under what conditions, if any, can violent action be said to be "legitimate"? ...

NOAM CHOMSKY: My general feeling is that this kind of question can't be answered in a meaningful way when it's abstracted from the context of particular historical concrete circumstances. Any rational person would agree that violence is not legitimate unless the consequences of such action are to eliminate a still greater evil. Now there are people of course who go much further and say that one must oppose violence in general, quite apart from any possible consequences. I think that such a person is asserting one of two things. Either he's saying that the resort to violence is illegitimate even if the consequences are to eliminate a greater evil; or he's saying that under no conceivable circumstances will the consequences ever be such as to eliminate a greater evil. The second of these is a factual assumption and it's almost certainly false. One can easily imagine and find circumstances in which violence does eliminate a greater evil. As to the first, it's a kind of irreducible moral judgment that one should not resort to violence even if it would eliminate a greater evil. And these judgments are very hard to argue. I can only say that to me it seems like an immoral judgment.

Oh, really? So, could you please point me to some examples of wars that successfully eliminated greater evils? The criteria are, you can agree that they are sane:

Without the war, very large number of people would have died or otherwise suffered.
There was no other plausible way of saving them, and all alternatives have been tried.
Thanks to the war, most of those people were saved.
Few people died or otherwise suffered in the war, especially civilians, but also participants. Some suffering is allowed, but only if it's vastly overwhelmed by suffering prevented.
It must have been possible to predict it advance that this success was very likely. Lucky wins don't count. And of course, it must have worked. Failed good intentions don't count.

I have really difficult time finding out wars that fit these criteria, even with very liberal grading.

The latest war in Iraq

The latest war in Iraq is easily not anywhere close to a just war. Saddam wasn't a nice guy, but amount of suffering he was causing recently was quite insignificant as for third world dictator standards, so it fails on count one, and automatically two and three.

According to Lancet estimates there were 1.3 million excess deaths due to it. Many other Iraqis suffered in other ways. So fail on count four.

Only fools would have expected an easy win, and being greeted with flowers, it's the Middle East after all. All claims about WMD, Saddam's connections to Al-Qaeda and so on don't count, as they were known lies, and even if Bush believed them, they would fail by count five.

Score: 0/5

The latest war in Afghanistan

This war is slightly better, as number of casualties on all sides is closer to 60 thousand, not counting another 10 thousands or so in the spill-off to Pakistan, what's most likely an underestimate in a poorly researched place like Afghanistan, but still much lower than Iraq's. And Bin Laden was in Afghanistan.

Still, it fails on all counts. There were people who died in Al-Qaeda terrorism, but other than a single well-known case, repetition of which was fairly easy to prevent, their numbers were really low. Even with 9/11 much lower than number of people who could be expected to die in the war, so count one fails.

There were plenty of plausible alternatives for getting or killing Bin Laden, and dealing with terrorism, so fail at criterion two. See IRA and ETA for good examples. Or even Clinton-style targeted killings.

Terrorism increased greatly after the war, so it fails criterion three. Count four doesn't fail that badly, assuming figures are not overly underestimated (they usually are, counting only direct deaths), but it's still pretty large number just to catch one guy. Let's be generous and count it as a point.

Anyone who expected an easy win in Afghanistan must have slept through history classes. And Afghanistan is still in chaos, terrorists still terrorize, and Bin Laden was never caught or killed, so criterion five spectacularly fails.

Score: 1/5

The Second World War

Let's move back in time to World War II, widely proclaimed to be the most just war in history, without any basis in fact.

So first, suffering. This is very likely - if everyone just gave Hitler what he wanted (mostly Central Europe to share with Stalin), millions of people would be reduced to status of second class citizens, and depending on how it would go, hundreds of thousands or even millions would die. It's not certain if large scale death camps would be implemented without the war - it seems Nazis were fine with Jews and everyone else they didn't like just getting out of their country (500,000 or so emigrated from Germany before the war, also see the famous Madagascar Plan), but let's assume they really wanted to perform massive genocide. In any case, the first criterion is quite plausible, so full point.

The second criterion fails. The obvious plausible alternative of letting people persecuted by Nazis emigrate to other countries, most were not terribly happy about letting them in, even through Nazis were quite glad about just kicking them out (first confiscating most of their property). Forced resettlement would suck, but it still beats getting gassed.

Now, the third one fails miserably - Nazis killed pretty much everyone they wanted to kill. Most people who would have suffered living under Nazi dictatorship, were instead forced to live under Soviet dictatorship. Approximately 70 million people have died, most of them outside territories that Nazis would have if the world decided against going to war with them. So it's spectacular failure on the counts three and four.

There could be no expectation of an easy victory - spectacular fall of France might have been unexpected, but World War I style trench warfare with millions of casualties was the primary alternative. It's not even obvious if Allies really cared about defeating Nazis - if they were serious about winning, going after them in September 1939 while they were busy in Poland seemed like the best moment, but they didn't do anything until they were attacked.

Score: 1/5

American Civil War

Here's another "glorious" war - American Civil War. It's really historically inaccurate, but let's grant the common myth that the cause of war was abolition of slavery. There were four million slaves in USA in 1860, to be freed. So a somewhat mythical point, but granted.

As for alternatives being untried, most of the world abolished already or soon after 1860, without getting into any major wars. Even countries highly dependent on slave labour, which quite smoothly transitioned into poor waged labour, with or without compensations for former slave owners. There's no reason why it wouldn't work in United States, sooner or later. No points here.

As for point three, yes, all slaves were freed, so entire point here. Sure, there was a century of discrimination ahead, but let's not get too nitpicking.

Point four utterly fails. 620 thousand soldiers were killed, plus some unspecified number of civilians. It was quite clean as far as wars go, so I wouldn't expect a huge value for civilian excess deaths. Still, that's only 6 slaves freed for one death. I'm sure most abolitionist activists had far better than this number - by collecting money, and buying off slaves' freedom, or by nagging slave-owners' conscience until they agreed to free slaves one person could surely free far more than 6 slaves with lifetime's effort.

As for point five, it was successful, and due to North massively overwhelming South in industrial capacity, military strength, and population, it could be expected to be an easy win. It wasn't all that easy, but point granted.

Score: 3/5

American War of Independence

American Revolutionary War is another big one, but I really don't see why.

First, suffering - there were some extra taxes white protestant elites weren't terribly fond of. That's zero points, or else Washington, D.C. has as much right to rebel right now.

Second, alternatives. It's automatic fail due to lack of significant suffering, but other countries like Canada successfully got as much autonomy as they wanted without armed rebellion.

Third, prevention of suffering. Also automatic fail, but notice how United States kept slavery long after the rest of British Empire abolished it. It's not at all obvious that it was more representative, or more democratic than other colonies.

Four, casualties. There were about 52 thousand soldiers killed, plus unknown number of civilians (but not too many, as it was one of the rare clean wars). Vast majority of those were from disease, most commonly scurvy, a disease that was already known to be easily preventable. Battle deaths were barely over 10 thousand. That's small by modern standards, but not trivial relative to two million people living in rebelling States back then - just about 2%. I'll grant a point, but it's just being generous.

As for point five, expecting an easy victory against British Empire at peak of its power was foolish. It was a lucky win. No points.

Score: 1/5

Cambodian–Vietnamese War

That covers the usual suspects, which I all mentioned mostly to point out how they fail at being just wars. Now let's look at some obscure wars that might actually succeed.

Cambodian–Vietnamese War is my best candidate. Khmer Rouge killed approximately 1.5 million people out of population of 7.1 million, and weren't stopping. They didn't even limit themselves to their borders, but also raided some towns in Vietnam and massacred civilians there. Overall, spectacular victory at point one.

It's hard to tell what alternatives could be tried, but none are obvious, and due to massive urgent suffering we cannot blame the Vietnamese for not looking too hard. A second point.

Pol Pot was successfully overthrown. Guerrilla lasted for another decade, but it was on much smaller scale. Full point.

Point four is not entirely successful, as China attacked Vietnam to stop their Cambodian operation, with wide variety of estimated, but something of order of 100,000 soldiers and civilians killed. Still, that's far less than number of Khmer Rouge casualties, so point granted.

War could be expected to be easy - huge, well equipped, and experienced Vietnamese army could and did easily overwhelm Khmer army, and it did so successfully. Full point.

Score: 5/5

Most other wars of independence

Cambodian-Vietnamese War proves a just war is possible, even if really rare. Let's go back to other wars for a moment. One very popular kind are all independence/liberation wars, and uprisings against dictatorships.

They usually fail on suffering front - most colonial powers don't engage in massive killing, so the suffering is mostly widespread discrimination, and lack of representative democracy.

They fail on the second front, as countless countries succeeded in getting significant autonomy, or independence, by political action. It worked even against the Soviets, so it doesn't depend on the colonial power being "nice".

As for the third point, most governments that get into power after such wars are dominated by military, or otherwise very far from democratic ideal, and hardly much better than previous colonial powers. You might be lucky sometimes, but I wouldn't count on it.

Fourth - there's usually plenty of civilian casualties. Independence wars, unless the colonial power simply decides to give up early, tend to be really dirty. No points.

Fifth - by balance of power, it's really naive to expect a victory here. Yes, sometimes it happens, not that rarely even, but lucky wins don't count. And there's even lower chance of the new government being much better than the one being replaced. No points.

Related reading: From Dictatorship To Democracy.

Score: usually 0/5, occasionally higher score.

Terrorism

If we go even further from guerrilla uprisings to terrorism, it gets even worse.

Now occasionally like in Palestine the suffering is real, so point granted. In many places like Basque Country it's fairly insignificant.

As for alternatives, political methods not involving violence have been quite successful in countless cases. So zero points.

One interesting thing about terrorism, is that is pretty much always fails. Of very long list of organizations employing terrorism, the only totally successful case is Hezbollah, which kicked Israelis out of Lebanon. Tamil Tigers - who had semi-independent state in North Sri Lanka for quite a long time before being defeated recently, and Hamas and other Palestinian groups, which fought out some very limited autonomy for Palestinians, can be counted as partially successful. Everything else was a spectacular failure. Notice how the successful cases are really guerrillas employing some limited terrorism against mostly military targets as a small side activity, so they shouldn't even count as primarily terrorist organizations. Zero points here.

As for random unrelated casualties, they're present by design. Usually they're not terribly numerous compared to a proper war, so actually we could grant half a point here, but that would be stretching it. And that would only count if it worked, what not a single case of terrorism so far did.

As for chances of success - well, so far they're 0 successes in far too many tries, so no points.

Score: 0/5

Bombing of Bosnia and Herzegovina

Let's look at operations like bombing of Bosnia and Herzegovina.

First, suffering. Over 100,000 people died in the war, about half of them civilians, so it was a pretty good cause. Plus millions displaced, usual looting, rape, and what you usually see in dirty wars. Full point.

Second, alternatives. Some effort was taken, and out of three fighting parties, UN managed to get two to agree to stop. There were also attempts to limit civilian casualties by creating safe zones (openly abused for military advantage, but that's a completely different issues). You could argue about lack of fairness by UN, and if really everything that was possible was tried, but I'll grant the full point. Also military action was quite limited, aimed at getting quick political solution, not full annihilation of Serbian military capacity.

Third, prevention of further suffering - that's an easy point, operation was fully successful.

Fourth, no extra casualties - bombing is estimated to kill over 100 Serbian soldiers, a trivial number, and many more of just Serbian soldiers would have died if the fighting continued. Full point.

As for estimating chances of victory... well, wars are rarely won by just dropping a few bombs, but NATO was overwhelmingly stronger than any of the fighting parties in Bosnia, and it was successful, so full point.

Score: 5/5

Kosovo and other operations

While bombing of Bosnia was really successful, other peace enforcement campaigns like one in Kosovo much less so.

There were around 2000 casualties of the war between Serbian military and police and KLA, but no massive genocide ongoing. The best example of war crimes found was Račak massacre of about 40-45 people, and even that isn't really terribly solid. There was killing due to fighting, and large scale displacement, but that's rather underwhelming, so just half point this time.

Alternatives were under-explored. There was some attempt at negotiating, but it's not really obvious if NATO seriously wanted a solution that would be acceptable to both parties, as opposed to just getting some cover. No pressure was put on KLA, even though they would be fairly easy to press. And during the negotiations all sides issued a joint statement agreeing to "substantial autonomy for Kosovo", and "invited international civilian and military presence in Kosovo", so it's not really obvious what the subsequent fighting was about. No points.

Prevention of suffering was also quite unsuccessful, the war erupted even more strongly after bombing started, with some proper massacres this time. 80% of people killed, and 90% of people displaced happened after the bombing started. 250 thousand Serbian refugees never returned to Kosovo. No points.

Bombing as such didn't kill terribly many people, around 500 civilians, but due to massive increase in fighting on the ground huge number of people died. There was also a later spill-over of fighting into Macedonia, but it wasn't that severe. No points again.

I'm undecided if the last point should be granted or not - previous successes like Bosnia suggested that it would be an easy victory, but it worked only so-so, and large scale ground fighting caused a lot more suffering that was prevented. Half point.

Score: 1/5

Summary

According to criteria of successfully and reliably preventing significant suffering, the only good wars I could find where overwhelming force is used to stop already happening war or other large scale atrocities, and they limit themselves to minimalist goals instead of trying major political restructuring.

All wars where one side is not overwhelming, like WW2, and American Civil War, tend to cause massive suffering, comparable or worse than whatever can be prevented, even if one of the sides is clearly doing a lot of evil.

"Wars to End All Wars" are as old as known history, and include not only WW1, and WW2, but events as ancient as civil wars of late Roman Empire, and probably many before it. Success rate - 0%. Using this argument to support any war is just ridiculous.

All fighting from position of weakness - anywhere from terrorism to full scare insurgency - tend to cause massive suffering, and have even few successes at preventing major suffering.

If you know of any other wars that score very high according to my criteria, or have problems with my rating system (ratings are quite subjective, but it's hard to come up with anything much better), post it in comments.

Chomsky is a total idiot

Which all means that Chomsky is a total idiot, as he is a big fan of the kind of unsuccessful wars from position of weakness like Tamils in Sri Lanka (80 thousand or more people killed with nothing to show for it), and a big critic of successful wars like Bosnia (apparently Srebrenica was "not really a massacre" with only 8,000 or so people killed).

Yes, American government engages in many stupid wars, and causes a lot of suffering. But that doesn't mean that everyone who happens to criticize American government is automatically right and entitled to the high moral ground. Reversed Stupidity Is Not Intelligence, as my favourite philosopher Eliezer Yudkowsky says.

Lasik retrospective

taw — Thu, 27 Aug 2009 12:14:00 -0700

It's been over six month since I had my eye operation, so it's time for a quick retrospective.

I was a fairly atypical patient. Before operation I had mild short-sightedness, about -1.5 in each eye. It wasn't annoying enough for me to ever bother with glasses or contact lenses. I might had some use for them if I ever bothered driving or something like that, but driving in London is a ridiculous idea. And perhaps not even that, I cycled without problems for years.

If you think about it, there are surprisingly few common activities that require highly precise long distance vision. Most of our lives are spent in small rooms, interacting with people and objects no further than a few meters away from us, and the only benefit from long distance vision is knowing that the bus is coming towards you and you should stay off road, not its registration number. Long-sightedness seems like a much worse problem in modern civilization - what use is precise distant vision if you cannot even read what's on the screen, or see faces of people standing next to you clearly?

Anyway, perhaps more due to my transhumanist drive towards self-improvement than an actual pressing need, I decided to go for surgery. To get laser eye correction you need your vision to be stable - unlike with glasses and lenses you cannot change the lasering every couple of years. And you need money, as unfortunately NHS doesn't yet fund corrective eye surgery. What good is socialized medicine, if the only time I needed it I had to pay with my own money? Oh well, that's a rant for another day. Here's the clinic I used.

What operation looks like

Before the operation I underwent a series of eye tests with a lot of fancy machinery. There was obviously the "focus on the picture" machine, looking at letters through a series of lenses, and also machines I haven't seen before - one that puffed into my eyes and observed effects of puffing (I was supposed not to blink), and another that projected rotating light line onto my eye, also no blinking allowed. And there were a few more tests I don't remember too well.

On the day of operation, it went really quickly. First I got anesthetizing eye drops and had to wait there for half an hour or so. Then I was laid on a table, and the operation begun. To prevent blinking some plastic contraption was put around my eye, keeping the eyelid open. Water was poured onto it to clean it up and lubricate, and then some big machine pressed it so it stayed in place. Normally eyes wander around all the time, it feels really weird to have an eye in place and pressed upon. Really weird. It was time for the first laser - it cut a flap in my eye. Then the machine was removed and the flap was moved aside by the surgeon. Of course with a flap removed from my eye, I could barely see anything, it was extremely blurry. I was told to look at a green dot in front of my, while then the second laser reshaped my eye. After a few seconds the lasering was over, and the surgeon put the flap back on its place. I was given a minute of rest, and the same procedure was applied to the second eye. The whole thing lasted perhaps 10 minutes.

After that, I was left in a darkened room to rest for 20 minutes. A good thing I remembered to take my MP3 player with me, otherwise I would be bored to death (of course no reading allowed). Then a quick vision check, I got a bunch of eye drops and was sent home.

First weeks after operation

There was no waiting for the results - I instantly had superclear vision. But obviously that's not the end of story. Eyes after operation are extremely sensitive and fragile - it takes some time for them to heal. I was issued with a long list of activities that I wasn't allowed to do - no Tube on the first day (due to some air pressure issues). Not letting water or sweat into the eyes for the first week. Going outside only with sunglasses. No exercise, no alcohol, no cycling, no make up, no flying, no swimming, no gym, I think the list pretty much implied no sex too, even though it wasn't spelled explicitly. And a complicated schedule of eye drop - lubricating ones to take every 1-2 hours for a week, anesthetic every two hours on treatment day, antibiotic four times a day for 6 days, steroids every 2 hours for 3 days, then every 4 hours for another three days. And sleeping at night only with very uncomfortable eye shields, to make sure they don't touch the pillow.

Fortunately most of the limitations were lifted within a week - the ones that were left for longer were swimming, long list of sports I don't care about like football and skiing, and definitely under no circumstances rubbing, touching, or putting pressure on my eyes. Then a few appointments to check progress (1 day, 1 week, 2 months, 6 months) and that's pretty much it.

There was no pain involved. My eyes were extremely ridiculously oversensitive to sunlight at first - hence the sunglasses - but that passed quickly. They were also very dry, and I had to use lubricating eye drops for a few months. At first it was every 1-2 hours, just as the leaflet said, then it was maybe twice a day, and now I perhaps use them once a week if my eyes feel particularly dry, but usually I don't bother.

After six months

Apparently my vision is "better than 20/20", which is better than human average, but not by a huge margin. No problems left, other than occasional dryness. And I cannot cry, but that's more likely a personality issue than anything related to my eyes.

So it's top human values, not superhuman values yet. But we might get there. Right now, if your vision is bad and unchanging, you should get lasered. But I can imagine that with time we might get to the point when eye lasering with simply be a smart thing to do for pretty much everyone. We are superhumans already. We are resistant to diseases that could kill wild people with vaccination, we have access to ridiculously good memory with Google search, now we can even get superior orientation in unknown territory with Google Maps and a phone. We can communicate with people who are on the other side of the world. We got so far away from our wild state, it would just be another step on the way to get better vision than they had.

What makes a good sniper game

taw — Wed, 26 Aug 2009 19:54:00 -0700

As a big fan of FPS, especially stealthy/sniperish kind, I'm highly opinionated about what makes a good sniper game. Actually, ignore that part. I'm highly opinionated about pretty much everything. Yes, that's more like it. So here are my thoughts on what constitutes a good sniper game.

Realism

First - the game needs to completely and utterly ignore realism.

You know what real war is like? It's basically walking around in large groups, and not killing anyone for years and years. Most deaths are civilians - by way of intentional killing, not caring much if they live or not, and hunger and diseases that tend to accompany war. Just look at one of the cleanest wars ever - American Civil War. There were 3.1M soldiers, and 625k deaths - that's a fairly respectable 20% death rate (lets skip civilians among them), but only 213k combat deaths - barely 6.7% death rate. Yes - even in a clean war like syphilis is far more dangerous than enemy bullets. In other world, you can expect to kill 0.067 enemies in all four years of that war. Wouldn't that make an amazing game? Play 60 years 24/7 to get one kill. How interesting would that be?

This number is of course nonsense - even most combat deaths are artillery, air bombardment, tanks, machine guns, and other big military hardware. An infantryman's chance of killing anyone is just ridiculously low. It's basically years and years of walking around.

Do you want to know what are the chances of a bullet fired killing someone? The world produces something between 10-14 billion rounds of military small arms ammunition a year. Well, we don't have any war now, but using reasonable (within order of magnitude) assumptions that World War II era's ammo production was more or less the same as today's, and that 20% of military deaths were from small arms fire (as opposed to non-combat and big guns), that gives 1 kill per 17,500 rounds shot. And now imagine a typical game where you're given 100 rounds and sent against a hundred or so of enemies. Ridiculous much?

[Probably the best sniper in history]
OK, you might say, but we're talking elite snipers here, not masses of lowly grunts. Then yes, it gets much more interesting indeed. Sniper kill rates are one kill per five round or better. The best sniper in history of the world - Simo Häyhä killed between 500-800 Russians with his sniper rifle, plus over 200 with submachine gun. And he used iron sights, not any fancy hardware.

Still this completely ridiculous number of 700-1000 kills took place over 105 days - that's less that one kill per hour, even assuming he didn't overwork himself and took plenty of rest. And that's so far beyond what a typical sniper can except to get, unless of course it's a video game sniper - those get 5 kills a minute, if it goes well.

[I'm not making up rocket pistols]
Does it mean realism is completely unnecessary, and the games should be using plasma rifles, and rocket pistols instead? Well, that might be cool as well, but reasonably accurate reproduction of battlefield mechanics as an emergent property can also result in reasonably accurate balance of different kinds of weapons, and different strategies, and reproduce richness that would be difficult to program manually.

Hollywood realism

The second thing we should abandon is Hollywood realism. There are so many ridiculous tropes in movies and video games that are repeated over and over again - people shooting without aiming, or shooting machine guns while holding them in their hands, grenades having ridiculously small casualty range (in most games you're safe if 2-3 meters away, and you need to be pretty much hit in your head to die from a grenade). Fragments from real granades can hit you even more than 200 meters away. And of course in movies wounded people either die instantly, or keep fighting. Nobody is ever wounded in need of hospitalization.

I'm not a fanatic of realism. Real war sucks. If you want some, volunteer to Afghanistan. You don't even need to hurry, that war isn't going anywhere. But often just going the easy way and making it realistic does improve gameplay, so why not do it?

Elements of good sniper games

Pretty much the best sniper game ever is Crysis. Now I'll wait for half of the readers to close this tab. Done? All right. So what makes Crysis such an amazing sniper game?

First, it's a PC game. Sniper games pretty much need mouse and keyboard, no controller gets even close to that. Unfortunately more and more crappy FPS games move to consoles-only, or are console games with crappy PC ports like FarCry 2, what's not much better.

Next, you do have peripheral vision. Humans have plenty of senses they can use to know what's happening in their vicinity. In addition to narrow field of view in front of them, our eyes can see large space around us, though quite poorly. The most practical way to represent peripheral vision in game is with some sort of radar view, just like Crysis does it.

And you can stay camouflaged, while moving at reasonable pace - in so many other games you can spend five minutes quietly crawling through grass, and then being instantly killed by an enemy sniper. In Crysis as long as you're in invisibility mode (which lasts only short time between rechargings), behind cover, hidden in vegetation, or sufficiently far away, you'll be safe.

Enemies behave reasonably sanely, as far as games of this kind go, even though I'd hope for a lot more in the future. They try to get cover, and shoot where they think you might be, and if you're not careful you will die. Crysis did not have to resort to giving them ridiculous aiming powers, or other dirty tricks like that as many other games do.

What's wrong with Crysis

Hey, I like a computer game? That doesn't sound much like me, does it? So let's start talking about what's wrong with Crysis.

First, weapons balance. You're most likely to use the assault rifle for everything. It beats everything else at every distance. It has much higher chance of killing a nearby enemy without aiming than either shotgun or SMG; with single fire mode, decent scope, and Maximum Strength mode it's extremely competitive with sniper rifles on long distances; and it's far better for short to medium distance kills from concealment (where silencer is a must) than anything else. For medium distance burst fire shootouts there's no alternative. And it has far more ammo than sniper rifle and the briefly featured gauss rifle. As a bonus you get a good grenade launcher to instantly get rid of enemy vehicles. There's simply no situation where anything except assault rifle is significantly better than it.

I have some more complaints about weapons. First, ammo is ridiculously sparse. You have to salvage ammo from your kills, and each Korean has just one magazine on him, and no grenades. What of course doesn't stop him from firing infinitely many rounds, and throwing as many grenades as he feels like throwing. It's especially bad with sniper rifles - forcing you to use assault rifles on long range - or move to midrange and use assault rifle from cover, what's actually pretty cool.

Another problem I have is that cover is too strong. Yes, you can blow most kinds of cover like metal sheets, but as long as they stand they protect whoever is behind them. Games like Call of Duty 4 get it right, thin cover offers protection, but not terribly much. I blame Hollywood realism. There's also the usual problem with grenades and RPGs being ridiculously underpowered, even if not as much as in many other games.

There are also some Crysis-specific issues. Invisibility mode, while really cool, can be abused - all energy is lost if you shoot while invisible, forcing you to become visible just like with D&D Invisibility spell. But - if you turn it off, headshot, and turn it on back, you can kill someone in a crowded room without losing any energy! It's an useful combo to learn.

Also rather annoyingly many seemingly cool features are pretty much useless. Tranquilizer darts would be useful for quietly taking down enemies, except they get up in 5 minutes, and set up the alarm - making them pretty much useless. Strength mode for melee attack is quite pointless, as you can as well be invisible, and shoot your target. Speed mode can only be used while high on energy - but if you're high on energy you can get invisible for safety, and if you're low on energy and have to escape, well too bad, speed mode won't work either. Vehicles are pretty much suicidal to use - their weapons have ridiculously low accuracy compared to your small arms, low firepower, and while inside you're a sitting duck waiting to be killed.

That's all still pretty cool for a sniper game.

Games that get it wrong

First, FarCry, no matter how much I liked it when it came out. You have sniper rifle which is good for what it says on the tin, except for its severely limited range. So you had a choice of either assault rifle - which instantly killed the enemy on headshot, but you needed to be close (for headshot), and it was loud enough to alert everyone. And MP5 SMG, which is silenced, but it doesn't even kill on headshot. So in either way, you get detected straight away. But the binoculars/radar mechanism is brilliant (and reused in Crysis).

Call of Duty 4 - there are sniper rifles, but because enemies respawn almost instantly, sniping is completely and utterly useless. You cannot clear the building and then move, or even clear the building, and order your team to move. Just forget about sniping and get the assault rifle back.

Then there are sniper games like Sniper: Art of Victory, which gets everything wrong.
Concealment being hit and miss for no reason, retarded AI, no peripheral vision, amazingly crappy grenades (less than 1 meter casualty range), no penetration of even glass windows, missing point blank shots with SMG, ridiculously low ammo, especially SMG ammo (like 10 bullets), and a very long list of bugs. And these are just some of the gameplay mechanics issues. Boring missions and crappy graphics can be ignored if sniping in fun, but the game is atrocious in every possible way imaginable.

And of course every single console FPS gets sniping wrong by using a controller.

That's it. Now you can go and design a good sniping game. Good luck!

Get better at go with gnugo

taw — Sat, 22 Aug 2009 20:10:00 -0700

Usually I write about things that work quite well - who wants to learn about stuff that doesn't? Or about spectacular failures. Near misses - things that almost work but not really - are a rare subject. This post is about one of such near misses.

I'm sometimes casually playing go at KGS. My account is at 10k now, not too impressive, but then I don't play that much. That's just what my personality is like - I don't do that many things with any regularity, I start something, get really into it, and then a few weeks later my enthusiasm wanes. Not always, but it's quite typical.

So of course I would like to get better at go, without too much effort if possible. I know what's your first thought, but it's not about not losing. You know what's the biggest problem with multiplayer games? For every winner there's a loser, and 50% of people must lose, and the winners tend to be asshats without life who spend all night playing the game, making it impossible for casual players to get anywhere - sure, a casual player like me could slowly learn how to play in multiplayer mode, but losing to asshats a few times too many tends to be too discouraging.

So go is not like that. Everyone has rank assigned, and when people with different ranks are playing the weaker player gets big head start. If you win too much, your rank increases. If you lose too much, your rank decreases. So typically everyone wins half of the time, and loses half of the time - unusually convenient situation for casual players. My rate of wins and loses is pretty much independent of how good I am.

If that's the case, why am I interested in getting better? Mostly because I really hate losing due to spectacularly bad moves. Sure, there are legitimate ways of losing the game - playing too slowly and not getting enough territory, losing a big fight, the normal cases. What is not cool is losing the game by moral equivalent of putting yourself in atari, like this 9dan pro did:

How can one get better at go? First one can play a lot. That definitely helps, but there are quite a few permanoobs - players with way too many games that never progress much. And just by playing and not doing anything else it would take ages to get better.

The second way is doing go problems. They let you get better at fighting and endgame, maybe teach you some common joseki - all very useful things, but they won't teach you any big concepts, or improve your grand strategy. I've seen quite a few players who could win every fight with me, and then lose 20 points by playing some completely useless move, many times during one games.

The third thing is serious studying, like watching games of pros and good amateurs, reading Sensei's Library and books etc. I tried that, but I don't really see what's going on in pro games, and Sensei's Library is mostly targeted at dan players, so it's of limited use to me. And of course it takes a lot of time.

The fourth way, and one that I really like, is reviewing your games. You can review alone or with your fellow player - especially useful in handicap games where White can teach Black something, no matter who won. Or you can send your game to The Go Teaching Ladder, to get a review by a stronger player. Unfortunately that takes ages, and you cannot review every single game this way.

So I thought - bots aren't that horrible at go, some are near shodan level, why not use bots to do a quick review? Now certainly, it won't be as good as a human review, but it's really quick, and hopefully would be of some value.

Gnugo already has some analysis mode - it can take sgf and annotate each step with its idea of what possible moves for Black and White there are and how good they are.

Because its interface is quite inconvenient, I wrote this tiny Ruby script, so I can say annotate_sgf foo.sgf to get annotated-foo.sgf:

level = 15
ARGV.each{|sgf|
 annotated_sgf = File.dirname(sgf) + "/" + "annotated-" + File.basename(sgf)
 system 'gnugo', '--level', "#{level}", '--output-flags', 'dv',
 '--replay', 'both', '-l', sgf, '-o', annotated_sgf
}

And here the problems start. First, while top bots are around shodan level, gnugo is not. It's more like 8k, not horrible (I wouldn't mind a review by an 8k player), but not impressive either. Especially for weak double kyu players it might be pretty attractive.

What gnugo is pretty decent about is noticing groups that were unsettled but both players forgot about them.

Here both players ignore T16, which is obviously vital point for White group if you look at it. I haven't bothered thinking if S6 is another such point, but if gnugo think so, that's at least reason enough to check.

Here's another example - players didn't notice that Black had a ko to mess with White. Gnugo is pretty decent at endgames, and it's surprising how often double digit kyu, and weaker single digit kyu players make such big mistakes.

Gnugo can also point at players making moves that are too small. N1 is a valid move in endgame, where nothing else interesting is going on. At this point it's just small gote reduction. This is extremely common among weak players - territory is real, and influence is vague, so they're willing to fight for scraps of territory instead of grabbing major influence over the board.

Gnugo can also help you stop making unusual fuseki and joseki moves. Unfortunately it's mostly just applying its joseki dictionary without thinking much, so you must think some more if the move is really bad, or gnugo just doesn't like it. Unlike with endgame and semeai, it's hard for you to verify if gnugo is right.

So why don't I like gnugo in spite of all this? Well, I don't really think I'm learning much from it. It doesn't tell me why a move is good or bad, it doesn't create variations showing what are the consequences of different moves (even though it obviously tries them internally). It might tell you which move lost you the fight if you're really curious, but it's difficult to generalize this to new games.

Basically, I don't think I got much better thanks to gnugo. But try it if you want.

Richard Dawkins' The Ancestor's Tale - audiobook review

taw — Mon, 17 Aug 2009 20:21:00 -0700

I don't really read paper books much. Hours of staring at computer and reading blogs and other stuff online are enough effort for my laser-powered eyes. More often than not when I want to read a book, I get an audiobook version of it. Yes, audio has a lot of problems, like lack of searchability, and difficulty of just skimming through less interesting bits. And virtually all audiobook recordings are way too slow. Fortunately my wonderful MP3 player supports increasing audiobook speeds. Side effects include everyone sounding considerably higher-pitched than they really do, but that's something you can really get used to it. By the way - is there any way to avoid that? I know that the most naive algorithm of resampling the original audio wave changes pitch and speed at the same time, and that's what seems to be used, but there surely have to be some smarter ways, right? Especially since I wouldn't really mind even higher speeds.

So, today's audiobook review is Richard Dawkins's The Ancestor's Tale. The audiobook version is unfortunately abridged, what's the dumbest solution to low audiobook speed problem you can think of. MP3 player manufacturers should really do something about it. Or surely there must be some programs out there to speed up audiobooks without making it unreasonably high-pitched right? At least for The Pirate Bay's version, as opposed to DRMed versions.

Humans

Anyway, the book starts with extremely long and boring preface explaining what the book is going to be about. I'm pretty sure many people will just turn it off before the real material starts. Then there are 40 concestor steps - concestor being what's more commonly known as most recent common ancestor of some clade of organisms. Yes, if you want that explained, go ahead and listen to the book preface. Or read some Wikipedia.

Then we move back concestor after concestor, and tell a story of what happened to organisms that split from our branch at that time. The first stories are about Neolithic Revolution, and Behavioral Modernity - two major changes on the way from humans being more sophisticated chimpanzees to humans as we know them. I find the notion of sudden rise of Behavioral Modernity extremely dubious - it disagrees with any molecular evidence, and the only proposed mechanism - acquisition of language - is highly dubious. Our best evidence suggest Neanderthals could most likely speak just as well as us (their FOXP2 gene is identical to ours and anatomy of their speech organs seems to allow speech), what pushes language way back to our common ancestor with Neanderthals, at least 660 thousand years ago. If that was so, it would render all theories of language-behaviour connection completely ridiculous. If Neanderthals couldn't speak, our best estimate for acquisition of language is around 200 thousand years ago, about the time Homo sapiens arose.

In the audiobook Dawkins acknowledges the problem, but postulates that maybe around 50 thousand years ago (supposed time of Behavioral Modernity revolution) language became "more advanced" as opposed to old and primitive kinds of language - this radically disagrees with the data, as modern languages take just one generation of children to reach full complexity and richness, as attested by Nicaraguan Sign Languages and all world's creoles. Yes, I'm going to argue with Dawkins a lot, about things that's not directly related to biology.

In the early chapters Dawkins talks a lot about human culture. An amazingly cool idea is that sheep and alikes "domesticated" grasses. Here's how it supposedly happened - sheep graze on grasses and other plans, what's individual grass plants are not too happy about. But - grasses take grazing reasonably well, while most other plants are far more devastated. As the result sheep give grasses marginal advantage over other plants - sure you're hurt, but the competition is hurt more, and now you have more sunlight, soil, water etc. for yourself. Isn't that great?

Something similar might have happened with domestication, and there's a very interesting domestication story - about tame silver foxes. You know the list of domesticated animals, right? Wolves (aka dogs), boars (aka pigs), aurochs (aka cows) and so on. Foxes are very definitely not on the list - few carnivores are for that matter. So how hard would it be to take a random wild species like a fox, and domesticate it into a pet? It turns out just a couple of decades by selecting for tameness. No genetic engineering or advanced biotechnology necessary. You could run this experiment on your own, given enough money for food and keeping a few hundred animals of whatever kind you want. As a side effect, selection for tameness seems to select for a variety of different characteristics useful for domestication, such foxes behave a lot more like dogs than like wild foxes. It will obviously take longer with species that breed slower than foxes, like most primates. That was your first thought, wasn't it? Domesticated marmosets!

Actually am I reviewing the audiobook or just randomly talking about various subjects vaguely related to it? Oh well, that's my blog, so you know what to expect, and staying on topic it is not! So I want to say I think Jared Diamond's argument that Europeans "had to" win the race because of geographical opportunities makes a lot less sense if you look at the data. The argument goes something like this - there are only 14 species of large domesticated animals. Therefore, there are only 14 species of large domesticable animals, and every animals that wasn't domesticated was obviously impossible to. This is bullshit, as silver foxes prove - just a couple of decades of effort, and you add another animal to the list. OK, it's not a "large" animal, so it doesn't count according to Diamond's criteria, but it proves that domesticated and easily domesticable are nowhere near the same thing.

Oh, and it's not like there weren't any other large animals elsewhere in the world. The Earth was full of them. Not only large land masses like Afroeurasia (silly name, but it makes sense here, as animals can walk from one end of it to another easily, climate issues notwithstanding), but also in Americas, and even on many small islands like Madagascar (elephant bird) and New Zealand (moa). The ones in Eurasia and Africa were just best at not getting extinct, probably by being exposed to Homo for millions of years, as opposed to suddenly.

Dawkins talks about it a lot, not just in the first chapters but all through the books. One amusing idea of his is the "wild Homo sapiens". What does it even mean? We're certainly not wild since the Neolithic Revolution, and arguable we never were. A simple question - do humans form a single species? That's an absurd question, of course they do, after all they can and do interbreed.

But isn't the criterion interbreeding in the wild? Do you know any wild Homo sapiens? Even modern hunter-gatherers are most emphatically not wild, with all the "contamination by civilization", and it's difficult position to claim that Paleolithic humans were really wild. Dawkins gives an example of a pair of grasshopper species that can interbreed perfectly well in zoos, but in the wild they never do so because they use incompatible mating songs. And according to him - how is it different from different groups of humans not interbreeding due to incompatible language and religion? Yes! Dawkins pretty much blames religion for development of human races.

Did I mention how the book doesn't bother with disclaimers, or political correctness, and whenever Dawkins disagrees with someone, he pretty much calls them names - with traditional British eloquence rather than common blogger vulgarity like I do, but it's pretty much the same thing. Oh by the way if you want some solid evidence that religion is a good barrier to human interbreeding, apparently European Ashkenazi Jews rate of interbreeding with surrounding European population, without any geographical or linguistic barriers, purely religious ones, was barely 0.5% per generation - a Jewish person was 200 more likely to mate with another Jew than with a non-Jew. It can be realistically imagined that with addition of some linguistic and geographic isolation barriers between tribes might have been much more significant, leading to a situation where an impartial observer might call different human tribes "distinct species". Of course this situation isn't permanent, and for example intermarriage rate for modern American Jews is 47%, far higher than historical rate even just a couple of decades ago. This makes such barriers temporary at worst, not enough to cause speciation, but plausible partial explanation of the origin of races.

Yes, Dawkins does talk about races. One of his claim about reality of races is how much different observers agree who belongs to which race. Like Colin Powell, who is supposedly "black". By the way, he doesn't look the slightest bit "black" to me - he's a total whitey to me, and Obama is obviously mixed race, not "black". But what do I know, I'm not American. By the way at various stages of history English-speaking people used to believe Irish, or Jews, or Southern Europeans, or Central/Eastern Europeans, or Middle Easterners, or Hispanics, or Indians etc., were "distinct races", even though they're all obviously totally white to me. Oh wait, they still believe Middle Easterners and Hispanics are "distinct races". Silly Americans. There's even plenty of Japanese people who look quite white-ish to me, even though most don't. So I don't really buy the "interobserver agreement" argument.

Dawkins talks a lot about how we shouldn't be giving to much weight to labels like species, and higher clades like phyla. There's plenty of cases like ring species, and gradual divergence of distinct clades where such thinking leads to unnecessary confusion.

Before humans

Wow, I wrote so much and I didn't even get to concestor 1 - of us, Common Chimpanzee, and Bonobo. Numeration starts from concestor 0 - ancestor of all humans. I'm not following the audiobook chapter by chapter - Dawkins talks about human issues a lot, all throughout the book. So first, I don't really buy the idea that there's some relevance of "a common ancestors of all living humans", supposedly living just a few thousand years ago, by statistical argument. Because humans reproduce sexually, we have a gene pool, and it doesn't really matter if there was such an individual or not. Imagine there's an island with 1 man and 10 women. That guy will be the common ancestor of everyone on the island starting from the next generation, but that very much does not mean all the genetic divergence only starts then. Perhaps the women are from 10 different places is the world, isolated for tens of thousands of years - genetic divergence would go hundreds of thousands years back. This is an exaggerated example, but with sufficient mixing of gene pool the time of one of the most recent common ancestors can easily be orders of magnitude than the time of the last genetic divergence.

This is less of a problem for earlier concestors. But there's a second problem. Dawkins claims there are only 40 concestors on our road from humans to concestor of all currently living organism. But that's just limits of our knowledge. It's quite likely that a few more concestors will be inserted around our merger with other Eukaryota, and then with Archaea and Bacteria. Or perhaps not. There are fairly strong limits on our knowledge of evolution that far back.

I think Dawkins commits something I'm going to call the "cladistic fallacy" - the idea that for example humans cannot possible be more closely related to Bonobo than to Common Chimpanzee, just because Bonobo and Common Chimpanzee have a common ancestor not shared by humans. This is technically defensible position, and well calibrated molecular clocks will confirm it - but how many years in the past we shared an ancestor is not a meaningful measure of relatedness!

For one, the same number of years doesn't mean the same number of generations - otherwise you would be equally related to your brother and your nephew, just because your last common ancestor in both cases is the same - that's obviously nonsense. More importantly, different evolutionary lines can diverge at different rates.

One example is Canine transmissible venereal tumor, a single-celled parasite of canines that evolved from dog tumor tissue hundreds or thousand years ago. Yes, it's a different species by all means. But it would be ridiculous to say dogs are more closely related to CTVT than to let's say coyotes. Sure, genetic clock says so, but that's not a meaningful measure of relatedness. CTVT is not a canine mammal is any meaningful sense of the word. Another such parasite is DFTD. There's also an arguable case of HeLa (Helacyton gartleri) - a similar species derived from humans - but in this case we could ignore it as a artificially created. A less drastic example are hippos and whales - yes, technically hippos are "more closely related" (as measured by time) to whales (Cetacea) than to any other Artiodactyla, but is it really the best measure? By the way, it's not really mentioned in the book, but there's an interesting even though most likely false theory that humans are more closely related to orangutans than to Chimpanzees, using similar arguments. Read if you're curious, even if it's unlikely.

Dawkins also panics for a moment about the problem of comparing all species against all others. Hippos and pigs both look like perfectly good Artiodactyla, so if you naively assume that it's a proper clade and try to just take one of these species for genetic comparisons, you might get different family trees. I understand his point, but DNA sequencing is getting ridiculously cheaper at exponential rate (I mean, genuinely not just metaphorically exponential). When the book was released apparently humans and mice were the only two mammals with fully sequenced genomes. Not any more. Right now there are at least 10 fully sequenced mammalian species. And you don't even need full sequences for genetic tree reconstruction, just a selection of a small number of representative genes will give a pretty decent approximation. It's funny how quickly books about biology get outdated.

Most of the audiobook is about different animal groups, and interesting things about them. Like how Deuterostomia (that's you) can be thought of as upside-down Protostomia (like insects). And how to explain Cambrian explosion - is it just an artifact, or was it real? How to explain the paradox of sex - why do you throw away half of your genes when you make children, instead of just cloning yourself and doubling your genetic success? How truly irreducible complexity (which we haven't found at all, so don't worry) would be a good evidence of directed panspermia, as opposed to creationism. About endosymbiosis, fish lungs, Platypus's bill, some theories on origins of life, and a lot more. Half of the stories are excluded from the abridged audiobook, unfortunately, so go on get a paper version if you want to get them all.

That's more or less what the book talks about. It's not the best Dawkins ever (especially the overly long preface annoyed me), but it's pretty decent Dawkins. Highly opinionated (and I don't always agree with his opinions), eloquent, and about something extremely interesting. It's definitely popular science, so hardly any biological background is needed. Enjoy.

Security magical thinking in ssh

taw — Sun, 16 Aug 2009 19:05:00 -0700

The authenticity of host 'delicious.cake.net (110.120.130.140)' can't be established.
RSA key fingerprint is 18:93:ea:38:81:84:ae:34:b4:56:0e:51:7c:28:ef:cd.
Are you sure you want to continue connecting (yes/no)?

How about a few bets? So many stupid things are said about security, more often than not by drastically overestimating importance of security measures to people... Should we try prediction markets instead? I haven't heard about any research on prediction markets with thought-experiment money, but they do work with both real and play money, so I'm accepting your though-experiment bets!

First, let's find a random ssh user trying to access delicious.cake.net or some other server. I bet they're not going to check the ssh key fingerprint. Instead, they'll just blindly type yes without thinking. What rates do you take?

A second bet. Let's find someone who did not bother to check the ssh key fingerprint. I bet they're not under any kind of man in the middle attack, and everything will be perfectly fine. What are your rates here?

And one more. Assume someone is under man in the middle attack. I bet they would be owned by some other channel anyway, and so ssh key fingerprint check wouldn't help them at all.

My initial guesses as for the rates are 100:1, 10000:1, 10:1. I bet 9000 internets on each one of the possibilities. Yes, I know this is ridiculous, but go on and offer counter-bets if you think I'm wrong.

Why the magical thinking?

So assuming I'm right, and the ssh key fingerprint check is useless, why don't we turn it off, or at least provide an option of doing so? This silly question is asked even with EC2 instances, where the instance key is randomly generated, so it's impossible for you to know it; and the instance already has your public key, so hopefully man in the middle attack shouldn't be possible (unless ssh works in some perverted way where that's not enough).

I bet the process goes something like this - ssh developers realized that man in the middle attack is possible. So what were their options? First, they could bother with the public key infrastructure of some sort, but seriously, that would be too much work, both for them and for admins. Second, they could just accept the possibility of man in the middle on the first connection. Both options make sense.

But no - instead they did some magical thinking, and added a completely useless security ritual, a bit of security fairy dust that magically "makes things secure" even though in reality it does nothing other than annoying the user.

It's not ssh's fault really - we're all trapped between irrational belief in importance of security, and reality of most kinds of attacks never happening due to high cost and low benefit to attackers in absence of economies of scale. So instead of reevaluating our beliefs we implement useless security rituals like that. And they work! After all, I don't see any ssh man in the middle around, do you?

Tastes of movie critics and tastes of the public

taw — Sat, 11 Jul 2009 11:19:00 -0700

Continuing last week's post where I noted that Transformers 2 was quite enjoyable even if unremarkable action movie, I'll continue with some analysis of tastes of movie critics versus tastes of the public.

In case you missed the story Aircraft Careers 2 is a movie with huge box office success, decent public reaction, and completely hated by pretty much every single movie critic. Go to the cinema and watch it to see what the controversy is all about ;-)

As proxy for taste of the public I'm going to use The IMDB Top 250 list. I'm interested in what the public actually liked, not just what they were duped into watching by pretty trailers and marketing buzz, and this list is pretty much it. Top 250 list contains movies that were both very highly rated and rated by large number of people, to exclude ones that nobody cares about other than the director and the lead actress, who both voted 10/10.

As proxy for taste of movie critics - The Tomatometer, or more or less a weighted percentage of movie critics that liked a particular movie.
And... it turns out movie critics and public agree most of the time. 18.8% of the top 250 has very rare perfect 100 tomatometer score, 77.6% has extremely good score of 90 or more, 91.6% has still very good 80 or more score - perhaps critics are not as amazed by the movie as the audience, but they liked it anyway. There are just 19 (7.6%) movies with scores in their 70s, and 2 (0.8%) in their 60s - where we can genuinely says the critics were largely unimpressed. Not a single movie has a horrible score like Transformers 2's 19.

Here's the entire list:

Tomatometer	Movies
61	Changeling (#219)
64	Public Enemies (#215)
71	Snatch. (#156), Lock, Stock and Two Smoking Barrels (#190)
72	Forrest Gump (#43), The Curious Case of Benjamin Button (#188)
73	V for Vendetta (#176)
74	Léon (#35)
75	The Prestige (#83), Star Wars: Episode VI - Return of the Jedi (#114)
76	Braveheart (#102), Crash (#223)
77	La vita è bella (#87), Sin City (#98), Gladiator (#115), The Big Lebowski (#155), The Thing (#174), Big Fish (#232)
78	The Green Mile (#111), The Hangover (#138)
79	Requiem for a Dream (#64)
80	Gran Torino (#76)
81	Fight Club (#21), Oldboy (#120), Dial M for Murder (#200), In Bruges (#204)
83	American History X (#39), Donnie Darko (#130), Into the Wild (#146), Casino (#195)
84	Se7en (#32), Batman Begins (#106), Brief Encounter (#166), Magnolia (#235), Harold and Maude (#248)
85	The Sixth Sense (#140), Kill Bill: Vol. 1 (#142), Gandhi (#164), Scarface (#171), The Exorcist (#209), Kill Bill: Vol. 2 (#230)
86	The Matrix (#27), Platoon (#144)
87	The Shawshank Redemption (#1), The Shining (#52), Twelve Monkeys (#189), Mystic River (#238)
88	Nuovo cinema Paradiso (#88), Harvey (#181)
89	The Usual Suspects (#23), American Beauty (#37), Indiana Jones and the Last Crusade (#108), Heat (#133), Judgment at Nuremberg (#143), The Graduate (#159), Trainspotting (#170)
90	Le fabuleux destin d'Amélie Poulain (#45), A Clockwork Orange (#51), The Apartment (#84), Butch Cassidy and the Sundance Kid (#150), Hotaru no haka (#196), Network (#226), Arsenic and Old Lace (#236)
91	Saving Private Ryan (#54), Der Untergang (#81), The Elephant Man (#86), The Great Escape (#93), The Sting (#96), Blade Runner (#112), Per qualche dollaro in più (#118), The General (#126), There Will Be Blood (#129), The Deer Hunter (#139), Ben-Hur (#141), Million Dollar Baby (#149), Letters from Iwo Jima (#203), Ed Wood (#212), Bonnie and Clyde (#220), Little Miss Sunshine (#239), Glory (#240)
92	Cidade de Deus (#17), The Lord of the Rings: The Fellowship of the Ring (#20), It's a Wonderful Life (#31), Paths of Glory (#44), The Departed (#53), Hotel Rwanda (#105), Amores perros (#167), The Lion King (#169), Children of Men (#192), Safety Last! (#218)
93	Memento (#29), Das Leben der Anderen (#56), Eternal Sunshine of the Spotless Mind (#61), Once Upon a Time in America (#91), Det sjunde inseglet (#103), Mononoke-hime (#128), The Bourne Ultimatum (#151), Le scaphandre et le papillon (#216), Rocky (#244)
94	The Dark Knight (#7), The Lord of the Rings: The Return of the King (#13), Raiders of the Lost Ark (#18), Slumdog Millionaire (#67), Monty Python and the Holy Grail (#68), Touch of Evil (#100), No Country for Old Men (#109), Die Hard (#122), Fargo (#124), Smultronstället (#131), Stand by Me (#161), The Ox-Bow Incident (#194), Duck Soup (#197), The Day the Earth Stood Still (#233)
95	Star Wars (#12), The Pianist (#59), The Bridge on the River Kwai (#69), Reservoir Dogs (#70), El laberinto del fauno (#71), Ladri di biciclette (#94), Star Trek (#99), High Noon (#119), The Big Sleep (#123), Sleuth (#178), Spartacus (#250)
96	Pulp Fiction (#5), Schindler's List (#6), One Flew Over the Cuckoo's Nest (#9), Goodfellas (#15), The Silence of the Lambs (#25), The Lord of the Rings: The Two Towers (#34), WALL·E (#42), 2001: A Space Odyssey (#89), Full Metal Jacket (#92), Back to the Future (#95), Mr. Smith Goes to Washington (#97), Unforgiven (#113), Ran (#137), Les diaboliques (#145), The Killing (#162), Ratatouille (#163), Groundhog Day (#175), The Princess Bride (#185), Umberto D. (#186), Stalag 17 (#193)
97	Star Wars: Episode V - The Empire Strikes Back (#10), Casablanca (#11), C'era una volta il West (#19), Up (#22), Psycho (#24), To Kill a Mockingbird (#48), Alien (#49), Terminator 2: Judgment Day (#57), Sen to Chihiro no kamikakushi (#60), Some Like It Hot (#79), Amadeus (#85), Strangers on a Train (#110), Notorious (#117), It Happened One Night (#132), Yojimbo (#134), Le notti di Cabiria (#147), 8½ (#152), Dog Day Afternoon (#165), Gone with the Wind (#168), Sunrise: A Song of Two Humans (#172), The Wild Bunch (#173), The Hustler (#182), The Best Years of Our Lives (#183), The Incredibles (#187), The Lady Vanishes (#198), La strada (#211), Who's Afraid of Virginia Woolf? (#213), His Girl Friday (#214), Rope (#217), All Quiet on the Western Front (#222), Patton (#229), Wo hu cang long (#237), Good Will Hunting (#243), In the Heat of the Night (#246)
98	The Godfather: Part II (#3), Il buono, il brutto, il cattivo. (#4), Sunset Blvd. (#26), Apocalypse Now (#36), Vertigo (#40), Lawrence of Arabia (#41), Double Indemnity (#47), Raging Bull (#73), The Wrestler (#104), The Manchurian Candidate (#116), Annie Hall (#136), Life of Brian (#153), The Night of the Hunter (#158), Finding Nemo (#160), Låt den rätte komma in (#191), A Streetcar Named Desire (#206), The Conversation (#207), Rosemary's Baby (#221), Manhattan (#227), Nosferatu, eine Symphonie des Grauens (#241), Roman Holiday (#242), La dolce vita (#247)
99	L.A. Confidential (#65), Metropolis (#82), La battaglia di Algeri (#201)
100	The Godfather (#2), 12 Angry Men (#8), Shichinin no samurai (#14), Rear Window (#16), Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb (#28), North by Northwest (#30), Citizen Kane (#33), Taxi Driver (#38), M (#46), The Treasure of the Sierra Madre (#50), The Third Man (#55), Chinatown (#58), City Lights (#62), Aliens (#63), Das Boot (#66), Rashômon (#72), All About Eve (#74), The Maltese Falcon (#75), Modern Times (#77), Singin' in the Rain (#78), Rebecca (#80), The Great Dictator (#90), On the Waterfront (#101), Jaws (#107), Cool Hand Luke (#121), The Wizard of Oz (#125), Witness for the Prosecution (#127), Kind Hearts and Coronets (#135), Le salaire de la peur (#148), The Grapes of Wrath (#154), The Gold Rush (#157), The Terminator (#177), The Kid (#179), Toy Story (#180), Shadow of a Doubt (#184), The African Queen (#199), Anatomy of a Murder (#202), The Adventures of Robin Hood (#205), King Kong (#208), The Lost Weekend (#210), Sweet Smell of Success (#224), The Philadelphia Story (#225), Frankenstein (#228), Great Expectations (#231), Laura (#234), My Man Godfrey (#245), The Red Shoes (#249)

If you hoped to see some overmarketed garbage among the public-beloved-but-critically-panned movies, sorry to disappoint you. I'm surprised by how many movies that won Oscars for the Best Picture are there - Braveheart, Forrest Gump, and Crash, plus some nominations like The Curious Case of Benjamin Button.

I haven't seen all the movies critics disliked, but for the ones I've seen I have no idea what was their problem. In particular - Fight Club? Léon? V? The original Matrix? Kill Bill? In Bruges? What the heck critics? These are masterpieces of the modern cinema!

To summarize:

IMDB's unwashed masses and Rotten Tomatoes' distinguished critics mostly agree, and when they don't IMDB tends to be right, so keep using IMDB.

Plan 9 from Outer Forks

taw — Sat, 04 Jul 2009 08:13:00 -0700

I love IMDB. For one, it might be one of the very few non-personalized recommendation systems that actually work. Scores on IMDB correlate very highly with likelihood of me enjoying a movie, especially if I apply a correction for genres that are overrated (like very old and very long movies) or underrated (like zombie flicks). In any case IMDB is vastly more accurate than professional critics' reviews.

The most drastic example of IMDB and pro critics disagreeing was Transformers: Revenge of the Fallen, which I absolutely loved! It got an okish 6.5 score from IMDB - which is accurate enough, it's a really nice action-packed movie vaguely following Transformers canon, with as much concern for the plot as a typical action movie (that is not terribly much), and perhaps far higher aircraft carrier to robot ratio than you'd expect from the title.

It was also universally panned by the pro critics getting Rotten Tomatoes metascore of just 20%. Seriously, critics? Is it really one of the worst movies in existence? Have you expected anything else than robots meaninglessly fighting each other and destroying valuable stuff? It's almost as if critics panned Transformers 2 to signal their sophisticated taste, not to provide good service to the audience.

Anyway, back to my point. IMDB not only provides great recommendation service, they also make a lot of the underlying data available in convenient formats. Yes, they charge huge money for some of the data, but even the free portion is very useful.

Some voting patterns are very interesting. Here are two examples. First, one of the "so bad it's good" movies - Troll 2. Normal movies have sort-of Gaussian distribution of votes. OK, not really Gaussian but with a fairly definite single peak. Not so with the "so bad it's good" movies - these tend to have mostly 1s and 10s, with perhaps a few 2s and 3s, but amusingly very rarely many 9s and 8s - it's either very low, or a 10!

The second interesting bit is a demographically polarizing movie like Twilight. People of different ages and genders tend to like pretty much the same movies.

Top 100 for men, and Top 100 for women contain virtually the same movies, just slightly rearranged - neither stereotypically feminine chick flicks nor stereotypically masculine action flicks get into top lists, and movies rarely have very different male and female scores, or much different scores by age group. But exceptions do happen, here's one:

Under-18 men scored it almost 2 points lower than under-18 women! I have a perfectly good explanation for the Twilight ratings effect - normally movies are watched by people who like the genre, so only men who like chick flicks and only women who like action movies watch them. So there will
be very little discernible gender-genre bias. But with Twilight, millions of girlfriends worldwide must have forced their boyfriends to watch Twilight against their wills, to what the aforementioned boyfriends reacted in a passive-aggressive way by downvoting Twilight on IMDB. I don't have any way of
testing this theory, but I watched Twilight on my own free will, and liked it a lot.

Anyway, back to "so bad it's good" movies. I really like the genre, but it's difficult to tell the "so bad it's good" movies from straightforwardly bad movies. So like a good hacker I am, I decided to grab IMDB's database, and find out movies. Here's the list, criteria being - at least 10% of votes are 1s, at least 10% of votes are 10s, ordered by number of votes.

	Votes	10s %	1s %	IMDB score	Title
1	77257	12.6%	11.0%	6.2	The Blair Witch Project
2	69978	24.6%	12.7%	6.1	Twilight
3	68560	30.4%	10.5%	7.6	Fahrenheit 9/11
4	38459	20.0%	19.2%	5.5	Sex and the City
5	33628	15.0%	10.1%	6.0	Halloween
6	24265	31.7%	23.0%	5.1	High School Musical
7	24044	10.6%	10.3%	5.1	The Pink Panther
8	22873	14.9%	12.1%	5.6	House of 1000 Corpses
9	22324	18.6%	27.5%	3.9	Freddy Got Fingered
10	21704	15.3%	12.4%	5.0	White Chicks
11	19898	14.5%	10.7%	7.6	Midnight Express
12	19418	11.0%	26.0%	3.7	Norbit
13	19230	10.6%	43.9%	2.8	Crossroads
14	18410	11.6%	10.3%	6.3	Funny Games U.S.
15	17779	12.2%	10.0%	4.7	Queen of the Damned
16	17606	10.5%	27.9%	3.8	In the Name of the King: A Dungeon Siege Tale
17	17103	25.3%	10.4%	5.7	Kung Pow: Enter the Fist
18	16840	13.6%	54.9%	2.5	You Got Served
19	16743	14.7%	35.0%	2.8	Spice World
20	16706	10.5%	74.0%	1.6	From Justin to Kelly
21	16596	17.2%	10.8%	5.7	Southland Tales
22	16338	16.2%	31.9%	3.6	Plan 9 from Outer Space
23	16191	19.1%	11.8%	5.5	Step Up 2: The Streets
24	16179	15.1%	40.5%	3.8	Get Rich or Die Tryin'
25	15728	22.4%	44.0%	3.7	High School Musical 3: Senior Year
26	15417	12.5%	38.1%	3.4	Little Man
27	15177	15.0%	11.0%	5.6	Alvin and the Chipmunks
28	14938	13.5%	16.0%	3.7	Super Mario Bros.
29	14431	10.8%	14.6%	5.4	Star Wars: The Clone Wars
30	13960	11.0%	10.4%	4.9	DOA: Dead or Alive
31	13812	41.3%	10.2%	8.1	Ikiru
32	13643	23.6%	27.6%	4.5	High School Musical 2
33	13377	35.3%	61.5%	1.3	Jonas Brothers: The 3D Concert Experience
34	12764	11.2%	43.2%	3.3	Dragonball Evolution
35	12625	10.5%	62.6%	2.0	Glitter
36	12596	18.3%	26.7%	4.2	Stomp the Yard
37	12560	17.8%	11.3%	6.1	Cannibal Holocaust
38	12554	11.6%	11.2%	4.8	Caligola
39	12382	13.8%	23.6%	3.8	D-War
40	12168	10.7%	12.7%	4.9	An American Haunting
41	12006	13.0%	11.9%	4.4	Stay Alive
42	11770	14.0%	18.4%	3.5	Grease 2
43	11740	20.3%	29.5%	4.1	Postal
44	11554	29.3%	10.4%	8.3	Le salaire de la peur
45	11383	14.9%	13.9%	4.6	Honey
46	11257	12.2%	10.4%	5.0	Just My Luck
47	11150	10.6%	24.7%	3.7	Prom Night
48	11054	11.4%	18.9%	4.5	The Marine
49	10942	11.0%	44.5%	2.4	Kazaam
50	10742	10.9%	77.7%	1.7	Who's Your Caddy?
51	10589	11.7%	16.7%	4.7	Meet Dave
52	10397	19.2%	11.9%	6.1	Sal? o le 120 giornate di Sodoma
53	10378	13.2%	16.6%	3.8	Bio-Dome
54	9774	12.4%	20.6%	4.0	Big Momma's House 2
55	9255	45.0%	10.2%	8.1	La passion de Jeanne d'Arc
56	9234	15.3%	10.9%	5.8	Georgia Rule
57	9145	11.1%	71.9%	1.9	Troll 2
58	9121	13.1%	11.0%	5.1	Strange Wilderness
59	9085	12.2%	16.9%	4.3	Black Christmas
60	8906	12.0%	13.0%	5.3	The Ten
61	8888	17.9%	15.8%	4.7	The Lizzie McGuire Movie
62	8840	12.0%	19.9%	3.3	Universal Soldier: The Return
63	8678	18.4%	45.3%	2.9	Material Girls
64	8654	18.5%	22.8%	3.8	Pok?mon: The First Movie
65	8486	21.4%	12.2%	5.8	Gummo
66	8419	10.1%	17.5%	3.8	On Deadly Ground
67	8378	13.3%	13.4%	5.7	Last Days
68	8210	12.7%	76.4%	1.4	SuperBabies: Baby Geniuses 2
69	8179	20.1%	58.3%	2.9	Hannah Montana: The Movie
70	8103	12.0%	52.7%	3.1	Beverly Hills Chihuahua
71	8045	12.2%	29.0%	3.6	Soul Plane
72	8030	20.3%	13.6%	4.5	Aquamarine
73	8013	41.1%	11.9%	8.1	Idi i smotri
74	7898	14.6%	32.4%	2.7	Hercules in New York
75	7854	23.1%	11.6%	6.4	17 Again
76	7804	20.6%	19.5%	5.2	What the #$*! Do Wenow!?
77	7726	26.4%	10.9%	7.1	The Birth of a Nation
78	7714	13.8%	10.4%	5.0	Annapolis
79	7684	10.7%	23.7%	3.3	Problem Child 2
80	7643	25.6%	15.0%	5.1	Raise Your Voice
81	7605	12.6%	11.2%	5.0	Eight Crazy Nights
82	7588	25.4%	12.4%	8.1	Du rififi chez les hommes
83	7563	11.4%	51.8%	2.7	Hannah Montana/Miley Cyrus: Best of Both Worlds Concert Tour
84	7499	12.6%	20.3%	4.0	Van Wilder 2: The Rise of Taj
85	7417	14.1%	12.6%	6.2	Gerry
86	7237	12.7%	55.4%	2.7	Larry the Cable Guy: Health Inspector
87	7169	11.7%	10.2%	4.4	Superstar
88	7095	31.4%	10.4%	4.9	Good Burger
89	7093	28.6%	11.3%	8.1	Z
90	7093	11.7%	75.1%	1.7	Crossover
91	7043	44.5%	10.2%	8.0	Les enfants du paradis
92	6930	23.9%	18.3%	3.9	Mighty Morphin Power Rangers: The Movie
93	6925	23.2%	13.1%	5.8	Pink Flamingos
94	6915	45.5%	16.8%	5.8	Kurtlar vadisi - Irak
95	6766	13.5%	12.9%	4.4	Yours, Mine and Ours
96	6744	10.5%	10.5%	4.2	Cyborg
97	6714	12.9%	40.7%	3.1	Are We Done Yet?
98	6687	20.9%	57.1%	3.7	Expelled: No Intelligence Allowed
99	6586	27.9%	11.6%	8.1	Tengoku to jigoku
100	6555	11.4%	36.8%	3.3	Zoom

Trivia from the list:

Transformers 2 almost got to the list by having 9.9% 1 votes.
The most controversial movie in every way is Jonas Brothers: The 3D Concert Experience with 35.3% 10s, 61.5% 1s, and only 3.2% everything else. It also has huge 5.0 to 1.3 under-18 gender gap, but as you can see even most teenage females aren't huge fans of it. It's also the worst scored movie on the list, with score of just 1.3, what suggests IMDB's filters think most of the 10s are attempt at ballot stuffing, and get thrown away.
The highest rated controversial movie is Le salaire de la peur with IMDB score of 8.3 and somehow still more 1s (10.4%) than Transformers 2.
The highest rated controversial movies that seem popular (by number of votes they received) are Fahrenheit 9/11, and Twilight, both of which I liked, and not because of the "so bad it's good effect", and The Blair Witch Project, which I hated with passion for being so unbelievably boring.
The lowest scored movie from the list I watched was Troll 2 and I liked it because of the "so bad it's good" effect.
Celebrity movies are huge on the list, like Crossroads, Glitter, Spice World, Hannah Montana: The Movie, and the aforementioned Jonas Brothers' movie. The only one I watched of them was Crossroads, and I genuinely enjoyed it, in semi-ironic way.

Things I hate about Stepmania

taw — Mon, 29 Jun 2009 01:48:00 -0700

It's been such a long time since my last flaming rant. In this episode of taw's flaming flames I focus on Stepmania. And I don't intend to differentiate between Stepmania the software, DDR as such, songs, and everything else - all of it is Fair Game.

First, I hate all the delays. The only part of Stepmania I care about is the one playing the songs and checking my steps, everything else is just an extra that should try its best not to get in the way. Stepmania spectacularly fails at it - there are delay screens and delay animations everywhere, some possible to turn off, most not really. I do not need the "CLEARED" screen, I especially do not need the score spin, and then 5 second delay back to the song selection wheel. Just give me the damn score, and get me back to the selection wheel immediately when I press START! PC is not an arcade machine!

Second, slowness. I mean, the non-deliberate delays. Now Stepmania is fast enough with 50 songs like arcade machines, but real installations will have over 9000 songs - what means a few minutes to start the damn thing. What is it doing during the startup, checking for rickrolls?

A minor related annoyance is the menu system. Configuration options are divided into way too many submenus that don't mean anything. You think they do? Now quickly tell me where are the options to speed up the wheel, and turn off all the delays that can be turned off... Yeah, I fail to see the logic behind it too. And while it makes sense to make the dance mat the primary controller, it would be nice if it was possible to use keyboard sometimes too. For the sake of example, let's say I'm looking for a Zelda song I really like but I don't remember the title. Right now I have to sort by title, go to Z(elda), L(egend of Zelda), T(the Legend of Zelda) and probably ten other places! How difficult would it be to add an option of search by keyboard for those times when I'm looking for something in particular? Remember, there are over 9000 songs, it's not an arcade machine!

Oh, and there entire difficulty levels system - I have so many complaints about it! For one, what does it matter to me if a song is an 8-feeter on standard, or 8-feeter on expert? It's an 8-feeter either way, but there's no way to browse all 8-feeters together. (for every possible value of 8 of course)

And I wouldn't really mind if the feet ratings were replaced with something else - foot ratings are sort of reasonably accurate for official releases, but for fan-made ones they can be horribly wrong. I wonder if we could take this 5-difficulty diagram or something like that, measure what player is good at, and convert that to player-specific difficulty ratings... That would be cool, wouldn't it?

Now something about songs - I really hate songs that do stupid tricks with arrows display. The root of the problem is that arrows move along with the beat timer, not along with wall clock time. This causes so many problems like:

Low bpm songs have extremely densely packed slowly moving arrows, and are hard to read
High bpm songs have very quickly moving arrows that don't last on the screen very long and provide very little reaction time
Songs that change bpm a lot, and you don't know when you have to hit the arrows
Songs that freeze beat timer and unfreeze it by surprise - unless you know when the unfreeze happens you'll miss the arrows.

In all cases you basically need to learn the song's arrows instead of looking at them on screen. These tricks could be fun if used occasionally, but they're overused to the point of being extremely annoying.

But most of all let me dance instead of waiting and I'll be happy enough.

Arduino and Wii Nunchuck to control iTunes

taw — Sun, 26 Apr 2009 10:06:00 -0700

Here's my latest project. It involves way too many technologies for the sole purpose of controlling iTunes with Wii Nunchuck. Signal flow is almost uni-directional, so let's go from step by step from the Nunchuck to iTunes.

Wii Nunchuck and I2C

I2C is a very simple protocol back from the 80s. There are four wires. Vcc (traditionally +5V, but lower voltages are used with I2C too) and ground are obvious. Clock and data lines require some more explanation.

I2C is a bus protocol - so multiple devices can operate on the same wires. Even in the simplest configuration we have a master (Arduino board), and a slave (Wii Nunchuck), both of which can read and write to the same lines. And it doesn't take much science to know that when one device writes 0 while other writes 1 to the same line hilarity ensues, also known as a circuit shorted, everyone dies.

The way I2C and many other protocols solve this problem is by using "open drain" design. In open drain no device is allowed to write 1 to the bus - you can only write 0 (connect to the ground), or leave the connection in high impedance state. The bus is connected to Vcc using a pull-up resistor. This way if any device writes 0, bus will be at 0. If none is writing, it will be at 1. That's somewhat unintuitive at first, if it confuses you just check the Internet for explanation.

Fortunately Arduino has hardware support for I2C over its analog input pins 4 and 5, and library for that (called Wire) is included in the distribution. So it would seem that we don't have anything to do... except it all doesn't work.

Now I'm not 100% sure about that, but here's my guess what happens. Arduino analog input pins can be set to output low, output high, high impedance, and high impedance with pull-up (allegedly 20k). For I2C we're interested in output low, and high impedance with pull-up. However - this built-in pull-up seems too weak, or it's not working for some other reason. Directly connecting clock and data lines to Vcc via 10k pull-up resistors (lower resistance = more pull-up, at least until you fry it) makes it all work. Nice way to spend an entire night, isn't it?

So after cutting the Wii Nunchuck wire, the connections are:

Nunchuck white (GND) to Arduino GND
Nunchuck red (VCC) to Arduino 5V
Nunchuck yellow (clock) to Arduino analog input pin 5
Nunchuck green (data) to Arduino analog input pin 4
Clock line to VCC via 10k resistor
Data line to VCC via 10k resistor

Speaking with Wii Nunchuck

Now that we have electric signals handled, we need to engage Wii Nunchuck in a meaningful conversation. It has device ID 0x52 (82), and we need to write 0x40 0x00 (64 0) to initialize it, and then request data in packets of six bytes, and write 0x00 (0) to confirm we got it.

Now I try to avoid word like "retarded" on this blog, but Wii Nunchuck data is "encrypted", and we need to "decrypt" it first:

decrypted = (0x17 XOR encrypted) + 0x17

Seriously, whoever came up with this fully deserves this adjective.

After that it's just a straightforward decoding, and we dump data onto serial interface. Here's full code:

#define CPU_FREQ 16000000L
#define TWI_FREQ 100000L

#include <Wire.h>

int diode = HIGH;

void blink() {
 digitalWrite(13, diode);
 diode = (diode == HIGH) ? LOW : HIGH;
}

void setup() {
 Serial.begin(19200);
 pinMode(13, OUTPUT);
 Wire.begin();
 Wire.beginTransmission(0x52);
 Wire.send(0x40);
 Wire.send(0x00);
 Wire.endTransmission();
}

void send_zero() {
 Wire.beginTransmission(0x52);
 Wire.send(0x00);
 Wire.endTransmission();
}

void loop() {
 int cnt = 0;
 uint8_t outbuf[6];

 Wire.requestFrom(0x52, 6);
 while(Wire.available()) {
 if(cnt < 6) {
 outbuf[cnt] = (0x17 ^ Wire.receive()) + 0x17;
 }
 cnt++;
 }

 int b = ~outbuf[5] & 3;
 int joy_x = outbuf[0]-125;
 int joy_y = outbuf[1]-128;
 int accel_x = (outbuf[2] << 2 | ((outbuf[5] >> 2) & 0x03)) - 500;
 int accel_y = (outbuf[3] << 2 | ((outbuf[5] >> 4) & 0x03)) - 488;
 int accel_z = (outbuf[4] << 2 | ((outbuf[5] >> 6) & 0x03)) - 504;

 Serial.print("B=");
 Serial.print(b);
 Serial.print(" XY=");
 Serial.print(joy_x);
 Serial.print(",");
 Serial.print(joy_y);
 Serial.print(" XYZ=");
 Serial.print(accel_x);
 Serial.print(",");
 Serial.print(accel_y);
 Serial.print(",");
 Serial.print(accel_z);
 Serial.print("\n");
 
 send_zero();

 blink();
 delay(50);
}

Controlling iTunes

iTunes is an outrageously bad music player, but at least it has one redeeming quality of being controllable from command line via AppleScript.

By issuing silly commands like

tell application iTunes
 next track
end tell

we can control its basic functionality, what will be good enough for us.

We also need to get some information from iTunes. It doesn't have "increase/decrease volume" commands, so we need to find current volume, and then set it to higher/lower values. It also doesn't have a single play/pause command, so we need to figure out what state we're in - if we're playing then pause, otherwise play. Some of the nasty code refactored to OSA class to keep ITunes class a bit cleaner, but not by much.

The code doesn't depend on the rest of the project, so you can use it in your own projects, or just look what it's doing and use it directly from command line.

class OSA
 def get(var)
 `osascript -e 'tell application "#{name}" to #{var}'`
 end
 def do!(cmd)
 system 'osascript', '-e', %Q[tell application "#{name}"], '-e', cmd, '-e', 'end tell'
 end
end

class ITunes < OSA
 def name
 'iTunes'
 end
 def get_volume
 get('sound volume as integer').to_i
 end
 def get_state
 get('player state as string').chomp
 end
 def set_volume(v)
 system 'osascript', '-e', %Q[tell application "iTunes" to set sound volume to #{v}]
 end
 def next!
 do! 'next track'
 end
 def prev!
 do! 'previous track'
 end
 def pause!
 do! 'pause'
 end
 def play!
 do! 'play'
 end
 def pause_flip!
 if get_state == 'playing'
 pause!
 else
 play!
 end
 end
 def vol_up!
 set_volume(get_volume + 5)
 end
 def vol_down!
 set_volume(get_volume - 5)
 end
end

Reading data from Arduino

Communication is strictly uni-directional. Arduino writes to virtual serial interface, and we're reading from it as the data arrives. The data is current position of buttons, analog stick, and accelerometer. We're not really interested in any of it directly - we care about button presses and releases, and about stick going up/down/left/right (analog stick is used only as fake D-pad here). So change of state, not state as such.

For buttons it's straight forward. We get a bit for each button, so we just check it. For the stick, there's a small trickery involved. The range on each axis is from about -100 to +100, so we could just set points like +64/-64 from which we count it as a up/down. But what if stick is hold around that level? Sensor is analog (and user's hand is shaking), so if user hold the stick around +64, the reading would go +64, +63, +65, +63, +64, +64, +65 ..., what such naive implementation would treat as plenty of up presses, even though there weren't any.

This is an extremely common problem in electronics, and there's a simple solution - hysteresis. We simply take two slightly different levels for two sides of the same transition. If stick goes above +64 it's up. If it goes below +48 it's neutral. If it's betweer +48 and +64, we just keep it where it was. So unless the user has Parkinsons's and wiggles it a lot, it won't register any spurious clicks.

Other than that the code is pretty straightforward. We throw away first five readings because electronics tend to take some time after power-up to stabilize. And with every reading we set state to what we read, put all transitions into @cur_events instance variable, and yield.

require 'rubygems'
require 'serialport'

class SerialDevice
 def get_device
 while true
 dev = Dir["/dev/cu.usb*"][0]
 return dev if dev
 sleep 1
 end
 end
 def initialize
 @sp = SerialPort.new(get_device, 19200)
 end
end

class Nunchuck < SerialDevice
 attr_reader :cur_events, :b, :sx, :sy, :sxs, :sys
 attr_accessor :ax, :ay, :az
 def initialize
 super
 @b,@sx,@sy,@ax,@ay,@az,@sxs,@sxs = 0,0,0,0,0,0,0,0
 end
 def each_state
 skip = 5
 while line = @sp.readline
 unless line =~ /B=(\d) XY=(\S+),(\S+) XYZ=(\S+),(\S+),(\S+)/
 puts line
 next
 end
 if skip > 0
 skip -= 1
 next
 end
 @cur_events = []
 self.b, self.sx, self.sy, self.ax, self.ay, self.az = [$1,$2,$3,$4,$5,$6].map{|x| x.to_i}
 yield
 end
 end
 def b=(nv)
 @cur_events << :press_lo if nv & ~@b & 1 != 0
 @cur_events << :release_lo if @b & ~nv & 1 != 0
 @cur_events << :press_hi if nv & ~@b & 2 != 0
 @cur_events << :release_hi if @b & ~nv & 2 != 0
 @b = nv
 end
 def hysteresis(new_measurement, old_state, *ranges)
 while true
 state, bounds = ranges.shift, ranges.shift
 if bounds.nil? or
 (new_measurement < bounds.first) or
 (new_measurement < bounds.last and old_state <= state)
 return state
 end
 end
 end
 def sxs=(nv)
 @cur_events << :sxs_change if nv != @sxs
 @sxs = nv
 end
 def sys=(nv)
 @cur_events << :sys_change if nv != @sys
 @sys = nv
 end
 def sx=(nv)
 self.sxs = hysteresis(nv, sxs, -1, -64..-48, 0, 48..64, 1)
 @sx = nv
 end
 def sy=(nv)
 self.sys = hysteresis(nv, sys, -1, -64..-48, 0, 48..64, 1)
 @sy = nv
 end
 def raw
 [@b,@sx,@sy,@ax,@ay,@az]
 end
end

All of it together

The final bit of code is trivial. We just instantiate ITunes and Nunchuck objects, and connect them in obvious way:

Press C to play/pause
Analog stick right/left for next/previous
Analog stick up/down to increase/decrease volume

This style of volume control sucks, but if I tried to run two osascript commands on every reading (if stick is above +64 increase volume, if under -64 decrease volume), it would be too slow, at least in this naive implementation, and you'd only see there would be a pretty big lag between moving the stick and iTunes reacting.

i = ITunes.new
n = Nunchuck.new
n.each_state do
 if n.cur_events.include?(:sxs_change)
 if n.sxs == 1
 i.next!
 elsif n.sxs == -1
 i.prev!
 end
 end
 if n.cur_events.include?(:sys_change)
 if n.sys == 1
 i.vol_up!
 elsif n.sys == -1
 i.vol_down!
 end
 end
 if n.cur_events.include?(:press_hi)
 i.pause_flip!
 end
 p(n.raw + n.cur_events)
end