Sun Storage Blogs

Deirdre' Straughan: Lots and Lots of Storage Video

Deirdre' Straughan — Fri, 20 Nov 2009 02:42:54 -0800

Storage (and other) videos - technical talks, presentations, interviews - have gone to live on blogs.sun.com/video. There's a tag cloud to help you find what you're looking for, and don't miss the videos that have been translated into português-brasil, español, русский 日本語简体中文

bobp: IT Infrastructure Products, Partnerships and Paraphernalia

bobp — Wed, 18 Nov 2009 18:08:08 -0800

Announcements have certainly accelerated lately with respect to integrated infrastructure technology. Multiple vendors who provide infrastructure hardware for compute, storage and network are either partnering with each other or pursuing a "go alone" strategy. Few will disagree that more tightly integrated solutions will benefit IT customers.

However an IT solution has always had the expectation and requirement to be integrated. That is why the IT industry has developed over the years standards and protocols. In other words, common industry accepted methods to move, process and protect business as well as consumer information. Traditionally large complex IT solutions have been integrated by VARs, management consulting firms or consulting services for a given customer. Is that beginning to change? Also the IT solution is comprised of more than just tightly integrated hardware blocks.

Let's not forget about the various applications that need to run on any given vendors hardware platform. In addition to the applications, virtualization is becoming a standard requirement to maximize the utilization of any vendors integrated hardware platform. Infrastructure providers will need to adjust to the fact that hypervisor technology may sell less hardware because utilization and efficiency will be largely improved.

Does this new focus on consolidation favor software stacks that are both heterogeneous and therefore ubiquitous? The IT industry has become a mature business. New ideas will always continue but IT will still be comprised of both hardware and software.

A good analogy is the automobile industry... Automobiles have had tremendous amounts of progress over the past 100 years. For example there have been many optimizations and efficiencies with car manufacturing, car fuel efficiency, etc. Over the past 20 years the automobile industry has highly leveraged the use of embedded electronics in cars. But the basic components of a car (tires, engine, brakes, steering wheel, etc.) have (and for the foreseeable future) not changed.

Blog is available also at: http://bobporras.wordpress.com/

dom: ZFS HSP Demo

dom — Wed, 18 Nov 2009 01:57:51 -0800

Here is a demo of the use of ZFS Hybrid Storage Pools that I put together. Its quite neat in that it is reproduceable fairly easily. My thanks to Mo Beik for doing the heavy lifting.

dom: Open Storage Demos

dom — Wed, 18 Nov 2009 01:37:58 -0800

Here is series of Brief Open Storage Demo's that I put together on various aspects of Solaris storage software such as ZFS, NFS, CIFS, iSCSI - just the basics to get you started.

I originally created them for SuperComputer 2008 and have only just rediscovered them. Enjoy.

timf: New zfs-auto-snapshot implementation on it's way

timf — Mon, 16 Nov 2009 19:09:35 -0800

Niall wrote a post to the zfs-auto-snapshot alias announcing his new time-sliderd implementation of the ZFS Automatic Snapshot service.

I'm looking forward to this new implementation: I wrote the old ksh-based code back in 2006 and have been adding features & fixing bugs ever since. Over time, it's started creaking at the seams - there were a few issues with it that were tricky to deal with in it's existing implementation. I've long felt the desire to start again, but just couldn't give it the time it needed. Well, Niall's done just that - many thanks Niall!

I've got commentary on the thread on the auto snapshot mailing list and have also forwarded Niall's announcement to zfs-discuss. The old README on this blog has been updated, with a pointer to the original heads-up message.

So now I get to focus on my day-job again :-)

mac: DMTF shows it's Cloud hand

mac — Mon, 16 Nov 2009 00:34:39 -0800

The DMTF has released a white paper titled: Interoperable Clouds. From the abstract: "This white paper describes a snapshot of the work being done in the DMTF Open Cloud Standards Incubator, including use cases and reference architecture as they relate to the interfaces between a cloud service provider and a cloud service consumer. The goal of the Incubator is to define a set of architectural semantics that unify the interoperable management of enterprise and cloud computing. This paper summarizes the core use cases, reference architecture, and service lifecycle. These building blocks will be used to specify the cloud provider interfaces, data artifacts, and profiles to achieve interoperable management."
The paper lays out several great models for cloud services, and I encourage everyone to read the whole whitepaper. Here is a picture from chapter 5 on the Cloud Service Reference Architecture:

The group has all the right players participating, including: AMD, CA, Cisco, Citrix, EMC, Fujitsu, HP, Hitachi, IBM, Intel, Microsoft, Novell, Rackspace, RedHat, Savvis, SunGard, Sun Microsystems, and VMware. The white paper is just the first milestone in the group's work. Chapter 6 lays out the next steps for the group, including future Alliances work.

: Fun With DTrace: The Windows-Key Prank

Mon, 16 Nov 2009 08:32:14 -0800

The current episode of the German HELDENFunk podcast features an interview with Chris Gerhard about one of his favourite subjects: DTrace (in English, beginning at 14:58):

After the interview, we hear a guy called "Konteener Kalle" express his love (in German) for DTrace by playing a prank on his boss: Whenever he presses the Windows key (on an OpenSolaris system, mind you), he's punished by watching the XScreensaver BSOD hack (of course not knowing that it's just a screensaver).

That little joke challenged me to actually implement this prank. Here's how to do it.

The Idea

The idea of this prank is to start the XScreensaver Blue-Screen-of-Death screensaver (which simulates a Windows crash experience) on an OpenSolaris system whenever the user presses a certain key a certain number of times. This could be the Windows-Key (which doesn't have any real use on an OpenSolaris machine) or any other key. We count the number of key presses and only execute the BSOD after a certain number of key presses in order to make the prank less obvious.

Step 1: Identify the Windows (or any other) Key

If you have a Windows-Keyboard, this is easy: Run xev and press the Windows-Key. Take note of the keycode displayed in the xev output. Of course you can use any other key as well to play this prank. In this case, I'm using the left Control-Key, because I don't have a Windows-Key on the system I'm working on. The Control key has the keycode 37.

Step 2: Configure XScreensaver for BSOD

XScreensaver comes with a great collection of "hacks" that do interesting stuff on the screen when the screensaver activates. Check out the /usr/lib/xscreensaver/hacks directory. Each hack can be run individually, but then it will only execute inside a new window. For the BSOD illusion to be realistic, we want to execute the BSOD hack in full-screen.

This can be achieved by telling XScreensaver to demo the BSOD hack for us. It will then create a full-screen window and execute the BSOD hack inside the new window. The following command will tell XScreensaver to run a hack for us:

xscreensaver-command -demo <number>

The <number> part is a little complicated: XScreensaver looks at its config file ~/.xscreensaver where it stores a list of programs and arguments after the keyword "programs:". <number> simply refers to the number of the hack on that list. Therefore, we must create an entry in our admin user's .xscreensaver file that starts bsod(6) with the right parameters and that gives us a known number to call xscreensaver-command with.

Let's put our entry at the top of the list so we can simply use the number "1" to execute the BSOD screensaver. Somewhere in our .xscreensaver, the programs section should look like this:

 ... textFile: /etc/motd textProgram: date textURL: http://blogs.sun.com/roller/rss programs: \ - "BSOD Windoze" bsod -root -only nt \n\ - "Qix (solid)" qix -root -solid -segments 100 \n\ - "Qix (transparent)" qix -root -count 4 -solid -transparent \n\ ...

You can test this by running xscreensaver-command -demo 1.

Step 3: Write a DTrace Script That Sets Up the Trap

Now it gets more interesting. How do we use DTrace to find out when a user presses a certain key? All we know is that the Xorg server processes the keystrokes for us. So let's start by watching Xorg in action. The following DTrace command will trace all function calls within Xorg:

pfexec dtrace -n pid`pgrep Xorg`:::entry'{ @func[probefunc] = count(); }'

Let's start it, press the desired key 10 times, then stop it with CTRL-C. You'll see a long list of Xorg functions, sorted by the number of times they've been called. Since we pressed the key 10 times, it's a good idea to look for functions that have been called ca. 10 times. And here, we seem to be lucky:

 ... miUnionO 8 DeviceFocusInEvents 9 CommonAncestor 10 ComputeFreezes 10 CoreLeaveNotifies 10 key_is_down 11 FreeScratchPixmapHeader 12 GetScratchPixmapHeader 12 LookupIDByType 12 ProcShmDispatch 12 ProcShmPutImage 12 ...

The key_is_down function looks like exactly the function we're looking for! In fact, some googling tells us that this function's 2nd argument is the keycode of the key that is down when the function is called.

Why do we see "11" and not "10" function calls to key_is_down? Because it also counted my pressing of the Ctrl-Key when I stopped the DTrace script through Ctrl-C :).

This gives us enough knowledge to create the following DTrace script:

 #!/usr/sbin/dtrace -s /* * BSODKey.d */ /* * This D script will monitor a certain key in the system. When this key is * pressed, a shell script will be executed that simulates a BSOD. * * The script needs the process id of the Xorg server to tap into as its * first argument. * * One example of using this script is to punish a user pressing the * Windows key on an OpenSolaris system by launching the BSOD screen saver. */ #pragma D option quiet #pragma D option destructive BEGIN { ctrlcount = 0; keycode=37 } pid$1::key_is_down:entry /arg1 == keycode/ { ctrlcount ++; } pid$1::key_is_down:return /ctrlcount == 10/ { ctrlcount = 0; system("/usr/bin/xscreensaver-command -demo 1"); }

First, we need to enable DTrace's destructive mode (ever heard of a "constructive prank"?) otherwise we can't call the system-command at the end. The script uses the pid provider to tap into Xorg. Therefore, we need to give it the PID of the Xorg server as an argument:

pfexec ./BSODKey.d `pgrep Xorg`

It then sets up a probe that fires whenever key_is_down is called with our keycode and counts the key presses. At the end of the key_is_down function call, it checks whether we reached 10 keypresses, then executes the BSOD screen saver and resets the counter. You may need to make sure that the DISPLAY variable is set correctly for the BSOD program to show up on the victim's screen when starting this script.

After hitting the Control-Key 10 times, we're rewarded with our beloved BSOD:

Conclusion

That wasn't too difficult, was it? Yes, one could have done the same thing by writing a regular script that taps into /dev/kbd or something similar. But the beauty of DTrace lies in the simplicity of this script (Tap into the right function while it's running) and in the fact that it now can be modified very easily to fire BSODs at any kind of event, including the user hitting a certain area of the screen with his mouse or selecting a particular text or whatever you choose it to be.

So, have fun with this script and let me know in the comments what kind of pranks (or helpful actions) you can imagine with DTrace!

jmcp: Migrating from Solaris Express to OpenSolaris

jmcp — Thu, 12 Nov 2009 14:54:53 -0800

There's currently no way to do an in-place upgrade0 from Solaris Express's SysV packaging to OpenSolaris' IPS packaging, so you have to think outside the box just a little.

My Ultra 40 M2 has been happily chugging away with SXCE builds since I took delivery of it, but with build 131 fast approaching (when we start delivering a nightly IPS repo rather than SysV packages), I figured I should put some effort into migrating to the new world. Fortunately for me, I've been running with ZFS root since it was first available (build 80 or so), and when I reinstalled my laptop last year I put OpenSolaris on it. It's been upgraded to snv_126 in the last week, too.

Here's what I did.

After making sure I had enough space in my u40m2 ("blinder") root pool, I created a zfs snapshot of the current BE on the laptop ("gedanken"). Then destroyed it, pruned sundry filesystems and a bunch of packages which I don't need (eg, I have no use for almost all the localisations), and re-created the snapshot.

gedanken# zfs snapshot rpool/ROOT/opensolaris-7@yay
blinder# zfs create rpool/ROOT/opensolaris-7

Then I sent it from gedanken to blinder:

gedanken# zfs send rpool/ROOT/opensolaris-7@yay | \ ssh blinder zfs recv -v rpool/ROOT/opensolaris-7 [trundle] received 20.8GB stream in 2707 seconds (7.85MB/sec)

Now the action switches to blinder:

(add entry to grub menu.lst, remember to set the default)

# zpool set boots=rpool/ROOT/opensolaris-7 rpool
# zfs set canmount=noauto rpool/ROOT/opensolaris-7
# zfs set mountpoint=/ rpool/ROOT/opensolaris-7

Time for some customisations on the new pool:

# zfs set mountpoint=/mnt rpool/ROOT/opensolaris-7
# cp /etc/ssh/sshd*key* /mnt/etc/ssh
# cp /etc/hostid /mnt/etc/hostid
# cp /etc/inet/hosts /mnt/etc/inet/hosts
# cp /etc/X11/xorg.conf /mnt/etc/X11
# cp /etc/hostname.nge0 /mnt/etc
# cp /var/spool/cron/cronjobs/onhg /mnt/var/spool/cron/cronjobs

I think that's it, I really should have run this from within script(1)!

# cd /
# zfs umount rpool/ROOT/opensolaris-7
# zfs set mountpoint=/ rpool/ROOT/opensolaris-7

Time for the acid test!

# init 6

[dammit, this "fast reboot" stuff is TOO FAST!]

root@blinder:~# cat /etc/release OpenSolaris Development snv_127 X86 Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Use is subject to license terms. Assembled 06 November 2009
root@blinder:~# root@blinder:~# uname -a
SunOS blinder 5.11 snv_127 i86pc i386 i86pc
root@blinder:~#

and X came up just fine.

After making sure that all the bits that I expected were there, I was very pleased to call this exercise a success.

The only bit that remains to be done is configuring my non-global zone but I'll leave that for another post.

bobp: A Time Capsule for the Enterprise?

bobp — Wed, 04 Nov 2009 18:08:08 -0800

On Halloween I upgraded the desk side PC from Windows XP to Windows 7. Between multiple reboots, application installs, recovery of email, print server configuration, etc. my wife asked "Why does it have to be so hard?" As she said this I was looking at the Apple Time Capsule sitting on top of a small cabinet in our home basement. The only item coming out of the Time Capsule was the power cord connected to the power outlet. I run the Time Capsule as a wireless client in our home network for data backup. The data consists of many pictures, video clips, songs, the kids homework and basically many various files (some of which are important).

My home (LAN) network has grown over the years from a few PCs connected via an old 802.11b router to a dual band (802.11g/802.11n) router connecting a multitude of wireless clients. These include a XBox, iTouch, PrintServer, OpenSolaris, Ubuntu, Mac, Windows XP, Windows Vista and Windows 7 devices. Basically a heterogeneous environment. Will the older Window versions be phased out at home eventually? Yes. The Unix/Linux clients will remain out of necessity as well as for hobby. It's difficult to currently beat the Mac experience. Adding a Time capsule into my existing home network was relatively easy. The Airport setup updated the Time Capsule firmware and configured the device in straight forward steps. It wasn't as "fun" for a "techie" as say CLI commands, but simple is defined as "fun" for most consumers. The Windows XP to Windows 7 full upgrade was painfull but I have to admit the network configuration experience was much improved. I was pleasantly surprised freeware Bonjour discovery services just worked and the W7 system configured the Time Capsule as a usable share.

One could imply a similar situation in the enterprise space. For an enterprise business "fun" is defined as high margin dollars on a growing revenue stream. This usually means your costs are contained, you implement continuous improvements on efficiency and you simplify. Vendors in the technology industry are all trying to provide a truly "Enterprise Time Capsule" or Appliance. There has been a large amount of innovation over the past 20 years. However today's innovation is tomorrow's PDP-11. Minicomputers were appliances that Mainframes couldn't be. The evolution has continued over the years in every technology segment.

Enterprise customers want it simple as do consumers. While the stakes are much higher in the enterprise, the bar is significantly raised for "just working" each and every time. It doesn't matter if you are playing catchup, you are the incumbent or you are the new thought leader-- the winners will be the set of vendors who provide the tool that "just works" each and every time in the harshest and most complex environments.

Blog is available also at: http://bobporras.wordpress.com/

Walter Moriconi: SAS advanced connectivity roadmap

Walter Moriconi — Wed, 04 Nov 2009 06:37:51 -0800

...

As an important part of the announcement, STA is promoting a converged
Mini-SAS High-Density (HD) standard connector to create a vibrant and
consistent infrastructure to expand SAS usability and scalability. The
STA objectives include a consistent connectivity management scheme in
addition to the active copper and optical cable extensions supported by
the new Mini-SAS HD connector....

source SNS Europe

Walter Moriconi: SNIA Announces SMI-S Conformance Test Enhancements in Version 1.4

Walter Moriconi — Wed, 04 Nov 2009 06:37:03 -0800

...

"As the SMI specification continues to evolve and become integrated
into an increasing number of storage products, SMI has focused on
increasing the functionality of CTP for end users," said Paul von
Behren, chair of the Storage Management Initiative. "Version 1.4 now
assists end users to better evaluate their storage purchase options by
helping them determine whether a product simply provides basic
discovery and monitoring or whether it allows for configuration
management via SMI-S."

CTP for SMI-S 1.4 features:
·   Easy-to-understand CTP reports
·   Delivery of event notifications
·   Identification of tested management tasks using business-oriented terms

....

source SNS Europe

Walter Moriconi: LSI Launches High-Performance 6Gb/s SAS Host Bus Adapters

Walter Moriconi — Wed, 04 Nov 2009 06:36:09 -0800

LSI has announced availability of a new family of high-performance
SATA+SAS host bus adapters (HBAs) based on 6Gb/s SAStechnology. The
LSI™ SAS 9200 series HBAs, based on the LSISAS2008 6Gb/s SAS controller
IC, offer new levels of performance and scalability for internal and
external direct-attached storage deployments using SATA and SAS disk
drives.

source SNS Europe

Walter Moriconi: Emulex Announces Support for SAS and SSDs with Next Generation InSpeed Bridging Technology

Walter Moriconi — Wed, 04 Nov 2009 06:34:16 -0800

Emulex is expanding the performance and versatility of its
market-leading InSpeed® Bridging technology through the addition of
enterprise storage system support for Serial Attached SCSI (SAS) to
Serial Advanced Technology Attachment (SATA) bridging in SAS
environments, support for SAS Hard Disk Drives (HDDs) in Fibre Channel
environments, and support for ultra-high performance Solid State Drives
(SSDs) in both SAS and Fibre Channel environments.

source SNS Europe

Walter Moriconi: FCIA Announces Milestone towards Completing New 16Gb/s Fibre Channel Standard

Walter Moriconi — Wed, 04 Nov 2009 06:23:48 -0800

ANSI INCITS T11 Committee Completes Work on 16GFC.

The Fibre Channel Industry Association (FCIA) says that the ANSI INCITS
T11 Committee completed the technical work on the FC-PI-5 for 16Gb/s
Fibre Channel ("16GFC" blogs.sun.com/images/smileys/wink.gif" class="smiley" alt=";)" title=";)" /> and voted in early October to send the document
out for letter ballot. This milestone marks the technical stability
and completeness necessary for vendors to commit to silicon their
upcoming designs based upon the FC-PI-5 standard.

...

source SNS Europe

mac: LISA BOF - OpenSolaris, Storage and the Cloud

mac — Tue, 03 Nov 2009 07:38:55 -0800

I am giving LISA BOF today at USENIX LISA here in Baltimore on OpenSolaris availability in the Cloud (EC2 instances mostly) as well as an update on the Cloud Storage standards efforts Sun is leading in the SNIA.
You can download my slides: OpenSolarisCloud.pdf if you like or view them in SlideShare.
Deirdre will be videoing the presentation, so I'll update this entry later to point to that. There are a number of great BOFs following mine, so stick around and attend those as well.

bonwick: ZFS Deduplication

bonwick — Sun, 01 Nov 2009 21:14:18 -0800

You knew this day was coming: ZFS now has built-in deduplication.

If you already know what dedup is and why you want it, you can skip the next couple of sections. For everyone else, let's start with a little background.

What is it?

Deduplication is the process of eliminating duplicate copies of data. Dedup is generally either file-level, block-level, or byte-level. Chunks of data -- files, blocks, or byte ranges -- are checksummed using some hash function that uniquely identifies data with very high probability. When using a secure hash like SHA256, the probability of a hash collision is about 2^-256 = 10^-77 or, in more familiar notation, 0.00000000000000000000000000000000000000000000000000000000000000000000000000001. For reference, this is 50 orders of magnitude less likely than an undetected, uncorrected ECC memory error on the most reliable hardware you can buy.

Chunks of data are remembered in a table of some sort that maps the data's checksum to its storage location and reference count. When you store another copy of existing data, instead of allocating new space on disk, the dedup code just increments the reference count on the existing data. When data is highly replicated, which is typical of backup servers, virtual machine images, and source code repositories, deduplication can reduce space consumption not just by percentages, but by multiples.

What to dedup: Files, blocks, or bytes?

Data can be deduplicated at the level of files, blocks, or bytes.

File-level assigns a hash signature to an entire file. File-level dedup has the lowest overhead when the natural granularity of data duplication is whole files, but it also has significant limitations: any change to any block in the file requires recomputing the checksum of the whole file, which means that if even one block changes, any space savings is lost because the two versions of the file are no longer identical. This is fine when the expected workload is something like JPEG or MPEG files, but is completely ineffective when managing things like virtual machine images, which are mostly identical but differ in a few blocks.

Block-level dedup has somewhat higher overhead than file-level dedup when whole files are duplicated, but unlike file-level dedup, it handles block-level data such as virtual machine images extremely well. Most of a VM image is duplicated data -- namely, a copy of the guest operating system -- but some blocks are unique to each VM. With block-level dedup, only the blocks that are unique to each VM consume additional storage space. All other blocks are shared.

Byte-level dedup is in principle the most general, but it is also the most costly because the dedup code must compute 'anchor points' to determine where the regions of duplicated vs. unique data begin and end. Nevertheless, this approach is ideal for certain mail servers, in which an attachment may appear many times but not necessary be block-aligned in each user's inbox. This type of deduplication is generally best left to the application (e.g. Exchange server), because the application understands the data it's managing and can easily eliminate duplicates internally rather than relying on the storage system to find them after the fact.

ZFS provides block-level deduplication because this is the finest granularity that makes sense for a general-purpose storage system. Block-level dedup also maps naturally to ZFS's 256-bit block checksums, which provide unique block signatures for all blocks in a storage pool as long as the checksum function is cryptographically strong (e.g. SHA256).

When to dedup: now or later?

In addition to the file/block/byte-level distinction described above, deduplication can be either synchronous (aka real-time or in-line) or asynchronous (aka batch or off-line). In synchronous dedup, duplicates are eliminated as they appear. In asynchronous dedup, duplicates are stored on disk and eliminated later (e.g. at night). Asynchronous dedup is typically employed on storage systems that have limited CPU power and/or limited multithreading to minimize the impact on daytime performance. Given sufficient computing power, synchronous dedup is preferable because it never wastes space and never does needless disk writes of already-existing data.

ZFS deduplication is synchronous. ZFS assumes a highly multithreaded operating system (Solaris) and a hardware environment in which CPU cycles (GHz times cores times sockets) are proliferating much faster than I/O. This has been the general trend for the last twenty years, and the underlying physics suggests that it will continue.

How do I use it?

Ah, finally, the part you've really been waiting for.

If you have a storage pool named 'tank' and you want to use dedup, just type this:

zfs set dedup=on tank

That's it.

Like all zfs properties, the 'dedup' property follows the usual rules for ZFS dataset property inheritance. Thus, even though deduplication has pool-wide scope, you can opt in or opt out on a per-dataset basis.

What are the tradeoffs?

It all depends on your data.

If your data doesn't contain any duplicates, enabling dedup will add overhead (a more CPU-intensive checksum and on-disk dedup table entries) without providing any benefit. If your data does contain duplicates, enabling dedup will both save space and increase performance. The space savings are obvious; the performance improvement is due to the elimination of disk writes when storing duplicate data, plus the reduced memory footprint due to many applications sharing the same pages of memory.

Most storage environments contain a mix of data that is mostly unique and data that is mostly replicated. ZFS deduplication is per-dataset, which means you can selectively enable dedup only where it is likely to help. For example, suppose you have a storage pool containing home directories, virtual machine images, and source code repositories. You might choose to enable dedup follows:

zfs set dedup=off tank/home

zfs set dedup=on tank/vm

zfs set dedup=on tank/src

Trust or verify?

If you accept the mathematical claim that a secure hash like SHA256 has only a 2^-256 probability of producing the same output given two different inputs, then it is reasonable to assume that when two blocks have the same checksum, they are in fact the same block. You can trust the hash. An enormous amount of the world's commerce operates on this assumption, including your daily credit card transactions. However, if this makes you uneasy, that's OK: ZFS provies a 'verify' option that performs a full comparison of every incoming block with any alleged duplicate to ensure that they really are the same, and ZFS resolves the conflict if not. To enable this variant of dedup, just specify 'verify' instead of 'on':

zfs set dedup=verify tank

Selecting a checksum

Given the ability to detect hash collisions as described above, it is possible to use much weaker (but faster) hash functions in combination with the 'verify' option to provide faster dedup. ZFS offers this option for the fletcher4 checksum, which is quite fast:

zfs set dedup=fletcher4,verify tank

The tradeoff is that unlike SHA256, fletcher4 is not a pseudo-random hash function, and therefore cannot be trusted not to collide. It is therefore only suitable for dedup when combined with the 'verify' option, which detects and resolves hash collisions. On systems with a very high data ingest rate of largely duplicate data, this may provide better overall performance than a secure hash without collision verification.

Unfortunately, because there are so many variables that affect performance, I cannot offer any absolute guidance on which is better. However, if you are willing to make the investment to experiment with different checksum/verify options on your data, the payoff may be substantial. Otherwise, just stick with the default provided by setting dedup=on; it's cryptograhically strong and it's still pretty fast.

Scalability and performance

Most dedup solutions only work on a limited amount of data -- a handful of terabytes -- because they require their dedup tables to be resident in memory.

ZFS places no restrictions on your ability to dedup. You can dedup a petabyte if you're so inclined. The performace of ZFS dedup will follow the obvious trajectory: it will be fastest when the DDTs (dedup tables) fit in memory, a little slower when they spill over into the L2ARC, and much slower when they have to be read from disk. The topic of dedup performance could easily fill many blog entries -- and it will over time -- but the point I want to emphasize here is that there are no limits in ZFS dedup. ZFS dedup scales to any capacity on any platform, even a laptop; it just goes faster as you give it more hardware.

Acknowledgements

Bill Moore and I developed the first dedup prototype in two very intense days in December 2008. Mark Maybee and Matt Ahrens helped us navigate the interactions of this mostly-SPA code change with the ARC and DMU. Our initial prototype was quite primitive: it didn't support gang blocks, ditto blocks, out-of-space, and various other real-world conditions. However, it confirmed that the basic approach we'd been planning for several years was sound: namely, to use the 256-bit block checksums in ZFS as hash signatures for dedup.

Over the next several months Bill and I tag-teamed the work so that at least one of us could make forward progress while the other dealt with some random interrupt of the day.

As we approached the end game, Matt Ahrens and Adam Leventhal developed several optimizations for the ZAP to minimize DDT space consumption both on disk and in memory, key factors in dedup performance. George Wilson stepped in to help with, well, just about everything, as he always does.

For final code review George and I flew to Colorado where many folks generously lent their time and expertise: Mark Maybee, Neil Perrin, Lori Alt, Eric Taylor, and Tim Haley.

Our test team, led by Robin Guo, pounded on the code and made a couple of great finds -- which were actually latent bugs exposed by some new, tighter ASSERTs in the dedup code.

My family (Cathy, Andrew, David, and Galen) demonstrated enormous patience as the project became all-consuming for the last few months. On more than one occasion one of the kids has asked whether we can do something and then immediately followed their own question with, "Let me guess: after dedup is done."

Well, kids, dedup is done. We're going to have some fun now.

timf: Halloween 2009

timf — Sat, 31 Oct 2009 04:43:46 -0700

Happy Z-Day everyone!

Breaking with tradition somewhat, I'm not sure there's going to be any fireworks photos here this year: we're down at my parents house, and are less likely to get the sort of sustained shelling that we normally experience in Raheny each year.

On the plus side, we had homemade pumpkin soup for lunch, so was at least able to get this shot. If there's any fireworks later on, I'll update this post - but otherwise, here's some Halloween cheer:

The day's been great so far - lovely birthday presents (a Merino base-layer and a copy of Neverwhere from the lovely missus, and DVDs of the first two Ice Age films from E (I think she had an ulterior motive there!) and a nice fleece from my folks)

I also popped out for a birthday run around Greystones this afternoon, only 10k but it was enough for me to realise I'm far from being back in running form: the recovery is going to last a few more weeks I think!

My folks are baby-sitting tonight, so myself and the missus get to go out for a grown-up dinner, which I'm really looking forward to. Happy Halloween!

Update: There were fireworks after all, here's a few shots: fantastic, the tradition continues!

timf: 26.2 miles

timf — Mon, 26 Oct 2009 23:21:54 -0700

I ran the Dublin City Marathon yesterday in approx. 3h 30m (more on that later) - that's my first marathon, but I suspect not my last: to anyone even half-thinking of running 26.2 miles, you have to try it.

My motivation for running started several months back on a low note. The background was that I was increasingly working from home, with work being busy and having a desire to eat dinner with the family and be around to put the kids to bed, I was noticing that there were days where I wasn't leaving the house at all. At the same time, the rumours started about Sun being in talks with various companies about a possible acquisition.

When the Oracle deal was announced it made things worse - what had previously just been rumours in the press became a lot more believable. Usually when something like this is going on, I'll write my thoughts about it here: but doing so would have been unprofessional. The furthest I ever went was the occasional emoticon on my twitter feed.

The solution for both problems, I decided, was to get out in the mornings and run: exercise, and a little time to think.

As the months went by, I was running further and further, logging my progress to @timfoster as I went ,and generally having a whale of a time. One weekend, we had a few friends over for lunch - Kev, Nic, Mike & Maria. A bit of the conversation went something like this:

"That's great running you're doing Tim, you should do the marathon"

Well, that was it, I'd had vague thoughts of doing it one day, but hearing someone else say it out loud was enough to make me register that night - with some trepidation, I might add. The registration form grouped entrants into three categories in order of expected finishing time: 3h or less, 3:30-4:15, 4:15+. I had no clue where I belonged, so popped myself in the middle and got on with it - that was July 28th this year.

I figured I ought to be following some sort of formal training plan. The athletics forum on boards.ie was a great resource, and were pointing newbies like me to Hal Higdon's running site . I chose the Intermediate II schedule, as I figured I was already reasonably fit with the daily commute on bike from time to time, and all the running I'd done so far. I'd missed a few weeks at the beginning of the formal program, so worked out where I should be (which turned out to be a good place to start given the running I'd already done) and stuck to it.

As I got closer to the race, I was dutifully doing my Long Slow Runs each weekend and was coming up to running the last of the three 20 milers when I became anxious about what sort of pace I'd manage in the race - could I go fast over a long distance? I didn't know. I didn't want to risk burning myself out in the early stages of the race, and end up not finishing. I decided to push it, and do a Long Fast Run instead - you're not supposed to do this during training, I found out why. Yes, I discovered that I actually could manage a 7:15 min/mile pace over 20 miles, but I also managed to injure my leg in the process. To make matters worse, I'd miscalculated where I was joining the schedule, and it left me with only 2 weeks to taper before the race rather than 3, and most of those 2 weeks were spent just resting my leg rather than doing the suggested mileage.

With that, race day was upon me. I was aiming for a 3h15m finish - but given the last few weeks, this was probably unrealistic.

The atmosphere around the start was tense, but an amazing experience - very very well organised I thought: a record turn-out of 12,500 people this year.

After a lot of limbering up, and shedding of bin-liners, the starting gun fired. We moved very slowly at first, eventually getting past the starting line. The race started gently: I was in the middle of the 3:30-4:15 pen and first few miles were depressingly slow, difficult to overtake slower runners and I was already well down on my target. They say one of the most common mistakes by new marathon runners is starting too fast, so I kept repeating that to myself, and tried to stay calm.

By mile 5, I'd escaped the crowds in Phoenix Park and picked up the pace, passing Kev & Nic at mile 10 who were out cheering for me (thanks!!) and managed to keep pretty much on target till about mile 16 where I started slowing down, only to slow down further on miles 20/21 (the dreaded Roebuck/Foster Avenue hill) The missus & the kids, and Mum & Dad were cheering for me there, and I spotted my friend Barry too, who was marshaling for the race: familiar faces making the run a lot easier.

In general, the support from the crowd on the day was phenomenal: I really hope the people who got up early on an October Bank Holiday Monday appreciate what a difference them cheering really makes to runners - and, to the lady watching the race who gave me a jelly baby around Kimmage to whom I forgot to say "thankyou" - many many thanks, it was yummy, and much needed!

Things took a turn for the worse though around mile 22 - going down Nutley Lane, I felt a twinge in my thighs that I'd felt once before during training: the onset of cramp. I had to stop stretch/shake out my legs periodically, eat more jelly babies and start again. My times were tumbling now, and I watched in dismay as the 3:30 pacer balloons passed me on Merrion Road. For the rest of the race, I kept going as fast as I could manage, but it wasn't enough.

Rounding the final corner onto Pearse St. I went for it, eating my remaining sweeties and telling myself it'd soon be over. I crossed the line, and stopped my watch, which told me 3:30:46. I was tired but happy - I'd missed the 3:15 target, but there's always the next marathon.

As for official timing, I'm still a wee bit confused - the timing service that was tied to the chip on my bib told me I'd finished in 3:32:04, yet when I visit the results page on the Dublin Marathon website and search for my number (4871), at the time of writing, it confirms that chip time of 3:34:04, but tells me my finish time is 3:30:34. I know there's a difference between chip time and gun-time, but I'd always thought gun-time should be longer than chip-time (as it doesn't account for the time it takes you to actually reach the starting line, whereas chip time is registered from the moment you cross the start-line) Perhaps they got the numbers mixed up - 3:30:34 is closer to what was on my stopwatch. Anyway - it doesn't really matter.

I had a ball on my first marathon - I finished 1516^th out of a field of 12,500, which I'm happy about. I've got memories I'll never forget and a belief in myself that I never had before. Sure, I'm walking around like Boris Karloff today (legs rather sore) and am wishing our house didn't have stairs, but I'm extremely proud of what I've achieved. I think I'll keep running and would strongly recommend everyone to have a go at completing a marathon: it's a truly unforgettable experience.

Here's the route map, and the splits from my watch - I missed a few mile markers here & there, so just rolled up those times into a single row. As you can see, I lack consistency here - so that's something to work on for next time.

Mile	Time by my watch (min:sec)
1	8:02
2, 3, 4	26:12
5	7:55
6	7:40
7	7:21
8	7:06
9	7:02
10	7:10
11	7:06
12	7:36
13	7:33
14	7:16
15	7:33
16	7:50
17	7:40
18	7:47
19	8:02
20	8:31
21	8:41
22	7:57
23, 24	18:34 (yeah, cramps started about here)
25	9:33
26	8:33
0.2	1:52

rohrer: Join us at LISA (Large Installation System Administration Conference)

rohrer — Mon, 26 Oct 2009 10:39:20 -0700

The good folks at USENIX are hosting the 23rd annual LISA (Large Installation System Administration Conference) this year in Baltimore, Maryland November 1-6.

I hope you can join us there... We have quite a few Solaris topics and we'll be showcasing some of our leading OS technologies like Solaris Networking, ZFS, Solaris Cluster, our built-in virtualization technologies including LDOMs, Solaris Containers and the like.

Our Solaris System Administration Training and Certification folks will also be on hand with the latest updates on courses and certification paths.

You'll also want to meet Rick Ramsey, the creative force behind BigAdmin, the definitive source for Solaris and Sun system administrators. Get your latest tips and tricks here and earn BigAdmin bucks for submitting your content.

We'll have some of the latest Solaris administration books there to hand out to lucky folks who take our surveys -- Solaris System Administration Essentials, Solaris 10 Security Essentials, Solaris Internals and Solaris Performance and Tools as well as some of our OpenSolaris titles -- OpenSolaris Bible and Pro OpenSolaris.

On November 3 we'll also be hosting the Solaris Security Summit. Come hear our crack team of security experts and get the latest updates on Solaris security features. Register for this free event. Walk-ins will also be welcome.

Check out the complete list of Solaris activities including some great Solaris training hosted by USENIX.

We'll be in booths 100-104 in the Vendor Exhibit area... please drop by and introduce yourself.

: A Small and Energy-Efficient OpenSolaris Home Server

Sun, 25 Oct 2009 15:22:05 -0700

In an earlier entry, I outlined my most important requirements for an optimal OpenSolaris Home Server. It should:

Run OpenSolaris in order to fully leverage ZFS,
Support ECC memory, so data is protected at all times,
Be power-efficient, to help the environment and control costs,
Use a moderate amount of space and be quiet, for some extra WAF points.

So I went shopping and did some research on possible components. Here's what I came up with:

Choosing a Platform: AMD or Intel?

Disclosure: My wife works for AMD, so I may be slightly biased. But I think the following points are still very valid.

Intel is currently going through a significant change in architecture: The older Core 2 microarchitecture was based on the Front Side Bus (FSB), where the CPU connects to the Northbridge which contains the memory controller and connects to the memory, while also connecting to the Southbridge which connects to I/O.

Now, they are switching to the new Nehalem microarchitecture which has a memory-controller built into the CPU and a scalable I/O bus called Quickpath Interconnect that connects CPUs with other CPUs and/or IO.

Unfortunately, none of these architectures seem to support ECC memory at consumer level pricing. The cheapest Intel-based ECC-motherboard I could find had still more than double the cost of an AMD-based one. Even though the new Intel Core i7 series is based on Nehalem and thus could support ECC memory easily in theory, Intel somehow chose to not expose this feature. In addition, Core i7 CPUs are relatively new and there are not yet any power efficient versions available.

The Intel Atom processor series may be interesting for a home server from a pure power-saving perspective, but again, Atom motherboards don't support ECC and once your workload becomes a little more demanding (like transcoding or some heavier compiling), you'll miss the performance of a more powerful CPU.

AMD on the other hand has a number of attractive points for the home server builder:

AMD consumer CPUs use the same microarchitecture than their professional CPUs (currently, it's the K10 design). They only vary by number of cores, cache size, number of HT channels, TDP and frequency, which are all results of the manufacturing process. All other microarchitecture features are the same. When using an AMD consumer CPU, you essentially get a "smaller brother" of their high end CPUs.
This means you'll also get a built-in memory-controller that supports ECC.
This also means less chips to build a system (no Northbridge needed) and thus lower power-consumption.
AMD has been using the HyperTransport Interconnect for quite a while now. This is a fast, scaleable interconnect technology that has been on the market for quite a while so chipsets are widely available, proven and low-cost.

So it was no suprise that even low-cost AMD motherboards at EUR 60 or below are perfectly capable of supporting ECC memory which gives you an important server feature at economic cost.

My platform conclusion: Due to ECC support, low power consumption and good HyperTransport performance at low cost, AMD is an excellent platform for building a home server.

AMD Athlon II X2 240e: A Great Home Server CPU

While I was shopping around for AMD Athlon CPUs and just before I was about to decide for an AMD Athlon II X2 variant, AMD offered me one of their brand new AMD Athlon II X2 240e for testing, provided that I blogged about it. Thank you, AMD!

Introduced in October 20th, this CPU is part of the newest energy-efficient range of consumer CPUs from AMD. It has 2 cores (hence X2), snazzy 2.8 GHz and a 2 MB L2 cache. What's most important: The TDP for this CPU is only 45W, meaning that even under the highest stress, this CPU will never exceed 45W of power consumption. Including the memory controller. As you've guessed already, the "e" in the model number stands for "efficient".

There's an important trade-off to consider for home server CPUs: For instance, the AMD Phenom II series would have been more powerful because it has an additional L3 cache, but their TDP is at 65W minimum. While big caches (both with AMD and Intel) are good for compute-intensive operations and games, they can't help much in a home server context: Home servers spend most of their non-idle time transferring data from A to B (files, videos, music) and a cache doesn't help much here, cause it's just another stop between I/O and CPU to pass by. Transferred data hardly gets re-used.

Instead, for home servers, sacrificing the L3 cache for lower power consumption makes a lot of sense: You pay less for the CPU and you pay less for your power bill without sacrificing too much (if any) server relevant performance.

My CPU conclusion: For home servers, AMD Athlon II "e" series are perfect, because they save power and money and do the job very well. For games you might choose a more powerful Phenom II processor, which delivers better compute power at a slightly higher power bill.

Finding the Right Motherboard

After nailing the Platform and CPU question, I needed a motherboard. This can be a confusing process: For each CPU there are different chipsets, then there are different vendors offering motherboards based on these chipset, and then they offer different variants with different features. What should a good home server motherboard offer?

OpenSolaris support: Most motherboards "just work" with OpenSolaris, especially if they've been available for some time and/or use some well-known chipsets. To remove any doubts, consult the OpenSolaris Hardware Compatibility List.
ECC: Yes, AMD's ECC support is in the CPU, but just in case, we want to make sure that the motherboard exposes this feature to the user.
Integrated graphics: We only need a graphics card for installation, BIOS-settings/updates or debugging. The graphics shouldn't consume much power and doesn't need to deliver a lot of performance. Therefore, integrated graphics is just fine for a home server, and it saves some precious slot space, too.
Enough SATA ports: You need two ports for a mirrored root pool, another two for a mirrored data pool and then some more for hot-spare or if you want to spread your data pool across 4 or more disks. Many motherboards come with 6 SATA ports, which is ok.
Stability: If you have the choice between different variants, try to go for the "business" or "quality" version. It will likely have slightly better components and be better tuned towards 24/7 operation than the "performance", "gaming" or "overclocker" type boards which try to squeeze out more performance at the expense of durability.

Here's a very useful email thread on the OpenSolaris ZFS-discuss mailing list about CPU and motherboard options, pros and cons and user experiences. In this discussion, F.Wessels recommended the M3A78 series from Asus so I went for the M3A78-CM motherboard, which is their "business class" variant, priced at around 60 Euros and it has 6 SATA and 12(!) USB ports.

My motherboard conclusion: The Asus M3A78-CM motherboard has everything I need for a home server at a very low cost, and it's proven to run OpenSolaris just fine.

The Case: Antec NSK-1380

I won't go into much details about the case. My goal was to find one that can support at least 4 disks while being as compact as possible. The Antec NSK-1380 was the smallest case I could find that supports 4 disks. It comes with a built-in power supply, an extra fan, some features to help with keeping noise down and it looked ok for a PC case.

Miscellaneous Tips&Tricks

While putting everything together, I ran into some smaller issues here and there. Here's what I came up with to solve them:

CPU cooler: My CPU being a gift from AMD, it came without a cooler. I chose the Zalman CNPS 7000C AL-CU because it was the cheapest cooler that was less than 6cm high, which is a limit imposed by the case. Unfortunately, it collides with one of the 4 harddisk slots by a mere couple of millimeters, so I need to figure out how to best cut or bend the blades to make room for the disk without hurting the cooler too much. I'm not very familiar with PC CPU coolers, but I suspect that with 45W TDP one could even get away with a passive cooler. The Zalman came with a throttle circuit which I set to the minimum speed. This seems to be more than enough and it makes the system really silent, but I need some more thorough testing to confirm. Drop me a comment if you are familiar with passive cooling of 45W TDP CPUs.
Boot disks: We need something to boot our home server from, so boot disks are unavoidable. I say "disks", because they should always be mirrored. But other than providing a way of booting, they tend to get in the way, especially when space is a precious resource in a small case. Some people therefore boot from CF cards or other small flash media. This can be a nice solution, especially combined with an IDE-to-CF adapter, but consumer level flash media is either very slow (a typical OS install can take many hours!) or very expensive. While looking for alternatives, I found a nice solution: The Scythe Slot Rafter fits into an unused PCI slot (taking up the breadth of two) and provides space for mounting four 2.5" disks at just EUR 5. These disks are cheap, good enough and I had an unused one lying around anyway, so that was a perfect solution for me.
Extra NIC: The Asus M3A78-CM comes with a Realtek NIC and some people complained about driver issues with OpenSolaris. So I followed the advice on the aforementioned Email thread and bought an Intel NIC which is well supported, just in case.
USB boot: I couldn't get the M3A78-CM to boot from USB at all. I tried a USB stick and a USB disk and different boot settings in the BIOS to no avail. I gave up and built in a DVD-ROM drive from my old server just to install OpenSolaris, then built it out again. Jan Brosowski has the same motherboard and he found out that you need to update to the latest BIOS revision and use the USB port that's right below the Ethernet port for USB boot to work. YMMV :).

The Result

And now for the most important part: How much power does the system consume? I did some testing with one boot disk and 4GB of ECC RAM and measured about 45W idle. While stressing CPU cores, RAM and the disk with multiple instances of sysbench, I could not get the system to consume more than 80W. All in all, I'm very pleased with the numbers, which are about half of what my old system used to consume. I didn't do any detailed performance tests yet, but I can say that the system feels very responsive and compile runs just rush along the screen. CPU temperature won't go beyond the low 50Cs on a hot day, despite using the lowest fan speed, so cooling seems to work well, too.

I just started full 24/7 operation of my new home server this weekend, so I hope I'll have some more long-term experience about performance and stability in a few months. Meanwhile, I'm in the middle of configuring the system, installing some services and implementing a new way of managing my home server. But that's probably the topic of another blog post...

Do you agree with the home server conclusions I reached in this post? Or would you suggest alternatives? Do you have experiences to share with the mentioned components? Or do you have suggestions and tips on how to get the most out of them? Let me know by posting a comment here!

Many thanks go to Michael Schmid of AMD for sending me the AMD Athlon II X2 240e CPU.

bobp: Around Oracle Open World in less than 180 hours

bobp — Wed, 21 Oct 2009 19:08:08 -0700

The turnout of customers and partners of the enterprise technology segment certainly did not disappoint at Oracle Open World this year. While other large IT events have been canceled this year do to the economic downturn, Oracle Open World attendance of 42000 IT professionals was basically unaffected from the 2008 attendance. Even more impressive was that virtually every enterprise vendor that partners, competes and analyzes Oracle attended this yearly gathering in downtown San Francisco. The multiple exhibit halls, sessions, events, activities and networking certainly created an environment for plenty of information exchange.

Every vendor at Oracle Open World is a cog (of varying sizes from small to large) that builds into the enterprise IT stack of:

applications

middleware

database

Every item from storage management, computational speeds, networking feeds, disaster recovery, hosted IT, employee productivity tools and various communication mediums all factor back into connecting the above 3 areas of the enterprise stack.

In my opinion enterprise IT is becoming much less driven by vendor loyalty and a great price to the vendors that can provide competitive advantage to their customers. During an economic downturn as well as post recovery, the competitive advantage will more than outshine a good price.

Blog is available also at: http://bobporras.wordpress.com/

eschrock: Shadow Migration Internals

eschrock — Sun, 18 Oct 2009 14:49:40 -0700

In my previous entry, I described the overall architecture of shadow migration. This post will dive into the details of how it's actually implemented, and the motivation behind some of the original design decisions.

VFS interposition

A very early desire was that we wanted something that could migrate data from many different sources. And while ZFS is the primary filesystem for Solaris, we also wanted to allow for arbitrary local targets. For this reason, the shadow migration infrastructure is implemented entirely at the VFS (Virtual FileSystem) layer. At the kernel level, there is a new 'shadow' mountpoint option, which is the path to another filesystem on the system. The kernel has no notion of whether a source filesystem is local or remote, and doesn't differentiate between synchronous access and background migration. Any filesystem access, whether it is local or over some other protocol (CIFS, NFS, etc) will use the VFS interfaces and therefore be fed through our interposition layer.

The only other work the kernel does when mounting a shadow filesystem is check to see if the root directory is empty. If it is empty, we create a blank SUNWshadow extended attribute on the root directory. Once set, this will trigger all subsequent migration as long as the filesystem is always mounted with the 'shadow' attribute. Each VFS operation first checks to see if the filesystem is shadowing another (a quick check), and then whether the file or directory has the SUNWshadow attribute set (slightly more expensive, but cached with the vnode). If the attribute isn't present, then we fall through to the local filesystem. Otherwise, we migrate the file or directory and then fall through to the local filesystem.

Migration of directories

In order to migrate a directory, we have to migrate all the entriest. When migrating an entry for a file, we don't want to migrate the complete contents until the file is accessed, but we do need to migrate enough metadata such that access control can be enforced. We start by opening the directory on the remote side whose relative path is indicated by the SUNWshadow attribute. For each directory entry, we create a sparse file with the appropriate ownership, ACLs, system attributes and extended attributes.

Once the entry has been migrated, we then set a SUNWshadow attribute that is the same as the parent but with "/ " appended where "name" is the directory entry name. This attribute always represents the relative path of the unmigrated entity on the source. This allows files and directories to be arbitrarily renamed without losing track of where they are located on the source. It also allows the source to change (i.e. restored to a different host) if needed. Note that there are also types of files (symlinks, devices, etc) that do not have contents, in which case we simply migrate the entire object at once.

Once the diretory has been completely migrated, we remove the SUNWshadow attribute so that future accesses all use the native filesystem. If the process is interrupted (system reset, etc), then the attribute will still be on the parent directory so we will migrate it again when the user (or background process) tries to access it.

Migration of files

Migrating a plain file is much simpler. We use the SUNWshadow attribute to locate the file on the source, and then read the source file and write the corresponding data to the local filesystem. In the current software version, this happens all at once, meaning the first access of a large file will have to wait for the entire file to be migrated. Future software versions will remove this limitation and migrate only enough data to satisfy the request, as well as allowing concurrent accesses to the file. Once all the data is migrated, we remove the SUNWshadow attribute and future accesses will go straight to the local filesystem.

Dealing with hard links

One issue we knew we'd have to deal with is hard links. Migrating a hard link requires that the same file reference appear in multiple locations within the filesystem. Obviously, we do not know every reference to a file in the source filesystem, so we need to migrate these links as we discover them. To do this, we have a special directory in the root of the filesystem where we can create files named by their source FID. A FID (file ID) is a unique identifier for the file within the filesystem. We create the file in this hard link directory with a name derived from its FID. Each time we encounter a file with a link count greater than 1, we lookup the source FID in our special directory. If it exists, we create a link to the directory entry instead of migrating a new instance of the file. This way, it doesn't matter if files are moved around, removed from the local filesystem, or additional links created. We can always recreate a link to the original file. The one wrinkle is that we can migrate from nested source filesystems, so we also need to track the FSID (filesystem ID) which, while not persistent, can be stored in a table and reconstructed using source path information.

Completing migration

A key feature of the shadow migration design is that it treats all accesses the same, and allows background migration of data to be driven from userland, where it's easier to control policy. The downside is that we need the ability to know when we have finished migrating every single file and directory on the source. Because the local filesystem is actively being modified while we are traversing, it's impossible to know whether you've visited every object based only on walking the directory hierarchy. To address this, we keep track of a "pending" list of files and directories with shadow attributes. Every object with a shadow attribute must be present in this list, though this list can contain objects without shadow attributes, or non-existant objects. This allows us to be synchronous when it counts (appending entries) and lazy when it doesn't (rewriting file with entries removed). Most of the time, we'll find all the objects during traversal, and the pending list will contain no entries at all. In the case we missed an object, we can issue an ioctl() to the kernel to do the work for us. When that list is empty we know that we are finished and can remove the shadow setting.

ZFS integration

The ability to specify the shadow mount option for arbitrary filesystems is useful, but is also difficult to manage. It must be specified as an absolute path, meaning that the remote source of the mount must be tracked elsewhere, and has to be mounted before the filesystem itself. To make this easier, a new 'shadow' property was added for ZFS filesystems. This can be set using an abstract URI syntax ("nfs://host/path"), and libzfs will take care of automatically mounting the shadow filesystem and passing the correct absolute path to the kernel. This way, the user can manage a semantically meaningful relationship without worrying about how the internal mechanisms are connected. It also allows us to expand the set of possible sources in the future in a compatible fashion.

Hopefully this provides a reasonable view into how exactly shadow migration works, and the design decisions behind it. The goal is to eventually have this available in Solaris, at which point all the gritty details should be available to the curious.

Walter Moriconi: Virtualization video

Walter Moriconi — Thu, 08 Oct 2009 08:12:44 -0700

roch: CMT, NFS and 10 Gbe

roch — Wed, 07 Oct 2009 22:00:53 -0700

Now that we have Gigabytes/sec class of Network Attached OpenStorage and highly threaded CMT servers to attach from you figure just connecting the two would be enough to open the pipes for immediate performance. Well ... almost.

Our openstorage system can deliver great performance but we often find limitation on the client side. Now that NAS servers can deliver so much power, their NAS client can themselve be powerful servers trying to deliver GB/sec class services to the internet.

CMT servers are great throughput engines for that, however they deliver the goods when the whole stack is threaded. So in a recent engagement, my collegue David Lutz found that we needed one tuning at each of 4 levels in Solaris : IP, TCP, RPC and NFS.

Service	Tunable
IP	ip_soft_ring_cnt
TCP	tcp_recv_hiwat
RPC	clnt_max_conns
NFS	nfs3_max_threads
NFS	nfs4_max_threads

ip_soft_rings_cnt requires tuning up to Solaris 10 update 7. The default value of 2 is not enough to sustain the high throughput in a CMT environment. A value of 16 proved beneficial.

In /etc/system :

 * To drive 10Gbe in CMT in Solaris 10 update 7 : see blogs.sun.com/roch set ip_soft_rings_cnt=16

The receive socket buffer size is critical to the TCP connection performance. The buffer is not preallocated and memory is only used if and when the application is not reading the data it has requested. The default at 48K is from the age of 10MB/s Network cards and 1GB/sec systems. Having a larger value allows the peer to not throttle it's flow pending the returning TCP ACK. This is specially critical in high latency environment, urban area networks or other large fat network but it's also critical in the datacenter to reach a reasonable portion of the 10Gbe available in today's NIC. It turns out that NFS connection inherit the TCP default for the system and so it's interesting to run with a value between 400K and 1MB :

 ndd -set /dev/tcp_recv_hiwat 400000

But even with this, a single TCP connection is not enough to extract the most out of 10Gbe on CMT. And the solaris rpc client will establish a single connection to any of the server it connects to. The code underneath is highly threaded but did suffer from a few bugs when trying to tune that number of connections notably 6696163, 6817942 both of which are fixed in S10 update 8.

With that release, it becomes interesting to tune the number of RPC connections for instance to 8.

In /etc/system :

 * To drive 10Gbe in CMT in Solaris 10 update 8 : see blogs.sun.com/roch set clnt_max_conns=8

And finally, above the RPC layer, NFS does implement a pool of threads per mount point to service asynchronous requests. These will be mostly used in streaming workloads (readahead and writebehind) while other synchronous requests will be issued within the context of the application thread. The default number of asynchronous requests is likely to limit performance in some streaming scenario. So I would experiment with

In /etc/system :

 * To drive 10Gbe in CMT in Solaris 10 update 7 : see blogs.sun.com/roch set nfs3_max_threads=32 set nfs4_max_threads=32

As usual YMMV and use them with the usual circumspection, remember that tuning is evil but it's better to know about these factors than being in the dark and stuck with lower than expected performance.

bobp: Technology spending? Economic growth? Have you heard of UAE?

bobp — Wed, 07 Oct 2009 19:08:08 -0700

Recent news of increased economic stability in the world is comforting. However how much of it is based on sound fundamentals? If you are an investor or run a business where in the world do you want to invest for growth and equity? United Arab Emerates may be a place of interest. The UAE growth rate in GDP is astounding. While not all economic indicators are objectively sound, you have to speculate where can you find a better risk/reward ratio.

Western based banks have recovered their share price as well as balance sheets, but new credit lending is still tight. Loan losses are still surging in the West. In my opinion there remains too many toxic, complex, leveraged, convoluted, imaginary assets out there that have not been exposed to date. In order for the world economy to move forward the majority of bad credit instruments need to be exposed and liquidated. Otherwise we will continue to regain the false sense of consumer comfort that eventually got us into trouble.

Most world economies are driven by consumer spending, but consumer savings is the buffer from repeating what we are seeing today. One issue is where (globally) to place your assets into safe saving instruments with a secure and viable return.

I'm not sure anyone out there has all the answers...

Blog is available also at: http://bobporras.wordpress.com/

Walter Moriconi: Gartner Says Measurement and Monitoring of Data Centre Energy Use Will Remain Immature Through 2011

Walter Moriconi — Mon, 05 Oct 2009 22:22:49 -0700

The Gartner webinar conducted in April 2009 among
more than 130 attendees from the infrastructure and operations
(I&O) management found that although green IT issues remain at the
top of the agenda, respondents consider vendor and green procurement a
low priority activity for the next 18 months. Although 68 per cent of
respondents thought data centre energy management is their most
important green IT issue for the next 18 months, only 7 per cent
consider green procurement and pushing vendors to create more energy
efficient and greener solutions as their top priority.

...

source Gartner

Walter Moriconi: Gartner Says IT Organisations Will Invest More in Private Cloud Services than in External Cloud Providers Through 2012

Walter Moriconi — Mon, 05 Oct 2009 22:21:49 -0700

....

“With cloud offerings coming in the form of
services, this means that the IT organisation will be replaced by
relationships to many cloud computing service providers, each for one
or a handful of services,” said Phil Dawson, research vice president at
Gartner.

.....

source Gartner

Walter Moriconi: Western European Mobile Phone Market Shows Signs of Recovery, But Still Expected to Shrink in 2009, Says IDC

Walter Moriconi — Mon, 05 Oct 2009 22:20:43 -0700

LONDON, September 18, 2009 — The
Western European Mobile Phone market recorded another quarter of
year-on-year declines in the second quarter of 2009 (2Q09). According
to the IDC European Mobile Phone Tracker, handset vendors shipped 42
million units to Western Europe, down 6% from 2Q08. The second quarter
results are an improvement on the 14% decrease in 1Q09, but the crisis
will continue to impact the region.

.....

source IDC

Walter Moriconi: A Orvieto si parlerà di OpenOffice

Walter Moriconi — Mon, 05 Oct 2009 22:10:05 -0700

E' la splendida città umbra di Orvieto a fare da cornice alla prossima conferenza interamente dedicata ad OpenOffice.org.
La manifestazione si svolgerà dal 3 al 6 novembre prossimi e proporrà
una serie di sessioni durante le quali parleranno sia i singoli membri
della comunità sia gli esponenti delle principali aziende che
sostengono o il progetto OpenOffice.org o il formato OpenDocument
Format (ODF), tra cui Sun, IBM, Redflag 2000, Novell e Microsoft. Il
primo giorno della manifestazione sarà riservato alla comunità mentre i
successivi saranno aperti al pubblico.

...

source 01Net

Walter Moriconi: Gartner: il mercato dei semiconduttori resta in attesa

Walter Moriconi — Mon, 05 Oct 2009 20:29:46 -0700

Secondo Gartner, nel terzo trimestre dell'anno l'indice Dasi (Dataquest
Semiconductor Inventory) è sceso da un punteggio di 1.21, che
corrisponde a una situazione di “grave eccesso di stock”, a un più
contenuto 1.10-1.20, che corrisponde a un “moderato surplus di stock”,
appena al di sopra del livello di normalità.

....

source 01Net

Walter Moriconi: Lsi dà più efficienza allo storage

Walter Moriconi — Mon, 05 Oct 2009 20:29:01 -0700

La società ha reso disponibile agli Oem la versione aggiornata del
sistema di fascia alta Engenio 7900, che promette più elevati livelli
di efficienza e capacità. Il nuovo sistema storage è stato attrezzato
con alloggiamenti per drive Sata ad alta densità (da 3,5”) oltre che
per memorie a stato solido (Ssd). Puo montare interfacce host iScsi da
1Gb/s doppia porta a fianco degli adattatori fibre channel da 4 e 8
Gbps. Il sistema Lsi è, inoltre, fornito con lo strumento Storage
Profiler Tek-Tools che svolge compiti di monitoraggio e di reportistica.

.... source 01Net

Walter Moriconi: Audiweb pubblica i dati di AW Database del mese di luglio 2009

Walter Moriconi — Fri, 02 Oct 2009 09:49:43 -0700

Nel mese di luglio risultano 30,8 milioni gli utenti che hanno accesso a internet,
21,7 milioni gli utenti attivi nel mese, 10,4 milioni gli utenti attivi
nel giorno medio, per 1 ora e 38 minuti di tempo speso e 179 pagine
viste per persona nel giorno medio

....

source Audiweb

Walter Moriconi: Cloud Storage Standard Released for Public Review

Walter Moriconi — Fri, 02 Oct 2009 09:46:55 -0700

The SNIA Cloud Storage Technical Work Group has completed the first
public draft of the new cloud storage standard: Cloud Data Management
Interface (CDMI),
Version 0.8 is avalable for review and feedback from:

http://www.snia.org/tech_activities/publicreview

....source SNIA

Walter Moriconi: Virtualization World Snia - Agenda

Walter Moriconi — Fri, 02 Oct 2009 09:45:48 -0700

Tuesday 27th October:

08:00 – 18:00: Conference Registration Open

08:00 – 19:30: Expo Areas Open, Hall 5

09:00 – 17:30: Conference Sessions*, Congress Centre

17:30 – 19:30: Reception

Wednesday 28th October:

09:00 – 14:30: Conference Registration Open

09:00 – 16:00: Expo Areas Open, Hall 5

09:30 – 16:10: Conference Sessions*, Congress Centre

complete agend source www.virtualizationworld.net

Walter Moriconi: Mediaset, Regione Siciliana e i-Faber, eccellenze italiane

Walter Moriconi — Fri, 02 Oct 2009 09:26:27 -0700

Mediaset per le Grandi aziende, Regione Siciliana nella categoria
Pubblica amministrazione e i-Faber per le Pmi sono i vincitori della
seconda edizione del Customer Success Award, il premio che Oracle
assegna come riconoscimento alle organizzazioni italiane, pubbliche o
private, che, a partire dal 2007, hanno realizzato i progetti più
innovativi affidandosi alle sue soluzioni It.

....source 01Net

Walter Moriconi: L'It a picco con hardware e servizi

Walter Moriconi — Fri, 02 Oct 2009 09:23:38 -0700

...

-9% per l'It e -2,5% per le telecomunicazioni (-6,7 il fisso e -3,4%
il mobile) sono le cifre che riassumono la situazione e che riportano
il settore a una situazione simile a quella del primo semestre del 1991.

Entrando nel dettaglio, l'hardware lascia sul terreno il 15,7%, il
software il 4,1%, l'assistenza tecnica il 6,2% e i servizi il 7,3%.

....

source 01Net

Walter Moriconi: Meno risorse e più richieste per i Cio

Walter Moriconi — Fri, 02 Oct 2009 09:19:59 -0700

.....

Lo studio, condotto su 100 Chief information officer europei di aziende
operanti in tutti i settori ha evidenziato per il Cio un ruolo evoluto
in tutte le componenti: è un manager che catalizza il cambiamento,
frenato da budget in contrazione, da complessità applicativa e da
eccessive richieste interne, e che vive le sfide dell'ottimizzazione
processuale, della razionalizzazione informativa, della sicurezza e
dell'innovazione.

....

source 01Net

Walter Moriconi: Premio innovazione: i vincitori

Walter Moriconi — Fri, 02 Oct 2009 09:19:02 -0700

L'elenco dei premiati della prima edizione del riconoscimento che Smau assegna a Pmi e Pa che utilizzano al meglio l'Ict

...

source 01Net

Walter Moriconi: Pmi più mature. L'Ict ora è meno lontana

Walter Moriconi — Fri, 02 Oct 2009 09:18:11 -0700

...

"Negli ultimi due anni - spiega Balocco, responsabile della ricerca sulle Ict nelle Pmi - abbiamo assistito ad un aumento del livello di maturità Ict: in particolare, le imprese Lungimiranti,
che stanno utilizzando in modo evoluto sia l'infrastruttura Ict sia il
parco applicativo sono passate dal 12% al 17%, mentre si è ridotta la
percentuale di imprese Immature, passate dal 42% al 34% ".

....

source 01Net

Walter Moriconi: Pa: Brunetta, al via sperimentazione posta elettronica certificata

Walter Moriconi — Fri, 02 Oct 2009 09:16:35 -0700

....

"Questa è una rivoluzione - ha detto il ministro della Pubblica amministrazione e dell'innovazione, Renato Brunetta illustrando l'iniziativa -. La
sperimentazione con Inps e Aci durera fino a dicembre. Entro novembre
tutti i professionisti dovranno dotarsi di posta elettronica
certificata per dialogare con la pubblica ammnistrazione, da gennaio
sarà avviato la posta elettronica certificata gratuita per tutti i
cittadini". ......

source 01Net

: Gonzalez Goes GeekAndPoke: My First Co-Authored Webcomic

Fri, 02 Oct 2009 00:55:40 -0700

A while ago, I highlighted a few of my favourite web comics. Little did I know then, that today I was going to be part of one. Here's the story:

Yesterday, @moellus complained yet again about his eternal nemesis, the NT admin, by saying something like: "Damn, can't unfollow the NT admin because he doesn't twitter - here's why - that's the sh*# he sends me!" (Original Tweet)

That reminded me of the famous insult by the late Douglas Adams from "The Hitchhiker's Guide to the Galaxy", in which Arthur Dent says about the Vogons: "I wish I had a daughter so I could forbid her to marry one..."

So I twittered back that "I wish you were on Twitter so I could unfollow you!" must be the new worst insult you can do in 2009 and asked Oliver whether that would make a nice Geek And Poke cartoon.

Today, I'm proud to be part of the "Geek And Poke"-uversum, here's the cartoon:

I guess, to be "GeekAndPoked" is the new "Slashdotted" :). Thanks, Oliver!

P.S.: @moellus: Actually, to a geek, in 2009, sending someone a box of Windows 7 is like the medieval slapping with a glove. Don't take it lightly and get your LART-whip ready!

bobp: China & India Fuel Growth With Infrastructure Projects

bobp — Mon, 21 Sep 2009 19:08:08 -0700

While the U.S. is still trying to stimulate spending of U.S. consumers with its economic stimulus programs, countries such as India and China appear to be making much better progress. On a recent trip in September to China and India via Singapore one can visibly see that Asia Pacific is not standing still and putting idled factory workers toward new infrastructure projects. Talking with customers and partners in China and India only confirms what could be described as a best practice for any nation looking for economic recovery.

Earlier this year Beijing announced a $585 billion (U.S. dollars) stimulus package (~13% of China's 2008 GDP). As part of this stimulus package China will spend this year $50 billion (U.S. dollars) alone on the ambitious worlds biggest high-speed (started in 2005 and expected to finish in 2020) railway which will connect Beijing, Dalian, Xuzhou, Lanzhou, Shanghai, Kunming and Guangzhou. Beijing's economic stimulus is directly targeted at the ~20 million idled workers in China due mostly to the 20% decline in foreign sales. Notable statistics of the new high-speed railway include:

The buttresses which carry the tracks will consume 117 million tons of concrete

16,000 miles (25,749 km) of new track will be built

The Beijing to Shanghai line will consume enough steel to build 120 "Birds Nest" Olympic Stadiums

In addition to construction labor the railway will create jobs for engineers and lot's of them. The railway will be computerized and require a lot of IT spending and resources. You can be certain the large IT companies are looking to win this business and provide hardware, software, services and support. A great opportunity. Bill Powell wrote a great article in FORTUNE magazine with many more exciting facts which you can read here.

India's growth is equally impressive despite the global recession. India still needs to build massive amounts of infrastructure while China is already upgrading infrastructure previous built. I like to describe China as organized chaos while India is still dealing with unorganized chaos. In Beijing you see cars, buses, trains, people, bikes, motorcycles all in congested traffic in harmony with the traffic signals. In Bangalore you have the same as Beijing but with everyone only in harmony with themselves. Traffic signals get ignored, motorcycles are riding on the sidewalk and intersections, at rush hour, give no clear indication as to who has the right of way. It's simply something that every business professional should experience because you need to know the customer you are selling. Both are beautiful cultures and part of the solution of economic recovery in the world. India too will consume massive amount of engineers, software, hardware and services not only to fuel their IT outsourcing industry but to rapidly build their own nation, infrastructure and ultimately their own economy as a world leader.

Blog is available also at: http://bobporras.wordpress.com/

timf: OpenSolaris at OSS BarCamp

timf — Sun, 20 Sep 2009 19:25:10 -0700

I presented An Introduction to OpenSolaris last Saturday at OSS BarCamp - my contribution to Software Freedom Day 2009.

You can download the odp presentation or the pdf, which I've exported with my notes for the talk that explain each of the slides a little more - I hope this is useful if you're planning on giving a similar talk.

A few things struck me while preparing for and giving the presentation. Firstly, it seemed odd to be giving an introduction to an operating system that's been around for quite a while now: clearly we haven't been doing enough of this sort of thing (and from personal experience, yes, ie-osug isn't as active as I'd like, I just sadly don't have the bandwidth)

Then secondly, it's really hard to cover all of the interesting features of OpenSolaris in sufficient detail over the course of an hour. My take, was to try to whet the appetite, rather than explain every feature fully - and in some cases, go for the features I thought the audience might be interested in (for example, mentioning NWAM as one of the major networking features - perhaps it's not as full of rocket science as other aspects of Solaris networking, but it makes a huge difference to the novice user)

Finally, and slightly embarrassingly, I had to spend about 5 minutes in front of an expectant audience futzing around with the display settings on my R500 laptop to get it to talk to the projector. It was doubly annoying that both xrandr (and indeed gnome-display-properties) were able to see the separate screen, but try as I might, I couldn't get any output to appear externally. Ultimately, a kind audience member offered me a USB key by which I transferred the pdf over to my EeePC (running nv_122) which was able to see the projector, but didn't have any of my demo material setup (some ZFS settings, some zones, crossbow, flows etc.) Oh well.

Here's hoping that at least some of the audience left with an impression that OpenSolaris was worth taking a second (or perhaps a first?) look at, despite the brevity of my talk and the initial teething problems I had. My new mantra:

Never work with children, animals, or weird exernal VGA projectors.

: New OpenSolaris ZFS Home Server: Requirements

Sat, 19 Sep 2009 08:24:29 -0700

A few months ago, I decided it was time for a new home server. The old one (see picture) is now more than 3 years old (the hardware is 2 years older), so it was time to plan ahead for the inevitable hardware failure. Curiously enough, my old server started to refuse working with some of my external USB disks only a few weeks ago, which confirmed my need for a new system. This is the beginning of a series of blog articles around building a new OpenSolaris home server.

Home Server Goals

Let's go over some goals for a home server to help us decide on the hardware. IMHO, a good home server should:

Run OpenSolaris. This means I don't want an appliance because this is too limiting and I'd end up hacking it to make it run something it doesn't do by itself anyway. It should therfore use a real OS.
It also means I don't want to use Linux, because quite frankly the whole Linux landscape is too unstable, confused and de-focused (don't get me wrong: It's nice for experimentation and as a hobbyist-OS, but I want something more serious to guard my data).
Windows is out of the question because it delivers too little for too high a price.
I like BSD (and used to run NetBSD on my Amiga 4000 back then in the mid nineties), but it seems to be more oriented to some (albeit interesting) niches for my taste.
Right now I prefer OpenSolaris because it's rock-solid, clean, well-documented, well-designed and it has lots of advanced features that other OSes only dream of. Yes, I'd still write and do the same if I weren't a Sun employee.
Leverage ZFS. This should be a no-brainer, but I just wanted to point out that any system that is serious about its data should absolutely run ZFS because of the end-to-end-integrity. Period. And then there are many useful features such as compression, send/receive, snapshots, ease of administration, no fscks and much more. Oh, and I'm looking forward to leveraging encryption and de-duplication at home in the near future, too!
Use ECC Memory: What's the use of having end-to-end data integrity with ZFS if your data is corrupted before ZFS can create it's checksum? That's why you need ECC Memory. Simply put: Use ECC memory and kiss those unexpected, unexplicable system crashes and broken data surprises good bye.
Be Power Efficient: Think 1.5 Euros of electricity bill per Watt per Year for a system running 24/7. The difference between your typical gaming PC and a power-efficient home server can easily be 50W or more when idle, so you're looking at an extra 75 Euros and more of free cash if you just pick your components more carefully. Notice that I'm not saying "Low-Power". There are a lot of compromises when trying to reach absolute low-powerness. Like many optimization problems, squeezing the last few Watts out of your system means investing a lot of money and effort while sacrificing important features. So I want this system to be power-efficient, but without too many sacrifices.
Use a Moderate Amount of Space: While my home server sits in the basement, form doesn't matter. But I may move into a new apartment where networking to the basement is not an option. Then the server needs to be living-room capable and have a decent WAF. Which also brings us to:
Be quiet: A power-efficient server needs less cooling which helps with being quiet. Again, we don't want to stretch the limits of quietness at all costs, but we want to make sure we don't do any obvious mistakes here that sacrifice the living-room capabilities of the system

What's Next

In the next blog entry, we'll discuss a few processor and platform considerations and reveal a cool, yet powerful option that presented itself to me. Meanwhile, feel free to check out other home server resources, such as Simon Breden's blog, Matthias Pfuetzner's blog, Jan Brosowski's Blog (German) or one of the many home server discussions on the zfs-discuss mailing list.

What are your requirements for a good home server? What do you currently use at home to fulfill your home server needs? What would you add to the above list of home server requirements? Feel free to add a comment below!

eschrock: User and Group Quotas in the 7000 Series

eschrock — Thu, 17 Sep 2009 03:19:49 -0700

When ZFS was first developed, the engineering team had the notion that pooled storage would make filesystems cheap and plentiful, and we'd move away from the days of /export1, /export2, ad infinitum. From the ZFS perspective, they are cheap. It's very easy to create dozens or hundreds of filesystems, each which functions as an administrative control point for various properties. However, we quickly found that other parts of the system start to break down once you get past 1,000 or 10,000 filesystems. Mounting and sharing filesystems takes longer, browsing datasets takes longer, and managing automount maps (for those without NFSv4 mirror mounts) quickly spirals out of control.

For most users this isn't a problem - a few hundred filesystems is more than enough to manage disparate projects and groups on a single machine. There was one class of users, however, where a few hundred filesystems wasn't enough. These users were university or other home directory environments with 20,000 or more users, each which needed to have a quota to guarantee that they couldn't run amok on the system. The traditional ZFS solution, creating a filesystem for each user and assigning a quota, didn't scale. After thinking about it for a while, Matt developed a fairly simple architecture to provide this functionality without introducing pathological complexity into the bowels of ZFS. In build 114 of Solaris Nevada, he pushed the following:

PSARC 2009/204 ZFS user/group quotas & space accounting

This provides full support for user and group quotas on ZFS, as well as the ability to track usage on a per-user or per-group basis within a dataset.

This was later integrated into the 2009.Q3 software release, with an additional UI layer. From the 'general' tab of a share, you can query usage and set quotas for individual users or groups quickly. The CLI allows for automated batch operations. Requesting a single user or group is significantly faster than requesting all the current usage, but you an also get a list of the current usage for a project or share. With integrated identity management, users and groups can be specified either by UNIX username or Windows name.

There are some significant differences between user and group quotas and traditional ZFS quotas. The following is an excerpt from the on-line documentation on the subject:

User and group quotas can only be applied to filesystems.
User and group quotas are implemented using delayed enforcement. This means that users will be able to exceed their quota for a short period of time before data is written to disk. Once the data has been pushed to disk, the user will receive an error on new writes, just as with the filesystem-level quota case.
User and group quotas are always enforced against referenced data. This means that snapshots do not affect any quotas, and a clone of a snapshot will consume the same amount of effective quota, even though the underlying blocks are shared.
User and group reservations are not supported.
User and group quotas, unlike data quotas, are stored with the regular filesystem data. This means that if the filesystem is out of space, you will not be able to make changes to user and group quotas. You must first make additional space available before modifying user and group quotas.
User and group quotas are sent as part of any remote replication. It is up to the administrator to ensure that the name service environments are identical on the source and destination.
NDMP backup and restore of an entire share will include any user or group quotas. Restores into an existing share will not affect any current quotas. (There is currently a bug preventing this from working in the initial release, which will be fixed in a subsequent minor release.)

This feature will hopefully allow the Sun Storage 7000 series to function in environments where it was previously impractical to do so. Of course, the real person to thank is Matt and the ZFS team - it was a very small amount of work to provide an interface on top of the underlying ZFS infrastructure.

: New OpenSolaris ZFS Auto-Scrub Service Helps You Keep Proper Pool Hygiene

Thu, 17 Sep 2009 07:25:34 -0700

One of the most important features of ZFS is the ability to detect data corruption through the use of end-to-end checksums. In redundant ZFS pools (pools that are either mirrored or use a variant of RAID-Z), this can be used to fix broken data blocks by using the redundancy of the pool to reconstruct the data. This is often called self-healing.

This mechanism works whenever ZFS accesses any data, because it will always verify the checksum after reading a block of data. Unfortunately, this does not work if you don't regularly look at your data: Bit rot happens and with every broken block that is not checked (and therefore not corrected), the probability increases that even the redundant copy will be affected by bit rot too, resulting in data corruption.

Therefore, zpool(1M) provides the useful scrub sub-command which will systematically go through each data block on the pool and verify its checksum. On redundant pools, it will automatically fix any broken blocks and make sure your data is healthy and clean.

It should now be clear that every system should regularly scrub their pools to take full advantage of the ZFS self-healing feature. But you know how it is: You set up your server and often those little things get overlooked and that cron(1M) job you wanted to set up for regular pool scrubbing fell off your radar etc.

Introducing the ZFS Auto-Scrub SMF Service

Here's a service that is easy to install and configure that will make sure all of your pools will be scrubbed at least once a month. Advanced users can set up individualized schedules per pool with different scrubbing periods. It is implemented as an SMF service which means it can be easily managed using svcadm(1M) and customized using svccfg(1M).

The service borrows heavily from Tim Foster's ZFS Auto-Snapshot Service. This is not just coding laziness, it also helps minimize bugs in common tasks (such as setting up periodic cron jobs) and provides better consistency across multiple similar services. Plus: Why invent the wheel twice?

Requirements

The ZFS Auto-Scrub service assumes it is running on OpenSolaris. It should run on any recent distribution of OpenSolaris without problems.

More specifically, it uses the -d switch of the GNU variant of date(1) to parse human-readable date values. Make sure that /usr/gnu/bin/date is available (which is the default in OpenSolaris).

Right now, this service does not work on Solaris 10 out of the box (unless you install GNU date in /usr/gnu/bin). A future version of this script will work around this issue to make it easily usable on Solaris 10 systems as well.

Download and Installation

You can download Version 0.5b of the ZFS Auto-Scrub Service here. The included README file explains everything you need to know to make it work:

After unpacking the archive, start the install script as a privileged user:

pfexec ./install.sh

The script will copy three SMF method scripts into /lib/svc/method, import three SMF manifests and start a service that creates a new Solaris role for managing the service's privileges while it is running. It also installs the OpenSolaris Visual Panels package and adds a simple GUI to manage this service.

After installation, you need to activate the service. This can be done easily with:

svcadm enable auto-scrub:monthly

or by running the GUI with:

vp zfs-auto-scrub

This will activate a pre-defined instance of the service that makes sure each of your pools is scrubbed at least once a month.

This is all you need to do to make sure all your pools are regularly scrubbed.

If your pools haven't been scrubbed before or if the time or their last scrub is unknown, the script will proceed and start scrubbing. Keep in mind that scrubbing consumes a significant amount of system resources, so if you feel that a currently running scrub slows your system too much, you can interrupt it by saying:

pfexec zpool scrub -s <pool name>

In this case, don't worry, you can always start a manual scrub at a more suitable time or wait until the service kicks in by itself during the next scheduled scrubbing period.

Should you want to get rid of this service, use:

pfexec ./install.sh -d

The script will then disable any instances of the service, remove the manifests from the SMF repository, delete the scripts from /lib/svc/method, remove the special role and the authorizations the service created and finally remove the GUI. Notice that it will not remove the OpenSolaris Visual Panels package in case you want to use it for other purposes. Should you want to get rid of this as well, you can do so by saying:

pkg uninstall OSOLvpanels

Advanced Use

You can create your own instances of this service for individual pools at specified intervals. Here's an example:

 constant@fridolin:~$ svccfg svc:> select auto-scrub svc:/system/filesystem/zfs/auto-scrub> add mypool-weekly svc:/system/filesystem/zfs/auto-scrub> select mypool-weekly svc:/system/filesystem/zfs/auto-scrub:mypool-weekly> addpg zfs application svc:/system/filesystem/zfs/auto-scrub:mypool-weekly> setprop zfs/pool-name=mypool svc:/system/filesystem/zfs/auto-scrub:mypool-weekly> setprop zfs/interval=days svc:/system/filesystem/zfs/auto-scrub:mypool-weekly> setprop zfs/period=7 svc:/system/filesystem/zfs/auto-scrub:mypool-weekly> setprop zfs/offset=0 svc:/system/filesystem/zfs/auto-scrub:mypool-weekly> setprop zfs/verbose=false svc:/system/filesystem/zfs/auto-scrub:mypool-weekly> end constant@fridolin:~$ svcadm enable auto-scrub:mypool-weekly

This example will create and activate a service instance that makes sure the pool "mypool" is scrubbed once a week.

Check out the zfs-auto-scrub.xml file to learn more about how these properties work.

Implementation Details

Here are some interesting aspects of this service that I came across while writing it:

The service comes with its own Solaris role zfsscrub under which the script runs. The role has just the authorizations and profiles necessary to carry out its job, following the Solaris Role-Based Access Control philosophy. It comes with its own SMF service that takes care of creating the role if necessary, then disables itself. This makes a future deployment of this service with pkg(1) easier, which does not allow any scripts to be started during installation, but does allow activation of newly installed SMF services.
While zpool(1M) status can show you the last time a pool has been scrubbed, this information is not stored persistently. Every time you reboot or export/import the pool, ZFS loses track of when the last scrub of this pool occurred. This has been filed as CR 6878281. Until that has been resolved, we need to take care of remembering the time of last scrub ourselves. This is done by introducing another SMF service that periodically checks the scrub status, then records the completion date/time of the scrub in a custom ZFS property called org.opensolaris.auto-scrub:lastscrub in the pool's root filesystem when finished. We call this service whenever a scrub is started and it deactivates itself once it's job is done.
As mentioned above, the GUI is based on the OpenSolaris Visual Panels project. Many thanks to the people on its discussion list to help me get going. More about creating a visual panels GUI in a future blog entry.

Lessons learned

It's funny how a very simple task like "Write an SMF service that takes care of regular zpool scrubbing" can develop into a moderately complex thing. It grew into three different services instead of one, each with their own scripts and SMF manifests. It required an extra RBAC role to make it more secure. I ran into some zpool(1M) limitations which I now feel are worthy of RFEs and working around them made the whole thing slightly more complex. Add an install and de-install script and some minor quirks like using GNU date(1) instead of the regular one to have a reliable parser for human-readable date strings, not to mention a GUI and you cover quite a lot of ground even with a service as seemingly simple as this.

But this is what made this project interesting to me: I learned a lot about RBAC and SMF (of course), some new scripting hacks from the existing ZFS Auto-Snapshot service, found a few minor bugs (in the ZFS Auto-Snapshot service) and RFEs, programmed some Java including the use of the NetBeans GUI builder and had some fun with scripting, finding solutions and making sure stuff is more or less cleanly implemented.

I'd like to encourage everyone to write their own SMF services for whatever tools they install or write for themselves. It helps you think your stuff through, make it easy to install and manage, and you get a better feel of how Solaris and its subsystems work. And you can have some fun too. The easiest way to get started is by looking at what others have done. You'll find a lot of SMF scripts in /lib/svc/method and you can extract the manifests of already installed services using svccfg export. Find an SMF service that is similar to the one you want to implement, check out how it works and start adapting it to your needs until your own service is alive and kicking.

If you happen to be in Dresden for OSDevCon 2009, check out my session on "Implementing a simple SMF Service: Lessons learned" where I'll share more of the details behind implementing this service including the Visual Panels part.

Edit (Sep. 21st) Changed the link to CR 6878281 to the externally visible OpenSolaris bug database version, added a link to the session details on OSDevCon.

roch: iSCSI unleashed

roch — Wed, 16 Sep 2009 19:23:50 -0700

One of the much anticipated feature of the 2009.Q3 release of the fishworks OS is a complete rewrite of the iSCSI target implementation known as Common Multiprotocol SCSI Target or COMSTAR. The new target code is an in-kernel implementation that replaces what was previously known as the iSCSI target deamon, a user-level implementation of iSCSI.

Should we expect huge performance gains from this change ? You Bet !

But like most performance question, the answer is often : it depends. The measured performance of a given test is gated by the weakest link triggered. iSCSI is just one component among many others that can end up gating performance. If the daemon was not a limiting factor, then that's your answer.

The target deamon was a userland implementation of iSCSI : some daemon threads would read data from a storage pool and write data to a socket or vice versa. Moving to a kernel implementation opens up options to bypass at least one of the copies and that is being considered as a future option. But extra copies while undesirable do not necessarily contribute to the small packet latency or large request throughput; For small packets requests, the copy is small change compared to the request handling. For large request throughput the important things is that the data path establishes a pipelined flow in order to keep every components busy at all times.

But the way threads interact with one another can be a much greater factor in delivered performance. And there lies the problem. The old target deamon suffered from one major flaw in that each and every iSCSI requests would require multiple trips through a single queue (shared between every luns) and that queue was being read and written by 2 specific threads. Those 2 threads would end up fighting for the same locks. This was compounded by the fact that user level threads can be put to sleep when they fail to acquire a mutex and that going to sleep for a user level thread is a costly operation implying a system call and all the accounting that goes with that.

So while the iSCSI target deamon gave reasonable service for large request, it was much less scalable in terms of the number IOPS that can be served and the CPU efficiency in which it could do that. IOPS being of course a critical metrics for block protocols.

As an illustration of that with 10 client initiators and 10 threads per initiators (so 100 outstanding request) doing 8K cache-hit reads, we observed

Old Target Daemon	Comstar	Improvement
31K IOPS	85K IOPS	2.7X

Moreover the target daemon was consuming 7.6 CPU to service those 31K IOPS while comstar could handle 2.7X more IOPS consuming only 10 cpus, a 2X improvement in iops per cpu efficiency.

On the write side, with a disk pool that had 2 striped write optimised SSD, comstar gave us 50% more throughput (130 MB/sec vs 88MB/sec) and 60% more cpu efficiency.

Immediatedata

During our testing we noted a few interesting contributor to delivered performance. The first being the setting of iSCSI immediatedata parameter iSCSIadm(1M). On the write path, that parameter will cause the initiator iSCSI to send up to 8K of data along with the initial request packet. While this is a good idea to do so, we found that for certain sizes of writes, it would trigger some condition in the zil that caused ZFS to issue more data than necessary through the logzillas. The problem is well understood and remediation is underway and we expect to get to a situation in which keeping the default value of immediatedata=yes is the best. But as of today, for those attempting world record data transfer speeds through logzillas, setting immediatedata=no and using a 32K or 64K write size might yield positive result depending on your client OS.

Interrupt Blanking

Interested in low latency request response ? Interestingly, a chunk of that response time is lost in the obscure setting of network card drivers. Network cards will often delay pending interrupts in the hope of coalescing more packets into a single interrupt. The extra efficiency often results in more throughput at high data rate at the expense of small packet latency. For 8K request we manage to get 15% more single threaded IOPS by tweaking one such client side parameter. Historically such tuning has always been hidden in the bowel of each drivers and specific to ever client OS so that's too broad a topic to cover here. But for Solaris clients, the Crossbow framework is aiming among other thing to make latency vs throughput decision much more adaptive to operating condition relaxing the need for per workload tunings.

WCE Settings

Another important parameter to consider for comstar is the 'write cache enable' bit. By default all write request to an iSCSI lun needs to be committed to stable storage as this is what is expected by most consumers of block storage. That means that each individual write request to a disk based storage pool will take minimally a disk rotation or 5ms to 8ms to commit. This also why a write optimised SSD is quite critical to many iSCSI workloads often yeilding 10X performance improvements. Without such an SSD, iSCSI performance will appear quite lackluster particularly for lightly threaded workloads which more affected by latency characteristics.

One could then feel justified to set the write cache enable bits on some luns in order to recoup some spunk in their engine. One good news here is that in the new 2009.Q3 release of fishworks the setting is now persistent across reboots and reconnection event, fixing a nasty condition of 2009.Q2. However one should be very careful with this setting as the end consumer of block storage (exchange, NTFS, oracle,...) is quite probably operating under an unexpected set of condition. This setting can lead to application corruption in case of outage (no risk for the storage internal state).

There is one exception to this caveat and it is ZFS itself. ZFS is designed to safely and correctly operate on top of devices that have their write cached enabled. That is because ZFS will flush write caches whenever application semantics or its own internal consistency require it. So a ZPOOL created on top of iSCSI luns would be well justified to set the WCE on the lun to boost performance.

Synchronous write bias

Finally as described in my blog entry about Synchronous write bias, we now have to option to bypass the write optimised SSDs for a lun if the workload it receive is less sensitive to latency. This would be the case of a highly threaded workload doing large data transfers. Experimenting with this new property is warranted at this point.

roch: Synchronous write bias property

roch — Wed, 16 Sep 2009 19:13:49 -0700

With the release of 2009.Q3 release of fishworks along with a new iSCSI implemtation we're coming up with a very significant new feature for managing performance of Oracle database : the new dataset Synchronous write bias property or logbias for short. In a nutshell, this property takes the default value of Latency signifying that the storage should handle synchronous writes in urgency, the historical default handling. See Brendan's comprehensive blog entry on the Separate Intent Log and synchronous writes. However for datasets holding Oracle Datafiles, the logbias property can be set to Throughput signifying that the storage should avoid using log devices acceleration instead trying to optimize the workload's throughput and efficiency. We definitely expect to see a good boost to Oracle performance from this feature for many types of workloads and configs; workloads that generate 10s of MB/sec of DB writer traffic and have no more than 1 logzilla per tray/JBOD.

The property is set in the Share Properties just above database recordsize. You might need to unset the Inherit from projet checkbox in order to modify the settings on a particular share:

The logbias property addresses a peculiar aspect of Oracle workloads : namely that DB writers are issuing a large number of concurrent synchronous writes to Oracle datafiles, writes which individually are not particularly urgent. In contrast to other types of synchronous writes workloads, the more important metrics for DB Writers is not about individual latency. The important metric is that the storage keep up with the throughput demand in order to have database buffers always available for recycling. This is unlike redo log writes which are critically sensitive to latency as they are holding up individual transactions and thus users.

ZFS and the ZIL

A little background; with ZFS, synchronous writes are managed by the ZFS Intent Log ZIL. Because synchronous writes are typically holding up applications, it's important to handle those writes with some level of urgency and the ZIL does an admirable job at that.

In the Openstorage hybrid storage pool the ZIL itself is speeded up using low latency write-optimized SSD devices : the logzillas. Those devices are used to commit a copy of the in-memory ZIL transaction and retain the data until an upcoming transaction group commits the in-memory state to the on-disk pooled storage (Dynamics of ZFS, The New ZFS write throttle).

So while the ZIL speeds up synchronous writes, logzillas speeds up the zil. Now SSDs can serve IOPS at a blazing 100μs but also have their own throughput limits: currently around 110MB/sec per device. At that throughput, committing, for example, 40K of data will need minimally 360μs. The more data we can divert away from log devices, the lower the latency response of those devices will be.

It's interesting to note that other types of raid controllers will be hostage of their NVRAM and require, for consistency, that data be committed through some form of acceleration in order to avoid the Raid Write Hole (Bonwick on Raid-Z). ZFS, however, does require that data passes through its SSD commit accelerator and it can manage consistency of commits either using disk or using SSDs.

Synchronous write bias : Throughput

With this newfound ability of storage administrators to signify to ZFS that some datasets will be subject to highly threaded synchronous writes for which global throughput is more critical than individual write latency, we can enable a different handling mode. By setting Logbias=Throughput ZFS is able to divert writes away from the Logzillas which are then preserved for servicing low latency sensitive operations (e.g. redo log operations).

A setting of `Synchronous write bias : Throughput` for a dataset allows synchronous writes to files in other datasets to have lower latency access to SSD log devices.

But that's not all. Data flowing through a logbias=Throughput dataset is still served by the ZIL. It turns out that the ZIL has different internal options in the way it can commit transactions one of which being tagged WR_INDIRECT. WR_INDIRECT commits issue an I/O for the modified file record and record a pointer to it in the zil chain. (see WR_INDIRECT in zil.c, zvol.c, zfs_log.c ).

ZIL transaction of type WR_INDIRECT might use more disk I/Os and slightly higher latency immediately but less I/Os and less total bytes during the upcoming transaction group update. Up to this point, the heuristics that lead to using WR_INDIRECT transactions, were not triggered by DB writer workloads. But armed with the knowledge that comes with the new logbias property, we're now less concerned about the slight latency increase that WR_INDIRECT can have. So from efficiency consideration the logbias=Throughput datasets are now set to use this mode leading to more leveled latency distributions of Transactions.

`Synchronous write bias : Throughput` is a dataset mode that reduces the number of I/Os that need to be issued on behalf of this dataset during the regular transaction group updates leading to more leveled response time.

A reminder that such kind of improvements sometimes can go unnoticed in sustained benchmarks if the downstream Transaction group destage is not given enough resources. Make sure you have enough spindles (or total disk KRPM) to sustain the level of performance you need. A pool with 2 logzillas and a single JBOD, might have enough SSD throughput to absorb DB writer workloads without adversely affecting redo log latency and so would not benefit from the special logbias settings, however for 1 logzillas per JBOD the situation might be reversed.

While the DB Record Size property is inherited by files in a dataset and is immutable, the logbias setting is totally dynamic and can be toggled on the fly during operations. For instance, during database creation or some lightly threaded write operations to Datafiles, it's expected that logbias=Latency should perform better.

Logbias deployments for Oracle

As of the 2009.Q3 release of fishworks, the current wisdom around deploying Oracle DB an Openstorage system with SSD acceleration, is to segregate, at the filesystem/dataset level, but within the single storage pool, Oracle datafiles, index files and redo Log files. Having each type of files in different dataset allows better observability into each one using the great analytics tool. But also, each dataset can then be tuned independantly to deliver the most stable performance characteristics. The most important parameter to consider is the ZFS internal recordsize used to manage the files. For Oracle datafiles the established (ZFS Best Practice) is to match the recordsize and the DB block size. For redo log files using default 128K records means that fewer file updates will be stradling multiple filesystem records. With 128K records we expect to have fewer transaction needing to wait for redo log input I/Os leading more leveled latency distribution for transactions. As for Index files, using smaller blocks of 8K offers better cacheability feature for both the primary and secondary caches (only cache what is used from indexes), but using larger blocks offers better index-scan performance. Experimenting is in order, depending on your use case, but an intermediate block size of maybe 32K might also be considered for mixed usage scenario.

For Oracle datafiles specifically, using the new setting of Synchronous write bias : Throughput has potential to deliver more stable performance in general and higher performance for redo log sensitive workloads.

Dataset	Recordsize	Logbias
Datafiles	8K	Throughput
Redo Logs	128K(default)	Latency(default)
Index	8K-32K?	Latency(default)

Following these guidelines yielded a 40% boost in our Transaction processing testing in which we had 1 logzillas for a 40 disk pool.

ahl: 2009.Q3 Storage Configuration

ahl — Wed, 16 Sep 2009 10:38:50 -0700

Today we shipped our 2009.Q3 release. Amidst the many great new features, enhancements and bug fixes, we've added new storage profiles for triple-parity RAID and three-way mirroring. Here's an example on a 9 JBOD system of what you'll see in the updated storage configuration screen:

Note that the new Triple parity RAID, wide stripes option replaces the old Double parity RAID, wide stripes configuration. With RAID stripes that can easily be more than 40 disks wide, and resilver times that can be quite long as a result, we decided that the additional protection of triple-parity RAID trumped the very small space efficiency of double-parity RAID.

Ryan Matthews has updated the space calculator for the 7310 and 7410 to include the new profiles. Download the new update and give it a shot.

eschrock: What is Shadow Migration?

eschrock — Wed, 16 Sep 2009 10:06:01 -0700

In the Sun Storage 7000 2009.Q3 software release, one of the major new features I worked on was the addition of what we termed "shadow migration." When we launched the product, there was no integrated way to migrate data from existing systems to the new systems. This resulted in customers rolling it by hand (rsync, tar, etc), or paying for professional services to do the work for them. We felt that we could present a superior model that would provide for a more integrated experience as well as let the customer leverage the investment in the system even before the migration was complete.

The idea in and of itself is not new, and various prototypes of this have been kicking around inside of Sun under various monikers ("brain slug", "hoover", etc) without ever becoming a complete product. When Adam and myself sat down shortly before the initial launch of the product, we decide we could do this without too much work by integrating the functionality directly in the kernel. The basic design requirements we had were:

We must be able to migrate over standard data protocols (NFS) from arbitrary data sources without the need to have special software running on the source system.
Migrated data must be available before the entire migration is complete, and must be accessible with native performance.
All the data to migrate the filesystem must be stored within the filesystem itself, and must not rely on an external database to ensure consistency.

With these requirements in hand, our key insight was that we could create a "shadow" filesystem that could pull data from the original source if necessary, but then fall through to the native filesystem for reads and writes one the file has been migrated. What's more, we could leverage the NFS client on Solaris and do this entirely at the VFS (virtual filesystem) layer, allowing us to migrate data between shares locally or (eventually) over other protocols as well without changing the interpositioning layer. The other nice thing about this architecture is that the kernel remains ignorant of the larger migration process. Both synchronous requests (from clients) and background requests (from the management daemon) appear the same. This allows us to control policy within the userland software stack, without pushing that complexity into the kernel. It also allows us to write a very comprehensive automated test suite that runs entirely on local filesystems without need a complex multi-system environment.

So what's better (and worse) about shadow migration compared to other migration strategies? For that, I'll defer to the documentation, which I've reproduced here for those who don't have a (virtual) system available to run the 2009.Q3 release:

Migration via synchronization

This method works by taking an active host X and migrating data to the new host Y while X remains active. Clients still read and write to the original host while this migration is underway. Once the data is initially migrated, incremental changes are repeatedly sent until the delta is small enough to be sent within a single downtime window. At this point the original share is made read-only, the final delta is sent to the new host, and all clients are updated to point to the new location. The most common way of accomplishing this is through the rsync tool, though other integrated tools exist. This mechanism has several drawbacks:

The anticipated downtime, while small, is not easily quantified. If a user commits a large amount of change immediately before the scheduled downtime, this can increase the downtime window.
During migration, the new server is idle. Since new servers typically come with new features or performance improvements, this represents a waste of resources during a potentially long migration period.
Coordinating across multiple filesystems is burdensome. When migrating dozens or hundreds of filesystems, each migration will take a different amount of time, and downtime will have to be scheduled across the union of all filesystems.

Migration via external interposition

This method works by taking an active host X and inserting a new appliance M that migrates data to a new host Y. All clients are updated at once to point to M, and data is automatically migrated in the background. This provides more flexibility in migration options (for example, being able to migrate to a new server in the future without downtime), and leverages the new server for already migrated data, but also has significant drawbacks:

The migration appliance represents a new physical machine, with associated costs (initial investment, support costs, power and cooling) and additional management overhead.
The migration appliance represents a new point of failure within the system.
The migration appliance interposes on already migrated data, incurring extra latency, often permanently. These appliances are typically left in place, though it would be possible to schedule another downtime window and decommission the migration appliance.

Shadow migration

Shadow migration uses interposition, but is integrated into the appliance and doesn't require a separate physical machine. When shares are created, they can optionally "shadow" an existing directory, either locally (see below) or over NFS. In this scenario, downtime is scheduled once where the source appliance X is placed into read-only mode, a share is created with the shadow property set, and clients are updated to point to the new share on the Sun Storage 7000 appliance. Clients can then access the appliance in read-write mode.

Once the shadow property is set, data is transparently migrated in the background from the source appliance locally. If a request comes from a client for a file that has not yet been migrated, the appliance will automatically migrate this file to the local server before responding to the request. This may incur some initial latency for some client requests, but once a file has been migrated all accesses are local to the appliance and have native performance. It is often the case that the current working set for a filesystem is much smaller than the total size, so once this working set has been migrated, regardless of the total native size on the source, there will be no perceived impact on performance.

The downside to shadow migration is that it requires a commitment before the data has finished migrating, though this is the case with any interposition method. During the migration, portions of the data exists in two locations, which means that backups are more complicated, and snapshots may be incomplete and/or exist only on one host. Because of this, it is extremely important that any migration between two hosts first be tested thoroughly to make sure that identity management and access controls are setup correctly. This need not test the entire data migration, but it should be verified that files or directories that are not world readable are migrated correctly, ACLs (if any) are preserved, and identities are properly represented on the new system.

Shadow migration implemented using on-disk data within the filesystem, so there is no external database and no data stored locally outside the storage pool. If a pool is failed over in a cluster, or both system disks fail and a new head node is required, all data necessary to continue shadow migration without interruption will be kept with the storage pool.

In a subsequent post, I'll discuss some of the thorny implementation detail we had to solve, as well as provide some screenshots of migration in progress. In the meantime, I suggest folks download the simulator and upgrade to the latest software to give it a try.