How to tell when a performance project succeeds?

The Volo project is an effort to improve the interface between sockets and any socket-driven subsystems, including the TCP/IP stack. During their testing, they panicked during some IPsec tests. See this bug for what they reported.

In our IPsec, we have LARVAL IPsec security associations (SAs). These are SAs that reserve a unique 32-bit Security Parameters Index (SPI), but have no other data. If a packet arrives for a LARVAL SA, we queue it up, so that when it gets filled in by key management, the packet can go through. We do this because of the IKE Quick Mode Exchange, which looks like this:

        INITIATOR                               RESPONDER
        ---------                               ---------

        IKE Quick Mode Packet #1  ------->

                                <----------     IKE Quick Mode Packet #2

        IKE Quick Mode Packet #3  -------->

Now once the initiator receives Quick Mode packet #2, it has enough information to transmit an IPsec-protected packet. Unfortunately, the responder cannot finish completing its Security Association entries until it receives packet #3. It is possible, then, that the initiator's IPsec packet may arrive before the responder has finished processing IKE. Let's look at the packets again:

        INITIATOR                               RESPONDER
        ---------                               ---------

        IKE Quick Mode Packet #1  ------->

                                <----------     IKE Quick Mode Packet #2

        ESP or AH packet        ---------->     Does this packet...

        IKE Quick Mode Packet #3  -------->     ... get processed after my
                                                receipt of #3, also after
                                                which I SADB_UPDATE my
                                                inbound SA, which changes it
                                                from LARVAL to MATURE?

Now the code that queues up an inbound IPsec packet for a LARVAL SA is sadb_set_lpkt(), as was shown in the bug's description. It turns out there was a locking bug in this function - and we even had an ASSERT()-ion that the SA manipulated by sadb_set_lpkt() was always larval. The problem was, we discounted the possibility of IKE finishing between the detection of a LARVAL SA and the actual call to sadb_set_lpkt().

The Volo project improved UDP latency enough so that the IKE packet wormed its way up the stack and into in.iked faster than the concurrent ESP or AH packet. The aformentioned ASSERT() tripped during Volo testing, because we did not check the SA's state while holding its lock. Had we, we could tell that the LARVAL SA was promoted to ACTIVE, and we could go ahead and process the packet.

This race condition was present since sadb_set_lpkt() was introduced in Solaris 9, but it took Volo's improved performance to find it. So hats off to Volo for speeding things up enough to find long-dormant race conditions!

---

Addendum - IKEv2 does not have this problem because its equivalent to v1's Quick Mode is a simpler request/response exchange, so the responder is ready to receive when it sends the response back to the initiator.

Racoon2 on OpenSolaris - first tiny steps

NOTE: A version of this was sent to the racoon2-users alias also.

I've been spending some of my time bringing up racoon2 (an IKEv2 and IKEv1 daemon) on OpenSolaris.

Because of vast differences in PF_KEY implementations between OpenSolaris and other OS kernels, I've spent my racoon2 time actually getting IKEv1 to work first, instead of IKEv2. Right now, what's working is:

• IKEv1 initiates and derives IPsec SAs for single-algorithm IPsec policies.

That's it! IKEv1 responder needs work, as does all of IKEv2, as does work
for multiple-choice of algorithms. But there's enough change in there to say
something now.

ARCHITECTURAL DIFFERENCES

The most noteworthy change in the OpenSolaris work so far is that literally
there's no spmd (a separate IPsec SPD daemon racoon2 uses) required for now. This is because:

• We don't have the indirection between ACQUIREs and the appropriate policy entry. Our extended ACQUIREs contain everything needed to construct a proposal. There's no SPD consultation required with an OpenSolaris ACQUIRE.


• Our responder-side logic uses inverse-ACQUIRE, which will provide the same structure as ACQUIRE w.r.t. proposal construction. This is the closest we get to needing something like spmd, and given its syntactic equality to an extended ACQUIRE, we can use it on rekeying if the responder initiates the next time.

If spmd serves another purpose, we will revisit it. As it stands, however, I cannot see us using it.

CODE DIFFERENCES

In OpenSolaris, we use the "webrev" tool to generate easy-to-review web pages
with diffs of all varieties. The webrev for what I have so far in racoon2 is
available at:

http://cr.opensolaris.org/~danmcd/racoon2-opensolaris/

Feel free to make comments or suggestions about what I've done.

How to rescue data from an iBook with thermal problems

My wife Wendy has had her iBook G4 for not-quite four years now. We had to return it once before via AppleCare due to thermal problems. Well, the thermal problems are back, and this time, there's no AppleCare for us to invoke. I managed to get the machine to behave itself only after leaving it powered off for a bit, but then it would lock again. I'd heard stories about putting computers in refrigerators to keep them cool enough to run. I never thought I'd try it myself.

We do, however, have a freezer in the basement. So check this out:



I managed to get Wendy's home-directory off, and that's what mattered. I'm heading off to the Apple Store to get a new MacBook (thank goodness for the just-arrived George-and-Nancy "Will you be my friend with this stimulus?" check). I hope to do the frozen data transfer one more time to bootstrap the new MacBook.

Can't let this one slip by

I'm not sure if this picture represents extreme stupidity in the protestor, or if it's merely a clever use of Photoshop to make a joke. If the latter, it's pretty funny. If the former... I have NO idea what to say.

Thanks to Fake Steve Jobs for bringing this to my attention.

BTW, for folks who need a quick history lesson - click here and follow your favorite search hit.

Kebe's Home Data Center (or f''(Bart's new home server))

A little over a year ago, Bart Smaalders blogged about his new home server. Subsequently Bill built a similarly-configured one. (I thought that he had blogged about his too, but he hadn't.)

I'd been toying with the idea of following in Bill's and Bart's footsteps for some time. A recent influx allowed me to upgrade lots of home technology (including a new Penryn-powered MacBook Pro), and finally allowed me to build out what I like to think of as my home data center. I mention f''(Bart's...) because this box really is the second-derivative of Bart's original box (with Bill's being the first-derivative).

And the starting lineup for this box is:

• An AMD Opteron Model 185 - I was lucky enough to stumble across one of these. 2 cores of 2.6GHz AMD64 goodness.


• A Tyan S2866 - I bought the one with two Ethernet ports - one nVidia (nge) and one Broadcom (bge). It has audio too, but I haven't tested it as I've my Macs for such things. It has all of the goodies Bart mentioned, but I *think* that the SATA might be native now. (Please comment if you know.)


• 2GB ECC RAM - with room for two more if need be.


• A two-port old Intel Pro Ethernet 10/100 - good thing the driver (iprb) for this is now open-source. I'll explain why I need four Ethernet ports in a bit.


• Two Western Digitial "green" 750GB SATA drives Each drive has 32GB root partitions (yes that's large, until Indiana matures, though, I'll stick with UFS roots), 4 GB swap (for core dumps), and the remaining large areas combine to make one mirrored ZFS pool with ~700 decimal GB of storage.


• A cheap MSI nVidia 8400GS - It's more than enough to drive my 1920x1200 display.


• An overkill Antec 850W power supply - obtained for only $100 from the carcass of CompUSA.


• A Lian Li U60 case - My brother-in-law, who has years in the trenches of PC care, feeding, and repair, recommended Lian Li to me. It has all the space I need and more for drives, and its fan layout is pretty comprehensive. Since this box lives in my office, noise isn't that much of an issue.


• OpenSolaris build 83 - While I'm pumped about what's going on with Indiana it's still under development, and I want something a bit more stable.
  

So why four ethernet ports (covering three drivers)? Well, like Indiana, Crossbow is exciting, but not yet integrated into the main OpenSolaris tree. I do, however, very much like the idea of Virtual Network Machines and I'll be using these four ports to build three such machines on this server using prerequisite-to-Crossbow IP Instances. Two ports will form the router zone. The router will also be a firewall, and maybe an IPsec remote-access server too. With Tunnel Reform in place, I can let my or my wife's notebook Macs access our internal home network from any location. One port will be the public web server, and assuming Comcast doesn't screw things up too badly on their business-class install, the new home of www.kebe.com. The last port will be the internal-server and global-zone/administrative station. All of that ZFS space needs to be accessible from somewhere, right?

I'd like to thank Bart and Bill for the hardware inspiration, and to my friends in OpenSolaris networking for offering up something I can exploit immediately to create my three machines in one OpenSolaris install. I'll keep y'all informed about how things are going.

Who am I rooting for this weekend? Michigan!

If the oddsmakers and pundits are to be believed, my two favorite NFL teams (my current-home Patriots and my childhood-home Packers) will be facing each other in a couple of weeks at the Super Bowl. Remember I said *if* the oddsmakers and pundits are right. If they are, I'm going to have to hide my Packers sweatshirt for the rest of January. And worse (well, worse for the football fan in me), I'm going to miss the game 'cause I'm on vacation with my family!

So what am I going to do? Simple: Root for Michigan!

Huh? Sure Dan - Michigan won their bowl game already... showing the SEC that not ALL Big Ten teams are pushovers, and sending Lloyd on with one more bowl win. What's this got to do with the NFL conference championships?

Well let's see here:

• Chargers - Well, nobody's perfect!


• Giants - Amani Toomer, who SI's Peter King calls, "one of the good guys... also one of the underrated guys.


• Packers - Michigan Heisman winner Charles Woodson (who I saw at the 1997 Rose Bowl). The last Heisman winner Michigan had also spent some of his late career with Green Bay... and Desmond Howard took home a Super Bowl ring!


• Patriots - Sitting on the bench at that 1997 Rose Bowl was a 6th round draft pick named Tom Brady.

No matter who you're rooting for this weekend, I've two words for you: GO BLUE!