Also in Solaris 10 01/06 (aka. Update 1)

Solaris 10 Update 1 has shipped. There are nifty things like new-boot in there, but here's a small, subtle, but perhaps blog-worthy entry.

RFC 3947 NAT-Traversal is now part of Solaris. The RFCs were published literally days after the Solaris 10 development gate closed its doors for anything but critical bug fixes. We had used the draft-09 version of NAT-T for Solaris 10, but now we have the RFC-compliant one, which should insure maximum interoperability.

Not a big deal, but IPsec-savvy folks out there ought to know.

It's time to enjoy my Christmas/New-Year's break. Enjoy your own end-of-year activities (whatever they are) and catch you in 2006.

Put IPsec to work in YOUR application

Hello coders!

Most people know that you can use ipsecconf(1m) to apply IPsec policy enforcement to an existing application. For example, if you wish to only allow inbound telnet traffic that's under IPsec protection, you'd put something like this into /etc/inet/ipsecinit.conf or other ipsecconf(1m) input:

# Inbound telnet traffic should be IPsec protected
{ lport 23 } ipsec { encr_algs any(128..) encr_auth_algs md5 sa shared}
             or ipsec { encr_algs any(128..) encr_auth_algs sha1 sa shared}

Combine that with appropriate IKE configuration or manual IPsec keys, and you can secure your telnet traffic against eavesdropping, connection hijacking, etc.

For existing services, using ipsecconf(1m) is the most expedient way to bring IPsec protection to bear on packets.

For new services, or services that are being modified anyway, consider using per-socket policy as an alternative. Some advantages to per-socket policy are:

• Per-socket policy is stored internally in network session state (the conn_t structure in OpenSolaris). Entries from ipsecconf(1m) are stored in the global Security Policy Database (SPD). No global SPD entries means lower latency for fresh flow creation, and less lock acquisition.


• Per-socket bypass means fewer bypass entries in global SPD. If I bypass remote-port 80 using ipsecconf(1m), I can, in theory, enter the system with a remote TCP packet with port=80. There's an RFE (6219908) to work around this, but per-socket is still quicker. I'd love a web proxy with the ability to set per-socket bypass.

The newly SMF-ized inetd(1m) would be a prime candidate for per-socket policy. See RFE 6226853, and this might be something someone in the OpenSolaris community would like to tackle!

Let's look at the ipsec_req_t structure that's been around since Solaris 8 in /usr/include/netinet/in.h:

/*
 * Different preferences that can be requested from IPSEC protocols.
 */
#define IP_SEC_OPT              0x22    /* Used to set IPSEC options */
#define IPSEC_PREF_NEVER        0x01
#define IPSEC_PREF_REQUIRED     0x02
#define IPSEC_PREF_UNIQUE       0x04
/*
 * This can be used with the setsockopt() call to set per socket security
 * options. When the application uses per-socket API, we will reflect
 * the request on both outbound and inbound packets.
 */

typedef struct ipsec_req {
        uint_t          ipsr_ah_req;            /* AH request */
        uint_t          ipsr_esp_req;           /* ESP request */
        uint_t          ipsr_self_encap_req;    /* Self-Encap request */
        uint8_t         ipsr_auth_alg;          /* Auth algs for AH */
        uint8_t         ipsr_esp_alg;           /* Encr algs for ESP */
        uint8_t         ipsr_esp_auth_alg;      /* Auth algs for ESP */
} ipsec_req_t;

The ipsec_req_t is a subset of what one can specify with ipsecconf(1m) in Solaris 9 or later, but it matched what one could do with Solaris 8's version. Algorithm values are derived from PF_KEY (see /usr/include/net/pfkeyv2.h for values), as below. One could also use getipsecalgbyname(3nsl). If I wish to set a socket to use ESP with AES and and MD5, I'd set it up as follows:

        int s;  /* Socket file descriptor... */
        ipsec_req_t ipsr;

        .....

        /* NOTE: Do this BEFORE calling connect() or accept() for TCP sockets. */
        ipsr.ipsr_ah_req = 0;
        ipsr.ipsr_esp_req = IPSEC_PREF_REQUIRED;
        ipsr.ipsr_self_encap_req = 0;
        ipsr.ipsr_auth_alg = 0;
        ipsr.ipsr_esp_alg = SADB_EALG_AES;
        ipsr.ipsr_esp_auth_alg = SADB_AALG_MD5HMAC;
        if (setsockopt(s, IPPROTO_IP, IP_SEC_OPT, &ipsr, sizeof (ipsr)) == -1) {
                perror("setsockopt");
                bail();  /* Ugggh, we failed. */
        }
        /* You now have per-socket policy set. */

Notice I mentioned setting the socket option BEFORE calling connect() or accept? This is because of a phenomenon we implement called connection latching. Basically, connection latching means that once an endpoint is connect()-ed, the IPsec policy (whether set per-socket or inherited from the state of the global SPD at the time) latches in place. We made this decision to avoid keeping policy-per-datagram state for things like TCP retransmits.

One thing per-socket policy does not address is the case of unconnected datagram services. In a perfect world, we could have IPsec policy information percolate all the way to the socket layer, where an application can make fully-informed per-datagram decisions on whether or not a particular packet was secured or not. It's a hard problem, requiring XNET sockets (to use sendmsg() and recvmsg() with ancillary data).

BTW, if you want to bypass whatever global entries are in the SPD, you can zero out the structure, and set all three (ah, esp, self_encap) action indicators to IPSEC_PREF_NEVER. You need to be privileged (root or "sys_net_config") to use per-socket bypass, however.

So modulo the keying problem (setting up IKE or having both ends agree on IPsec manual keys), you can put IPsec to work right in your application. In fact, if you use IKE, you can let IKE sort out permissions and access control (by using PKI-issued certificates, self-signed certificates, or preshared keys) and have policy merely determine the details of the protection required.

---

EDITED: This entry brought to you by the Technorati tags IPsec, Solaris, and OpenSolaris.

Sooner than I thought!

Well, it looks like our friends on the OpenSolaris source team have been working more quickly than I thought. If you look at certain files (like the ESP source), they've been mysteriously fleshed-out. While crypto hasn't been sorted out, some other IPsec-specific files are now present in the tree!

Since crypto hasn't shown up yet, I guess we can't fully celebrate, but once it does, maybe Bill or I will have some more to say on the IPsec source.

COMING SOON: Full IPsec kernel code

AWESOME... TOTALLY AWESOME!

Pardon my channelling of Jeff Spicoli, but Mike Kupfer is working on delivering the with-kernel-crypto build of OpenSolaris. This means that OpenSolaris users will have a working implementation of ESP at their fingertips. It also means Bill and I can (time permitting... uggh) give some more details and explanations about what's going on under the covers.

Watch this space for more!

PF_KEY in Solaris, or "Dude, Where's My Spec?"

An OpenSolaris IPsec Hello

Hi! Those of you visiting here probably know I'm one of the IPsec guys (actually, I'm the original IPsec guy) here in Solaris-land. Bill may also have some stuff to say about the IPsec source in Solaris.

The kernel source for IPsec (AH, ESP, and the internal databases) lives in usr/src/uts/common/inet/ip/, because we're an integral part of our IP implementation. I should warn you now that there's a mixture of STREAMS boundaries and function calls between the different parts of the IPsec subsystem. It used to be almost all STREAMS, because of broken US Export restrictions (across all political party lines, BTW). We figured we could sell it as exportable to the powers that be more easily if we used a "general-purpose interface" which allowed for easy module perforation for moving our data around. As the restrictions loosened up, we were able to streamline things somewhat. We hope to do even more now that OpenSolaris is available. There are bits and pieces of the actual Solaris IPsec missing from OpenSolaris (especially from ESP) that will show up on OpenSolaris soon as well, now that we're officially open-source. (It's a bit of a chicken & egg problem.)

This entry will be discussing the PF_KEY implementation in Solaris. I assume you know something about how IPsec works, have read RFC 2367, and have a handle on TCP/IP protocol suite principles.

A Brief PF_KEY Synopsis

PF_KEY is analagous to the PF_ROUTE routing socket. See Keith Sklower's Radix-Tree paper at his site for the introduction to routing sockets. Where the routing socket manipulates IP forwarding entries (or routes), the PF_KEY socket manipulates IPsec Security Associations (SAs). A user-space application sends a message to the kernel telling it to ADD, DELETE, or UPDATE SAs, and the kernel sends back a message indicating either success or failure.

The paper makes mention of a message that's little-used in most PF_ROUTE implementations -- RTM_RESOLVE. RTM_RESOLVE allows a user-space application to resolve an address, e.g. a user-space ARP. This inspired PF_KEY's similar message,
SADB_ACQUIRE, which is used to tell a user-space key management (KM) daemon that an outgoing IPsec SA is needed. RFC 2367 has the specification for a PF_KEY socket.

Solaris Changes from RFC 2367

Most, if not all, existing PF_KEY implementations either alter or add to the message types in RFC 2367. Most changes were made because:

• RFC2367 does not mesh well in to some KM protocols (esp. IKEv1).


• The UNIX errno space is not sufficient to describe some failures.


• Some implementors thought PF_KEY would be a suitable place to put their IPsec Security Policy Database (SPD) manipulations.

Solaris addresses the last bullet by introducing a separate PF_POLICY socket for SPD manipulation. The other issues, however, were a problem for us.

All of our changes to PF_KEY were a direct result of implementing The Internet Key Exchange (IKE) as part of our work in Solaris 9. They are summarized below:

• Extended ACQUIRE - Instead of sending up a message for every IPsec SA that is not present, send up a list of what is needed to protect the packet to the listening KM daemon. This allows a packet that requires AH and ESP to express that protection in one ACQUIRE message.


• Extended REGISTER - Goes hand-in-hand with Extended ACQUIRE. You tell the kernel that you can handle the Extended ACQUIRE.


• Inverse ACQUIRE - The closest to policy manipulation we come in PF_KEY... it's a one-time consultation of the IPsec SPD, and you get as an answer an Extended ACQUIRE, just like if a outbound packet was triggering it. This is useful for IKE responders, and for diagnostic listeners on the PF_KEY socket.


• Diagnostic codes - EINVAL is a frequently occurring value for sadb_msg_errno. Was it a weak DES key? Was it a botched sockaddr structure? The reserved field in struct sadb_msg now contains useful extra data when an EINVAL occurs.


• typedefs - It's easier to type sadb_ext_t instead of struct sadb_ext.


• 64-bit alignment - RFC 2367 claims all PF_KEY structures can be aligned on 64-bit boundaries. In Solaris, we force it to happen. That's why net/pfkeyv2.h has a lot of unions in most of its structure definitions.

But Dan... weren't you an author of RFC 2367?!?

Yes I was. Hence the question: Dude, Where's My Spec?

I wasn't allowed (yes, I'm serious; and no, it had nothing to do with any government interference) to work on IPsec or IKE when I first got to Sun, but the RFC was work that was a continuation from my previous job. In hindsight, I think we should've been paying more attention to the customers (authors of KM daemons, of which I'd be one someday). I was wrapped up in non-IPsec work at Sun when I wasn't working on what would become RFC 2367, and I split my attention in a non-optimal fashion.

Enough yapping, let's see some code!

The first place to look is usr/src/uts/common/net/pfkeyv2.h, which gets deposited into /usr/include/net/ on a running system. You'll notice every structure that doesn't have a field of type uint64_t will have a union in it. Here's the base PF_KEY message:

typedef struct sadb_msg {
        uint8_t sadb_msg_version;       /* Version, currently PF_KEY_V2 */
        uint8_t sadb_msg_type;          /* ADD, UPDATE, etc. */
        uint8_t sadb_msg_errno;         /* Error number from UNIX errno space */
        uint8_t sadb_msg_satype;        /* ESP, AH, etc. */
        uint16_t sadb_msg_len;          /* Length in 64-bit words. */
        uint16_t sadb_msg_reserved;     /* must be zero */
/*
 * Use the reserved field for extended diagnostic information on errno
 * responses.
 */
#define sadb_x_msg_diagnostic sadb_msg_reserved
        /* Union is for guaranteeing 64-bit alignment. */
        union {
                struct {
                        uint32_t sadb_x_msg_useq;       /* Set by originator */
                        uint32_t sadb_x_msg_upid;       /* Set by originator */
                } sadb_x_msg_actual;
                uint64_t sadb_x_msg_alignment;
        } sadb_x_msg_u;
#define sadb_msg_seq sadb_x_msg_u.sadb_x_msg_actual.sadb_x_msg_useq
#define sadb_msg_pid sadb_x_msg_u.sadb_x_msg_actual.sadb_x_msg_upid
} sadb_msg_t;

Notice that every extra field that is not in RFC 2367 uses the _X_ naming convention. In the case above, we took the two uint32_ts and merged them into a union with a uint64_t so that we can force 64-bit alignment on sadb_msg_t. This makes PF_KEY message manipulations 64-bit happy.

If you look at an extension that has a 64-bit type in it already, you'll see that there's no alignment-forcing union inserted into the definition:

typedef struct sadb_lifetime {
        uint16_t sadb_lifetime_len;
        uint16_t sadb_lifetime_exttype;         /* SOFT, HARD, CURRENT */
        uint32_t sadb_lifetime_allocations;
        uint64_t sadb_lifetime_bytes;
        uint64_t sadb_lifetime_addtime; /* These fields are assumed to hold */
        uint64_t sadb_lifetime_usetime; /* >= sizeof (time_t). */
} sadb_lifetime_t;

(Yes, PF_KEY is Y2038-ready, as long as the KM application is compiled as 64-bit.)

For people interested in applications that use PF_KEY, I'd suggest investigating the ipseckey(1m) command. Its source can be found in usr/src/cmd/cmd-inet/usr.sbin/ipseckey.c. This program does not use command-line editing yet, but as an example of a PF_KEY consumer, it does the job.

All relevant PF_KEY internal-implementation headers live in usr/src/uts/common/inet. Relevant header files are:

• ipsec_info.h
  - The structures for M_CTL messages prepended to data that gets passed around between IPsec STREAMS modules. The keysock consumer interface definitions are important here. Note the keysock_in_t, especially ks_in_extv[]. This vector allows easy access to all PF_KEY extension headers. (As we remove STREAMS from IPsec, this file will shrink. If all goes well, all that will remain (in some form) are IPSEC_IN and IPSEC_OUT messages, and that's because you can't enforce policy without some form of packet tagging.)


• keysock.h
  - The keysock driver implements the PF_KEY socket interface at its most basic. A keysock_t represents an open PF_KEY socket, and a keysock_consumer_t represents a consumer of PF_KEY messages (i.e. AH and ESP).

All relevant PF_KEY internal-implementation source lives one level down from the headers, in the ip/ directory. They are:

• keysockddi.c
  - DDI and module loading glue.


• keysock.c
  - STREAMS driver that implements the PF_KEY socket as /dev/keysock. It is also a STREAMS module that sits atop AH and ESP listening for their messages.


• sadb.c
  - Where IPsec's Security Association Database (SADB) is mostly implemented. You'll notice an ip_sadb.c file, because we want the fast-path lookups to be in IP without going through modstubs.

Most of this entry will be spent in keysock.c. Other portions of the subsystem will either be visited by one of Team IPsec when we have cycles, or if there are enough requests.

keysock either handles messages from a PF_KEY socket, or from the SADB and its consumers. The heavy lifting for user-generated PF_KEY messages is in keysock_parse().

The first thing keysock_parse() does is perform some reality checks. First off, the actual data length should match what's in the sadb_msg_len field. (Note the first of may SADB_nnTOmm() macros, for converting units of 64-bits to units of 8-bits, etc.) The first really interesting part of reality-checking is the "extension vector", or as it's called in the IPsec code, the extv. A PF_KEY message has a base header, followed by one or more extension headers. Let me quote this section from RFC 2367:

There MUST be only one instance of a extension type in a message. (e.g.
Base, Key, Lifetime, Key is forbidden). An EINVAL will be returned if there
are duplicate extensions within a message.

The keysock code takes this to heart in keysock_get_ext(). The extv is a vector of sadb_ext_t pointers, where the specific extension type (SADB_EXT_foo) can be found by merely indexing into the vector by that value. Say you want the sadb_sa_t extension:

        sadb_sa_t *sa;

        sa = (sadb_sa_t *)extv[SADB_EXT_SA];

The above code snippet shows what you need to do. As we generate the extv, if we see a collision, we return EINVAL (with an appropriate diagnostic). We do not enforce extension ordering inbound or outbound. Once keysock is done with first-pass reality-checks, the extv is sent around (as part of the KEYSOCK_IN M_CTL that is prepended to the data) to all who need it.

Most messages are shuttled off to keysock_passdown() for sending off via STREAMS to either AH or ESP. Unusual inbound messages
for keysock are the SADB_REGISTER, SADB_FLUSH, SADB_DUMP, SADB_ACQUIRE, and SADB_X_INVERSE_ACQUIRE ones.

SADB_REGISTER sets socket state, as well as informs consumers about the register. If an SADB_REGISTER is sent for a specific SA type (e.g. ESP, AH), then the message is treated like a common-case message, except that when its reply arrive, keysock_t state is altered to indicate a registered socket. If the sadb_msg_satype is set to 0, then the message is an EXTENDED register, and an extended-register extension (sadb_x_ereg_t) is required. keysock converts the 0-terminated list of one-byte values into a bit vector internally. Then it sends the message to consumers like a normal REGISTER.

An inbound SADB_ACQUIRE can be used to signal other KM applications. (If PF_KEY is used to keep keys in the kernel for user-space consumers.) The more common case, however, is a negative ACQUIRE, which means a KM negotiation failed and the internal ACQUIRE record (more on this in a bit) needs to be cancelled.

SADB_FLUSH and SADB_DUMP messages need to lock down the keysock module until their respective operations are finished. FLUSH doesn't take as long, but DUMP needs to keep track of all consumer-originated replies until the consumer indicates it is done.

SADB_X_PROMISC merely changes some keysock state. It never goes to a consumer.

The SADB_X_INVERSE_ACQUIRE handling is a glimpse of things to come for keysock. It does not use the keysock_passdown() method of calling a consumer. It instead calls directly into IPsec (and if we had other in-kernel consumers, it would directly call to those) and returns a message to the user immediately.

The keysock_rput() function handles all messages from consumers. The KEYSOCK_OUT portion of the switch checks for FLUSH and DUMP messages, and releases the clamps on keysock if the final message for a FLUSH or a DUMP has been received. Otherwise, the keysock_passup() does the work.

keysock_passup() is conceptually much simpler than
keysock_parse(). It merely has to make one of three delivery decisions:

• Do I deliver to all PF_KEY sockets?


• Do I deliver to all registered PF_KEY sockets?


• Do I deliver merely to the sender?

The more interesting parts of PF_KEY are handled inside the SADB code (sadb.c) and in the consumers. Those will be the subject of one or more other entries, because of all of the interaction with the IPsec SADB.

---

This entry was brought to you by the Technorati Tags OpenSolaris and Solaris.

No hard-drive wipe needed for Solaris x64 support!

Someone sent me this link about the new 64-bit Windows XP. There's some interesting text on that page I would like to quote here (assuming the link doesn't change...):

Back up your data and settings. Windows XP Professional x64 Edition requires a "clean installation," meaning the contents of your hard drive will be erased during the installation.

I can tell you with confidence that if you go from Solaris 9 on x86 to Solaris 10 with both x86 and x64, you will NOT have to erase your hard drive (assuming you've enough room for the new x64-native binaries, and you probably do already if you're in possession of an x64-capable box).

US Navy and Open Source - how last decade! :)

An old friend/colleague sent me this url:

http://linux.slashdot.org/article.pl?sid=05/03/26/028200&tid=103&tid=106

And in particular wanted me to read this response to the posting in question.

The response is pretty good, and right up front talks about two projects (see here and here respectively) that I had the pleasure to work on.

The work we did at NRL lives on today in many places either directly or indirectly. We have our war stories (no pun intended), including locking horns with GPL-bigots who did not like that we were going to put out our software with a BSD license, in spite of our funding sources' requirements that our work be directly transferable into commercial products.

So raise your glasses to the new, folks, but don't forget the old. And also don't forget that just because you remember history, it doesn't mean that you can't make the good parts repeat!

Grudge Match (TM) to Jump the Shark (TM)?

Now that I've got my "I like end-systems doing the heavy lifting" rant out there, it's time to lighten up for a bit.

One of my favorite humor sites is the World-Wide Web Fights - Grudge Match page. They take the 4th-grade (age 9 for our international readers) concept of "oooh, what if the Enterprise took on the Death Star?" to its maximum silliness and fun. I've been a regular reader (and occasional Bronze-Medal contributor as well) for most of its 10 years of existence.

So now, their current match has the interesting property of having cursed-actor Ted McGinty as one of the contestants... and their hint for their next match is "Jumping the Shark", which I'll let readers follow this link to see background information on the term.

I was concerned that one of my favorite sites would not make it to its 10-year anniversary. So I sent a mail, and got this enigmatic response (full poor-form top-posted reply included):

To: danmcd@kebe.com
Subject: Re: You're not going off the air, are you?

I can neither confirm nor deny these rumors.

- Brian (tm)

Dan McDonald wrote:
> 
> Please tell me it's a 10-year anniversary stunt... bringing Ted McGinley
> aboard, and then teasing us about Jumping the Shark... please, please, please
> don't go off the air!
> 
> A fan since 1995 (and recipient of both a Final Word and a Bronze Grudgie),
> Dan McD.

-- 
- Brian Wright
WWWF Grudge Match
http://www.grudge-match.com/

"So consistently funny it ought to be developed into
a series for Comedy Central!"
                                - Entertainment Weekly

So I'm hoping it's just a stunt - and a funny one at that. If you're semi-knowledgible about American pop culture, you'll probably bust a gut at some of the matches on this site. Please click that link above and visit... before it's too late?!?

I hate middleboxes...

Ever since I worked on one of the early IPv6 implementations, I've had a sour taste in my mouth for any node in the middle of a network that did more than forward packets (or maybe one or two other things, which I'll talk about later).

Nodes that are in the middle of the network and do not merely forward packets I will call middleboxes in this entry. The granddaddy middlebox was the firewall. These were designed to prevent classes of packets (often very large classes of packets) from reaching a network that was connected to the Internet. Given the poor state of end-node implementations (anyone remember when sendmail was the biggest source of vulnerabilities?) at the time, firewalls were a sensible solution to the problem.

To cope with people demanding Internet services while still being behind their safe firewall, the proxy came into play. These boxes pretend to be servers for machines on the inside of a firewall, and clients to machines outside the firewall.

To cope with address space "shortages" (especially when the addresses in question might never be seen on the Internet, thanks to firewalling), the Network Address Translator (NAT) was invented. To the outside world, the NAT presents a small number of Internet addresses (often that number is one). Behind the NAT, there is a whole network (usually carved out of addresses that are reserved just for this
purpose) of nodes.

These nodes often cause as many problems as they solve. A server cannot depend on IP addresses (whether or not that's good design is a discussion for another time) any more (even if they're secured with IPsec). IPsec itself does not work without the NAT-Traversal hack (now available in Solaris 10). Anything using a proxy is prone to failure if the proxy is flaky - my outbound ssh sessions often disappear because the proxy times them out. With most middleboxes, state that should only be on the two the end-nodes is now distributed to a third, or more. And often, that third node is out of the user's control.

There are some good cases for having some set of middlebox functionality. Alec Muffet has convinced me with
defense-in-depth arguments that some set of firewall functionality is useful in various environments. I can understand (but still despise) NAT in places where stingy ISPs only allow one address per subscriber, or worse, do NAT for you. We are where we are today because a lot of this was a market-driven reaction to poor implementations of network service applications (and in some cases the OS as well). We in OS and network-service development can, of course, fix this if we try hard enough. I'd suggest looking at recent articles about what historical development practices have wrought.

I think we all finally understand what our responsibilities are as end-node engineers. The sooner we act on those responsibilities, the sooner we can remove any excuses for the new classes of bridge trolls to exist. (Did I mention anyone who likes this blog should visit David Reed's excellent site?) And if any folks out there are dissatisfied with their current choice, I hope that they realize that the market will provide other choices.