Kerberos: A Practical Overview of Principals and Keytabs

Reading time: 20 minutes Published date: 25 May 2018

Table of Contents

Kerb Your Enthusiasm - What is this all about?

Despite the readily available documentation on Kerberos, I’ve found that many people are still confused about kerberos concepts such as keytabs, principals and tickets. The issue is compounded by the fact that there are also terms outside of the protocol specification that get thrown around such as “headless keytabs”.

There have already been excellent posts written on this topic, such as this one, and I acknowledge that I may be covering some of the same material. However, I am also going to talk about Kerberos within the context of Redhat IdM / FreeIPA.

Throughout this article, I will also be providing command examples for manipulating kerberos using ktutil for those who run their own KDC , and ipa commands for those using either FreeIPA or Redhat IdM. Windows users - i’ve got nothing :(

What is Kerberos?

Kerberos is simply an authentication protocol specified under RFC4120 that allows clients to be authenticated over a network without sending the client secret over the wire. Kerberos utilises shared-key/symmetric cryptography (analogous to SSL), rather than asymmetric key cryptography (which is used in Public Key Infrastructure). The encrypted keys have a short expiry time, so the tradeoff is that even if keys (in the form of tickets) get compromised, they will become invalidated within a short period of time.

There are a number of implementations of Kerberos - including Active Directory and MIT. It’s possible to run and manage your own KDC (Key Distribution Center), which hands out tickets (TGS), and performs Authentication, such as MIT. Typically corporations tend to use federated solutions which combine a directory service to store all the users and keys, so that authentication using the KDC can be more tightly integrated. This is the case with both Windows Active Directory and Redhat IdM (or FreeIPA if using Open Source).

How does Kerberos Work?

Kerberos works as a third party authentication service. Users request access to a TGT (Ticket Granting Ticket) which they cannot decrypt, and also expires. This ticket is used to request any number of additional tickets to different services the user needs access to. Each service (or application) will then accept or reject the ticket based on the user’s permissions. The application is responsible for authorizing the user and should not rely on the Kerberos server to do so.

There are some great articles on the interwebs explaining all the steps in the workflow of kerberos authentication, so I will encourage you to read this, rather than me attempting to explain it here. I must admit I had to read it at least twice to fully understand the flow.

The first point to note is that when you run kinit, a plaintext request is made to the KDC - requesting a TGT. The details sent to be KDC can be observed with a simple tcpdump listening on the client interface:


15:31:51.546544 IP 10.1.2.102.44668 > 10.1.1.31.88: Flags [P.], seq 4051788158:4051788334, ack 828312661, win 210, options [nop,nop,TS val 57697181 ecr 402095938], length 176
        0x0000:  4500 00e4 02e3 4000 4006 1fab 0a01 0266  E.....@.@......f
        0x0010:  0a01 011f ae7c 0058 f181 617e 315f 0c55  .....|.X..a~1_.U
        0x0020:  8018 00d2 185d 0000 0101 080a 0370 639d  .....].......pc.
        0x0030:  17f7 7f42 0000 00ac 6a81 a930 81a6 a103  ...B....j..0....
        0x0040:  0201 05a2 0302 010a a30e 300c 300a a104  ..........0.0...
        0x0050:  0202 0095 a202 0400 a481 8930 8186 a007  ...........0....
        0x0060:  0305 0040 0100 10a1 1430 12a0 0302 0101  ...@.....0......
        0x0070:  a10b 3009 1b07 6172 7479 636c 65a2 171b  ..0...artycle...
        0x0080:  154d 494c 4b59 5741 5950 4d59 4741 4c41  .MILKYWAY.MYGALA
        0x0090:  5859 2e43 4f4d a32a 3028 a003 0201 02a1  XY.COM.*0(......
        0x00a0:  2130 1f1b 066b 7262 7467 741b 154d 494c  !0...krbtgt..MIL
        0x00b0:  4b59 5741 592e 4d59 4741 4c41 5859 2e43  KYWAY.MYGALAXY.C
        0x00c0:  4f4d a511 180f 3230 3138 3035 3233 3035  OM....2018052305
        0x00d0:  3331 3531 5aa7 0602 0412 08c5 cfa8 0530  3151Z..........0
        0x00e0:  0302 0117                                ....

If you managed to trace the flow, the second point to note is that the TGT (Ticket Granting Ticket) is not directly able to be decrypted by you - the user. This is instead stored in your kerberos cache (location configurable in /etc/krb5.conf), and sent to the TGS (Ticket Granting Server) in its encrypted form to request other service tickets.

The TGT is identified by the service principal named “krbtgt”, which can be viewed by running klist as shown below:

[myuser@myhost ~]$ klist
Ticket cache: FILE:/run/user/krb5cc/krb5cc_myuser
Default principal: myuser@GALAXY.ORG

Valid starting       Expires              Service principal
05/22/2018 15:31:55  05/23/2018 15:31:51  krbtgt/MILK.GALAXY.ORG@GALAXY.ORG

To be more precise, the krbtgt service is actually a string which identifies the TGS (Ticket Granting Service) in the KDC. The composition of this principal is actually defined in the Kerberos RFC:

The principal identifier of the ticket-granting service shall be composed of three parts: the realm of the KDC issuing the TGS ticket, and a two-part name of type NT-SRV-INST, with the first part “krbtgt” and the second part the name of the realm that will accept the TGT. Source

So due to our extremely creative realm naming, the TGT krbtgt/MILK.GALAXY.ORG@GALAXY.ORG indicates that the ticket was issued by a KDC from the realm GALAXY.ORG, and it will be accepted by services in the realm MILK.GALAXY.ORG.

But what exactly is a realm?

Kerberos Realms

According to the RFC:

Each organization wishing to run a Kerberos server establishes its own “realm”. The name of the realm in which a client is registered is part of the client’s name and can be used by the end-service to decide whether to honor a request

So a realm is exactly what it sounds like - a logical grouping of kerberos elements usually tied to an organisation, and mapped to each KDC (Server).

Kerberos allows authentication between realms (i.e between different KDC’s) by configuring trust policies within each server. In which case, tickets issues by one KDC allows access to services within another KDC, for example when we obtained the TGT krbtgt/MILK.GALAXY.ORG@GALAXY.ORG. In most cases, authentication occurs within the same realm.

Realms Vs. Domains

People often get confused between realms and domains. A domain refers to a domain name, which is:

…an identification string that defines a realm of administrative autonomy, authority or control within the Internet. Domain names are formed by the rules and procedures of the Domain Name System (DNS)

Source

So domain names are used by DNS, and realm names are used by the KDC - generally for analogous purposes. They both uniquely identify elements (hostnames for DNS, and principals for KDC) within an organisation.

DNS and Kerberos

When a client performs kinit, how does it know where to send the request to? The client needs to know the KDC server’s hostname.

One method is to explicitly configure the client’s /etc/krb5.conf with the location of the KDC using the kdc, master_kdc, admin_server, and kpasswd_server options. However, this is not very scalable, and if you wanted to change the hostname of the server you would have to update all of the client configs.

Alternatively, the RFC also defines a process called KDC Discovery which can use DNS to retrieve the KDC hostname(s). The latest MIT implementation supports the use of both SRV and URI DNS Records.

If using SRV Records, you will need to configure:

If using URI records,

Source

A URI record is more efficient than the SRV records in certain situations. For example, when a client fails over to other transport types or when a client has to rediscover the master KDC, multiple SRV requests may be triggered compared to a single URI query.

You can read (a lot) more about URI DNS records in its RFC

Kerberos Principals

So far, we have been throwing around this term “principal” quite casually. But what is it?

According to the [MIT Kerberos documentation] (https://web.mit.edu/kerberos/krb5-1.5/krb5-1.5.4/doc/krb5-user/What-is-a-Kerberos-Principal_003f.html):

A Kerberos principal is a unique identity to which Kerberos can assign tickets.

A Kerberos principal is a string which looks like this:

component1/component2/.../componentN@REALM

The principal is made up of an arbitrary number of components, separated by a component separator (typically a “/”), as shown above. The last component is the Realm (we’ll discuss what this is soon)

Typically there are usually 3 components, including the realm. In this format, Kerberos principals can be described as follows:

primary/instance@REALM

The primary is the first part of the principal, and the instance is the qualifier which adds context. The last component is always the REALM, and it is separated from the rest of the principal by the realm separator (usually a “@”). Together, these components can form different types of principals. Principals can be created to represent hosts, users, certificates installed on a host or services running on a host.

In most implementations, the instance is often used to define the host to which the primary (service or user) is bound to. In such cases, the KDC will only accept authentication requests for principals if they originate from the host listed in the instance. For example, an authentication request from the principal HTTP/my.webserver.hostname will be refused if it originates from my.database.hostname.

Service Principals

In the Kerberos world (or realm), a service principal can be thought of as an identifier used to represent a client application running on the OS, such as nfs or yarn.

For example, in IPA, a service principal is defined as follows:

A IPA service represents a service that runs on a host. The IPA service record can store a Kerberos principal, an SSL certificate, or both.` Source

If you re-visit the kerberos authentication flow, you can see that a user can request any number of service tickets (if they have permission to do so) for access using their TGT. With the service ticket, they can access/utilise/manage this service.

I’ve kept the wording vague here because how an application uses a service ticket is entirely up to the application. An application typically configures what type of service principal it can handle. For example, a webserver could be configured to require HTTP/my.webserver.hostname@MY.REALM.NAME.

Having said this, there are some special services that Kerberos knows about out of the box - the krbtgt service was already mentioned in the previous section. The host service is used to represent a host itself - and is often called a “host principal”. We will explore this further on.

When using your own KDC, you can create a service principal by:

# Add a Service Principal using kadmin
## -q option allows a specific kerberos query/command to be executed
## addprinc command adds a kerberos principal
## -r: specifies the default realm to use
## -randkey sets the principal's key to a random value
kadmin -q "addprinc -randkey HTTP/my.webserver.hostname" -r MY.REALM.NAME

# View a Service Principal using kadmin
kadmin -q "getprinc HTTP/my.webserver.hostname" -r MY.REALM.NAME

In Redhat IPA (or FreeIPA the OpenSource version), a service principal can be created as follows:

# Create a Service Principal
ipa service-add HTTP/my.webserver.hostname@MY.REALM.NAME

# Show a Service Principal
ipa service-show HTTP/my.webserver.hostname@MY.REALM.NAME

User Principals

User principals are intuitive enough to understand. They are used to identify users, and use their username as the primary of the principal. Since the creation of all principals follow the same concept, creating them is easy enough:

# Create a user principal using admin
kadmin -q "addprinc -randkey myusername" -r MY.REALM.NAME

In a federated system such as IPA, user principals are created every time a new user is created. A new entry is created in the directory server, with the user’s attributes as well as their client secret key (password) and user principal.

In IPA, a new user is created using the user-add command:

ipa user-add starlord --title="mr" --first="Star" --last"Lord" --principal="starlord@GALAXY.ORG"
## NOTE: The --principal is optional, and is an alias to the one automatically created by the user-add call.

A user’s principal can then be confirmed using user-show:

[admin@myhost ~]$ ipa user-show starlord
WARNING: yacc table file version is out of date
  User login: starlord
  First name: Star
  Last name: Lord
  Home directory: /home/users/starlord
  Login shell: /bin/bash
  Principal name: starlord@GALAXY.ORG
  Principal alias: starlord@GALAXY.ORG
  Email address: star.lord@galaxy.org
  UID: 46589332
  GID: 100
  User authentication types: password
  Account disabled: False
  Password: True
  Member of groups: admins
  Roles: User Administrator
  Indirect Member of group: admin_group
  Indirect Member of Sudo rule: allow_all
  Indirect Member of HBAC rule: allow_admins
  Kerberos keys available: True

Host Principals

As previously mentioned, there is a special service called ‘host’. Service principals containing this service have special uses - they are used to uniquely identify a host, and authenticate network services. For example:

host/milkyway.galaxy.org@GALAXY.ORG

Would uniquely identify the host milkyway.galaxy.org.

Why would you want to identify a host? In the introduction to principals, I mentioned that often principals are often qualified with a hostname in the instance to restrict authentication of the principal to a host (i.e principals of the format HTTP/my.host.name@MY.REALM.NAME).

In order to prove to users and other clients/applications that the service is in fact running on a host that it claims to, the host itself needs to be verifiable and identifiable. Otherwise a service could just claim to run on any host in its principal, and the KDC would have to trust that it is. Host principals fulfill that role.

The fact that host principals contain a fully qualified domain name should alert you to the fact that DNS is somewhere around the corner, and is instrumental to this verification. Don’t worry - we’ll turn the corner in the next section.

Host principals must be unique across a realm - and when created, they register a new host to the KDC. This is essentially like joining the realm. They are also used directly by some services using the network such as OpenSSH to authenticate users logging in.

Host principals can be created similarly to service principals using kadmin:

# Create a host principal using admin
kadmin -q "addprinc -randkey host/milkyway.galaxy.org" -r MY.REALM.NAME

In IPA, hosts are managed using the host-* commands. A host principal is also created when an IPA client is installed on a host using ipa-client-install. You can check the principal for a host using:

[admin@myhost ~]$ ipa host-show milkyway
WARNING: yacc table file version is out of date
  Host name: milkyway.galaxy.org
  Principal name: host/milkyway.galaxy.org@GALAXY.ORG
  Principal alias: host/milkyway.galaxy.org@GALAXY.ORG
  ...
  Password: False
  Member of host-groups: all-servers
  Indirect Member of Sudo rule: allow_all
  Keytab: True
  Managed by: milkyway.galaxy.org

Principal Names and DNS Lookups

When a client provides a service principal to the KDC to authenticate to, the KDC can confirm whether or not this principal has been registered with it. However there is no way for the KDC to know that the principal requested is the one the client intends to talk to.

In order to solve this, most implementations use a DNS lookup to determine the target hostname to then define the service principal that is requested.

For example, the MIT Kerberos implementation says that:

MIT Kerberos clients currently always do forward resolution (looking up the IPv4 and possibly IPv6 addresses using getaddrinfo()) of the hostname part of a host-based service principal to canonicalize the hostname.` Source

It is interesting to note however that the Kerberos RFC says:

Implementations of Kerberos and protocols based on Kerberos MUST NOT use insecure DNS queries to canonicalize the hostname components of the service principal names. … Implementation note: Many current implementations do some degree of canonicalization of the provided service name, often using DNS even though it creates security problems.

The reason being:

…one should not rely on an unprotected DNS record to map a host alias to the primary name of a server, accepting the primary name as the party that one intends to contact, since an attacker can modify the mapping and impersonate the party. Source

To understand exactly how DNS is used, lets illustrate by way of an example. Lets say that we want to ssh from server1 to server2.

  1. The user first performs kinit on server1 to receive a TGT.

  2. The kerberos client takes the hostname specified by the user and performs a DNS lookup on it. The result of this is then used to perform a reverse DNS lookup to derive the fully qualified domain name to the target host.

  3. The client then uses this FQDN to construct the name of the principal that it will request. Since the client application is SSH, we will be requesting the ‘host’ service. The principal would therefore be: host/server2.my.domain. In order to determine the realm for the principal, the client by default converts the domain name of the host to uppercase. If this realm does not exist, the client checks if any domain name to realm mappings have been configured in /etc/krb5.conf. This whole process is the Service principal canonicalization described in this section.

  4. The KDC confirms that the requested principal exists, and provides the service ticket back to the client.

  5. The client contacts the kerberos client on server2 by providing it with the service ticket for the ssh (‘host’) service. Server1 knows how to contact server2, because the FQDN in the principal name is used to perform a DNS lookup to receive the IP that the service ticket will be sent to for authentication.

  6. The client on server2 can optionally verify that the requesting client is who it says it is - by verifying the client ID with the KDC. Otherwise, it trusts that the client is valid since it was already authenticated by the KDC previously.

You can see that service principal canonicalization can cause serious problems. If for example, there were CNAME DNS records setup for hostnames, then the client could request a hostname that is not in the host principal for the host.

In such cases, it is always advisable to create host alias principals and associate them back to the original host principal in the KDC. This can be done in IPA as follows:

ipa host-add-principal <original host name> host/<host name alias>@<realm>

# For Example:
ipa host-add-principal not-a-planet.galaxy.com host/pluto.galaxy.com@GALAXY.COM

Keytabs

In most cases, end-users would authenticate to the KDC using their client secret (i.e their password). However, it would be cumbersome for automated scripts and applications to regularly (re)authenticate using a password.

This is where it might make sense to use a keytab. A keytab (Key Table), is a file storing pairs of Kerberos principals and their keys. When users generally start the authentication process using kinit, they are prompted for their password - which triggers the KDC to provide it the TGT, and then initiate the follow-up requests for service tickets. What the keytab does is when the client wishes to initiate authentication, the password is sent automatically to the KDC (in encrypted form) from the keytab file, rather than prompting for it.

The consequences of this are fairly obvious. Anyone who has access to a keytab can essentially impersonate the principal(s) contained within it. So its safe to say that keytabs should be protected just like passwords.

A keytab can store any type of principal, including all the ones we have previously discussed. For example:

# A Keytab storing a user principal
sudo klist -k -t username.headless.keytab
Keytab name: username.headless.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
  11 12/09/2016 11:38:17 username@MY.REALM.NAME (aes256-cts-hmac-sha1-96)
  11 12/09/2016 11:38:17 username@MY.REALM.NAME (aes128-cts-hmac-sha1-96)
  11 12/09/2016 11:38:17 username@MY.REALM.NAME (des3-cbc-sha1)
  11 12/09/2016 11:38:17 username@MY.REALM.NAME (arcfour-hmac)

# A Keytab storing a user principal tied to a host
sudo klist -k -t -e username.service.keytab
Keytab name: username.headless.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
  11 12/09/2016 11:38:17 username/my.domain.name@MY.REALM.NAME (aes256-cts-hmac-sha1-96)
  11 12/09/2016 11:38:17 username/my.domain.name@MY.REALM.NAME (aes128-cts-hmac-sha1-96)
  11 12/09/2016 11:38:17 username/my.domain.name@MY.REALM.NAME (des3-cbc-sha1)
  11 12/09/2016 11:38:17 username/my.domain.name@MY.REALM.NAME (arcfour-hmac)

There are two things to note here:

A keytab can be generated using ktutil:

[admin@myhost]# ktutil

ktutil: addent -password -p starlord/myhost.galaxy.com@GALAXY.COM -k 1 -e aes256-cts-hmac-sha1-96
Password for starlord/myhost.galaxy.com:

ktutil: addent -password -p starlord/myhost.galaxy.com@GALAXY.COM -k 1 -e aes128-cts-hmac-sha1-96
Password for starlord/myhost.galaxy.com:

ktutil: addent -password -p starlord/myhost.galaxy.com@GALAXY.COM -k 1 -e des3-cbc-sha1
Password for starlord/myhost.galaxy.com:

ktutil: wkt /path/to/starlord.keytab

Note that you have to create principals for all authentication methods (encryption types) that you wish to support.

Wrangling with Keytabs in IPA

For IPA users, the above steps to generate a keytab are more automated - but this also means that there are more gotchas. In IPA, there is a single command to generate a keytab:

# Generate a Keytab for a principal
ipa-getkeytab
-s my.ipa.server -p principal.to.retrieve -k keytab.file.name

There are a couple of things to watch out for:

To allow certain users and groups to retrieve keytabs in IPA:

# Allow a service principal to be retrieved by a user or group
ipa service-allow-retrieve-keytab my.principal.name --users="user1,user2" --groups="group1,group2"

# Allow a host principal to be retrieved by a user or group
ipa host-allow-retrieve-keytab my.principal.name --users="user1,user2" --groups="group1,group2"

ipa-getkeytab also has an option to specify a password using “-P”. This is not the same as the addent -password command when using ktutil. ipa-getkeytab regenerates the principal’s keys everytime it is run (unless using -r), and so it can either generate a random password or use the password specified with “-P”.

Password-less Users and “Headless” Keytabs

For user principals, whenever you reset your password - the IPA server regenerates the keys associated to the principal. This also means that any keytabs the user has with their user principals will be invalidated, as the keys stored along with the principal are no longer valid.

Conversely, if a new keytab is generated for a user principal, the KDC generates a new set of keys for the principal, meaning the user’s password will now be invalid - meaning you can lock them out of everything managed by IPA… :/

Additionally, if you have setup a password policy for users, they should expire within a configured time. This will mean that any keytabs storing the existing principals keys will be invalidated, and the password will need to be reset, and the keytab retrieved again.

From this, it is clear that IPA makes using keytabs with user principals a little tricky, since the same principal is also used for federated authentication. A safer approach is to create user principals tied to hosts - IPA treats these as service principals making them easier to manage, and avoiding conflict with federated user login/authentication. This will allow anyone with access on to the keytab file on the host to authenticate as the user principal within it (without specifying a password).

If for some reason you want to create a keytab for a user that can be authenticated anywhere without a password (I’m guessing this is what a “headless” keytab is):

  1. You can disable the password authentication mechanism for the user in IPA:

    # Types of supported user authentication
# --user-auth-type=['password', 'radius', 'otp']
ipa user-mod <username> --user-auth-type=[]

  2. Then, new keytabs can be generated, using the user principal:

    ipa-getkeytab -s my.ipa.server -p starlord@GALAXY.COM -k starlord.headless.keytab

  3. Some applications may not allow non-password authentication mechanisms, such as sudo - so you may have to change the sudo rule on the hosts to allow the user to sudo without a password:

    sudorule-add-option <MY SUDO RULE> --sudooption="!authenticate"

Clearing Kerberos Cache

When you update access policies on the IPA server, this will not immediately be updated on the client, as the kerberos ticket will be cached within the /tmp directory for that user. This will still be reflecting the old access policies, until the ticket expires or the cache is cleared. To force the cache to be cleared:

# sss_cache invalidates records in SSSD cache.
## -E performs a cleanup of all records.
## You can specify just a specific set of records to be invalidated. For example --groups.
sss_cache -E

# Restart the sssd client
sudo service sssd restart

# Re-authenticate
kinit