GENERAL KERBEROS QUESTIONS
Q #1: What Is Kerberos?
In Greek mythology, Kerberos is the 3-headed dog that guards the gates of hell.
Kerberos is also the authentication protocol used to protect the contents of your Linux NFSv4 home directory from prying eyes.
Q #2: What Is NFSv4?
NFS refers to “Network File System” which is where Linux home directories are kept. It allows you to use the same home directory on any Linux machine on the Campus wired network.
NFSv4 refers to version 4 of this Network File System, which allows for Kerberization.
Q #3: What Is Kerberization?
Kerberization is the process by which Kerberos is enabled on your Linux NFSv4 home directory (and other things), such that nobody can view its contents without a Kerberos Ticket.
Q #4: What Is A Kerberos Ticket?
A Kerberos Ticket is a chunk of encrypted data that gives you access to things on the Campus wired network such as your NFSv4 home directory.
Q #5: How Do I Get A Kerberos Ticket?
Upon successfully logging in to a Linux Workstation on the Campus wired network, you will (if the system is working correctly) automatically be issued a Kerberos Ticket known as a “Ticket Granting Ticket”. This ticket is then used to gain access to other things, such as your Linux NFSv4 home directory. Each such thing you access causes a new Kerberos Ticket to be generated, specific to the service you’re accessing.
Q #6: How Long Does A Kerberos Ticket Last?
By default, all Kerberos Tickets have a 10 hour lifetime before they expire, and a maximum renewal period of 1 week. If you want to renew your ticket, you must do so before it expires. If you wait until after the 10 hours is up, then it is too late, and you must get a new one.
Q #7: What Happens When My Kerberos Ticket Expires?
You lose acceess to your home directory.
You will regain access to your home directory when you get a new Kerberos Ticket.
Q #8: How Do I Renew My Kerberos Ticket Before It Expires?
See Q #13 or Q #23.
Q #9: How Do I Get A New Kerberos Ticket After My Old One Expires?
See Q #14 or Q #23.
Q #10: Do I Need To Worry About Kerberos On Linux Servers?
On many Linux Servers, your NFSv4 home directory is mounted without Kerberization.
This includes all the Colony Cluster nodes. Cluster jobs may run for weeks or sometimes months. We wouldn’t want you to have to renew your Kerberos Ticket every 10 hours in order to keep your jobs running!
Most other Linux Servers, including the Linux terminal servers (rcg-linux-ts1.rcg.sfu.ca & rcg-linux-ts2.rcg.sfu.ca), will use Kerberization, with the exception of rcg-linux-ts3.rcg.sfu.ca, which we are providing without Kerberization for the convenience of those users who need to run long-term jobs, such as screen or tmux.
Q #11: Why Do We Have To Suffer The Inconvenience Of Kerberos On Linux Workstations?
Non-Kerberized NFS is inherently insecure, in that anybody with sudo access to a Linux computer can su to another user and read/modify the contents of their home directory.
Most of our Linux Servers are locked up in a server room with restricted physical access, and only trusted IT staff have sudo access. This minimizes the risk and therefore we can allow many of these servers to use non-Kerberized NFS.
The Linux Workstations, on the other hand, have a more liberal policy with regards to who is granted sudo access. This is because Faculty members often demand sudo access to their own Workstations, and Lab Administrators of various research labs need to install software for their users. In order to accomodate this greater risk, we need to use Kerberized NFSv4 on all the Linux Workstations.
LINUX COMMAND-LINE KERBEROS QUESTIONS
Q #12: How Do I View My Kerberos Tickets?
Just type:
klist
You’ll see something akin to the following:
Ticket cache: FILE:/tmp/krb5cc_12345_A6b7C8
Default principal: username@AD.SFU.CA
Valid starting Expires Service principal
05/02/20 15:29:10 06/02/20 01:29:10 krbtgt/AD.SFU.CA@AD.SFU.CA
renew until 12/02/20 15:27:42
on computers that use a file-based Kerberos Ticket cache.
If the computer is using the kernel keyring for kerberos tickets, then you’ll see something akin to the following:
Ticket cache: KEYRING:persistent:12345:krb_ccache_4Ybfz5f
Default principal: username@AD.SFU.CA
Valid starting Expires Service principal
05/02/20 15:29:10 06/02/20 01:29:10 krbtgt/AD.SFU.CA@AD.SFU.CA
renew until 12/02/20 15:27:42
In either case, krbtgt is the Kerberos Ticket Granting Ticket. You’ll see that it was valid starting at 15:29 on Feb 5th, 2020, and it expires 10 hours later.
You’ll also see that it can be renewed for up to a week before you need to get a new Kerberos ticket. However, it must be renewed before it expires. If you wait until after the 10 hours is up, then it is too late!
If you have accessed your Linux NFSv4 home directory, then might also see another Kerberos Ticket:
02/05/20 16:00:57 02/06/20 02:00:55 nfs/nfs-home.its.sfu.ca@AD.SFU.CA
renew until 02/12/20 16:00:51
This is the nfs Kerberos Ticket that allows you to read and write to your home directory located on the nfs-home.its.sfu.ca server.
And if you used Kerberos authentication to ssh into a Linux host such as rcg-linux-ts1.rcg.sfu.ca, then you might also see a Kerberos Ticket like this:
02/05/20 16:02:26 02/06/20 02:00:55 host/rcg-linux-ts1.rcg.sfu.ca@AD.SFU.CA
renew until 02/12/20 16:00:51
This is a host Kerberos Ticket.
Q #13: How Do I Renew My Kerberos Tickets Before They Expire?
Just type:
kinit -R
This will renew your Kerberos Ticket Granting Ticket. The other tickets will disappear. But that’s okay. New ones will be auto-generated when you access things such as your Linux NFSv4 home directory.
Q #14: What Happens If I Forget To Renew My Kerberos Tickets Before They Expire?
Just type:
kinit
and then give your password. You’ll get a new Kerberos Ticket Granting Ticket.
Q #15: What Happens To My Kerberos Tickets After The Maximum Renewal Period Has Passed?
Your Kerberos Tickets cannot be renewed after the maximum renewal period, which is one week.
But you can always get a new one. Just type:
kinit
and then give your password. You’ll get a new Kerberos Ticket Granting Ticket.
Q #16: Help! I Can No Longer Access My Home Directory! Why?
This probably means that your Kerberos Ticket expired. A sign of this is that you type:
ls
and the system responds with:
ls: cannot open directory '.': Key has expired
Another possibility is that your Kerberos Ticket has been destroyed and is no longer present. In which case the system responds with:
ls: cannot open directory '.': Permission denied
In any case, the fix is straightforward. Don’t panic! Just feed the 3-headed dog. Its favourite food is kinit. Simply type:
kinit
and then give your password. You’ll get a new Kerberos Ticket Granting Ticket.
Q #17: Help! My Home Directory Isn’t Available At Login! Why?
Normally, a Kerberos Ticket is generated at login time. However, under some circumstances this doesn’t happen. Upon login you might see the following:
Could not chdir to home directory /home/user: Key has expired
or:
Could not chdir to home directory /home/user: Permission denied
The first is an indication that you had a Kerberos Ticket but that it has expired.
The second is an indication that you don’t have a Kerberos Ticket at all.
Either of these error messages will immediately be followed by something like:
Loading default RCG module...
ENVIRONMENTS/RCG(26):ERROR:102: Tcl command execution failed: set MACHINE_OSVER [exec facter lsbdistdescription]
use.own(49):ERROR:102: Tcl command execution failed: if [ module-info mode load ] {
if { ! [ file exists $ownmoddir ] } {
file mkdir $ownmoddir
set null [open $ownmoddir/null w]
puts $null "#%Module########################################################################"
puts $null "##"
puts $null "## null modulefile"
puts $null "##"
puts $null "proc ModulesHelp { } {"
puts $null " global version"
puts $null ""
puts $null " puts stderr \"\tThis module does absolutely nothing.\""
puts $null " puts stderr \"\tIt's meant simply as a place holder in your\""
puts $null " puts stderr \"\tdot file initialization.\""
puts $null " puts stderr \"\n\tVersion \$version\n\""
puts $null "}"
puts $null ""
puts $null "module-whatis \"does absolutely nothing\""
puts $null ""
puts $null "# for Tcl script use only"
puts $null "set version 3.2.10"
}
}
This is an indication that the modules environment has failed to load because it couldn’t access your home directory.
In any case, the fix is straightforward. Don’t panic! The 3-headed dog is hungry. Feed it with:
kinit
and then give your password. You’ll get a new Kerberos Ticket Granting Ticket. Then log out and log in again. This time, the login should proceed normally and the modules environment will be loaded as usual.
Why do these things sometimes happen? It might be that you set up ssh public/private key exchange, in order to log in without giving a password. This may have worked fine in the past, but you can’t generate a Kerberos Ticket without typing your password!
Q #18: How Do I Destroy My Kerberos Tickets? Why Would I Want To?
To destroy your Kerberos Tickets, just type:
kdestroy
It is recommended that you do this before logging out of a Linux workstation.
If you fail to do this, any valid Kerberos Tickets will remain in the Kerberos Ticket cache. On computers that are using file-based Kerberos Ticket caches, these will be files in /tmp. Anybody with sudo access to the Linux workstation could potentially use these these kerberos ticket cache files to access your home directory!
Even if you do kdestroy there are often other Kerberos Ticket caches that get generated by various activities. They may also be stored in /tmp. To be safe, it is best to take a look in /tmp and make sure that there are no files there belonging to you starting with krb5cc_. If there are, delete them.
Q #19: Why Can I Still Read My Home Directory After Destroying My Kerberos Tickets?
kdestroy will instantly remove your Kerberos Tickets from a cache file in /tmp/. And rm /tmp/krb5cc_[your_uid]* will remove other Kerberos Tickets you may have. However, the Linux kernel also caches Kerberos Tickets in memory, and it takes a while before the memory cache is cleared.
Q #20: How Do I Use Kerberos Authentication To Ssh Into A Linux Host?
Assuming you are logged-into a Linux host on the Campus wired network and that you have a valid Kerberos Ticket, you can ssh into any other such Linux host simply via:
ssh -K hostname.domainname
Using this method, you can log in without having to use a password. The advantage of this method over using ssh public/private key exchange is that your Kerberos Ticket will be propagated to the second host.
Q #21: Help! I Run Long-Term Processes On My Linux Workstation And Kerberos Is Locking My Home Directory!
If you run long-term processes on your Linux Workstation and you don’t want to have to keep manually renewing your Kerberos tickets every 10 hours, you can set up a cron job to do it for you.
In order to do this, you need to edit your crontab with crontab -e. Before doing so, make sure that your EDITOR environment variable is set to your favourite Linux editor. Eg. setenv EDITOR emacs or setenv EDITOR vim, assuming that your shell is tcsh (or export EDITOR=emacs or export EDITOR=vim, if your shell is bash).
For example, you could set up the following crontab entry on your Linux computer:
MAILTO=username@sfu.ca
30 * * * * /usr/bin/kinit -R
This will renew your Kerberos ticket at 30 minutes past every hour of every day. (See man 5 crontab for more information on what the various fields mean.)
Note, however, that this will work for at most 1 week, until the renewal period expires, and then one must manually kinit and type one’s password. This is where setting MAILTO=username@sfu.ca comes in handy, because you will receive mail when the cron job fails. (Be sure to use your real username here, not “username”!)
Q #22: The Answer To Q #21 Isn’t Good Enough! I Need My Processes To Run Unattended For Weeks!
You can achieve this via a crontab entry and your own private keytab file.
Suppose you want to keep your keytab within your home directory in /home/username/myprivatestuff/username.keytab. You can create it as follows:
mkdir myprivatestuff
chmod 0700 myprivatestuff
cd myprivatestuff
ktutil
addent -password -p username@AD.SFU.CA -k 1 -e aes256-cts-hmac-sha1-96
addent -password -p username@AD.SFU.CA -k 1 -e aes128-cts-hmac-sha1-96
(enter your password here when asked)
wkt username.keytab
q
(Make sure you use your real username here, not “username”!)
You can call the directory whatever you want; just make sure it’s private – that’s what the chmod 0700 is for. If somebody gains access to your keytab file then they can use it to gain access to all your stuff!
You can then set up a crontab entry similar to the one above, but instead of using kinit -R, you use the following:
MAILTO=username@sfu.ca
* */2 * * * /usr/bin/kinit username@AD.SFU.CA -k -t /home/username/myprivatestuff/username.keytab
(Again, make sure you use your real username here, not “username”!)
This will get you a new Kerberos Ticket every 2 hours, without having to type a password, by using the credentials stored in your private keytab file.
This will work virtually forever* (until the computer reboots), unless for some reason the cron job fails for more than 10 hours and the Kerberos Ticket expires. In this case your home directory becomes unavailable, and the cron job can no longer read the contents of /home/username/privatestuff. It’s a bit of a catch-22, but one can get around it by manually typing kinit and giving one’s password.
If the computer is rebooted, then the Kerberos Ticket caches get cleared, resulting in all the Kerberos tickets going away. Subsequent to a reboot, you need to kinit manaully to get a new Kerberos ticket before your cron job will work.
* virtually forever unless one changes one’s Campus password, in which case the steps of creating a private keytab need to be redone.
LINUX GUI KERBEROS QUESTIONS
Q #23: How To I Renew My Kerberos Ticket In The Linux GUI?
If you’ve been away from your Linux workstation for a while, the screen saver kicks in and the screen is locked until you type your password. Every time you unlock your screen, a new Kerberos Ticket is generated.
If you’ve configured your workstation not to lock the screen (or if you’re using the Linux GUI via VNC which doesn’t usually lock the virtual screen), then your Kerberos Ticket will expire after 10 hours and you’ll notice that various things in the GUI will no longer work.
You should, however, still be able to open a Linux Terminal. For example, in Ubuntu’s MATE GUI, go to the Applications:System Tools menu and select “MATE Terminal”:
It will load with error messages like this:
Loading default RCG module...
ENVIRONMENTS/RCG(26):ERROR:102: Tcl command execution failed: set MACHINE_OSVER [exec facter lsbdistdescription]
use.own(49):ERROR:102: Tcl command execution failed: if [ module-info mode load ] {
if { ! [ file exists $ownmoddir ] } {
file mkdir $ownmoddir
set null [open $ownmoddir/null w]
puts $null "#%Module########################################################################"
puts $null "##"
puts $null "## null modulefile"
puts $null "##"
puts $null "proc ModulesHelp { } {"
puts $null " global version"
puts $null ""
puts $null " puts stderr \"\tThis module does absolutely nothing.\""
puts $null " puts stderr \"\tIt's meant simply as a place holder in your\""
puts $null " puts stderr \"\tdot file initialization.\""
puts $null " puts stderr \"\n\tVersion \$version\n\""
puts $null "}"
puts $null ""
puts $null "module-whatis \"does absolutely nothing\""
puts $null ""
puts $null "# for Tcl script use only"
puts $null "set version 3.2.10"
}
}
This is an indication that the modules environment has failed to load because it couldn’t access your home directory.
Now feed the 3-headed dog by typing:
kinit
and then giving your password. You’ll get a new Kerberos Ticket Granting Ticket.
You can then close the MATE Terminal. If you want to open another one, it should open this time without the error messages.
If Kerberos has so badly messed-up your Linux workstation that the GUI is completely frozen and you can’t do anything, then your best option is to ssh-in remotely from another computer and kinit on the command-line.
Q #24: Help! I Double-Clicked On My “Home” Folder And Was Denied Access!
In Ubuntu’s MATE GUI, you might see something like this (substitute “hebron” [that’s me, the author of this documentation] with your username):
and in CentOS 6’s GNOME GUI, you might see something like this:
The message in MATE about there being no application installed for symbolic link files is a red herring. Do not hit ‘yes’ to search for an application to open the file!
The real problem is that your Kerberos Ticket has expired and therefore the system cannot follow the symbolic link that points to your home directory.
The solution is to get a new Kerberos Ticket. See Q#23 for details.
Q #25: Help! The Linux GUI Says That My Firefox Profile Is Missing!
In Ubuntu’s MATE GUI, you might see this:
and in CentOS 6’s GNOME GUI, you might see this:
This most likely means that your Kerberos Ticket has expired, and therefore Firefox can no longer see your home directory.
To fix this, you need to get a new Kerberos Ticket. See Q#23 for details.