Background
Traditionally, the Linux Network File System (NFS) served files unencrypted with access based on host. This provided little real security, especially with modern threats.
ASC Linux file servers use Kerberos ticket based, encrypted NFS. This is much more secure but tickets have lifetimes of a little over a week, which means unless the ticket is renewed access will be denied. A ticket is granted when a password based login happens.
Two situations where this is potentially problematic:
- Continually logging in bypassing the password using an SSH key, whether based on the filesystem, directory, or certificate.
- Running a job for longer than a week - or starting a job on a session with an older ticket.
FAQ / Using Kerberized NFS
When I SSH into a computer I get "Could not chdir to home directory"
This usually happens when a user has SSH keypairs set up for passwordless SSHing. Please try the following in order:
- SSH using password only:
ssh -o PreferredAuthentications=password user@hostname
- Less preferred: Forward kerberos tickets when SSHing:
ssh -K user@hostname
If neither work, try SSHing normally and then run the following commands:
kinit
cd ~
If no errors occur, please exit the session, and ssh back in. If errors persist, please email asctech@osu.edu.
When I open a terminal I get "lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/0000000/gvfs"
This usually happens when users have not logged out of a workstation in a long time (over 12 days). Since locking/unlocking a computer does not generate a new session, Kerberos will not automatically generate a new ticket. Users can fix this issue by either logging out of the system, or by manually generating a new ticket. To generate a new ticket, open the terminal and enter the command:
kinit
This will ask for a password before generating a ticket. The terminal may be closed if desired, since Kerberos tickets are shared across a user's sessions on the same computer. If the error persists, please email asctech@osu.edu.
A long-running job that keeps getting terminated after a few hours
First, we recommend you run any long-running jobs from within a screen or tmux session (see the starting guide article for details). This should assure your session is active, which is important for Kerberos to continue to allow your processes access to NFS mounts. If your jobs are still being terminated (probably after exactly 8 hours) then your session might be appearing inactive, which will cause issues with Kerberos and NFS. To fix this, it may be necessary to log in to your system at least once every 8 hours and assure you still have a valid Kerberos ticket. You can see your tickets using:
klist
It may also be necessary to manually generate a ticket using:
kinit
If you expect your job to last for longer than 12 days, manually generating a ticket will be required at some point before the 12th day (note, manually generating a ticket will add an additional 12 days from the day it was generated). See the Kerberos section below for more details on ticket lifespan and renewal periods. If the jobs is still being prematurely terminated despite having valid Kerberos tickets, please email asctech@osu.edu
My computer freezes randomly and requires a reboot
This is usually caused by some kind of NFS problem, or if sessions are left open for too long without renewing Kerberos tickets. Please try:
- SSHing in. If you can gain access to the computer via SSH, try running
kinit (see above)
- Pressing Alt-F2, and then typing "r" and pressing Enter. This will restart Gnome without restarting the entire computer.
If issues persist, please email asctech@osu.edu.
Kerberos - more info
Kerberos is used to protect data shared via NFS. It requires a password to generate a "ticket" before it will allow a user access to NFS mounts. Most research data directories in CBC are mounted to workstations via NFS, and all home directories on CBC workstations are NFS mounts, and are thus protected by Kerberos.
Kerberos tickets last for 8 hours at a time. Tickets will automatically be renewed if there is an active session for up to 12 days from the original start date. At the end of the 12 days, a new ticket has to be generated (meaning a password is required). There are many ways to generate a new ticket; every time a new session is started using a form of password authentication, a new Kerberos ticket is generated. Kerberos includes a tool to force a ticket generation. That includes:
- Logging out of any existing GUI sessions on the host, and then logging back in (locking and unlocking will not generate a new ticket, since it is not starting a new session)
- Starting a new SSH session using password authentication
- Running
kinit
Tickets are shared between sessions on the same host. To list all of your tickets, run:
klist
This will tell you the starting date and time, when each ticket expires, and what date and time it will stop automatically renewing. You may see several tickets.
Most of the time, there is no need to interact with Kerberos directly, but sometimes users may encounter strange errors and may be unable to access their data. This is almost always because Kerberos did not receive a password, and so cannot generate a ticket. The most common cause is SSH keys, since they don't require a password. The best fix for this is to SSH using the -K option, which forwards Kerberos tickets. You could add an alias to your shell's rc file:
- For Bash, add the following line to
~/.bashrc
alias ssh='ssh -K'
- For tcsh, add the following line to ~/.tcshrc
alias ssh ssh -K
This will allow you to SSH normally (without needing to remember or type any extra options), maintain the SSH key functionality, and assure Kerberos will function normally.
Note that if SSHing to a non-OSU computer that also uses Kerberos, you may encounter errors. To fix these errors, please add the following lines to ~/.ssh/config:
Host HOSTNAME
GSSAPIAuthentication no
(replace "HOSTNAME" with the name of the computer you are trying to SSH to)