July 18 2018: Directory Difficulties

Today one of our file servers had a drive fail. The normal failover process worked, but unfortunately took long enough to confuse a bunch of machines.

The usual symptom is that directories (home directories, project directories) will not attach. Often your login session will just hang.

Unfortunately, this takes a hard reboot to fix reliably. We are checking a bunch of the open login and compute servers (Wednesday afternoon), and rebooting as we can.