>>> Update 2>>>

Dear Expanse User,

The Lustre filesystem issues on Expanse have been resolved and we have mounted it back on the login nodes and on most compute and GPU nodes. The remaining nodes have been drained from the queue and will be rebooted to fix the mounts. The system is available for logins now and jobs can run on the nodes that are already fixed. Users are reminded that the Lustre filesystem locations (both scratch and projects) are **not** backed up. So please make offsite copies of anything critical. Also, it is important to control Lustre usage levels to help with performance and stability. Users are requested to clean up any data that is not in use for active runs or not needed in the near future. The filesystem **should not** be used as an archival location and users are strongly urged to make offsite copies of anything they want to keep long term.

Thanks

SDSC User Services Staff

>>> Update 1>>>>

Dear Expanse User,

We are continuing to work on the Lustre filesystem issue. We were able to recover the metadata servers last night but there are still continuing issues with one of the metadata servers that we are working on. Additionally, logins to Expanse are being impacted (with errors regarding malformed TOTP) and we are looking into the source of the problem. Updates will be provided as we resolve both of these problems.

Thanks

SDSC User Services Staff

>>>>> 

Dear Expanse User,

We are currently seeing problems with the metadata servers of the Expanse Lustre filesystem. The issue started around 5PM (PT), December 16, 2025,  and is currently resulting in filesystem access problems on all Expanse nodes. This will impact logins if your default path has Lustre locations in it. We will update once the issue is resolved.

Thanks

SDSC User Services Staff