Checking How Much Space you are Using
There are two main storage quotas that exist on the SCC. One is the Home Directory quota for each user and the other is the Project Space quotas, which includes /project, /projectnb, and STASH directories.
All users have a limit of 10 GB of storage for their Home Directory and this cannot be increased. You can exceed your Home Directory quota but not your hard limit (11 GB) for a maximum of 7 days. You should NEVER go up to your hard limit or you may be unable to log in as you will be unable to write any files. If you have exceeded your Home Directory quota, review the section Home Directory Quota Exceeded to determine what files and directories are consuming storage.
Each project gets a maximum of 1 TB of Free Baseline storage and additional storage can be purchased via our Buy-in program. If your project needs more space, the project LPI or IT/Admin Contact can request additional space but there is a charge for requests over 1000 GB.
One can check the storage quotas either through the OnDemand portal or via a terminal using the pquota command for Project Disk Space or quota command for your Home Directory.
Checking Home Directory and Project Disk Space Quotas Using the SCC OnDemand Portal
Within the SCC OnDemand Portal, in the top menu bar, click on “Quotas” and then “Quota”.
This will open a new page which will contain a summary of your Home Directory quota and Project Disk Space quotas for all projects you are a member of. Note that the data used by this page is updated approximately every five minutes, and therefore if you delete some files it may take a few minutes to see the changes reflected on this page.
Under “Project Directory Quotas” you can click the “show” button to give a breakdown of disk usage by members of the project team. Some login names listed may contain just numbers because the original user who created the files no longer has an account on the SCC and the numbers are a substitute for the username. A project’s Lead Project Investigator (LPI) or IT/Admin Contact can request that we delete or make accessible to him or her any files in a given project’s Project Disk Space or STASH areas. This request should be sent to help@scc.bu.edu.
Checking Project Quotas Using the Terminal
In a terminal connected to the SCC, use the command pquota -u to see the breakdown of usage for each project partition (e.g. /project and /projectnb) you have access to by user. To view the quota for a specific project specify the project name at the end of the command. The example below shows the disk usage for project animate:
scc1% pquota -u animate
quota quota usage usage
project space (GB) (files) (GB) (files)
----------------------------------- ------ --------- --------- --------
/project/animate 50 1638400 0.00 2
kmischo 0.00 1
/projectnb/animate 50 1638400 3.37 4328
dcornell 0.29 104
laura 1.22 2177
rcrnl 1.66 1723
15407 0.09 80
82363 0.11 243Some login names listed may contain just numbers because the original user who created the files no longer has an account on the SCC and the numbers are a substitute for the username. A project’s Lead Project Investigator (LPI) or IT/Admin Contact can request that we delete or make accessible to him or her any files in a given project’s Project Disk Space or STASH areas. This request should be sent to help@scc.bu.edu.
Note that the data used by pquota is updated approximately every five minutes, and therefore if you delete some files it may take a few minutes to see the changes reflected by the pquota command.
Checking Home Directory Quota Using the Terminal
The terminal command to show the disk usage and quota in your Home directory is quota -s:
scc1% quota -s
Home Directory Usage and Quota:
Name GB quota limit in_doubt grace | files quota limit in_doubt grace
adftest2 0.00212 10.0 11.0 0.0 none | 287 200,000 200,000 0 none
The important items are highlighted in yellow. They show your usage (in GB), quota (in GB), hard limit (in GB) and number of files you have.
Note that the data used by quota is updated approximately every five minutes, and therefore if you delete some files it may take a few minutes to see the changes reflected by the quota command.
Home Directory Quota Exceeded
Exceeding the quota limit in your Home directory can result in difficulties using the SCC. This may include issues running compute jobs or the inability to log in to the SCC. You will receive an email in the morning each day, while your Home directory is over quota.
To see which directories, files, and dotfiles are taking up the most space in your home directory, connect to the SCC using a terminal and run the command du -s .[^.]* * | sort -n; the largest items will be listed last. Please, note, this command can take several minutes to complete, so be patient when using it:
scc1% du -s .[^.]* * | sort -n
...
32 helloWorld.o9436314
224 newdir
1760 .matlab
The Usual Suspects
Frequently, the cause of home directory quota issues are the .cache and .local directories. If the above command shows that the .cache directory is consuming the most disk space you can change to that directory and re-run the command:
scc1% cd .cache
scc1% du -s .[^.]* * | sort -n
...
1000 pip
2003 RStudio
These directories can be safely deleted provided you don’t have any running jobs or OnDemand sessions using them. For example, before deleting the .cache/RStudio directory, make sure to shut down any RStudio sessions you have running.
The .local and .cache/pip directories are frequently filled when the Python pip command is used to install Python packages into the home directory. Typically the pip directory can be deleted provided that you’re not actively running pip. The pip tool can be used to install Python packages into your /projectnb/project_name space. Refer to the SCC instructions for installing packages via pip. Alternatively, Python virtual environments can be used to store Python packages. Once you have set up your Python packages outside of your home directory you can remove the contents of the .local directory.
If you are using conda environments, the .conda directory can rapidly consume home directory space. Instructions for using conda properly with /projectnb storage and for migrating your conda environments are provided.
More details on Project Disk Space are available.