How to check your file quota

There is a limit on the amount of disk space on the fileservers which students and staff are allowed to use. This is known as disk quota and (from October 2015) is set at 8GB and 40,000 files for everyone in DoC. Of course CSG can increase a user's quota where necessary, this is discussed further right at the end of this guide.

This quota covers any files and sub-directories created in your home directory. Your home directory is your initial working space when you log in to any DoC Linux system, and is the H: drive on DoC Windows systems.

Your disk quota actually comprises four separate limits, two on space and two on number of files. For each quantity (space and files) there are two limits - a soft limit ("quota") and a hard limit ("limit"). Currently these are:

space quota	8GB
space limit	8.1GB
files quota	40,000
files limit	41,000

This allows you to store up to 8GB at all times, and allows you to store between 8GB and 8.1GB for up to 7 days at a time. Similarly, it allows to store up to 40,000 files at all times, and between 40,000 and 41,000 files for up to 7 days.

Checking your quota and disk space used

We recommend that you check your quota (showing how much disk space, and how many files, you are using) a couple of times a week by typing:

quota -Q

on the shell command line of any DoC Linux system. The only way to check your quota when running Windows is to log in to a Linux host via an ssh client (eg. putty) and run the quota command as above.

The output will be similar to:

Disk quotas for user zzzz10 (uid 9999):
     Filesystem  blocks    quota   limit   grace   files   quota   limit   grace
redkite:/export/users/z/zzzz10
                  22400  8000000 8100000            1800   40000   41000

This shows you the various file spaces on which you have a quota. In this example, your home directory is held on a filesystem called /export on the file server redkite, you're using 22MB (megabytes) of disk space at present, with a quota of 8GB and a hard limit of 8.1GB (8100MB). You have 1800 files, and a file quota of 40K files, 41K in total. On a busy machine (eg. a shell server) you might even see your own quota several times in a confusing fashion; just ignore all but one.

What happens when I go over my quota?

When you exceed your quota, the above quota report will show a timer that starts counting down to zero (in one of the "grace" columns), and you must delete enough files to come back under quota before the counter expires.

If you either hit your hard limit, or allow the over-quota timer to expire (i.e. leave yourself over quota for a whole week) the consequences will be:

every single write or file create to your home directory will then start to fail, as if the disk was full.
you will probably not be able to log into DoC linux machines graphically. Read the section "How can I reduce my usage once I've reached my hard limit" of how to dig yourself out of this hole.
you could easily lose data (edit a file and then write out a zero-length file because the write fails and the editor is not perfectly written to handle the write failure case)!

How can I reduce my usage once I have reached my hard limit?

If you reached the total amount of disk space you are allowed to use you will not be able to log in via X-windows (KDE/Gnome). This is because the system creates a temporary file in your account when you log in. If there is no space to do so, you can't log in. You can log in if you avoid X-windows.

On Linux workstations type Ctrl-Alt-F1 to get a text-mode login prompt. Log in, find where you're using the space (using the commands from the following sections), then clear some space by removing or compressing files until your quota comes under the soft limit, using the command quota -Q to check periodically). Then log out of the text-mode login, and then press Ctrl-Alt-F8 to return the machine to X-Windows, and login in graphically.

You can also clear space by logging in via Windows and either removing files from the H: drive or ssh'ing into a shell server. Take care not to remove any important Linux setup files!

Where am I using all the space?

People often think they are only using say 200MB of space and when quota reports they are using 8GB they don't believe it! The truth is that many applications you use store data in files you don't even realise are there. KDE, Gnome, web browsers and miscellaneous junk like pulseaudio are notorious for storing stuff behind the scenes. Make sure you empty your Gnome or KDE trashcan, and clear up all the junk on your desktop for a start! The following commands will show which directories or files are taking up most space in the top-level of your home directory.

cd
/vol/linux/bin/usage

The largest files or directories will appear at the end of the list. The usage command is a short shell script that essentially runs du | sort. du can summarise disk usage in various ways - see the manual page (type man du) for other options.

You might want to add that command to the end of your .cshrc file as an alias:

alias usage '/vol/linux/bin/usage'

then on future logins (or immediately, after running source ~/.cshrc) you can simply type:

cd
usage

When you found a suspicuously large directory, you can repeat the process one level down (and so on):

cd BIGDIR
usage
cd BIGGEST_SUBDIR_OF_THAT
usage

until you find a flat directory containing an unexpectedly large amount of space - then you might consider deleting some of the contents (once you understand what it is, of course).

Where am I using all the files?

Checking which directories contain the most number of files is slightly trickier. For a single directory such as ~/public_html, the command:

find ~/public_html | wc -l

will show the total number of files and directories [both of which count towards your file limits] within that directory.

So, in order to do that for all sub-directories in your current directory, producing the output in sorted order, one needs to write a short script. Fortunately, we have already written one for you, so just run:

/vol/linux/bin/nfiles

This produces a list of sub-directories [under the current directory] sorted by number of files contained. The directories with the largest number of files appear at the bottom of the list.

As with usage, you may want to use nfiles to drill down, as in:

cd
/vol/linux/bin/nfiles

When you find a directory with a suspicuously large number of files in it, you can repeat the process one level down (and so on):

cd DIR_WITH_MOST_FILES
/vol/linux/bin/nfiles
cd SUBDIR_OF_THAT_WITH_MOST_FILES
/vol/linux/bin/nfiles

until you find a flat directory containing an unexpectedly large number of files - then you might consider deleting some of those files (once you understand what the directory and it's files are, of course).

Making the most of your quota

There are several recommended methods for getting the most out of your file space.

Regularly empty your Gnome or KDE trash can
Check that your web browser (Chromium, Firefox, IE, Opera, or whatever) is not caching a large number of files in your home directory. For example, in Firefox, go to:

Edit -> Preferences -> Advanced -> Network

Set the disk cache to 0 and clear the disk cache.
In whatever application you use to read email (eg. Thunderbird, Outlook, Safari), make sure it's not keeping a local copy of all your email.
When a program or application fails it will sometimes dump diagnostic information out to a file called core. These files can be quite large and should be removed unless you intend to use them. If you like you can prevent such files from being created by adding a line as follows to your .cshrc file in your home directory:

limit coredumpsize 0

After submission of an exercise, use the rm command to remove all unnecessary files, especially the binaries, object code and backups. Binaries can aways be recreated if the necessary source code is available. If an exercise directory has a "Makefile" in it, make clean may remove object code and binaries.
Similarly, Windows users should note that most Microsoft compilers (like Visual Studio) have a Build/Clean menu item which should be used at the end of an exercise.
Compress and compact the remaining source code into a single file using the tar or zip commands. For example:

tar czvf <filename.tgz> <list of files to archive>

Other storage areas

A separate area called /vol/bitbucket is available from all systems and is for the use of all members of DoC, for temporary storage of material that can be regenerated (downloads, compilation of large software etc). Files in bitbucket are not backed up, and there are limited file protections. You must put any files you wish to keep in a directory named after your username, i.e. /vol/bitbucket/YOUR_USERNAME; create this directory via mkdir /vol/bitbucket/YOUR_USERNAME if it doesn't yet exist. Files in other locations in bitbucket may be deleted at any time by CSG.
Bitbucket is available from Windows if you map the network drive: \\fs-vol-bitbucket\bitbucket
You must not store any copyright material in your home dir OR bitbucket; both you and the College could get into serious trouble if you do. Do not try to link bitbucket into your personal web pages.
Each separate Linux machine makes any local unused disk space available as a local filesystem called /data, only accessible from that machine (not across the network via NFS). You are welcome to use this space as you like, remember that it's not backed up, and again not storing any copyright material in there.
/vol/lab and /vol/labsrc are controlled by the Lab Organisers (Teaching Fellows). If you install application software in your account, which is likely to be used by other users, then it could be moved to this partition by agreement with the Lab Organisers. Users wishing to suggest software to be stored into /vol/lab or /vol/labsrc, should contact the Lab Organisers and discuss it with them. Software in this partition is not supported by CSG.
Removable media :- DVD writers are installed on most desktop computers. USB drives should automatically mount under Windows and Linux (at least for Gnome and KDE users, some Linux users may need to use something like pmount /dev/sda1 to mount a newly inserted USB drive onto /media/sda1, and pumount /media/sda1 to unmount it before you detach the USB drive!).
Shared disk space in /vol/project is set up for organised group project work. We sort this out directly with the group project organiser and will email students with group project info when we have the data.

Getting your quota extended

The disk space quota is set by discussion with the academic staff, in particular with Peter McBrien, the Director of UG Studies. Your quota is considered adequate for most academic work and you would need a good reason to be allocated any more (eg. final year project work involving video streams or coursework involving a 10,000 file git repository:-)). You're welcome to email help@doc.ic.ac.uk and explain how much more quota (space or files) you think you need, why, and for how long. We'll discuss it with you and decide whether it's a reasonable request.