CSG, Disk space quota

Department of Computing	Imperial College London
Disk quota and temporary storage space

There is a limit on the amount of disk space on the fileservers which students and staff are allowed to use. This is known as disk quota and (from October 2005) is set at 500MB for everyone in DoC. This quota covers any files and sub-directories created in your home directory, including all your email. Your home directory is your initial working space when you log in to any Linux system, and is the area known as the H: drive on Windows systems.

Checking your quota and disk space used

You can check your quota and the space you are using by typing:

quota

on any Linux system. The only way to check your quota when running Windows is to log in to a Linux host via an ssh client (eg. putty) and run the quota command as above.

The output will be similar to:

Disk quotas for user zzzz07 (uid 9999):
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
stork:/export2/users/z/zzzz07
                  22400  500000  600000              18   20000   21000

This shows you the various file spaces on which you have a quota. In this example, the user's home directory is held on a filesystem called /export2 on the file server stork, the user is using 22MB (megabytes) of disk space at present, has a quota of 500MB and a hard limit of 600MB. This user would start getting warning messages when they pass 500MB, and would never be allowed to use more than 600MB. The user also has 18 files, and is limited to 21000 in total.

Which files are taking up most space?

People often think they are only using say 20MB of space and when quota reports they are using 50MB they don't believe it! The truth is that many applications you use store data in files you don't even realise are there. Email can also take up a lot of space.

The following command will show which files are taking up most space in your account:

du -s * .[a-zA-Z]* | sort -n

The largest files or directories will appear at the end of the list. The du command can summarise disk usage in various ways - see the manual page (type man du) for other options.

You might want to add that command to the end of your .cshrc file as an alias:

alias usage 'du -s * .[a-zA-Z]* | sort -n'

then on future logins (or immediately, after running source ~/.cshrc) you can simply type:

cd
usage

What happens when I go over my quota?

Suppose you have a quota of 500MB, with a hard limit of 600MB. This allows you to store up to 500MB at all times, and allows you to exceed 500MB (up to an absolute upper limit of 600MB) for up to 7 days at a time. When you exceed your quota, quota will show a timer that starts counting down to zero, and you must delete enough files to come back under quota before the counter expires.

If you either hit your hard limit, or allow the over-quota timer to expire (i.e. leave yourself over quota for a whole week) the consequences will be:

all writes to your files will then start to fail, as if the disk was full.
you will probably not be able to log in.
mail will back up on the mail servers waiting to be delivered for you, and will eventually be returned to the sender.
you could easily lose data (edit a file and then write out a zero-length file because the write fails)!

How can I reduce my usage once I have reached my hard limit?

If you reached the total amount of disk space you are allowed to use you will not be able to log in via X-windows (KDE/Gnome). This is because the system creates a temporary file in your account when you log in. If there is no space to do so, you can't log in. You can log in if you avoid X-windows.

On Linux workstations type Ctrl-Alt-F1 to get a text-mode login prompt. Log in, find where you're using the space (usage alias above), then clear some space by removing or compressing files until your quota comes under the soft limit, using the command quota -v to check). Then log out of the text-mode login, and then press Ctrl-Alt-F7 to return the machine to X-Windows, and login in graphically.

You can also clear space by logging in via Windows and removing files from the H: drive. Take care not to remove any Linux setup files!

Making the most of your quota

There are several recommended methods for getting the most out of your file space.

Check that your web browser (Mozilla, Firefox, IE, Opera, or whatever) is not caching a large number of files in your home directory. For example, in Firefox, go to:
```
Edit -> Preferences -> Advanced -> Network
```
Set the disk cache to 0 and clear the disk cache.
In whatever application you use to read email, please remember to regularly delete unwanted mail, particularly copies of sent and trash messages.
When a program or application fails it will sometimes dump diagnostic information out to a file called core. These files can be quite large and should be removed unless you intend to use them. If you like you can prevent such files from being created by adding a line as follows to your .cshrc file in your home directory:
```
limit coredumpsize 0
```
After submission of an exercise, use the rm command to remove all unnecessary files, especially the binaries, object code and backups. Binaries can aways be recreated if the necessary source code is available. If an exercise directory has a "Makefile" in it, make clean may remove object code and binaries.
Similarly, Windows users should note that most Microsoft compilers (like Visual Studio) have a Build/Clean menu item which should be used at the end of an exercise.
Compress and compact the remaining source code into a single file using the tar or zip commands. For example:
```
tar czvf <filename.tgz> <list of files to archive>
```

Other storage areas

A separate area called /vol/bitbucket is available from all systems and is for the use of all members of DoC, for temporary storage of material that can be regenerated (downloads, compilation of large software etc). Files are not backed up, and there are limited file protections. You must put any files you wish to keep in a directory named after your username, i.e. /vol/bitbucket/username; create this directory via mkdir /vol/bitbucket/username if it doesn't yet exist. Files in other locations in bitbucket may be deleted at any time.
Bitbucket is available from Windows if you map the network drive: \\fs-vol\bitbucket
You must not store any copyright material in bitbucket; both you and the College could get into serious trouble if so. Do not try to link bitbucket into your personal web pages.

Each separate Linux machine makes any local unused disk space available as a local filesystem called /data. You are welcome to use this space as you like, remembering that it's not backed up, and again not storing any copyright material in there.

/vol/lab and /vol/labsrc :- These are controlled by the Lab Organisers (Teaching Associates). If you have application software in your account, which is likely to be used by other users, then it could be moved to this partition by agreement with the Lab Organisers. Users wishing to suggest software to be stored into /vol/lab or /vol/labsrc, should contact the Lab Organisers (mail lab). Software in this partition is not supported by CSG.
Removable media :- CD/DVD writers are installed on many systems. USB drives should automatically mount under Windows and Linux (at least for KDE and Gnome users, some Linux users may need to use something like pmount /dev/sda1 to mount a newly inserted USB drive onto /media/sda1, and pumount /media/sda1 to unmount it before you detach the USB drive!).

Getting your quota extended

The disk space quota is considered adequate for most academic work and you would need a good reason to be allocated any more (eg. final year project work involving video streams:-)). You're welcome to email help@doc however and ask. Shared disk space is set up for group project work. We sort this out directly with the group project organiser and will email students with group project info when we have the data.

Other Linux notes related to this topic are Managing your file space and Important files in your account.