# FAQ on data coordination

This page list frequently asked questions on data coordination. See our lab user storage FAQ for practical use of our storage services.

TIP

This section prosper on questions. Contact us with burning topics so we can grow the page together.

# Access

# Can all users access all data in a lab?

Yes, principally. Practically, maybe. On the home-machine, all lab users can by default access all folders on the collaborative data volumes: /mnt/archive, /mnt/work and /mnt/scratch. Lab users may add individually restrictions to files or folders they upload or generate, although such separation is not guaranteed by us. When needed, we recommend to separate user access by establishing separate labs labs rather than to separate data in folders inside a lab.

# What happens with a user's data when the user is deactivated in a lab?

All data that is uploaded or generated by lab users are under the control of the Lab leader's organization. This includes data in the user's home folder. When lab access is deactivated for a lab user, then Lab leader and/or Data space leader can ask HUNT Cloud to change ownership and permissions of such data to allow other lab users to access the data.

# Volumes

# What is a storage volume?

You can think of a storage volume as a named virtual hard disk that is attached and reachable from a lab. This is the places there your lab users storage their data. Read more about the four default volumes.

# Can we add new volumes to our labs?

Yes. Nothing is better than seeing your data grow. We are more than happy to include additional volumes to your lab machines to make your science fly. Order your new volumes in our service desk.

# Can we expand existing volumes?

You can expand volumes on both and IaaS machines in terabyte (TB) steps. Lab leaders and lab coordinators can expand volume expansion in our service desk.

# Can we reduce the size of existing volumes?

Not directly. You can reduce your storage volumes in the following way: (1) request a new volume with your new desired size in our service desk, (2) transfer data from your old volume to your new volume, and (3) request deletion of your original volume.

# When should we expand our volumes?

Full storage means that your lab users no longer can utilize this volume in your scientific activities. We recommend that you expand a storage volume at about 85 percent usage. See how to check your current usage here.

# Will data disappear during a volume expansion?

No. The data in your volume will still be there after the volume expansion. For safety reasons, we duplicate your volume before we start the expansion as a safeguard against unexpected failures. That said, your data will be unavailable during the period of the expansion procedures, typically for a few hours.

# What's the response time for a volume expansion?

During the expansion, the machine where the volume is attached will need to be restarted. Thus, all running analysis on the lab machine will be stopped. We recommend that you plan the expansion with your lab users about 2 weeks before the storage is needed. Then, we will together schedule a two (2) hour time slot for the technical expansion, usually from 9-11 am.

# Why is volume sizes less than claimed?

The volumes sizes that you see on your machines are slightly less that the volume sizes that you claim. This is expected. Read why below.

Terabyte definition

One (1) terabyte of claimed storage is defined as 2^40 bytes. Computers allocate storage in base 2 mathematics (2^40 bytes = 1,099,511,627,776 bytes) and report the allocation attached to machines in base 10 mathematics (10^12 bytes = 1,000,000,000,000 bytes). The discrepancy between the two systems cause volume allocations to be observed in the machines as a lower number than the terabytes claimed. To illustrate, an allocation of one (1) terabyte storage will be reported, and have an operational capacity, of 0.91 terabytes when attached to a machine (10^12 bytes / 2^40 bytes = 0.91 terabytes).

# What is the upper size of a volume?

We aim to keep individual volumes below 20 TB if possible. The soft limit is set for practical purposes when it comes to volume handling on our side, both so we are able to handle duplicate copies and complete file checks within reasonable times. The limit is also set to help distribute large collections across several logical units in the unlikely case of volume destructions.

# Restore

# Do we have 'backup' in our lab?

Yes and no. You have restore functionality by default for two folders on your home machine, /mnt/work and /mnt/archive. No other volumes on your home machine, nor your IaaS and blue machines have restore activated by deployment. We strongly recommend to store a copy of data that you care about in one of the above folders.

# How does restore work?

The restore is intended to protection your data against accidental overwrites and accidental deletion of files in your lab. Therefore, we do nightly copies of data that has changed during the day from your /mnt/work and /mnt/archive volumes. This allows you to revert to previous versions of files from the period that your restore copies are preserved.

DANGER

Your data are stored in one (1) physical location. This means that your restore do -not- protect against accidents that destroy this location. Keep at least one copy of your data outside HUNT Cloud, or use our tape service to keep encrypted offsite tapes, to ensure improved geographical distribution for data that need such protection.

# How long is our restore copies preserved?

Restore files are preserved by default for thirty (30) days. This means that you can revert to daily versions of your files in your /mnt/work and /mnt/archive volumes as they existed on an individual day over this period.

# Where is our restore files located?

Your restore files are located in the .restore folder in the /mnt/work and /mnt/archive volumes.

ls /mnt/work/.restore
ls /mnt/archive/.restore

The above command should give you the list of folders with the date and time for your individual restores in the respective folders, such as:

2020-01-29-054701  2020-02-10-054701  2020-02-23-054701
2020-01-30-054701  2020-02-11-054701  2020-02-24-054701
2020-01-31-054701  2020-02-13-054701  2020-02-25-054701
2020-02-01-054701  2020-02-14-054701  2020-02-26-054701
2020-02-02-054701  2020-02-15-054701  2020-02-27-054701
2020-02-03-054701  2020-02-16-054701
2020-02-04-054701  2020-02-17-054701
2020-02-05-054701  2020-02-18-054701
2020-02-06-054701  2020-02-19-054701
2020-02-07-054701  2020-02-20-054701

# How can we restore a file or folder?

You will find your preserved files inside the .restore folders. Change directory into the folder where you want the preserved file to be copied, for example by using the cd <foldername>. Then, copy your file of interest back into your current lab folder of choice. Contact us if you need assistance with file identification.

# -- Principal example
cp /mnt/archive/.restore/2020-02-27-054701/<my-file.txt> .
ls

# Who is responsible for our restore?

We are responsible to ensure that the technical side of your restore work for the /mnt/work and /mnt/archive volumes. One or more lab users are assigned to the restore group that can remove files for your restore folders. These users are responsible for the data inside the restore folders.

# Can I delete restore folders?

Labs users assigned to the restore group can edit and delete files inside your restore folders. Lab leaders can authorize individual lab users to be listed in the restore group. Lab users can see if they are attached to this group by running the groups command:

groups

Members of the restore group should be able to delete preserved files in the .restore folders. Note that permissions for individual files and folder may restrict some deletions. Contact us if your need help to clear your restore folders.

# Do restore consume storage space?

Yes. Your restore folders are stored inside your /mnt/work and /mnt/archive volumes. This will consume storage space in these volumes as a regular folder. Typically, restore will take one duplicate copy of your data and then incremental changes over the 30 day period. You should therefore plan your volume sizes for at least twice the amount of data that you plan to store in these folders.

# What is the size of our restore folders?

You can see the consumption of your default restore folders with the following command:

du -sh /mnt/work/.restore/*
du -sh /mnt/archive/.restore/*

# Why is our restore folders filling our volumes?

We take nightly copies of data that has changed over the last day. Thus, folders with data that change on a daily basis will continuously grow your restore. For example, if you store temporary files from your analysis in work or archive.

Use scratch for temporary files

One good way to avoid filling of your restore volumes is to use the /mnt/scratch volume for such temporary files.

# Can we deactivate our restore?

Yes. You can deactivate restore for the entire volume or individual folders on your home machine. Deactivation or deletion of restore folders needs authorization from lab leaders since this directly affect the integrity of your lab data. Lab leaders can request such deactivations on email.

DANGER

Deactivation of restore means that your data holds -no- protection for accidental deletion or accidental overwrites. We are unable to recreate any data in folders without restore if your data gets deleted or overwritten.

# Can we control if our restore works as intended?

Yes. You can check for yourself that your restore is working as intended:

  1. Start by controlling that your restore folders do exist, and if so hold duplicate copies from the last 30 days. You should expect to see 30 subfolders with daily time stamps within each .restore folder:
ls /mnt/work/.restore/
ls /mnt/archive/.restore/
  1. Check the size of your daily restore folders. You should expect the earliest date to hold most of your date (largest storage size) and the others to show sizes according to your data changes. Depending on your volume sizes, these commands may take up to many minutes.
du -sh /mnt/work/.restore/*
du -sh /mnt/archive/.restore/*
  1. Manually inspect the content of individual folders inside your daily restores using cd <folder>.
cd /mnt/work/.restore/
cd /mnt/archive/.restore/
  1. Test that you are able to [copy out a file or folder] into your regular lab storage (see above).

Celebrate the success if you are able to successfully export a file, or contact us if you think something is wrong with your restore.

Last Updated: 4/25/2024

← Introduction →