Skip to content
Snippets Groups Projects
Commit a2791e6a authored by Laurent Heirendt's avatar Laurent Heirendt :airplane:
Browse files

Merge branch... 

Merge branch '2-create-a-copy-of-the-backup-and-hpc-lab-card-and-integrate-with-nice-layout' into 'master'

Created copies of two existing cards

Closes #2

See merge request R3/R3LabCards!4
parents 7b920705 6fee1e61
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
Showing
with 171 additions and 89 deletions
backup/R3labCARD-backup.md
+ 6
− 58
View file @ a2791e6a
......@@ -23,13 +23,13 @@ Below table lists storage resources available to the LCSB research staff. We als
| Storage | Has Built-In Backup | Recommended As Master Location | Recommended as Backup Location | Notes |
| -------------|:-------------:|:-------------:|:-------------:|:-----|
| Atlas-Isilon | Yes | Yes | Yes | Request information on Atlas backup frequencies using [ServiceNow](https://service.uni.lu/).|
| HPC Folders | Yes | Yes | No | Only **$HOME** and **$PROJECTWORK** folders are backed up. Frequencies per cluster listed [here](https://hpc.uni.lu/blog/2018/faq-backup-policy/R3labCARD-backup.md).|
| LCSB Managed Virtual Machines | On-Demand | Yes | No |Request backup service by following instructions on this [Lab Card](https://git-r3lab.uni.lu/R3/R3LabCards/blob/master/R3labCARD-backup_services.md). |
| HPC Folders | Yes | Yes | No | Only **$HOME** and **$PROJECTWORK** folders are backed up. Frequencies per cluster listed [here](https://hpc.uni.lu/blog/2018/faq-backup-policy/).|
| LCSB Managed Virtual Machines | On-Demand | Yes | No | See Lab Card [Backup of Servers](./R3labCARD-backup_services.html).|
| owncloud.lcsb.uni.lu | Yes | No | No | Last 10 daily backups, one per month for the last 6 months|
| Staff Desktop | No | No | Yes | See Section **Backup Staff Computer**.|
| Staff Laptop | No | No | Yes | See Section **Backup Staff Computer**. |
| External Drive | No | No | Yes | See Section **Backup Staff Computer**.|
| dropit.uni.lu | Yes | No | Yes | See Section **Backup Staff Computer**.|
| Staff Desktop | No | No | Yes | See Lab Card [Backup Staff Computer](./R3labCARD-backup_computer.html).|
| Staff Laptop | No | No | Yes | See Lab Card [Backup Staff Computer](./R3labCARD-backup_computer.html). |
| External Drive | No | No | Yes | See Lab Card [Backup Staff Computer](./R3labCARD-backup_computer.html).|
| dropit.uni.lu | Yes | No | Yes | See Lab Card [Backup Staff Computer](./R3labCARD-backup_computer.html).|
**You should store the master copy of your data on server-side storage**; specifically on Atlas, HPC Folders or Virtual Machines (if the data is in databases or applications). As these storages provide built-in backup support, by using them you ensure 1 backup copy.
......@@ -37,57 +37,5 @@ Often, staff computers, laptop or desktop, contain other (working) copies of dat
Finally, LCSB's ownCloud is not recommended as an ultimate destination for the master or backup copies of data. It is intended as a data sharing platform to support collaborative work. Therefore, any data on ownCloud should be moved to recommended server-side storage once joint work is completed.
## Backup Staff Computer
### Backup to External Disk
#### macOS
macOS platform comes with Time Machine, a very easy to use backup feature. Time Machine backups can be set up as follows:
1. Connect an external disk to your computer.
2. Using Finder open up **/Applications/System Preferences** and Select **Time Machine**. <br/> ![Alt](mac_tm_1.png "Title")
3. From the **Time Machine** main screen click **Select Backup Disk** &nbsp; ![Alt](mac_tm_2.png "Title")
4. From the list select your external disk. It is also recommended that you select the option to **encrypt backups**. <br/> ![Alt](mac_tm_3.png "Title")
5. If you opt for encryption you will be asked for an encryption password. You will need this password whenever you re-connect the disk to your computer or whenever you restart your Mac. You should keep this password safe using [Password Management software](../passwords/). <br/> ![Alt](mac_tm_4.png "Title")
6. Time Machine requires an apple-specific disk format to work. Your external disk may not be in this format (e.g. may be in exFAT format which is accessible both by Windows and Mac). In such a case Time Machine will prompt you to approve the erasure and re-formatting of your external disk (see below). If you do not want to re-format, you may [create a disk image](../encryption/file/) on your external disk and then point Time Machine to this image. <br/> ![Alt](mac_tm_5.png "Title")
7. After you select a backup disk, provide password and confirm disk format (if applicable), Time Machine will automatically start the backup process in the background. You may continue to work. You will receive a notification, when the first backup is complete. <br/> ![Alt](mac_tm_6.png "Title")
8. If you want to initiate another backup manually, click the Time Machine logo in your menu bar and select **Back Up Now**. It is recommended that you enable the automatic backup feature of Time Machine. To do so, go to **/Applications/System Preferences/Time Machine** and select the option **Back Up Automatically**. <br/> ![Alt](mac_tm_7.png "Title")
Official instructions for restoring your Mac using a Time Machine backup can be found [here](https://support.apple.com/en-us/HT201250)
#### Windows
Windows's offerings for backup has different names in different versions.
* _Microsoft Backup and Restore_, which is available in [Windows 7] (http://windows.microsoft.com/en-US/windows7/Back-up-your-files ) and [Windows 10] (https://support.microsoft.com/en-us/help/17143/windows-10-back-up-your-files), allows backup of your entire system or selected folders/files on your computer.
* _File History_, which is is available in [Windows 8 and 10] (http://windows.microsoft.com/en-us/windows-8/set-drive-file-history), provides an easy to use option with which you can backup select files/folders. _File History_ runs periodically, tracks versions of your files, and allows you to go back older versions.
The steps to switch **File History** on your computer is as follows:
1. Connect an external disk to your computer.
2. From the **Start** menu select **Settings/Update & Security/Backup**. <br/> ![Alt](win_fh_1.png "Title")
3. Click the "+" next to Add a drive. You'll see a prompt to choose an external drive, choose the disk you have connected.
4. File History has now started archiving your data. An on/off slider will now appear under a new heading called **Automatically back up my files.** <br/> ![Alt](win_fh_2.png "Title")
5. By default, Windows 10's File History will back-up all the folders in the **User** folder. And it will run the backup on an hourly schedule. To change any of those settings click on More options under the on/off slider (see below). <br/> ![Alt](win_fh_2_2.png "Title")
6. In the **Backup Options** dialog, under the **Back up these folders.** you can remove existing folders and add new ones. **Important Note:** Make sure that any application that read/write your data files/folder are closed, when you're making this _File History_ configuration. <br/> ![Alt](win_fh_3.png "Title")
The instructions for restoring your files from **File History** can be found [here](https://it.nmu.edu/docs/backup-using-windows-file-history)
### Backup to dropit.uni.lu
The University of Luxembourg's cloud service [DropIt](dropit.uni.lu) provides all staff with 100GB of storage.
To use this service, first, visit [dropit.uni.lu](https://dropit.uni.lu/). You should be able to login with your university credentials. In case of login problems, request IT support using [ServiceNow](https://service.uni.lu/).
Second, in order to create backups of your computer on DropIt cloud, you need to install a client application. The University IT Guideline [here](https://intranet.uni.lux/the_university/siu/Documents/Dropit%20client%20-%20configuration%20guide.pdf#search=dropit) provides the details on how to install this client application for macOS, Windows and Linux platforms, and also, how to configure the backup process on your computer.
backup/R3labCARD-backup_computer.md 0 → 100644
+ 47
− 0
View file @ a2791e6a
---
layout: page
---
# Backup Staff Computer
## Backup to External Disk
### macOS
macOS platform comes with Time Machine, a very easy to use backup feature. Time Machine backups can be set up as follows:
1. Connect an external disk to your computer.
2. Using Finder open up **/Applications/System Preferences** and Select **Time Machine**. <br/> ![Alt](mac_tm_1.png "Title")
3. From the **Time Machine** main screen click **Select Backup Disk** &nbsp; ![Alt](mac_tm_2.png "Title")
4. From the list select your external disk. It is also recommended that you select the option to **encrypt backups**. <br/> ![Alt](mac_tm_3.png "Title")
5. If you opt for encryption you will be asked for an encryption password. You will need this password whenever you re-connect the disk to your computer or whenever you restart your Mac. You should keep this password safe using [Password Management software](../passwords/). <br/> ![Alt](mac_tm_4.png "Title")
6. Time Machine requires an apple-specific disk format to work. Your external disk may not be in this format (e.g. may be in exFAT format which is accessible both by Windows and Mac). In such a case Time Machine will prompt you to approve the erasure and re-formatting of your external disk (see below). If you do not want to re-format, you may [create a disk image](../encryption/file/) on your external disk and then point Time Machine to this image. <br/> ![Alt](mac_tm_5.png "Title")
7. After you select a backup disk, provide password and confirm disk format (if applicable), Time Machine will automatically start the backup process in the background. You may continue to work. You will receive a notification, when the first backup is complete. <br/> ![Alt](mac_tm_6.png "Title")
8. If you want to initiate another backup manually, click the Time Machine logo in your menu bar and select **Back Up Now**. It is recommended that you enable the automatic backup feature of Time Machine. To do so, go to **/Applications/System Preferences/Time Machine** and select the option **Back Up Automatically**. <br/> ![Alt](mac_tm_7.png "Title")
Official instructions for restoring your Mac using a Time Machine backup can be found [here](https://support.apple.com/en-us/HT201250)
### Windows
Windows's offerings for backup has different names in different versions.
* _Microsoft Backup and Restore_, which is available in [Windows 7] (http://windows.microsoft.com/en-US/windows7/Back-up-your-files ) and [Windows 10] (https://support.microsoft.com/en-us/help/17143/windows-10-back-up-your-files), allows backup of your entire system or selected folders/files on your computer.
* _File History_, which is is available in [Windows 8 and 10] (http://windows.microsoft.com/en-us/windows-8/set-drive-file-history), provides an easy to use option with which you can backup select files/folders. _File History_ runs periodically, tracks versions of your files, and allows you to go back older versions.
The steps to switch **File History** on your computer is as follows:
1. Connect an external disk to your computer.
2. From the **Start** menu select **Settings/Update & Security/Backup**. <br/> ![Alt](win_fh_1.png "Title")
3. Click the "+" next to Add a drive. You'll see a prompt to choose an external drive, choose the disk you have connected.
4. File History has now started archiving your data. An on/off slider will now appear under a new heading called **Automatically back up my files.** <br/> ![Alt](win_fh_2.png "Title")
5. By default, Windows 10's File History will back-up all the folders in the **User** folder. And it will run the backup on an hourly schedule. To change any of those settings click on More options under the on/off slider (see below). <br/> ![Alt](win_fh_2_2.png "Title")
6. In the **Backup Options** dialog, under the **Back up these folders.** you can remove existing folders and add new ones. **Important Note:** Make sure that any application that read/write your data files/folder are closed, when you're making this _File History_ configuration. <br/> ![Alt](win_fh_3.png "Title")
The instructions for restoring your files from **File History** can be found [here](https://it.nmu.edu/docs/backup-using-windows-file-history)
## Backup to dropit.uni.lu
The University of Luxembourg's cloud service [DropIt](dropit.uni.lu) provides all staff with 100GB of storage.
To use this service, first, visit [dropit.uni.lu](https://dropit.uni.lu/). You should be able to login with your university credentials. In case of login problems, request IT support using [ServiceNow](https://service.uni.lu/).
Second, in order to create backups of your computer on DropIt cloud, you need to install a client application. The University IT Guideline [here](https://intranet.uni.lux/the_university/siu/Documents/Dropit%20client%20-%20configuration%20guide.pdf#search=dropit) provides the details on how to install this client application for macOS, Windows and Linux platforms, and also, how to configure the backup process on your computer.
backup/R3labCARD-backup_services.md 0 → 100644
+ 69
− 0
View file @ a2791e6a
---
layout: page
---
This card describe the procedure to follow in order to **Backup
services** running on a server e.g. a *database*, *configuration
scripts*, or *any substantial file* to ensure that the service is
running. Here are the steps to follow:
> If your machine is provided by LCSB sysadmins, you only have to
take care about step 3 and 4, and maybe 5. Enjoy ;)
1. Add a user on your machine that will hold backups. We call it
“backup” as default. On Debian/Ubuntu
sudo adduser backup --gecos 'User holding backups' --disabled-password --home /opt/backup --system --shell '/bin/bash'
or on CentOS
adduser backup -r -m -b '/opt/' --shell '/bin/bash'
2. Add the public ssh key to allow the backup remote user to connect to
your machine.
``` {.bash language="bash"}
echo "<SSH PUBLIC KEY>" > /opt/backup/.ssh/authorized_keys
```
Ask a [sysadmin](mailto:lcsb-sysadmins@uni.lu) to provide you the
public key.
3. Prepare your backup. It can be any file, prefer a *tarball* if you
have many. Place the tarball into backup user home directory.
``` {.bash language="bash"}
mv mybackup.tgz /opt/backup/.
```
4. Set up a *cron job* to do it automatically every day. Don't forget to `chown` it to the backup user ;)
5. Open a ticket on [ServiceNow](https://service.uni.lu/sp?id=bs_dep_catalog_entry&sys_id=a9f9fcf71da932000a53049ff5773060) (LCSB -> BioCore: Server Provisioning Services -> Request for backup) and provide some settings about your backup:
1. hostname of the machine. Given by `hostname -f` (or the IP if
the server is not resolvable `hostname -i`).
2. Which user and the home directory you have chosen in step 1.
3. Additional SSH connection parameters that can be mandatory to
connect to your server. By default, SSH port used is 8022.
4. A list of files to backup. For instance “mybackup.tgz”. It could
also be a pattern. By default, “\*” is used.
5. A hash function that will allow to identify same files on
your machine. Default is *md5sum*.
6. Decide if you want to delete or not backup files from your
machine when after they are fetched by the backup server.
Default is not to delete.
7. Where to put your backup on the backup server. You may have
specific requirement. By default it goes to a standard location.
8. Decide your [backup strategy](https://en.wikipedia.org/wiki/Backup_rotation_scheme).
Default is 7, 4, 12, 10. Which means that we keep data for the
last 7 days, then 1/week for 4 weeks, 1/month for 12 months and
1/year for 10 years approximately. *Note that if your backup
data is devouring space, we may choose a Ten-Tape
Hanoi schedule.*
\ No newline at end of file
exchange-channels/R3labCARD-exchange-channels.md
+ 14
− 24
View file @ a2791e6a
......@@ -2,7 +2,6 @@
layout: page
permalink: /exchange-channels/
---
# Exchanging Research Data with Collaborators
LCSB provides two channels for the exchange of data with research collaborators:
......@@ -13,27 +12,25 @@ LCSB provides two channels for the exchange of data with research collaborators:
## Using LCSB ownCloud
A LUMS account is needed in order to use LCSB ownCloud. LUMS accounts for staff are normally created within the first few days of starting work at the LCSB. In addition to staff, researchers at Partner Institutes may be given LUMS accounts. If you have inquiries/requests on LUMS accounts, please create a ticket on [ServiceNow](https://service.uni.lu/) using catalog item _Home/Catalog/LCSB/BioCore_.
A LUMS account is needed in order to use LCSB ownCloud. LUMS accounts for staff are normally created within the first few days of starting work at the LCSB. In addition to staff, researchers at Partner Institutes may be given LUMS accounts. If you have inquiries/requests on LUMS accounts, please create a ticket on [ServiceNow](https://service.uni.lu/) using catalog item **Home/Catalog/LCSB/BioCore**.
Similar to other cloud storage systems, ownCloud is accessible both via a browser and also via a client application. On the web, LCSB's ownCloud is at [https://owncloud.lcsb.uni.lu/](https://owncloud.lcsb.uni.lu/) ![Alt](./images/owcld_1.png "Title")
Similar to other cloud storage systems, ownCloud is accessible both via a browser and also via a client application. On the web, LCSB's ownCloud is at [https://owncloud.lcsb.uni.lu/](https://owncloud.lcsb.uni.lu/) <br/> ![Alt](owcld_1.png "Title")
You can download the ownCloud client suitable for your staff computer [here](https://owncloud.lcsb.uni.lu/). User documentation on ownCloud tools and portal can be found [here](https://doc.owncloud.com/server/index.html).
LCSB staff should observe the following guidance when using ownCloud:
* Limit folder shares to only the personnel that needs to access data.
* When sharing via Links, always set a **password** and an **expiration date** for the link. \
![Alt](./images/owcld_2.png "Title")
* When sharing via Links, always set a **password** and an **expiration date** for the link. <br/> ![Alt](owcld_2.png "Title")
* When sharing via Links, **please do not send the link passwords to collaborators in email**. Instead, use secure password sharing utilities listed [here](https://address_of the_password_lab_card).
* When sharing via Links, **please do not send the link passwords to collaborators in email**. Instead, use secure password sharing tools listed [here](../passwords/).
## Using LCSB DUMA
The use of DUMA is mediated by the LCSB Data Stewardship team.
DUMA can be accessed via a web browser or via a command-line utility. In order to use DUMA, create a ticket on [ServiceNow](https://service.uni.lu/) using the catalog item _Home/Catalog/LCSB/BioCore_.
DUMA can be accessed via a web browser or via a command-line utility. In order to use DUMA, create a ticket on [ServiceNow](https://service.uni.lu/) using the catalog item **Home/Catalog/LCSB/BioCore**.
* If you want to make data available for download, then please specify in your ticket the current source location of your data (e.g. HPC, staff computer).
* If collaborators will be uploading data, then please specify in your ticket the target storage location you want your data to arrive (e.g. HPC, staff computer).
......@@ -43,28 +40,21 @@ DUMA can be accessed via a web browser or via a command-line utility. In order t
- a **token** (for command line)
- an **access link and a token** (for web)
When sharing DUMA access tokens with collaborators, use password link generator tools listed [here](https://address_of_the_password_lab_card).
When sharing DUMA access tokens with collaborators, use password link generator tools listed [here](../passwords/).
In the following we describe instructions on how to use DUMA web interface. The instructions for using the command-line client can be found [here](https://asperaweb.lcsb.uni.lu/help/-Sc1j1kj_zkkFAVjwKgMwbGMa7_iyiv6Ob7GwKV_)
1. you will receive an **access link** and a **token** from the Data Steward. Access link is a download/upload area that can be reached by you and/or collaborator. You will be prompted for the token to reach the access link (see below). ![Alt](./images/duma_1.png "Title")
1. you will receive an **access link** and a **token** from the Data Steward. Access link is a download/upload area that can be reached by you and/or collaborator. You will be prompted for the token to reach the access link (see below). ![Alt](duma_1.png "Title")
2. When you visit a DUMA access link for the first time, you will be prompted to install **IBM Aspera Connect** client.
* click **Download** or **Download latest version** links (see below). ![Alt](./images/duma_2.png "Title")
* Your browser will download the client installer. \
![Alt](./images/duma_3.png "Title")
* Run the installer till you receive the completion message. \
![Alt](./images/duma_4.png "Title")
* Once **IBM Aspera Connect** client is installed, refresh the **access link**. You will be prompted whether you want to open **IBM Aspera Launcher**, click **Open**. \
![Alt](./images/duma_5.png "Title")
* click **Download** or **Download latest version** links (see below). ![Alt](duma_2.png "Title")
* Your browser will download the client installer. <br/> ![Alt](duma_3.png "Title")
* Run the installer till you receive the completion message. <br/> ![Alt](duma_4.png "Title")
* Once **IBM Aspera Connect** client is installed, refresh the **access link**. You will be prompted whether you want to open **IBM Aspera Launcher**, click **Open**. <br/> ![Alt](duma_5.png "Title")
3. The **access link** page will display a **File Browser** section. Depending on the settings per access link, users can create or delete folders in the File Browser and upload or download data.\
![Alt](./images/duma_6.png "Title")
4. Clicking **Upload** or **Download** will launch the **IBM Aspera Connect** client on your computer. You first will be asked whether you allow the client to connect to aspera.lcsb.uni.lu. Choose **Allow**. \
![Alt](./images/duma_7.png "Title")
5. At any time you can launch **IBM Aspera Connect** to display the status of uploads to or downloads from your computer. \
![Alt](./images/duma_8.png "Title")
3. The **access link** page will display a **File Browser** section. Depending on the settings per access link, users can create or delete folders in the File Browser and upload or download data.<br/> ![Alt](duma_6.png "Title")
4. Clicking **Upload** or **Download** will launch the **IBM Aspera Connect** client on your computer. You first will be asked whether you allow the client to connect to aspera.lcsb.uni.lu. Choose **Allow**.<br/> ![Alt](duma_7.png "Title")
5. At any time you can launch **IBM Aspera Connect** to display the status of uploads to or downloads from your computer. <br/> ![Alt](duma_8.png "Title")
......
index.md
+ 5
− 6
View file @ a2791e6a
......@@ -9,17 +9,16 @@ Bioinformatics Core assists LCSB researchers with the organization, management,
## Current LabCards
* [Exchange data with collaborators](./exchange-channels/)
* [Backup guidance](./backup/)
* [Backup staff computer](./backup/R3labCARD-backup_computer.html)
* [Backup servers](./backup/R3labCARD-backup_services.html)
* [Checksums for data integrity](./checksum/)
* [De-Identification of human-subject data](./de-identification/)
* Encrypt [disks](./encryption/disk/) or [files and folders](./encryption/file/)
* [Password management](./passwords/)
* [Deleting data and sanitising media](./sanitisation/)
* [Backup of servers](R3labCARD-backup_services.md)
* [Link HPC to OwnCloud](R3labCARD-link_HPC_to_owncloud.md)
* [Link HPC to OwnCloud](./misc/)
\ No newline at end of file
misc/R3labCARD-link_HPC_to_owncloud.md 0 → 100644
+ 29
− 0
View file @ a2791e6a
---
layout: page
permalink: /misc/
---
# How to link a directory on the HPC cluster to your Owncloud
It's possible to configure your home directory or a project directory on the HPC clusters (Gaia and Chaos) as external storage in your Owncloud.
1. In the Owncloud webinterface, click on your name in the top right corner and select "Personal".
2. On the left in the menu, click on "External Storage".
3. Choose an appropriate folder name and in the "Add storage" dropdown menu select "SFTP".
4. Set "Authentication" to "RSA public key".
5. The value for "Host" is `access-gaia.uni.lu:8022` or `access-chaos.uni.lu:8022`, depending on which cluster you want to link.
6. In the next field select the directory on the cluster you want to link, e.g. `/home/users/<your_username>`. **NEVER KEEP "Root"**, because it will try to synchronise all files on the cluster you have access to to your computer, which might be several Terabytes! Also be careful with project directories, they often contain big amounts of data.
7. In the following field enter your HPC account's username.
8. Copy the ssh key from the last field, connect to the cluster and add it to your `~/.ssh/authorized_keys` file. Make sure it's in a single line.
Here's an example how the finished configuration in Owncloud looks like:
![Alt](link_HPC_owncloud.jpg)
\ No newline at end of file
misc/link_HPC_owncloud.jpg 0 → 100644
+ 0
− 0
View file @ a2791e6a
misc/link_HPC_owncloud.jpg

25 KiB

passwords/R3labCARD-passwords.md
+ 1
− 1
View file @ a2791e6a
......@@ -6,7 +6,7 @@ permalink: /passwords/
# Managing Your Passwords
There are several tools that can be used for password management. Some provide limited (basic) features for, some require a fee. Below are some password management tools that can be used:
- [LastPass](https://www.lastpass.com) (Free for most features). Paid access allows secure sharing of passwords e.g. for shared infrastrucuture. If you're in charge of the archival and management of research data for your group, please check with the LCSB IT Team whether they can provide you a LastPass subscription.
- [LastPass](https://www.lastpass.com) (Free for most features). Paid access allows secure sharing of passwords e.g. for shared infrastructure. If you're in charge of the archival and management of research data for your group, please check with the LCSB IT Team whether they can provide you a LastPass subscription.
- [dashlane](https://www.dashlane.com) (Free for limited features).
- [KeePass](https://keepass.info) (Free).
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment