Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • R3/school/courses
  • vilem.ded/courses
  • todor.kondic/courses
  • noua.toukourou/courses
  • nene.barry/courses
  • laurent.heirendt/courses
  • marina.popleteeva/courses
  • jenny.tran/courses
8 results
Show changes
Showing
with 653 additions and 0 deletions
../../../../2020/2020-06-09_IT101-DM/slides/img/red-cross.png
\ No newline at end of file
../../../../2020/2020-06-09_IT101-DM/slides/img/reproducibility_nature.png
\ No newline at end of file
../../../../2020/2020-06-09_IT101-DM/slides/img/rudi_balling.jpg
\ No newline at end of file
../../../../2020/2020-06-09_IT101-DM/slides/img/scripts
\ No newline at end of file
../../../../2020/2020-06-09_IT101-DM/slides/img/undraw_secure_server_s9u8.png
\ No newline at end of file
../../../../2020/2020-06-09_IT101-DM/slides/img/wordcloud.png
\ No newline at end of file
# IT101 - Working with computers
<br>IT101 - Working with computers<br>
## March 09th, 2021
<div style="top: 6em; left: 0%; position: absolute;">
<img src="theme/img/lcsb_bg.png">
</div>
<div style="top: 5em; left: 60%; position: absolute;">
<img src="slides/img/r3-training-logo.png" height="200px">
<br><br><br><br>
<h3></h3>
<br><br><br>
<h4>
Vilem Ded<br>
Data Steward<br>
vilem.ded@uni.lu<br>
<i>Luxembourg Centre for Systems Biomedicine</i>
</h4>
</div>
# Data housekeeping
## Available data storage
<div class='fragment' style="position:absolute">
<img src="slides/img/LCSB_storages_full.png" height="750px">
</div>
<div class='fragment' style="position:absolute">
<img src="slides/img/LCSB_storages_personal-crossed.png" height="750px">
</div>
<div style="position:absolute; width:45%; left:50%; top:28em; text-align:right">
<a href=" https://howto.lcsb.uni.lu/?policies:LCSB-POL-BIC-02" style="color:grey; font-size:0.8em;">Data Storage and Backup Policy</a>
</div>
# Data ingestion/transfer
## Receiving and sending data
<img height="400px" style="position:relative;left:10%" src="slides/img/banned_exchange_channels.png"><br>
<div style="position:absolute; left:10%;width:30%">
## E-mail is not for data transfer
* Avoid transfer of any data by e-mail
* E-mail is a poor repository
* Data can be read on passage
</div>
<div class="fragment" style="left:50%; width:30%; position:absolute">
## Exchanging data
* Share on Atlas server
* OwnCloud share (LCSB - BioCore)
* DropIt service (SIU)
* LFT (IBM Aspera) share for sensitive data
</div>
</div>
<div style="position:absolute; width:45%; left:50%; top:28em; text-align:right">
<a href=" https://howto.lcsb.uni.lu/?policies:LCSB-POL-BIC-05" style="color:grey; font-size:0.8em;">Research Human Data Sharing Policy</a>
</div>
# Data ingestion/transfer
Data can be corrupted:
* (non-)malicious modification
* faulty file transfer
* disk corruption
<div class="fragment">
### Solution
* disable write access to the source data
* generate checksums!
<div style="position:absolute;left:40%">
<img src="slides/img/checksum.png" width="500px">
</div>
</div>
<div class="fragment" style="position:relative; left:0%">
## When to generate checksums?
* before data transfer
- new dataset from collaborator
- upload to remote repository
* long term storage
- master version of dataset
- snapshot of data for publication
</div>
<div style="position:absolute; width:45%; left:50%; top:28em; text-align:right">
<a href=" https://howto.lcsb.uni.lu/?policies:LCSB-POL-BIC-02" style="color:grey; font-size:0.8em;">Data Storage and Backup Policy</a>
</div>
# Introduction
<div class="fragment" style="position:absolute">
<img height="450px" src="slides/img/wordcloud.png"><br>
## Learning objectives
* How to manage your data
* How to look and analyze your data
* Solving issues with computers
* Reproduciblity in the research data life cycle
</div>
<div class="fragment" style="position:relative;left:50%; width:40%">
<div >
<center>
<img height="405px" src="slides/img/rudi_balling.jpg"><br>
Prof. Dr. Rudi Balling, director
</center>
</div>
## Pertains to practically all people at LCSB
* Scientists
* PhD candidates
* Technicians
* Administrators
</div>
[
{"filename": "index.md"},
{"filename": "introduction.md"},
{"filename": "data-introduction.md"},
{"filename": "data_flow.md"},
{"filename": "ingestion.md"},
{"filename": "storage_setup.md"},
{"filename": "data-housekeeping.md"},
{"filename": "howtos.md"},
{"filename": "reproducibility.md"},
{"filename": "code_versioning.md"},
{"filename": "visualization.md"},
{"filename": "problem_solving.md"},
{"filename": "fair-principles.md"},
{"filename": "r3_group.md"},
{"filename": "thanks.md"}
]
## Overview
0. Introduction - learning objectives + targeted audience
1. Data workflow
1. Ingestion:
* receiving/sending/sharing data
* file naming
* checksums
* backup
1. making data tidy
* what is table
*
1. Learning to code workflows and analyses - excel files, coding
1. Code versioning and reproducibility
1. Visualization
* see the data
1. problem solving
* guide
* rubberducking
* google for help
* oracle
1. R3 team
1. Acknowledgment
1. data minimization
# Problem solving
A guide for solving computing issues
1. Express the problem
* Write down what you want to achieve
2. Search for help
* Read **FAQs**, **help pages** and the **official documentation** well before turning to Google
* Use stack exchange, forums and related resources carefully
3. Ask an expert
* You have to submit the problem in writing
* Make the question interesting
* If you supply a trivial problem, it will stop answering
# Responsible and Reproducible Research (R<sup>3</sup>)
## What is R<sup>3</sup>?
A multi-facetted change management
process built on 3 pillars:
- R3 pathfinder
- R3 school
- R3 accelerator
Common link module: R3 clinic
<div style="top: -1em; left: 50%; position: absolute;">
<img src="slides/img/3pillars-full.png">
</div>
<br>
<br>
<br>
<br>
<aside class="notes">
Pathfinder - policies, finding optimal data management changes<br>
School - courses, howtos, trainnings<br>
Accelerator - advanced teams and their boost/support, CI/CD setup<br>
Clinic - hands-on, meetings in groups, code review + suggestions<br>
</aside>
## R<sup>3</sup> Training
* LCSB's Monthly Data Management and Data Protection training
* ELIXIR Luxembourg's trainings <br>
https://elixir-luxembourg.org/training
* R<sup>3</sup> school Git basics - every 4 months
<aside class="notes">
Direct newcommers to this monthly training
</aside>
# Responsible and Reproducible Research (R<sup>3</sup>)
<section data-transition="none" data-background-image="slides/img/r3-training-logo.png" data-background-size="1000px" data-background-opacity="0.1">
</section>
<div style="display:block;text-align:center;position:relative;" >
<div class="profile-container">
* Reinhard Schneider
* <img src="slides/img/R3_profile_pictures/reinhard_schneider.png">
* Head of Bioinformatics Core
</div>
<div class="profile-container">
* Pinar Alper
* <img src="slides/img/R3_profile_pictures/pinar_alper.png">
* Datasteward
</div>
<div class="profile-container">
* Yohan Yarosz</li>
* <img src="slides/img/R3_profile_pictures/yohan_yarosz.png">
* Development
</div>
<div class="profile-container">
* Laurent Heirendt</li>
* <img src="slides/img/R3_profile_pictures/laurent_heirendt.png">
* Git, CI
</div>
<div class="profile-container">
* Sarah Peter</li>
* <img src="slides/img/R3_profile_pictures/sarah_peter.png">
* Infrastructure
</div>
<div class="profile-container">
* Valentine Groues</li>
* <img src="slides/img/R3_profile_pictures/valentine_groues.png">
* Development
</div>
<div class="profile-container">
* Vilem Ded</li>
* <img src="slides/img/R3_profile_pictures/vilem_ded.png">
* Datasteward
</div>
<div class="profile-container">
* Noua Toukourou</li>
* <img src="slides/img/R3_profile_pictures/noua_toukourou.png">
* Infrastructure
</div>
<div class="profile-container">
* Alexey Kolodkin</li>
* <img src="slides/img/R3_profile_pictures/alexey_kolodkin.png">
* Datasteward
</div>
<div class="profile-container">
* Maharshi Vyas</li>
* <img src="slides/img/R3_profile_pictures/maharshi_vyas.png">
* Infrastructure
</div>
<div class="profile-container">
* Nene Barry</li>
* <img src="slides/img/R3_profile_pictures/nene_barry.png">
* Datasteward
</div>
<div class="profile-container">
* Karim Chaouch</li>
* <img src="slides/img/R3_profile_pictures/karim_chaouch.png">
* Development
</div>
<div class="profile-container">
* Christophe Trefois
* <img src="slides/img/R3_profile_pictures/christophe_trefois.png">
* R<sup>3</sup> team lead
</div>
</div>
# Reproducibility
* ensures credibility
* key requirement for follow-up and collaborative studies
<div style="position:absolute">
<img src="slides/img/reproducibility_nature.png" height="650px">
</div>
<div class="fragment" style="position:relative;left:50%">
## Why is our workflow not reproducible?
Lack of provenance:
* Input data downloaded from “some website”
* Copy & paste operations
* Manual text entry
* Analysis not coded
* Intermediate and final data not generated by deterministic processes
</div>
# Reproducibility
## Learning to code workflows and analyses
<div style="display:inline-grid;grid-gap: 40px;grid-template-columns: auto auto;position:relative;left:12%">
<div class="fragment">
<div class="content-box">
<div class="box-title red">Spreadsheets alone</div>
<div class="content">
* Is great for looking at data.
* Data entry is fast.
* Analysis flow is hidden and not in focus.
</div>
</div>
<div style="text-align:center">
<img src="slides/img/excel_data-sheet.png" height="280px">
</div>
</div>
<div class="fragment">
<div class="content-box">
<div class="box-title">Coding</div>
<div class="content">
* Is great for controlling analysis
* Data is hidden.
* Flow is visible.
</div>
</div>
<img src="slides/img/code-example.png" height="280px">
</div>
</div>
<div class="content-box fragment" style="left:15%;width:60%;position:relative">
<div class="box-title green">Develop data science skills</div>
<div class="content">
* Develop good data management and analysis habits.
* Start coding your analysis within spreadsheets.
* Make yourself familiar with a statistics environment such as R, Python or Matlab
* No need to learn a high level programming language such as C++ or Java.
</div>
</div>
</div>
# Table
<div style="position:absolute">
"Tabular format of data"
### Header
* one line!
* **good** names of columns
### Rows
* represent observations/entities
### Columns
* represent property of the observations
* one data type
</div>
<div style="left:50%; position:relative; top:-2em">
<img src="slides/img/excel_data-sheet.png" width="700px">
<div class="fragment" data-fragment-index="3" style="position:absolute">
<img src="slides/img/excel_analyses-sheet.jpeg" width="700px"><br>
</div>
<div class="fragment" data-fragment-index="4" style="position:relative">
<img src="slides/img/red-cross.png" width="700px"><br>
</div>
</div>
# Storage set-up
* Download Anti-virus software
* Regularly update your SW/OS
* Encrypt movable media
### Passwords
* Strong passwords
* Password manager
* Safe password exchange channels
* Expiration time on password share
### Backup
* take care of your own backups!
* don't work on your backup copy!
* minimum is <b>3-2-1 backup rule</b>
<div style="position:absolute;right:10%;top:10%">
<img src="slides/img/undraw_secure_server_s9u8.png" height="750px">
</div>
<div style="position:absolute; width:45%; left:50%; top:28em; text-align:right">
<a href=" https://howto.lcsb.uni.lu/?policies:LCSB-POL-BIC-02" style="color:grey; font-size:0.8em;">Data Storage and Backup Policy</a>
</div>
# Storage set-up
## Backup - Central IT/LCSB
<div style="position:relative">
<img src="slides/img/LCSB_storages_backed-up.png" height="750px">
</div>
<div style="position:absolute;left:65%;top:60%">
Server administrators take care of:
* server backups
* LCSB OwnCloud backups
* group/application server backups (not always)
</div>
# Storage set-up
## Backup - personal research data
<div style="position:relative">
<img src="slides/img/LCSB_storages_backup.png" height="750px">
</div>
<div style="position:absolute;left:55%;top:70%">
<font color="red">One version should reside on Atlas!</font>
</div>
# Thank you.<sup> </sup>
<center><img src="slides/img/r3-training-logo.png" height="200px"></center>
<br>
<br>
<br>
<br>
<center>
Contact us if you need help:
<a href="mailto:lcsb-r3@uni.lu">lcsb-r3@uni.lu</a>
</center>
Links:
HowTo Cards / Policies: https://howto.lcsb.uni.lu/
Course Slides: https://courses.lcsb.uni.lu/
Internal Presentations: https://presentations.lcsb.uni.lu/
LCSB GitLab: https://git-r3lab.uni.lu/
HPC: https://hpc.uni.lu/
Service Portal: https://service.uni.lu/sp
# Visualization
<center>
**Plot your data!**
<figure>
<img src="slides/img/DinoSequentialSmaller.gif" height="500px">
<blockquote>"never trust summary statistics alone; always visualize your data"</blockquote>
<figcaption>--Alberto Cairo</figcaption>
</figure>
</center>
# Visualization
<center>
**Plot your data!**
<figure>
<img src="slides/img/plot-data.png" height="800px">
</figure>
</center>
# Best practices
* `pull` before `push` and, generally, before starting to work
* Work on your <font color="red">own</font> branch (in your own fork), and **not** on `master` and **not** on `develop`
* Do **not push** to `master`, but **submit a Pull Request (PR)**
* Get your code **reviewed** by your peers (submit a PR!)
* Submit a PR **often**!
* `clone` a repository, do not download the `.zip` file.
* Do **not** combine `git` commands
```bash
$ git commit -am "myMessage" # do not do this
```
* Stage only 1 file at once using
```bash
$ git add myFile.md
```
* Commit **only a few files** at once (after multiple separate `git add` commands)
* `Push` often - avoid conflicts
Remember: **A `push` a day keeps conflicts away!**
# Development scheme
Generally, in a repository, there are guidelines for contributing.
<div class="fragment">
A common development scheme is dual with a:
- **development** version of the code on `develop`
- **stable** version of the code on `master`
A **version** of the code is referred to as a **branch**.
<div class="fragment">
<img src="slides/img/icon-live-demo.png" height="100px">
<font color="red">In the practice repository, the development branch is called `develop`!</font>
<div class="fragment">
![bulb](slides/img/bulb.png) Use this dual development scheme for your own repositories!
# Branches
A **version** of the code (i.e., a **branch**) is made up of a sequence of code changes.
<div class="fragment">
These individual code changes are called **commits**.
For instance, the `master` and `develop` branches can be represented as a timeline:
<img src="slides/img/branch-master.png" class="branch-master" height="500em"/>
# Switch between branches
List all branches of the repository with
```bash
$ git branch -a
```
Exit by typing `q`. The branch with the * is the current branch.
<div class="fragment">
Checkout another branch
```bash
$ git checkout <branchName>
```
<div class="fragment">
You can switch to the `develop` branch with
```bash
$ git checkout develop
```
If the local branch does not exist but the remote does, it is created automatically.
<div class="fragment">
<img src="slides/img/icon-live-demo.png" height="100px">
# Create your own version
Assume that you want to work on a file:
<div class="fragment">
<font color="red">Create a new **branch**!</font>
```bash
$ git checkout -b myBranch
```
The `-b` flag creates the branch. Locally, you have your own version now:
<img src="slides/img/branch-create.png" class="branch-create" height="500em"/>
Push your version to your fork:
```bash
$ git push origin myBranch
```
<div class="fragment">
<img src="slides/img/icon-live-demo.png" height="100px">
# How do I start working on a repository?
You have to `clone` it first:
```bash
$ git clone git@github.com:LCSB-BioCore/basic-git-practice.git
```
If you did not configure your SSH key, clone using HTTPS:
```bash
$ git clone https://github.com/LCSB-BioCore/basic-git-practice.git
```
You will be prompted to enter your credentials.