PathoFact issueshttps://git-r3lab.uni.lu/laura.denies/PathoFact/-/issues2020-06-22T14:40:26+02:00https://git-r3lab.uni.lu/laura.denies/PathoFact/-/issues/45Reduce output size2020-06-22T14:40:26+02:00Valentina Galatavalentina.galata@uni.luReduce output sizeRemove all intermediate files which are not relevant to the user.
Files with relevant results should be compressed, e.g. output created by `PlasFlow`.
Check if there are very large log files.
* [x] make not relevant intermediate files t...Remove all intermediate files which are not relevant to the user.
Files with relevant results should be compressed, e.g. output created by `PlasFlow`.
Check if there are very large log files.
* [x] make not relevant intermediate files temporary w/ `temp(...)`
* [ ] redirect `hmmer`s stdout to `/dev/null` instead of the log file
* [ ] compress relevant intermediate files
* [x] add rule to removes files which cannot be made temp() (files used in snakemake checkpoints) Paper review - Microbiome - 1Laura DeniesLaura Denieshttps://git-r3lab.uni.lu/laura.denies/PathoFact/-/issues/13spring-clean: add .gitignore2020-04-20T14:04:47+02:00Valentina Galatavalentina.galata@uni.luspring-clean: add .gitignore* [x] add `.gitignore`
Impact: Will not affect the pipeline* [x] add `.gitignore`
Impact: Will not affect the pipelinespring-cleanValentina Galatavalentina.galata@uni.luValentina Galatavalentina.galata@uni.luhttps://git-r3lab.uni.lu/laura.denies/PathoFact/-/issues/33spring-clean: add test workflow2020-04-30T08:04:55+02:00Valentina Galatavalentina.galata@uni.luspring-clean: add test workflowAdd a test workflow. Purpose:
- check whether the pipeline can be run
- check whether the pipeline produces the expected results
Tasks:
* [x] create a new `snakemake` file
* [x] create workflow
* [x] create rules
Impact: Will extend th...Add a test workflow. Purpose:
- check whether the pipeline can be run
- check whether the pipeline produces the expected results
Tasks:
* [x] create a new `snakemake` file
* [x] create workflow
* [x] create rules
Impact: Will extend the pipeline, old workflows will remain untouchedspring-cleanValentina Galatavalentina.galata@uni.luValentina Galatavalentina.galata@uni.luhttps://git-r3lab.uni.lu/laura.denies/PathoFact/-/issues/24spring-clean: add workflow parameter2020-04-21T13:47:28+02:00Valentina Galatavalentina.galata@uni.luspring-clean: add workflow parameter* [x] Add `workflow` variable to `config.yaml`
* [x] Adjust `Snakemake`* [x] Add `workflow` variable to `config.yaml`
* [x] Adjust `Snakemake`spring-cleanValentina Galatavalentina.galata@uni.luValentina Galatavalentina.galata@uni.luhttps://git-r3lab.uni.lu/laura.denies/PathoFact/-/issues/15spring-clean: all scripts in scripts/2020-04-23T12:31:27+02:00Valentina Galatavalentina.galata@uni.luspring-clean: all scripts in scripts/* [x] mv scripts in `rules/` to `scripts/`
* [x] adjust paths when calling these scripts
Impact: Will not affect the pipeline* [x] mv scripts in `rules/` to `scripts/`
* [x] adjust paths when calling these scripts
Impact: Will not affect the pipelinespring-cleanValentina Galatavalentina.galata@uni.luValentina Galatavalentina.galata@uni.luhttps://git-r3lab.uni.lu/laura.denies/PathoFact/-/issues/31spring-clean: args in virulence_prediction.py and rule classifier in the vir ...2020-04-23T12:32:59+02:00Valentina Galatavalentina.galata@uni.luspring-clean: args in virulence_prediction.py and rule classifier in the vir stepScript `scripts/virulence_prediction.py` uses `snakemake` variable to access parameters but the rule `classfier` in `rules/Virulence/Virulence.snk` uses `shell` instead of `script`.Script `scripts/virulence_prediction.py` uses `snakemake` variable to access parameters but the rule `classfier` in `rules/Virulence/Virulence.snk` uses `shell` instead of `script`.spring-cleanValentina Galatavalentina.galata@uni.luValentina Galatavalentina.galata@uni.luhttps://git-r3lab.uni.lu/laura.denies/PathoFact/-/issues/23spring-clean: clean up YAML files2020-04-21T15:26:09+02:00Valentina Galatavalentina.galata@uni.luspring-clean: clean up YAML filesRemove `name` and `prefix` from `conda` YAML files, `conda` does not use them anyway.
Impact: Will not affect the pipelineRemove `name` and `prefix` from `conda` YAML files, `conda` does not use them anyway.
Impact: Will not affect the pipelinespring-cleanValentina Galatavalentina.galata@uni.luValentina Galatavalentina.galata@uni.luhttps://git-r3lab.uni.lu/laura.denies/PathoFact/-/issues/19spring-clean: DeepARG2020-04-21T15:09:05+02:00Valentina Galatavalentina.galata@uni.luspring-clean: DeepARGDependency: `DeepARG`
* [x] `git submodule`
* [x] rm dir. in `scripts/`
* [x] rm `git clone` from `set-up.sh`
* [x] update path in `config.yaml`
* [x] set-up commands: replace path and make `DIAMOND` bin executable
```shell
git lfs ins...Dependency: `DeepARG`
* [x] `git submodule`
* [x] rm dir. in `scripts/`
* [x] rm `git clone` from `set-up.sh`
* [x] update path in `config.yaml`
* [x] set-up commands: replace path and make `DIAMOND` bin executable
```shell
git lfs install # if not already done
git submodule add https://gaarangoa@bitbucket.org/gusphdproj/deeparg-ss.git submodules/deeparg-ss
cd submodules/deeparg-ss/
git checkout 14b8dce
```
Impact: Could affect the results if there are any discrepancies between the versions of any dependency. Therefore:
* Confirm `git` version
* Use same YAML as in original repospring-cleanValentina Galatavalentina.galata@uni.luValentina Galatavalentina.galata@uni.luhttps://git-r3lab.uni.lu/laura.denies/PathoFact/-/issues/17spring-clean: DeepVirFinder2020-04-21T15:09:05+02:00Valentina Galatavalentina.galata@uni.luspring-clean: DeepVirFinderDependency: `DeepVirFinder`
* [x] `git submodule` (for version see below)
* [x] rm dir. in `scripts/`
* [x] rm `git clone` from `set-up.sh`
* [x] update path in `config.yaml`
```shell
git submodule add https://github.com/jessieren/Deep...Dependency: `DeepVirFinder`
* [x] `git submodule` (for version see below)
* [x] rm dir. in `scripts/`
* [x] rm `git clone` from `set-up.sh`
* [x] update path in `config.yaml`
```shell
git submodule add https://github.com/jessieren/DeepVirFinder.git submodules/DeepVirFinder
cd submodules/DeepVirFinder
git checkout ddb4a94
```
Impact: Could affect the results if there are any discrepancies between the versions of any dependency. Therefore:
- Confirm `git` version
- Use same YAML as in original repospring-cleanValentina Galatavalentina.galata@uni.luValentina Galatavalentina.galata@uni.luhttps://git-r3lab.uni.lu/laura.denies/PathoFact/-/issues/32spring-clean: escape special chars in rules properly2020-04-30T08:05:22+02:00Valentina Galatavalentina.galata@uni.luspring-clean: escape special chars in rules properlyEscape special chars in rules' shell call properly, e.g. replace `\t` by `\\t`.
Reason: Prevents `snakemake` from interpreting these characters.
Impact: Should not affect the pipelineEscape special chars in rules' shell call properly, e.g. replace `\t` by `\\t`.
Reason: Prevents `snakemake` from interpreting these characters.
Impact: Should not affect the pipelinespring-cleanValentina Galatavalentina.galata@uni.luValentina Galatavalentina.galata@uni.luhttps://git-r3lab.uni.lu/laura.denies/PathoFact/-/issues/18spring-clean: give snakemake files an extension2020-04-24T10:43:37+02:00Valentina Galatavalentina.galata@uni.luspring-clean: give snakemake files an extensionAdd `.smk` to all `snakemake` files in `rules/` and `workflows/`.
Makes it easier to recognize `snakemake` files and to have code highlighting in editors.Add `.smk` to all `snakemake` files in `rules/` and `workflows/`.
Makes it easier to recognize `snakemake` files and to have code highlighting in editors.spring-cleanValentina Galatavalentina.galata@uni.luValentina Galatavalentina.galata@uni.luhttps://git-r3lab.uni.lu/laura.denies/PathoFact/-/issues/22spring-clean: HMMER2020-04-21T15:21:16+02:00Valentina Galatavalentina.galata@uni.luspring-clean: HMMERDependency: 'HMMER'
* [x] `conda` YAML file
* [x] update path in `config.yaml`
* [x] update rule(s)
Impact: Could affect the results if there are any discrepancies between the versions of any dependency. Therefore
- Confirm version: `3...Dependency: 'HMMER'
* [x] `conda` YAML file
* [x] update path in `config.yaml`
* [x] update rule(s)
Impact: Could affect the results if there are any discrepancies between the versions of any dependency. Therefore
- Confirm version: `3.2.1`spring-cleanValentina Galatavalentina.galata@uni.luValentina Galatavalentina.galata@uni.luhttps://git-r3lab.uni.lu/laura.denies/PathoFact/-/issues/28spring-clean: logging2020-04-21T15:53:04+02:00Valentina Galatavalentina.galata@uni.luspring-clean: logging- Add `log` to rules (if applicable)
- Add logging in R-scripts
```R
# logging in R scripts
sink(file=file(snakemake@log[[1]], open="wt"), type="message")
```
Impact: Will not affect the pipeline- Add `log` to rules (if applicable)
- Add logging in R-scripts
```R
# logging in R scripts
sink(file=file(snakemake@log[[1]], open="wt"), type="message")
```
Impact: Will not affect the pipelinespring-cleanValentina Galatavalentina.galata@uni.luValentina Galatavalentina.galata@uni.luhttps://git-r3lab.uni.lu/laura.denies/PathoFact/-/issues/16spring-clean: main YAML file2020-04-20T14:18:45+02:00Valentina Galatavalentina.galata@uni.luspring-clean: main YAML fileCreate main `conda` YAML file containing
- `python=3.6.4`
- `snakemake=5.5.4`Create main `conda` YAML file containing
- `python=3.6.4`
- `snakemake=5.5.4`spring-cleanValentina Galatavalentina.galata@uni.luValentina Galatavalentina.galata@uni.luhttps://git-r3lab.uni.lu/laura.denies/PathoFact/-/issues/14spring-clean: mv conda_environments and YAML files2020-04-20T13:46:18+02:00Valentina Galatavalentina.galata@uni.luspring-clean: mv conda_environments and YAML files* [x] mv `conda_environments/` to `envs/` - shorter and more convenient
* [x] mv YAML files: name based on tool, `*.yaml`
* [x] adjust paths in scripts
Impact: Will not affect the pipeline* [x] mv `conda_environments/` to `envs/` - shorter and more convenient
* [x] mv YAML files: name based on tool, `*.yaml`
* [x] adjust paths in scripts
Impact: Will not affect the pipelinespring-cleanValentina Galatavalentina.galata@uni.luValentina Galatavalentina.galata@uni.luhttps://git-r3lab.uni.lu/laura.denies/PathoFact/-/issues/20spring-clean: PlasFlow2020-04-21T15:09:05+02:00Valentina Galatavalentina.galata@uni.luspring-clean: PlasFlowDependency: `PlasFlow`
* [x] `conda` YAML file
* [x] rm dir. in `scripts/`
* [x] update path in `config.yaml`
* [x] add YAML to rule(s) calling `PlasFlow`
Impact: Could affect the results if there are any discrepancies between the versi...Dependency: `PlasFlow`
* [x] `conda` YAML file
* [x] rm dir. in `scripts/`
* [x] update path in `config.yaml`
* [x] add YAML to rule(s) calling `PlasFlow`
Impact: Could affect the results if there are any discrepancies between the versions of any dependency.
Using `conda` installation (`plasflow=1.1.0`) instead of `git` repo (commit `v1.1-11-g82e9c75`).
**NOTE**: [release v1.1 vs. v1.1-11-g82e9c75](https://github.com/smaegol/PlasFlow/compare/v1.1...v1.1-11-g82e9c75)
- README update
- changed description of arg parser in `PlasFlow.py`
- no other changes in the code
--> No critical changesspring-cleanValentina Galatavalentina.galata@uni.luValentina Galatavalentina.galata@uni.luhttps://git-r3lab.uni.lu/laura.denies/PathoFact/-/issues/25spring-clean: rm `configfile` from snakemake rule files2020-04-21T14:08:06+02:00Valentina Galatavalentina.galata@uni.luspring-clean: rm `configfile` from snakemake rule filesRemove `configfile` in all `snakemake` files except in `Snakemake`.
Otherwise it is impossible to use a custom file to call the pipeline.
Impact: Will not affect the pipelineRemove `configfile` in all `snakemake` files except in `Snakemake`.
Otherwise it is impossible to use a custom file to call the pipeline.
Impact: Will not affect the pipelinespring-cleanValentina Galatavalentina.galata@uni.luValentina Galatavalentina.galata@uni.luhttps://git-r3lab.uni.lu/laura.denies/PathoFact/-/issues/26spring-clean: snakemake rules indentation2020-04-21T14:08:28+02:00Valentina Galatavalentina.galata@uni.luspring-clean: snakemake rules indentationFix formatting in all `snakemake` files (do **NOT** change the commands)
- Indentation: 4 spaces
Impact: Will not affect the pipelineFix formatting in all `snakemake` files (do **NOT** change the commands)
- Indentation: 4 spaces
Impact: Will not affect the pipelinespring-cleanValentina Galatavalentina.galata@uni.luValentina Galatavalentina.galata@uni.luhttps://git-r3lab.uni.lu/laura.denies/PathoFact/-/issues/29spring-clean: snakemake rules: use names in shell/run/script2020-04-30T09:08:33+02:00Valentina Galatavalentina.galata@uni.luspring-clean: snakemake rules: use names in shell/run/scriptWhen accessing `snakemake` variables use names if provided, e.g. `snakemake@input[["faa"]]` in an `R` script.
Impact: Will not affect the pipelineWhen accessing `snakemake` variables use names if provided, e.g. `snakemake@input[["faa"]]` in an `R` script.
Impact: Will not affect the pipelinespring-cleanValentina Galatavalentina.galata@uni.luValentina Galatavalentina.galata@uni.luhttps://git-r3lab.uni.lu/laura.denies/PathoFact/-/issues/30spring-clean: test data set2020-04-29T16:30:25+02:00Valentina Galatavalentina.galata@uni.luspring-clean: test data setAdd a test data set to be used to check whether the pipeline runs through and whether the results are consistent.
* [x] Create/copy input files (from `IMP3` test data set)
* [x] Create expected output (using `PathoFact`'s version of the...Add a test data set to be used to check whether the pipeline runs through and whether the results are consistent.
* [x] Create/copy input files (from `IMP3` test data set)
* [x] Create expected output (using `PathoFact`'s version of the `master` branch)
* [x] Create a config file
*Note: Scripts to compare the output will be part of a separate issue*spring-cleanValentina Galatavalentina.galata@uni.luValentina Galatavalentina.galata@uni.lu