IMP issueshttps://git-r3lab.uni.lu/IMP/IMP/-/issues2019-08-27T08:50:26+02:00https://git-r3lab.uni.lu/IMP/IMP/-/issues/92IncompleteFilesException error2019-08-27T08:50:26+02:00Federico BaldiniIncompleteFilesException errorFollowing error:
```bash
$ impy init --index ~/hg38.fa
[x] Found IMP docker-r3lab.uni.lu/imp/imp 1.4.1
IMP image found
Executing IMP command:
docker run --rm --name home_federico_impyinstall_imp_db -v /home/federico/impyinstall/imp-...Following error:
```bash
$ impy init --index ~/hg38.fa
[x] Found IMP docker-r3lab.uni.lu/imp/imp 1.4.1
IMP image found
Executing IMP command:
docker run --rm --name home_federico_impyinstall_imp_db -v /home/federico/impyinstall/imp-db:/home/imp/databases -e "LOCAL_USER_ID=`id -u $USER`" -e "LOCAL_GROUP_ID=`id -g $USER`" -e FILTER="hg38" docker-r3lab.uni.lu/imp/imp:1.4.1 snakemake -s /home/imp/code/rules/ini/init
IncompleteFilesException:
The files below seem to be incomplete. If you are sure that certain files are not incomplete, mark them as complete with
snakemake --cleanup-metadata <filenames>
To re-generate the files rerun your command with the --rerun-incomplete flag.
Incomplete files:
filtering/hg38.fa
```
I downloaded the file `hg38.fa` using:
```bash
$ wget http://hgdownload.cse.ucsc.edu/goldenPath/hg38/bigZips/hg38.fa.gz
```
and then unzipped it using:
```bash
$ gunzip hg38.fa.gz
```
Running the following command:
```bash
$ snakemake --cleanup-metadata ./imp-db/filtering/hg38.fa
```
yields:
```bash
Error: Snakefile "Snakefile" not present.
```
Can you help me out please? Thanks.Yohan Jaroszyohan.jarosz@uni.luYohan Jaroszyohan.jarosz@uni.luhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/88Migrate extract-unmapped rule to common rules2019-08-27T08:50:20+02:00Shaman NarayanasamyMigrate extract-unmapped rule to common rulesAt present this rule is duplicated to hybrid and single-omic. This should be preferably be made common at some point.At present this rule is duplicated to hybrid and single-omic. This should be preferably be made common at some point.Yohan Jaroszyohan.jarosz@uni.luYohan Jaroszyohan.jarosz@uni.luhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/39Security of docker container2019-08-27T08:50:12+02:00Shaman NarayanasamySecurity of docker containerLooking at the fact that docker requires super-user rights. It would be nice if the IMP script is able to throw error if users try to mount important OS directories (such a the root dir: / ).Looking at the fact that docker requires super-user rights. It would be nice if the IMP script is able to throw error if users try to mount important OS directories (such a the root dir: / ).Yohan Jaroszyohan.jarosz@uni.luYohan Jaroszyohan.jarosz@uni.luhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/136Accelerating the rRNA filtering step2018-10-10T12:18:37+02:00Cedric LacznyAccelerating the rRNA filtering step`sortmerna` appears to be a relatively slow program for the metatranscriptomic files we are typically processing.
An acceleration of this step would be nice!
There do not seem to be too many alternatives out there though: https://omicto...`sortmerna` appears to be a relatively slow program for the metatranscriptomic files we are typically processing.
An acceleration of this step would be nice!
There do not seem to be too many alternatives out there though: https://omictools.com/rrna-filtering-category
Some of them are web-based, hence do not really represent an alternative.
Maybe `rrnafilter` would be one, but from superficially scanning over it, it seems to be based on k-mer abundances.
While this enables the identification of candidate rRNA reads without the need for reference sequences, it could also lead to false positives, i.e., non-rRNA gene-derived sequences of high (or divergent) abundance) that are wrongly classified as rRNA gene-derived.
Not sure how the tool was tested in the original publication.
A dedicated test might be required, though.https://git-r3lab.uni.lu/IMP/IMP/-/issues/135Read preprocessing using fastp2018-10-08T16:32:54+02:00Cedric LacznyRead preprocessing using fastpMaybe [fastp](https://github.com/OpenGene/fastp) could be integrated as a preprocessing step.
According to its documentation is is (supposedly)
>like FASTQC but faster and more informative
and it integrates trimming etc. while automati...Maybe [fastp](https://github.com/OpenGene/fastp) could be integrated as a preprocessing step.
According to its documentation is is (supposedly)
>like FASTQC but faster and more informative
and it integrates trimming etc. while automatically detecting the adapters as well as
>trim polyG in 3' ends, which is commonly seen in NovaSeq/NextSeq datahttps://git-r3lab.uni.lu/IMP/IMP/-/issues/27Deprecate DATADIR parameter2018-02-09T09:10:34+01:00Shaman NarayanasamyDeprecate DATADIR parameterI think this simply adds complexity to the running of the pipeline. The entire paths can be specified in the MG and MT parameters.I think this simply adds complexity to the running of the pipeline. The entire paths can be specified in the MG and MT parameters.Yohan Jaroszyohan.jarosz@uni.luYohan Jaroszyohan.jarosz@uni.luhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/29tmp directory location2018-02-09T09:10:34+01:00Shaman Narayanasamytmp directory locationThe tmp directory is being created in the `build/tmp` of the IMP folder. This is causing the "real" analysis to crash due to lack of space.The tmp directory is being created in the `build/tmp` of the IMP folder. This is causing the "real" analysis to crash due to lack of space.Yohan Jaroszyohan.jarosz@uni.luYohan Jaroszyohan.jarosz@uni.luhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/49R migrate functions2018-02-09T09:10:34+01:00Shaman NarayanasamyR migrate functionsClean up R functions from the plot script by moving to a function script.
Make sure handling of NA's are optimal!!Clean up R functions from the plot script by moving to a function script.
Make sure handling of NA's are optimal!!Shaman NarayanasamyShaman Narayanasamyhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/6MG and MT only assembly2018-02-09T09:10:34+01:00Shaman NarayanasamyMG and MT only assemblyCreate `snakemake` rules for MG and MT only stepwise assemblies.Create `snakemake` rules for MG and MT only stepwise assemblies.Shaman NarayanasamyShaman Narayanasamyhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/7MG and MT only analysis2018-02-09T09:10:34+01:00Shaman NarayanasamyMG and MT only analysisRelated to issue #6.
Once the individual omic (MG and MT) assemblies are working, prepare workflow for MT and MG only analysis.Related to issue #6.
Once the individual omic (MG and MT) assemblies are working, prepare workflow for MT and MG only analysis.Shaman NarayanasamyShaman Narayanasamyhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/8Fast MGMT assembly option using MEGAHIT2018-02-09T09:10:34+01:00Shaman NarayanasamyFast MGMT assembly option using MEGAHITSince, the refactoring of the pipeline in it's entirity into `Snakemake`, it is rather easy to add modules. Therefore I added some rules that will enable a fast option: 14b56d1b
Note that while individual steps already work, it has n...Since, the refactoring of the pipeline in it's entirity into `Snakemake`, it is rather easy to add modules. Therefore I added some rules that will enable a fast option: 14b56d1b
Note that while individual steps already work, it has not been incorporated into the core workflow as this will be an option. This is not a priority, and can be incorporated towards the end.https://git-r3lab.uni.lu/IMP/IMP/-/issues/10Module for Trinity2018-02-09T09:10:34+01:00Shaman NarayanasamyModule for TrinityBecause we can! :)Because we can! :)Shaman NarayanasamyShaman Narayanasamyhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/36IMP main script python2 portability2018-02-09T09:10:34+01:00Shaman NarayanasamyIMP main script python2 portabilityAdd header in IMP main script for python2 portability. Not a big issue, but just in case.Add header in IMP main script for python2 portability. Not a big issue, but just in case.Shaman NarayanasamyShaman Narayanasamyhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/38Too many containers.2018-02-09T09:10:34+01:00Yohan Jaroszyohan.jarosz@uni.luToo many containers.After using IMP for a while, you have too many containers on your system. By default, we may rm the container after use. (docker run .... --rm)
Can disable with a `--norm`` option on the wrapper.
@shaman.narayanasamy What do you think?...After using IMP for a while, you have too many containers on your system. By default, we may rm the container after use. (docker run .... --rm)
Can disable with a `--norm`` option on the wrapper.
@shaman.narayanasamy What do you think?
related doc: https://docs.docker.com/reference/run/#clean-up-rmYohan Jaroszyohan.jarosz@uni.luYohan Jaroszyohan.jarosz@uni.luhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/55IMP with reference genome/metagenome2018-02-09T09:10:34+01:00Shaman NarayanasamyIMP with reference genome/metagenomeMake IMP run such that one provides a genome or metagenome of interest.
Input:
1. Reference genome/metagenome (fasta)
2. Reads MG and/or MT
3. Annotation for genome/metagenome (gff) - optional
Steps will include:
1. *Preprocess...Make IMP run such that one provides a genome or metagenome of interest.
Input:
1. Reference genome/metagenome (fasta)
2. Reads MG and/or MT
3. Annotation for genome/metagenome (gff) - optional
Steps will include:
1. *Preprocess reads
2. *Mapping reads to reference
3. Annotate reference genome/metagenome (if gff file not provided)
4. *Depth calling for contigs
6. *Depth calling for genes
7. *Variant calling for contigs
8. *Variant calling for genes
Will add more later on...
NOTE: * For MG and/or MT data sets
Shaman NarayanasamyShaman Narayanasamyhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/45Changing ownership of files within docker2018-02-09T09:10:34+01:00Shaman NarayanasamyChanging ownership of files within dockerIs it possible to change ownership of files within docker? For instance, create a rule that converts the directory and file ownership to the user that launched the job.
I realized that we are having issues moving the files because we ...Is it possible to change ownership of files within docker? For instance, create a rule that converts the directory and file ownership to the user that launched the job.
I realized that we are having issues moving the files because we do not own them. This is rather inconvenient as the user needs to be root in order to change the file. It is not a problem at present for bigbug users as there are only two people actively running IMP, but will be in the future...
Thanks @anne.kaysen for bringing this to light.
Update: Some googling showed me that this is quite a troublesome and complicated task. Maybe this might work:
http://stackoverflow.com/questions/26500270/understanding-user-file-ownership-in-docker-how-to-avoid-changing-permissions-oYohan Jaroszyohan.jarosz@uni.luYohan Jaroszyohan.jarosz@uni.luhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/46Compress (zip) report2018-02-09T09:10:34+01:00Shaman NarayanasamyCompress (zip) reportcreate a rule to compress the report and the necessary files such that users can easily download it and open it on their local browser.create a rule to compress the report and the necessary files such that users can easily download it and open it on their local browser.Yohan Jaroszyohan.jarosz@uni.luYohan Jaroszyohan.jarosz@uni.luhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/58Work on snakemake step names2018-02-09T09:10:34+01:00Shaman NarayanasamyWork on snakemake step namesChange names to be more meaningful and understandable.
NOTE: Remember to also change log file names
Change names to be more meaningful and understandable.
NOTE: Remember to also change log file names
Shaman NarayanasamyShaman Narayanasamyhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/47Add new rule for assembly validation2018-02-09T09:10:34+01:00Shaman NarayanasamyAdd new rule for assembly validationShaman NarayanasamyShaman Narayanasamyhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/59Add condition for MEGAHIT k-mer values2018-02-09T09:10:34+01:00Shaman NarayanasamyAdd condition for MEGAHIT k-mer valuesMEGAHIT requires the min and max k-mers to be odd values and k-step value to be even.
Should add condition to the MEGAHIT rules to deal with this issue.
Add also a notification such that the user is aware of what kmers are being us...MEGAHIT requires the min and max k-mers to be odd values and k-step value to be even.
Should add condition to the MEGAHIT rules to deal with this issue.
Add also a notification such that the user is aware of what kmers are being used.
Shaman NarayanasamyShaman Narayanasamy