IMP issueshttps://git-r3lab.uni.lu/IMP/IMP/-/issues2017-05-31T09:51:20+02:00https://git-r3lab.uni.lu/IMP/IMP/-/issues/82Make tests for the init script2017-05-31T09:51:20+02:00Yohan Jaroszyohan.jarosz@uni.luMake tests for the init scriptCurrently there are no tests to check if the CL works properly. We need to test for #81 .Currently there are no tests to check if the CL works properly. We need to test for #81 .Yohan Jaroszyohan.jarosz@uni.luYohan Jaroszyohan.jarosz@uni.luhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/70IMP db default path2017-05-31T09:51:21+02:00Shaman NarayanasamyIMP db default pathThe path to database is defaulted where IMP is launched.
For instance, when launching in my work directory, the database path is set to :`/mnt/internal/work/users/snarayanasamy/Work/db`.
Isn't it better for the default setting to ...The path to database is defaulted where IMP is launched.
For instance, when launching in my work directory, the database path is set to :`/mnt/internal/work/users/snarayanasamy/Work/db`.
Isn't it better for the default setting to be in the IMP home directory? Below is the example.
E.g.:
```
$ pwd
/home/snarayanasamy/Work
$ python3.4 ~/Work/tools/IMP_new/IMP/IMP -m /mnt/md1200/snarayanasamy/IMP_data/TEST_DATA/MG.R1.5_percent.fq -m /mnt/md1200/snarayanasamy/IMP_data/TEST_DATA/MG.R1.5_percent.fq -t /mnt/md1200/snarayanasamy/IMP_data/TEST_DATA/MT.R1.5_percent.fq -t /mnt/md1200/snarayanasamy/IMP_data/TEST_DATA/MT.R2.5_percent.fq -o /home/snarayanasamy/Work/IMP_MGMT_TEST -c /home/snarayanasamy/Work/IMP_conf_files/userconfig.imp.json -a megahit --current
Executing " docker run -v /mnt/md1200/snarayanasamy/IMP_data/TEST_DATA:/home/imp/data -v /home/snarayanasamy/Work/IMP_MGMT_TEST:/home/imp/output -v /mnt/internal/work/users/snarayanasamy/Work/db:/home/imp/databases -v /home/snarayanasamy/Work/IMP_conf_files:/home/imp/conf -v /home/snarayanasamy/Work/tools/IMP_new/IMP:/home/imp/code -e MG="/home/imp/data/MG.R1.5_percent.fq /home/imp/data/MG.R1.5_percent.fq" -e MT="/home/imp/data/MT.R1.5_percent.fq /home/imp/data/MT.R2.5_percent.fq" -e CONFIGFILE=/home/imp/conf/userconfig.imp.json -e IMP_ASSEMBLER=megahit --rm -e "LOCAL_USER_ID=`id -u $USER`" -e "LOCAL_GROUP_ID=`id -g $USER`" --name home_snarayanasamy_Work_IMP_MGMT_TEST docker-r3lab.uni.lu/imp/imp:1.3 snakemake ALL -s /home/imp/code/Snakefile "
MissingInputException in line 1 of /home/imp/code/rules/Preprocessing/trimming.rules:
Missing input files for rule trimming:
/home/imp/databases/adapters/adapters.done
```
Yohan Jaroszyohan.jarosz@uni.luYohan Jaroszyohan.jarosz@uni.luhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/92IncompleteFilesException error2019-08-27T08:50:26+02:00Federico BaldiniIncompleteFilesException errorFollowing error:
```bash
$ impy init --index ~/hg38.fa
[x] Found IMP docker-r3lab.uni.lu/imp/imp 1.4.1
IMP image found
Executing IMP command:
docker run --rm --name home_federico_impyinstall_imp_db -v /home/federico/impyinstall/imp-...Following error:
```bash
$ impy init --index ~/hg38.fa
[x] Found IMP docker-r3lab.uni.lu/imp/imp 1.4.1
IMP image found
Executing IMP command:
docker run --rm --name home_federico_impyinstall_imp_db -v /home/federico/impyinstall/imp-db:/home/imp/databases -e "LOCAL_USER_ID=`id -u $USER`" -e "LOCAL_GROUP_ID=`id -g $USER`" -e FILTER="hg38" docker-r3lab.uni.lu/imp/imp:1.4.1 snakemake -s /home/imp/code/rules/ini/init
IncompleteFilesException:
The files below seem to be incomplete. If you are sure that certain files are not incomplete, mark them as complete with
snakemake --cleanup-metadata <filenames>
To re-generate the files rerun your command with the --rerun-incomplete flag.
Incomplete files:
filtering/hg38.fa
```
I downloaded the file `hg38.fa` using:
```bash
$ wget http://hgdownload.cse.ucsc.edu/goldenPath/hg38/bigZips/hg38.fa.gz
```
and then unzipped it using:
```bash
$ gunzip hg38.fa.gz
```
Running the following command:
```bash
$ snakemake --cleanup-metadata ./imp-db/filtering/hg38.fa
```
yields:
```bash
Error: Snakefile "Snakefile" not present.
```
Can you help me out please? Thanks.Yohan Jaroszyohan.jarosz@uni.luYohan Jaroszyohan.jarosz@uni.luhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/88Migrate extract-unmapped rule to common rules2019-08-27T08:50:20+02:00Shaman NarayanasamyMigrate extract-unmapped rule to common rulesAt present this rule is duplicated to hybrid and single-omic. This should be preferably be made common at some point.At present this rule is duplicated to hybrid and single-omic. This should be preferably be made common at some point.Yohan Jaroszyohan.jarosz@uni.luYohan Jaroszyohan.jarosz@uni.luhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/39Security of docker container2019-08-27T08:50:12+02:00Shaman NarayanasamySecurity of docker containerLooking at the fact that docker requires super-user rights. It would be nice if the IMP script is able to throw error if users try to mount important OS directories (such a the root dir: / ).Looking at the fact that docker requires super-user rights. It would be nice if the IMP script is able to throw error if users try to mount important OS directories (such a the root dir: / ).Yohan Jaroszyohan.jarosz@uni.luYohan Jaroszyohan.jarosz@uni.luhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/44Add "list" option to IMP python script2017-12-19T11:29:22+01:00Shaman NarayanasamyAdd "list" option to IMP python script@yjarosz, could you please add an option to the IMP python script such that it would list all/remaining steps?
something like `IMP -l` or `IMP -list` for all steps within the pipeline such that users will be able to invoke them from o...@yjarosz, could you please add an option to the IMP python script such that it would list all/remaining steps?
something like `IMP -l` or `IMP -list` for all steps within the pipeline such that users will be able to invoke them from outside the container.
Or if we had an analysis that failed/stopped halfway, we can do something like `IMP -m MG.R1 -m MG.R2 -t MT.R1 -t MT.R2 -o output_dir --list-incomplete`, to list the remaining steps.
Not sure if the above is the best way to do it, but it would be cool and quite convenient instead of entering the container each time. Tell me what you think. This is not urgent, please leave it for last :)Yohan Jaroszyohan.jarosz@uni.luYohan Jaroszyohan.jarosz@uni.luhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/11Modularize assembly strategies2017-12-19T11:29:22+01:00Shaman NarayanasamyModularize assembly strategiesSince there are so many different types/preferred assembly strategies, then we should make modules for diffierent assembly protocols.
This is not urgent.Since there are so many different types/preferred assembly strategies, then we should make modules for diffierent assembly protocols.
This is not urgent.https://git-r3lab.uni.lu/IMP/IMP/-/issues/26htstat migration2018-02-09T09:10:34+01:00Shaman Narayanasamyhtstat migrationMigrate ht_stat steps into analysis instead of preprocessing. Let's keep preprocessing clean such that it only focuses on essential output.
I made a new branch and removed the ht_stat steps to the Analysis rules:
a71db616
Still ...Migrate ht_stat steps into analysis instead of preprocessing. Let's keep preprocessing clean such that it only focuses on essential output.
I made a new branch and removed the ht_stat steps to the Analysis rules:
a71db616
Still need to get it to work in the Analysis rules...Shaman NarayanasamyShaman Narayanasamyhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/3Merge ANALYSIS_MG_GENE_COVERAGE and ANALYSIS_MT_GENE_COVERAGE:2018-02-09T09:10:34+01:00Shaman NarayanasamyMerge ANALYSIS_MG_GENE_COVERAGE and ANALYSIS_MT_GENE_COVERAGE:Produces redundant files. Or maybe edit the output to reduce the number of redundant files to keep the modules compact.Produces redundant files. Or maybe edit the output to reduce the number of redundant files to keep the modules compact.Shaman NarayanasamyShaman Narayanasamyhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/60IMP execution outside of code directory2018-02-09T09:10:34+01:00Shaman NarayanasamyIMP execution outside of code directoryIMP is currently not executable outside of the directory containing the code. Or I am doing it the wrong way. This is an issue because it is not efficient. At present, everyone running IMP has their own copy or the repository, the databa...IMP is currently not executable outside of the directory containing the code. Or I am doing it the wrong way. This is an issue because it is not efficient. At present, everyone running IMP has their own copy or the repository, the databases etc. Is it possible to have a centralized repository which everyone can run from?
Below is the error I get when trying to execute outide the IMP directory.
```
fatal: Not a git repository (or any parent up to mount point /mnt/md1200)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Traceback (most recent call last):
File "/home/snarayanasamy/Work/tools/IMP/IMP", line 252, in <module>
args = docopt(__doc__, version=get_git_version(), options_first=True)
File "/home/snarayanasamy/Work/tools/IMP/IMP", line 122, in get_git_version
['git', '--no-pager', 'log', '-n', '1', '--pretty=format:%H']
File "/usr/local/lib/python3.4/subprocess.py", line 620, in check_output
raise CalledProcessError(retcode, process.args, output=output)
subprocess.CalledProcessError: Command '['git', '--no-pager', 'log', '-n', '1', '--pretty=format:%H']' returned non-zero exit status 128
```
Is there any way we can fix this? My IMP code directory is getting so messy with all the log files etc :six:
Yohan Jaroszyohan.jarosz@uni.luYohan Jaroszyohan.jarosz@uni.luhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/23Incorporate VizBin Java implementation2018-02-09T09:10:34+01:00Shaman NarayanasamyIncorporate VizBin Java implementationFor consistency reasons.For consistency reasons.Yohan Jaroszyohan.jarosz@uni.luYohan Jaroszyohan.jarosz@uni.luhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/59Add condition for MEGAHIT k-mer values2018-02-09T09:10:34+01:00Shaman NarayanasamyAdd condition for MEGAHIT k-mer valuesMEGAHIT requires the min and max k-mers to be odd values and k-step value to be even.
Should add condition to the MEGAHIT rules to deal with this issue.
Add also a notification such that the user is aware of what kmers are being us...MEGAHIT requires the min and max k-mers to be odd values and k-step value to be even.
Should add condition to the MEGAHIT rules to deal with this issue.
Add also a notification such that the user is aware of what kmers are being used.
Shaman NarayanasamyShaman Narayanasamyhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/47Add new rule for assembly validation2018-02-09T09:10:34+01:00Shaman NarayanasamyAdd new rule for assembly validationShaman NarayanasamyShaman Narayanasamyhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/46Compress (zip) report2018-02-09T09:10:34+01:00Shaman NarayanasamyCompress (zip) reportcreate a rule to compress the report and the necessary files such that users can easily download it and open it on their local browser.create a rule to compress the report and the necessary files such that users can easily download it and open it on their local browser.Yohan Jaroszyohan.jarosz@uni.luYohan Jaroszyohan.jarosz@uni.luhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/58Work on snakemake step names2018-02-09T09:10:34+01:00Shaman NarayanasamyWork on snakemake step namesChange names to be more meaningful and understandable.
NOTE: Remember to also change log file names
Change names to be more meaningful and understandable.
NOTE: Remember to also change log file names
Shaman NarayanasamyShaman Narayanasamyhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/45Changing ownership of files within docker2018-02-09T09:10:34+01:00Shaman NarayanasamyChanging ownership of files within dockerIs it possible to change ownership of files within docker? For instance, create a rule that converts the directory and file ownership to the user that launched the job.
I realized that we are having issues moving the files because we ...Is it possible to change ownership of files within docker? For instance, create a rule that converts the directory and file ownership to the user that launched the job.
I realized that we are having issues moving the files because we do not own them. This is rather inconvenient as the user needs to be root in order to change the file. It is not a problem at present for bigbug users as there are only two people actively running IMP, but will be in the future...
Thanks @anne.kaysen for bringing this to light.
Update: Some googling showed me that this is quite a troublesome and complicated task. Maybe this might work:
http://stackoverflow.com/questions/26500270/understanding-user-file-ownership-in-docker-how-to-avoid-changing-permissions-oYohan Jaroszyohan.jarosz@uni.luYohan Jaroszyohan.jarosz@uni.luhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/55IMP with reference genome/metagenome2018-02-09T09:10:34+01:00Shaman NarayanasamyIMP with reference genome/metagenomeMake IMP run such that one provides a genome or metagenome of interest.
Input:
1. Reference genome/metagenome (fasta)
2. Reads MG and/or MT
3. Annotation for genome/metagenome (gff) - optional
Steps will include:
1. *Preprocess...Make IMP run such that one provides a genome or metagenome of interest.
Input:
1. Reference genome/metagenome (fasta)
2. Reads MG and/or MT
3. Annotation for genome/metagenome (gff) - optional
Steps will include:
1. *Preprocess reads
2. *Mapping reads to reference
3. Annotate reference genome/metagenome (if gff file not provided)
4. *Depth calling for contigs
6. *Depth calling for genes
7. *Variant calling for contigs
8. *Variant calling for genes
Will add more later on...
NOTE: * For MG and/or MT data sets
Shaman NarayanasamyShaman Narayanasamyhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/38Too many containers.2018-02-09T09:10:34+01:00Yohan Jaroszyohan.jarosz@uni.luToo many containers.After using IMP for a while, you have too many containers on your system. By default, we may rm the container after use. (docker run .... --rm)
Can disable with a `--norm`` option on the wrapper.
@shaman.narayanasamy What do you think?...After using IMP for a while, you have too many containers on your system. By default, we may rm the container after use. (docker run .... --rm)
Can disable with a `--norm`` option on the wrapper.
@shaman.narayanasamy What do you think?
related doc: https://docs.docker.com/reference/run/#clean-up-rmYohan Jaroszyohan.jarosz@uni.luYohan Jaroszyohan.jarosz@uni.luhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/10Module for Trinity2018-02-09T09:10:34+01:00Shaman NarayanasamyModule for TrinityBecause we can! :)Because we can! :)Shaman NarayanasamyShaman Narayanasamyhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/36IMP main script python2 portability2018-02-09T09:10:34+01:00Shaman NarayanasamyIMP main script python2 portabilityAdd header in IMP main script for python2 portability. Not a big issue, but just in case.Add header in IMP main script for python2 portability. Not a big issue, but just in case.Shaman NarayanasamyShaman Narayanasamyhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/8Fast MGMT assembly option using MEGAHIT2018-02-09T09:10:34+01:00Shaman NarayanasamyFast MGMT assembly option using MEGAHITSince, the refactoring of the pipeline in it's entirity into `Snakemake`, it is rather easy to add modules. Therefore I added some rules that will enable a fast option: 14b56d1b
Note that while individual steps already work, it has n...Since, the refactoring of the pipeline in it's entirity into `Snakemake`, it is rather easy to add modules. Therefore I added some rules that will enable a fast option: 14b56d1b
Note that while individual steps already work, it has not been incorporated into the core workflow as this will be an option. This is not a priority, and can be incorporated towards the end.https://git-r3lab.uni.lu/IMP/IMP/-/issues/7MG and MT only analysis2018-02-09T09:10:34+01:00Shaman NarayanasamyMG and MT only analysisRelated to issue #6.
Once the individual omic (MG and MT) assemblies are working, prepare workflow for MT and MG only analysis.Related to issue #6.
Once the individual omic (MG and MT) assemblies are working, prepare workflow for MT and MG only analysis.Shaman NarayanasamyShaman Narayanasamyhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/6MG and MT only assembly2018-02-09T09:10:34+01:00Shaman NarayanasamyMG and MT only assemblyCreate `snakemake` rules for MG and MT only stepwise assemblies.Create `snakemake` rules for MG and MT only stepwise assemblies.Shaman NarayanasamyShaman Narayanasamyhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/29tmp directory location2018-02-09T09:10:34+01:00Shaman Narayanasamytmp directory locationThe tmp directory is being created in the `build/tmp` of the IMP folder. This is causing the "real" analysis to crash due to lack of space.The tmp directory is being created in the `build/tmp` of the IMP folder. This is causing the "real" analysis to crash due to lack of space.Yohan Jaroszyohan.jarosz@uni.luYohan Jaroszyohan.jarosz@uni.luhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/49R migrate functions2018-02-09T09:10:34+01:00Shaman NarayanasamyR migrate functionsClean up R functions from the plot script by moving to a function script.
Make sure handling of NA's are optimal!!Clean up R functions from the plot script by moving to a function script.
Make sure handling of NA's are optimal!!Shaman NarayanasamyShaman Narayanasamyhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/27Deprecate DATADIR parameter2018-02-09T09:10:34+01:00Shaman NarayanasamyDeprecate DATADIR parameterI think this simply adds complexity to the running of the pipeline. The entire paths can be specified in the MG and MT parameters.I think this simply adds complexity to the running of the pipeline. The entire paths can be specified in the MG and MT parameters.Yohan Jaroszyohan.jarosz@uni.luYohan Jaroszyohan.jarosz@uni.lu