IMP issueshttps://git-r3lab.uni.lu/IMP/IMP/-/issues2019-08-27T08:50:26+02:00https://git-r3lab.uni.lu/IMP/IMP/-/issues/92IncompleteFilesException error2019-08-27T08:50:26+02:00Federico BaldiniIncompleteFilesException errorFollowing error:
```bash
$ impy init --index ~/hg38.fa
[x] Found IMP docker-r3lab.uni.lu/imp/imp 1.4.1
IMP image found
Executing IMP command:
docker run --rm --name home_federico_impyinstall_imp_db -v /home/federico/impyinstall/imp-...Following error:
```bash
$ impy init --index ~/hg38.fa
[x] Found IMP docker-r3lab.uni.lu/imp/imp 1.4.1
IMP image found
Executing IMP command:
docker run --rm --name home_federico_impyinstall_imp_db -v /home/federico/impyinstall/imp-db:/home/imp/databases -e "LOCAL_USER_ID=`id -u $USER`" -e "LOCAL_GROUP_ID=`id -g $USER`" -e FILTER="hg38" docker-r3lab.uni.lu/imp/imp:1.4.1 snakemake -s /home/imp/code/rules/ini/init
IncompleteFilesException:
The files below seem to be incomplete. If you are sure that certain files are not incomplete, mark them as complete with
snakemake --cleanup-metadata <filenames>
To re-generate the files rerun your command with the --rerun-incomplete flag.
Incomplete files:
filtering/hg38.fa
```
I downloaded the file `hg38.fa` using:
```bash
$ wget http://hgdownload.cse.ucsc.edu/goldenPath/hg38/bigZips/hg38.fa.gz
```
and then unzipped it using:
```bash
$ gunzip hg38.fa.gz
```
Running the following command:
```bash
$ snakemake --cleanup-metadata ./imp-db/filtering/hg38.fa
```
yields:
```bash
Error: Snakefile "Snakefile" not present.
```
Can you help me out please? Thanks.Yohan Jaroszyohan.jarosz@uni.luYohan Jaroszyohan.jarosz@uni.luhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/88Migrate extract-unmapped rule to common rules2019-08-27T08:50:20+02:00Shaman NarayanasamyMigrate extract-unmapped rule to common rulesAt present this rule is duplicated to hybrid and single-omic. This should be preferably be made common at some point.At present this rule is duplicated to hybrid and single-omic. This should be preferably be made common at some point.Yohan Jaroszyohan.jarosz@uni.luYohan Jaroszyohan.jarosz@uni.luhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/39Security of docker container2019-08-27T08:50:12+02:00Shaman NarayanasamySecurity of docker containerLooking at the fact that docker requires super-user rights. It would be nice if the IMP script is able to throw error if users try to mount important OS directories (such a the root dir: / ).Looking at the fact that docker requires super-user rights. It would be nice if the IMP script is able to throw error if users try to mount important OS directories (such a the root dir: / ).Yohan Jaroszyohan.jarosz@uni.luYohan Jaroszyohan.jarosz@uni.luhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/27Deprecate DATADIR parameter2018-02-09T09:10:34+01:00Shaman NarayanasamyDeprecate DATADIR parameterI think this simply adds complexity to the running of the pipeline. The entire paths can be specified in the MG and MT parameters.I think this simply adds complexity to the running of the pipeline. The entire paths can be specified in the MG and MT parameters.Yohan Jaroszyohan.jarosz@uni.luYohan Jaroszyohan.jarosz@uni.luhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/29tmp directory location2018-02-09T09:10:34+01:00Shaman Narayanasamytmp directory locationThe tmp directory is being created in the `build/tmp` of the IMP folder. This is causing the "real" analysis to crash due to lack of space.The tmp directory is being created in the `build/tmp` of the IMP folder. This is causing the "real" analysis to crash due to lack of space.Yohan Jaroszyohan.jarosz@uni.luYohan Jaroszyohan.jarosz@uni.luhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/49R migrate functions2018-02-09T09:10:34+01:00Shaman NarayanasamyR migrate functionsClean up R functions from the plot script by moving to a function script.
Make sure handling of NA's are optimal!!Clean up R functions from the plot script by moving to a function script.
Make sure handling of NA's are optimal!!Shaman NarayanasamyShaman Narayanasamyhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/6MG and MT only assembly2018-02-09T09:10:34+01:00Shaman NarayanasamyMG and MT only assemblyCreate `snakemake` rules for MG and MT only stepwise assemblies.Create `snakemake` rules for MG and MT only stepwise assemblies.Shaman NarayanasamyShaman Narayanasamyhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/7MG and MT only analysis2018-02-09T09:10:34+01:00Shaman NarayanasamyMG and MT only analysisRelated to issue #6.
Once the individual omic (MG and MT) assemblies are working, prepare workflow for MT and MG only analysis.Related to issue #6.
Once the individual omic (MG and MT) assemblies are working, prepare workflow for MT and MG only analysis.Shaman NarayanasamyShaman Narayanasamyhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/8Fast MGMT assembly option using MEGAHIT2018-02-09T09:10:34+01:00Shaman NarayanasamyFast MGMT assembly option using MEGAHITSince, the refactoring of the pipeline in it's entirity into `Snakemake`, it is rather easy to add modules. Therefore I added some rules that will enable a fast option: 14b56d1b
Note that while individual steps already work, it has n...Since, the refactoring of the pipeline in it's entirity into `Snakemake`, it is rather easy to add modules. Therefore I added some rules that will enable a fast option: 14b56d1b
Note that while individual steps already work, it has not been incorporated into the core workflow as this will be an option. This is not a priority, and can be incorporated towards the end.https://git-r3lab.uni.lu/IMP/IMP/-/issues/10Module for Trinity2018-02-09T09:10:34+01:00Shaman NarayanasamyModule for TrinityBecause we can! :)Because we can! :)Shaman NarayanasamyShaman Narayanasamyhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/36IMP main script python2 portability2018-02-09T09:10:34+01:00Shaman NarayanasamyIMP main script python2 portabilityAdd header in IMP main script for python2 portability. Not a big issue, but just in case.Add header in IMP main script for python2 portability. Not a big issue, but just in case.Shaman NarayanasamyShaman Narayanasamyhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/38Too many containers.2018-02-09T09:10:34+01:00Yohan Jaroszyohan.jarosz@uni.luToo many containers.After using IMP for a while, you have too many containers on your system. By default, we may rm the container after use. (docker run .... --rm)
Can disable with a `--norm`` option on the wrapper.
@shaman.narayanasamy What do you think?...After using IMP for a while, you have too many containers on your system. By default, we may rm the container after use. (docker run .... --rm)
Can disable with a `--norm`` option on the wrapper.
@shaman.narayanasamy What do you think?
related doc: https://docs.docker.com/reference/run/#clean-up-rmYohan Jaroszyohan.jarosz@uni.luYohan Jaroszyohan.jarosz@uni.luhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/55IMP with reference genome/metagenome2018-02-09T09:10:34+01:00Shaman NarayanasamyIMP with reference genome/metagenomeMake IMP run such that one provides a genome or metagenome of interest.
Input:
1. Reference genome/metagenome (fasta)
2. Reads MG and/or MT
3. Annotation for genome/metagenome (gff) - optional
Steps will include:
1. *Preprocess...Make IMP run such that one provides a genome or metagenome of interest.
Input:
1. Reference genome/metagenome (fasta)
2. Reads MG and/or MT
3. Annotation for genome/metagenome (gff) - optional
Steps will include:
1. *Preprocess reads
2. *Mapping reads to reference
3. Annotate reference genome/metagenome (if gff file not provided)
4. *Depth calling for contigs
6. *Depth calling for genes
7. *Variant calling for contigs
8. *Variant calling for genes
Will add more later on...
NOTE: * For MG and/or MT data sets
Shaman NarayanasamyShaman Narayanasamyhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/45Changing ownership of files within docker2018-02-09T09:10:34+01:00Shaman NarayanasamyChanging ownership of files within dockerIs it possible to change ownership of files within docker? For instance, create a rule that converts the directory and file ownership to the user that launched the job.
I realized that we are having issues moving the files because we ...Is it possible to change ownership of files within docker? For instance, create a rule that converts the directory and file ownership to the user that launched the job.
I realized that we are having issues moving the files because we do not own them. This is rather inconvenient as the user needs to be root in order to change the file. It is not a problem at present for bigbug users as there are only two people actively running IMP, but will be in the future...
Thanks @anne.kaysen for bringing this to light.
Update: Some googling showed me that this is quite a troublesome and complicated task. Maybe this might work:
http://stackoverflow.com/questions/26500270/understanding-user-file-ownership-in-docker-how-to-avoid-changing-permissions-oYohan Jaroszyohan.jarosz@uni.luYohan Jaroszyohan.jarosz@uni.luhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/46Compress (zip) report2018-02-09T09:10:34+01:00Shaman NarayanasamyCompress (zip) reportcreate a rule to compress the report and the necessary files such that users can easily download it and open it on their local browser.create a rule to compress the report and the necessary files such that users can easily download it and open it on their local browser.Yohan Jaroszyohan.jarosz@uni.luYohan Jaroszyohan.jarosz@uni.luhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/58Work on snakemake step names2018-02-09T09:10:34+01:00Shaman NarayanasamyWork on snakemake step namesChange names to be more meaningful and understandable.
NOTE: Remember to also change log file names
Change names to be more meaningful and understandable.
NOTE: Remember to also change log file names
Shaman NarayanasamyShaman Narayanasamyhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/47Add new rule for assembly validation2018-02-09T09:10:34+01:00Shaman NarayanasamyAdd new rule for assembly validationShaman NarayanasamyShaman Narayanasamyhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/59Add condition for MEGAHIT k-mer values2018-02-09T09:10:34+01:00Shaman NarayanasamyAdd condition for MEGAHIT k-mer valuesMEGAHIT requires the min and max k-mers to be odd values and k-step value to be even.
Should add condition to the MEGAHIT rules to deal with this issue.
Add also a notification such that the user is aware of what kmers are being us...MEGAHIT requires the min and max k-mers to be odd values and k-step value to be even.
Should add condition to the MEGAHIT rules to deal with this issue.
Add also a notification such that the user is aware of what kmers are being used.
Shaman NarayanasamyShaman Narayanasamyhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/23Incorporate VizBin Java implementation2018-02-09T09:10:34+01:00Shaman NarayanasamyIncorporate VizBin Java implementationFor consistency reasons.For consistency reasons.Yohan Jaroszyohan.jarosz@uni.luYohan Jaroszyohan.jarosz@uni.luhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/60IMP execution outside of code directory2018-02-09T09:10:34+01:00Shaman NarayanasamyIMP execution outside of code directoryIMP is currently not executable outside of the directory containing the code. Or I am doing it the wrong way. This is an issue because it is not efficient. At present, everyone running IMP has their own copy or the repository, the databa...IMP is currently not executable outside of the directory containing the code. Or I am doing it the wrong way. This is an issue because it is not efficient. At present, everyone running IMP has their own copy or the repository, the databases etc. Is it possible to have a centralized repository which everyone can run from?
Below is the error I get when trying to execute outide the IMP directory.
```
fatal: Not a git repository (or any parent up to mount point /mnt/md1200)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Traceback (most recent call last):
File "/home/snarayanasamy/Work/tools/IMP/IMP", line 252, in <module>
args = docopt(__doc__, version=get_git_version(), options_first=True)
File "/home/snarayanasamy/Work/tools/IMP/IMP", line 122, in get_git_version
['git', '--no-pager', 'log', '-n', '1', '--pretty=format:%H']
File "/usr/local/lib/python3.4/subprocess.py", line 620, in check_output
raise CalledProcessError(retcode, process.args, output=output)
subprocess.CalledProcessError: Command '['git', '--no-pager', 'log', '-n', '1', '--pretty=format:%H']' returned non-zero exit status 128
```
Is there any way we can fix this? My IMP code directory is getting so messy with all the log files etc :six:
Yohan Jaroszyohan.jarosz@uni.luYohan Jaroszyohan.jarosz@uni.lu