IMP issueshttps://git-r3lab.uni.lu/IMP/IMP/-/issues2022-04-30T04:31:41+02:00https://git-r3lab.uni.lu/IMP/IMP/-/issues/166Error in hybrid assembly occurred2022-04-30T04:31:41+02:00MikaError in hybrid assembly occurred
Hi, When I run the assembly step, the following error in hybrid assembly occurred.
How can I fix this? Thank you.
```
$impy assembly -m ${MG1} -m ${MG2} -m ${MG_single} -t ${MT1} -t ${MT2} -t ${MT_single} -o ${Dir_out}
.
.
.
(omitted...
Hi, When I run the assembly step, the following error in hybrid assembly occurred.
How can I fix this? Thank you.
```
$impy assembly -m ${MG1} -m ${MG2} -m ${MG_single} -t ${MT1} -t ${MT2} -t ${MT_single} -o ${Dir_out}
.
.
.
(omitted)
.
.
.
rule megahit_assembly_from_unmapped:
input: Assembly/mt.r1.unmapped.fq, Assembly/mt.r2.unmapped.fq, Assembly/mt.se.unmapped.fq
output: Assembly/mt.megahit_unmapped.2/final.contigs.fa, Assembly/mt.megahit_unmapped.2.fa
wildcards: loop=2, type=mt
[x] Performing mt assembly step '2' using MEGAHIT
MEGAHIT v1.0.6
--- [Tue Apr 19 16:27:43 2022] Start assembly. Number of CPU threads 4 ---
--- [Tue Apr 19 16:27:43 2022] Available memory: 33675894784, used: 24000000000
--- [Tue Apr 19 16:27:43 2022] k list: 25,29,33,37,41,45,49,53,57,61,65,69,73,77,81,85,89,93,97,99 ---
--- [Tue Apr 19 16:27:43 2022] Converting reads to binaries ---
[read_lib_functions-inl.h : 209] Lib 0 (Assembly/mt.r1.unmapped.fq,Assembly/mt.r2.unmapped.fq): pe, 159143522 reads, 100 max length
[read_lib_functions-inl.h : 209] Lib 1 (Assembly/mt.se.unmapped.fq): se, 0 reads, 0 max length
[utils.h : 126] Real: 747.1210 user: 148.4221 sys: 71.0971 maxrss: 155120
--- [Tue Apr 19 16:40:10 2022] Extracting solid (k+1)-mers for k = 25 ---
--- [Tue Apr 19 16:54:03 2022] Building graph for k = 25 ---
--- [Tue Apr 19 16:54:08 2022] Assembling contigs from SdBG for k = 25 ---
--- [Tue Apr 19 16:54:22 2022] Local assembling k = 25 ---
--- [Tue Apr 19 16:56:46 2022] Extracting iterative edges from k = 25 to 29 ---
--- [Tue Apr 19 17:09:31 2022] Building graph for k = 29 ---
--- [Tue Apr 19 17:09:34 2022] Assembling contigs from SdBG for k = 29 ---
--- [Tue Apr 19 17:09:48 2022] Local assembling k = 29 ---
--- [Tue Apr 19 17:12:15 2022] Extracting iterative edges from k = 29 to 33 ---
--- [Tue Apr 19 17:25:23 2022] Building graph for k = 33 ---
--- [Tue Apr 19 17:25:27 2022] Assembling contigs from SdBG for k = 33 ---
--- [Tue Apr 19 17:25:43 2022] Local assembling k = 33 ---
--- [Tue Apr 19 17:28:13 2022] Extracting iterative edges from k = 33 to 37 ---
--- [Tue Apr 19 17:41:24 2022] Building graph for k = 37 ---
--- [Tue Apr 19 17:41:27 2022] Assembling contigs from SdBG for k = 37 ---
--- [Tue Apr 19 17:41:45 2022] Local assembling k = 37 ---
--- [Tue Apr 19 17:43:45 2022] Extracting iterative edges from k = 37 to 41 ---
--- [Tue Apr 19 17:55:27 2022] Building graph for k = 41 ---
--- [Tue Apr 19 17:55:31 2022] Assembling contigs from SdBG for k = 41 ---
--- [Tue Apr 19 17:55:48 2022] Local assembling k = 41 ---
--- [Tue Apr 19 17:57:48 2022] Extracting iterative edges from k = 41 to 45 ---
--- [Tue Apr 19 18:08:51 2022] Building graph for k = 45 ---
--- [Tue Apr 19 18:08:55 2022] Assembling contigs from SdBG for k = 45 ---
--- [Tue Apr 19 18:09:12 2022] Local assembling k = 45 ---
--- [Tue Apr 19 18:11:13 2022] Extracting iterative edges from k = 45 to 49 ---
--- [Tue Apr 19 18:21:23 2022] Building graph for k = 49 ---
--- [Tue Apr 19 18:21:27 2022] Assembling contigs from SdBG for k = 49 ---
--- [Tue Apr 19 18:21:44 2022] Local assembling k = 49 ---
--- [Tue Apr 19 18:23:45 2022] Extracting iterative edges from k = 49 to 53 ---
--- [Tue Apr 19 18:34:01 2022] Building graph for k = 53 ---
--- [Tue Apr 19 18:34:05 2022] Assembling contigs from SdBG for k = 53 ---
--- [Tue Apr 19 18:34:20 2022] Local assembling k = 53 ---
--- [Tue Apr 19 18:36:22 2022] Extracting iterative edges from k = 53 to 57 ---
--- [Tue Apr 19 18:45:32 2022] Building graph for k = 57 ---
--- [Tue Apr 19 18:45:35 2022] Assembling contigs from SdBG for k = 57 ---
--- [Tue Apr 19 18:45:48 2022] Local assembling k = 57 ---
--- [Tue Apr 19 18:47:52 2022] Extracting iterative edges from k = 57 to 61 ---
--- [Tue Apr 19 18:55:52 2022] Building graph for k = 61 ---
--- [Tue Apr 19 18:55:55 2022] Assembling contigs from SdBG for k = 61 ---
--- [Tue Apr 19 18:56:06 2022] Local assembling k = 61 ---
--- [Tue Apr 19 18:58:09 2022] Extracting iterative edges from k = 61 to 65 ---
--- [Tue Apr 19 19:05:34 2022] Building graph for k = 65 ---
--- [Tue Apr 19 19:05:36 2022] Assembling contigs from SdBG for k = 65 ---
--- [Tue Apr 19 19:05:45 2022] Local assembling k = 65 ---
--- [Tue Apr 19 19:07:48 2022] Extracting iterative edges from k = 65 to 69 ---
--- [Tue Apr 19 19:15:06 2022] Building graph for k = 69 ---
--- [Tue Apr 19 19:15:08 2022] Assembling contigs from SdBG for k = 69 ---
--- [Tue Apr 19 19:15:17 2022] Local assembling k = 69 ---
--- [Tue Apr 19 19:17:18 2022] Extracting iterative edges from k = 69 to 73 ---
--- [Tue Apr 19 19:23:38 2022] Building graph for k = 73 ---
--- [Tue Apr 19 19:23:40 2022] Assembling contigs from SdBG for k = 73 ---
--- [Tue Apr 19 19:23:46 2022] Local assembling k = 73 ---
--- [Tue Apr 19 19:26:08 2022] Extracting iterative edges from k = 73 to 77 ---
--- [Tue Apr 19 19:31:26 2022] Building graph for k = 77 ---
--- [Tue Apr 19 19:31:28 2022] Assembling contigs from SdBG for k = 77 ---
--- [Tue Apr 19 19:31:32 2022] Local assembling k = 77 ---
--- [Tue Apr 19 19:33:29 2022] Extracting iterative edges from k = 77 to 81 ---
--- [Tue Apr 19 19:38:04 2022] Building graph for k = 81 ---
--- [Tue Apr 19 19:38:06 2022] Assembling contigs from SdBG for k = 81 ---
--- [Tue Apr 19 19:38:09 2022] Local assembling k = 81 ---
--- [Tue Apr 19 19:40:01 2022] Extracting iterative edges from k = 81 to 85 ---
--- [Tue Apr 19 19:43:38 2022] Building graph for k = 85 ---
--- [Tue Apr 19 19:43:40 2022] Assembling contigs from SdBG for k = 85 ---
--- [Tue Apr 19 19:43:42 2022] Local assembling k = 85 ---
--- [Tue Apr 19 19:45:34 2022] Extracting iterative edges from k = 85 to 89 ---
--- [Tue Apr 19 19:48:30 2022] Building graph for k = 89 ---
--- [Tue Apr 19 19:48:31 2022] Assembling contigs from SdBG for k = 89 ---
--- [Tue Apr 19 19:48:33 2022] Local assembling k = 89 ---
--- [Tue Apr 19 19:50:22 2022] Extracting iterative edges from k = 89 to 93 ---
--- [Tue Apr 19 19:52:38 2022] Building graph for k = 93 ---
--- [Tue Apr 19 19:52:39 2022] Assembling contigs from SdBG for k = 93 ---
--- [Tue Apr 19 19:52:40 2022] Local assembling k = 93 ---
--- [Tue Apr 19 19:54:39 2022] Extracting iterative edges from k = 93 to 97 ---
--- [Tue Apr 19 19:56:32 2022] Building graph for k = 97 ---
--- [Tue Apr 19 19:56:33 2022] Assembling contigs from SdBG for k = 97 ---
--- [Tue Apr 19 19:56:34 2022] Local assembling k = 97 ---
--- [Tue Apr 19 19:58:26 2022] Extracting iterative edges from k = 97 to 99 ---
--- [Tue Apr 19 20:00:00 2022] Building graph for k = 99 ---
--- [Tue Apr 19 20:00:00 2022] Assembling contigs from SdBG for k = 99 ---
--- [Tue Apr 19 20:00:01 2022] Merging to output final contigs ---
--- [STAT] 66 contigs, total 24000 bp, min 203 bp, max 599 bp, avg 364 bp, N50 358 bp
--- [Tue Apr 19 20:00:02 2022] ALL DONE. Time elapsed: 12739.113557 seconds ---
5 of 15 steps (33%) done
rule idba_hybrid_assembly_1:
input: Preprocessing/mg.r1.preprocessed.fq, Preprocessing/mg.r2.preprocessed.fq, Preprocessing/mg.se.preprocessed.fq, Preprocessing/mt.r1.preprocessed.fq, Preprocessing/mt.r2.preprocessed.fq, Preprocessing/mt.se.preprocessed.fq, Assembly/mt.megahit_preprocessed.1/final.contigs.fa, Assembly/mt.megahit_unmapped.2/final.contigs.fa
output: Assembly/mgmt.idba_hybrid.1.fa
[x] Performing first hyrbid assembly step using IDBA
[x] Interleave MG and MT fastq files
[x] Join MG and MT interleaved fasta files
[x] Concatenate MT contigs, MT and MG single end files
number of threads 4
bash: line 14: 12672 Killed idba_ud -r $TMPD/merged.fa -l $TMPD/MT_contigs-MG_MT.SE.fa -o $TMPD --mink 25 --maxk 99 --step 4 --num_threads 4 --similar 0.98 --pre_correction
Error in job idba_hybrid_assembly_1 while creating output file Assembly/mgmt.idba_hybrid.1.fa.
RuleException:
CalledProcessError in line 16 of /home/imp/code/rules/Assembly/hybrid/idba.hybrid.rules:
Command '
echo "[x] Performing first hyrbid assembly step using IDBA"
echo "[x] Interleave MG and MT fastq files"
TMPD=$(mktemp -d -t --tmpdir=/home/imp/output/tmp "XXXXXX")
fq2fa --merge Preprocessing/mg.r1.preprocessed.fq Preprocessing/mg.r2.preprocessed.fq $TMPD/merged_MG.fa
fq2fa --merge Preprocessing/mt.r1.preprocessed.fq Preprocessing/mt.r2.preprocessed.fq $TMPD/merged_MT.fa
echo "[x] Join MG and MT interleaved fasta files"
cat $TMPD/merged_MG.fa $TMPD/merged_MT.fa > $TMPD/merged.fa
echo "[x] Concatenate MT contigs, MT and MG single end files"
cat <(cat Assembly/mt.megahit_preprocessed.1/final.contigs.fa Assembly/mt.megahit_unmapped.2/final.contigs.fa | awk '/^>/{print ">contig_MT_" ++i; next}{print}') <(cat Preprocessing/mg.se.preprocessed.fq | sed -n '1~4s/^@/>/p;2~4p') <(cat Preprocessing/mt.se.preprocessed.fq | sed -n '1~4s/^@/>/p;2~4p') > $TMPD/MT_contigs-MG_MT.SE.fa
idba_ud -r $TMPD/merged.fa -l $TMPD/MT_contigs-MG_MT.SE.fa -o $TMPD --mink 25 --maxk 99 --step 4 --num_threads 4 --similar 0.98 --pre_correction
mv $TMPD/contig.fa Assembly/mgmt.idba_hybrid.1.fa
rm -rf $TMPD
' returned non-zero exit status 137
File "/home/imp/code/rules/Assembly/hybrid/idba.hybrid.rules", line 16, in __rule_idba_hybrid_assembly_1
File "/usr/lib/python3.4/concurrent/futures/thread.py", line 54, in run
Will exit after finishing currently running jobs.
/home/imp/data/1018_metagenome_1.fastq => Preprocessing/mg.r1.fq
/home/imp/data/1018_metagenome_2.fastq => Preprocessing/mg.r2.fq
/home/imp/data/1018_metagenome_single.fastq => Preprocessing/mg.se.fq
/home/imp/data/1018_transcriptome_1.fastq => Preprocessing/mt.r1.fq
/home/imp/data/1018_transcriptome_2.fastq => Preprocessing/mt.r2.fq
/home/imp/data/1018_transcripptome_single.fastq => Preprocessing/mt.se.fq
symlink mg.r1.fq => Preprocessing/mg.r1.preprocessed.fq
symlink mg.r2.fq => Preprocessing/mg.r2.preprocessed.fq
symlink mg.se.fq => Preprocessing/mg.se.preprocessed.fq
symlink mt.r1.fq => Preprocessing/mt.r1.preprocessed.fq
symlink mt.r2.fq => Preprocessing/mt.r2.preprocessed.fq
symlink mt.se.fq => Preprocessing/mt.se.preprocessed.fq
Exiting because a job execution failed. Look above for error message
```https://git-r3lab.uni.lu/IMP/IMP/-/issues/165input data error2022-04-25T17:51:44+02:00EEinput data errorhello
i use docker for running IMP as non-root, after following instruction of installation i tried to run IMP using this code: impy --threads 8 --memtotal 32 --memcore 4 run -m /root/esraa/dna_data/dna_sample1_R1.fq -m /root/esraa/dna_...hello
i use docker for running IMP as non-root, after following instruction of installation i tried to run IMP using this code: impy --threads 8 --memtotal 32 --memcore 4 run -m /root/esraa/dna_data/dna_sample1_R1.fq -m /root/esraa/dna_data/dna_sample1_R2.fq -o output --single-omics
**i got this error:**
\*\*MissingInputException in line 90 of /home/imp/code/rules/data.input.rules: Missing input files for rule prepare_input_data: /home/imp/data/dna_sample1_R1.fq /home/imp/data/dna_sample1_R2.fq \*\*
in this error, it says my data is located in /home/imp/data/ directory but really i don't have this directory
Also running **impy --enter ...** showed that i don't have the data/ directory
**\~/code$ ls CHANGELOG MANIFEST.in README.rst VERSION docker impy.py outpu requirements.txt setup.py test LICENSE PyPI.md Snakefile conf imp-db lib output rules src workflows**
how can i fix this problem?
thank youhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/163Installation error with "impy install-imp-container"2022-04-22T15:00:37+02:00MimiInstallation error with "impy install-imp-container"
Hi, I tried to install using impy and got an error complaining of urlopen error. help? Thank you.2
```
$ impy install-imp-container
[x] Downloading IMP TARBALL at 'https://webdav-r3lab.uni.lu/public/R3lab/IMP/dist/imp-1.4.1.tar.gz'
Tr...
Hi, I tried to install using impy and got an error complaining of urlopen error. help? Thank you.2
```
$ impy install-imp-container
[x] Downloading IMP TARBALL at 'https://webdav-r3lab.uni.lu/public/R3lab/IMP/dist/imp-1.4.1.tar.gz'
Traceback (most recent call last):
File "/home/user/.pyenv/versions/anaconda3-2018.12/envs/IMP/lib/python3.10/urllib/request.py", line 1348, in do_open
h.request(req.get_method(), req.selector, req.data, headers,
File "/home/user/.pyenv/versions/anaconda3-2018.12/envs/IMP/lib/python3.10/http/client.py", line 1282, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/home/user/.pyenv/versions/anaconda3-2018.12/envs/IMP/lib/python3.10/http/client.py", line 1328, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/home/user/.pyenv/versions/anaconda3-2018.12/envs/IMP/lib/python3.10/http/client.py", line 1277, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/home/user/.pyenv/versions/anaconda3-2018.12/envs/IMP/lib/python3.10/http/client.py", line 1037, in _send_output
self.send(msg)
File "/home/user/.pyenv/versions/anaconda3-2018.12/envs/IMP/lib/python3.10/http/client.py", line 975, in send
self.connect()
File "/home/user/.pyenv/versions/anaconda3-2018.12/envs/IMP/lib/python3.10/http/client.py", line 1447, in connect
super().connect()
File "/home/user/.pyenv/versions/anaconda3-2018.12/envs/IMP/lib/python3.10/http/client.py", line 941, in connect
self.sock = self._create_connection(
File "/home/user/.pyenv/versions/anaconda3-2018.12/envs/IMP/lib/python3.10/socket.py", line 845, in create_connection
raise err
File "/home/user/.pyenv/versions/anaconda3-2018.12/envs/IMP/lib/python3.10/socket.py", line 833, in create_connection
sock.connect(sa)
OSError: [Errno 113] No route to host
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/user/.pyenv/versions/anaconda3-2018.12/envs/IMP/bin/impy", line 8, in <module>
sys.exit(cli())
File "/home/user/.pyenv/versions/anaconda3-2018.12/envs/IMP/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/home/user/.pyenv/versions/anaconda3-2018.12/envs/IMP/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/home/user/.pyenv/versions/anaconda3-2018.12/envs/IMP/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/user/.pyenv/versions/anaconda3-2018.12/envs/IMP/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/user/.pyenv/versions/anaconda3-2018.12/envs/IMP/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/home/user/.pyenv/versions/anaconda3-2018.12/envs/IMP/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/user/.pyenv/versions/anaconda3-2018.12/envs/IMP/lib/python3.10/site-packages/impy.py", line 235, in install_imp_container
with urlopen(ctx.obj['image-repo']) as response, open(fname, 'wb') as out_file:
File "/home/user/.pyenv/versions/anaconda3-2018.12/envs/IMP/lib/python3.10/urllib/request.py", line 216, in urlopen
return opener.open(url, data, timeout)
File "/home/user/.pyenv/versions/anaconda3-2018.12/envs/IMP/lib/python3.10/urllib/request.py", line 519, in open
response = self._open(req, data)
File "/home/user/.pyenv/versions/anaconda3-2018.12/envs/IMP/lib/python3.10/urllib/request.py", line 536, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
File "/home/user/.pyenv/versions/anaconda3-2018.12/envs/IMP/lib/python3.10/urllib/request.py", line 496, in _call_chain
result = func(*args)
File "/home/user/.pyenv/versions/anaconda3-2018.12/envs/IMP/lib/python3.10/urllib/request.py", line 1391, in https_open
return self.do_open(http.client.HTTPSConnection, req,
File "/home/user/.pyenv/versions/anaconda3-2018.12/envs/IMP/lib/python3.10/urllib/request.py", line 1351, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 113] No route to host>
```https://git-r3lab.uni.lu/IMP/IMP/-/issues/162install issue2021-05-14T05:10:30+02:00Edward-Tristaninstall issue![2021-05-12_20-09-56屏幕截图](/uploads/fcc2d22f1ea33f7f0de31faf5f91e651/2021-05-12_20-09-56屏幕截图.png)
When I import the ktUpdateTaxonomy.sh comand line it report "Update failed.Is your internet connection okay?"
But there is no problem with ...![2021-05-12_20-09-56屏幕截图](/uploads/fcc2d22f1ea33f7f0de31faf5f91e651/2021-05-12_20-09-56屏幕截图.png)
When I import the ktUpdateTaxonomy.sh comand line it report "Update failed.Is your internet connection okay?"
But there is no problem with my internet connection
ktUpdateTaxonomy.sh
>>>>> Updating GI to taxID dump (nucleotide)...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0
curl: (19) Given file does not exist
>>>>> Update failed.
Is your internet connection okay?https://git-r3lab.uni.lu/IMP/IMP/-/issues/161Installation error2021-04-12T04:53:42+02:00Edward-TristanInstallation error
![2021-04-11_15-09-23屏幕截图](/uploads/14bcbcfdf2bf16969afccf67399aad0c/2021-04-11_15-09-23屏幕截图.png)
![2021-04-11_15-09-23屏幕截图](/uploads/14bcbcfdf2bf16969afccf67399aad0c/2021-04-11_15-09-23屏幕截图.png)https://git-r3lab.uni.lu/IMP/IMP/-/issues/160IMP image not installed2021-01-17T01:55:33+01:00songb1230IMP image not installedI just started to install IMP and had the following error:
IMP image not installed. Please run `impy install_imp_container` first.
Please help me to resolve this error.
Thanks
BKI just started to install IMP and had the following error:
IMP image not installed. Please run `impy install_imp_container` first.
Please help me to resolve this error.
Thanks
BKhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/153can't find trimmomatic2020-04-29T21:29:57+02:00Billy Tajcan't find trimmomaticHi,
I followed all the instructions from the install guide (a mixture of the new-readme, and the webpage instructions as both were incomplete) and I'm seeing an error complaining that Trimmomatic can't be found.
I'm trying to run the c...Hi,
I followed all the instructions from the install guide (a mixture of the new-readme, and the webpage instructions as both were incomplete) and I'm seeing an error complaining that Trimmomatic can't be found.
I'm trying to run the conda version of the pipeline, and testing the sample sets.
help?https://git-r3lab.uni.lu/IMP/IMP/-/issues/151CalledProcessError in line 59 of /home/imp/code/rules/Preprocessing/filtering...2019-01-16T14:41:00+01:00TANKENGUOCalledProcessError in line 59 of /home/imp/code/rules/Preprocessing/filtering.rules:![error](/uploads/973a8107dede9b0e258f726fd1ffe53d/error.png)![error](/uploads/973a8107dede9b0e258f726fd1ffe53d/error.png)https://git-r3lab.uni.lu/IMP/IMP/-/issues/149git clone results in HTTPS error2019-01-09T07:43:42+01:00Shea Connellgit clone results in HTTPS errorWhen attempting to initially clone the bioconda repo (or any other repo actually), the request fails as the info/refs needs a valid username and password:
```
git clone -b bioconda https://git-r3lab.uni.lu/IMP/IMP.git # Get the code base...When attempting to initially clone the bioconda repo (or any other repo actually), the request fails as the info/refs needs a valid username and password:
```
git clone -b bioconda https://git-r3lab.uni.lu/IMP/IMP.git # Get the code base
Initialized empty Git repository in /gpfs/home/pvy17zzu/IMP/IMP/.git/
error: while accessing https://git-r3lab.uni.lu/IMP/IMP.git/info/refs
fatal: HTTP request failed
```https://git-r3lab.uni.lu/IMP/IMP/-/issues/147Does skip_preprocessing work (snakemake)?2018-11-20T14:36:55+01:00Javier TamamesDoes skip_preprocessing work (snakemake)?Hi all
I am trying to run a job skipping trimming and filtering since I already have clean reads. My config.imp.json file is attached. I run it with snakemake as:
CONFIGFILE="/home/jtamames/Huinay/IMP/config.imp.json" snakemake -s /hom...Hi all
I am trying to run a job skipping trimming and filtering since I already have clean reads. My config.imp.json file is attached. I run it with snakemake as:
CONFIGFILE="/home/jtamames/Huinay/IMP/config.imp.json" snakemake -s /home/jtamames/software/IMP/IMP/Snakefile --use-conda -p
and got this error:
Building DAG of jobs...
MissingInputException in line 1 of /home/jtamames/software/IMP/IMP/rules/Preprocessing/trimming.rules:
Missing input files for rule trimming:
/home/my/Huinay/IMP/imp-db/adapters/adapters.done
Indeed the file /home/my/Huinay/IMP/imp-db/adapters/adapters.done is missing, but I set skip_preprocessing and preprocessing_filtering to "false" in the config file. Isn't that enough to skip the filtering step?
Thanks!
[config.imp.json](/uploads/cfca72cebf9644574764c9ef8af4e3ad/config.imp.json)https://git-r3lab.uni.lu/IMP/IMP/-/issues/141IMP-specific prokka conda recipe needed2018-10-04T14:26:35+02:00Cedric LacznyIMP-specific prokka conda recipe needed`prokka` comes by default with `tbl2asn`.
While this makes sense when using prokka for isolate genomes, the use of this step for metagenomes is less apparent.
In particular, this step is very time-consuming, i.e., can take 10 hours or mo...`prokka` comes by default with `tbl2asn`.
While this makes sense when using prokka for isolate genomes, the use of this step for metagenomes is less apparent.
In particular, this step is very time-consuming, i.e., can take 10 hours or more.
Moreover, the output is currently not used at all by the IMP pipeline.
Hence, creating a dedicated recipe for prokka is suggested.
A dedicated recipe would allow us to apply a patch to prokka's perl script to disable the tbl2asn step.
While it could also be described in the installation instructions of IMP to comment-out the respective line when using the prokka bioconda recipe, having a dedicated recipe seems to be the cleaner solution.https://git-r3lab.uni.lu/IMP/IMP/-/issues/140Why are Fasta index and BAM index not created by distinct rules, but in the v...2018-10-01T15:52:58+02:00Cedric LacznyWhy are Fasta index and BAM index not created by distinct rules, but in the variant_calling rule?The following seems to be suboptimal to me:
https://git-r3lab.uni.lu/IMP/IMP/blob/master/rules/Analysis/variant.rule#L11
https://git-r3lab.uni.lu/IMP/IMP/blob/master/rules/Analysis/variant.rule#L18
Shouldn't the respective files be part...The following seems to be suboptimal to me:
https://git-r3lab.uni.lu/IMP/IMP/blob/master/rules/Analysis/variant.rule#L11
https://git-r3lab.uni.lu/IMP/IMP/blob/master/rules/Analysis/variant.rule#L18
Shouldn't the respective files be part of the `input:` **and** have a respective rule, rather than being created within the `variant_calling` rule, if they do not originally exist?
`{input[1]}.bai` is probably less of an issue as it is *independent* for metaG and metaT, but since `{input[0]}.fai` is the merged assembly of both types (`mg` and `mt`), it needs to be created only *once*.
Could it be that executing this rule for `mg` and `mt` *in parallel* could create issues/overwritings/collisions here?
Best,
Cedrichttps://git-r3lab.uni.lu/IMP/IMP/-/issues/139Error in idba_hybrid_assembly_22018-09-24T11:33:30+02:00Cedric LacznyError in idba_hybrid_assembly_2Dear all,
there seems to be an error in
https://git-r3lab.uni.lu/IMP/IMP/blob/master/rules/Assembly/hybrid/idba.hybrid.rules#L63
It seems that this would translate to:
`input[2]` translate to `'Assembly/mgmt.se.idba_hybrid.mt.unmapped...Dear all,
there seems to be an error in
https://git-r3lab.uni.lu/IMP/IMP/blob/master/rules/Assembly/hybrid/idba.hybrid.rules#L63
It seems that this would translate to:
`input[2]` translate to `'Assembly/mgmt.se.idba_hybrid.mt.unmapped.fq',`and `input[3]` to `'Assembly/mgmt.r1.idba_hybrid.mg.unmapped.fq',`
Put differently, this looks to me like it would merge M*G*-SE and M*T*-R1…
Best,
Cedrichttps://git-r3lab.uni.lu/IMP/IMP/-/issues/137samtools-1.9 might break host filtering step2018-10-08T16:28:24+02:00Cedric Lacznysamtools-1.9 might break host filtering stepAs part of the effort to enable conda-based installation, it occured that `samtools-1.9` was installed, despite that `0.1.19` **should** be installed (iris had newer version for an as-of-yet unknown reason, i.e., `all.yaml` specified `sa...As part of the effort to enable conda-based installation, it occured that `samtools-1.9` was installed, despite that `0.1.19` **should** be installed (iris had newer version for an as-of-yet unknown reason, i.e., `all.yaml` specified `samtools==0.1.19` and was **identical** between iris and litcrit).
It is speculated that this lead to the host filtering step failing on iris:
```
rule mt_filtering:
input: Preprocessing/mt.r1.trimmed.rna_filtered.fq, Preprocessing/mt.r2.trimmed.rna_filtered.fq, Preprocessing/mt.se.trimmed.rna_filtered.fq,
/home/users/claczny/projects/imp-devel/dbs/hg38-db/filtering/hg38.fa, /home/users/claczny/projects/imp-devel/dbs/hg38-db/filtering/hg38.fa, /home/
users/claczny/projects/imp-devel/dbs/hg38-db/filtering/hg38.fa.amb, /home/users/claczny/projects/imp-devel/dbs/hg38-db/filtering/hg38.fa.ann, /hom
e/users/claczny/projects/imp-devel/dbs/hg38-db/filtering/hg38.fa.bwt, /home/users/claczny/projects/imp-devel/dbs/hg38-db/filtering/hg38.fa.pac, /h
ome/users/claczny/projects/imp-devel/dbs/hg38-db/filtering/hg38.fa.sa
output: Preprocessing/mt.r1.trimmed.rna_filtered.hg38_filtered.fq, Preprocessing/mt.r2.trimmed.rna_filtered.hg38_filtered.fq, Preprocessing/mt
.se.trimmed.rna_filtered.hg38_filtered.fq
jobid: 13
TMP_FILE=$(mktemp --tmpdir=/tmp -t "alignment_XXXXXX.bam")
BUFFER=$(mktemp --tmpdir=/tmp -t "alignment_XXXXXX.bam")
bwa mem -v 1 -t 26 /home/users/claczny/projects/imp-devel/dbs/hg38-db/filtering/hg38.fa Preprocessing/mt.r1.trimmed.rna_filtered.fq Prepro
cessing/mt.r2.trimmed.rna_filtered.fq | samtools view -@ 26 -bS - > $TMP_FILE
samtools merge -@ 26 -u - <(samtools view -@ 26 -u -f 4 -F 264 $TMP_FILE) <(samtools view -@ 26 -u -f 8 -F 260 $TMP_FILE)
<(samtools view -@ 26 -u -f 12 -F 256 $TMP_FILE) | samtools view -@ 26 -bF 0x800 - | samtools sort -o -@ 26 -m 2G -n - $BUFFER -
| bamToFastq -i stdin -fq Preprocessing/mt.r1.trimmed.rna_filtered.hg38_filtered.fq -fq2 Preprocessing/mt.r2.trimmed.rna_filtered.hg38_fi
ltered.fq
if [[ -s Preprocessing/mt.se.trimmed.rna_filtered.fq ]]
then
bwa mem -v 1 -t 26 /home/users/claczny/projects/imp-devel/dbs/hg38-db/filtering/hg38.fa Preprocessing/mt.se.trimmed.rna_filtered.fq | samt
ools view -@ 26 -bS - | samtools view -@ 26 -uf 4 - | bamToFastq -i stdin -fq Preprocessing/mt.se.trimmed.rna_filtered.hg38_filtered.fq
else
echo "Preprocessing/mt.se.trimmed.rna_filtered.fq is empty, skipping single end human sequence filtering, but creating it anyway..."
touch Preprocessing/mt.se.trimmed.rna_filtered.hg38_filtered.fq
fi
rm -rf $BUFFER* $TMP_FILE
[...]
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[main] Version: 0.7.9a-r786
[main] CMD: bwa mem -v 1 -t 26 /home/users/claczny/projects/imp-devel/dbs/hg38-db/filtering/hg38.fa Preprocessing/mt.r1.trimmed.rna_filtered.fq Preprocessing/mt.r2.trimmed.rna_filtered.fq
[main] Real time: 255.606 sec; CPU: 4680.023 sec
Usage: samtools sort [options...] [in.bam]
Options:
-l INT Set compression level, from 0 (uncompressed) to 9 (best)
-m INT Set maximum memory per thread; suffix K/M/G recognized [768M]
-n Sort by read name
-t TAG Sort by value of TAG. Uses position as secondary index (or read name if -n is set)
-o FILE Write final output to FILE rather than standard output
-T PREFIX Write temporary files to PREFIX.nnnn.bam
--input-fmt-option OPT[=VAL]
Specify a single input file format option in the form
of OPTION or OPTION=VALUE
-O, --output-fmt FORMAT[,OPT[=VAL]]...
Specify output format (SAM, BAM, CRAM)
--output-fmt-option OPT[=VAL]
Specify a single output file format option in the form
of OPTION or OPTION=VALUE
--reference FILE
Reference sequence FASTA FILE [null]
-@, --threads INT
Number of additional threads to use [0]
[W::bam_merge_core2] No @HD tag found.
Error in rule mt_filtering:
jobid: 13
output: Preprocessing/mt.r1.trimmed.rna_filtered.hg38_filtered.fq, Preprocessing/mt.r2.trimmed.rna_filtered.hg38_filtered.fq, Preprocessing/mt.se.trimmed.rna_filtered.hg38_filtered.fq
```
This needs further inspection.https://git-r3lab.uni.lu/IMP/IMP/-/issues/136Accelerating the rRNA filtering step2018-10-10T12:18:37+02:00Cedric LacznyAccelerating the rRNA filtering step`sortmerna` appears to be a relatively slow program for the metatranscriptomic files we are typically processing.
An acceleration of this step would be nice!
There do not seem to be too many alternatives out there though: https://omicto...`sortmerna` appears to be a relatively slow program for the metatranscriptomic files we are typically processing.
An acceleration of this step would be nice!
There do not seem to be too many alternatives out there though: https://omictools.com/rrna-filtering-category
Some of them are web-based, hence do not really represent an alternative.
Maybe `rrnafilter` would be one, but from superficially scanning over it, it seems to be based on k-mer abundances.
While this enables the identification of candidate rRNA reads without the need for reference sequences, it could also lead to false positives, i.e., non-rRNA gene-derived sequences of high (or divergent) abundance) that are wrongly classified as rRNA gene-derived.
Not sure how the tool was tested in the original publication.
A dedicated test might be required, though.https://git-r3lab.uni.lu/IMP/IMP/-/issues/135Read preprocessing using fastp2018-10-08T16:32:54+02:00Cedric LacznyRead preprocessing using fastpMaybe [fastp](https://github.com/OpenGene/fastp) could be integrated as a preprocessing step.
According to its documentation is is (supposedly)
>like FASTQC but faster and more informative
and it integrates trimming etc. while automati...Maybe [fastp](https://github.com/OpenGene/fastp) could be integrated as a preprocessing step.
According to its documentation is is (supposedly)
>like FASTQC but faster and more informative
and it integrates trimming etc. while automatically detecting the adapters as well as
>trim polyG in 3' ends, which is commonly seen in NovaSeq/NextSeq datahttps://git-r3lab.uni.lu/IMP/IMP/-/issues/134idba hybrid assembly error2018-05-21T10:08:21+02:00Adam Blanchardidba hybrid assembly errorHi,
I have been running the pipeline with my own test dataset. It runs fine up until the idba step and it produces this error:
14 of 58 steps (24%) done
rule idba_hybrid_assembly_1:
input: Preprocessing/mg.r1.preprocessed.fq, Preproc...Hi,
I have been running the pipeline with my own test dataset. It runs fine up until the idba step and it produces this error:
14 of 58 steps (24%) done
rule idba_hybrid_assembly_1:
input: Preprocessing/mg.r1.preprocessed.fq, Preprocessing/mg.r2.preprocessed.fq, Preprocessing/mg.se.preprocessed.fq, Preprocessing/mt.r1.preprocessed.fq, Preprocessing/mt.r2.preprocessed.fq, Preprocessing/mt.se.preprocessed.fq, Assembly/mt.megahit_preprocessed.1/final.contigs.fa, Assembly/mt.megahit_unmapped.2/final.contigs.fa
output: Assembly/mgmt.idba_hybrid.1.fa
[x] Performing first hyrbid assembly step using IDBA
[x] Interleave MG and MT fastq files
[x] Join MG and MT interleaved fasta files
[x] Concatenate MT contigs, MT and MG single end files
number of threads 4
reads 60039998
long reads 35534055
extra reads 0
read_length 83
kmer 62
bash: line 14: 9630 Killed idba_ud -r $TMPD/merged.fa -l $TMPD/MT_contigs-MG_MT.SE.fa -o $TMPD --mink 25 --maxk 99 --step 4 --num_threads 4 --similar 0.98 --pre_correction
Error in job idba_hybrid_assembly_1 while creating output file Assembly/mgmt.idba_hybrid.1.fa.
RuleException:
CalledProcessError in line 16 of /home/imp/code/rules/Assembly/hybrid/idba.hybrid.rules:
Command '
echo "[x] Performing first hyrbid assembly step using IDBA"
echo "[x] Interleave MG and MT fastq files"
TMPD=$(mktemp -d -t --tmpdir=/home/imp/output/tmp "XXXXXX")
fq2fa --merge Preprocessing/mg.r1.preprocessed.fq Preprocessing/mg.r2.preprocessed.fq $TMPD/merged_MG.fa
fq2fa --merge Preprocessing/mt.r1.preprocessed.fq Preprocessing/mt.r2.preprocessed.fq $TMPD/merged_MT.fa
echo "[x] Join MG and MT interleaved fasta files"
cat $TMPD/merged_MG.fa $TMPD/merged_MT.fa > $TMPD/merged.fa
echo "[x] Concatenate MT contigs, MT and MG single end files"
cat <(cat Assembly/mt.megahit_preprocessed.1/final.contigs.fa Assembly/mt.megahit_unmapped.2/final.contigs.fa | awk '/^>/{print ">contig_MT_" ++i; next}{print}') <(cat Preprocessing/mg.se.preprocessed.fq | sed -n '1~4s/^@/>/p;2~4p') <(cat Preprocessing/mt.se.preprocessed.fq | sed -n '1~4s/^@/>/p;2~4p') > $TMPD/MT_contigs-MG_MT.SE.fa
idba_ud -r $TMPD/merged.fa -l $TMPD/MT_contigs-MG_MT.SE.fa -o $TMPD --mink 25 --maxk 99 --step 4 --num_threads 4 --similar 0.98 --pre_correction
mv $TMPD/contig.fa Assembly/mgmt.idba_hybrid.1.fa
rm -rf $TMPD
' returned non-zero exit status 137
File "/home/imp/code/rules/Assembly/hybrid/idba.hybrid.rules", line 16, in __rule_idba_hybrid_assembly_1
File "/usr/lib/python3.4/concurrent/futures/thread.py", line 54, in run
Will exit after finishing currently running jobs.
/home/imp/data/1D_S64_L006_R1_001.fastq => Preprocessing/mg.r1.fq
/home/imp/data/1D_S64_L006_R2_001.fastq => Preprocessing/mg.r2.fq
/home/imp/data/4_1D_S4_L001_R1_001.fastq => Preprocessing/mt.r1.fq
/home/imp/data/4_1D_S4_L001_R2_001.fastq => Preprocessing/mt.r2.fq
Exiting because a job execution failed. Look above for error message
Any ideas on how to rectify?
Ahttps://git-r3lab.uni.lu/IMP/IMP/-/issues/132Error in cap32018-05-06T20:32:30+02:00easternbluebirdError in cap3Assembly runs fine, but cap3 throws an error. Any idea what is the cause of this?
```bash
bash: line 5: 20 Segmentation fault (core dumped) cap3 $NAME.cat.fa -p 98 -o 100
Error in job cap3 while creating output file Assembly/mg....Assembly runs fine, but cap3 throws an error. Any idea what is the cause of this?
```bash
bash: line 5: 20 Segmentation fault (core dumped) cap3 $NAME.cat.fa -p 98 -o 100
Error in job cap3 while creating output file Assembly/mg.assembly.merged.fa.
RuleException:
CalledProcessError in line 18 of /home/imp/code/rules/Assembly/common/merge-assembly.rules:
Command '
NAME=Assembly/mg.assembly
cat Assembly/mg.megahit_preprocessed.1.fa Assembly/mg.megahit_unmapped.2.fa > $NAME.cat.fa
# Run cap3
cap3 $NAME.cat.fa -p 98 -o 100
# Concatenate assembled contigs, singletons and rename the contigs
cat $NAME.cat.fa.cap.contigs $NAME.cat.fa.cap.singlets | awk '/^>/{print ">contig_" ++i; next}{print}' > $NAME.merged.fa
' returned non-zero exit status 139
File "/usr/lib/python3.4/concurrent/futures/thread.py", line 54, in run
Will exit after finishing currently running jobs.
Exiting because a job execution failed. Look above for error message
```Shaman NarayanasamyShaman Narayanasamyhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/131Singularity support?2018-05-03T14:47:44+02:00Scott DanielSingularity support?Are there any plans to package the program in singularity containers instead of docker? My infrastructure supports singularity but not docker due to security concerns.Are there any plans to package the program in singularity containers instead of docker? My infrastructure supports singularity but not docker due to security concerns.Deepti MittalDeepti Mittalhttps://git-r3lab.uni.lu/IMP/IMP/-/issues/130How can I rebuild the docker file?2018-03-12T14:31:32+01:00easternbluebirdHow can I rebuild the docker file?I get this error
```
Step 1/18 : FROM docker-r3lab.uni.lu/imp/imp-tools:1.4.1
Get https://docker-r3lab.uni.lu/v2/imp/imp-tools/manifests/1.4.1: no basic auth credentials
```
when I try to rebuild the Docker file. I found some errors in...I get this error
```
Step 1/18 : FROM docker-r3lab.uni.lu/imp/imp-tools:1.4.1
Get https://docker-r3lab.uni.lu/v2/imp/imp-tools/manifests/1.4.1: no basic auth credentials
```
when I try to rebuild the Docker file. I found some errors in the file and would like to fix them because otherwise I do not get any taxonomic information.
Some links in the Docker file are dead and the quast version is outdated.
How can I fix this? Or can anyone give me credentials so I can rebuild the fixed docker file?