Bioinformatics Asked by Kamil S Jaron on December 2, 2020
Snakemake is not really designed to store temporary files to local disk, but in the documentation, they suggest a workaround using "shadow rules".
The problem I have is, that our front node does not have /scratch/..
directory and therefore snakemake complains right away
snakemake my_result.tsv -p --jobs 10 --shadow-prefix /scratch/$USER --cluster "qsub "
...
PermissionError: [Errno 13] Permission denied: '/scratch'
Every single computational node does have the directory, only the frontend does not. Is there any easy workaround?
— edit —
I will try to clarify where I think is the problem.
The head node does not have anything mounted to /scratch
, I think that’s why I am getting that error. I sort of expected that snakemake will attempt do create the shadow environments for individual executed jobs, because if it does, everything would be fine. All the computing nodes do have their local disks mounted to /scratch
.
Of course, I could use a different shadow directory, but all the directories that are mounted both at the front-end and computing nodes are NOT local disks of individual nodes, which would cause more network traveling of the temporary data.
— edit 2 —
Thanks @Chris_Rands for finding the relevant lines. I opened an Issue on Snakemake GitHub. Once I will know more, I will get back with the answer here.
What is happening is your Admin is not allowing you access to /scratch ... which is undertandable. Essentially you are attempting to access the headnode /scratch
. If every user on the cluster had access, it would be a bit chaotic.
I am surprised by the error, not that I would use Snakemake I'd code it, because you would expect a data pipeline tool as established as Snakemake to have a solution to working on a cluster. What you are describing is a standard cluster set-up between headnode and 'sub-node' (the technical term is slavenode, but I hate that term). The headnode is mirrored across the cluster, hence the error. Once you submit via qsub
the mirror will 'display' the headnode layout on the subnode.
/scratch
, this is root privilege. It is a bit unlikely they will say 'yes' but its worth a try.Snakemake
do a Q&A interaction on Overflow SE (but please come back here!) ... they must know the work around.Frankly I am astonished because the whole point of Snakemake, is the ease of constructing a data pipeline and data pipelines are processed on clusters, thus this error seems to up-end it, but who am I?
Answered by Michael on December 2, 2020
Get help from others!
Recent Questions
Recent Answers
© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP