All Questions
Tagged with parallel-processing cluster-computing
231 questions
0
votes
1
answer
27
views
Error in R: "one node produced an error: Argumento NA/NaN" when extracting raster values in package parallel version 4.2.2
I'm working on an R script to build a correlation matrix based on values extracted from WorldClim bioclimatic layers at specific geographic coordinates (WGS84).
I have a dataset (db) with 842 obs. of ...
0
votes
0
answers
29
views
Pre-staging large data files for parallel job execution
Apologies in advance if this is a mundane or unclear question.
I want to scale up a workflow on on a cluster to run a program concurrently on several nodes. The program in question references a large, ...
- 81
1
vote
1
answer
107
views
parpool MATLAB running in the command-line on a slurm cluster
I've been running a matlab script in a cluster and I got a warning saying that my MATLAB programm was comsuming in excess 500% CPU, so somehow the MATLAB access the head-node.
I've been submitting my ...
0
votes
1
answer
81
views
Setup Azure doAzureParallel for first time does not work Error in getCluster() AccountCoreQuotaReached AccountLowPriorityCoreQuotaReached
In R programming, I am trying doAzureParallel for the first time and I encountered this cluster quota error:
Error in getCluster(clusterName = "myPoolName") : Code: AccountCoreQuotaReached - ...
- 721
2
votes
1
answer
116
views
Problem running single-node job on computing cluster in conda environment for all processors in R
I am trying to run a single-node job on my university computing cluster, submitted via qsub, within a conda environment "myenv". As soon as I start my parallelization within my R script, the ...
- 23
1
vote
1
answer
870
views
Running an Independent SLURM Job with mpirun Inside a Python Script - Recursive Call Error
I'm currently using a Python script that requires MPI to operate. This script runs on a SLURM system.
In order to run my Python script, I define the number of nodes to use and launch the following ...
- 105
0
votes
1
answer
45
views
How can I separate cores in different group and used by multiple function in MATLAB
I am currently writing a matlab program. There's a function, denote it as myfunc(..). myfunc has a parfor which requires 4 CPUs. I need to run myfunc for 3 times, and it happens that my PC has 12 ...
- 3
0
votes
1
answer
501
views
How to run a .sh in different nodes using different data in slurm?
I'm a beginner in Slurm and I would like some help with the following problem. I've made a .sh, within which, initially, a matlab script creates two arrays (based on the value of a parameter i). Then,...
- 1
0
votes
1
answer
75
views
Real Time computation
I have a algorithm written in python and mysql which takes inputs in csv file and some properties and then run for 20-25 mins to produce output.
I want to make it realtime such that if input csv is ...
0
votes
1
answer
452
views
How to decouple nodes and cores with MPI?
I have code currently using multiprocessing with joblib to distribute computations from a loop over multiple cores within one cluster node, like so:
from joblib import Parallel, delayed
def function():...
- 588
0
votes
0
answers
57
views
where to use multiprocessing in multi-loop program
I am running a program that uses [a lot of] looping to perform a type of bootstrap analysis. The computational step can take a long time (60+) seconds, which means that collectively this program could ...
- 588
1
vote
1
answer
347
views
How to convert a loop to a Job-array in LSF cluster
I have 100 files, and I want to parallelise my submission to save time instead of running jobs one by one. How can I change this script to a Job-array in LSF using bsub submission system and run 10 ...
- 3,158
1
vote
1
answer
1k
views
Why are the workers failing to connect when calling `makePSOCKcluster` from an external `R` process?
I am experiencing a problem already raised before
here
and
here,
but in a different context.
Suppose I have an R script file called psock.R that contains the following
code:
cat("Setup cluster...\...
- 2,987
0
votes
0
answers
58
views
Controlling division of workload between parallelized MKL, openMP and MPI
I am trying to learn MPI for HPC applications, using Fortran. I have a question regarding the distribution of workload when I have a code that is parallelized for single node computation using ...
- 199
0
votes
2
answers
320
views
How to fetch data from one cluster to another cluster in dask cluster?
I have created a 1st cluster using this in my jupyter notebook:
from dask.distributed import Client, LocalCluster
cluster = LocalCluster(name='clus1',n_workers=1,dashboard_address='localhost:8789')
...
- 377