help > RE: conn_batch distributed preprocessing
Apr 25, 2024  12:04 AM | Alfonso Nieto-Castanon - Boston University
RE: conn_batch distributed preprocessing

Hi,


That script looks perfectly fine in both cases, with or without the two BATCH.parallel lines (i.e. without those two lines preprocessing will be run locally, and with those two lines preprocessing will be run remotely). Could you please check first that CONN's Condor default settings are working correctly in your computer? e.g. using something like:


  conn_jobmanager setdefault 'HTCondor' save current
  conn_jobmanager test


(and if you have access to a GUI use "conn_jobmanager settings" to look at the HTCondor profile options in more detail, it should look like the attached image)


Best


Alfonso


Originally posted by max345:



Hi CONN expertes,


I am currently trying to do my preprocessing using the conn_batch (there are some technical reasons, which don't want to explain here, but I cannot run the preprocessing via the GUI). So I've set up my CONN file via the GUI (import of the data etc.) and have written a MATLAB script, which should then be exectued by conn_batch, but I get some error messages at the moment. That's why I would like to get some feedback on my script here, so that it will work soon:


 


%prepare batch structure
clear BATCH;


%load project
BATCH.filename ='path/to/my/CONN/mat/file';


%insert command to use distributed processing
%printf " OPTS\n error = "STDERR"\n output = "STDOUT"\n executable = "SCRIPT"\n\n queue\n" > SCRIPT.sub && chmod a+x SCRIPT && condor_submit SCRIPT.sub


%defining of preprocessing steps
BATCH.Setup.preprocessing.steps = 'default_mni'; %use CONNs default pipeline
BATCH.Setup.preprocessing.sliceorder = 'interleaved (bottom-up)'; %select right slice order
BATCH.Setup.preprocessing.fwhm = 5; 
BATCH.parallel.profile = 'HTCondor computer cluster';
BATCH.parallel.N = 8; %how many parallel jobs
%BATCH.Setup.done = 1;
%BATCH.Setup.overwrite = 'Yes';%run batch script
conn_batch(BATCH)


 


When I run the script without the two lines about parallel jobs (BATCH.parallel.profile and BATCH.parallel.N), my script works perfectly fine. Do I need to re-write these two lines or do I need to un-comment the line starting "printf .." etc?


Thanks for your help!


Best,


Max


 



 

Threaded View

TitleAuthorDate
max345 Apr 18, 2024
RE: conn_batch distributed preprocessing
Alfonso Nieto-Castanon Apr 25, 2024