We have successfully run subjects through the GUI, and are now looking for ways to batch and run hundreds of subjects through AutoEACSF.   What's the best way to do that?  

The GUI creates a series of python scripts with filenames hardcoded into the scripts as parser argument defaults/constants.  I could adapt those scripts to un-hardcode the filenames, but I suspect that would be reinventing the wheel. 

Do you have command-line only code available or advice for the easiest way to batch process?

Thanks for all your help!!  -C

We planned for this from the start. Most of our tools (including Auto_EACSF) can be run as a command line command. Thus, with a bit of scripting experience, it should be straightforward to batch process them.

You can run Auto_EACSF from the terminal using the "--help" tag and you will see a brief usage information:

USAGE:

Auto_EACSF [--returnparameterfile ]
[--processinformationaddress ] [--xml] [--echo]
[-t ] [--output_dir ] [--cerebmask
] [--subjectVentricleMask ]
[--segmentation ] [--brainmask ]
[--t2 ] [--t1 ] [--noGUI] [-p
] [--] [--version] [-h]
Where:
--returnparameterfile
Filename in which to write simple return parameters (int, float,
int-vector, etc.) as opposed to bulk return parameters (image,
geometry, transform, measurement, table).
--processinformationaddress
Address of a structure to store process information (progress, abort,
etc.). (value: 0)
--xml
Produce xml description of command line arguments (value: 0)
--echo
Echo the command line arguments (value: 0)
-t , --
--output_dir
Output directory
--cerebmask
Input cerebellum mask
--subjectVentricleMask
Input subject ventricle mask
--segmentation
Input brain segmentation
--brainmask
Input brainmask
--t2
Input T2 image
--t1
Input T1 image
--noGUI
Use AutoEACSF without the GUI (value: 0)
-p , --param
Input parameters in a json file
--, --ignore_rest
Ignores the rest of the labeled arguments following this flag.
--version
Displays version information and exits.
-h, --help
Displays usage information and exits.
Description: Automatically extract external CSF from T1 or T1 and T2
images.

What you need:
- save the json for the parameters and executable information
- have a way to edit the json file for all datasets (e.g. via json reading/writing in python using the json python library)
- run Auto_EACSF --noGUI --param yourParam.xml for each case with its edited json file

Martin
Originally posted by cccoleman:

Originally posted by Martin Styner:
We downloaded the default_config.json and edited it to include absolute paths for our copies of the relevant files.  I then saved it AS a JSON but with a .xml filename.  (File is atttached)

We then run the recomended commands:

The program appears to hang after delivering the follow message:

In the output directory, we do get a few files (ABCparam.xml + the five scripts in the PythonScripts directory), but no actual outputs.
Any idea where what going astray?
C

Sorry for creating a bit of confusion:
1. do not save the json as an xml, but rather leave it as json, so the call should be:

2. the error you get is not fatal, and we get the same error when we run it, but the tool should continue and work if the json is setup correctly. So, my expectation is that the json is incorrect.

I looked at your json and can see that your are using a T1w image only (not T1w and T2w). While that should work, we mainly tested the tool with both T1w and T2w, so that could be an issue (i.e. that we have a bug in the T1w only processing).

Also your json has a section entitled "scripts". That is not needed. Can you delete that entire section

And finally, your python3 binary does not have a path in the json file, so that needs to be adjusted.

Hope some of this helps
Martin

Originally posted by cccoleman:

Your last message was extremely helpful, and we're now able to run the bulk of the script as expected.  Fixed json file attached.

We are still getting errors, however, and no QCistern final output file.      

The first error, I'm pretty sure, is just the false alarm from ABC.

While running : /home/cory/anaconda3/bin/python3
Following error(s) occured :
While running : /data/proj/app/pipeline/eacsf/Auto_EACSF_v1.4.1/bin/ABC_CLI /data/proj/app/pipeline/eacsf/Auto_EACSF_v1.4.1/bin/ABC_CLI /data/proj/app/pipeline/eacsf/Outputs/100604-200_1/ABCparam.xml Following error(s) occured : Run without any arguments to see command line options
Failed opening /data/proj/app/pipeline/eacsf/Outputs/100604-200_1/ABC_Segmentation/100604-200_1_OwlOrient_to_template_EMS.affine

The next error, however, appears to be more substantial: 

While running : /home/cory/anaconda3/bin/python3
Following error(s) occured :
While running : /data/proj/app/pipeline/eacsf/Auto_EACSF_v1.4.1/bin/ImageMath /data/proj/app/pipeline/eacsf/Auto_EACSF_v1.4.1/bin/ImageMath
   /data/proj/app/pipeline/eacsf/Outputs/100604-200_1/ABC_Segmentation/100604-200_1_OwlOrient.nii_stripped_labels_EMS.nrrd
   -mul /data/proj/app/pipeline/eacsf/Outputs/100604-200_1/VentricleMasking/Vent_CSF-INVMask_Registered.nrrd
   -outfile /data/proj/app/pipeline/eacsf/Outputs/100604-200_1/VentricleMasking/100604-200_1_OwlOrient.nii_stripped_EMS_withoutVent.nrrd
Following error(s) occured :
   terminate called after throwing an instance of 'itk::ImageFileReaderException' what(): /work/alemaout/install/include/ITK-4.13/itkImageFileReader.hxx:143:
  Could not create IO object for reading file /data/proj/app/pipeline/eacsf/Outputs/100604-200_1/ABC_Segmentation/100604-200_1_OwlOrient.nii_stripped_labels_EMS.nrrd The file doesn't exist.
Filename = /data/proj/app/pipeline/eacsf/Outputs/100604-200_1/ABC_Segmentation/100604-200_1_OwlOrient.nii_stripped_labels_EMS.nrrd Subject specific ventricle mask is not a file and cannot be applied

Twenty-seven errors follow in the error log, but presumably they all go back to a failure to read the "_stripped_labels_EMS.nrrd" file. 
We _do_ have an image in the ABC_Segmentation folder called 100604-200_1_OwlOrient_labels_EMS.nrrd which is skullstripped..

Any idea what's going astray?

Originally posted by cccoleman: I figured out what was going on here.  Our T1 image has an extension of ".nii.gz". 
This leads to a skullstripped file ending in ".nii_stripped.nrrd"
It looks like that ".nii" in the stripped filename  is confusing ABC, leading it to generate a label file ending in   "_labels_EMS.nrrd" instead of "_stripped_labels_EMS.nrrd"

Simply unzipping the file before we begin solves this problem.  

Looks like things are running smoothly.   For future users visiting this thread, my batch script is attached.

Great! This is clearly a bug in Auto-EACSF in that (could be an issue of ABC, rather than Auto-EACSF). 

The issue is that rather than removing all extensions from a filename to generate the output filename (image.nii.gz => image), here only one extension is removed (image.nii.gz => image.nii) and then downstream in the workflow other tools do not find the expected output. This is probably due to that my lab prefers the use of NRRD (.nrrd) and thus there is only one extension. 

So, your work-around is great, but we need to fix this. I added this as an issue/bug to the tracker https://github.com/NIRALUser/auto_EACSF/issues/8

Martin
Originally posted by cccoleman: