JIST: Java Image Science Toolkit News

Access Private Amazon Clouds with JIST – Powered by NITRC-CE

Bennett Landman — Fri, 26 Feb 2016 1:46:44 GMT

In work published in SPIE-MI 2016 (lead author Shunxing Bao), JIST now supports distribution of processing to Amazon Web Services (AWS) with virtual machines powered by NITRC-CE. All cloud services, including virtual machine spin up, software initialization, data transfer, process control, and cloud shutdown, are controlled programmatically within JIST. Just enter your AWS and let JIST do it’s thing.

For information on AWS and HIPAA, please see: https://aws.amazon.com/compliance/hipaa-compliance/

Abstract:

Adopting high performance cloud computing for medical image processing is a popular trend given the pressing needs of large studies. Amazon Web Services (AWS) provide reliable, on-demand, and inexpensive cloud computing services. Our research objective is to implement an affordable, scalable and easy-to-use AWS framework for the Java Image Science Toolkit (JIST). JIST is a plugin for Medical-Image Processing, Analysis, and Visualization (MIPAV) that provides a graphical pipeline implementation allowing users to quickly test and develop pipelines. JIST is DRMAA-compliant allowing it to run on portable batch system grids. However, as new processing methods are implemented and developed, memory may often be a bottleneck for not only lab computers, but also possibly some local grids. Integrating JIST with the AWS cloud alleviates these possible restrictions and does not require users to have deep knowledge of programming in Java. Workflow definition/management and cloud configurations are two key challenges in this research. Using a simple unified control panel, users have the ability to set the numbers of nodes and select from a variety of pre-configured AWS EC2 nodes with different numbers of processors and memory storage. Intuitively, we configured Amazon S3 storage to be mounted by pay-for-use Amazon EC2 instances. Hence, S3 storage is recognized as a shared cloud resource. The Amazon EC2 instances provide pre-installs of all necessary packages to run JIST. This work presents an implementation that facilitates the integration of JIST with AWS. We describe the theoretical cost/benefit formulae to decide between local serial execution versus cloud computing and apply this analysis to an empirical diffusion tensor imaging pipeline.

Merry JISTmas and JIST 3.2

Stephen Damon — Tue, 16 Dec 2014 1:55:42 GMT

Merry JISTmas fellow JISTers!

Please see the minor release (JIST 3.2)

Since our previous release, we have fixed the following bugs:

1. The “Submit a Bug” now works correctly with all special characters.

2. XML returned by using the CLI interface now includes input/output for the directory channel which should help expedite integration with NiPy (thanks Chris!)

3. CLI STDOUT and STDERR are fixed to more logically represent what “went wrong” when you tried to run your module. Many times if a compulsory input was not passed to the CLI a generic error message about the module failing in an ungraceful manor would be returned. This generally reflects a developer error. However, in this case, this is not a developer error, but a parser error. Now it will return the required argument that you did not input.

4. Fixed many modules that would not run from CLI due to various parse errors which were all linked to having a special character for the argument.

5. Fixed a minor bug related to CLI which caused numerous modules to fail even though the args were correct. This was related to how the authors list was parsed.

New Features:

1. New Module under the MASI/External/Solar directory to run a polygenic model using SOLAR-ECLIPSE imaging genetics software: http://www.nitrc.org/projects/se_linux. Note that the solar executable will need to be in your path.

2. We have expanded on our previous methods to implement various external CLI programs in the JIST environment. Utilizing a simple JSON framework, you are able to auto-generate modules for external CLI programs. Please see the wiki here: for a quick overview: http://www.nitrc.org/plugins/mwiki/index.php/jist:Jist_External_Programs

3. XML returned by the CLI is automatically tested to ensure that it is well-formed. If it is not, the first line of the output will be an error message. The Slicer XML will still follow.

Our next plans are to implement multi-shot DTI reconstruction and ASL processing modules.

Best wishes and have a safe and happy holiday,
Benentt and Steve @ Vanderbilt

JIST Release 3.1 - Exciting news on new JIST developments.

Bennett Landman — Thu, 18 Sep 2014 5:27:46 GMT

Fellow JIST’ers,

Please see that we have released a minor version (3.1). We plan to release regular updates from here on out as Steve Damon has joined our team. Our immediate plans are to clean up bugs and ramp up support for advanced diffusion weighted MRI methods (HARDI, compressed sensing, etc.).

The new release fixes: (1) the bug reporting tool, (2) XML parsing errors, and all bugs otherwise addressed in the last 18 months.

Cheers,
Bennett and Steve @ Vanderbilt

PyJIST - Now with pip / easy_install

Bennett Landman — Wed, 09 Apr 2014 6:05:39 GMT

https://pypi.python.org/pypi/PyJIST/0.1

PyJIST

Bennett Landman — Mon, 07 Apr 2014 3:52:34 GMT

PyJIST is in pretty good shape. See the tutorials below for installation instructions and common use cases.

https://www.nitrc.org/plugins/mwiki/index.php/jist:PyJIST

JIST 3.0 Release

Bennett Landman — Fri, 06 Sep 2013 2:18:50 GMT

Good evening,

We have released JIST 3.0 which is compatible with MIPAV 7.0+. I just posted final JIST 2.0 build which are compatible with MIPAV 5.4.X. These are SIGNIFICANT upgrades and the .layout formats are not fully backward compatible. Please read through the changes and consider upgrading.

Best,
Bennett

JIST 3.0 Changelog
Major Changes
• Removed all duplicate/conflicting jar files in JIST / JIST Plugins that conflict with new MIPAV7
o E.g., j3dcore, j3d-core, j3dutils, camino, etc.
• Default preferences have changed
o Filetype -> nii
o Compression -> gzip
o Layout type LayoutXML
o Hypervisor off
• LayoutXML format should be compatible with all layouts and, as far as we know, there is no reason not to use the LayoutXML format for all layouts.
o All .layout specific code will be removed from a future release.
• Several of the “Source” types have changed, specifically, for Volumes, Surfaces, and Files there are three types of sources
o Volume/Surface/File Set
 A set of files/volumes/surfaces specified manually, via text file, or through directory search that are passed individually (one at a time).
o Volume/Surface/File Collection
 A collection of files/volumes/surfaces specified manually, via text file, or through directory search that are passed en masse (as a single collection).
o Volume/Surface/File Collection Set
 A set of file/volume/surface collections that are passed individually and specified via a comma separated text file (a set of collections are passed).
• Added “post-it note” module for adding notes to your layout. This module does not affect the process manager in any way.
• Added ability to change color of modules (right-click on module). This can be useful for large layouts to differentiate between different sections.
Minor Changes/Bug Fixes
• Fixed bug causing GlobalPreferences to ask library to re-build twice.
• Made AlgorithmInformation and all sub-classes implement serializable to avoid no-args constructor errors.
• Removed nearly all “System.out/err” and replaced with JistLogger system.
• All file/volume/surface selections should use the previously used directory.
• Fixed bug related to saving empty layouts.
• Empty layouts no longer ask you to save before exiting.
• Fixed bug submission system for automatically posting bugs to nitrc tracker site (please use!).
• Fixed tons of unnecessary checks that were causing layout to require saving unnecessarily.
• Fixed updater dying when process manager closed (should make process manager much more stable).
• Fixed JGraph bug causing parameters not to update when edges/modules changed/removed/deleted.
• Dramatically simplified code to display input/output parameters (JIST-II stuff).
• Fixed bug causing previous layout list to be read inappropriately.
• All sources/destinations are now presented in a sorted order.
• Updated JISTLayoutAdapter to use better names for input/output parameters
• JISTPluginSelector is now compatible with MIPAV 7.
• Selecting “View in process manager” changes the layout in the process manager if necessary.
• Created a new port color for parameters that are invalid (i.e., empty) but are not mandatory
o Note, this only really matters for files/volumes/surfaces.
• Open/save layout uses previous directory – a directory that is separate from previous file/surface/volume directory).
Known Bugs / Future Changes
• JISTLayoutAdapter needs a little bit of work to make it more useful for simplifying large layouts.
• JISTPluginSelector needs better visualization for several of the input/output parameters.
• The JIST Command line interfaces needs work to simplify the usage process.
• External sources should keep the mandatory settings of their ancestors.
• Avoidance routing (how JGraph selects the path) could be better (less overlapping edges).
• Several known graphics bugs
o In particular, bugs that never actually touch the JIST code but occur when interacting with the layout/process manager.

JIST CVS Cleanup

Bennett Landman — Tue, 09 Jul 2013 8:19:10 GMT

JIST Developers - I am cleaning up "junk" (temp files, module cache files, etc.) that have migrated into the repository. With CVS, I cannot remove the empty directories, so I suggest that you use "cvs update -dP" option to get rid of the cruft. This is in preparation for a full migration to MIPAV 7.0 (hopefully in August 2013).

JISTmas 2012 - Please submit bugs

Bennett Landman — Wed, 19 Dec 2012 2:35:29 GMT

We are planning the 4th annual JISTmas celebration and bug fixing event. Please submit any lingering annoyances, bugs, and feature request to the JIST bug portal on NITRC. We'll do our best to address them.

Cheers,
Bennett

NEW JIST Article - "JIST 2.0"

Bennett Landman — Wed, 05 Sep 2012 4:35:18 GMT

Please see our new InSight article about the new JIST. It describes many of the new features, provides updated navigation, and should help new users get started more quickly. Please use the InSight review system to like (or dislike) the article.

http://www.insight-journal.org/browse/publication/874

NEW JIST Hypervisor - Resource Estimation

Bennett Landman — Wed, 05 Sep 2012 2:45:24 GMT

Merry Band of JIST Users,

I am thrilled to announce that our hypervisor is now estimating resource (memory/cpu) requirements. If you download any new JIST bundle, the hypervisor will be on by default. This will send our server (by default, but you can run your own hypervisor if you like – it’s all politely virtualized) the name of the module, size information (# bytes) for each input parameter, grid status, and your MAC address. Based on this information, we use a regression tree to predict memory and CPU (in a few ms). When your module completes, the actual memory/CPU usage are sent back to the server.

In version 1, our data are rather limited so our regression tree is simple. We’ve found it converges fairly well once you run a module 10-20 times on similar machines. Please let me know if you experience any issues. All code is committed and public. I am editing the paper for submission now. I’m happy to send a confidential pre-submission copy upon request if ya’ll want to prod the innards.

Cheers,
Bennett