Hi Zhoubing -

To be very clear: I have no experience whatsoever with IRTK.

But generally, when registering thresholded data, I would be wonder whether both images have been thresholded, or only either source or target. Also, what happens with the masked-out pixels? Are they treated as missing, or taken at face value (e.g., zero?)

Interestingly, from the "rreg" documentation:

Parameter name : "Padding value"

Parameter value :
A number used to determine if voxels in the target image are used
during registration. All voxels with an intensity less than or equal to
the padding value are ignored.

If you were using this option, try doing without it. In your case, setting the non-bone pixels to zero and using them in the registration as such will likely help the alignment. More so than ignoring the data at least.

That's pretty much all I can think of. Like I said, no experience specifically with IRTK here. Or with niftyreg or flirt, to be honest ;)

Good luck!

Torsten