help
help > Inconsistencies between CONN NBS and NBS tool
Aug 5, 2017 01:08 AM | Alfonso Nieto-Castanon - Boston University
Inconsistencies between CONN NBS and NBS tool
Dear Rob
My guess on the first difference (190 vs. 95 suprathreshold connections using the same threshold) is that it may probably be due to CONN simply counting twice each edge when computing the "network size" measure (CONN does this just because it counts individual elements of the adjacency matrix, even if that matrix is symmetrical by design, just for consistency with future implementations on directional graphs). If you display the adjacency matrix in NBS you should find that it is exactly the same as the one displayed in CONN's second-level ROI-to-ROI results under the same thresholding conditions (if it is not please check whether perhaps you may be using two-sided thresholds in CONN and one-sided thresholds in NBS, or something along those lines)
Regarding the second difference, NBS uses permutation for two-sample t-tests and randomization for one-sample t-tests, while CONN uses randomization of residuals in all cases, so there may be minor differences expected between the statistics derived from these different methods. That said, the difference in p-values that you describe seems too large to be due to these differences. One potential issue to explore is which particular p-value you are comparing in CONN: CONN reports uncorrected p-values "p-unc", family-wise corrected p-values "p-FWE", and False Discovery Rate- corrected p-values "p-FDR". The p-value reported in NBS corresponds to the one named "p-FWE" in CONN (if you are comparing the p-value in NBS to the "p-unc" value in CONN that may explain the liberal values observed)
Hope this helps
Alfonso
Originally posted by Rob McCutcheon:
My guess on the first difference (190 vs. 95 suprathreshold connections using the same threshold) is that it may probably be due to CONN simply counting twice each edge when computing the "network size" measure (CONN does this just because it counts individual elements of the adjacency matrix, even if that matrix is symmetrical by design, just for consistency with future implementations on directional graphs). If you display the adjacency matrix in NBS you should find that it is exactly the same as the one displayed in CONN's second-level ROI-to-ROI results under the same thresholding conditions (if it is not please check whether perhaps you may be using two-sided thresholds in CONN and one-sided thresholds in NBS, or something along those lines)
Regarding the second difference, NBS uses permutation for two-sample t-tests and randomization for one-sample t-tests, while CONN uses randomization of residuals in all cases, so there may be minor differences expected between the statistics derived from these different methods. That said, the difference in p-values that you describe seems too large to be due to these differences. One potential issue to explore is which particular p-value you are comparing in CONN: CONN reports uncorrected p-values "p-unc", family-wise corrected p-values "p-FWE", and False Discovery Rate- corrected p-values "p-FDR". The p-value reported in NBS corresponds to the one named "p-FWE" in CONN (if you are comparing the p-value in NBS to the "p-unc" value in CONN that may explain the liberal values observed)
Hope this helps
Alfonso
Originally posted by Rob McCutcheon:
Dear Alfonso and other CONN users,
I have been comparing NBS results obtained using CONN and those obtained using the NBS toolbox (https://www.nitrc.org/projects/nbs/), and find the results with the two methods are somewhat different. I am using the same connectivity matrices and design matrices in both.
It seems that using the same initial threshold in both leads to a larger maximum network size in CONN (e.g. using p<0.05 one sided in CONN, and t<1.72 in NBS gives networks of 190 and 95 connections respectively). Also regardless of the edge level threshold, when calculating the p value for the network CONN seems to be less stringent than NBS (e.g. the 190 edge network has a p value of 0.0037 in CONN, while the 95 edge NBS one has a P value of 0.06. If one reduces the edge threshold to get a 190 edge network in NBS its p value is 0.09).
I was wondering does CONN use exactly the same methods as NBS, or is there a difference that explains this discrepancy?
Many thanks
Rob
I have been comparing NBS results obtained using CONN and those obtained using the NBS toolbox (https://www.nitrc.org/projects/nbs/), and find the results with the two methods are somewhat different. I am using the same connectivity matrices and design matrices in both.
It seems that using the same initial threshold in both leads to a larger maximum network size in CONN (e.g. using p<0.05 one sided in CONN, and t<1.72 in NBS gives networks of 190 and 95 connections respectively). Also regardless of the edge level threshold, when calculating the p value for the network CONN seems to be less stringent than NBS (e.g. the 190 edge network has a p value of 0.0037 in CONN, while the 95 edge NBS one has a P value of 0.06. If one reduces the edge threshold to get a 190 edge network in NBS its p value is 0.09).
I was wondering does CONN use exactly the same methods as NBS, or is there a difference that explains this discrepancy?
Many thanks
Rob
Threaded View
| Title | Author | Date |
|---|---|---|
| Rob McCutcheon | Jul 31, 2017 | |
| Alfonso Nieto-Castanon | Aug 5, 2017 | |
| Rob McCutcheon | Aug 7, 2017 | |