open-discussion
open-discussion > RE: AAAS: Your Paper MUST include Data and Code
Mar 13, 2011 09:03 PM | Matthew Brett
RE: AAAS: Your Paper MUST include Data and Code
Originally posted by Luis Ibanez:
As maintainers of ITK, we are in the business of working with the ITK community on taking published papers and converting them in usable code (tested, multiplatform, open source). In that process of translating "papers" into "real code", we have seen many skeletons in many closets..., and many emperors without clothes. The lack of full specification of parameters and the absence of the "real recipe for use" is blatant in most papers. Curiously this is particularly done when the operation is deemed to be too simple (despite of being essential). For example, in one case it took us several months to realize that the authors have taken all of their input data and renormalized their intensities before feeding it into the processing pipeline described in the paper. A trivial step, that made a huge difference in the results, but was not mentioned in the paper. Authors (in the effort of presenting their papers as "scientific" tend to overlook the pieces of the work that do not look "complicated" enough or "smart" enough, hence their attachment to anything that looks like a differential equation...)
As you can see from this thread, one of the primary problems is that most people believe that the system as it is at the moment is more or less OK.
I'm sure that close investigation of workflow in individual labs would reveal just the kind of waste that you are describing. Unfortunately, just because this kind of waste is so ordinary, it's often difficult for people to see that is it a problem.
I wonder if you have some way of documenting for each algorithm what you had to do to replicate it as a record? I think this is one of those cases where some well-described evidence will help remind us that it doesn't have to be the way that it is.
As maintainers of ITK, we are in the business of working with the ITK community on taking published papers and converting them in usable code (tested, multiplatform, open source). In that process of translating "papers" into "real code", we have seen many skeletons in many closets..., and many emperors without clothes. The lack of full specification of parameters and the absence of the "real recipe for use" is blatant in most papers. Curiously this is particularly done when the operation is deemed to be too simple (despite of being essential). For example, in one case it took us several months to realize that the authors have taken all of their input data and renormalized their intensities before feeding it into the processing pipeline described in the paper. A trivial step, that made a huge difference in the results, but was not mentioned in the paper. Authors (in the effort of presenting their papers as "scientific" tend to overlook the pieces of the work that do not look "complicated" enough or "smart" enough, hence their attachment to anything that looks like a differential equation...)
As you can see from this thread, one of the primary problems is that most people believe that the system as it is at the moment is more or less OK.
I'm sure that close investigation of workflow in individual labs would reveal just the kind of waste that you are describing. Unfortunately, just because this kind of waste is so ordinary, it's often difficult for people to see that is it a problem.
I wonder if you have some way of documenting for each algorithm what you had to do to replicate it as a record? I think this is one of those cases where some well-described evidence will help remind us that it doesn't have to be the way that it is.
Threaded View
| Title | Author | Date |
|---|---|---|
| Luis Ibanez | Mar 10, 2011 | |
| hongtu zhu | Mar 13, 2011 | |
| Luis Ibanez | Mar 13, 2011 | |
| Matthew Brett | Mar 13, 2011 | |
| Isaiah Norton | Mar 13, 2011 | |
| Torsten Rohlfing | Mar 10, 2011 | |
| Luis Ibanez | Mar 11, 2011 | |
| Daniel Kimberg | Mar 10, 2011 | |
| Cinly Ooi | Mar 10, 2011 | |
| Torsten Rohlfing | Mar 10, 2011 | |
| Cinly Ooi | Mar 10, 2011 | |
| Torsten Rohlfing | Mar 10, 2011 | |
| Cinly Ooi | Mar 10, 2011 | |
| Torsten Rohlfing | Mar 10, 2011 | |
| Cinly Ooi | Mar 10, 2011 | |
| Matthew Brett | Mar 10, 2011 | |
| Pierre Bellec | Mar 10, 2011 | |
| Luis Ibanez | Mar 11, 2011 | |
| Matthew Brett | Mar 10, 2011 | |
| Cinly Ooi | Mar 10, 2011 | |
| Cinly Ooi | Mar 10, 2011 | |
| Torsten Rohlfing | Mar 10, 2011 | |
| Daniel Kimberg | Mar 10, 2011 | |
| Cinly Ooi | Mar 10, 2011 | |