A very nice initiative, which should get a few problems out of the world. However, I am not sure how that is supposed to work in practice. Does that mean that we should soon add an ORCID number (without typos) below the title and the author name? And cite works by citing the ORCID and the DOI (digital object identifier)? And will we write these numbers with less errors than the author names now?

It makes me indeed think of that other unique number: DOI, which was introduced to uniquely identify a document (publication, for as far as I have seen them). I’ve seen it for some time now when I look up articles, and I have no doubt it uniquely identifies those articles, but what is it used for? Maybe they have their use… but I haven’t seen it yet.

People who do know of practical cases where the DOI is used, feel free to comment! (others too, of course)

In academia, the game is all about publishing, and getting others to cite your articles. And I guess, to a certain extent, article counts and citation counts indeed give a measure of someone’s work. Until you start overfitting your system. But anyway, that’s another story…

So, to get back to my story, citations measure the quality of a work. In general, people try to be correct, and cite the researchers that started a certain work. And then, once work gets really well known, it’s somehow not cited anymore. So the ultimate reward for good work is not to be cited anymore. Or did you cite a reference when writing about the Fourier transform, wavelets, least squares or filtering? For some of them I don’t even know who it was, but someone must have invented them…

I recently experienced it again in some of my work. In the stress of preparing a publication for a submission deadline, it is very challenging to take the (precious) time to verify all of the results once more and make sure all the results are perfectly reproducible. A result or figure so easily slips in for which the exact parameter settings have not been checked or written down…

Repeatability Criteria: The CVPR 2010 reviewer form will include the following additional criteria, with rating and associated comment field: “Are there sufficient algorithmic and experimental details and available datasets that a graduate student could replicate the experiments in the paper? Alternatively, will a reference implementation be provided?”. During paper registration, authors will be asked to answer the following two checkbox questions: “1. Are the datasets used in this paper already publicly available, or will they be made available for research use at the time of submission of the final camera-ready version of the paper (if accepted)? 2. Will a reference implementation adequate to replicate results in the paper be made publicly available (if accepted)?” If either these boxes are checked, the authors should specify in the submitted paper the scope of such datasets and/or implementations so that the reviewers can judge the merit of that aspect of the submission’s contribution. The Program Chairs realize that for certain CVPR subfields providing such datasets, implementations, or detailed specification is impractical, but in other areas it is reasonable and sometimes even standard, so on balance repeatability is a relevant criteria for reviewer consideration. “N.A.” will be an available reviewer score for this field, as it is for other fields.

Very exciting developments!

Keith Baggerly and Kevin Coombes have just published a great paper about forensic bioinformatics: “Deriving chemosensitivity from cell lines: forensic bioinformatics and reproducible research in high-throughput biology.” The article will appear in the next issue of Annals of Applied Statistics and is available here.

The blog post Make up your own rules of probability discusses a couple of the innovative rules of probability Baggerly and Coombes discovered while reverse engineering bioinformatics articles.

Note that the Reproducible Ideas blog is winding down. I’m in the process of handing over the reproducibleresearch.org domain to the owners of reproducibleresearch.net. Eventually the .net site will move over to the .org name. ]]> http://www.reproducibleresearch.net/blog/2009/10/13/new-paper-in-forensic-bioinformatics/feed/ 1 http://www.reproducibleresearch.net/blog/2009/09/16/research-data-who-cares/ http://www.reproducibleresearch.net/blog/2009/09/16/research-data-who-cares/#comments Wed, 16 Sep 2009 19:17:10 +0000 Patrick http://www.reproducibleresearch.net/blog/?p=143 Today I attended the mini-symposium “Research data. Who cares?“, organized by Leon Osinski at TU Eindhoven. The symposium was organized at the startup of the 3TU.Data Centre, an organization by the 3 Dutch Technical Universities’ libraries for the storage, sharing and preservation of research data. I gave a presentation there about my experiences with reproducible research.

Another presentation there that I liked a lot was given by Pieter Van Gorp, about “SHARE” (Sharing Hosted Autonomous Research Environments). This is an exciting setup he developed. It allows a researcher to put his research results in a safe and well-controlled environment on a virtual machine. Other researchers can then login to that virtual machine, and reproduce the results in exactly the same environment as used by the author, as if they are working on the author’s machine. While I am not entirely sure yet about its advantage for my typical Matlab scripts (that do not require complex installations), it is certainly of tremendous help when presenting more complex tools and results. Seems like a great step towards one-click reproducing of results, and I am certainly going to try it out!

First of all, I’d like to welcome all readers from John Cook’s Reproducible Ideas blog at reproducibleresearch.org. I hope I’ll be able to live up to the standards John has set.

And of course also welcome to reproducibleresearch.net readers, and readers that join from pixeltje.be.

This blog has been created when merging reproducibleresearch.net and reproducibleresearch.org. I’ve taken this occasion to merge John’s posts (and thus keep those links valid) with my earlier posts on two different sites: blog.epfl.ch/rr and blog.pixeltje.be that are related to reproducible research.

I hope I’ll be able to write many interesting posts here. Please feel free to comment on any of my writings! If you would be interested in writing guest posts, please let me know!

K. Price, Anything You Can Do, I Can Do Better (No You Can’t)…, Computer Vision, Graphics, and Image Processing, Vol. 36, pp. 387-391, 1986, doi:10.1016/0734-189X(86)90083-6.

Abstract: Computer vision suffers from an overload of written information but a dearth of good evaluations and comparisons. This paper discusses why some of the problems arise and offers some guidelines we should all follow.

Very nice reading material, and (although I know these ideas are around for quite some time already) I was amazed to see so many parallels to our recent IEEE Signal Processing Magazine paper, already in this paper by Price from 1986. That’s more than 20 years ago! Price talks about the reproducibility problems in computer vision and image processing, writing we should “stand on other’s shoulders, not on other’s toes”. He also did a study on reproducibility of a set of about 42 papers, verifying the size of the dataset and clarity of the problem statement. Price concludes as follows: “Researchers should make the effort to obtain implementations of other researchers’ systems so that we can better understand the limitations of our own work.”

Again, interesting to see how these issues and worries have been around for more than 20 years in the field of image processing. It’s about time to drastically improve our standards, I think!

I would really recommend this article to anyone interested in issues around reproducible research.

A. J. Rossini and F. Leisch, Literate statistical practice, UW Biostatistics Working Paper Series 194, University of Washington, WA, USA, 2003.

Although I am not a statistician, this was a very interesting paper to me. It gives a nice description of a possible literate programming approach in statistics. The authors propose a very versatile type of document combining documentation and code/statistical analyses, interweaved as in the original description of literate programming by Knuth. From this versatile document, which contains a complete description of the research work, multiple reports can be extracted, such as an article, an internal report, an overview of the various analyses that were performed, etc.