It’s not a good idea to copy someone else’s article, only changing the author list, and submitting it to another conference/journal. I’m glad to see IEEE takes appropriate measures when such a thing happens. And I am still amazed it actually happened…

More information can be found in the call for papers.

Machine Learning for Signal Processing (MLSP) TC Announces the Winners of the 6th Annual Data Analysis Competition

See here for more info.

As it says on the site, this workshop was on the following topic, which very well described to me both the content and the spirit of the workshop: “Modeling and simulation are essential tools in systems biology and many other branches of science. This workshop is an invitation to step back from the day-to-day struggle with our simulations and to reflect about the nature of modeling and its relation to simulation: How do modeling and simulation contribute to the development of knowledge? Is a simulation per se a valid scientific experiment?”

Both the speakers and the audience consisted of people with a very diverse background, ranging from physicists, chemists and engineers (like me) all the way to philosophers in metaphysics. This resulted in often very enthusiast and interesting discussions. It was also very interesting for me to see how scientists in neuroscience struggle with similar issues as me, and to see how they approach things. I learned a lot of new things, some of which will pop up in separate posts on this blog in the future. Stay tuned!

We’ve only got one planet Earth to validate models on. And it takes soooo long to check whether a model is correct, that we’d better be open about it, collaborate, check each other’s assumptions, and make sure it’s the best model we can make!

For some more discussion on the recent climate study scandal and reproducible research, see also Victoria Stodden’s blog (or also here).

The tool is also described in the following paper:

D. Ramage and A. J. Oliner, RA: ResearchAssistant for the Computational Sciences, Workshop on Experimental Computer Science (ExpCS), June 2007.

I recently experienced it again in some of my work. In the stress of preparing a publication for a submission deadline, it is very challenging to take the (precious) time to verify all of the results once more and make sure all the results are perfectly reproducible. A result or figure so easily slips in for which the exact parameter settings have not been checked or written down…

It also shows something about the amazing power of open source software developed by a community of authors (and typically users at the same time).

reproducibleresearch.org is an overview website by John D. Cook containing links to reproducible research projects, articles about the topics, and relevant tools. It also contains a blog about reproducible ideas.

reproducibleresearch.com is owned by the people at Blue Reference, who created Inference for Office, a commercial tool to perform reproducible research from within Microsoft Office.

reproducibility.org is used by Sergey Fomel and his colleagues as home for their Madagascar open source package for reproducible research experiments.

reproducible.org is a reproducible research archive maintained by R. Peng at Johns Hopkins School, where the goal is to host a place for reproducible research packages.

Quite a range of domain names containing the word “reproducible” (or a derivative), if you ask me! And then I didn’t even start about the Open Research or Research 2.0 sites. Let’s hope this also means that research itself will soon see a big boost in reproducibility!

Why does this worry me so much?

I think web pages are the easiest, fastest, and most practical way of making reproducible research and articles available to colleagues. But if these web pages have a short lifetime, it also makes that research reproducible (by others) for only a short time. We conducted a study last year about reproducibility of articles, and when going through the obtained URLs now, I can already see quite a few that don’t work anymore. No way to retrieve the information…

And the worst is: what can we do about this? I do believe institutional repositories, by their larger scale nature, and more long-term support, will have a longer lifetime. But those are often not very flexible to allow for adding code, data, or other metadata. Each paper these days also has a unique identifier, its DOI. This allows one to track a paper using something more permanent than a URL. So a DOI can resolve the site redesign problem. But if a page disappears, it is still gone. And which individual can/wants to guarantee that he will maintain a page forever?

From some recent discussions, I get the impression that quite a few people would not mind uploading their code and data to a centralized service. But currently I don’t feel like setting such a service up, as it is a lot of work, and I do not think I want to be responsible for keeping it alive “forever”, taking care of back-up, etc. So I think such a service should definitely not be an individual initiative, but be backed by an institution. And even then, what will the lifetime be?

Any better solutions?