Comments for Monitoring, Modeling, and Memory

Comment on Science special issue on “Dealing with Data” by the genuine articles

the genuine articles — Mon, 06 Feb 2012 00:43:57 +0000

Proves My Point…

The discussion was wrapped up after I discovered this article yesterday…

Comment on New climate variability results: models and data, again by Climate smackdown: models vs. data? « Monitoring, Modeling, and Memory

Climate smackdown: models vs. data? « Monitoring, Modeling, and Memory — Tue, 22 Mar 2011 13:02:54 +0000

[...] Just published a long post on A Vast Machine about the recent Nature and QJRMS results, about which I’ve blogged here earlier. [...]

Comment on New climate variability results: models and data, again by archer

archer — Fri, 18 Feb 2011 17:48:46 +0000

So one straightforward observation is cherry-picking of model results by climate skeptics. They are willing to accept the model-produced data in the reanalysis, but not the model-produced data for the extreme events. Charitably, this could be a misunderstanding of what data really are, and how models work in climate science. Or it could just be a matter of selective vision. This also discredits the myth that climate scientists report selectively or are stuck in group-think.

One other thought: I wonder if the re-analysis models are calibrated based on (the more comprehensive) data from the latter part of the 20th century, when greenhouse gases were already beginning to have an impact. Then when the models reconstruct missing data for the earlier years they generate data that reproduces greenhouse-based effects even though they may not have existed. So since the NAO changes in a certain way now, the model would be biased to reproduce that and is gently tugged by the sparse data of the early years. Perhaps the few data points of a different NAO pattern are not enough to pull it off of the (modern) baseline. Am I thinking about this correctly?

Comment on Replicating results with published data + code: very hard to do… by michaeldcohen

michaeldcohen — Fri, 11 Feb 2011 23:18:14 +0000

A key observation in several comments, and implicit in the replications studies, is that quite a few publications, maybe even a large majority, engender no demand for replicability. Given that – as the other remarks point out – it’s costly and likely to remain so, perhaps insisting that all submissions, or funded research projects, provide replicability may be too high a tax on everyone, and publishers and funders should instead put resources into making it cheaper and/or more reliable for researchers who might do it voluntarily — and perhaps make it more reputation-costly for those who decline.

“The data are publicly available” could then be a distinction some authors might do extra work to include in their paper. Supplying some high quality but optional standards and services might cut the costs a bit. And an annual list (along with the “thanks to our reviewers”) of “our authors [grantees] who are reported to have met [or declined?] reasonable replication requests” might have some impact as well.

Comment on Replicating results with published data + code: very hard to do… by archer

archer — Fri, 11 Feb 2011 21:43:29 +0000

This reminds me of the idea of having the whole paper compile via one script. Just typing “make” and having it run all the data analysis, produce graphs, and compile the PDF from .tex. Should work in principle but requires a lot of work – even an impractical standard for most papers. And still doesn’t address problems like that of tools changing after a few years. There’s even talk of needing to have virtual machines to ensure that we can run old tools after the hardware changes, which can get to be a problem if you want the data available for analysis 10-20 years down the road.

Comment on Replicating results with published data + code: very hard to do… by Will Riley

Will Riley — Fri, 11 Feb 2011 19:07:22 +0000

I think automatic software testing practices would provide a useful lens through which to view the problem of reproducibility. In software testing, one wants to assert that certain conditions apply before running a test. Test fixtures are used to make sure the testing environment is fixed (same dataset, same operating system, dependency checks etc.) before the main processing begins. I’m not an expert in software testing, but I think there would be many parallel concepts and concerns. For example, one might have a suite of tests to run to make sure the software is producing the same outputs as the scientists claim in their paper. Scientists could write additional tests with known results to give them greater confidence that the analysis code is working properly.

Comment on Replicating results with published data + code: very hard to do… by Jeff MacKie-Mason

Jeff MacKie-Mason — Fri, 11 Feb 2011 17:56:26 +0000

Nice post Paul (and nice work by Howison. Most of my early research was empirical (economics). Some of it attracted enough attention that I got a number of years to provide the data and code over the years. It was a fair bit of work, but the culture in which I was trained expected and I provided it (besides, it’s nice to have someone do follow-on work that cites you Usually I had to correspond multiple times with the other user to explain my semi-coherently commented code, or clarify data transformations I did. I’m pleased to be able to say that we always succeeded in enabling replication and reuse. But if back then journals had required me to deposite the code and data, what I originally turned in would indeed have the same problems JMCB found: until someone tried to actually use it I surely would not have provided adequate metadata or code documentation.

Comment on Replicating results with published data + code: very hard to do… by [Taxacom]: Data persistence « Monitoring, Modeling, and Memory

[Taxacom]: Data persistence « Monitoring, Modeling, and Memory — Fri, 11 Feb 2011 16:49:08 +0000

[...] points here, following up on another post from earlier today. First, DPHEP is upping the metadata ante considerably by requiring not just publication of code, [...]

Comment on Time by James Howison

James Howison — Fri, 11 Feb 2011 16:32:16 +0000

To this I’d add (all far more prosaic, but perhaps of interest nontheless):

Bluedorn, A. C. and Standifer, R. L. (2006). Time and the temporal imagination. Academy of Management Learning and Education, 5:196–206. Relates perceptions of a time-scape within organizations (and teams within organizations in a nested way) to their organizational structure and performance.

Gersick, C. J. G. (1989). Marking time: Predictable transitions in task groups. Academy of Management Journal, 32(2):274–309. A classic Org. Science result showing how “time remaining” in group work shapes types of activities.

Comment on Replicating results with published data + code: very hard to do… by James Howison

James Howison — Fri, 11 Feb 2011 16:05:13 +0000

Great post Paul. Far too often we treat software (and data) as an artifact that goes to accomplishing aims (reproducibility, knowledge diffusion, innovation etc). Artifacts, particularly information artifacts, are great, since they appear to require no additional work to accomplish these aims. Yet time and time again we learn that these aims are the result of (artifact supported) practice. And that practice is virtually invisible to (or actively undermined by) our incentive systems, whether those be in academia or in other types of organizations (as the IS literature has found for “Knowledge Management” systems, see [1-3]).

[1]: McDermott, R. (1999). Why Information Technology Inspired but Cannot Deliver
Knowledge Management. *California Management Review, 41*(4), 103-117.

[2]: Wasko, M. M., & Faraj, S. (2000). “It is What One Does”: Why People
Participate and Help Others in Electronic Communities of Practice. *Journal
of Strategic Information Systems, 9*, 155-173.

[3]: Kankanhalli, A., Tan, B. C. Y., & Wei, K. (2005). Contributing Knowledge to
Electronic Knowledge Repositories: An Empirical Investigation. *MIS
Quarterly, 29*(1), 113-143.