“Say, doesn’t that look remarkably similar to something else that I saw…”
Compiling data for a scientific publication is challenging. What to include and exclude is a tacit consideration for every scientist. Before you might get angry that scientists are ‘fiddling with the data’ or ‘only showing you what they want’, please consider the following: you yourself are an information gathering and filtering machine. In fact, you harvest information involuntarily all the time. However, your evolutionary self has learnt to filter and select information and pitch it according to your need. When you walk in to a room you don’t actively notice that the walls are flat or that you are wearing shoes; but if the walls were to spontaneously turn green or your shoes disappear, you’d notice it right away. Is that a fire on my desk?!
When a scientist presents data, we hope they do so openly and honestly, and normally expect a broad overview to guide the reader as to where this data should be placed in the grand scheme of things. There will inevitably be an element of self-selection in any scientist’s data; this is unavoidable and yet totally understandable. Include everything and any scientific work will become unreadable, boring and uninformative. Include too little and you run the risk of not making your point clear enough. Finding that balance is an art-form. The peer review process aspires to produce a higher publication standard. In reality, it can only do so up to a certain point. However, once a publication is out there in cyber space, the general public can always weigh in to scrutinize its finer details. Which brings me to the purpose of this post...
I recently noticed a peculiarity in a scientific paper *. I initially identified it as an error, because I had seen exactly the same graph elsewhere. After a quick check of the old publication, I noticed that these ‘new’ data points were exactly the same as the old ones, although they were coloured differently, the x-axis scale was changed and some of the data points were missing. See for yourselves:
Compiling data for a scientific publication is challenging. What to include and exclude is a tacit consideration for every scientist. Before you might get angry that scientists are ‘fiddling with the data’ or ‘only showing you what they want’, please consider the following: you yourself are an information gathering and filtering machine. In fact, you harvest information involuntarily all the time. However, your evolutionary self has learnt to filter and select information and pitch it according to your need. When you walk in to a room you don’t actively notice that the walls are flat or that you are wearing shoes; but if the walls were to spontaneously turn green or your shoes disappear, you’d notice it right away. Is that a fire on my desk?!
When a scientist presents data, we hope they do so openly and honestly, and normally expect a broad overview to guide the reader as to where this data should be placed in the grand scheme of things. There will inevitably be an element of self-selection in any scientist’s data; this is unavoidable and yet totally understandable. Include everything and any scientific work will become unreadable, boring and uninformative. Include too little and you run the risk of not making your point clear enough. Finding that balance is an art-form. The peer review process aspires to produce a higher publication standard. In reality, it can only do so up to a certain point. However, once a publication is out there in cyber space, the general public can always weigh in to scrutinize its finer details. Which brings me to the purpose of this post...
I recently noticed a peculiarity in a scientific paper *. I initially identified it as an error, because I had seen exactly the same graph elsewhere. After a quick check of the old publication, I noticed that these ‘new’ data points were exactly the same as the old ones, although they were coloured differently, the x-axis scale was changed and some of the data points were missing. See for yourselves:
The data in question. The x-axis is different and yet the data looks the same... although with different colours and some points are missing... (I've expanded part of the data with a dotted box, for clarity)
Above are two graphs (1 and 2) from two different publications on different years (I’ve made enlarged dotted boxes for clarity). Interestingly, the publications had some shared authors, some of whom (lets say) weren’t strangers to high profile journal publications. Without going in to the details of what the graphs show, it seemed reasonable that the data was a copy. Given the curious nature of the data, I decided to contact the editor of the journal; especially since changing the x-axis scale could have implications for the interpretation of the data. This wasn't a malicious attack on those authors, it could very well have been an oversight, which it fine. But in the interest of integrity, it was worth at least asking.
Scientists are routinely reminded of the importance of the expected high standards required for publication. For a good reason: to uphold the integrity of skeptical inquiry. I remember attending a lecture on exactly such a topic from the editor-in-chief of one scientific journal, and the message was crystal clear: “no issue is too small, let us know if you have a query”. So that’s what I did.
After some routine preamble to the editor, I signed off my email “… the data looks similar, including the noise, although the x-axis scale differs. I thought it important to point this out.” I was a little nervous because I wanted to maintain anonymity (which they do categorically adhere to). But, being the coy inquirer that I am, I at least wanted someone to check this data anomaly without it being seen as a personal attack on the authors. I can confidently say that the editor's response (up until the eventual correction) was very professional and reassuring.
In one back and forth, the journal editor responded thusly: “… thank you for bringing to our attention this discrepancy. We believe _*journal*_ should be of the highest order of rigor and greatly appreciate people who carefully read our work and call to our attention these issues. If you have any further questions… once again, I want to give you a sincere thank you for giving this manuscript such a careful reading to find this error and bringing it to our attention.”
A few short months later both journals published corrections and the world is a better place.
After note: I would highly encourage any reader of science to be involved in the material that you read (which you undoubtedly do already!). Interact with it somehow. Whether it’s a scientific journal or a newspaper article, many authors are only a click away. Ask them something whether you agree or disagree, or just want clarification. Sometimes when you dig you’ll find enlightenment and a shining ray of optimism. Other times you may find sheer malevolence (honestly) and a stark peak through a door that someone didn’t realise was left ajar **. Either way, there's a chance you could learn something, and that’s not such a bad proposition.
* if you would like to know which articles these are please PM me
** a follow up post on malevolent responses to some past inquiries will be sure to follow shortly
Scientists are routinely reminded of the importance of the expected high standards required for publication. For a good reason: to uphold the integrity of skeptical inquiry. I remember attending a lecture on exactly such a topic from the editor-in-chief of one scientific journal, and the message was crystal clear: “no issue is too small, let us know if you have a query”. So that’s what I did.
After some routine preamble to the editor, I signed off my email “… the data looks similar, including the noise, although the x-axis scale differs. I thought it important to point this out.” I was a little nervous because I wanted to maintain anonymity (which they do categorically adhere to). But, being the coy inquirer that I am, I at least wanted someone to check this data anomaly without it being seen as a personal attack on the authors. I can confidently say that the editor's response (up until the eventual correction) was very professional and reassuring.
In one back and forth, the journal editor responded thusly: “… thank you for bringing to our attention this discrepancy. We believe _*journal*_ should be of the highest order of rigor and greatly appreciate people who carefully read our work and call to our attention these issues. If you have any further questions… once again, I want to give you a sincere thank you for giving this manuscript such a careful reading to find this error and bringing it to our attention.”
A few short months later both journals published corrections and the world is a better place.
After note: I would highly encourage any reader of science to be involved in the material that you read (which you undoubtedly do already!). Interact with it somehow. Whether it’s a scientific journal or a newspaper article, many authors are only a click away. Ask them something whether you agree or disagree, or just want clarification. Sometimes when you dig you’ll find enlightenment and a shining ray of optimism. Other times you may find sheer malevolence (honestly) and a stark peak through a door that someone didn’t realise was left ajar **. Either way, there's a chance you could learn something, and that’s not such a bad proposition.
* if you would like to know which articles these are please PM me
** a follow up post on malevolent responses to some past inquiries will be sure to follow shortly
RSS Feed
