My Interview with Ivan Oransky at #scio12 – The Transcript
[This is a cross-post from Field Notes, my other blog, where I write about science across the media.]
When I was at Science Online 2012 earlier this year, I had the opportunity to interview Ivan Oransky of Retraction Watch and Embargo Watch. When I tweeted about transcribing the interview as part of the preparations for this article in Spektrum, some people asked if they could read the transcript. After Ivan kindly agreed, I can now say: Yes, you can.
Just a note in advance, though: It’s a transcript (minus some off-topic remarks and outside distractions…), not an edited interview, so it may meander a bit here and there. We talked about blogging and the role of journalism, specifically science journalism, about potential business models and a retraction mystery novel, about science as a human endeavor and a developing culture of error reporting online, about commenters and German readers, about dialogue and the role of humor – and much more. I immensely enjoyed listening to the kind and generous innovator I found in Ivan, and I hope you do, too.
So without further ado – the interview:
K: What motivated you to start Embargo Watch and later Retraction Watch?
IO: It was actually this conference that inspired me to start these blogs. I have been here every year, this is the sixth year, and I wouldn’t miss it. Because it’s just a wonderful group of people and you learn so much. And it’s just such a supportive goup and we can all criticize each other’s work in an incredibly constructive way, without it ever being a personal issue. You can even criticize speakers while they are speaking, on Twitter, and people will respond and we all are the better for it at the end of it.
On the subject matter, though: I had always thought a lot about embargoes. This goes back years. When I was at The Scientist, I wrote a few blogposts for the Scientist-Blog, about a couple of situations. And there is a book by Vincent Kiernan called „Embargoed Science“. It’s his thesis. He was a reporter at the Chronicle of Higher Education for many years and then went back to get a PhD in communications or science communication. And his thesis was about embargoes and what effect they have on science and science communication, in particular journalism. So he eventually published that thesis as a book and I always followed his work, I thought very highly of it. And then there were some incidents with WHO, the World Health Organization, the New York Times, the New England Journal of Medicine had something else that happened with embargoes, so I would always write little items about it, you know, think about it a little bit, used the „Ingelfinger Rule“, it’s a fun word to use…
The nice thing about blogging is that it can give voice and a regular rhythm to your obsessions, right? And embargoes really are one of my obsessions…
So I came back from this conference two years ago and just decided – well I didn’t „just“ decide it, but it finally got me to say: You know what? I should be blogging all this stuff that I think about all the time. And so I started it, that was in February 2010, and it just – it had such an almost immediate response. It has never developed that much traffic, really, and it slowed down because I haven’t been posting as much because I have been on Retraction Watch, but clearly the right people are reading it. I get tons of tipps. Scientific societies, a number of them now, have changed their policy in response to my criticisms. In particular, I have a real problem with societies that make abstracts or papers freely available, but then say they’re embargoed. That makes no sense at all.
And a number of them have changed that, because I wrote about it. And they say, this is why we are changing it. And that says to me – well, that’s the power of the media. I mean, I didn’t come at it from nowhere. I was at Reuters, I’ve been at other places, I am not the most well-known person in the science-writing world, but enough people know who I am, so that I was not starting from scratch.
K: Do you think, this is why they didn’t dismiss you as just another troublemaker?
IO: I think the trick is to go about it seriously. That’s one of the things I learned, or one of the things that was confirmed for me, and that was my plan: I do things journalistically, I do them with a – what I think is a rigourous approach. So the trick is to be serious about things and, even when you criticize people, to give them the opportunity to explain themselves. And often those explanations make at least as much sense as your criticisms and – that’s a dialogue. And so I was always very careful with people. I use humour… I think that’s appropriate. Partly because it is okay to make fun of human foibles, when you acknowledge that you yourself make those same kinds of mistakes. And sometimes, quite frankly, ridicule isn’t so inappropriate, and… that happens more on Retraction Watch, where people are clearly lying or not being forthright.
So I did Embargo Watch for about six months – like I said, I got great response… and Adam Marcus, who is my co-founder at Retraction Watch… – we had known each other for a number of years, worked together on small projects, he freelanced for me at various places – and, separately, he and I had always been very interested in retractions, because whenever we looked at them we found big stories, very exciting stories. For example, Adam wrote about an anaesthesiologist who ended up going to jail, for making up evidence and things. [...]
So, we had always been interested – again: separately – in retractions, and we were talking one day, I remember having a phone conversation with him, and I said, you know, I have been having all this fun with Embargo Watch and you and I seem to have the same sort of obsession, where we learn things and we don’t do anything with them, we don’t have a platform or an outlet. You want to blog? [...]
And that really just took of! We are now avaraging 150.000 page views a month, and this month [January 2012] will probably hit far more than that, actually. People are picking us up and linking to us…
K: And that’s for a very specific topic, a niche.
IO: Right. I think, it’s partly because we publish a lot, in average about six posts a week. [...]
But the response has been just phantastic. We are asked to speak all over the world. I was in Berlin, for a conference several months ago, I’m going to Barcelona next month, and I have just been invited somewhere else… [...]
We certainly have our critics, people from within science – we don’t know who some of them are, a lot of them are anonymous on the blog, and we let them have their say – who think we are focussing on the negative and all that. But we have far more people who say, we’re just doing something that… it’s not that they think the idea is so brilliant. The idea is not even that smart, it’s just taking advantage of technology to really continue and use this… the way a beat reporter would, someone who is just on the same idea all the time.
We just have been really humbled by the response and also very happy about it.
K: Apparently there is a need…
IO: Yeah, it filled something. We are not quite sure what that need is or was. We just happened to think that retractions, and corrections for that matter, need more publicity. Sunlight being the best desinfectant.
Also, because journalism is not doing a very good job, even when the retractions are clear, which is not always the case, as we have learnt… They are not doing a very good job at publicising it. And we think, that whatever publicity they gave to the original paper, they should give to the retraction.
Especially with technology today, any time you find the original piece of content, you should be able to find – well, not just find, it should be yelling at you from the screen: here is the correction or the retraction!
K: Why did you choose a blog? You could have put it on a newsletter, or on Reuters, in a feed or something…
IO: Actually, I couldn’t have. Or I could have, if I had pitched the whole idea… But the activation energy is so much greater. I mean, literally, especially since I had already set up a blog and made some mistakes and learnt from them on Embargo Watch, that it took me… – to set it up and to get out the first post was less than a day. That’s what makes it so powerful.
But it’s not just that. It’s actually the social media aspect of it. People talk about social media and what is all this… I don’t care about the definition. What I do know is that once you start blogging you are connected to this world of people who care about similar things and who get interested in your work. They really become your… „advertisers“. I don’t mean advertisers paying you money, I mean, they become your marketers, they get the word out about your work and that is what made it so powerful.
With Reuters, I actually wanted to keep it separate and it still is separate. And Adam similarly, he has a day job, too. [...]
What we are doing is what someone ten years ago, say, when you had a group of people who cared about something, might have done with a list, an email-list. But now we can just… anyone can read it anytime. People sign up for the emails, there are 1300 people signed up for our emails, many more are getting an RSS, things like that… The blog, it’s just software.
We also have begun to categorize our posts.
Because after one year, we realized, there is actually some data here. We should be able to search better and categorize. So we actually set up categories. And now, each time we have a new journal we add a category. I mean the list… it’s floor to ceiling, but it’s actually quite robust, because if you are suddenly just interested in this particular author or this journal or this country, or anything, or the reason why something has been retracted or whether something is behind a paywall or not, one of the retraction notices, we categorize in all those ways… and it helps people find things.
K: So it does become more like a database…
IO: Yes! Not yet a formal one, we would be careful about saying that. It’s much better than a list of posts, much more robust than that, but it’s not quite a database. We are not librarians, we don’t have training in all of that. I would love some help with it, and at some point one of the ideas that we want to move forward on is creating a robust database. That anyone could check, a librarian or anyone writing a paper could check papers against they want to cite. [...]
At some point maybe we will do that, work with people, maybe even get a grant to build something, who knows.
K: But at this point, you are not making any money from it…
No, and there is something to be said for that. It’s not that it has to remain revenue free forever, but some commenters sometimes say: „ It’s obvious that you are not having a financial agenda, and we appreciate that…“
The reason why we would want one day to develop some real revenue on it is, that we’d want to spend more time with it. It’s not that we suddenly wanted to get incredibly wealthy or we saw an opportunity to… in a cynical way, some people would say, oh, you are criticizing this group and now you are going to take their funding. None of those things. We love our day jobs, this is not about that, but if we could spend more time with this, wouldn’t that be great. And in order to do that, realistically in life, you have to pay the mortgage and all that…
K: One wonders how you do this anyway. You never sleep, right?
IO: Obviously, you have to learn how to be efficient. One benefit is that, just a detail, my wife happens to work evenings, she works in television, so I can work later without anyone caring, which is actually not such a small thing. But also, you make choices about what you end up covering and we certainly always have stories on the backburner, that we are working on, but we tend to chose things that can get done quickly…
And there are two of us, that helps a lot. It’s six posts a week, and we do it pretty much fifty-fifty. Three posts a week is still a lot, given what we are doing with them. I mean, it’s not „I read something interesting and you should, too.“ – It’s call people, report, maybe fact-check… but it’s still three as opposed to six. Except weeks where one of us is on vacation…
K: So what will be your next step? Turn it into a databank?
IO: I think that’s one of the possible next steps. We have just started to think a bit about what we might do. There are a couple of things. One step is to simply make it bigger. We have so much more material, but we are reaching the limit of how much time we can spend on it. Even just having a third person would be really helpful. But it’s not the sort of thing where I can say, oh let’s just hire a third person – I can’t hire anybody! If there is someone who is similarly passionate and who wants to join comrades and not work for anything, sure, welcome, but I don’t want to take advantage of people. If we had some funding, that would be one place to use it for. The database would be another, building an ability to check articles against, which would require more than just a database, it would need some software, an algorithm.
We might also, although we haven’t really explored this a lot, we might also look at opportunities to educate. I am more and more being asked to speak to groups of people as part of this Responsible Conduct of Research Requirement that we have here in the US. [...] There is funding for that. The government actually requires it, but also gives out grants for it. It’s a mandate and they don’t want to make it an unfunded mandate. We would have to set up a company, but there are ideas… Or write a book! We have actually started, Adam and I, kicking around the idea of a novel, sort of a thriller or a mystery novel. It would be fun. Because there is intrigue. We can’t always prove it, so we don’t run with them; but in a novel you can write the story the way you are pretty sure it happened and just have some fun with it.
K: If you look at the broader picture – is there a trend towards this kind of reporting?
IO: I’d love to see it take off. There are a couple of things… Do you know „Abnormal Science“ in Germany by Jörg Zwirner? [...] It’s interesting, it’s almost like Darwin’s finches with the way that some of these things have evolved. Because we have a blog, that is really just about – it’s about the scientific process, but we start most of the time with when a retraction happens. Sometimes we cover papers, that haven’t been retracted, but there is a lot of investigations and things… What Jörg does is, he actually looks at papers that look dodgy, that look suspicious. And he draws in a lot of readers who look at different images and say „Oh, there is something funny about this image“ or „This image was copied from this other…“ – and things that he does often end up on Retraction Watch later and we give him credit for that. [...]
I’d love to see more of this kind of reporting. The question is: how does it sustain itself? You know, Retraction Watch may not get bigger as long as Adam and I are doing exactly what we are doing, but it is sustainable in the sense that there are two guys who are willing to do the work that sustains it. Adding a new [blog] would require someone else to be that way. You could have a Peer Review Watch, Data Watch, you could have everything…
I would love to see more of that because I do think that it’s a shame that in many cases science journalists don’t look at their role as being sceptical. It’s not that they avoid scepticism, they just don’t do it with any enthusiasm. It’s not what they feel that their role is. And I would differ with that. I think the core of journalism is scepticism and keeping people honest and challenging authority when appropriate. And when you are only writing about how wonderful things are, how wonderful science is – you know, science definitely is wonderful and has made lots of strides – but it’s a human endeavor. And to me the most interesting stories are the ones that demonstrate just how human it is.
But I think the question is: How feasible is it? You know, we do very nicely with traffic, without being part of a network and all that, could we sell advertising and what have you – in other words: what is the revenue model? Other than creating a database that maybe people would license… I’m not sure that would [work]. That needs a lot of thought and I don’t have the answer for that. But I would like to see science and medicine covered more sceptically.
K: Do you think this kind of reporting will always remain in the „semi-official“ world of the blogosphere or can it go into mainstream media?
IO: I think, to be fair, the mainstream media, when it does stories like this, they mostly do a pretty good job. They have uncovered all sorts of things. And we are happy to pick those stories up.
We are certainly covering these on a more granular level… I don’t know if they will remain in the province of the „semi-official“. I mean, you have organizations here in the US like ProPublica and there may be one that is specifically interested in science or medicine. There isn’t one now, and maybe that will happen one day. So I don’t know, I really don’t know what will happen with it. But I think, what we have learnt is that there is an interest in it and we are hoping that this will work out somewhere.
K: It seems that the mainstream are more focused on big scandals and not so much on ongoing processes like you can cover them in your blogs.
IO: And working in my day job in mainstream media I know why. It’s understandable. I am not condoning it, but you need to produce a certain amount… and these things take effort.
And the question really would be… let’s say that we did, for example – and this is completely hypothetical – let’s say that we partner with Reuters or someone else who had a daily [outlet]… how often would what we are doing be of enough interest for them? I don’t know. A lot of the papers that we cover are barely of interest to the people who wrote them, right? And yet, lately, there had been a number of major cases, where, because I worked at Reuters and I happened to know something about retractions now, I end up writing the story for Reuters.
K: With this granular way of reporting, it might be easy to lose sight of the bigger picture. Do you have a way of summarizing your observations?
IO: I do an annual. We did one on our first birthday, which was in August, I did one at the end of the year. We tried to summarize some of the big themes, we were looking at. Also our columns [in LabTimes], because the newspaper comes out only 6 or 8 times a year, those are a bit more „big picture“.
At the same time, I’ll say that the interest that we’ve had, even just this year, in this subject of retractions and of scientific integrity – I’ve done Science Friday on NPR, and On the Media and All Things Considered, these are national programs, and the Wall Street Journal, the New York Times [...] – so I do think there is an interest. How sustained that interest remains, I don’t know. That’s an open question.
K: Do you think this is part of a more general trend of how the new media are dealing with errors?
IO: Yes. Do you know Craig Silverman? He was basically the first reporter, who – literally, I think it was the day or the week we launched Retraction Watch – he just immediately got it. Because he saw that what we are doing for the scientific literature is what he has been doing for the newspapers, for the mass media. So he wrote a column about us for the Columbia Journalism Review.
We were also invited to a sort of fact-checking [think tank]… So, it is part of a movement – well, that sounds grandiose… But there are other people thinking about it. I think that there is – in certain circles, and I think those circles are growing – there is an interest in making sure that the facts that people have at their disposal are as up to date and true as possible. And that’s what we are about with Retraction Watch. It’s what Craig Silverman is about with Regret the Error. [...] Here in the States – and I imagine there are some more efforts – there are PolitiFact, FactCheck.org. They mostly focus on politics, but most of the media tends to focus first on politics, too. And I think we are part of that, yes, I think that’s legitimate.
K: Last year, here at the conference, there was a session about maintaining standards on the internet. And I think this is one of the ways…
IO: And make it easy, make technology help you. Adam and I use technology to blog and to link and categorize etc., but we are still doing a lot of the reporting by hand, because we don’t have a better way to do that yet. For example, if we find a retraction that we don’t think says enough, we have to call or email and contact.
On the other hand, what has happened is – our commenters, they are very… they are just intense. They will put up comments: „Oh, look into this paper…here’s an image and you should look at that…“, so it’s almost crowdsourced. And what will happen sometimes, when I am away for a few hours and I can’t be at my computer, a thread will start and by the time I get back it is concluded and people basically deliverd us a new post. And we know, we only have to place one phonecall and we can confirm everything that’s in there.
K: There is one prominent example for crowdsourced fact-checking in Germany concerning the thesis of former Minister of Defence zu Guttenberg…
IO: Yes, we covered that briefly. It’s funny, we have a big readership in Germany. WordPress is not very granular in telling us where people are coming from, because it is the free version that we are using, but we know just from the feedback we get and some of the IP-addresses and things, it is very well read in Germany. Per capita, I think, probably more than in any other country. In the US and the UK we have lots of readers and obviously it’s in English, so it’s easier, and the US is where most of our sources are, so we end up covering more. But Germany certainly had a lot of retractions this year. And a lot of high profile ones, Bulfone-Paus for example, and still actually ongoing, we just published [...] another one, a 13th one. And there are others, Boldt and Savaskan… [...]
K: Who alerts you to these cases?
IO: [...] Bulfone-Paus was an ongoing story when we started the blog and I don’t honestly remember how we first heard about it. But once we did, in that case in particular, as I am sure you know, there is this whole group of anonymous people that keep feeding that story. We worked hard, once we’ve learnt things, to learn more, but we always had people feeding us all sorts of things.
One is not anonymous at all, and we always credit her, Karin Wiebauer, who is, I think, at this point well-known for this in certain circles, but she is really [open], she uses her name. In fact, she is quite careful. She sometimes sends us things and says: Look into this, but please don’t use it unless you verify it, I don’t want my track record to be [compromised]. She is very careful. There are others who – depending on how you consider anonymous or pseudonymous – they always use the same name, or a number of names. We don’t know who they are. And our argument has always been: Look, we would like to know who they are, we want to know who they are, their conflicts and what have you – but: they are giving us facts, right. Or they are giving us questions. We can check them. And if they check out, I’m going to write it, and if they don’t, I’m not. [...] A lot of them are clearly either working or former scientists or medical researchers who know something like that, probably the only people who would have the capability to figure it out…
We are not experts. In medicine, or the things that are more medical, certainly anaesthesiology for Adam, we understand what’s happening. But when we write about physics? Even microbiology… I have a colleague who studied biology and I often send things to him and ask him: „Explain this to me, I don’t understand what’s happening…“ Which means, we are really generalists. We look at the science and we understand what’s wrong with it, but we are not deep experts. Frequently, pople in chemistry and physics come and say, oh, why don’t you cover more in physics and chemistry – we cover whatever we see!
But at the same time, it is sometimes a little more… clear.
In physics, in particular, there just aren’t as many [retractions], just because of the way studies are published. You have archive, you have pre-print service, and it does seem – with the exception of Henrik Schön, the Bell Labs guy from about 20 years ago – it seems that physicists are quite straightforward about mistakes. They just say, here is the mistake and then move on. (K: It’s probably easier to spot the mistakes as well.) Yes, it’s math. Or it’s theoretical, or something. But it’s not a Western Blot with a fuzzy band… It’s interesting, because Western Blots – this is something we are going to take up – Western Blots seem to be the common denominator in many of these misconduct cases. They seem, they sort of – and again: I did work in a lab, in the college, in medical school, so I did Western Blots, I can understand how they get manipulated; it’s quite simple… well, it’s always simple, but it’s actually quite easy and – it’s something that we would need to sit down, like really talk to people about it, put together a bigger picture, but…
On the blog people have these rich discussions about Western Blots, whether or not they are too easy to fake and things like that. I have to say: the level of comments, that is really, to us – you know, there is obvious spam that is just robots and that the software takes care of – but we reject very few comments. The level of discourse is extremely high. Occasionally, when there is a post that gets a lot of traffic, we will get some other sort of comments. But [most of the time] they are helpful, they are critical, sometimes they are critical of us. But those are people who come back. Especially when we respond… And that’s, by the way, is one of the other things, that’s part of the power of the medium: you are connected to people online. You do a newsletter or you put it out on Reuters. Reuters is global, it has huge reach, but you don’t hear back. It’s very one way.
K: Do you moderate comments?
IO: Yes. Which is a little bit of a job in itself. Most posts are getting a dozen or 20 comments… It’s a place where people discuss the issues, which is what we really want. It works for us and I think when you have a niche, it is going to work really well. The question is – like big newspaper sites, they have hundreds of comments on something… It’s not really a dialogue, it’s just a lot of people saying things and there is no way to keep up with it. But I think on niche sites, it will work very well.