Great lies of our time: “journalists and coders should sit together to create amazing stuff” (updated)

The Thomson Reuters newsroom. Note papers stacked all over the place. No idea if journalists and developers “sit together” here – but I’d bet they don’t. Photo by Targuman on Flickr.

I keep seeing people saying “you know how journalism and the internet can work better? Have the news org’s journalists and coders sit beside each other. Wonderful things will happen.”

Postscript, but at the top: this post generated a lot of reaction – so be sure to read the followup, which pulls together the many people saying that it can and does work./Postscript.

Let me tell you: when someone spins you this line, it’s pure unadulterated 100% bullshit. Anyone who says this has never looked at what happens when you do this, or considered the differences in work patterns between the two. (It pains me to point out that Wolfgang Blau is only the latest to suggest this – see point 4. I like and respect Wolfgang a lot and he’s really smart – he suggested adding a chapter about China to the second edition of my book – but on this, he’s wrong, as I’ll explain.)

Edit: let me be clear just before you dive all the way in: my criticism is that a seating plan will not solve this conundrum. It is possible to solve it. But not by simply putting people together. You need much more subtlety. /Edit.

Five stars for the idea..

Let’s begin with Anecdote 1. When I worked at the Guardian as technology editor, part of my remit was the games side. We had great games reviewers who turned out games reviews that people wanted to read. As time went on, though, I began to worry that these reviews (giving games between one and five stars) were being lost in the haystack of content. There were years of reviews which could all be relevant to a puzzled buyer looking to get the best stuff.

What, I wondered, about the people – parents especially – who had just got a new games console and wanted to know which were the top-rated games to get for them? Where did they go?

The obvious answer, it seemed to me, was to create a page for each console which would show the highest-ranking games – perhaps in a grid. Obviously you’d need to update it from time to time. Finding which games were for which console wouldn’t be hard because each review would be tagged “PS3” or “Xbox360” or “Wii”, but grabbing the star ranking and then ordering them was tougher. But it wouldn’t be a problem for a developer who knew their way around our CMS (content management system), I figured.

My cunning plan: make it possible to buy the games directly through the CMS, either from Amazon or a white-label seller, so that the pages would earn money.

So I went to one of the developers who I knew had some spare time and was a dab hand with the CMS. I explained the problem and the potential. He looked at me for a while. I also suggested that we might be able to get a list of upcoming games – which our reviewers could supply – and create a page where people could pre-order them. He looked at me a little longer.

Then he went away and spent a few months developing a way to display the Guardian on Google TV. After making no appreciable impact, Google TV got canned. We never got the “five-star games” pages or the “upcoming games” page.

Love ya, Excel

This leads to Anecdote 2, which followed on pretty directly from that. I’m a journalist, but I can code – PHP and Applescript, principally. Nodding acquaintance with Cobol, Fortran, Pascal, Lisp; blind date with Objective-C where I fell asleep in the wine; blind date with C where I walked out. Anyhow, I can wrangle simple-ish tasks to automate extremely boring stuff that is often the basis of work in many newspaper offices.

So, for example, every week the UK games industry puts out an Excel spreadsheet showing the top 10 games sold in the previous week by various platforms, with information such as the publisher, previous week’s position, and so on. Here’s what it looked like:

UKIE games chart, raw form

The raw UKIE games chart: an Excel file with lots of columns and spare rows. A formatting nightmare

I liked the idea of putting up a weekly chart, and making it possible for people to buy games directly from the chart. We already had a “buy” button embedded in the reviews. So you could find the relevant game, pull the “buy” button from it, embed it in a chart, and put it on the site. Money-spinner, potentially; and certainly not a money-loser.

I considered trying to get developer time for someone to write this for, oh, about two seconds. Then I decided to write it myself.

It wasn’t trivial. I had to assume access only to the tools and programs our subeditors (who would create the “story” with the chart) had, which meant a standard installation – no extras. That meant no Excel – we used OpenOffice. But that meant no Applescript, because OpenOffice has woeful Applescript support. That meant the sub had to copy the required cells out of the OpenOffice spreadsheet and paste them into TextEdit, which has fairly bad Applescript support, but at least it’s there. More wrangling (stepping through the cells to find the ones we wanted, spotting the many lines of padding in the spreadsheet and ignoring them, using Google’s API to find the link to the Guardian’s link to the game in question, pulling the source of that link to find the affiliate “buy” links) ensued. Once I had the set, we had to create another TextEdit document which could then be copied and pasted into OpenOffice, and saved as an Excel format spreadsheet, because that was the only one which our CMS recognised if you were going to include a chart.

It took me a couple of weeks of spare time, but I got it done. The result was stuff like this:

UKIE games chart, formatted

After using my script, you could buy games directly, or follow the links to reviews. (It’s a different week from the chart above.)

The outcome was that work that would have taken any subeditor about an hour, and been prone to all sorts of errors (reading the wrong column, picking the wrong game) became a two-minute task in which the toughest part was choosing which picture to use. And it earned money through affiliate buying; not huge amounts, but more than zero.

Now – you’re probably saying “hey, wouldn’t it have been easier to ask the UKIE for a link to their API so that you could get it direct, and format it?” Answer: if they’d had an API, it would have been easier. But asking and getting wouldn’t have been. That’s the sort of thing which requires lots of meetings, chinstroking, and discussions about whether it’s possible. All those things are anethema to the culture of daily newsrooms, which is to Get Stuff Done.

(Oh, but of course: there came a time when UKIE changed the formatting of the spreadsheet. And by then, I couldn’t be bothered to work back through the code to adjust for it. So the chart died.)

Update/edit: Robert Rees, who is the developer manager at Guardian News & Media, has written about this subsequently:

The central assumption that runs through Arthur’s narrative is that it is valuable to let readers pre-order computer games via Amazon. One of the pieces of work I’ve done at the Guardian is to study the value of the Amazon links in the previous generation of the Guardian website. I can’t talk numbers but the outcome was that the expense of me looking at how much money was earned resulted in all the “profits” being eaten up by cost of my time. You open the box but the cat is always dead.

Now, that’s interesting. I had seen some of the affiliate revenue numbers too; they weren’t huge by any means. My point was more that we already had the Amazon buttons (that had been set up as an automated process some time before by our development team) and so we might as well use them. But Rees makes a good point – you have to look at the opportunity cost too of the developers’ time if you go ahead with the work. (No developer time was given to my idea.) Also, this was a time – still is – when we were being encouraged to “experiment” with revenue-generating ideas; I didn’t believe that affiliate money would make us rich, but thought that if we could get a white-label deal so we got bought at wholesale and sold at retail, rather than selling at retail-affiliate, then it could be worthwhile. (The Guardian does this with books.)

Sub-update: a little discussion at Gawker threw up this gem, from its former edit honcho Joel Johnson, discussing Gawker’s finances:

Gawker Media is an advertising-based business, with revenues of around 35- to 45-million dollars a year. There are a few other sources of income: a couple of million for international licensing fees (from the companies that publish international versions, such as Kotaku Australia); and affiliate fees, largely from Amazon, that add another $5-10 million a year.

$5-10m per year? Maybe this affiliate game can be played to win if you do it right./sub-update

Rees titles his post “No one loves a bad idea”, which is a fair criticism – my idea wasn’t useful in the wider scheme of things. Equally, though, that was never pointed out. In effect, I did a little skunkworks project, which died. /update

The difference is time

It’s this key difference – that journalists tend to want to get something up there for people to read/watch/listen to, dammit – which in my experience marks the giant gulf between journalists (including those who can code a bit, like me) and “developers”. Journalism, especially daily journalism, and increasingly all journalism, is about getting a result, and doing so effectively, quickly and (optionally) thoroughly. It’s a culture where getting it done right now is what matters. And then you move right on to the next thing, because there’s always a next thing.

Update: Rees calls this (correctly) out as an example of the rush to get stuff done:

British journalism favours action and instinct and sometimes that combination generates results. Mostly however it just fails and regardless of whom is sitting next to whom, who can get inspired by a muddle-minded last-minute joyride on the Titanic except deadline-loving action junkies?

(OK, I consider myself upbraided.) /update

Developers I’ve come across, by contrast, iterate around the same thing repeatedly, just as I did in trying to get the code to work in Anecdote 2. They don’t grab an idea, use their existing knowledge to turn it into a story, and put it out there. (OK, there are some gonzo programmers who do that sort of thing, but you don’t find them in newsrooms I’ve been in.)

My coding in the newsroom was all about timesaving. I wrote scripts that wrangled InCopy so that articles which needed to be strictly formatted (Ask Jack, in its printed days) could be, in different colours, in 10 seconds rather than the 20 eyestraining minutes that it took by hand. (Ask Kate Bevan and Stuart O’Connor about that.) I wrote a script that meant the subeditors on the Media desk could get the day’s Guardian Media stories from its web page in one second rather than five minutes, ready for the daily email which had to go out before 8am. What I did was all about saving time, because that tends to be the most precious commodity for the journalist.

For the developer – in my experience – things are different. I recall Dan Catt (one of the early Flickr developers), when he worked at the Guardian, pointing out how amazing it is that the paper starts with effectively nothing every morning and has a full, complete newspaper ready by 8pm or so. And then keeps doing it again and again and again. That’s organisation. That’s deadlines.

The task of developers in a news organisation, on the other hand, isn’t on that timescale at all. It’s about building apps; building great mobile sites; building great ways for people to interact; building systems to manage and interlink content so the almighty Goooooogle indexes it better. Sometimes it’s about building new CMSs, or ways to link multiple CMSs, but in my experience (again) it’s hardly ever about spotting kinks in workflow and easing them.

And as for the idea of journalists and developers sitting beside each other and the developer saying “hey! Why don’t we..” (or vice-versa) – forget it. They just don’t have that common ground. Journalism tends to be surface-skimming, fast-moving; coding/development tends to be deep-diving, strongly focussed.

Edit: one thing that occurs to me is that many journalists who don’t have awareness of coding don’t realise that boring, repetitive tasks can be automated. So they don’t know to ask. /Edit

Update: of course as soon as I hit “publish” I could think of a counter-example: the team at the Mirror’s @ampp3d team, such as Tom Phillips and Martin Belam, who created things such as the “badgers moving the goalposts” game.

Phillips’s description of how this came about is worth reading:

October 9, 2013. Owen Paterson, the Secretary of State for Environment, Food and Rural Affairs, made a comment in the Commons that morning about the failed badger cull – he blamed the badgers for “moving the goalposts”, which was obviously hilarious. What I love is that you can trace, to the minute, the time from when my mate Francis suggested to UsVsTh3m that they do an “Owen Paterson’s Badger Penalty Shootout” game, to when they put it live. 4 hours and 21 minutes to publish a fully playable game about a news event that happened that morning. Nobody else could come close to doing that then; nobody else has got anywhere close since. They were, in the most literal sense possible, playing a different game to everybody else.

Does this disprove my argument?

You’ll be unsurprised to hear me say: I don’t think so. I see that as an example of developers who are interested in current events basically being given free rein (in a good way). The aim was to get virality, a la Buzzfeed, not to “do journalism”. Repeatable? Effective? Maybe, but the journalism element isn’t exactly blinding.

Similar for the “photo comparison” slider that you see on, I think, the WSJ and Verge (possibly others) for literally side-by-side comparison of photos taken by different phone cameras: I’m sure a journalist or developer expressed frustration at the need for it, but getting it done is an iterative process, focussed on a particular target. Deep diving, not skimming.

The data question

But wait, you say: what about data journalism? My answer: it’s not programming. It’s not coding. Most of the time, it’s great works done with Excel. And most of the time, that’s all it needs. It’s certainly not something where you want a valuable developer spending their time. Journalists are cheaper, and they know how to spot a story, or ought to.

So should journalists learn to code?

I’ve answered this before, on my personal blog – jeez, it was 2009. Here’s the posts’s title: “If I had one piece of advice to a journalist starting out now, it would be: learn to code”. See if you can figure it out.

My advice on developer/journalist seating arrangements? Keep them well apart. But also have an effective manager who listens to the two groups and can tell each to do things. Have a manager who can direct them. Don’t stop the two groups talking. But let each be aware that they’re like animals living at different speeds, which means that they simply won’t be able to comprehend why the other side reacts as it does to some things (“why can’t they understand that hunting down bugs is hard?” “Why can’t they understand that we can’t have this sort of thing cause a screwup in a live blog?”). They can live a mutually beneficial existence. They just can’t live in the same cage.

Postscript: huge amount of pushback on Twitter over this, from Wolfgang (thanks, Wolfgang), Aaron Pilhofer and many others. And lots of good points and examples raised, which I collated in a Storify linked from this followup. It would be great to know that my experience was just an outlier, though I suspect it’s more that there are good ways to do it, and rather less good ways. What we want are the good ones.

12 thoughts on “Great lies of our time: “journalists and coders should sit together to create amazing stuff” (updated)

  1. I agree with your assertion about the difference between the professions, timescales can be very different. However as a developer working in a team of mixed disciplines of journalists, designers and developers where we are all sat together, I don’t agree with your overall argument.

    Maybe its the outcomes that should be argued about rather than the seating plan. Just because a developer is part of the journalism team it doesn’t mean that they should be delivering code against the same kinds of deadlines as editorial content is. A developer’s role on the news desk should consist of:

    * Acting as a tech consultant (helping with Google maps API for example)
    * Provide support for content tooling that journalists use
    * Creating and delivering *simple* coded content
    * To understand first hand requirements and come up with future solutions

    I really empathise with your example around creating tooling to increase timesaving for specific tasks. I often find it very frustrating that other technical people in my org (especially management from none journalism teams) don’t appreciate the benefits in reducing what seems to them as micro timescales. I’ve recently come up against (but gotten over) hurdles where I’ve tried to decrease the time it takes to publish content to the live site (I want the time it takes to be under 1 minute, compared to others who think 10 to 15 minutes is fine).

    However in order to deliver technical content against hard deadlines the two things that we always come up against are testing and complexity.

    If something is complex then it takes longer to make. And in today’s climate where everything you make has to work on a hugely diverse range of devices from mobiles to tablets to computers it can be a challenge. I find that if you design something to be really simple, then its easier to get it working on more devices. Something like a Google map interface, where the interface and proposition never changes, it just expands into the available space, is simple to deliver across multiple devices.

    Then making sure what you build works on everything you have to support can be very time consuming. Just turning on 4 devices, connecting them to the wifi and then getting them to look at what you built can easily take 45 minutes.

    Another issue I often come across (and I’m pointing at journalists for this one) is when people say “but this is easy why can’t you do it fast?” and “Why can’t we just put this piece of code onto the page?”. There are two reasons for why its not as trivial as it looks:

    1) The best thing about making web pages is that its easy to make web pages, but the worst thing about making web pagesis that its easy to make web pages. Making something that works for 1 person is easy, but making it work for a large audience creates new, unforeseen issues. Scaling web tech means having to make sure that you don’t DDOS attack yourself, or you give yourself a huge AWS bill. To build something for a large audience like a newspapers means creating a webpage in a very robust manner. This can and should take time.

    2) Building pages in isolation can be simple, but integrating it into a legacy platform (i.e. a huge codebase that is a 4 years old and has been contributed to by many developers) can create its own problems. Code written by inexperienced people tend to define presentation and logic globally – this can impact negatively on the entire page.

    To get around these two issues you need to invest in training so that journalists and designers create prototypes or very small snippets of code that do scale and can be passed on to developers to quickly make it more robust and production ready. You also deploy your code to the live site in a way that is completely isolated (i.e. via an iFrame).

    In my team we’re rapidly building up a collection of tooling to help journalists build richer content. We’re also looking at how we can increase the efficiency of daily tasks (use IT to make our lives easier). Maybe some of the issues that you speak about come from the fact that developers and their managers are coming from the web development team so they think all solutions should be web based. A better way might be to see the IT resource as systems analysts and business system developers who are there to increase efficiency.

    I have a anecdote of the relationship between journalist and developer working well. A few weeks ago as I got into the office I was immediately pulled over to the daily meet up (devs aren’t typically involved in this) and asked if we could get a Google map up and running on a news page. Ed the journalist had experience building Google maps and had a working prototype. I took his prototype and changed the way Google maps is loaded onto the simple web page. I then converted the page to use the standard jQuery inclusion my site has (we use require.js for dependency management) and then made the page a fragment of HTML that could be inserted into the site.

    It took Ed a further day to finish his prototype but we got the whole thing up and running in under 2 days. An outcome of this is a simple, generic snippet of reusable code that journalists use as a starting point to make Google maps. The next time we do this the bottleneck will be the time it takes the journalist to make the content.

  2. Not all developers are the same. I get the blank stares from other developers too. It’s amazing the things they don’t get. But I’ve mostly gotten the same or worse from journalists.

    I’m the same way you are, I want to get a result, fast.

    The moral of the story is you have to work with the people who want to work with you.

    I’ve found people are very very picky about who they’ll work with.

    No matter what they do for a living.

  3. Pingback: No-one loves bad ideas | Echo One

  4. I read this article with interest because I spent most of my working life in journalism in the hot metal age, long before digital technology reared its ugly/beautiful* head (delete if inappropriate!) but couldn’t stand the heat in the late 70s and early 80s so got out of that kitchen, leaving behind the fumes from the Linotypes and the war of the Fleet Street unions and jumped right into a whole new ball game, taking a crash course in IT and starting a new career as a college lecturer at the age of 50-plus.

    Journalism today is not the game I remember with affection; so I am unable to comment with any degree of authority, but I still enjoy reading about it occasionally, so thank you for this. I’ll go back and read it in more detail and look out for more – as a bystander now.

  5. Charles, one problem in this piece is an implicit assumption that “coder” is a fungible category. But it’s more like “doctor” in a way. As in, there’s doctors treating “this patient has cancer, what combination of chemo/radiation/surgery will help them survive over the next several years?”. And there’s doctors treating “this patient has just been hit by a car and is bleeding heavily, what needs to be done *right now* or they’re going to die within the hour?”. Both are doctors, but those are separate specializations operating on different timescales which involve distinct skillsets. You were basically talking to a chronic disease specialist, when you apparently wanted an accident/emergency specialist.

    The things you want(ed) are not part of the general basic “coding” curriculum, and require a certain interest in topics that are often off the beaten path of getting a steady job. There are of course many coders who do have those interest and skills. But it’s not something you should expect of anyone who has a coding job. Or even most such people.

  6. Interesting observations Charles. In book publishing I’ve often had the same suggestion from authors — wouldn’t it be great if they could sit beside the editor at the computer and work through the book together? This suggestion comes, I think, from mixed impulses. One is not wanting to relinquish control (handing over the baby to a stranger: needing to hover over the process like a helicopter parent), but another is simply lack of understanding of what editors do when they edit a book, and failure to recognise that to a publisher a book is a product in production, not a cherished baby whose special needs have to be explained. The author/editor perspectives are so far apart that the suggestion just isn’t going to work (and the idea horrifies editors — at least of non-fiction; it might conceivably work better for fiction, but as an editor I still wouldn’t like it.) The closest I’d ever get would be jotting notes on a hard copy manuscript over a few glasses of red. But an author perched over me at my computer? Guaranteed to treble the time taken to edit, for one thing. And it would be an enormous distraction from the nitty-gritty, mundane bits of editing (standardising spelling and style, etc) – quite apart from being watched while you work. (Recalls the old motor mechanic sign: “FEES: $20 per hour; $50 if you watch; $100 if you help.”)

  7. I’ve been in technology for a long time, but not with journalists. Reading your post’s title got my interest though because at the outset I can’t think of professions with more disparate briefs. There has been a general trend the last several years, since we ushered in the Age of the App, to cut out middle men and move developers closer to users, in this case, journalists. This trend has been received with mixed results.

    Having the right title and right desire won’t get the job done. It has to be he right personalities – users who can make a step toward describing processes in detail, with intent, and can balance what they need right now with what the future direction should be; and coders who can and do take a true interest in a business, can suss out procedural pain, start to translate that to solutions and be respectful of timelines.

    Seth’s comparison to the medical profession is spot on. I’ve had the pleasure to work with a few developers and users who I think could make great partners, but they are the exceptional, not the rule.

  8. Fascinating. I read this from a slightly different perspective: a marketer, charged with driving revenue, whose development team splits its time between process/workflow issues (which are important) and visionary, world-changing projects (that never seem to do that).

    I recognize the criticism of being addicted to the adrenaline of deadlines as one of the critical voices you cited suggested. But I also think it’s a personal preference. Our developers work on a 25 project roster. My department will produce nearly 1500 projects of varying complexity.

    That’s sort of difference results in a huge gap in understanding and management is really in no position to help sort it all out. At times I think I work in the institutionalized form of Berlin’s hedgehogs and foxes.

  9. Pingback: Journalists & developers: the power of collaboration | inso

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s