Debugging tech journalism

In November, a few days after Sam Altman was fired — and then rehired — as CEO of OpenAI, Reuters reported on a letter that may have played a role in Altman’s ouster. Several staffers reportedly wrote to the board of directors warning about “a powerful artificial intelligence discovery that they said could threaten humanity.”

The discovery: an AI system called Q* that can solve grade-school math problems.

“Researchers consider math to be a frontier of generative AI development,” the Reuters journalists wrote. Large language models are “good at writing and language translation,” but “conquering the ability to do math — where there is only one right answer — implies AI would have greater reasoning capabilities resembling human intelligence.”

This was a bit of a head-scratcher. Computers have been able to perform arithmetic at superhuman levels for decades. The Q* project was reportedly focused on word problems, which have historically been harder than arithmetic for computers to solve. Still, it’s not obvious that solving them would unlock human-level intelligence.

The Reuters article left readers with a vague impression that Q* could be a huge breakthrough in AI — one that might even “threaten humanity.” But it didn’t provide readers with the context to understand what Q* actually was — or to evaluate whether feverish speculation about it was justified.

For example, the Reuters article didn’t mention research OpenAI published last May describing a technique for solving math problems by breaking them down into small steps. In a December article, I dug into this and other recent research to help to illuminate what OpenAI is likely working on: a framework that would enable AI systems to search through a large space of possible solutions to a problem.

Reuters is hardly the only news organization to publish superficial or confusing articles about technology. Today’s highly competitive media industry creates bad incentives that discourage reporters from doing in-depth or nuanced journalism. Shallow or sensational stories about technology require less resources to produce and often attract more attention from readers.

There is great technology journalism being done today, but it tends to appear in specialist publications that cater to tech-savvy audiences. If we want a well-informed public, we need to figure out how to raise the quality of journalism consumed by ordinary people — those who lack the expertise and financial means to identify and subscribe to specialist publications.

It’s hard to cover technical subjects for normies

The first few paragraphs of a news story — known as the lead — are often the hardest to write. Indeed, I rewrote the first six paragraphs of this article at least a dozen times.

A lead needs to accomplish two things: convince the reader the story is interesting enough to continue reading, and provide the background necessary to understand the rest of the story.

These two goals are connected, since a reader who doesn’t understand a story is unlikely to find it interesting. But they are also in tension because the longer the explanation gets, the more likely the reader is to give up before reaching the “nut graf” that sums up what the article will be about.

When I’m writing the lead of a complex piece, I can almost hear a timer ticking in the background, counting the seconds before the reader gets bored and closes the browser tab.

This is a much bigger challenge for some topics than others. I was at Vox during the 2016 election and would sometimes help out on election nights. Writing leads about presidential politics wasn’t difficult: Everyone knew who Donald Trump and Hillary Clinton were and why it mattered which of them became our next president.

In 2017, I moved from Vox to Ars Technica, where the leads of technology stories became much easier to write. Ars caters to software engineers and IT professionals, so I could assume a high level of knowledge about — and interest in — technology topics. This meant I could spend fewer words providing basic background information or convincing readers that a story was interesting — and more words on the meat of the story.

But a journalist who writes for a general news publication doesn’t have that luxury. Many readers know very little about technology and need to be enticed to learn more. So journalists have developed some standard templates for making technology news interesting to ordinary readers.

One approach: Identify an emerging technology that has the potential to be the “next big thing.” One of my first feature stories for Vox followed this template: I argued that the “Internet of Things” was likely to dramatically improve people’s lives.

In hindsight, I overstated my case. Lots of people now own internet-connected light bulbs, doorbells, and smart speakers, but I’m not sure they have significantly improved our standard of living. And that’s typical of the genre — many of the world-changing technologies trumpeted by the media turn out not to be so world-changing after all.

Reporters pitching these stories to their editors have an obvious incentive to exaggerate the importance of the technology or company they are writing about. And once they’ve started work on a story, they have a strong incentive not to ask too many skeptical questions. After all, if they learn that the technology isn’t actually a big deal, they might have to kill the story and be left with nothing to show for their work.

These stories are particularly common in business magazines, whose readers are interested in up-and-coming companies but not the details of how their technology works. And sometimes that lack of curiosity proves embarrassing.

Elizabeth Holmes was featured on the covers of Forbes, Fortune, and Inc. before The Wall Street Journal revealed that Theranos’s technology didn’t work in 2015. Sam Bankman-Fried appeared
on the covers of Fortune and Forbes before the 2022 implosion of FTX.

Of course most companies that receive magazine cover treatment don’t turn out to be massive frauds. But the tendency to take subjects at their word can lead to superficial and credulous coverage.

The scandal frame

Conversely, reporters love to catch big companies doing something illegal, unethical, or anticompetitive.

Reporters working on these stories have the opposite incentive of those writing puff pieces about startup founders. It doesn’t pay to give subjects the benefit of the doubt because a story is more compelling if it’s framed in simplistic good-versus-evil terms.

For example, in 2014 The Verge published a scoop about Uber’s “playbook for sabotaging Lyft”: contractors hailing Lyft rides and then trying to recruit the drivers to work for Uber. As I wrote at the time, it’s not obvious Uber did anything wrong here. After all, workers benefit when companies compete vigorously for their services. But that kind of nuance was nowhere to be found in the Verge story, which argued that Uber’s “ruthless efforts” revealed its “cutthroat nature.”

This starkly moralistic framing not only made the story more clickable, it also saved the reporter the trouble of researching recruiting industry norms to see whether Uber had actually crossed any ethical lines.

Another example: In 2017 there was a wave of stories about an iPhone software update that slowed down the performance of older iPhones. Reporters dutifully included some context: a quote from Apple stating that this was necessary to prevent devices with aging batteries from shutting down unexpectedly. But most stories about the controversy still framed it as a scandal. And few, if any, reporters actually tried to figure out if Apple’s solution had been a good one.

A more subtle example comes from 2022, when The New York Times reported that “LinkedIn ran social experiments on 20 million users over five years.” The “social experiment” here was A/B testing — a technique technology companies routinely use to test new features. LinkedIn experimented with different versions of its “People You May Know” feature to see whether people are more likely to get new jobs from arms-length acquaintances than close friends. Results from their research were published in the journal Science.

The piece ominously suggested that conducting such experiments on users “could affect their job prospects.” It only mentioned in passing, in a quote from one of the study’s authors, that failing to conduct such experiments — and hence to learn whether a feature improves the user experience — could affect users’ job prospects too, before moving on to discuss other examples of A/B testing, such as Facebook’s “quiet” manipulation of news feeds.

The fourth paragraph of the Times piece acknowledges that A/B testing is a “longstanding practice” in the industry and that even The New York Times uses it. But the piece was still framed as a potential scandal rather than a worthwhile bit of social science. And I worry that such coverage could discourage press-shy companies from doing research like this in the future.

The growing antagonism between the media and Silicon Valley

Over the last decade, Silicon Valley elites have grown increasingly frustrated with media coverage of their industry. And they aren’t wrong that coverage has grown increasingly negative. But I think they’re wrong to assume this reflects a hostility toward Silicon Valley in particular.

A more banal explanation is that companies like Google, Facebook, and Uber aren’t startups anymore. It no longer makes sense to publish positive profiles introducing readers to these companies. So reporters have switched to treating Silicon Valley giants like other big companies, which means mostly writing about them when they do something wrong.

Think, for example, about the way oil and drug companies get covered by the media. Stories about oil companies typically involve accusations of environmental damage or harms to human rights. Stories about drug companies frequently focus on addiction, misleading marketing, or price gouging.

In part, this happens because a big company doing something wrong is a more colorful story than a big company just delivering good products at a reasonable price. It also happens because many journalists got into the profession to “speak truth to power.” As big technology companies have grown wealthier and more powerful over the last decade, journalists increasingly see them as powerful institutions that deserve more scrutiny.

I think that’s entirely appropriate up to a point. Writing about misconduct by big companies is an important function of an independent press. But I worry that the pendulum might have swung so far in this direction that other worthwhile angles get neglected.

An example: self-driving technology

One of the stories I’m most excited about right now is the slow but steady progress of Waymo, formerly the Google Self-Driving Car Project. In December, Waymo reported that it had driven more than 7 million miles with no one behind the wheel and that its vehicles were involved in injury-causing crashes 75% less often than humans driving the same roads.

Waymo held a video call with reporters shortly before it released its report. Afterward, I compared notes with another reporter. We both agreed that the story was unlikely to generate much attention.

“I was just talking to my editor about how I’ll write something short that no one will want to read,” he told me in a direct message. “It does feel newsworthy, but they did a terrible job providing a punchline.“

I find attitudes like this frustrating, but he wasn’t wrong. My write-up of the news did not attract much attention, and I bet his didn’t either.

What does drive attention is crashes. Any time there’s a significant crash involving a self-driving vehicle, it gets wide coverage in the media — even if the self-driving vehicle isn’t at fault. And stories that report on broader crash trends can be extremely one-sided.

For example, a Washington Post story last summer reported that the number of fatal crashes involving Tesla’s Autopilot technology had “surged over the past four years.” Conspicuously missing from the story was any data about the denominator: How many total miles did Tesla vehicles travel with Autopilot engaged?

Without that information it’s impossible to tell whether Autopilot makes Tesla’s vehicles safer or less safe. But a more nuanced analysis wouldn’t have fit the attention-grabbing headline the Post chose about the “shocking toll of Tesla’s Autopilot.”

To be clear, I think there are legitimate grounds for concern about the safety of Tesla’s Autopilot technology. Tesla does seem to be more cavalier about safety than Waymo.

But I worry about the perverse incentives created by our intensely competitive media environment. I bet the Post’s story about Tesla last summer got a lot more attention than anyone’s write-up of Waymo’s positive safety data in December. And this could push the public toward unwarranted skepticism of a technology that has the potential to save thousands of lives.

The power of a knowledgeable audience

There are lots of other kinds of stories to be written about the technology world — explaining how technology works, exploring its legal or policy implications, examining how people are using technology, and so forth. These kinds of stories are not only more work to report, they are often harder to sell to readers.

A related challenge is that technology talent is expensive. I have a master’s degree in computer science, and I like to think this background enables me to write more sophisticated technology stories. But few technology reporters have this kind of training.

A big reason for this is that computer scientists can make more money working as software engineers. But another factor is that most readers are not technology experts themselves and don’t know or especially care if technology stories get the details right. So a publication that pays a premium to hire a reporter with deep expertise may not see a good return on that investment.

There are exceptions, and one of them is Ars Technica, the publication where I worked from 2017 to 2021. If I made even a slight error in an Ars story, our tech-savvy commenters would point it out within minutes of publication.

As a result, I felt pressure to stay at the top of my game. I made sure I understood a technology thoroughly before writing about it. Discerning readers could tell our stories were a cut above technology stories published elsewhere, and so they kept coming back to the site.

So why doesn’t everyone get their technology news from a site like Ars Technica? Ars doesn’t focus as much on making stories accessible by a broad audience. Writers spend fewer words explaining basic technology concepts and convincing readers a topic is interesting. Ars stories sometimes get deep into the technical weeds, which is catnip for our core audience but may bore casual readers.

In theory, a publication could deliver the best of both worlds, with the rigor of a specialist publication and the accessibility of a generalist publication. But it would be hard to make this pay off financially. Normie readers would be unlikely to notice or appreciate the publication’s greater technical rigor. Indeed, more rigorous reporters might actually cost their publications money by nixing sensational stories that have the potential to bring in more traffic.

Money matters

The awkward reality is that tech journalism is the way it is because that’s mostly what consumers want. Nobody is explicitly asking for technology coverage that lacks depth or nuance, of course. But journalism like this is often easier to produce and is often more clickable.

Moreover, journalism in the 21st century is in a state of perpetual crisis. Our industry has endured wave after wave of layoffs and publication closures. Already in 2024 we’ve seen announcements that Pitchfork will cease to operate as an independent site, and both Sports Illustrated and the Los Angeles Times have planned layoffs in the hundreds. Indeed, aside from a brief wave of investor enthusiasm between 2013 and 2015, the industry has been like this across my entire 17-year career.

So I would love it if publications invested more in in-depth reporting about technology and other topics. But the reality is that most news organizations are struggling just to stay in business. With little margin for error, publishers are understandably reluctant to invest in nuanced technology journalism that is unlikely to generate a clear financial return.

This seems like a promising direction for philanthropists.

There is great technology journalism out there, but it’s often expensive. My former employer, Ars Technica, is a free, ad-supported publication, though paying subscribers get an ad-free experience and other perks.

But other high-quality publications are very expensive. The Information, an excellent business publication focused on Silicon Valley, charges $399 for an annual subscription. There are newsletters like Casey Newton’s Platformer and Ben Thompson’s Stratechery that cost $100 and $120 per year, respectively. If you’re interested in artificial intelligence, you might enjoy my own newsletter, Understanding AI, which costs $75 annually.

A subscription business model insulates these publications against the vagaries of the advertising market. But it also puts them out of reach for casual news consumers. So the harder challenge is to improve the quality of the technology journalism that the average reader consumes.

This is tough because, as we’ve seen, these readers can’t tell — or maybe don’t care — if the journalism they consume is any good. So normal market mechanisms aren’t going to help.

I also don’t think it makes sense to try to build new mainstream news organizations from scratch because they would face the same perverse incentives as existing organizations.

Rather, the goal should be to encourage and assist existing news organizations to step up their game. Here are a few ideas.

Every year, America’s top news organizations devote significant resources to stories calibrated to win a Pulitzer Prize, with the most visible prizes awarded to the biggest scoops and the most ambitious investigative stories. There are surprisingly few awards for other types of journalistic excellence. An organization that gave out awards every year to the reporters who did the most rigorous yet accessible technology journalism could help raise the prestige of this kind of reporting and give the best technology reporters more clout within their own newsrooms.

One of the most important journalism nonprofits of the internet era has been ProPublica, which hires reporters to do in-depth investigations and then partners with mainstream publications to publish their findings. Partnering with existing large publications ensures that ProPublica’s findings reach a large audience. ProPublica focuses on ambitious monthslong investigative projects, but I could imagine a similar model working for other types of journalism, including stories explaining how technology works and how it’s likely to impact the world.

I think it might be helpful for the reporters doing this kind of work to write for two sets of readers, with a specialist newsletter that covers day-to-day developments on their beats as well as occasional pieces that get syndicated to a mainstream audience. The goal of the newsletter wouldn’t be to attract a broad audience, but rather an influential audience that could provide feedback, story ideas, and potential sources.

Vox.com has a section called Future Perfect whose journalists are supported by a number of charitable foundations. At first glance it seems odd for nonprofit foundations to pay the salaries of reporters in for-profit newsrooms, but the advantage of this approach is that the reporters become fully integrated into their host newsroom while being insulated from the commercial pressures their colleagues face.

The best sources are willing to talk about a fairly wide range of topics, are good communicators, and are highly responsive to email. Sources like this are especially hard to find in computer science. The most knowledgeable technologists often work for big companies that don’t let their employees talk to reporters. Academics are able to talk to reporters, but in my experience it’s harder to find computer scientists willing and able to do so — perhaps because being quoted in the press is considered less prestigious in computer science than in other fields like law or economics.

So I’d love to have a program that paid competitive salaries for articulate technical experts to spend a year or two at a university or think tank serving as a source for reporters working on technology stories. Alternatively, this could be structured as an award program, with reporters nominating their most valuable expert sources for cash prizes.

How Silicon Valley can help

One irony of the growing antagonism between Silicon Valley and the media is that Silicon Valley created the economic environment that has driven the transformation of media over the last 20 years.

When journalists write clickbait headlines about technology companies, their goal is often to improve their ranking on Google, Reddit, and other social media sites. Reporters who write shoddy stories about technology companies often do so because their editor didn’t give them enough time to do in-depth reporting. And often that’s because the publication is in a precarious financial position because Google and Facebook have soaked up a large share of advertising revenue that previously went to media companies.

No one specific person or company is to blame for this. The invention of the internet probably made it inevitable that the media industry would become more competitive and therefore less profitable. But design decisions by tech giants have certainly made the problem worse over the last decade. And people in Silicon Valley could be doing more to help if they wanted to.

In the last section I offered some ideas for how philanthropists — in Silicon Valley or elsewhere — could promote better technology journalism. I think similar ideas would likely work for other complex beats like law, science, or medicine.

But the people in Silicon Valley who could do the most good are those at Google — especially the ones who run Google News. Google is still one of the largest sources of traffic for news sites. The Google News algorithm doesn’t encourage clickbait the way Facebook’s News Feed did during the mid-2010s. But Google also doesn’t seem to work very hard to identify and promote exceptional journalism in search results.

If Google started doing this — and being transparent about how they are doing it — it might create a financial incentive for news organizations to invest more in quality.

If you have financial means, you can get high-quality journalism by subscribing to expensive specialist publications. But democracy depends on the general public being reasonably well-informed. The poor quality of mainstream news sources is ultimately going to hurt everyone.

Source Link