How We Built an AI Tool a Journalist Could Trust

In September, we presented a project at The New York Times for the Hacks/Hackers AI x Investigative Journalism conference. It started with a reporting question from Sony, an investigative journalist who covers the wellness industry. 

She wanted a way to analyze the often-controversial recommendations coming from the internet’s most popular longevity influencers, including podcasters like Andrew Huberman, Mark Hyman and Peter Attia. 

But analyzing their podcasts amounted to 531 hours of audio. Nobody can review that much by hand, so the natural instinct is to say "just use AI."

But "just use AI" is not a spec. It doesn't say what the AI should look for, how it should decide, or how anyone would know whether it got the answer right. For journalists in particular, that last part matters most. 

So we built the tool together, one decision at a time, and kept Sony's judgment in the loop at every step. What we learned is that people trust an AI tool's output when they can see, and shape, how it got there. Confidence in the process is confidence in the product.

Why We Couldn't Just Ask a Chatbot

Across the three shows, there are over 30 million characters of transcript, several times what even the largest AI context windows can hold. 

Asking a chatbot to find every supplement recommendation is a needle-in-a-haystack search. The answers come back coarse and imprecise, and there’s no reliable way to tell what the model caught and what it missed. For a reporter who has to stand behind every number, that’s not good enough.

So instead of asking AI for answers, we used it to build a dataset Sony could inspect, filter and verify herself.

How We Broke the Problem Down

At a high level, the process looked like this: break the content into small units, send each unit to an AI model, and have it label what's inside. Then turn those labels into something a reporter can sort, filter and check. 

Step 1: Choose the Unit of Content

The first decision was the core building block of this project: What counts as a unit of content? 

An episode may be the obvious choice. But these episodes run as long as two hours and wander across dozens of topics. If you ask an AI model what a two-hour episode is “about,” you get a summary too broad to be useful. Episode length also becomes a confounding variable, because a model will tend to find more tags in a longer episode simply because there’s more content to tag.

We chose one minute instead. Every unit is the same size, so how often something gets tagged reflects how often it came up, not how long the episode ran. We split all 531 hours into minutes and sent each one to Google’s Gemini API, asking it to label the products and topics mentioned.

Step 2: Define the Taxonomy

Labeling every minute produced more than 14,000 tags. So we still had a problem of abundance. 

A lot of the tags overlapped. For example, 'omega-3' and 'fish oil' are related enough to be interchangeable for our purposes. To solve this, we built a simple interface that allowed Sony to review and merge tags. This created the final taxonomy we used in our analysis.

Step 3: Add Context

Tags tell you what was mentioned, but not how it was mentioned. Take these two quotes, both tagged "magnesium":

• "Each chew also includes vitamin C, vitamin D, magnesium, and zinc…"
• "…sometimes it's as simple as just giving someone magnesium because 45% of the population is low in magnesium…”

The first is a sponsor read. The second is the host's own advice. If we counted every magnesium mention the same way, the dataset would blur the line between what a host is paid to say and what he believes. 

So we expanded the taxonomy to pair each tag with key context. Those context dimensions let Sony ask much sharper questions of the data, like which supplements a host recommends on his own, outside of sponsored segments.

Step 4: Filter Down to the Story

Put the tags and their context together and you get a structured dataset that can be organized and filtered. That makes it much easier to spot patterns worth reporting on. How many recommendations are the influencers making? How often do they mention certain products, such as GLP1s? 

We didn’t need to build our own analysis tools for this part. General-purpose AI tools are already good at analyzing a clean, well-structured dataset. Our job was to get 531 hours of audio into a form those tools could handle. From there, Sony could export the data and layer on new reporting questions: How much would it cost the average person to follow these “stacks”? How often do influencer recommendations go beyond, or even contradict, medical guidelines? When are recommendations for prescription medications considered “off label”?

But finding a promising headline isn’t the final step.

Step 5: Fact-Check Backwards to the Source

For any story to be published, it needs to be thoroughly fact checked. So we asked how a fact-checker would approach it. Under traditional editorial standards, a fact-checker goes back to the original transcript or recording and confirms every claim.

Answering that by hand would mean searching hundreds of hours of audio for every product in the final table. Instead, we built a feature that works backwards. It starts from each recommendation, pulls up every mention and endorsement behind it, and takes the reporter directly to the source minute.

Confidence in the Process = Confidence in the Product

Increasingly, reporters rely on AI tools to expand the boundaries of what’s possible in their investigations.

But AI can feel like a black box. Data goes in, headlines come out. That’s a problem, because reporters need to trust the results of their investigations before they ask audiences to do the same.

What we built lets Sony find stories grounded in concrete analysis and verify every claim against the original audio. But the bigger factor in her trust was being involved every step of the way. She made the editorial calls that shaped the data, so she understood how every number was produced.

That applies well beyond journalism. Anyone who has to stand behind an AI tool’s output should be able to see, and shape, how it got there.

--

Sony Salzman is an investigative journalist who covers the wellness industry. You can find her reporting on LinkedIn and MuckRack.