Before we celebrate, let’s read carefully.

A large field experiment compared AI voice interviews to human recruiters across 70,000 applicants for entry-level roles ( Jabarian, B., & Henkel, L. (working paper). Voice AI in Firms: A Natural Field Experiment on Automated Job Interviews. )

Headline results:

+12% more job offers

+18% more job starts

+17% higher 30-day retention

Impressive.

But here’s what’s I revealed when I dug deeper:

• The study was conducted with a recruiting vendor and one of the researchers later got a non paid position at the vendor's company.

• It ran in the Philippines, where candidates may be less likely to refuse assigned hiring processes compared to some Western markets.

• In the AI group, 5% refused to speak with AI and 7% failed due to technical issues. That’s roughly 1 in 8 candidates lost because of the AI setup.

• Recruiters knew which interviews were AI-led. They gave those candidates higher scores and extended more offers.

• “Hiring quality” was inferred mainly from short-term retention.

Does this invalidate the research? No.

Does it prove AI interviews are universally superior? Also no.

What it really shows is this:

Structured, standardized interviews improve signal quality.

AI enforced structure and consistency.

But structure is not magic. It is design.

The lesson for HR leaders is simple:

When you hear extraordinary AI results, read beyond the headline.

Understand the context.

Understand the tradeoffs.

Understand what was actually measured.

At Informed Decisions, we use AI intentionally:

• To transcribe and structure interviews

• To surface interviewer behavior patterns

• To detect potential bias signals

• To connect interview data to post-hire performance

We do not use AI to replace judgment.

We use it to make judgment measurable and improvable.

AI is powerful.

But hiring excellence comes from critical thinking, not automation alone.