Six Chatbots, 4,050 Medical Answers, and One Pattern Worth Memorizing

A study published in Annals of Internal Medicine this week did something unusually useful. Researchers took fifteen Choosing Wisely recommendations, the list of tests and treatments the medical specialty societies say are commonly ordered and rarely needed, built 225 clinical scenarios from them, and ran the lot past six commercial AI models three times each. That is 4,050 answers, graded.

The result was better than the alarmed version you may have read. Across everything, the models recommended an unnecessary test or treatment 5.4 percent of the time, and withheld care that was genuinely indicated just 0.6 percent of the time. In eight of the fifteen recommendations they got it right every single time.

Sponsored by StocksToTrade.com

They Said the Same Thing About Tesla

Dear Reader,

Has Elon Musk finally gone too far?

Since the SpaceX IPO, shares have struggled. The headlines have turned ugly. And the critics who’ve been wrong about Elon for 20 years are suddenly feeling very bold.

Sam Altman even called Elon’s new plan “ridiculous.”

But according to legendary investor Tim Bohen, the man who called Tesla before shares exploded 1,216%...

Elon is preparing to shock the world again.

He confirmed it himself…

“I see a path to SpaceX being worth orders of magnitude more than the current Earth economy.”
— Elon Musk

It’s why Tim has just released a free briefing with all the details of Musk’s bold new plan – and how to get positioned today without buying shares of SpaceX or Tesla.

Click Here to Watch 100% Free of Charge – No Credit Card Required.

Best,
Tim Bohen

P.S. Billionaire Peter Thiel once said: “I would never bet against Elon, in anything. That’s hard rule number one.”

The skeptics are lining up again. But you don’t have to miss out.

REVEALED: Free Ticker to Claim Your Stake in “Musk’s Mission X.”

Look Behind the Curtain

The interesting part is where the errors went, because they did not spread evenly. They piled up in two places.

Asked about a small incidental thyroid nodule, the kind found by accident on a scan, the models recommended an unnecessary ultrasound in nearly 43 percent of responses. Asked about a routine electrocardiogram for a low-risk adult with no symptoms, they recommended one in just over 28 percent. Strip those two out and the rest of the list sits between almost nothing and three percent.

What those two have in common is that they are exactly the questions real doctors argue about. Guidance says do not image the small nodule and do not run the routine heart tracing, and plenty of practicing clinicians do both anyway, for reasons that feel defensible in the room. The models were trained on all of that. So they are reliable where medicine has settled and wobbly precisely where medicine has not, which is a far more usable rule than "AI is good" or "AI is bad."

Three honest qualifications belong here. First, the headline 5.4 percent carries a confidence interval running from 0.6 to 12.2 percent, so treat it as a direction rather than a measurement. Second, the questions were put the way a clinician would put them, and the authors say plainly that the findings may not extend to consumer use. Third, they took no funding for this.

The mirror image matters too. False restraint, where the model talks a patient out of care they genuinely need, barely happened: 0.6 percent overall, none at all for eleven of the fifteen, highest for prostate-specific antigen screening at 3.3 percent. The machine's bias runs toward doing more, not less.

The Capability Multiplier

The leverage is in knowing which question you are asking.

When you want to understand what a word on your results page means, what a medication is for, what usually happens next, or what a procedure involves, you are asking a settled question, and that is the territory where these tools performed almost perfectly. Using one to turn a page of discharge instructions into plain English before an appointment is time well spent.

When the question is whether you should have a test at all, you have walked into the contested territory, and the tool tilts toward yes. Knowing that in advance changes how you read the answer. A suggestion to get the scan is the machine's default, not a finding about you.

So invert the prompt. Rather than asking whether you should get the scan, ask what the published guidance says about people in your situation, what the argument against the test is, and what to put to your doctor. Those three produce something you can take into the room. The first produces a recommendation nobody qualified has made.

Then bring it back to a person. The study scored answers against guidelines, not against the outcome for one patient, which is what your clinician is for.

The Sovereign Action

Use it to prepare, not to decide.

• Ask it to explain: results, terminology, medication purposes, what a procedure involves. That is the settled ground where it performed best.
• Do not ask it whether to have a test. The measured bias runs toward recommending one, and it runs strongest exactly where the guidelines are contested.
• Ask instead what the published guidance says, what the case against the test is, and which questions to raise. Take that list to the appointment.
• Treat an incidental finding carefully. The small thyroid nodule was the single worst category in the study, and it is a common accidental discovery.
• Check anything specific against a named source. The study tested guideline agreement, not whether the citations an assistant offers you are real.

We encourage all readers to consult a qualified financial professional — such as a registered investment adviser or licensed broker — before making any investment decision.