Yes blindly trusting it is not a good idea, but for complex issues it's very good at taking reams of data and synthesizing it into likely diagnosis. Far better than humans in my experience, probably mostly because humans don't have the time to review your full medical history or case history on an issue: AI can hold all that and examine it together. It's made very important diagnoses for me at least. As long as there's a real doctor in the loop, it works great as a second opinion to fill in gaps that the doctor may have missed.
It can be used successfully but you need to instruct it to only rely on science that can be shown in studies, just tell it the facts, give it your data, and ask a non-leading question, and finally confirm with a doctor. Research I did with frontier models when my parents were in hospital likely saved both of their lives. I think the HMO employed doctor (same one both times) was actively trying to kill them. Using them against my own medical records turned up ideas that my doctors missed but agreed to when brought up.
That's not hard. Nelson Mandela, Václav Havel, José Mujica, Kim Dae-jung. I lived in Czechia under Havel and it was certainly the greatest moment in history I will ever experience. It still gives me hope for humanity.
MMmmmm, more crocodile tears for me to lick up!!