Study: Dr ChatGPT tell me what I want to hear: How different prompts impact health answer correctness
As AI becomes increasingly integral to our daily lives, its ability to provide accurate and reliable information, particularly in sensitive areas such as health, is under intense scrutiny. The study conducted by CSIRO and The University of Queensland researchers brings to light the nuanced ways in which the formulation of prompts influences ChatGPT’s responses. In the realm of health information seeking, where the accuracy of the information can have profound implications, the findings of this study are especially pertinent.
Using the Text Retrieval Conference (TREC) Misinformation dataset, the study precisely evaluated ChatGPT’s performance across different prompting conditions. This analysis revealed that ChatGPT could deliver highly accurate health advice, with an effectiveness rate of 80% when provided with questions alone. However, this effectiveness is significantly compromised by biases introduced through the phrasing of questions and the inclusion of additional information in the prompts.
Comments are closed.