Since its inception in November 2022, ChatGPT-3.5 has gained increasing popularity.^1,2 12ChatGPT is an artificial intelligence (AI) language application that performs online tasks and mimics human conversations using a question-and-answer format.³Expand Reference The interface has found widespread use in several industries, with applications ranging from programming, business, education and healthcare.⁴Expand Reference

To contextualize this new technology, it is important to differentiate the types of recent developments and terminologies, including AI, large language models (LLMs), natural language processing (NLP) and generative pre-trained transformers (GPTs). AI describes the ability of a computer or other machine to learn human cognitive skills, such as problem-solving, pattern recognition and reasoning. The development of the field of NLP has revolutionized our interactions with AI, as NLP focuses on understanding and generating language in a natural way that mimics human language.⁵Expand Reference LLMs are AI models that are trained to process large amounts of text data drawn from internet and other sources, including human text or speech conversations, which can help the AI better generate meaningful, human-like language as an output.⁶Expand Reference GPT technology, such as the popularized ChatGPT, is a type of LLM that specifically has a training phase before being released for widespread use, and it specializes in producing context-sensitive responses to user input. After its release, the bot can draw from the pre-training text sets and take advantage of the linguistic optimization of NLP to communicate effectively with human users by generating relevant, human-like text in an engaging and seamless way.^3,737

A recent review on the utility of LLMs in ophthalmic care reveals their use in helping ophthalmologists, optometrists and ophthalmological researchers diagnose eye conditions and provide information and management guidance. Notably, LLMs have been shown to perform in general ophthalmology topics more accurately compared with ophthalmological subspecialities, mostly focusing on image-based analysis.⁸Expand Reference Prior studies have explicitly tested ChatGPT’s performance in answering board-style questions and found it to be fair to good.⁵Expand Reference Specifically in ophthalmology, a recent study by Mihalache et al. found that ChatGPT correctly answered about half of the multiple-choice questions for ophthalmology board certification preparation.⁹Expand Reference As patients today have more access to medical information than ever before and as ChatGPT grows in popularity, it is anticipated that patients will increasingly use this interface to seek healthcare information previously sought from clinicians. Other fields have performed studies to assess the utility of ChatGPT in answering patient questions, with each study yielding varying results.^10–12101112 However, thorough research into the limitations of ChatGPT in ophthalmology patient care has yet to be conducted.

In this study, our aim is to investigate the ability of ChatGPT to accurately answer inquiries about cataract surgery, the most commonly performed intraocular surgery. Our study comes at an opportune moment considering the rapid rise in popularity of ChatGPT since its inception, particularly because it is a highly convenient and readily available tool for patients. As mentioned, there is currently a modest but growing amount of literature on the use of ChatGPT in ophthalmology. However, this study on cataract surgery, in particular, is the first of its kind. It builds on and improves the methods of prior studies by expanding the analysis to not only include the topic of cataract surgery but also assess the most effective questioning technique to pose medical questions to ChatGPT to better inform its future utility. Our hope is that this study will better enlighten ophthalmologists about the scope of ChatGPT and help assess if it is a valuable tool in clinical care and patient education.

Methods

In this cross-sectional study, potential patient questions regarding cataract surgery were written by the authors based on what they knew to be commonly encountered in practice and then confirmed by several reputable ophthalmology websites.^13–18131415161718 The eight questions chosen were: ‘When should I consider cataract surgery?’, ‘How is cataract surgery done?’, ‘What are the risks and complications of cataract surgery?’, ‘How long does it take to recover from cataract surgery?’, ‘What medications should I take after cataract surgery?’, ‘Will I have any restrictions after cataract surgery?’, ‘What are the different types of lens implants used in cataract surgery?’ and ‘Will I need follow-up surgery after the cataract surgery?’ These questions were asked to ChatGPT version 3.5 on 28 August 2023.²Expand Reference To ensure consistency, the questions were entered by a single author. ChatGPT and other GPT models generate responses based on conversational context, suggesting that their responses could be influenced even by small differences in user inputs.¹⁹Expand Reference Thus, questions were asked to the AI bot in three different ways to assess variations in response depending on questioning techniques: all eight questions at once, four questions at a time and one question at a time. To prevent the browsing history from influencing subsequent responses, the study was conducted on a private browser, the web browsing history was cleared after each ChatGPT search and ChatGPT’s ‘chat-saving’ feature was disabled. No duplicate inquiries were posed to ChatGPT.

All ChatGPT responses were compiled and analysed individually by ophthalmologists-in-training. Each resident evaluated each of the responses in random order and was blinded to questioning techniques. The analysis was conducted using an evidence-based, validated 4-point scale adopted from a prior study.¹²Expand Reference A lower score on this 4-point scale indicated greater accuracy and completeness of response. A response score of 1 was given to responses that were ‘excellent’ and required no clarification because they were thorough and addressed all aspects of the question. Responses that were ‘satisfactory’ and required minimal clarification due to not being detailed enough received a response accuracy score of 2. Responses that were ‘satisfactory’ and required moderate clarification due to vague, outdated or irrelevant information were provided a response accuracy score of 3. Finally, responses that were ‘unsatisfactory’ and required significant clarification due to extremely general or incorrect information received a response accuracy score of 4. Final scores were calculated by averaging the scores given by all evaluators for each question and each questioning technique. The evaluator agreement was computed using Fleiss’ kappa.^20,212021 The differences in the scores by group based on questioning techniques were analysed using the Kruskal–Wallis test.²²Expand Reference

Institutional review board approval was not required for this study. Patients/public were not involved in the design or conduct of this study

Results

Table 1 presents all eight questions along with the average score of all evaluators for the answers that ChatGPT provided after being prompted using the three different questioning techniques: (1) all eight questions at once, (2) four questions at a time and (3) one question at a time. On average, when questions were asked as stand-alone inquiries, the generated ChatGPT responses earned higher evaluator scores compared with ChatGPT responses produced from the other two questioning methods. In contrast, responses produced when questions were asked all at once earned, on average, lower evaluator scores. As shown in Table 1 and Figure 1, when all eight questions were posed at the same time, the average evaluator score was 2.60. When four questions were posed at once, the average evaluator score was 2.15. Finally, the average evaluator score provided when questions were asked one at a time was 1.03. The differences in scores between the three questioning technique categories proved statistically significant (p<0.001). Additionally, evaluator scores returned a Fleiss’ kappa value of 0.55, indicating moderate agreement beyond chance alone (p<0.001).

Tables 2 and 3 show transcripts of the responses provided by ChatGPT when asked two of the eight questions (see Supplementary Material 1 for transcripts for the remaining six questions). As shown in Table 2, when asked the question, ‘What are the risks and complications of cataract surgery?’ in the setting of all eight questions, ChatGPT’s response correctly identifies that there are risks to cataract surgery, despite it being a generally safe and uncomplicated surgery.²³Expand Reference It lists some of the most common complications, consistent with those stated in reputable literature and online sources.^14–18,23141516171823 However, there is a lack of depth in this extremely brief answer, which may leave patients wondering about details regarding these complications and whether they are common. Most important, ChatGPT refers the patient to their surgeon for further discussion on the topic, which is necessary to obtain detailed and personally relevant medical advice. As shown in Table 2, the average score by evaluators for this questioning technique was 3.00.

Similarly, the ChatGPT response to this question in the setting of four total questions correctly identifies the risks of cataract surgical procedures, with a more extensive list of common complications than the previous answers, including the most common late complication of cataract surgery–posterior capsule opacification.^23,242324 Additionally, this answer provides more context on how common these complications are, informing the patient that they have a ‘low’ likelihood of occurring.²³Expand Reference However, there is still a lack of depth on what these complications entail. Again, the AI correctly refers the patient to their surgeon for more information and discussion on the risks and complications of the surgery. As shown in Table 2, the average score by evaluators for this questioning technique was 2.20.

When asked the question alone, ChatGPT’s response is more thorough and comprehensive than the previous two answers, providing the patient with a more extensive list of potential complications, along with an accurate explanation of each complication. This can be highly useful to patients who have no or limited medical knowledge and/or prior experience with intraocular surgeries. Throughout the explanations, ChatGPT identifies the likelihood of risks with words such as ‘rare’ and ‘extremely rare’, which can orient and reassure patients. In contrast to the previous two answers, this response also accurately includes precautions taken by surgeons, such as using sterile equipment to avoid post-operative infections, as well as monitoring for excessive bleeding during surgery and for glaucoma after surgery.^23,252325 Moreover, the AI even accurately provides the patient with the next steps in resolving such complications in order to help them understand what they might expect if any of these complications occur. The response appropriately touches upon medical management for swelling/inflammation and dry eye, as well as yttrium aluminum garnet (YAG) laser capsulotomy for posterior capsule opacification. YAG laser capsulotomy is a surgical intervention that uses laser to create an opening in the opacified capsule, thus helping to clear patient vision.^23,24,26232426 ChatGPT’s response also correctly mentions the improved quality of life due to improved visual acuity as the outcome for most patients who undergo cataract surgery.^27,282728 Additionally, once again, ChatGPT importantly refers the patient to their surgeon for personalized discourse of the risks and complications of cataract surgery in the context of their own risk factors and medical history. As shown in Table 2, the average score by evaluators for this questioning technique was 1.00.

As shown in Table 3, ChatGPT’s response to the question ‘What medications should I take after cataract surgery?’, when asked at the same time as all other eight questions, correctly states the functions of the major medications prescribed after cataract surgery but fails to name or provide details regarding the medications. As a general practice, patients are prescribed prophylactic antibiotic medications to prevent post-surgical infections and endophthalmitis, a serious complication that can lead to vision loss or even eye loss.^29,302930 Additionally, because cataract surgery can induce an intraocular inflammatory response that could lead to cystoid macular oedema and visual impairment, patients are generally prescribed steroid eye drops following the surgery.^29,312931 Thus, this ChatGPT response is a fair but insufficient starting point, as it is rather brief and lacks detail. As shown in Table 3, the average score by evaluators for this questioning technique was 3.60.

When asked this question within a set of four questions, ChatGPT takes it one step further than the previous response by providing more concrete answers as to what types of medications the patient can expect after surgery. Again, this includes the antibiotic eye drops and steroid eye drops, mentioned earlier to prevent post-surgical infection and inflammation.²⁵Expand Reference It additionally mentions artificial tears, which are important in preventing symptomatic dry eye, a highly common complication after cataract surgery secondary to eye incision, eye inflammation and disturbance to the natural tear production of the eye.^32,333233 In this response, the types of medications are also followed by accurate general descriptions of their indications. Although what is stated is accurate, this list is still limited in terms of the types of medications. Nonetheless, ChatGPT correctly states that the patient will receive more detailed instructions on their post-operative medications from their surgeon, who should always give specific advice regarding dosage and length of use based on the medical needs of the patient. As shown in Table 3, the average score by evaluators for this questioning technique was 2.20.

When asked this question alone, ChatGPT’s response is much more expansive than the previous two responses, providing an extended list of the medications the patient may likely be prescribed and setting general expectations for the medication regimens. This extended list not only includes the antibiotic eye drops, steroid eye drops and artificial tears mentioned previously but also adds pain relievers to help minimize discomfort.^29,33,34293334 Here, ChatGPT opens the list to other ‘prescription medications’, which removes the limits of a finite list and can prepare patients to be potentially prescribed medications that their surgeon deems appropriate in the patient’s particular case. ChatGPT also correctly prepares patients to carefully check the medication dosages and frequencies based on what their surgeon decides for them, and it even warns patients to look out for side effects that may require medical attention. This is important as ChatGPT recognizes that, although it provides helpful context for post-operative medication use in terms of the most common time course, patients should always defer to their surgeon. As shown in Table 3, the average score by evaluators for this questioning technique was 1.00.

Discussion

In summary, our study demonstrates that ChatGPT can provide basic, data-backed answers to medical questions about the pre-operative, operative and post-operative aspects of cataract surgery, making its increasing usage over the past years less daunting and more hopeful in the context of medical advice. However, there are significant changes in answer length, depth and quality depending on the questioning approach, with isolated questions receiving higher evaluator scores, indicating that they required significantly less clarification than the other approaches. Questions asked on their own elicited more extensive, exact explanations from the AI bot; however, questions posed within a collection of other questions generated a curtailed version of responses, likely leaving users with more follow-up questions. Notably, all ChatGPT responses were generic and devoid of patient context, including medical history and personal risk factors, decreasing confidence in its clinical utility. ChatGPT recognizes this limitation, specifically its lack of medical training, and, importantly, refers patients to their surgeon for further personalized discussion. While evaluator scores for each response to a stand-alone inquiry received a score of 1 most of the time, indicating excellence, patients would nonetheless need to speak directly with a surgeon to address individualized questions that ChatGPT does not have the proficiency to answer.

Our findings align with those from prior studies on the value of ChatGPT in ophthalmology and general medicine. Momenaei et al. revealed that when asked questions concerning vitreoretinal illnesses, such as macular holes, retinal detachments and epiretinal membranes, ChatGPT’s answers were appropriate most of the time.¹¹Expand Reference Our study similarly demonstrated overall appropriateness and accuracy in the ChatGPT responses but primarily when questions were asked in isolation. They also raised the issue of readability, as some of the responses were regarded as too complex for laypersons to comprehend.¹¹Expand Reference We similarly believe that the average layperson may find difficulty in understanding the ChatGPT responses, as some responses offer too much detail and create information overload. When consulting with an ophthalmologist, the patient would be able to ask clarifying questions about imprecise terminology or confusing language. Additionally, an ophthalmologist would also be able to examine their patient’s disposition to gauge comprehension and the need for further explanation. ChatGPT lacks this discernment, and despite its comprehensive replies, it will most likely never be able to match the level of care that can be offered by a physician for this reason alone.

Additionally, in another study, Biswas demonstrated that ChatGPT consistently produced ‘good’ quality responses to inquiries about myopia, including its causes, symptoms, treatment and prognosis.⁴Expand Reference This was determined by a panel of optometrists and research staff. However, they did draw attention to the fact that ChatGPT often provided inaccurate responses, calling into question the reliability of the tool.¹⁰Expand Reference The concerns they raised are validated by the fact that ChatGPT formulates responses based on internet sources, articles and books, and it may be difficult for the AI to decipher fraudulent websites and sources.¹Expand Reference ChatGPT and other GPTs have been shown to have the potential to mislead users due to their robust and seemingly comprehensive replies, as ChatGPT demonstrated the ability to provide justifications and explanations for incorrect responses to ophthalmic questions.^3,12312 Although we show that ChatGPT’s responses to our questions were supported by literature, we ultimately share this sentiment, as the spreading of misinformation can be especially dangerous when it comes to patient health and medical understanding.

Within the broader scope of ChatGPT usage in medicine, this AI ChatBot has been shown to produce detailed ophthalmic discharge summaries and operative notes, although these notes often required fine-tuning by humans to correct for inaccuracies.³⁵Expand Reference While this may reduce physician trust in ChatGPT for sharing medical information, it is nevertheless encouraging for ChatGPT’s future, as it suggests that with rigorous domain-specific pre-training, the AI may have sufficient ability to have a meaningful and practical niche in medicine in the future. Furthermore, current research indicates that many of the medical websites that patients often utilize to get medical knowledge score poorly on authorship metrics.³⁶Expand Reference Based on the findings of these and other studies, we can envision a more advanced future with generative AI that can sift through these websites, validate their facts and present accurate information to patients in seconds, removing the need for patients to struggle through confusing medical websites on their own.

This study is not without limitations. The concern with insufficient responses when presented with multiple questions at a time is likely not unique to ChatGPT; rather it is highly probable that the same phenomenon occurs in the doctor’s office. While physicians can use their human discernment to decide on when to provide more explanation or ask if the patient has any more follow-up questions, this is still a limitation to note. Future studies could benefit from having a human control model to assess how differently physicians would perform compared with LLM in answering patient questions based on questioning techniques. Another limitation is that, although this pre-trained generative model does not necessarily ‘learn’ or change its core model base in real time, there is still some short-term memory storage with each new user input that allows ChatGPT to fine-tune its language and interaction with the user based on recent conversation.^3,19319 Thus, the limitations of this study include ChatGPT user bias, as its responses may be influenced by recent chat history, despite our efforts to control for this by using a private browser and disabling the feature that saves the chat history for model adjustments. There is also a limitation due to the bias of self-reported scores, which are subjective based on the scorer’s personal experiences and understanding of patient knowledge. Additionally, the evaluators in this study were ophthalmologists-in-training who were not experienced cataract surgeons. It is possible that ChatGPT responses may have been graded differently if our chosen evaluators were experienced cataract surgeons. Moreover, as ophthalmologists-in-training with limited clinical exposure, the evaluators may have unintentionally introduced bias due to their lack of variation in practice environment and influence from a particular hospital system. We envision this study as pilot, laying the groundwork for future investigations that could explore how perceptions of AI vary across different levels of training (i.e. residents versus attendings). Although we attempted to cover a wide range of topics through the eight questions we examined in this study, the choice of the questions is also limiting in that they are not wholly representative of all patient questions about cataract surgeries. Moreover, as shown in this study that different questioning techniques affect the response, the wording of the questions may also have an effect on the ChatGPT responses, an influence not deeply examined in this study. The prompts that were used here were curated by medical professionals and do not necessarily reflect the way a layperson would casually enter a search query. On this note, future studies that investigate the effects of question wording on ChatGPT responses are encouraged. Furthermore, as ChatGPT has proven to be an effective tool in answering the cataract surgery questions in this study, similar studies should be pursued for other surgeries and procedures, using not only ChatGPT but also other AI models. It must be emphasized that ChatGPT is not originally designed for medical, and much less ophthalmic, purposes. Thus, the pre-training text and datasets were not specifically selected to focus on health topics such as cataract surgery. Therefore, the results and conclusions of this study apply specifically to ChatGPT version 3.5, and they cannot be extrapolated to all other LLMs, including future models that may in fact be designed more intentionally for the purposes of patient care and education.

Conclusion

This study investigates the utility of ChatGPT in answering frequently asked patient questions about cataract surgery. Although ChatGPT was often able to provide basic responses to questions, the questioning technique significantly influenced the quality and subsequent utility of these answers, with questions asked one at a time prompting responses that were deemed more comprehensive by evaluators. Given the real-life variability in patient question styles and the clear influence of question style on ChatGPT responses shown in our study, we conclude that ChatGPT should be limited to functioning as solely an adjunct to patient care and education regarding cataract surgery alongside medical expertise from ophthalmologists.