Generative Artificial Intelligence (AI): Limitations

Large language models (LLMs) cannot "understand" user input; they can only identify linguistic patterns and imitate them.

By default, language models optimize the next word prediction objective, which is only a proxy for what we want these models to do.

ChatGPT is fundamentally a text transformer—not an information retrieval system.

Consequently, LLMs will sometimes output text that appears credible but has no factual basis. In particular, LLMs have a known tendency to cite non-existent sources in convincing APA style. Even when citing real sources, LLMs may paraphrase them inaccurately.

If a prompt is ambiguously phrased, LLMs may (wrongly) guess user intent rather than asking clarifying questions or admitting that they do not "understand" what is being asked. Indeed, they can be "confidently wrong".³ This is, perhaps, because LLMs are optimized to provide answers satisfying to human users, who are biased in favor of confident responses over doubtful or noncommittal ones.²

ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers. Fixing this issue as challenging as [during reinforcement learning training], there's currently no source of truth.

• AI increasingly relies on paradigms, such as deep learning, in which the developers themselves do not understand how their models make decisions.
• For particularly complex models, such as LLMs, it becomes impossible to document and reproduce research methods.
  • As for humans learning from experience, an AI's methods are more comparable to intuition that to a set of instructions that can be followed or reproduced.¹
• For example, if we use an LLM to conduct a literature search, we cannot document the search strategy or reproduce the results:
  • For most LLMs, the same prompt will produce a different output each time.
  • We cannot determine or appraise the criteria by which articles were included or excluded.
  • This is antithetical to the principles and practice of systematic search, which is a cornerstone of evidence-based practice.

As machines, AI systems may give the false impression of being impartial and objective, but they are the products of the data inputs used to train them, which were created by human beings, and the choices of the developers, also human beings. Thus, AI is subject to many of the same biases and errors as humans are.

Some are concerned that existing datasets underrepresent particular sociodemographic groups¹, which, if used as training data, may result in inequitable AI models. It may be possible to counter this issue by use of carefully selected training data.² Racial and political biases have been observed in the outputs of ChatGPT.^3-5

We conclude that chatbots cannot comply with the Health Insurance Portability and Accountability Act (HIPAA) in any meaningful way despite industry assurances.

• Inputting even deidentified data into a chatbot may give the developers enough information to make inferences about a patient's health or a clinician's prescribing practices.
  • Such information is extremely valuable and can be sold to advertisers and data brokers.
• Experts can use AI to reidentify data.
• Prompting is a two-way street—chatbots prompt the user with follow-up questions.
  • The FTC has expressed concerns about the ability of chatbots to gain undeserved trust from users.² Healthcare providers may be tempted to divulge more than they originally intend.

The answer is to use chatbots sparingly despite the temptation to flood them with clinical information.

Chatbots (such as ChatGPT) should not be listed as authors because they cannot be responsible for the accuracy, integrity, and originality of the work.

The general consensus among prominent scientific publishing organizations^1-5 is that AI models cannot be credited as authors because they cannot be held accountable for their statements. Human authors alone must accept the responsibility of authorship. Simultaneously, presenting the output of an AI model as one's own work is unethical and compromises the integrity of your research.

Therefore, if you use these tools in your research:

• Use them to support and compliment your own efforts, not as a replacement for them.
• Declare that you have used AI tools, including which ones and how you used them.

• Outputs are often unnecessarily verbose and repetitive.
• Due to attempts at moderation of harmful content by the developers, LLMs will sometimes be excessively cautious.
• Conversely, because the developers' moderation practices cannot account for everything, some harmful outputs will still be possible.