Limitations of Large Language Models

ChatGPT, which unbelievably has only been around since November 2022, was already hugely popular when Dr. Ali Chaudhry proposed a joint event on generative artificial intelligence with the Leeds Institute for Data Analysis (LIDA). On December 7th, 2023, the Scientific Machine Learning at Leeds group (SciML), Applications and Implications of Artificial Intelligence (AI2) forum, and the Generative AI and Reinforcement Learning Community hosted a hugely relevant event on Large Language Models (LLMs), offering attendees from the University of Leeds and the wider community the chance to engage with two generative AI experts.

The evening began with a talk from Dr. Serge Sharoff, a Professor of Language Technology. His research interests are related to three domains: linguistics (primarily computational linguistics and corpus linguistics), cognitive science and communication studies. This field of study has been at the forefront of Serge’s focus since the end of his PhD in 1997. The most interesting bit in his recent research is digital curation of representative corpora automatically collected from the Web, i.e., their annotation in terms of genres, domains or linguistic properties. Teacher’s may be concerned that students will use generative AI models such as ChatGPT to complete their homework, but how much ChatGPT can help may depend on the language the homework is written in. Serge highlighted a potential bias that may limit the use of ChatGPT. The performance decreases as the proportion of texts in each language within the training dataset decreases, particularly when prompts are given in languages used in more limited regions.

Professor. Eric Atwell followed, his talk moving away from the more technical limitations of LLMs covered by Serge, and instead raised the point that ‘the main limitation is thinking up clever things to do with it’. Eric believes there are 2 main limitations: (1) LLM responses are combinations of snippets of training text which match snippets of input text, so they do not truly mimic imagination or creativity. Also, (2) the main limitation of use and application of LLMs is our own imagination: we need to think creatively about how to best use this new magic! He covered possible applications of LLMs from detecting fake news to simulating patient-doctor conversations to better train the next generation of doctors.

We really enjoyed these talks and would like to thank both speakers for their time and sharing their knowledge on LLMs. We hope that attendees now have sparked imaginations and view LLMs as exciting tools to utilise, rather than mysterious black boxes.

Blog written by: Eilish O’Grady and Zoe Hancox