A test of ChatGPT’s sample mock questions and technical responses found those relying on it for exam support are at risk of serious repercussions
Artificial intelligence (AI) has already had a significant impact across the financial services industry.
Its applications range from basic tasks, such as taking minutes or drafting emails, to higher level complex functions, such as enhancing a firm’s security.
I want to explore the issue around using AI for technical support, focusing on its readiness to provide reliable resources for candidates taking their financial services exams.
As candidates progress through the exam syllabus, they will encounter challenging subjects that require more clarification and alternative explanations. They will also look to assess their knowledge and prepare for exams by tackling mock questions.
None of the answer options provided by the AI were correct. It’s clear it had confused DB and DC schemes while generating the choices
With the emergence of free platforms such as ChatGPT, candidates are increasingly seeking assistance from AI. To gauge its efficacy, we decided to test ChatGPT by requesting sample mock questions and posing technical queries. Here are our findings.
Firstly, we asked it to: “Create five multiple choice questions to test my knowledge of the CII R03 exam syllabus”. (This is the diploma level 4 CII exam covering taxation).
The initial issue in the results was that all questions focused solely on pensions not taxation, which, though mentioned in R03, certainly isn’t the core of the exam. The first question and answer produced was as follows:
“When comparing defined benefit (DB) and defined contribution (DC) pension schemes, which statement is true?
A) DB schemes typically provide a guaranteed income based on contributions and investment growth.
B) DC schemes promise a fixed income throughout retirement, unaffected by market fluctuations.
C) DB schemes offer flexibility in contributions and investment choices.
D) DC schemes offer a predetermined income based on salary history and years of service.”
Option A was marked by ChatGPT as the correct answer, which is inaccurate. DB schemes offer a guaranteed pension based on salary and length of service. In fact, none of the answer options provided by the AI were correct. It’s clear the system had confused DB and DC schemes while generating the answer choices.
The questions asked about the 2023/24 tax year, but the answers were relevant to the 2022/23 tax year
While the other four questions did produce technically correct content, they were too basic to be worthwhile.
For example, ChatGPT asked: “Which of the following is NOT a type of DC pension scheme?” Answer options included GPP, Sipp, DB scheme and DC scheme.
Candidates might hope for such straightforward questions in their diploma-level exams, but the reality is they are never going to be this easy. In fact, questions this simple are unlikely to appear even in the level 2 CII exams.
Next, we tried being more specific with our prompts to ChatGPT to see if we can get it to focus the questions on taxation. We asked:
“Create five mock multiple-choice questions which would test my knowledge of the CII R03 exam syllabus, specifically capital gains tax (CGT) in the 2023/24 tax year”.
In this example, the results were worse – three out of the five questions generated were wrong. For example:
“What is the rate applicable for CGT in the UK for higher rate and additional rate taxpayers in the tax year 2023/24?
A) 10%
B) 20%
C) 28%
D) 40%”
ChatGPT gave the correct answer as 28% (C). It looks to us as though the answer may be referring to the CGT rate applied when there is a gain on residential property rather than general gains – however, residential property was not mentioned in the question. The AI is clearly not able to differentiate this nuance.
Relying solely on AI to prepare for an exam could lead to severe repercussions
The errors with another two of the AI generated questions were that the questions asked about the 2023/24 tax year, but the answers were relevant to the 2022/23 tax year. Again, it’s clear the AI isn’t able to differentiate between tax years within this format, therefore making any generated content significantly less useful during exam prep.
It was clear from the mock questions we generated that there are many inconsistencies and errors in the material, so we turned our attention to asking ChatGPT a number of technical questions.
The same problems persisted. In the main, the explanations were clear, however there were sporadic errors, the main issue being that out-of-date tax rates and allowances were used.
For example, we asked: “How do you calculate CGT in the UK for individuals in the tax year 2023/24?”
A clear step-by-step explanation was provided, however the CGT annual exempt amount (AEA) was given as £12,300. This is incorrect, as the AEA amount reduced to £6,000 by 2023/24.
Without a good understanding of the subject, it’s difficult to discern the accuracy of the response provided
It was interesting to see how far AI has come, but also concerning that candidates may now be relying on the information produced through sources such as this.
We know firsthand the process required to provide accurate, reliable exam support material. All our resources go through a rigorous three-point check when created, are technically reviewed and updated annually, and then additional checks are carried out outside of the annual review process to check the consistency, difficulty levels and technical content.
Each individual on our technical team is hugely experienced in their chosen subjects and some are also CII examiners. It’s not surprising that, when AI systems pull information from all over the internet on highly technical subjects without any secondary checks in place, there are errors to be found.
So, while some of the basic mock AI exam questions may yield questions and answers that are technically correct, their scope is considerably narrow and often unsuitable for most CII exams.
Relying solely on AI to prepare for an exam could lead to severe repercussions.
This is not to dismiss their potential usefulness entirely. For instance, posing a prompt such as, “Explain how the pension annual allowance works” might provide valuable insights. However, the inherent challenge lies in the fact that, without a good understanding of the subject, it’s difficult to discern the accuracy of the response provided.
Catriona Standingford is managing director of Brand Financial Training
For more practical information on how to get started in financial advice, please visit our Future Financial Adviser community. We tell you how to take the first steps, how to get ahead and what attributes you need to flourish. Even better, we can put you in touch with helpful contacts.
Comments