FRC’s thematic review of audit sampling
The Financial Reporting Council (FRC) has recently published its thematic review of audit sampling. Although the review was conducted of Public Interest Entity (PIE) audit registered firms, the outcomes of the sampling review should still resonate with all ICAS audit firms. We outline the main findings of this review and summarise what can be learned by ICAS firms.
The key takeaways for ICAS audit firms
While we have included the FRC’s main findings, a short summary of the findings most relevant to our firms is noted below:
- Key professional judgements over sampling were often not being executed or documented well.
- Assurance was placed on other audit procedures to reduce sample sizes without sufficient justification of why sufficient assurance could be derived from that other work.
- This reduction in sample size often resulted in too small a sample size.
- The judgements over the selection of key items required improvement. Often this was size driven, without explaining whether/why this particular size measure or approach was an appropriate threshold for the test.
- Haphazard sampling was used frequently rather than random sampling, leading to potential sampling bias and a failure to project errors accurately. Although haphazard sampling is permissible in the context of the International Standards on Auditing (ISAs), the FRC advocates the use of random sampling.
- There was audit team confusion over dual-purpose tests, resulting in monetary value samples being too small.
These findings are broadly consistent with those identified by ICAS Audit Monitoring as summarised in our 2023 ICAS Audit Monitoring Report. The additional issue of capping sample sizes that we find regularly on monitoring visits does not feature in the FRC findings, given large firms generally no longer allow sample capping.
Scope of the review
The audit sampling guidance and methodology of seven Tier 1 audit firms was the focus of this review and the three key areas within scope were:
- Methodologies relating to sampling as described in ISA (UK) 500 Audit Evidence and ISA (UK) 530 Audit Sampling as a means of selecting samples for tests of detail.
- Sampling methods deployed in testing information produced by the entity (‘IPE’) and attribute testing.
- Sampling methods deployed in tests of controls.
Summary of sampling approaches
Most of the firms reviewed make use of mandated internally developed tools to execute their sampling methodologies:
- Sample size calculators – These range from simple spreadsheet-based tools to more complex bespoke solutions.
- Monetary unit sampling (MUS) tools – These tools are used at some firms to help the semi-automated use of MUS and require similar inputs as more general sample size calculators, but will typically select a sample automatically for the engagement team to examine.
The FRC found that key professional judgements used in the approach to sampling were not well evidenced, despite being one of the biggest factors driving the quantum of samples used. This was especially the case where engagement teams relied on assurance gained from other procedures with very little explanation given as to why the team believed it had derived sufficient assurance from this other work. This often substantially reduced the number of items selected, with instances of teams selecting too few items to be able to conclude.
There was also confusion as to the function of testing IPE and the use of attribute testing. Some engagement teams did not understand that IPE testing assesses the reliability of the information to be used as audit evidence, rather than being a test over the monetary value of a population.
Sampling for test of details
In instances where engagement teams overestimate the amount of evidence obtained from other procedures, such as substantive analytical procedures, data analytics and tests of controls, they are likely to select too few items to allow them to conclude.
The confidence level (CL) that an engagement team is required to meet is a matter of professional judgement and is not defined in the ISAs (UK). Given the principles-based nature of the ISAs (UK) and the application of judgement, there is variation across the audit firms as to the required CL for different risk levels.
Given the importance of this key professional judgement on the sample size calculator, audit firms should ensure that they provide audit teams with sufficient guidance to support professional judgement in this area.
Key items selection and selecting specific items
All of the methodologies reviewed allow engagement teams to select and test key items from the population before then selecting a sample of the residual population. All of the sample size calculators reviewed allowed for the removal of key items tested elsewhere.
Audit Quality Review (AQR) comments in this area noted that in several reviews, there was insufficient documentation of the reasons for selecting items either as key items when audit sampling, or as specific items. When there was justification, it generally focused on size, such as “selecting everything over 50% of PM”, with no consideration or documentation of why that was an appropriate threshold.
Given the significance of these judgements, the FRC encourages firms to communicate the importance of recording judgements and to consider if their methodologies would benefit from additional guidance.
Haphazard sampling
A sample selected haphazardly rather than randomly has a greater risk of bias.
The reviews identified instances where sample calculators incorrectly stated that the sampling method was random, but haphazard sampling was actually used by the engagement team. This led to potentially inaccurate projection of errors and to improper consideration of bias in the sample.
In multiple reviews, there was no documentation or consideration of why haphazard sampling would be the most appropriate method when random was clearly a plausible option and would have reduced bias.
Although haphazard sampling is permissible in the context of the ISAs (UK), the FRC considers that firm methodologies should actively encourage the use of random sampling over haphazard sampling wherever possible.
All firms involved in this review agreed that they will consider amending their guidance to ensure random sampling is clearly labelled as the preferred method over haphazard.
Sampling methodologies for IPE and attribute testing
Dual-purpose testing is where a sample is selected for both IPE or attribute testing and additional procedures to obtain assurance over the monetary value of the population are undertaken.
IPE and attribute testing sample sizes are generally lower than those required to conclude on a population’s monetary value. IPE or attribute testing need only be performed on the number of samples required for that purpose, with the monetary value also being tested on these samples. Only the monetary value will be tested on the remaining samples.
Though most firms explain this concept clearly in methodologies, AQR findings indicate that engagement teams appear to struggle with practical application and are sometimes unclear on the dual objective of their tests.
For example, this led to teams only selecting the smaller number of samples and testing the monetary value on that sample, without including the additional samples needed to be able to conclude on the monetary value of the population.
Controls testing and sampling
Three of the firms have predominantly used the American Institute of Certified Public Accountants (AICPA) Audit Sampling Guide and three other firms have used the guide, as well as aspects from another statistical model, such as the Poisson probability distribution model, to inform their controls testing sample sizes. One firm, however, does not base its sample sizes on a statistical model.
When testing a control operating multiple times a day, sample sizes ranged from 10-60. Two firms allow the audit team to design their tests allowing for one deviation when testing controls operating multiple times a day. However, this requires an increase in sample sizes from 25 items to 40.
For less frequently operating controls, sample sizes are broadly in line with each other across the firms, as well as with AICPA guidance:
- Weekly: AICPA suggested 5-9 Range of all firms 4-11
- Monthly: AICPA suggested 2-4 Range of all firms 2-4
- Quarterly: AICPA suggested 2 Range of all firms 1-3
As with audit sampling in substantive testing, the application of appropriate professional judgement is the key to ensuring the effective use of audit sampling methodology in test of controls. Firms should ensure that engagement teams understand the importance of appropriate professional judgements and are able to evidence their judgements appropriately.
Audit firms should also remind engagement teams that the prescribed sample sizes are minimum levels.
Sampling and International Standard on Quality Management (ISQM) (UK) 1
All of the firms’ methodologies were driven by a global methodology. Almost all of the firms reviewed had additional UK-specific material to:
- Address either ISA (UK) requirements (where they are higher than the international version).and/or;
- Respond to specific inspection findings at a firm level. Three firms heavily relied on their global methodology teams to address the FRC’s questions. While support from the global central functions is appropriate, the FRC concluded that audit firms need to ensure they have a full understanding of the sampling techniques developed globally and are able to understand and apply those methodologies in the UK.
The FRC’s final observations and recommendations
FRC Observations
- Audit sampling for tests of details and controls is still prevalent, despite the advent of tools such as audit data analytics (ADA).
- Most firms’ methodologies are based on similar statistical models with firms building on these with their own guidance and preferences. This has led to substantial variation in the firms’ final methodologies.
- When applying methodology in practice, professional judgement is key. Judgement is needed to use firms’ sample size calculators, including to assess inherent risk and determine the contribution of evidence from other procedures. The extent of firms’ guidance to support these judgements is variable.
- There is insufficient evidence of the key professional judgements made when determining sample sizes. Evidencing these key judgements is vital.
FRC Recommendations
- Audit committees should understand how auditors obtain audit evidence to support their choice of auditor when tendering and to aid understanding of how their auditor undertakes the audit.
- Audit firms should ensure that they provide engagement teams with sufficient guidance and training to support their use of professional judgement in audit sampling.
- All audit firms should update their methodologies and guidance to drive better documentation of key judgements made when determining sample sizes.
