Audit Scotland: Improving audit quality with data science

Abstract

Audit Scotland is appointed by the Auditor General for Scotland and the Accounts Commission to perform audit services for most of Scotland's public organisations. An auditor must determine if the accounts presented by an organisation represent a true and fair view of their financial position. A detailed, methodical exploration of the audit client's general ledger will assist the auditor in coming to this conclusion. Advances in the private audit sector in general ledger analysis have been made possible thanks to the streamlined nature of many small/medium enterprise (SME) accounts packages and the accounting frameworks that apply. Public audit financial management systems are more diverse in nature, as are the accounting frameworks that apply across the sectors, therefore no commercially available ledger analysis tools have been successfully adapted for use in public audit. Audit Scotland had introduced a rudimentary ledger analysis tool, in the form of a Microsoft Excel add-in, which was struggling to cope with the volume of data to be processed for effective analysis. It also failed to meet standards for reproducibility and documentation required by data analytics tools used in an audit. The introduction of an ETL (extract, transform and load) data pipeline using data engineering principles during this project has increased Audit Scotland's capacity to ingest and prepare general ledger data from public organisations, ready for analysis. Additionally, a web application for general ledger analysis (Asc) has been created to enable Audit Scotland financial auditors transparent, efficient access to the general ledger data of their clients. An important aspect of audit work is journal risk assessment. Each journal in the ledger should be assessed and classified as 'risky' or 'non-risky' in terms of causing a material misstatement in the accounts. With some public organisations creating upwards of a million journals per year this is an impossible manual task for an Audit Scotland audit team. Attempts at producing unsupervised machine learning classification models for this task by the author, thus far, have proved unsuccessful in accurately classifying journals thanks, in no small part, to an ineffective evaluation method caused by a lack of labelled data and a lack of resources needed to evaluate the model’s output. Within Asc, a journal risk assessment module has been developed which allows an auditor to manually classify all journals more efficiently through the lens of optional risk factors. This human expert classification has been captured to produce a data labeller for public audit ledger data, opening future possibilities to train and test supervised classification models, using labelled data, whilst providing a useful journal risk assessment tool for use in Audit Scotland now. Using data visualisation tools and automated reporting within Asc, the general ledger can be examined to a significantly higher level of transparency. Auditors are reporting more confidence in their audit decisions, and the evidence they can provide to support those decisions, using Asc. Increased confidence, backed up by documented decisions lead to a higher quality audit which is the motivation for this work.