Checklist for AI Auditing
Algorithmic auditing is a way to inspect AI systems in their specific contexts. It is an approach and methodology that allows for a dynamic appraisal of regulation, standards and impacts. If its results are public, it is also a tool for transparency and accountability.
AI audits are key tools for regulators and society, who can use audit reports to assess how systems work and their impacts. But they are also useful for those developing and acquiring AI systems. An end-to-end, socio-technical approach like the one proposed here generates documentation that improves system accountability, organizational memory and compliance with AI and data regulations.
For those acquiring and incorporating AI systems into their operations, audits provide crucial evidence that enable due diligence and proper assessment and comparison of the characteristics between different systems and vendors.
The AI audit checklist proposed is specifically focused on AI impacts. This means that while it gathers information on compliance and trust and safety mechanisms both before and after an AI system is launched, the focus of the audit is to validate that AI developers and implementors have taken all necessary measures, at all different stages, to make sure that the impacts of their systems are in line with existing laws, trust and safety best practices and societal expectations.
It should be noted that an audit process in the framework of the controller implementation of the accountability principle and the inspection/investigation carried out by a Supervisory Authority could be different. Such differences rely, among others, in the final purpose of both activities (the SA could do the inspection to get evidence of an infringement), the scope (limited to the GDPR: applies on personal data processing activities but not on technologies) and the national regulations regarding inspection by control authorities.
This document includes a methodology in the form of a check-list to perform an audit of an AI system. We define an AI system as a logic with a specific outcome. An AI system may be composed of several algorithms, and an AI service or product may include several AI systems.
In addition, it should be noted thatthere are different techniques for developing artificial intelligence. This document is focused on auditing an algorithm for artificial intelligence based on machine learning, where its life cycle is divided in three totally independent stages from the point of view of data processing and these stages are: algorithm training (pre-processing), the operation of the algorithm implementing one (or more than one) operation in the framework of a personal data processing (inprocessing – inference) and the decision making and impact of the same in the processing (postprocessing – model deployment). It could be a fourth one, that is the algorithm evolution. All of those stages could be different processing activities and could involve different controllers.
Views: 8
