Security, Privacy and Change Management Risks

Security and Privacy Risks

The challenges of Al are not limited to the risks of algorithmic bias. The design and deployment of Al technologies often come with increased risks in security and privacy. Today’s Al systems involve complex processing of structured and unstructured data using extensive libraries of algorithms; thus, they are often interconnected with a great variety of third-party applications, systems, and devices (mobile computing, Internet-of-Things). This higher level of sophistication and interconnectivity makes them more vulnerable to potential misuse of administrative privileges, mismanaged access controls, malicious code, or attacks.

Besides, in big data, Al systems involve much more massive amounts of data than ever before. As Professor Joshua Kroll of the University of California Berkeley (2018) points out, an increasing number of business decisions are made automatically. These business decisions are driven by systems that employ machine learning, data analytics, and Al to derive decision rules using data instead of having humans code the rules. This shift to data-driven systems and software-mediated processes creates new data governance requirements for organizations.

Professor Kroll highlights, for example, that any data collected and retained pose some risk of breach or compliance issues with privacy laws or other hazards. He suggests that the data collected should be limited to only what is essential. Further, he recommends avoiding retaining data once they are no longer business-critical, secure the data at all times from outside hackers, and manage potential deliberate misuse by insiders (i.e., data should always be encrypted at rest and in transit). He also suggests that the retained data could be scrubbed and aggregated to a lower level of sensitivity. Certain categories of data

Challenges and Ethical Considerations of Al 89 could be identified, such as personally identifiable information (PII), and be treated safely. Other recommendations from Professor Kroll include:

• • Form a cross-functional Data Use Review Board within the organization to approve or deny the collection of new data, investigate sensitive questions using company data, and the deployment of related insights. Beyond the risk of a data breach, the Board would assess the risks of legal noncompliance and company reputational damage because of the use of the data with the input of outside experts and panels of trusted customers.
• • Review data collection and analysis practices periodically for risk assessment and provide the organization with formal social impact statements to communicate internally about the risks and the techniques or procedures to mitigate them.
• • Ensure that the organization can explain what the data-driven processes are doing and how decisions are reached (such as by providing adequate explanations or documentation of these processes)
• • Perform ongoing audits, including evaluating or testing systems for potential undesirable biases and designing systems that facilitate this type of review.

Change Management Risks

Al systems can be inaccurate, make mistakes, or even malfunction because of changes introduced in them or changes in their environment. Examples of change management risks include:

• • A change in the application code itself:, a correction or an enhancement introduced in the application triggered an error or defect in the system.
• • A change in the training data set: , a change made to the training data set resulted in defective machine learning, leading to inaccurate (or biased) results.
• • A change in another system with which the Al application is interacting: that is, a change in a third-party algorithm, third-party application (e.g., enterprise resource planning, accounting), application programming interface (API), or IT infrastructure system caused an error or a malfunction in the application.

Besides, the absence of change may also be the cause of the system’s deficiency: For example, an omission or a delay in updating the systemor its training data set following a change in industry rules and regulations or new compliance requirements could cause severe problems. Delays would be particularly critical in the world of accounting and tax, where rules, regulations, and compliance requirements are particularly complex and change frequently.

As for security and privacy risks, because Al systems are more complex and sophisticated, and they are interacting with many other systems and devices, they are subject to increased change management risks. Increased risks represent a significant challenge for the deployment and maintenance of these systems. For example, RPA systems or bots often break due to changes in their environment (software upgrades, system integrations, compatibility with infrastructure, etc.).

To mitigate these risks, it is critical that organizations plan for adequate resources to operate, control, monitor, and maintain Al systems beyond their implementation. Failure to budget for such resources might result in the failure of the initiatives.