NIST’s Artificial Intelligence Risk Management Framework

Rebellion Defense, Inc.’s comments regarding AI RMF Playbook

Rebellion submitted the following comments to NIST as feedback on the AI RMF Playbook (https://www.nist.gov/itl/ai-risk-management-framework/nist-ai-rmf-playbook).

Comment No. 1

The Playbook covers too many potential issues, handicapping the ability of consumers and regulators to reach a common understanding of what an organization’s use of the framework does or doesn’t signal. Socio-technical factors should be restricted to local issues that can be assessed through user research and testing.

Since compliance with this framework is voluntary, in order to be effective it has to provide value to the organizations that elect to use it. One possible value add is the framework provides a level of assurance that following it will result in better outcomes and greater customer confidence as a result. Another possible value add is that using the framework can serve as evidence that any negative outcomes were not a result of negligence, deterring actions from regulators.

Unfortunately, the AI RMF Playbook describes a dizzying array of factors for organizations to do risk assessments on. The options are neither restricted in scope, nor time or stage of product life cycle. It would be impossible for any organization to account for all the factors recommended in the framework. Application of the framework, therefore, will be inconsistent across cases, even when the technology itself is similar.

While it’s good to be comprehensive, a framework that attempts to cover everything from algorithmic bias to carbon usage of computation is one where organizations can effectively cherry pick the criteria and risks they assess. No one will understand what an organization using this framework signals about the safety of their AI products and instead of helping create more ethical technology, NIST will be lending its brand out to whitewash other organization’s brands.

The AI RMF Playbook should focus on a smaller subset of criteria that can be assessed with user research and testing. While this will leave some important issues out, there is no reason why an AI framework needs to cover areas that other risk management frameworks already cover.

Comment No. 2

NIST should shift the emphasis from fairness, trustworthiness, and ethics–which focus on how the outcomes of technology feel to people–to safety which has decades of peer reviewed research informing its best practices.

One of the critical lessons learned from safety research is that when a system produces a negative outcome, people often have trouble separating the incidental conditions leading up to the event with contributing factors (1). The AI RMF Playbook’s wide scope of potential considerations means that failure can always be attributed to incompleteness in analysis. This is counterproductive to the aims of the framework.

Furthermore, many of the broader issues the Playbook includes are part of the early theoretical conversations around responsible AI that are not standing up to scrutiny in controlled research environments. Algorithmic fairness can produce worse outcomes for vulnerable groups (2). Explainability has little impact on how the human operator weighs recommendations from AI systems (3)(4).

The Playbook also falsely equates safety with testing when in practice safety standards are less about testing and more about communication. For example, safety standards in aviation mandate what issues are allowed to trigger alerts in the cockpit, how those alerts are allowed to interrupt and when they are allowed to interrupt (5). They use verification not to test system behavior, but to establish contracts between systems that determine what inputs and outputs can be passed between them and at what timescale. Although the end result is more reliable and predictable behaviors, those results do not come from tests, they come from structured communication channels and processes. We do not define system behavior by building a system and running it many times until we figure out what its likely behavior should be, we define what behaviors are allowed and build systems to eliminate as much undefined behavior as possible.

Safety researchers advise organizations to assess risk based on operator agency rather than controlling and restricting negative outcomes. Technology can be thought of as safe when the operator can correctly assess the risk of operating it in a given situation and can mitigate failure when operating it. Risk is determined by the extent to which the design of the technology interferes with either one of those features, not the probability that something might go wrong or the extent to which long term the technology might prove to have negative outcomes for society. After all, is it really fair to expect the manufacturer to attempt to control for circumstances where the operator deliberately chooses to use the technology to injure?

The point of safety is not to eliminate all potential negative behaviors, but rather to eliminate those that normal systems of accountability and due process would have difficulty resolving.

Comment No. 3

More emphasis should be put on how organizations design the partnership between the human components of the system and the AI components

We have structures and institutions in place to handle unethical and irresponsible behaviors from human operators. There is no reason to reinvent the wheel for AI. Instead, NIST should focus on ensuring human operators maintain agency within a computer system acting on their behalf.

The current draft framework helpfully recommends user research but does not take advantage of any of the proposed techniques to do so. For example, researchers in generative models have started to coalesce around describing the exact nature of interaction between human and machine with standard set of stages (6)(7). Using such stages to demonstrate how to diagram and document the design of human ai interaction as part of the Playbook would both make NIST’s guidance consistent with the practices of researchers and also leave less up to the interpretation of organizations.

Alternatively, other researchers have proposed using Shneiderman’s 2D matrix of human control -vs- AI control and mapping a process map on top of it (8). This removes the restriction of standardized steps but decouples the concept of human control from the concept of AI control.

The advantage of standard stages over the Shneiderman model is that it allows product designers to consult design patterns for inspiration. Most AI products are not starting from scratch but intended to serve as a replacement for an existing non-AI process. A core challenge with the redesign process is overcoming the anchoring bias created by the existing system (9), and the case of AI that skews many products towards human replacement over collaboration. A playbook of design patterns can help organizations find less risky Human AI interaction patterns that accomplish the same goals.

References:

  1. Dekker, S, et al. The complexity of failure: Implications of complexity theory for safety investigations. Safety Sci. (2011).
  2. Mittelstadt, Brent and Wachter, Sandra and Russell, Chris, The Unfairness of Fair Machine Learning: Levelling down and strict egalitarianism by default (January 20, 2023).
  3. Zana Buçinca, Phoebe Lin, Krzysztof Z. Gajos, and Elena L. Glassman. 2020. Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems. In Proceedings of the 25th International Conference on Intelligent User Interfaces (IUI’20). ACM, New York, NY, USA.
  4. Vivian Lai, Han Liu, and Chenhao Tan. 2020. "Why is ’Chicago’ Deceptive?" Towards Building Model-Driven Tutorials for Humans. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13.
  5. Barnes, V. E., & Monan, W. P. (1990). Cockpit Distractions: Precursors to Emergencies. Proceedings of the Human Factors Society Annual Meeting, 34(16), 1142–1144.
  6. Towards A Framework for Human-AI Interaction Patterns in Co-Creative GAN Applications, Imke Grabe, Miguel González-Duque, Sebastian Risi and Jichen Zhu Workshop on Human-AI Co-Creation with Generative Models (HAI-GEN 2022)
  7. A. Spoto, N. Oleynik, Library of Mixed-Initiative Creative Interfaces, 2017. URL: http://mici.codingconduct.cc/
  8. Michael Muller and Justin Weisz. 2022. Extending a Human-AI Collaboration Framework with Dynamism and Sociality. In 2022 Symposium on Human-Computer Interaction for Work (CHIWORK 2022). Association for Computing Machinery, New York, NY, USA, Article 10, 1–12. https://doi.org/10.1145/3533406.3533407
  9. R. Mohanani, I. Salman, B. Turhan, P. Rodríguez and P. Ralph, "Cognitive Biases in Software Engineering: A Systematic Mapping Study," in IEEE Transactions on Software Engineering, vol. 46, no. 12, pp. 1318-1339, 1 Dec. 2020,