Researchers from George Washington University have published a paper examining whether the time and cause of AI going rogue can be predicted; and if predicted, prevented.
Since 50% of the world’s population carry devices that can run personal AI companions with no internet connection and limited security, they focused their research here. The lack of cloud-based safety filters, real-time telemetry, live monitoring, or the ability to patch weights once deployed provides a good test bed for analyzing AI’s chat-style transformer behavior when left to its own devices.
If there is a tipping point, it will stem from the AI’s Attention head. This is the computational component that determines which earlier AI tokens are most relevant when processing the current token. The decision lays the foundation for the next token and so on until the chatbot has finished responding to the user’s prompt. The token is loosely, but not precisely, related to individual words or parts of a word. Different tokens have different weights – an indication of the importance of individual tokens.
The primary argument is that the forward motion of tokens can slip from good to bad (that is, go rogue) due to competition in the Attention head between the conversation’s context and competing output basins. A conversation’s accumulated context can gradually shift Attention toward an undesirable basin until a tipping point is crossed and the model begins producing bad outputs.
Since the primary driver in a conversation is the user’s prompt sequence, it follows that both thoughtless and malicious prompts can hasten the AI’s slippage into bad outputs. This can be either immediate (one bad prompt) or delayed (the accumulated effect of poor prompts).
The researchers (Neil Johnson and Frank (Yingjie) Huo) have further developed a mathematical formula designed to estimate the tipping point represented by the number of good outputs that occur before the first undesirable output appears. Once that first bad output occurs, the AI is on the slippery slope to roguery since it starts to influence future tokens in unintended ways. Alignment with purpose is lost, and the AI can be categorized as ‘misaligned’.