Where My Ramblings Live

Among others, manipulative or subliminal messaging applications of AI fall under “unacceptable risk” in the AI Act, so to what extent is a recommender system subliminal or manipulative? If the system itself is not coded to be nor is it attempting to be manipulative, but is enabling and promoting manipulative and subliminal content in the course of its intended function, should it still be considered an unacceptable risk?

If we interpret the intent behind the law and the ‘unacceptable’ categorisation as being ‘to prevent AI systems from being used as manipulative tools’, then there is an implied responsibility for providers and deployers to prevent their systems being used in such a way, such that even AI systems being incidentally or unwittingly used in this manner should be considered within its remit. Although, to interpret the definition in this way risks outlawing recommender systems entirely, or at least obliging a degree of policing that would change their character fundamentally. The other interpretation is that only AI systems which are explicitly designed to be manipulative or subliminal can be considered unacceptable, but considering how ‘acceptable’ AI systems are easily capable of being exploited as amplifying platforms for manipulative and subliminal actors, it would become unnecessary to even commit to designing a manipulative AI system. Some middle-ground interpretation would be that incidentally-manipulation-enabling AI is permissible, but only to a certain degree. At this point, new questions arise – how much manipulation is a permissible amount? How much bad action can a deployer or provider of AI systems be expected to prevent before the expectation becomes unreasonable? What is the appropriate balance between “market freedom” and “human rights protections”?

There is a reading of Article 5 and Recital 29 that supports the first and third approaches, explicitly suggesting that the unacceptable AI system is ‘deployed with the objective to or the material effect of distorting human behaviour’. The ‘material effect’ line is significant, and suggests that the resultant outcome of the deployed AI system is more important than its intended outcome, so long as there is a causal link. However, it is also true that the AI system in question must ‘deploy subliminal techniques beyond a person’s consciousness or purposefully manipulative or deceptive techniques’. So in my opinion, there is a regulatory blind-spot here. Under the letter of the law an unacceptable system must:

1. Be intentionally deploying manipulative techniques

AND

2. Intentionally or not – result in a material distortion of behaviour

It is the contention that such an interpretation fails to account for the exploitation of already-established AI systems as vehicles for manipulation. A recommender system, for example, prioritises engagement metrics, and so is more inclined to recommend manipulative, misleading or subversive AI-generated content, so long as that content is able to drive sufficient engagement (e.g by being inflammatory or controversial). Under this interpretation, the recommender system itself is not engaging in manipulative techniques – it is simply recommending content – but the model is still able to be exploited and therefore turned into a vehicle for manipulative content which is shown to audiences who otherwise would not have been exposed to it but for the recommender system. So:

1. Intending or not – manipulation occurs as a result of the system

AND

2. There is a resultant material distortion of behaviour

This makes the recommender system a functional host for an otherwise unacceptable system. The obligations for GPAI and systemic-risk GPAI systems do include systemic risk-mitigation and monitoring obligations which would include such exploitation in their remit, but monitoring obligations are largely reactive; systems might be designed to mitigate manipulation after it occurs, rather than proactively preventing it. Conceivably, manipulative content can spread rapidly before being detected or it can be deployed at opportune moments (e.g an election month, a high-profile court case, etc), to have an outsized effect before detection. Further, systems that detect known manipulative techniques might be frustrated by novel strategies, risking a cat-and-mouse game or AI-based arms race where monitoring systems constantly lag behind evolving manipulation tactics.