Disarmament in Times of War: Interview with Ryan Gariepy

It comes down to the concept of collateral damage. We already know that to a certain degree machines have been making life and death decisions in war in areas where there are no civilians. But when it comes to the grayer areas of operating around civilians, making assessments of possible collateral damage, or what is an appropriate and proportional use of force, then I do believe that is morally objectionable if there is no chain of responsibility.

I would not necessarily say that it is about the machines per se as it is about abrogating the responsibility of a human or humans in making those decisions. It is easy to imagine a world where, when fatalities or excess civilian casualties occur, everyone just puts up their hand and says, it is just code, and we cannot do anything about it. It was the fault of the machine; therefore, nobody is at fault. But this goes against a lot of historical precedents and foundations of our society that people should be held accountable for their decisions.

Something that has come to my mind in the last few years is that this discussion of LAWS is also part of a broader discussion about when people are okay with a robot using some sort of force or agency on them. This leads to the more general question about the type of force that is being used. Lethal force is the ultimate degree of force, but there is a whole spectrum of other possible forces. There is less than lethal force. There is gently pushing someone.  There is collecting data about someone. There is simply existing in someone’s personal space.

It also comes down to what is the dynamic between the person deploying the robot and the person having that robot deployed on them. These two fundamental factors affect how a given autonomous system will be accepted. A civilian confronted with a faceless military machine that is employing lethal force is the ultimate example of a hostile dynamic. Conversely, a neighbor using a drone to inspect their roof around their own neighbors is likely to be considered acceptable. There are intermediate scenarios, too. For instance, the military using a drone for disaster response or search and rescue is likely to be accepted. Police using a drone to surveil an underrepresented population in their area would likely be less accepted. We have already seen examples of all these situations play out.

The way that I look at humans in the loop is that in the end the humans decide. They make the go or no–go decision, or they make the decision to employ a weapon or not; whereas, a human on the loop is merely observing things occurring.  But at least they are observing something occurring.  Obviously, there are certain nuances, for instance, regarding the question of how deep the human is in the loop. Is the human in the loop in the sense that they have approved a weapons system to be used? Or are they observing or approving every single action of that weapon system? For example, in automated weapons systems such as short–range point–defense, such as close–in weapon systems (CIWS) like Phalanx, or anti–missile systems like Iron Dome, humans are in the loop, because they enable and disable these systems. They are observing — on the loop — what is going on, but they are not in the loop in a way that they are controlling every single shell or missile that is being fired. 

This leads us to the concept of meaningful human control. You could say that if there is a human in the loop, who is ready to interrupt the weapon at any given point, then the weapon is technically under ‘human control’. However, this is not meaningful human control. If there is a human in the loop or on the loop, they need to be involved in such a way that it is accommodating the various limitations of humans in terms of reaction times, attention spans, and a range of other different factors.

To use an example that does not relate to weapons: there was an incident some years back with a safety driver for a self–driving car that was deployed on a public road. The safety driver was just expected to sit there and monitor for hours on end, intervening if necessary. The driver was not paying attention at an appropriate time and the car hit and killed someone crossing the road. That was not meaningful human control. It is impossible for a person to pay attention for this long in those conditions, and mitigating controls should have been put into place.

In my opinion, not really.  Since I became involved in these questions in 2014, there has always been a consensus that these conversations need to happen. But the concept of meaningful human control requires a lot more collaboration. It requires discussions on what is meaningful human control for an anti–missile system such as Iron Dome or Phalanx.  It requires discussions about meaningful human control for the growing amount of anti–drone systems that are likely to be deployed. And it requires discussions about the advancing space of armed ground based mobile robots.

There are all sorts of different discussions necessary, because, as I noted, it is not just that you have approved a single weapon to be used or not. It is also about in what space? How much authority does that weapon have? How much force are you allowing it to use? What is the area you are allowing it to deploy force within? For how long are you allowing it to operate? All of these aspects are part of that meaningful human control discussion.

These conversations need to be had. It requires more than just political discussions. It requires deep collaboration among people with real world military experience, with engineers and scientists, with people who have a background in what we would now call AI risk, with civil society, and with policy makers.

There will always be pressures to do so, but it is up to all of us to decide how we want to conduct war. I am not the kind of person to say that war can be outlawed or anything like that. We have got a lot of human history that makes it clear that armed conflict is not going away, but, at the same time, we have found ways to make it a little bit more civil over the years.

It is also up to us to decide what kind of new technology to deploy. Yes, military officials do want the ability to collect and assess information quickly from multiple sources. Yet I would not imagine that military officials want the systems to just react without considering their own military experience.

I do not have direct military experience myself, I am a roboticist. But having spoken to a great many people about this topic who do, there is universal agreement that it does not take much to teach someone how to employ a weapon, you can teach eighteen– and nineteen–year–olds how to do that. But that is no substitute for decades of military experience with using weapons in a context of how and where, particularly when civilians are involved.

That almost sounds like we are saying that we just cannot fight any more. I think the currently ongoing conflict in Ukraine is a very clear indication that the prospect of technological armed conflict is not something we can simply ignore. There is still a lot of tension out in the world. It is very unrealistic and naïve to say that we can just disarm and that that will lead to reduced military expenditure, reduced conflict, and reduced militarism.

But it is true that advanced and autonomous technologies are spreading and are being developed faster and faster. Treaty–based strategies are only one aspect of reigning in these developments. There are other aspects such as state–level regulations, norms, and industry discussions.

For example, the emergence of large language models has resulted in an incredibly fast regulatory process. This proves that, when governments believe a risk exists, they can move quickly. Globally, every major country has developed or is in the process of developing an AI act. Are they perfect? Absolutely not. But it is proof that regulations can move fast. I do hope that members of governments do reflect a bit on the fact that far less sophisticated models put in full control of weaponized robots pose the potential for great harm.

It was only about a month or two after the most recent Ukraine conflict broke out in 2022, that members from the private sector, government, and academia who have been involved in these discussions in the past started having various conversations about the conflict. Can we afford to sit in North America and say that countries should not be able to develop technologies to defend themselves? What is the reasonable thing to do? For instance, what should be done against a hypothetical swarm of drones that is designed to target dismounted infantry? The best defense against that is probably not another swarm of drones designed to target dismounted infantry, but probably a system that targets drones itself. When the cannon was developed, the answer was not to build other cannons that shoot the cannonballs out of the air. However, fortifications were built differently from there on out.

What is becoming increasingly clear in the Ukrainian conflict is the degree of capability that modern, low–cost robotics can have. But what we are seeing is more about low–cost and high–volume manufacturing and innovation than it is about the true use of autonomous weapons systems. Those drones are not doing anything particularly fancy. Moreover, the Ukrainians are defending their own soil and that is not the place to let a drone operate that is fully capable of using lethal force autonomously. If anything, there is a stronger impetus to make sure that there is meaningful human control being enforced.

The most important thing that we can do in the private sector is to really educate. To educate on the capabilities, on the risks, on what is considered doable, on what is controllable. And educate on the potential benefits or harms of the dual–use nature of the foundational technologies. A good example occurred in 2015 when a few experts, including myself, were at the United Nations in Geneva and brought up the very early incidences of adversarial attacks on convolutional neural networks. Nowadays, with the potential to jailbreak large language models, the problems have become much more complex. It has become clear that, much like with more basic approaches to deep learning, there are tradeoffs between robustness, predictability, and flexibility, which need to be navigated. These engineering topics are issues that the private sector is uniquely suited to address.

[1] The views expressed in this interview do not reflect the official policy or position of Rockwell Automation.