Virtual and Augmented Reality Tech Joins the Fight against Crime

Tucked away in a small corner of South Wales, United Kingdom, police officers are using a larger-than-life technology to tackle domestic abuse. Some 1,200 frontline officers from Gwent Police Force have already used a "virtual reality cave" to interact with a virtual victim, honing skills to identify controlling and coercive behavior.

The Cave Automatic Virtual Environment comprises three screens to give a wraparound virtual experience for a growing number of exercises, including vital "golden hour" training to increase rape convictions and a virtual custody suite complete with detainees, detention officers, nurses, and lawyers.

"We've got plans to build ten scenarios," says Superintendent Vicki Townsend. "Constables, sergeants, and detectives have all used [the cave] and the feedback has been wholly positive."

Gwent Police Force is hardly alone in its foray into the virtual world. The rise of VR and AR to fight crime follows years of worldwide research by forensic scientists, many of whom have developed prototype systems for reconstructed crime scenes. The Netherlands has been a hive of activity in this arena with computer scientists, systems engineers, and police professionals collaborating on a host of projects.

One such project known as CSI: The Hague launched at the turn of this decade and set out to digitize the crime scene. At the scene of any crime, all traces of evidence must be identified and secured as quickly as possible. With this in mind, the project turned to augmented reality to create a virtual version of the crime scene that would always remain intact so forensic investigators could revisit it time and time again.

To create the virtual crime scene, a first investigator at the actual crime scene would wear a head-mounted display receiving 3D video from attached cameras, controlled by a laptop in a backpack. In this way, he or she could see the surroundings, but also overlay and virtually tag objects using hand gestures. Blood stains and bullet holes could be documented, and projectile trajectories reconstructed, to build a clear record of the scene in 3D video before evidence was removed.

As Professor Stephan Lukosch from the Delft University of Technology recalls, "We had mounted two depth cameras onto a Carl Zeiss Cinemizer headset and used a simultaneous localization and mapping (SLAM) algorithm to [analyze] motion. Our end result was far from a detailed 3D model that you could run around and interact with, but back then it was really good."

A later project, "On the Spot," brought in the Port of Rotterdam fire brigade to collaborate with police in a number of virtual scenarios including a forensic investigation and ecstasy lab discovery. 3D video footage of the crime scene was streamed from an on-site police officer's body camera to a remote forensic expert.

The AR system, also using a SLAM algorithm, hand-tracking, and gesture recognition, allowed the remote expert to augment the scene with notes, geometric shapes, arrows, and more. In this way, the local police officer could view these virtual annotations via a smartphone or a head-mounted device and use them to collect the evidence at the actual crime scene, as directed by the expert.

However, according to Nick Koeman, Strategic Innovation Advisor from The Netherlands' National Police, the technology still wasn't mature enough in 2015. Visual tracking software couldn't keep pace with more complex tasks, and the Google Glass-type headsets weren't usable under direct sunlight. But, he adds, "We realized the potential of augmented reality here."

Fighting crime with "On the Spot": user interface for a remote investigator in a forensic investigation scenario.

And so Koeman and Lukosch, in collaboration with Dragoş Datcu from the AR organization Twnkls, started work on a third relatively low-key project, ARPro. This time, field officers would head to a potential crime hotspot, such as a suspicious building. Once there, and thanks to a localization system developed by Twnkls, they would instantly receive notes and images from previous investigations at that site to better assess the situation.

Having only recently ended, the project was not as technologically sophisticated as past AR setups, but as Datcu highlights, he and colleagues can reach tens of thousands of officers, massively enhancing collaboration. "The Dutch police has some 40,000 operational agents all with smartphones, and we can inject the right information to the right people at just the right moment," he explains.

What's more, Datcu is confident a good app for fighting crime will reach police forces in a few years. "This will be used to take images, extract meaningful evidence about markers, clues, and traces, store the semantic annotations related to the crime scene, and then present this when needed to the user," he says. "It could even detect license plates, recognize faces, and process fingerprints, giving you in-depth information about crime hotspots within moments."

Lukosch also believes that state-of-the-art mixed-reality smart glasses, such as Microsoft HoloLens, and virtual retinal display Magic Leap One will soon herald systems with a better user experience.

And for Koeman of the Dutch Police, AR and VR are the future. "Our ambition is to integrate these technologies and enhance training and our operations," he says. "We might want to act out an evacuation in a fire scenario in a government building, and we can't really light up Parliament just for a training exercise, can we?"

Beyond the scenes

Just as AR and VR infiltrate shady criminal worlds, the technologies are also beginning to tackle terrorism. On 7 April 2017, in a Stockholm shopping center, a hijacked lorry was deliberately driven into crowds before being crashed into a department store. Five people were killed, and fourteen others seriously injured.

Philip Engström, SPIE Member and head of the sensor technology group at the Swedish National Forensic Centre, has reconstructed the terrorist attack. Using a cutting-edge 3D laser scanner from the 3D measurement company FARO (Florida, USA), he carried out around 100 laser scans and collected some 800 million data points from the scene to create several videos from different perspectives. His work was broadcast in a Sweden courtroom.

"The prosecution wanted to show what it was like having this truck drive at 60 km an hour towards you," he says. "The video didn't include people, was as clean as possible...but everything was clearly based on fact."

The Swedish National Forensic Centre has been developing 3D sensing technology for crime scene investigations for a decade, with Engström now heavily involved with its VR development. Like The Netherlands researchers, he has been experimenting with VR headsets as well as using real-time streaming 360º cameras as data sources. Since acquiring the laser scanner in 2017, he has rapidly reconstructed some 15 crime scenes as well as the tragic Stockholm terror attack, and he is excited about the future use of VR to counter crime and terrorism.

"It's so easy to see the big role that virtual reality can play here," he says. "After discussions with the prosecutors who watched our video as well as the police, we know that many people think it has vast potential."

Indeed, big and vast aptly describe what is around the corner for AR and VR. Researcher and serious games developer Jonathan Saunders from the Centre of Excellence in Terrorism, Resilience, Intelligence and Organised Crime Research at Sheffield Hallam University, United Kingdom, concurs, and has been instrumental in developing a platform to train law enforcement agencies to respond to terrorist and organized crime threats.

As he puts it, "Augmented and virtual reality are still experimental but very useful for when you've got to make critical decisions under pressure, say during a terrorist attack or even a crisis such as a flood or earthquake."

His platform is part of the pan-European project AUGGMED—Automated Serious Game Scenario Generator for Mixed Reality Training—which set out to enhance the 3D simulation tools already used in police and paramedic training for extremely complex scenarios. "Simulations evolve depending on what the users do," explains Saunders. "During training, one user could choose to start an evacuation immediately, which has a very different effect than exploring a scene and trying to find the suspect first."

AUGGMED players can directly interact with agents via hand gestures or spoken commands to take control of a situation in a way that resembles a real-life scenario.

AUGGMED's key training pilot featured a terrorist attack at the Port of Piraeus in Greece. Here, HTC Vive VR headsets, with additional cameras to extend the field of view, provided video footage rendered with VR explosions, weapons, and hostages. And, as Saunders adds, "We also used custom-haptic feedback vests that would heat up if you walked too closely to the virtual fire, and you'd feel a thump on your chest if you were shot."

With the mediated reality platform now deemed a success, commercial development could ensue, but in the meantime, Saunders is already working a new Advanced Training, Learning, and Scenario (ATLAS) simulation for the United Nation's International Organization for Migration. Commissioned after two civilian staff were killed during a convoy attack in South Sudan, the simulations are training staff to respond to life-threatening situations.

"This time, we've used methods to reduce the motion sickness experienced by those using the VR headset," highlights Saunders. "A post-processing effect on the camera—virtual reality tunneling—subtly reduces peripheral vision by slowly darkening the edges, and stops you feeling uncomfortable and unwell during motion."

Given the rapid progress, Saunders is sure both VR and AR will feature in crime and the fight against terror.

Back in Gwent, Superintendent Vicki Townsend agrees. "I think ultimately this is something that we all need to do," she says. "Virtual reality is giving us the best opportunity to deliver consistent quality training, and the sooner we can take this forward, the better."