Weapons detection for security and video surveillance

Summary
Public datasets
Published studies
Papers
Authors

There are many places where the crime rate caused by firearms or knives is very high, especially in places where they are allowed. The early detection of potentially violent situations is of paramount importance for citizens security. One way to prevent these situations is by detecting the presence of dangerous objects such as handguns and knives in surveillance videos.

Current surveillance and control systems still require human monitoring and intervention. We present a system of automatic detection of weapons on video appropriate for surveillance and control purposes.

Our proposal is the early detection of these weapons using deep learning techniques through video security in real time and minimizing the computational burden.

Different researching works carried out in this line are described below. We list the different image datasets built in these works in Section 2, as well as the proposed methods for improving the weapon detection system and results are illustrated in Section 3. Finally, the published works information is provided for citation and research purposes and the authors in Section 4 and 5.

2. Public datasets

The weapon detection task can be performed through different approaches that determine the type of required images. Therefore, the created datasets follow the image classification and object detection scheme and annotation including different objects:

Handguns
Knives
Weapons vs similar handled object

All dataset are depicted and public researching purpose, more information in provided in OpenData Weapon Detection.

3. Published studies

In an automatic weapon detection system the alarm must be activated when the system is completely confident about the presence of weapons in the scene. We describe below the published studies focussed on making a weapon detection system more robust by minimizing false positives and maximizing detection skills.

3.1 Automatic handgun detection alarm in videos using deep learning

Olmos, R., Tabik, S., & Herrera, F. (2018). Automatic handgun detection alarm in videos using deep learning. Neurocomputing, 275, 66-72. doi.org/10.1016/j.neucom.2017.05.012

This work presents a novel automatic handgun detection system in videos appropriate for both, surveillance and control purposes. We reformulate this detection problem into the problem of minimizing false positives and solve it by i) building the key training data-set guided by the results of a deep Convolutional Neural Network (CNN) classifier and ii) assessing the best classification model under two approaches, the sliding window approach and region proposal approach.
The most promising results are obtained by Faster RCNN based model trained on our new database. The best detector shows a high potential even in low quality youtube videos and provides satisfactory results as automatic alarm system. Among 30 scenes, it successfully activates the alarm after five successive true positives in a time interval smaller than 0.2 s, in 27 scenes. We also define a new metric, Alarm Activation Time per Interval (AATpI), to assess the performance of a detection model as an automatic detection system in videos.

3.2 Brightness guided preprocessing for automatic cold steel weapon detection in surveillance videos with deep learning

Castillo, A., Tabik, S., Pérez, F., Olmos, R., & Herrera, F. (2019). Brightness guided preprocessing for automatic cold steel weapon detection in surveillance videos with deep learning. Neurocomputing, 330, 151-161. doi.org/10.1016/j.neucom.2018.10.076

The automatic detection of cold steel weapons handled by one or multiple persons in surveillance videos can help reducing crimes. However, the detection of these metallic objects in videos faces an important problem: their surface reflectance under medium to high illumination conditions blurs their shapes in the image and hence makes their detection impossible.
The objective of this work is two-fold: (i) To develop an automatic cold steel weapon detection model for video surveillance using Convolutional Neural Networks(CNN) and (ii) strengthen its robustness to light conditions by proposing a brightness guided preprocessing procedure called DaCoLT (Darkening and Contrast at Learning and Test stages).
The obtained detection model provides excellent results as a cold steel weapon detector and as an automatic alarm system in video surveillance.

Results of this study are ilustrated in next content: Results videos

3.3 A binocular image fusion approach for minimizing false positives in handgun detection with deep learning

Olmos, R., Tabik, S., Lamas, A., Perez-Hernandez, F., & Herrera, F. (2019). A binocular image fusion approach for minimizing false positives in handgun detection with deep learning. Information Fusion, 49, 271-280. doi.org/10.1016/j.inffus.2018.11.015

Object detection models have known important improvements in the recent years. The state-of-the art detectors are end-to-end CNN based models that reach good mean average precisions, around 73%, on benchmarks of high quality images. However, these models still produce a large number of false positives in low quality videos such as, surveillance videos. This paper proposes a novel image fusion approach to make the detection model focus on the area of interest where the action is more likely to happen in the scene.
We propose building a low cost symmetric dual camera system to compute the disparity map and exploit this information to improve the selection of candidate regions from the input frames. From our results, the proposed approach not only reduces the number of false positives but also improves the overall performance of the detection model which makes it appropriate for object detection in surveillance videos.

3.4 Object Detection Binary Classifiers methodology based on deep learning to identify small objects handled similarly: Application in video surveillance

Pérez-Hernández, F., Tabik, S., Lamas, A., Olmos, R., Fujita, H., & Herrera, F. (2020). Object detection binary classifiers methodology based on deep learning to identify small objects handled similarly: Application in video surveillance. Knowledge-Based Systems, 194, 105590. doi.org/10.1016/j.knosys.2020.105590

The capability of distinguishing between small objects when manipulated with hand is essential in many fields, especially in video surveillance. To date, the recognition of such objects in images using Convolutional Neural Networks (CNNs) remains a challenge. In this paper, we propose improving robustness, accuracy and reliability of the detection of small objects handled similarly using binarization techniques. We propose improving their detection in videos using a two level methodology based on deep learning, called Object Detection with Binary Classifiers. The first level selects the candidate regions from the input frame and the second level applies a binarization technique based on a CNN-classifier with One-Versus-All or One-Versus-One. In particular, we focus on the video surveillance problem of detecting weapons and objects that can be confused with a handgun or a knife when manipulated with hand. We create a database considering six objects: pistol, knife, smartphone, bill, purse and card. The experimental study shows that the proposed methodology reduces the number of false positives with respect to the baseline multi-class detection model.

Results of this study are ilustrated in next content: Results videos

3.5 Human pose estimation for mitigating false negatives in weapon detection in video-surveillance

Lamas, A., Tabik, S., Montes, A. C., Pérez-Hernández, F., García, J., Olmos, R., & Herrera, F. (2022). Human pose estimation for mitigating false negatives in weapon detection in video-surveillance. Neurocomputing. doi.org/10.1016/j.neucom.2021.12.059

Applying CNN-based object detection models to the task of weapon detection in video-surveillance is still producing a high number of false negatives. In this context, most existing works focus on one type of weapons, mainly firearms, and improve the detection using different pre- and post-processing strategies. One interesting approach that has not been explored in depth yet is the exploitation of the human pose information for improving weapon detection. This paper proposes a top-down methodology that first determines the hand regions guided by the human pose estimation then analyzes those regions using a weapon detection model. For an optimal localization of each hand region, we defined a new factor, called Adaptive pose factor, that takes into account the distance of the body from the camera. Our experiments show that this top-down Weapon Detection over Pose Estimation (WeDePE) methodology is more robust than the alternative bottom-up approach and state-of-the art detection models in both indoor and outdoor video-surveillance scenarios.

Results of this study are ilustrated in next content: Results videos

3.6 MULTICAST: MULTI Confirmation-level Alarm SysTem based on CNN and LSTM to mitigate false alarms for handgun detection in video-surveillance

Olmos, R., Tabik, S., Perez-Hernandez, F., Lamas, A., & Herrera, F. (2021). MULTICAST: MULTI Confirmation-level Alarm SysTem based on CNN and LSTM to mitigate false alarms for handgun detection in video-surveillance. arXiv preprint arXiv:2104.11653.

Despite the constant advances in computer vision, integrating modern single-image detectors in real-time handgun alarm systems in video-surveillance is still debatable. Using such detectors still implies a high number of false alarms and false negatives. In this context, most existent studies select one of the latest single-image detectors and train it on a better dataset or use some pre-processing, post-processing or data-fusion approach to further reduce false alarms. However, none of these works tried to exploit the temporal information present in the videos to mitigate false detections. This paper presents a new system, called MULTI Confirmation-level Alarm SysTem based on Convolutional Neural Networks (CNN) and Long Short Term Memory networks (LSTM) (MULTICAST), that leverages not only the spacial information but also the temporal information existent in the videos for a more reliable handgun detection. MULTICAST consists of three stages, i) a handgun detection stage, ii) a CNN-based spacial confirmation stage and iii) LSTM-based temporal confirmation stage. The temporal confirmation stage uses the positions of the detected handgun in previous instants to predict its trajectory in the next frame. Our experiments show that MULTICAST reduces by 80% the number of false alarms with respect to Faster R-CNN based-single-image detector, which makes it more useful in providing more effective and rapid security responses.

4. Papers

Olmos, R., Tabik, S., & Herrera, F. (2018).
Automatic handgun detection alarm in videos using deep learning.
Neurocomputing, 275, 66-72.
Link to the article - PDF

Castillo, A., Tabik, S., Pérez, F., Olmos, R., Herrera, F. (2019).
Brightness guided preprocessing for automatic cold steel weapon detection in surveillance videos using deep learning.
Neurocomputing, 330, 151-161.
Link to the article - PDF

Olmos, R., Tabik, S., Lamas, A., Pérez-Hernández, F., Herrera, F. (2019).
A binocular image fusion approach for minimizing false positives in handgun detection with deep learning
Information Fusion, 49, 271-280.
Link to the article - PDF

Pérez-Hernández, F., Tabik, S., Lamas, A., Olmos, R., Fujita, H., Herrera, F. (2020).
Object Detection Binary Classifiers methodology based on deep learning to identify small objects handled similarly: Application in video surveillance.
Knowledge-Based Systems, 194, 105590.
Link to the article - PDF

Lamas, A., Tabik, S., Montes, A. C., Pérez-Hernández, F., García, J., Olmos, R., & Herrera, F. (2022).
Human pose estimation for mitigating false negatives in weapon detection in video-surveillance.
Neurocomputing.
Link to the article - PDF

Olmos, R., Tabik, S., Perez-Hernandez, F., Lamas, A., & Herrera, F. (2021).
MULTICAST: MULTI Confirmation-level Alarm SysTem based on CNN and LSTM to mitigate false alarms for handgun detection in video-surveillance.
arXiv preprint arXiv:2104.11653.
Link to the article - PDF

5. Authors

Francisco Herrera Triguero Google Scholar Researcher ID Orcid E-Mail