A Multimodal Attention-Based Deep Learning Framework For Real-Time Activity Recognition At The Edge

Paper: Submitted

GitHub Link: https://github.com/iscaas/AFOSR-HAR-2021-2025/tree/main/OD-VIRAT (Dataset access link will be given in GitHub)

Description: OD-VIRAT is an object detection benchmark developed purely for object detection tasks in challenging surveillance environments. This dataset is constructed from videos of the VIRAT Ground 2.0 dataset, comprising 329 surveillance videos captured through stationary ground cameras mounted at significant heights (mostly atop buildings), spanning 11 distinct scenes. The recorded scenes include construction sites, parking lots, streets, and open outdoor areas. Different models of high definition (HD) cameras are used to capture scenes at different resolutions (1920×1080 and 1280×720) and frame rates (25∼30 frames per second (FPS)).

This dataset has two distinct versions that include OD-VIRAT-Large and OD-VIRAT-Tiny dataset. The statistical details of both datasets are given below in the table.

Codes

Description: OD-VIRAT is an object detection benchmark developed purely for object detection tasks in challenging surveillance environments.

1. OD-VIRAT: A Large-Scale Benchmark for Object Detection in Realistic Surveillance Environments

Paper link: Submitted

GiHub link: https://github.com/iscaas/AFOSR-HAR-2021-2025/tree/main/OD-VIRAT

Description: This paper introduces two visual object detection benchmarks named OD-VIRAT Large and OD-VIRAT Tiny, aiming at advancing visual understanding tasks in surveillance imagery. The video sequences in both benchmarks cover 10 different scenes of human surveillance recorded from significant height and distance. The proposed benchmarks offer rich annotations of bounding boxes and categories, where OD-VIRAT Large has 8.7 million annotated instances in 599,996 images and OD-VIRAT Tiny has 288,901 annotated instances in 19,860 images. This work also focuses on benchmarking state-of-the-art object detection architectures, including RETMDET, YOLOX, RetinaNet, DETR, and Deformable-DETR on this object detection-specific variant of VIRAT dataset.

2. A 3DCNN-Based Knowledge Distillation Framework for Human Activity Recognition

Paper link: https://www.mdpi.com/2313-433X/9/4/82

GiHub link: https://github.com/iscaas/AFOSR-HAR-2021-2025/tree/main/3DCNN-Knowledge-Distillation

Description: In this paper, we propose a knowledge distillation framework, which distills spatio-temporal knowledge from a large teacher model to a lightweight student model using an offline knowledge distillation technique. The proposed offline knowledge distillation framework takes the predictions of two models: a large pre-trained 3DCNN teacher model and a lightweight 3DCNN student model as an input and distills the knowledge from teacher to student model. During the knowledge distillation phase, the distillation algorithm trains the student model by minimizing the difference between the teacher’s and student’s prediction, resulting in a small yet robust model that maintains the same level of precision as of teacher model.

3. Human Activity Recognition Using Cascaded Dual Attention CNN And Bi-directional GRU Framework

Paper link: https://www.mdpi.com/2313-433X/9/7/130

GiHub link: https://github.com/iscaas/AFOSR-HAR-2021-2025/tree/main/DA-2DCNN

Description: This paper presents a computationally efficient yet generic spatial–temporal cascaded framework that exploits the deep discriminative spatial and temporal features for Human action recognition task. To learn robust representation of human actions from input video frames, we propose an efficient dual-attentional convolutional neural network (DA-CNN) architecture that leverages a unified channel–spatial attention mechanism to extract human-centric salient features in video frames. The dual channel–spatial attention layers together with the convolutional layers learn to be more selective in the spatial receptive fields having objects within the feature maps. The extracted discriminative salient features are then forwarded to a stacked bi-directional gated recurrent unit (Bi-GRU) for long-term temporal modeling and recognition of human actions using both forward and backward pass gradient learning.

4. Human Action Representation Learning Using an Attention-Driven Residual 3DCNN Network

Paper link: https://www.mdpi.com/1999-4893/16/8/369

GiHub link: https://github.com/iscaas/AFOSR-HAR-2021-2025/tree/main/DA-Residual%203DCNN

Description: In this paper, we present a computationally efficient yet robust approach, exploiting saliency-aware spatial and temporal features for human action recognition in videos. We propose an efficient approach called the dual-attentional Residual 3D Convolutional Neural Network (DA-R3DCNN). Our proposed method utilizes a unified channel-spatial attention mechanism, allowing it to efficiently extract significant human-centric features from video frames. By combining dual channel-spatial attention layers with residual 3D convolution layers, the network becomes more discerning in capturing spatial receptive field

5. ViT-ReT: Vision and Recurrent Transformer Neural Networks for Human Activity Recognition in Videos

Paper link: https://ieeexplore.org/abstract/document/10177697

GiHub link: https://github.com/iscaas/AFOSR-HAR-2021-2025/tree/main/ViR-ReT

Description:  This paper proposes and designs two transformer neural networks for human activity recognition: a recurrent transformer (ReT), a specialized neural network used to make predictions on sequences of data, as well as a vision transformer (ViT), a transformer optimized for extracting salient features from images, to improve speed and scalability of activity recognition. We have provided an extensive comparison of the proposed transformer neural networks with the contemporary CNN and RNN-based human activity recognition models in terms of speed and accuracy for four publicly available human action datasets.

Project Highlights