SOAR: Self-supervision Optimized UAV Action Recognition with Efficient Object-Aware Pretraining


Paper Code
FAR GitHub Code
Abstract: We introduce SOAR, a novel Self-supervised pre- training algorithm for aerial footage captured by Unmanned Aerial Vehicles (UAVs). We incorporate human object knowl- edge throughout the pretraining process to enhance UAV video pretraining efficiency and downstream action recognition performance. This is in contrast to prior works that primarily incorporate object information during the fine-tuning stage. Specifically, we first propose a novel object-aware masking strategy designed to retain the visibility of certain patches related to objects throughout the pretraining phase. Second, we introduce an object-aware loss function that utilizes object information to adjust the reconstruction loss, preventing bias towards less informative background patches. In practice, SOAR with a vanilla ViT backbone, outperforms prior best models, recording a 9.7% and 21.4% boost in top-1 accuracy on the NEC-Drone and UAV-Human datasets with 2× to 5× faster inference speed. Additionally, SOAR obtains comparable accuracy to prior self-supervised learning (SSL) methods while requiring 87.5% less pretraining time and 25% less memory usage.