Gait recognition based on 3D point cloud data augmentation
Abstract
Abstract: The goal of gait recognition is to recognize human identity through walking patterns. There are two main methods of gait recognition in the existing research. The first method is based on appearance to extract gait features from binary contour images, and the second method is based on model to extract gait features from key joints. However, the effect of appearance based methods will be affected by the changes of carrying objects and different clothing, while model-based methods will be affected by the effect of pose estimation in recognition, and include sparse gait features, which makes existing gait recognition performances highly depend on visual texture information (such as clothing, carrying and so on). Combining the advantages of the above two methods, we can not only use continuous contour sequences, but also remove the influence of clothing and occlusion. In this paper, we propose a gait feature extraction method based on 3D point cloud. The proposed method first extracts 3D point cloud based on each person's walking video. Two different methods are proposed to map the 3D point cloud data to 2D black and white images. Then the projected 2D images are combined with the original 2D gait samples to expand existing gait datasets. We evaluate 3D point cloud based gait recognition methods on popular gait datasets. The experimental results demonstrate that our proposed method can achieve improvements compared to existing methods, and can achieve the state-of-the-art recognition performances under several experimental settings.
Keywords: Gait recognition, 3D point cloud, 2D mapping, occlusion elimination, dress and carrying variances
DOI: 10.33440/j.ijpaa.20230601.221
Citation: Yang Q G, Chen X, Lan Y B, Deng X L. Gait recognition based on 3D point cloud data augmentation. Int J Precis Agric Aviat, 2023; 6(1): 69–83.
Full Text:
PDFReferences
L. Xue, M. Gao, C. Xing, R. Mart ??n-Mart ??n, J. Wu, C. Xiong, R. Xu, J. C. Niebles, and S. Savarese. Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 1179–1189.
H. Zhu, Z. Zheng, and R. Nevatia. Gait recognition using 3-d human body shape inference, in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2023, pp. 909–918.
A. Xiao, J. Huang, W. Xuan, R. Ren, K. Liu, D. Guan, A. El Saddik, S. Lu, and E. P. Xing. 3d semantic segmentation in the wild: Learning generalized models for adverse-condition point clouds, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 9382–9392.
M. Deng, Z. Fan, P. Lin, and X. Feng. Human gait recognition based on frontal-view sequences using gait dynamics and deep learning. IEEE Transactions on Multimedia, 2023.
T. Teepe, A. R. Khan, J. Gilg, F. Herzog, S. H ?ormann, and G. Rigoll. Gaitgraph: Graph convolutional network for skeleton-based gait recognition. IEEE International Conference on Image Processing, 2021. 1, II-C
G. Li, L. Guo, R. Zhang, J. Qian, and S. Gao. Transgait: Multimodal-based gait recognition with set transformer. Applied Intelligence, 2023, 53(2): 1535–1547.
T. Huang, X. Ben, C. Gong, B. Zhang, R. Yan, and Q. Wu. Enhanced spatial-temporal salience for cross-view gait recognition. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 1(1): pp.1–14.
R. Liao, S. Yu, W. An, and Y. Huang. A model-based gait recognition method with body pose and human prior knowledge. Pattern Recognition, 2020. I, II-C, II-C, II-C.
X. Liu, Z. You, Y. He, S. Bi, and J. Wang. Symmetry-driven hyperfeature gcn for skeleton-based gait recognition. Pattern Recognition, 2022, 125, pp. 1–13.
F. Han, X. Li, J. Zhao, and F. Shen. A unified perspective of classification-based loss and distance-based loss for cross-view gaitrecognition. Pattern Recognition, 2022, 125, pp. 1–10.
H. Qin, Z. Chen, Q. Guo, Q. M. J. Wu, and M. Lu. Rpnet: Gait recognition with relationships between each body-parts. IEEE Transactions on Circuits and Systems for Video Technology, 2022.
P. Delgado-Santos, R. Tolosana, R. Guest, F. Deravi, and R. Vera-Rodriguez. Exploring transformers for behavioural biometrics: A case study in gait recognition. Pattern Recognition, 2023, 143, p. 109798.
H. Chao, Y. He, J. Zhang, and J. Feng. Gaitset: Regarding gait as a set for cross-view gait recognition. National Conference on Artificial Intelligence, 2018. I, II-B, IV.
Y. Zhang, Y. Huang, S. Yu, and L. Wang. Cross-view gait recognition by discriminative feature learning. IEEE Transactions on Image Processing, 2020.
C. Fan, Y. Peng, C. Cao, X. Liu, S. Hou, J. Chi, Y. Huang, Q. Li, and Z. He. Gaitpart: Temporal part-based model for gait recognition. Computer Vision and Pattern Recognition, 2020. I, II-B, II-B, IV.
Y. Peng, K. Ma, Y. Zhang, and Z. He. Learning rich features for gait recognition by integrating skeletons and silhouettes. Multimedia Tools and Applications, 2023, pp. 1–22.
V. Narayan, S. Awasthi, N. Fatima, M. Faiz, and S. Srivastava. Deep learning approaches for human gait recognition: A review, in 2023 International Conference on Artificial Intelligence and Smart Communication (AISC). IEEE, 2023, pp. 763–768.
X. Chen, X. Luo, J. Weng, W. Luo, H. Li, and Q. Tian. Multi-view gait image generation for cross-view gait recognition. IEEE Transactions on Image Processing, 2021.
X. Chen, J. Weng, W. Luo, W. Lu, H. Wu, J. Xu, and Q. Tian,Sample balancing for deep learning-based visual recognition. IEEE Transactions on Neural Networks, 2020.
L. Yao, W. Kusakunniran, Q. Wu, J. Xu, and J. Zhang. Collaborative feature learning for gait recognition under cloth changes. IEEE Transactions on Circuits and Systems for Video Technology, 2021.
C. Shen, C. Fan, W. Wu, R. Wang, G. Q. Huang, and S. Yu. Lidargait: Benchmarking 3d gait recognition with point clouds, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 1054–1063.
J. Chen, Z. Wang, C. Zheng, K. Zeng, Q. Zou, and L. Cui. Gaitamr: Cross-view gait recognition via aggregated multi-feature representation. Information Sciences, 2023, 636, p. 118920.
Y. Shi, L. Du, X. Chen, X. Liao, Z. Yu, Z. Li, C. Wang, and S. Xue. Robust gait recognition based on deep cnns with camera and radar sensor fusion. IEEE Internet of Things Journal, 2023.
T. Teepe, J. Gilg, F. Herzog, S. H ?ormann, and G. Rigoll. Towards a deeper understanding of skeleton-based gait recognition, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 1569–1577.
Y. Cui and Y. Kang. Multi-modal gait recognition via effective spatial-temporal feature fusion, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 17 949–17 957.
Z. He, W. Wang, J. Dong, and T. Tan. Temporal sparse adversarial attack on sequence-based gait recognition. Pattern Recognition, 2023, 133, p. 109028.
K. Ma, Y. Fu, D. Zheng, C. Cao, X. Hu, and Y. Huang. Dynamic aggregated network for gait recognition, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 22 076–22 085.
A. Sepas-Moghaddam and A. Etemad. Deep gait recognition: A survey, IEEE transactions on pattern analysis and machine intelligence, 2022, 45(1): 264–284.
J. Zheng, X. Liu, W. Liu, L. He, C. Yan, and T. Mei. Gait recognition in the wild with dense 3d representations and a benchmark, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 20 228–20 237.
F. Han, X. Li, J. Zhao, and F. Shen. A unified perspective of classification-based loss and distance-based loss for cross-view gait recognition. Pattern Recognition, 2022, 125, p. 108519.
G. Pavlakos, V. Choutas, N. Ghorbani, T. Bolkart, A. A. A. Osman, D. Tzionas, and M. J. Black. Expressive body capture: 3d hands, face, and body from a single image. Computer Vision and Pattern Recognition, 2019.
G. Pavlakos, L. Zhu, X. Zhou, and K. Daniilidis. Learning to estimate 3d human pose and shape from a single color image. Computer Vision and Pattern Recognition, 2018.
M. Omran, C. Lassner, G. Pons-Moll, P. V. Gehler, and B. Schiele. Neural body fitting: Unifying deep learning and model-based human pose and shape estimation. Computer Vision and Pattern Recognition, 2018.
S. Hou, X. Liu, C. Cao, and Y. Huang. Gait quality aware network: toward the interpretability of silhouette-based gait recognition. IEEE Transactions on Neural Networks and Learning Systems, 2022.
H. Dou, P. Zhang, W. Su, Y. Yu, Y. Lin, and X. Li. Gaitgci: Generative counterfactual intervention for gait recognition, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 5578–5588.
N. Li and X. Zhao. A strong and robust skeleton-based gait recognition method with gait periodicity priors. IEEE Transactions on Multimedia, 2022.
C. Fan, J. Liang, C. Shen, S. Hou, Y. Huang, and S. Yu. Opengait: Revisiting gait recognition towards better practicality, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 9707–9716.
H. Dou, P. Zhang, Y. Zhao, L. Dong, Z. Qin, and X. Li. Gaitmpl: Gait recognition with memory-augmented progressive learning. IEEE Transactions on Image Processing, 2022.
X. Liu, Z. You, Y. He, S. Bi, and J. Wang. Symmetry-driven hyper feature gcn for skeleton-based gait recognition. Pattern Recognition, 2022, 125, p. 108520.
S. Hou, C. Fan, C. Cao, X. Liu, and Y. Huang. A comprehensive study on the evaluation of silhouette-based gait recognition. IEEE Transactions on Biometrics, Behavior, and Identity Science, 2022.
Z. Wu, Y. Huang, L. Wang, X. Wang, and T. Tan. A comprehensive study on cross-view gait based human identification with deep cnns. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017. II-B, II-B, IV-A, IV-C
C. Filipi Gonc ?alves dos Santos, D. d. S. Oliveira, L. A. Passos, R. Gonc ?alves Pires, D. Felipe Silva Santos, L. Pascotti Valem, T. P. Moreira, M. Cleison S. Santana, M. Roder, J. Paulo Papa, et al. Gait recognition based on deep learning: A survey. ACM Computing Surveys (CSUR), 2022, 55(2): 1–34.
S. Hou, X. Liu, C. Cao, and Y. Huang. Set residual network for silhouette-based gait recognition. IEEE Trans. Biometrics, Behav., Identity Sci., 2021, 3(3): 384–393.
H. Wu, J. Tian, Y. Fu, B. Li, and X. Li. Condition-aware comparison scheme for gait recognition. IEEE Transactions on image processing, 2021.
R. Liao, Z. Li, S. S. Bhattacharyya, and G. York. Posemapgait: A model-based gait recognition method with pose estimation maps and graph convolutional networks. Neurocomputing, 2022, 501, pp. 514–528.
S. Hou, C. Cao, X. Liu, and Y. Huang. Gait lateral network: Learning discriminative and compact representations for gait recognition, 2020.
Z. Zhang, L. Tran, X. Yin, Y. Atoum, X. Liu, J. Wan, and N. Wang. Gait recognition via disentangled representation learning. Computer Vision and Pattern Recognition, 2019.
B. Lin, S. Zhang, and F. Bao. Gait recognition with multiple-temporal-scale 3d convolutional neural network. ACM International Conference on Multimedia, 2020.
J. N. Mogan, C. P. Lee, K. M. Lim, and K. S. Muthu. Gait-vit: Gait recognition with vision transformer. Sensors, 2022, 22(19): 7362.
Q. Zou, Y. Wang, Q. Wang, Y. Zhao, and Q. Li. Deep learning-based gait recognition using smartphones in the wild. IEEE Transactions on Information Forensics and Security, 2020.
Y. Wang, X. Zhang, Y. Shen, B. Du, G. Zhao, L. Cui, and H. Wen. Event-stream representation for human gaits identification using deep neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(7): 3436–3449.
Z. Cao, G. Hidalgo, T. Simon, S.-E. Wei, and Y. Sheikh. Openpose: Realtime multi-person 2d pose estimation using part affinity fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018.
C. Xu, Y. Makihara, X. Li, and Y. Yagi. Occlusion-aware human mesh model-based gait recognition. IEEE transactions on information forensics and security, 2023, 18, pp. 1309–1321.
S. Choi, J. Kim, W. Kim, and C. Kim. Skeleton-based gait recognition via robust frame-level matching. IEEE Transactions on Information Forensics and Security, 2019, 14(10): 2577–2592.
T. Chai, A. Li, S. Zhang, Z. Li, and Y. Wang. Lagrange motion analysis and view embeddings for improved gait recognition, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 20249–20258.
W. Yu, H. Yu, Y. Huang, and L. Wang. Generalized inter-class loss for gait recognition, in Proceedings of the 30th ACM Internationa Conference on Multimedia, 2022, pp. 141–150.
R. Liao, C. Cao, E. B. Garcia, S. Yu, and Y. Huang. Pose-based temporal-spatial network (ptsn) for gait recognition with carrying and clothing variations. Chinese Conferenceon Biometric Recognition, 2017.
X. Li, Y. Makihara, C. Xu, Y. Yagi, S. Yu, and M. Ren. End-to-end model-based gait recognition. Asian Conference on Computer Vision, 2020.
X. Song, Y. Huang, C. Shan, J. Wang, and Y. Chen. Distilled light gaitset: Towards scalable gait recognition. Pattern Recognition Letters, 2022, 157, pp. 27–34.
D. Das, A. Agarwal, and P. Chattopadhyay. Gait recognition from occluded sequences in surveillance sites, in European Conference on Computer Vision. Springer, 2022, pp. 703–719.
S. Yan, Y. Xiong, and D. Lin. Spatial temporal graph convolutional networks for skeleton-based action recognition. Computer Vision and Pattern Recognition, 2018.
N. Li, X. Zhao, and C. Ma. A model-based gait recognition method based on gait graph convolutional networks and joints relationship pyramid mapping. Computer Vision and Pattern Recognition, 2020.
Y. F. Song, Z. Zhang, C. Shan, and L. Wang. Stronger, faster and more explainable: A graph convolutional baseline for skeleton-based action recognition. ACM International Conference on Multimedia, 2020.
H. M. Hsu, Y. Wang, C. Y. Yang, J. N. Hwang, H. L. U. Thuc, and K. J. Kim. Gaittake: Gait recognition by temporal attention and keypoint-guided embedding, in 2022 IEEE International Conference on Image Processing (ICIP). IEEE, 2022, pp. 2546–2550.
B. Lin, S. Zhang, and X. Yu. Gait recognition via effective global-local feature representation and local temporal aggregation. IEEE International Conference on Computer Vision, 2020.
H. Dou, P. Zhang, W. Su, Y. Yu, and X. Li. Metagait: Learning to learn an omni sample adaptive representation for gait recognition, in European Conference on Computer Vision. Springer, 2022, pp. 357–374.
A. Kanazawa, J. Y. Zhang, P. Felsen, and J. Malik. Learning 3d human dynamics from video. Computer Vision and Pattern Recognition, 2018.
Refbacks
- There are currently no refbacks.
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.