RFPose-OT: RF-based 3D human pose estimation via optimal transport theory

Cong YU; Dongheng ZHANG; Zhi WU; Zhi LU; Chunyang XIE; Yang HU; Yan CHEN

doi:10.1631/FITEE.2200550

Your Location：

Home >

Browse articles >

RFPose-OT: RF-based 3D human pose estimation via optimal transport theory

Regular Papers | Updated：2023-10-25

- RFPose-OT: RF-based 3D human pose estimation via optimal transport theory
- RFPose-OT：基于最优传输理论的无线三维人体姿态估计
- Frontiers of Information Technology & Electronic Engineering Vol. 24, Issue 10, Pages: 1445-1457(2023)
- Affiliations：
  
  1.School of Information and Communication Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China
  2.School of Cyber Science and Technology, University of Science and Technology of China, Hefei 230026, China
  3.School of Information Science and Technology, University of Science and Technology of China, Hefei 230026, China
- Author bio：
  
  †E-mail: congyu@std.uestc.edu.cn
  ‡Corresponding author
- Funds：
- DOI：10.1631/FITEE.2200550
  CLC： TP391.4
- Received：07 November 2022，
  
  Accepted：15 June 2023，
  
  Published：0 October 2023
- Accepted：
Scan QR Code
Cong YU, Dongheng ZHANG, Zhi WU, et al. RFPose-OT: RF-based 3D human pose estimation via optimal transport theory[J]. Frontiers of Information Technology & Electronic Engineering, 2023, 24(10): 1445-1457.
DOI：

Cong YU, Dongheng ZHANG, Zhi WU, et al. RFPose-OT: RF-based 3D human pose estimation via optimal transport theory[J]. Frontiers of Information Technology & Electronic Engineering, 2023, 24(10): 1445-1457. DOI： 10.1631/FITEE.2200550.

摘要

本文提出一个新颖的RFPose-OT模型框架以实现从无线射频信号中估计三维人体姿态。与现有直接从射频信号中预测人体姿态方法不同，本文考虑射频信号与人体姿态之间的结构特征差异，提出基于最优传输理论在特征空间上将射频信号变换到人体姿态域，再根据变换后的特征预测人体姿态。为评估RFPose-OT模型，本文构建了一个无线电系统和一个多视角相机系统获取无线信号数据以及真实的人体姿态标签。在室内基本环境、室内遮挡环境以及室外环境中的实验结果表明，RFPose-OT模型能精确地估计三维人体姿态，优于现有方法。

Abstract

This paper introduces a novel framework

i.e.

RFPose-OT

to enable three-dimensional (3D) human pose estimation from radio frequency (RF) signals. Different from existing methods that predict human poses from RF signals at the signal level directly

we consider the structure difference between the RF signals and the human poses

propose a transformation of the RF signals to the pose domain at the feature level based on the optimal transport (OT) theory

and generate human poses from the transformed features. To evaluate RFPose-OT

we build a radio system and a multi-view camera system to acquire the RF signal data and the ground-truth human poses. The experimental results in a basic indoor environment

an occlusion indoor environment

and an outdoor environment demonstrate that RFPose-OT can predict 3D human poses with higher precision than state-of-the-art methods.

关键词

Keywords

references

Bonneel N , van de Panne M , Paris S , et al. , 2011 . Displacement interpolation using Lagrangian mass transport . Proc SIGGRAPH Asia Conf , p. 1 - 12 . 10.1145/2024156.2024192 https://doi.org/10.1145/2024156.2024192

Cao Z , Simon T , Wei SE , et al. , 2017 . Realtime multi-person 2D pose estimation using part affinity fields . Proc IEEE Conf on Computer Vision and Pattern Recognition , p. 7291 - 7299 . 10.1109/CVPR.2017.143 https://doi.org/10.1109/CVPR.2017.143

Chen JB , Zhang DH , Wu Z , et al. , 2022 . Contactless electro-cardiogram monitoring with millimeter wave radar . IEEE Trans Mob Comput , early access . 10.1109/TMC.2022.3214721 https://doi.org/10.1109/TMC.2022.3214721

Chen Y , Su X , Hu Y , et al. , 2020 . Residual carrier frequency offset estimation and compensation for commodity WiFi . IEEE Trans Mob Comput , 19 ( 12 ): 2891 - 2902 . 10.1109/TMC.2019.2934106 https://doi.org/10.1109/TMC.2019.2934106

Chen Y , Deng HY , Zhang DH , et al. , 2021 . SpeedNet: indoor speed estimation with radio signals . IEEE Int Things J , 8 ( 4 ): 2762 - 2774 . 10.1109/JIOT.2020.3022071 https://doi.org/10.1109/JIOT.2020.3022071

Conte E , Filippi A , Tomasin S , 2010 . Ml period estimation with application to vital sign monitoring . IEEE Signal Process Lett , 17 ( 11 ): 905 - 908 . 10.1109/LSP.2010.2071382 https://doi.org/10.1109/LSP.2010.2071382

Fang HS , Xie SQ , Tai YW , et al. , 2017 . RMPE: regional multi-person pose estimation . Proc IEEE Int Conf on Computer Vision , p. 2334 - 2343 . 10.1109/ICCV.2017.256 https://doi.org/10.1109/ICCV.2017.256

He KM , Gkioxari G , Dollár P , et al. , 2017 . Mask R-CNN . Proc IEEE Int Conf on Computer Vision , p. 2961 - 2969 . 10.1109/ICCV.2017.322 https://doi.org/10.1109/ICCV.2017.322

He Y , Chen Y , Hu Y , et al. , 2020 . WiFi vision: sensing, recognition, and detection with commodity MIMO-OFDM WiFi . IEEE Int Things J , 7 ( 9 ): 8296 - 8317 . 10.1109/JIOT.2020.2989426 https://doi.org/10.1109/JIOT.2020.2989426

Hsu CY , Hristov R , Lee GH , et al. , 2019 . Enabling identification and behavioral sensing in homes using radio reflections . Proc CHI Conf on Human Factors in Computing Systems , p. 1 - 13 . 10.1145/3290605.3300778 https://doi.org/10.1145/3290605.3300778

Ito N , Godsill S , 2020 . A multi-target track-before-detect particle filter using superpositional data in non-Gaussian noise . IEEE Signal Process Lett , 27 : 1075 - 1079 . 10.1109/LSP.2020.3002704 https://doi.org/10.1109/LSP.2020.3002704

Ji HR , Hou CP , Yang Y , et al. , 2021 . A one-class classification method for human gait authentication using micro-Doppler signatures . IEEE Signal Process Lett , 28 : 2182 - 2186 . 10.1109/LSP.2021.3122344 https://doi.org/10.1109/LSP.2021.3122344

Jiang WJ , Xue HF , Miao CL , et al. , 2020 . Towards 3D human pose construction using WiFi . Proc 26 th Annual Int Conf on Mobile Computing and Networking , p. 1 - 14 . 10.1145/3372224.3380900 https://doi.org/10.1145/3372224.3380900

Kantorovich LV , 1942 . On the translocation of masses . Dokl Akad Nauk USSR , 37 : 199 - 201 (in Russian) .

Kim HI , Park RH , 2018 . Residual LSTM attention network for object tracking . IEEE Signal Process Lett , 25 ( 7 ): 1029 - 1033 . 10.1109/LSP.2018.2835768 https://doi.org/10.1109/LSP.2018.2835768

Kotaru M , Joshi K , Bharadia D , et al. , 2015 . SpotFi: decimeter level localization using WiFi . Proc ACM Conf on Special Interest Group on Data Communication , p. 269 - 282 . 10.1145/2785956.2787487 https://doi.org/10.1145/2785956.2787487

LeCun Y , Bengio Y , Hinton G , 2015 . Deep learning . Nature , 521 ( 7553 ): 436 - 444 . 10.1038/nature14539 https://doi.org/10.1038/nature14539

Li J , 2018 . Cyber security meets artificial intelligence: a survey . Front Inform Technol Electron Eng , 19 ( 12 ): 1462 - 1474 . 10.1631/FITEE.1800573 https://doi.org/10.1631/FITEE.1800573

Li TH , Fan LJ , Zhao MM , et al. , 2019 . Making the invisible visible: action recognition through walls and occlusions . Proc IEEE/CVF Int Conf on Computer Vision , p. 872 - 881 . 10.1109/ICCV.2019.00096 https://doi.org/10.1109/ICCV.2019.00096

Li YD , Zhang DH , Chen JB , et al. , 2021 . Towards domain-independent and real-time gesture recognition using mmWave signal . IEEE Trans Mob Comput , early access . 10.1109/TMC.2022.3207570 https://doi.org/10.1109/TMC.2022.3207570

Liu SP , Tian GH , Cui YC , et al. , 2022 . A deep Q-learning network based active object detection model with a novel training algorithm for service robots . Front Inform Technol Electron Eng , 23 ( 11 ): 1673 - 1683 . 10.1631/FITEE.2200109 https://doi.org/10.1631/FITEE.2200109

Ma L , Zhong QY , Zhang YY , et al. , 2021 . Associative affinity network learning for multi-object tracking . Front Inform Technol Electron Eng , 22 ( 9 ): 1194 - 1206 . 10.1631/FITEE.2000272 https://doi.org/10.1631/FITEE.2000272

Majeed K , Sorour S , Al-Naffouri TY , et al. , 2016 . Indoor localization and radio map estimation using unsupervised manifold alignment with geometry perturbation . IEEE Trans Mob Comput , 15 ( 11 ): 2794 - 2808 . 10.1109/TMC.2015.2510631 https://doi.org/10.1109/TMC.2015.2510631

Martinez J , Hossain R , Romero J , et al. , 2017 . A simple yet effective baseline for 3D human pose estimation . Proc IEEE Int Conf on Computer Vision , p. 2640 - 2649 . 10.1109/ICCV.2017.288 https://doi.org/10.1109/ICCV.2017.288

Monge G , 1781 . Mémoire sur la théorie des déblais et des remblais. Mémoires de Mathématique et de Physique , Presentés à l’Académie Royale des Sciences , p. 666 - 704 (in French) .

Niu K , Zhang FS , Wang XZ , et al. , 2022 . Understanding WiFi signal frequency features for position-independent gesture sensing . IEEE Trans Mob Comput , 21 ( 11 ): 4156 - 4171 . 10.1109/TMC.2021.3063135 https://doi.org/10.1109/TMC.2021.3063135

Patwari N , Wilson J , Ananthanarayanan S , et al. , 2014 . Monitoring breathing via signal strength in wireless networks . IEEE Trans Mob Comput , 13 ( 8 ): 1774 - 1786 . 10.1109/TMC.2013.117 https://doi.org/10.1109/TMC.2013.117

Qian K , Wu CS , Yang Z , et al. , 2018 . Enabling contactless detection of moving humans with dynamic speeds using CSI . ACM Trans Embed Comput Syst , 17 ( 2 ): 1 - 18 . 10.1145/3157677 https://doi.org/10.1145/3157677

Qiu CR , Zhang DH , Hu Y , et al. , 2022 . Radio-assisted human detection . IEEE Trans Multim , 25 : 2613 - 2623 . 10.1109/TMM.2022.3149129 https://doi.org/10.1109/TMM.2022.3149129

Rampa V , Savazzi S , Nicoli M , et al. , 2015 . Physical modeling and performance bounds for device-free localization systems . IEEE Signal Process Lett , 22 ( 11 ): 1864 - 1868 . 10.1109/LSP.2015.2438176 https://doi.org/10.1109/LSP.2015.2438176

Sengupta A , Jin F , Zhang RY , et al. , 2020 . mm-Pose: real-time human skeletal posture estimation using mmWave radars and CNNs . IEEE Sens J , 20 ( 17 ): 10032 - 10044 . 10.1109/JSEN.2020.2991741 https://doi.org/10.1109/JSEN.2020.2991741

Song RY , Zhang DH , Wu Z , et al. , 2022 . RF-URL: unsupervised representation learning for RF sensing . Proc 28 th Annual Int Conf on Mobile Computing and Networking , p. 282 - 295 . 10.1145/3495243.3560529 https://doi.org/10.1145/3495243.3560529

Wang F , Zhou S , Panev S , et al. , 2019 . Person-in-WiFi: fine-grained person perception using WiFi . Proc IEEE/CVF Int Conf on Computer Vision , p. 5452 - 5461 . 10.1109/ICCV.2019.00555 https://doi.org/10.1109/ICCV.2019.00555

Wang L , Sun K , Dai HP , et al. , 2021 . WiTrace: centimeter-level passive gesture tracking using OFDM signals . IEEE Trans Mob Comput , 20 ( 4 ): 1730 - 1745 . 10.1109/TMC.2019.2961885 https://doi.org/10.1109/TMC.2019.2961885

Wei SE , Ramakrishna V , Kanade T , et al. , 2016 . Convolutional pose machines . Proc IEEE Conf on Computer Vision and Pattern Recognition , p. 4724 - 4732 . 10.1109/CVPR.2016.511 https://doi.org/10.1109/CVPR.2016.511

Wu Z , Zhang DH , Xie CY , et al. , 2022 . RFMask: a simple baseline for human silhouette segmentation with radio signals . IEEE Trans Multim , early access . 10.1109/TMM.2022.3181455 https://doi.org/10.1109/TMM.2022.3181455

Xu XY , Yu JD , Chen YY , 2022 . Leveraging acoustic signals for fine-grained breathing monitoring in driving environments . IEEE Trans Mob Comput , 21 ( 3 ): 1018 - 1033 . 10.1109/TMC.2020.3015828 https://doi.org/10.1109/TMC.2020.3015828

Yang Y , Zhuang YT , Pan YH , 2021 . Multiple knowledge representation for big data artificial intelligence: framework, applications, and case studies . Front Inform Technol Electron Eng , 22 ( 12 ): 1551 - 1558 . 10.1631/FITEE.2100463 https://doi.org/10.1631/FITEE.2100463

Yu C , Wu Z , Zhang DH , et al. , 2022 . RFGAN: RF-based human synthesis . IEEE Trans Multim , 25 : 2926 - 2938 . 10.1109/TMM.2022.3153136 https://doi.org/10.1109/TMM.2022.3153136

Yue SC , He H , Wang H , et al. , 2018 . Extracting multi-person respiration from entangled RF signals . Proc ACM Interact Mob Wearab Ubiq Technol , 2 ( 2 ): 1 - 22 . 10.1145/3214289 https://doi.org/10.1145/3214289

Zeng YZ , Pathak PH , Mohapatra P , 2016 . WiWho: WiFi-based person identification in smart spaces . Proc 15 th ACM/IEEE Int Conf on Information Processing in Sensor Networks , p. 1 - 12 . 10.1109/IPSN.2016.7460727 https://doi.org/10.1109/IPSN.2016.7460727

Zhang BB , Zhang DH , Li YD , et al. , 2021 . Unsupervised domain adaptation for device-free gesture recognition . https://arxiv.org/abs/2111.10602v1 https://arxiv.org/abs/2111.10602v1

Zhang DH , He Y , Gong XY , et al. , 2018 . Multitarget AOA estimation using wideband LFMCW signal and two receiver antennas . IEEE Trans Veh Technol , 67 ( 8 ): 7101 - 7112 . 10.1109/TVT.2018.2827408 https://doi.org/10.1109/TVT.2018.2827408

Zhang DH , Hu Y , Chen Y , et al. , 2019 . BreathTrack: tracking indoor human breath status via commodity WiFi . IEEE Int Things J , 6 ( 2 ): 3899 - 3911 . 10.1109/JIOT.2019.2893330 https://doi.org/10.1109/JIOT.2019.2893330

Zhang DH , Hu Y , Chen Y , et al. , 2020 . Calibrating phase offsets for commodity WiFi . IEEE Syst J , 14 ( 1 ): 661 - 664 . 10.1109/JSYST.2019.2904714 https://doi.org/10.1109/JSYST.2019.2904714

Zhang DH , Hu Y , Chen Y , 2021 . MTrack: tracking multi-person moving trajectories and vital signs with radio signals . IEEE Int Things J , 8 ( 5 ): 3904 - 3914 . 10.1109/JIOT.2020.3025820 https://doi.org/10.1109/JIOT.2020.3025820

Zhang F , Chen C , Wang BB , et al. , 2018 . WiSpeed: a statistical electromagnetic approach for device-free indoor speed estimation . IEEE Int Things J , 5 ( 3 ): 2163 - 2177 . 10.1109/JIOT.2018.2826227 https://doi.org/10.1109/JIOT.2018.2826227

Zhang QS , Zhu SC , 2018 . Visual interpretability for deep learning: a survey . Front Inform Technol Electron Eng , 19 ( 1 ): 27 - 39 . 10.1631/FITEE.1700808 https://doi.org/10.1631/FITEE.1700808

Zhang Z , 2000 . A flexible new technique for camera calibration . IEEE Trans Patt Anal Mach Intell , 22 ( 11 ): 1330 - 1334 . 10.1109/34.888718 https://doi.org/10.1109/34.888718

Zhao MM , Yue SC , Katabi D , et al. , 2017 . Learning sleep stages from radio signals: a conditional adversarial architecture . Proc 34 th Int Conf on Machine Learning , p. 4100 - 4109 .

Zhao MM , Tian YL , Zhao H , et al. , 2018a . RF-based 3D skeletons . Proc Conf of the ACM Special Interest Group on Data Communication , p. 267 - 281 . 10.1145/3230543.3230579 https://doi.org/10.1145/3230543.3230579

Zhao MM , Li TH , Abu Alsheikh M , et al. , 2018b . Through-wall human pose estimation using radio signals . Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition , p. 7356 - 7365 . 10.1109/CVPR.2018.00768 https://doi.org/10.1109/CVPR.2018.00768

Zheng C , Zhu SJ , Mendieta M , et al. , 2021 . 3D human pose estimation with spatial and temporal transformers . Proc IEEE/CVF Int Conf on Computer Vision , p. 11656 - 11665 . 10.1109/ICCV48922.2021.01145 https://doi.org/10.1109/ICCV48922.2021.01145

Zhou L , Chen YY , Gao YZ , et al. , 2020 . Occlusion-aware Siamese network for human pose estimation . Proc 16 th European Conf on Computer Vision , p. 396 - 412 . 10.1007/978-3-030-58565-5_24 https://doi.org/10.1007/978-3-030-58565-5_24

Views

238

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

MltAuxTSPP: a unified benchmark for deep learning-based traffic state prediction with multi-source auxiliary data

Q-space-coordinate-guided neural networks for high-fidelity diffusion tensor estimation from minimal diffusion-weighted images

CRGT-SA: an interlaced and spatiotemporal deep learning model for network intrusion detection

Federated model with contrastive learning and adaptive control variates for human activity recognition

A comprehensive survey of physical adversarial vulnerabilities in autonomous driving systems

Related Author

Yusong ZHOU

Xiaoyu JIANG

Shu SUN

Xinmin ZHANG

Yuanqiu MO

Zhihuan SONG

Maokun ZHENG

Zhi LI

Related Institution

College of Control Science and Engineering, Zhejiang University

State Key Laboratory of Industrial Control Technology, Zhejiang University

School of Mathematics, Southeast University

State Key Laboratory of Public Big Data, College of Computer Science and Technology, Guizhou University

School of Electronic and Electrical Engineering, Shanghai University of Engineering Science

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰