STFNets: Learning sensing signals from the time-frequency perspective with short-time fourier neural networks

Shuochao Yao, Ailing Piao, Wenjun Jiang, Yiran Zhao, Huajie Shao, Shengzhong Liu, Dongxin Liu, Jinyang Li, Tianshi Wang, Shaohan Hu, Lu Su, Jiawei Han, Tarek Abdelzaher

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Recent advances in deep learning motivate the use of deep neural networks in Internet-of-Things (IoT) applications. These networks are modelled after signal processing in the human brain, thereby leading to significant advantages at perceptual tasks such as vision and speech recognition. IoT applications, however, often measure physical phenomena, where the underlying physics (such as inertia, wireless signal propagation, or the natural frequency of oscillation) are fundamentally a function of signal frequencies, offering better features in the frequency domain. This observation leads to a fundamental question: For IoT applications, can one develop a new brand of neural network structures that synthesize features inspired not only by the biology of human perception but also by the fundamental nature of physics? Hence, in this paper, instead of using conventional building blocks (e.g., convolutional and recurrent layers), we propose a new foundational neural network building block, the Short-Time Fourier Neural Network (STFNet). It integrates a widely-used time-frequency analysis method, the Short-Time Fourier Transform, into data processing to learn features directly in the frequency domain, where the physics of underlying phenomena leave better footprints. STFNets bring additional flexibility to time-frequency analysis by offering novel nonlinear learnable operations that are spectral-compatible. Moreover, STFNets show that transforming signals to a domain that is more connected to the underlying physics greatly simplifies the learning process. We demonstrate the effectiveness of STFNets with extensive experiments on a wide range of sensing inputs, including motion sensors, WiFi, ultrasound, and visible light. STFNets significantly outperform the state-of-the-art deep learning models in all experiments. A STFNet, therefore, demonstrates superior capability as the fundamental building block of deep neural networks for IoT applications for various sensor inputs 1.

Original languageEnglish (US)
Title of host publicationThe Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019
PublisherAssociation for Computing Machinery, Inc
Pages2192-2202
Number of pages11
ISBN (Electronic)9781450366748
DOIs
StatePublished - May 13 2019
Event2019 World Wide Web Conference, WWW 2019 - San Francisco, United States
Duration: May 13 2019May 17 2019

Publication series

NameThe Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019

Conference

Conference2019 World Wide Web Conference, WWW 2019
CountryUnited States
CitySan Francisco
Period5/13/195/17/19

Fingerprint

Physics
Neural networks
Sensors
Speech recognition
Natural frequencies
Brain
Fourier transforms
Signal processing
Ultrasonics
Experiments
Internet of things
Deep neural networks
Deep learning

Keywords

  • Deep learning
  • Internet of Things
  • IoT
  • Time frequency analysis

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Software

Cite this

Yao, S., Piao, A., Jiang, W., Zhao, Y., Shao, H., Liu, S., ... Abdelzaher, T. (2019). STFNets: Learning sensing signals from the time-frequency perspective with short-time fourier neural networks. In The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019 (pp. 2192-2202). (The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019). Association for Computing Machinery, Inc. https://doi.org/10.1145/3308558.3313426

STFNets : Learning sensing signals from the time-frequency perspective with short-time fourier neural networks. / Yao, Shuochao; Piao, Ailing; Jiang, Wenjun; Zhao, Yiran; Shao, Huajie; Liu, Shengzhong; Liu, Dongxin; Li, Jinyang; Wang, Tianshi; Hu, Shaohan; Su, Lu; Han, Jiawei; Abdelzaher, Tarek.

The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019. Association for Computing Machinery, Inc, 2019. p. 2192-2202 (The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Yao, S, Piao, A, Jiang, W, Zhao, Y, Shao, H, Liu, S, Liu, D, Li, J, Wang, T, Hu, S, Su, L, Han, J & Abdelzaher, T 2019, STFNets: Learning sensing signals from the time-frequency perspective with short-time fourier neural networks. in The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019. The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019, Association for Computing Machinery, Inc, pp. 2192-2202, 2019 World Wide Web Conference, WWW 2019, San Francisco, United States, 5/13/19. https://doi.org/10.1145/3308558.3313426
Yao S, Piao A, Jiang W, Zhao Y, Shao H, Liu S et al. STFNets: Learning sensing signals from the time-frequency perspective with short-time fourier neural networks. In The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019. Association for Computing Machinery, Inc. 2019. p. 2192-2202. (The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019). https://doi.org/10.1145/3308558.3313426
Yao, Shuochao ; Piao, Ailing ; Jiang, Wenjun ; Zhao, Yiran ; Shao, Huajie ; Liu, Shengzhong ; Liu, Dongxin ; Li, Jinyang ; Wang, Tianshi ; Hu, Shaohan ; Su, Lu ; Han, Jiawei ; Abdelzaher, Tarek. / STFNets : Learning sensing signals from the time-frequency perspective with short-time fourier neural networks. The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019. Association for Computing Machinery, Inc, 2019. pp. 2192-2202 (The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019).
@inproceedings{87400c0c58b14d16a7e774ad87a64ad2,
title = "STFNets: Learning sensing signals from the time-frequency perspective with short-time fourier neural networks",
abstract = "Recent advances in deep learning motivate the use of deep neural networks in Internet-of-Things (IoT) applications. These networks are modelled after signal processing in the human brain, thereby leading to significant advantages at perceptual tasks such as vision and speech recognition. IoT applications, however, often measure physical phenomena, where the underlying physics (such as inertia, wireless signal propagation, or the natural frequency of oscillation) are fundamentally a function of signal frequencies, offering better features in the frequency domain. This observation leads to a fundamental question: For IoT applications, can one develop a new brand of neural network structures that synthesize features inspired not only by the biology of human perception but also by the fundamental nature of physics? Hence, in this paper, instead of using conventional building blocks (e.g., convolutional and recurrent layers), we propose a new foundational neural network building block, the Short-Time Fourier Neural Network (STFNet). It integrates a widely-used time-frequency analysis method, the Short-Time Fourier Transform, into data processing to learn features directly in the frequency domain, where the physics of underlying phenomena leave better footprints. STFNets bring additional flexibility to time-frequency analysis by offering novel nonlinear learnable operations that are spectral-compatible. Moreover, STFNets show that transforming signals to a domain that is more connected to the underlying physics greatly simplifies the learning process. We demonstrate the effectiveness of STFNets with extensive experiments on a wide range of sensing inputs, including motion sensors, WiFi, ultrasound, and visible light. STFNets significantly outperform the state-of-the-art deep learning models in all experiments. A STFNet, therefore, demonstrates superior capability as the fundamental building block of deep neural networks for IoT applications for various sensor inputs 1.",
keywords = "Deep learning, Internet of Things, IoT, Time frequency analysis",
author = "Shuochao Yao and Ailing Piao and Wenjun Jiang and Yiran Zhao and Huajie Shao and Shengzhong Liu and Dongxin Liu and Jinyang Li and Tianshi Wang and Shaohan Hu and Lu Su and Jiawei Han and Tarek Abdelzaher",
year = "2019",
month = "5",
day = "13",
doi = "10.1145/3308558.3313426",
language = "English (US)",
series = "The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019",
publisher = "Association for Computing Machinery, Inc",
pages = "2192--2202",
booktitle = "The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019",

}

TY - GEN

T1 - STFNets

T2 - Learning sensing signals from the time-frequency perspective with short-time fourier neural networks

AU - Yao, Shuochao

AU - Piao, Ailing

AU - Jiang, Wenjun

AU - Zhao, Yiran

AU - Shao, Huajie

AU - Liu, Shengzhong

AU - Liu, Dongxin

AU - Li, Jinyang

AU - Wang, Tianshi

AU - Hu, Shaohan

AU - Su, Lu

AU - Han, Jiawei

AU - Abdelzaher, Tarek

PY - 2019/5/13

Y1 - 2019/5/13

N2 - Recent advances in deep learning motivate the use of deep neural networks in Internet-of-Things (IoT) applications. These networks are modelled after signal processing in the human brain, thereby leading to significant advantages at perceptual tasks such as vision and speech recognition. IoT applications, however, often measure physical phenomena, where the underlying physics (such as inertia, wireless signal propagation, or the natural frequency of oscillation) are fundamentally a function of signal frequencies, offering better features in the frequency domain. This observation leads to a fundamental question: For IoT applications, can one develop a new brand of neural network structures that synthesize features inspired not only by the biology of human perception but also by the fundamental nature of physics? Hence, in this paper, instead of using conventional building blocks (e.g., convolutional and recurrent layers), we propose a new foundational neural network building block, the Short-Time Fourier Neural Network (STFNet). It integrates a widely-used time-frequency analysis method, the Short-Time Fourier Transform, into data processing to learn features directly in the frequency domain, where the physics of underlying phenomena leave better footprints. STFNets bring additional flexibility to time-frequency analysis by offering novel nonlinear learnable operations that are spectral-compatible. Moreover, STFNets show that transforming signals to a domain that is more connected to the underlying physics greatly simplifies the learning process. We demonstrate the effectiveness of STFNets with extensive experiments on a wide range of sensing inputs, including motion sensors, WiFi, ultrasound, and visible light. STFNets significantly outperform the state-of-the-art deep learning models in all experiments. A STFNet, therefore, demonstrates superior capability as the fundamental building block of deep neural networks for IoT applications for various sensor inputs 1.

AB - Recent advances in deep learning motivate the use of deep neural networks in Internet-of-Things (IoT) applications. These networks are modelled after signal processing in the human brain, thereby leading to significant advantages at perceptual tasks such as vision and speech recognition. IoT applications, however, often measure physical phenomena, where the underlying physics (such as inertia, wireless signal propagation, or the natural frequency of oscillation) are fundamentally a function of signal frequencies, offering better features in the frequency domain. This observation leads to a fundamental question: For IoT applications, can one develop a new brand of neural network structures that synthesize features inspired not only by the biology of human perception but also by the fundamental nature of physics? Hence, in this paper, instead of using conventional building blocks (e.g., convolutional and recurrent layers), we propose a new foundational neural network building block, the Short-Time Fourier Neural Network (STFNet). It integrates a widely-used time-frequency analysis method, the Short-Time Fourier Transform, into data processing to learn features directly in the frequency domain, where the physics of underlying phenomena leave better footprints. STFNets bring additional flexibility to time-frequency analysis by offering novel nonlinear learnable operations that are spectral-compatible. Moreover, STFNets show that transforming signals to a domain that is more connected to the underlying physics greatly simplifies the learning process. We demonstrate the effectiveness of STFNets with extensive experiments on a wide range of sensing inputs, including motion sensors, WiFi, ultrasound, and visible light. STFNets significantly outperform the state-of-the-art deep learning models in all experiments. A STFNet, therefore, demonstrates superior capability as the fundamental building block of deep neural networks for IoT applications for various sensor inputs 1.

KW - Deep learning

KW - Internet of Things

KW - IoT

KW - Time frequency analysis

UR - http://www.scopus.com/inward/record.url?scp=85066908826&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85066908826&partnerID=8YFLogxK

U2 - 10.1145/3308558.3313426

DO - 10.1145/3308558.3313426

M3 - Conference contribution

AN - SCOPUS:85066908826

T3 - The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019

SP - 2192

EP - 2202

BT - The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019

PB - Association for Computing Machinery, Inc

ER -