In this study, we learn from the way spiders perceive prey through the subtle vibrations of their webs, design a framework that runs on Android called spider-inspired handwriting character capture (spider-inspired HCCapture) to speculate handwriting contents in the unrestricted area. Moreover, posture variation also needs to be handled when the victim’s data are not included in the training process of the speculation model. The other is that the sensors’ signals and noise patterns vary under different holding postures (e.g., sitting and standing), since the victim’s limb jitter and handwriting strength change significantly under different postures, called posture variation. Meanwhile, we observe that each person has his/her own handwriting habits, such as writing strength, sequence, and speed, called chirography difference. One issue is that victims are usually unknown to attackers before attacks, but the speculation model is built on the training data from the known users. Nevertheless, the speculation for more complex and meaningful inputs, such as handwritten contents, has not been sufficiently discussed. (2016) have shown that primitive operation actions (e.g., click and scroll in gesture control) on touchscreens can be recognized by analyzing sensor signals. On the other hand, more and more touchscreens support unrestricted-area input interface, in which users can tap any position/area on the touchscreen for input, such as gesture control and handwritten input. By far, most research work only focus on the restricted-area input interface, in which touchscreen information is entered by tapping a specified position/area on the touchscreen, such as the virtual keyboard ( Spreitzer et al., 2018a Mehrnezhad et al., 2018) and the pattern lock screen ( Aviv et al., 2012 Zhou et al., 2018). By exploiting those sensors as a side-channel, touchscreen information can be speculated, such as passwords tapped by users on the virtual keyboard ( Xu et al., 2012 Das et al., 2018 Zhao et al., 2019). Roughly speaking, even though those background applications cannot directly obtain the touchscreen information, they can access the signals of some built-in sensors, such as accelerometer and gyroscope. For example, users can enter passwords by tapping virtual keyboards, unlock devices by drawing patterns, and input messages by handwriting.Īt present, numerous research works have confirmed that touchscreen information (e.g., PIN and unlock pattern) can be speculated by some malicious background applications. The experimental results show that the accuracy of spider-inspired HCCapture reaches 96.1%.Īs the most popular human–computer interaction interface on mobile devices, touchscreen transmits a plenty of sensitive information from users to mobile applications. In conclusion, spider-inspired HCCapture completes the handwritten character speculation attack without obtaining the victim’s information in advance.
In addition, the Markov model is introduced into spider-inspired HCCapture, which is used as an enhancement feature when there is a correlation between adjacent characters. We also proposed a user-independent posture-aware approach to detect the user’s handwriting posture to select a suitable one from some pretrained models for speculation. Furthermore, each character is disassembled into basic strokes, which are used as recognition features. To alleviate the impact of different handwriting habits, we utilize the generality patterns of characters rather than the patterns of raw sensor signals. Spider-inspired HCCapture exploits the motion sensor as the side-channel and uses the neural network algorithm to train the recognition model. We learn from the way spiders perceive prey through the subtle vibrations of their webs an unrestricted-area handwriting information speculation framework, called spider-inspired handwriting character capture (spider-inspired HCCapture), is designed. The research of information leakage in the restricted area has been relatively mature, but in the unrestricted area, still there are two issues to be solved urgently: chirography difference and posture variation. Many researchers have proven that sensors can be used as side channels to leak touchscreen interactive information. On mobile devices, the most important input interface is touchscreen, which can transmit a large amount of sensitive information.