Yasha Iravantchi

Enabling computing systems to understand user interactions with everyday surfaces and objects can drive a wide range of applications. However, existing vibration-based sensors (e.g., accelerometers) lack the sensitivity to detect light touch gestures or the bandwidth to recognize activity containing high-frequency components. Conversely, microphones are highly susceptible to environmental noise, degrading performance. Each time an object impacts a surface, Surface Acoustic Waves (SAWs) are generated that propagate along the air-to-surface boundary. This work repurposes a Voice PickUp Unit (VPU) to capture SAWs on surfaces (including smooth surfaces, odd geometries, and fabrics) over long distances and in noisy environments. Our custom-designed signal acquisition, processing, and machine learning pipeline demonstrates utility in both interactive and activity recognition applications, such as classifying trackpad-style gestures on a desk and recognizing 16 cooking-related activities, all with >97% accuracy. Ultimately, SAWs offer a unique signal that can enable robust recognition of user touch and on-surface events.

Internet-connected cameras support many useful home monitoring and health applications. However, these same cameras indiscriminately capture sensitive and Personally Identifiable Information (PII), limiting their acceptance in certain settings, such as the home. Prior works removed Region of Interest (ROI) to secure images and improve privacy. However, the methods that rely solely on RGB information to find persons are susceptible to environmental and lighting conditions, causing them to fail and leak PII. From our deployment study, nearly half of the images containing persons had a PII leakage when using RGB-only methods. Furthermore, ROI removal is often performed off-device, requiring the server performing these operations to be trustworthy. This work presents the PrivacyLens system, where with the addition of thermal sensing, our system has a significantly enhanced ability to find persons in RGB images and video and efficiently remove them on the device before any data is stored or transmitted, all while staying under typical IoT power constraints. From our aforementioned deployment study in an office-building atrium, family home, and outdoor park environment, the PrivacyLens prototype effectively removes PII with a sanitization rate of 99.1%. Additionally, PrivacyLens can use its embedded GPU to generate on-device features for downstream CV/ML tasks, as shown in three illustrative applications, further reducing the collection and storage of PII.

As the number of smart devices increases in our lives, the data they collect to perform valuable tasks, such as voice assistant requests, comes at the cost of user privacy. To mitigate their privacy impact, emerging usable privacy-aware sensing (UPAS) research has relied on cross-disciplinary approaches that extend past the core focus of broader academic research communities, such as Security & Privacy or Human-Computer Interaction. These works incorporate privacy design principles, whereby systems include safeguards by combining usable privacy (UP) with privacy-aware sensing (PAS) design to protect users’ privacy. To better understand this emerging area of research, we conducted a mixed qualitative and quantitative Systematization of Knowledge (SoK). With a thorough review of pertinent literature, resulting in 114 selected works (reduced from 10,122 across 12 venues), we found that, despite the similarity of these works, many are dispersed across multiple communities, utilize community-specific jargon and keywords, and minimally overlap in design and evaluation approaches, potentially hindering cross-pollination across communities and thereby slowing the growth of this emerging research area. Thus, these factors helped reveal a research gap in this space. We use these findings to present four research themes and provide community and design recommendations to encourage cross-disciplinary UPAS research.