Sensor-enhanced LLM for Smart Health Systems

Date: Tuesday, 15th July 2025

Title: Sensor-enhanced LLM for Smart Health Systems

Abstract:

The widespread adoption of Large Language Models (LLMs) has transformed modern AI development. However, current deployments face significant challenges in integrating real-world sensory data from physical environments, including processing heterogeneous mixed-modality data, operating within computational constraints of edge devices, and maintaining user privacy.

 In this talk, I will introduce SensorLLM, a novel framework that enhances LLMs with real-world sensing capabilities for smart health applications. SensorLLM incorporates three key innovations: (1) an edge-cloud collaborative architecture enabling open-class classification on resource-constrained devices, (2) cost-efficient encoding methods for secure and privacy-preserving Transformer inference across distributed systems, and (3) an LLM-driven orchestration system that coordinates multi-modal sensors to address complex user queries.

 I will then present SensorLLM's adoption in three smart health applications. First, DrHouse serves as an LLM-based virtual consultation system that synthesizes wearable sensor data with medical knowledge to provide clinical recommendations. Second, Nuna is an LLM-powered smart necklace designed for accessible emotional tracking and continuous health monitoring. Finally, KoalaFM represents the first LLM-enabled platform for early diagnosis, personalized intervention, and complex cross-disease analysis of aging-related degenerative conditions. This system will undergo validation through a comprehensive five-year clinical trial involving 1,000 participants.