Current Projects
These projects reflect my current work on deployable sensing and machine learning systems, with an emphasis on on-device inference, multimodal perception, and robust evaluation in real-world environments.
Multi-Stage Thermal-Triggered VLM Framework for On-Device Eating Detection and Caloric Estimation
A resource-aware multimodal sensing system that introduces a thermal-triggered gating mechanism to selectively activate high-cost sensors such as RGB/depth cameras. The pipeline combines temperature-based filtering, connected-component analysis, and spatial constraints for event detection, followed by a NAS-optimized vision-language model and depth-based volumetric reconstruction for caloric estimation. It is designed for real-time, on-device deployment under strict power and compute budgets.
- Thermal-triggered sensing: Designed a low-cost event detection module using temperature thresholding, spatial clustering, and centroid constraints to gate RGB/depth sensing and reduce unnecessary sensor activation.
- Multimodal pipeline: Integrated thermal (MLX90640), RGB/depth, and IMU streams into a unified inference pipeline for eating detection and intake estimation.
- Model optimization: Applied neural architecture search (NAS) to design lightweight models optimized for on-device inference, balancing latency, accuracy, and energy consumption.
- Caloric estimation: Implemented depth-based volumetric reconstruction to estimate food portion size and combine portion estimates with classification outputs for kCal estimation.
- System deployment: Built an end-to-end embedded system supporting real-time inference, adaptive sensing, and efficient data logging under hardware constraints.
IGDA-MultiLoc: Label-Efficient Wi-Fi CSI Multi-Target Localization
A device-free indoor localization framework that uses Wi-Fi channel state information (CSI) to estimate multiple occupants' locations without requiring them to carry devices. IGDA-MultiLoc learns from labeled single-target CSI and unlabeled real multi-target observations, reducing the need for exhaustive multi-person fingerprint collection. The system combines structured CSI representation, geometry-aware learning, interaction modeling, and graph-based decoding for label-efficient multi-target localization in realistic indoor environments.
- Multi-view CSI representation: Transformed raw 2x3 MIMO CSI into amplitude, calibrated phase, temporal-difference, and inter-link relation views for hierarchical encoding.
- Geometry-aware learning: Used floor-plan-constrained distances to organize latent representations according to the physical structure of the indoor space.
- Interaction modeling: Built a permutation-invariant generator that composes virtual multi-target representations from single-target embeddings and geometry-conditioned interaction residuals.
- Graph-based localization: Decoded occupied reference points and target cardinality with a floor-plan-constrained graph decoder and uncertainty-guided node-conditional adaptation.
- System evaluation: Evaluated over five days in an approximately 100 m^2 laboratory with 28 reference points and two to four simultaneous occupants, achieving 0.420 m OSPA error, 88.7% target-count accuracy, and 89.4% macro-F1 with 25.9 ms processing after each CSI-window update.