EdgeSense
Active ProjectHardware
A comprehensive IoT sensor integration system for EdgeSense, a native macOS AI desktop application. The update enables real-time data streaming from edge devices—including Raspberry Pi, Arduino (ESP32/ESP8266), and Nvidia Orin Nano—directly into LLM conversations. By leveraging WebSocket communication, mDNS zero-configuration discovery, and an efficient in-memory ring buffer architecture, the system provides seamless contextual awareness to AI assistants about physical world conditions.
12
Total Feedback
Apr 12
Last Updated
Key Features
Remote DevicesEdge DevicesMacOS AppClient App+7 more
Tech Stack
RustTauri 2.0Axumtokio+8 more
Image Gallery
EdgeSense - LLM Chat Window
EdgeSense - Local and API Model Configuration
EdgeSense - Available Devices & Sensors
EdgeSense - Edit Edge Device Sensor Config
EdgeSense - Saved Tools
EdgeSense - Saved Apps
EdgeSense - Connect App Page
EdgeSense - Split View w/Network
EdgeSense - Network View
EdgeSense - Settings Page
EdgeSense - MCP Server Configuration
EdgeSense - Client Install Guide
EdgeSense - App Example
EdgeSense Project
Loading updates...
Project Roadmap
Loading timeline...
Upcoming Features
As a user, I want to be able to view real-time sensor data from the Gemma 4 device within the EdgeSense application because this integration will allow me to monitor and analyze sensor information seamlessly without disrupting existing functionalities. This will affect the main application layout, particularly the sections where sensor data is displayed, such as the newly created `GemmaSensorDisplay` component in the React frontend.Planned
Medium Priority
As a user, I want to be able to manage my EdgeSense instances remotely and share sensor suites and tools between them because a new cloud function will be added to the EdgeSense application to facilitate these capabilities. This will affect the EdgeSense application’s remote management functionalities, specifically updating the React frontend where I interact with instance management features. The changes will enhance the application by integrating a cloud-based service, allowing for secure communication and resource sharing across multiple EdgeSense instances.Planned
Medium Priority
As a user, I want to be able to manage EdgeSense instances remotely and share sensor suites and tools between them because this will enable efficient and seamless remote management of my EdgeSense environment. This will affect the Cloud Manager section, where I will interact with the new cloud function, and the state management within the application to reflect changes in real-time.Planned
Medium Priority
As a user, I want to be able to view devices as nodes connected to the MCP and AI Gateway Servers because this will allow me to visually understand and interact with the network connectivity status in real-time. This feature will affect the EdgeSense application’s main interface, specifically by introducing a new NodeGraph component that dynamically updates to reflect the current device connections. It will enhance the user experience by providing an interactive and visually distinct representation of connected devices, enabling me to access detailed information about each node directly from the visualization.Planned
Medium Priority
As a user, I want to be able to select and modify existing tools and applications directly within the chat interface because this will allow me to use relevant resources contextually, enhancing my chat interactions. This will affect the chat UI component, specifically the chat interface where a new tool/app selector will be integrated. The selector will enable me to view a list of available tools and apps, and I will be able to make modifications that are saved correctly. This change will ensure a seamless experience by allowing me to manage tools and applications without leaving the chat, thereby improving efficiency and productivity in my interactions.Planned
Medium Priority
Known Issues
No known issues
Documents & Files
EdgeSesne 1.20.0 Apple Silicon MacOS
Project Challenges
1.Cross-Platform mDNS Compatibility
Challenge: mDNS behaves differently across macOS, Windows, and Linux. The
Solution: Implemented explicit interface selection prioritising non-loopback IPv4 addresses using the
2. WebSocket Connection Stability on Embedded Devices
Challenge: ESP8266/ESP32 devices have limited memory and unstable WiFi connections, causing frequent disconnections that weren't always detected promptly.
Solution: Implemented heartbeat mechanism with configurable timeout, combined with exponential backoff reconnection.
3. Efficient State Synchronisation Between Rust and React
Challenge: Tauri's IPC is efficient but batching many small sensor updates created UI lag.
Solution: Used Tokio broadcast channels in Rust and Tauri's event system for push-based updates, rather than polling.
4. Token Budget Management for LLM Context
Challenge: Sensor context can grow unboundedly with many devices, potentially consuming the LLM's context window.
Solution: Implemented configurable token budget with intelligent truncation.
``
5. Ring Buffer Memory Estimation
Challenge: Estimating appropriate buffer sizes for diverse sensor reading frequencies without excessive memory consumption.
Solution: Settled on 1000 readings per sensor type as default, with configurable limits. At ~200 bytes per reading, this bounds memory to ~200KB per sensor type per device—acceptable for desktop use.
Challenge: mDNS behaves differently across macOS, Windows, and Linux. The
mdns-sd crate abstracts much of this, but network interface enumeration proved problematic on machines with multiple interfaces (Ethernet + WiFi + VPN).Solution: Implemented explicit interface selection prioritising non-loopback IPv4 addresses using the
getifaddrs crate.2. WebSocket Connection Stability on Embedded Devices
Challenge: ESP8266/ESP32 devices have limited memory and unstable WiFi connections, causing frequent disconnections that weren't always detected promptly.
Solution: Implemented heartbeat mechanism with configurable timeout, combined with exponential backoff reconnection.
3. Efficient State Synchronisation Between Rust and React
Challenge: Tauri's IPC is efficient but batching many small sensor updates created UI lag.
Solution: Used Tokio broadcast channels in Rust and Tauri's event system for push-based updates, rather than polling.
4. Token Budget Management for LLM Context
Challenge: Sensor context can grow unboundedly with many devices, potentially consuming the LLM's context window.
Solution: Implemented configurable token budget with intelligent truncation.
``
rust
let maxchars = self.config.maxtokens * 4; // ~4 chars per token
if context.len() > maxchars {
context.truncate(maxchars - 20);
context.push_str("\n... [truncated]");
}
``5. Ring Buffer Memory Estimation
Challenge: Estimating appropriate buffer sizes for diverse sensor reading frequencies without excessive memory consumption.
Solution: Settled on 1000 readings per sensor type as default, with configurable limits. At ~200 bytes per reading, this bounds memory to ~200KB per sensor type per device—acceptable for desktop use.
Project Solutions & Learnings
1.Rust's Ownership Model Shines for Concurrent State
Managing concurrent device connections would be error-prone in C++ or prone to race conditions in JavaScript. Rust's
2. mDNS is Powerful but Fragile
Zero-configuration discovery significantly improves user experience for IoT integrations, but mDNS has quirks:
- Some routers filter multicast traffic
- Virtual networks (Docker, VMs) may not propagate mDNS
- Resolution can be slow (2-5 seconds)
Always provide manual host configuration as a fallback.
3. Keep the Wire Protocol Simple
JSON over WebSocket may not be the most efficient choice, but its debuggability is invaluable. During development, being able to
4. Design for Graceful Degradation
The system continues functioning when:
- No devices are connected (empty context)
- mDNS fails (manual host fallback)
- Device disconnects mid-stream (heartbeat timeout cleanup)
- Rate limits are exceeded (device informed, server protected)
This resilience is essential for production IoT systems where reliability trumps feature completeness.
5. Frontend State Management Complexity
Real-time data introduces complexity that simpler CRUD applications avoid:
- Optimistic updates vs. server truth
- Stale data detection
- Event ordering guarantees
Zustand with persist middleware handled this well, but required careful thought about which state should persist (configuration) versus which should be ephemeral (device connections).
Managing concurrent device connections would be error-prone in C++ or prone to race conditions in JavaScript. Rust's
Arc> pattern with parking_lot's implementation provided both safety and performance. The compiler catches potential data races at compile time, which proved invaluable during development.2. mDNS is Powerful but Fragile
Zero-configuration discovery significantly improves user experience for IoT integrations, but mDNS has quirks:
- Some routers filter multicast traffic
- Virtual networks (Docker, VMs) may not propagate mDNS
- Resolution can be slow (2-5 seconds)
Always provide manual host configuration as a fallback.
3. Keep the Wire Protocol Simple
JSON over WebSocket may not be the most efficient choice, but its debuggability is invaluable. During development, being able to
wscat into the server and send hand-crafted messages accelerated debugging significantly. For IoT devices with limited bandwidth, binary protocols (MessagePack, Protobuf) could be future optimisations.4. Design for Graceful Degradation
The system continues functioning when:
- No devices are connected (empty context)
- mDNS fails (manual host fallback)
- Device disconnects mid-stream (heartbeat timeout cleanup)
- Rate limits are exceeded (device informed, server protected)
This resilience is essential for production IoT systems where reliability trumps feature completeness.
5. Frontend State Management Complexity
Real-time data introduces complexity that simpler CRUD applications avoid:
- Optimistic updates vs. server truth
- Stale data detection
- Event ordering guarantees
Zustand with persist middleware handled this well, but required careful thought about which state should persist (configuration) versus which should be ephemeral (device connections).