Connecting a Local LLM to a $30 ESP32 Display for Dynamic Screens

Dynamic Displays: A New Approach to Smart Screens

The advent of smart displays has been met with enthusiasm, yet many come with a fixed set of screens, restricting users to whatever the firmware developers deemed suitable. These displays typically feature basic functionalities like clocks, weather updates, and calendar features, leaving many users desiring more customizable experiences. Recently, Adam Conway, a technology enthusiast and editor at XDA, devised a way to change this dynamic by connecting a local language model (LLM) to a $30 ESP32 display, allowing it to dynamically create new screens based on user requests.

The Project Setup

For this project, Conway utilized the Hermes Agent running in an unprivileged LXC on a Proxmox server, which lets users interact with the model via Telegram. The hardware involved includes the Elecrow CrowPanel, a 1.46-inch rotary display featuring a 360x360 pixel round touchscreen. The entire setup enabled Conway to send messages to the agent and receive visually updated screens as responses, revolutionizing how information is presented on such a small device.

Technical Specifications

This display communicates through a USB connection where both power and UART functionality are achieved via a supplied cable. Despite its compact form, challenges arose during the user interface experience; using a finger to scroll on the small screen was cumbersome. However, the rotary encoder offered a practical solution, allowing users to change screens without obstructing their view.

Rendering Custom Screens

Conway devised an innovative method for generating screen content. Rather than parsing HTML on the ESP32 directly, the Hermes Agent would interpret user requests in Telegram, render them as HTML through a headless Chromium instance on the host server, and take a screenshot of that visual output. The resulting PNG is then sent over the local area network (LAN) to the display panel for viewing.

Each card, or screen update, is a static image, which means real-time information updates are not automatically reflected; they require the agent's input to refresh. Interestingly, the system employs SHA-256 digests to avoid sending duplicate data, only transferring new content that the device has yet to view. This mechanism minimizes unnecessary data transfer, making the entire system efficient.

Understanding Display Limitations

While the CrowPanel boasts a generous 360x360 resolution, there are practical limitations to its use. Approximately half of the display pixels are rendered unusable due to the device's round shape, which means the maximum area suitable for content is a rectangle measuring 272x236 pixels. This design constraint has required careful planning in the stylesheet rules that the agent follows when generating content.

Creating Smart Content

One of the standout features of this project is the model’s ability to intelligently adjust text and layout for the display's limited size. Upon requesting the latest headlines from XDA, the language model instantly modified article titles to fit the screen without losing their meaning. This capability represents a sophisticated understanding from the model, responding not only to text prompts but also to contextual limitations that define effective communication on a small display.

Unique Functionalities and User Experience

Conway also noted the various display functionalities built into the agent. Each button press on the rotary encoder initiates a JSON event via WebSocket, with each interaction serving as a means to further customize the experience without directly interacting with the touchscreen. While the display does not have a dedicated queue or retry mechanism for unsent messages, it is designed to respond to requests in a straightforward manner.

Long-term ambitions for the device lean towards automating information display based on user behavior. The agent even suggested creating a scheduled routine to refresh the display with specific data daily, enhancing its utility beyond simple requests.

The Challenge of Integration

During the integration process, Conway encountered various challenges, particularly in hardware compatibility and software configurations. For example, discrepancies in the instructions provided by Elecrow for the controller led to extensive troubleshooting. The board was reported to utilize a JD9855 controller, but it turned out that most firmware was erroneously built against a different chip's instruction set. Correcting these errors required substantial investigation, but ultimately the system began to operate as intended.

Conclusion and Future Prospects

Overall, Conway's project demonstrates fascinating innovation within the realm of accessible technology, particularly for users interested in DIY setups. The blend of a robust local LLM with a cost-effective ESP32 display presents a versatile and adaptable device capable of showing varying information as requested. Users looking to explore this setup can find detailed firmware and project considerations on GitHub, presenting an excellent opportunity for tech enthusiasts to adapt the design for their purposes. This project highlights how local models can enhance everyday technology, pushing the boundaries of what users can expect from inexpensive hardware.

Key Takeaways

• The Elecrow CrowPanel is a 1.46-inch round display with 360x360 pixel resolution.
• Dynamic screens are created via a local LLM that responds to user requests.
• Efficiency is enhanced by using SHA-256 to prevent duplicate data transfers.
• Text and layouts are intelligently modified to fit the small screen context.
• Conway aims to automate daily updates for a more seamless experience.