Creating a Phone AI Agent Using Local LLMs
Learn how RikkaHub Agent can transform smartphones into AI agents by utilizing local language models, allowing for impressive automation capabilities.
As smartphones advance, so does the potential for them to function as more than just communication devices. One technology enthusiast demonstrated how he transformed his phone into an AI agent using local language learning models (LLMs). By leveraging RikkaHub Agent, a robust fork of an open-source Android LLM client, he unlocked numerous features on his device, making it a surprisingly capable automation tool.
Transforming the Smartphone
Traditionally, mobile devices lack the capacity to run complex AI models due to limitations in memory bandwidth and thermal management. However, Adam Conway, XDA’s Lead Technical Editor, found a workaround by integrating a phone-based agent with local models, specifically the Qwen 3.6 27B and Qwen 3.6 35B. This combination allows for impressive capabilities that include file searching, SMS reading, and even automating browser tasks. RikkaHub Agent builds a bridge between device and software through a unique architecture, showcasing how the latest hardware can harness AI potential.
RikkaHub Agent and its Features
| Feature | Description |
|---|---|
| Device Tools | Over 80 tools including tapping, swiping, typing, and accessing notifications. |
| Browser Control | Full control of an in-app browser, enabling navigating and interacting with web pages. |
| File Manager | Ability to search and manage files directly from the device. |
| Job Scheduling | Scheduled jobs persist after device reboots. |
| Sub-Agents | Use additional agents for more complex tasks without user input. |
The addition of these features allows users to delegate a range of tasks to their smartphones without requiring detailed programming knowledge. The automated processes escalate as users define desired outcomes rather than dictating intricate automation procedures.
How It Functions
The RikkaHub Agent operates by splitting the workload between the phone and a home server. The local model runs on external hardware, while the phone simply acts as a controller that facilitates requests and executes basic commands. This architecture frees up power and resource restrictions inherent in mobile devices.
Conway utilized the Qwen 3.6 27B model, which, when executed on a suitable environment such as a dedicated desktop setup with ample GPU memory, can generate outputs at a rate of 40 to 50 tokens per second—offering adequate performance for various tasks.
Commanding the Phone
The real magic of this setup comes when interacting with the phone as an AI agent. Conway tested several actions, including opening a web browser and navigating to XDA. The agent accurately pulled headlines and article authors without human intervention, showcasing not only its versatility but also the seamless integration of control commands. This capability illustrates a significant advancement in how smartphones can serve as automated assistants.
Installation Capabilities
One of the standout moments in Conway’s exploration was when he asked the agent if the whisper.cpp tool was set up on his phone. Upon finding it was not, the agent offered to install the necessary dependencies, compile the software, and pull additional model files—all without any command-line input from the user. The setup process took around seven minutes and demonstrated a degree of autonomy rarely seen in mobile applications.
Safety and Security Features
With greater power comes responsibility; thus, safety mechanisms are intricately woven into the functionality of RikkaHub Agent. All tools are turned off by default, requiring user permission for activation. This approach allows users to create distinct assistant profiles tailored to individual preferences while maintaining a fundamental level of security. Each command that could potentially cause disruption requires a specific approval, adding another layer of safeguard against misuse.
Additionally, a deterministic blocklist prevents harmful commands from executing even when they appear benign. The developer transparently communicates the limits of the safety features, ensuring users are aware of potential vulnerabilities.
Pitfalls of Local Processing
While there are numerous advantages to running a local LLM on a mobile device, the implementation can be somewhat sluggish. Conway noted that when operating the Gemma 4 E2B model directly on his Oppo Find N5, the agent struggled with tool-calling tasks, resulting in confusing experiences such as the model printing tool calls as text instead of executing them. The process was improved after updates, but speed remained an issue due to hardware constraints.
Even with powerful models, there is a clear distinction between performance on mobile vs. desktop systems, particularly under multitasking scenarios. For casual tasks like reading messages, local models can thrive; however, for more complex commands, external hardware significantly enhances speed and responsiveness.
Key Takeaways
- RikkaHub Agent transforms smartphones into powerful AI assistants with extensive device control.
- Over 80 tools allow for multifaceted interactions, from browser control to job scheduling.
- The integration of local models like Qwen 3.6 enhances task execution despite inherent mobile limitations.
- Safety features prevent unauthorized actions, requiring user permission for tool activation.
- The performance of local models is often slower on mobile devices compared to desktops, especially for complex tasks.
This innovative approach to utilizing smartphones broadens the scope of personal automation and AI capabilities, making the prospect of running autonomous agents on mobile devices a tantalizing reality.
Frequently Asked Questions