One of my side projects is this little device which I call Sammy.
Sammy is an ESP-VoCat device, an ESP32-S3 with a round touchscreen and a dual microphone array, assembled in China. It runs the open source xiaozhi-esp32 firmware, which out of the box connects to a Chinese server and gives a fairly basic chatbot.
You can buy it on AliExpress for about £45, but pick the N32R16 for 32 MB of Flash memory and 16 MB of PSRAM.
Set up was not loads of fun, and included translating a 6 digit binding code, read to me in Cantonese. However after much headscratching I got it working. The demo server is xiaozhi.me, and whilst this is pretty nifty for starters, and had quite excellent response times (which I have not been able to replicate), I wanted to run my own version.
The base server software is xiaozhi-esp32-server which supports a variety of solutions for automatic speech recognition, voice activity detection, text-to-speech, and manages device session tracking. It’s pretty cool, and if you wanted - you could keep all of your data out of the cloud. I installed it on my VPS which provided a fairly functional chatbot with an LLM provider of my choice. But what I really wanted was the flexibility of speaking to the Hermes agent that lives on the same VPS. Using that agent gives me access to the skills and scripts my agent uses.
I am working on getting Sammy to work as a first-class Hermes platform, in the same way as Telegram or Discord. Xiaozhi handles the hardware-adjacent bits. Hermes handles the agent side.
The problem is that the Xiaozhi protocol expects a simple pattern: the user says something, the server returns one response, and that response is spoken. Hermes is more interactive. On Discord or Telegram, Hermes will show that it is thinking, running tools, searching files, retrying something or generating an artefact.
My current solution is to keep the final answer spoken, but send the intermediate activity to Sammy’s screen. So if Hermes is running a tool, Sammy can show something like web_search, terminal or patch on the display. It is a small architectural split, but you can see that the agent is doing something.
Here’s me testing Sammy to check that he can run tools. The latency is a big issue.
I have other plans too, including customising the emote animations and getting Sammy/Hermes to initiate messages of his own from cron jobs or other triggers.
For now, though, the important thing is that the loop works. I can talk to Sammy, it can reach the agent running on my VPS, it can use tools, and it can talk back. That is enough to make the next bit worth building.