Phone-as-Controller System
Turns any smartphone into a system-level virtual gamepad: a web controller streams touch and gyro input over WebSockets to a PC server that emulates a real Xbox controller via ViGEm (Windows) / uinput (Linux) so any controller-only game just works.
Skills involved
What This Is
A system that turns a smartphone into a virtual game controller for a PC. A web app on the phone renders on-screen sticks and buttons, streams input over WebSockets to a server on the computer, and the server emulates a gamepad at the OS level — so any game that supports controllers (FIFA/FC, racing titles, etc.) sees a regular Xbox controller and just works. No app-store install required; any Android or iOS phone connects from the browser.
Three-Layer Architecture
Phone UI (web app) → PC server (WebSocket) → Virtual controller backend
on-screen sticks / parses messages into ViGEm / uinput / foohid
buttons / gyro abstract input actions emits real gamepad events
- Phone UI layer — HTML/JS controller page served from the PC at
http://server-ip:port, rendering buttons, virtual joysticks, and optional accelerometer/gyro input for tilt-steering. - Transport layer — WebSockets over Wi-Fi for low-latency bidirectional messaging at 20–60 Hz.
- Server input layer — parses messages into abstract actions (
button A down,axis L_STICK x/y). - Emulation layer — OS-specific driver that presents a real gamepad to the OS and games.
Controller Emulation Backends
The hard part is making the OS believe a physical controller is plugged in. The backend is OS-specific behind a common interface:
- Windows — ViGEmBus kernel driver + ViGEmClient creates a virtual Xbox 360 / DualShock 4 controller recognised system-wide. Games natively understand XInput, so an emulated Xbox 360 pad is the ideal target. (
vJoy/vXboxis a user-mode alternative.) - Linux —
/dev/uinputlets a user-space program declare a virtual joystick with axes and buttons, then writeinput_eventstructs per network event. - macOS —
foohidkext publishes a virtual HID device defined by a custom HID report descriptor via Apple's HID Manager APIs.
Protocol
A small JSON message schema over the socket:
{ "type": "button", "id": "A", "state": 1 }
{ "type": "axis", "id": "L_STICK", "x": -1.0, "y": 0.4 }
State is sent only when it changes, with an optional heartbeat snapshot at a fixed interval, to keep bandwidth low.
Why Wi-Fi Over Bluetooth
A browser can't expose an arbitrary Bluetooth HID profile, and a full BT controller profile needs low-level, per-OS native work. Wi-Fi/WebSockets gives easy multi-client support, rich HTML5 UIs, and portability — the same pattern used by HappyFunTimes and AirConsole. Bluetooth/USB can be layered on later via native apps.
Security
The server should bind only to the LAN, gate connections behind a random room code / QR join, and consider TLS if ever exposed beyond localhost — any connected client can otherwise inject input to the host.
References
Builds on the patterns proven by open-source projects: XJoy (ViGEm), notroller and VirtualGamePad (uinput, web → gamepad), and the Mac Virtual Game Controller (Flask + foohid).
What's Next
- Pluggable backend interface (
BackendWindows/BackendLinux/BackendMac) selected at runtime - Multiple selectable layouts: D-pad, dual sticks, racing wheel, custom button maps
- Gyro-based steering for racing titles
- Multi-phone support for local co-op
Last updated Jun 5, 2026
← Back to all projects