All projects
Game / Systemsin-progress

Phone-as-Controller System

Turns any smartphone into a system-level virtual gamepad: a web controller streams touch and gyro input over WebSockets to a PC server that emulates a real Xbox controller via ViGEm (Windows) / uinput (Linux) so any controller-only game just works.

PythonFastAPIWebSocketsViGEmuinputJavaScript

Skills involved

PythonBackend APIsGame Dev

What This Is

A system that turns a smartphone into a virtual game controller for a PC. A web app on the phone renders on-screen sticks and buttons, streams input over WebSockets to a server on the computer, and the server emulates a gamepad at the OS level — so any game that supports controllers (FIFA/FC, racing titles, etc.) sees a regular Xbox controller and just works. No app-store install required; any Android or iOS phone connects from the browser.

Three-Layer Architecture

Phone UI (web app)  →  PC server (WebSocket)  →  Virtual controller backend
on-screen sticks /     parses messages into     ViGEm / uinput / foohid
buttons / gyro         abstract input actions   emits real gamepad events
  • Phone UI layer — HTML/JS controller page served from the PC at http://server-ip:port, rendering buttons, virtual joysticks, and optional accelerometer/gyro input for tilt-steering.
  • Transport layer — WebSockets over Wi-Fi for low-latency bidirectional messaging at 20–60 Hz.
  • Server input layer — parses messages into abstract actions (button A down, axis L_STICK x/y).
  • Emulation layer — OS-specific driver that presents a real gamepad to the OS and games.

Controller Emulation Backends

The hard part is making the OS believe a physical controller is plugged in. The backend is OS-specific behind a common interface:

  • Windows — ViGEmBus kernel driver + ViGEmClient creates a virtual Xbox 360 / DualShock 4 controller recognised system-wide. Games natively understand XInput, so an emulated Xbox 360 pad is the ideal target. (vJoy/vXbox is a user-mode alternative.)
  • Linux — /dev/uinput lets a user-space program declare a virtual joystick with axes and buttons, then write input_event structs per network event.
  • macOS — foohid kext publishes a virtual HID device defined by a custom HID report descriptor via Apple's HID Manager APIs.

Protocol

A small JSON message schema over the socket:

{ "type": "button", "id": "A", "state": 1 }
{ "type": "axis", "id": "L_STICK", "x": -1.0, "y": 0.4 }

State is sent only when it changes, with an optional heartbeat snapshot at a fixed interval, to keep bandwidth low.

Why Wi-Fi Over Bluetooth

A browser can't expose an arbitrary Bluetooth HID profile, and a full BT controller profile needs low-level, per-OS native work. Wi-Fi/WebSockets gives easy multi-client support, rich HTML5 UIs, and portability — the same pattern used by HappyFunTimes and AirConsole. Bluetooth/USB can be layered on later via native apps.

Security

The server should bind only to the LAN, gate connections behind a random room code / QR join, and consider TLS if ever exposed beyond localhost — any connected client can otherwise inject input to the host.

References

Builds on the patterns proven by open-source projects: XJoy (ViGEm), notroller and VirtualGamePad (uinput, web → gamepad), and the Mac Virtual Game Controller (Flask + foohid).

What's Next

  • Pluggable backend interface (BackendWindows / BackendLinux / BackendMac) selected at runtime
  • Multiple selectable layouts: D-pad, dual sticks, racing wheel, custom button maps
  • Gyro-based steering for racing titles
  • Multi-phone support for local co-op

Last updated Jun 5, 2026

← Back to all projects