bellsystems-cp/docs/mqtt-app-user-auth.md

MQTT Authentication for Phone-App Users

Why this exists

The remote phone app (FlutterFlow + Firebase Auth) talks to Vesper boards over MQTT. We did not want to create an MQTT account per app user, so app users log into Mosquitto with their Firebase identity, and the Console backend decides what each one may do, based on which devices are assigned to them.

Mosquitto uses mosquitto-go-auth with two backends:

• files (passwd file on the VPS): admin, bonamin, NodeRED.
• http (this Console): POST /mqtt/auth/user (on CONNECT) and POST /mqtt/auth/acl (on SUBSCRIBE, PUBLISH and every message delivery). Code: backend/mqtt/auth.py, backend/mqtt/app_users.py.

The Mosquitto / go-auth config lives on the VPS, not in this repo.

Connection contract for the app (read this if you build the app)

Setting	Value
Username	app_<firebase_uid>
Password	The user's current Firebase ID token (getIdToken())
Client ID	Must start with app_<firebase_uid>_, e.g. app_<uid>_<installId>
Transport	MQTT over secure WebSockets: wss://mqtt.bellsystems.net:443, path /mqtt
Wildcards	Not allowed (+ / # are denied)

Never connect the app to port 1883. It is plain TCP for the boards only. The broker would still check the login there, but by then the token has already been sent unencrypted. Keep the MQTT keepalive below 3600 s (e.g. 30–60 s).

Any other client ID is refused on every topic. The prefix rule stops one user from reusing another user's client ID to kick them off.

Allowed topics, only for serials assigned to that user:

Action	acc	Topics
Publish	2	vesper/{serial}/control/command
Subscribe / receive messages	4 / 1	vesper/{serial}/control/ack, vesper/{serial}/status/heartbeat, vesper/{serial}/status/playback

Everything else is denied (403).

Token lifetime: Mosquitto checks the password only at CONNECT. When the app reconnects, it must fetch a fresh ID token (getIdToken() refreshes an expired one). A connection stays open past token expiry. Blocking or unassigning still takes effect on that connection, because every publish or delivery goes through the ACL (within about 60 s of the go-auth ACL cache).

Which field is the device serial

The serial in vesper/{serial}/... is the device doc's serial_number: it is flashed into NVS and the firmware uses it as its MQTT id. Legacy docs only have it in device_id. users.service.device_serial_of() holds that rule; always use it.

How access is decided

• users/{doc}.device_serials: [string] lists the serials a user may reach. It mirrors the devices' user_list and is written in the same atomic Firestore batch by every Console path that edits user_list:
  • users.service.assign_device / unassign_device
  • POST / DELETE /api/devices/{id}/user-list (device Manage tab)
  • PUT /api/devices/{id} when the body contains user_list (devices.service.update_device)
• Users are resolved by the uid field (a query), never by doc ID. Console- created users get random doc IDs (.add()); FlutterFlow uses the uid as the doc ID.
• A user is refused when status == "blocked".
• Lookups are cached in-process for 60 s (mqtt/app_users.py). Assign/unassign, update, block/unblock and delete call invalidate(uid). If you add a new code path that changes user_list, device_serials or status, it must do the same. The cache is per process: if uvicorn ever runs multiple workers, invalidation only reaches one of them (entries still expire after 60 s).
• app_ usernames never go through the device HMAC or the legacy password.

Legacy "vesper" password

Boards on pre-HMAC firmware still use the shared password vesper. MQTT_ALLOW_LEGACY_PASSWORD (default true) controls it. It is only accepted for device-shaped usernames. Each use is logged at WARNING, once per board per hour:

MQTT legacy password accepted for <serial> — board still on pre-HMAC firmware

When those lines stop appearing, set MQTT_ALLOW_LEGACY_PASSWORD=false.

Rollout checklist

1. Deploy the backend.

2. Backfill device_serials for existing assignments (run it right after the deploy):

   cd backend
   python scripts/backfill_user_device_serials.py          # dry run, writes nothing
   python scripts/backfill_user_device_serials.py --apply  # writes
   

   It only writes the device_serials field on users docs. It is idempotent: a second run finds nothing to change. Until it runs, the Console shows existing users with no devices, and app users are denied everything.

3. go-auth config on the VPS (R = required, r = recommended):

   auth_opt_backends files, http                  # R  files first, then http
   auth_opt_http_host <host reaching the backend> # R
   auth_opt_http_port 8000                        # R
   auth_opt_http_getuser_uri /mqtt/auth/user      # R
   auth_opt_http_aclcheck_uri /mqtt/auth/acl      # R
   auth_opt_http_params_mode form                 # R  endpoints read form fields; go-auth's default is json
   auth_opt_http_response_mode status             # r
   auth_opt_http_method POST                      # r
   auth_opt_http_timeout 5                        # r
   auth_opt_disable_superuser true                # r  no superuser endpoint exists or is needed
   auth_opt_cache true                            # r
   auth_opt_cache_type go-cache                   # r
   auth_opt_auth_cache_seconds 60                 # r
   auth_opt_acl_cache_seconds 60                  # r
   auth_opt_auth_jitter_seconds 10                # r
   auth_opt_acl_jitter_seconds 10                 # r
   allow_anonymous false                          # R
   

4. Files-backend ACL: verified OK (2026-09-30). The VPS config sets no auth_opt_files_acl_path (nor acl_file). A live test confirmed the files backend does not grant ACLs to everyone, so every ACL check reaches the Console. If an ACL file is ever added, only lines under user admin / user bonamin / user NodeRED are safe. A pattern line, or a topic line placed before the first user line, applies to every user (app users included) and must not cover vesper/.... Re-run the isolation test after any broker config change (on the VPS host):

   A=PV25L22BP01R01; B=PV26B02BP01R01
   PW=$(docker exec bellsystems-backend python -c "from mqtt.auth import _derive_password; print(_derive_password('$A'))")
   # own topics -> should print messages
   docker exec mosquitto mosquitto_sub -h localhost -i acl-test-own -u "$A" -P "$PW" -t "vesper/$A/#" -v -W 5
   # foreign topics -> must say "All subscription requests were denied."
   docker exec mosquitto mosquitto_sub -h localhost -i acl-test-foreign -u "$A" -P "$PW" -t "vesper/$B/#" -v -W 5
   

5. TLS for the app: done (2026-09-30). See "Current VPS broker setup" below. Test it from the VPS host (should print connect: Success and a got: line):

   docker exec -i bellsystems-backend python - <<'EOF'
   import time, paho.mqtt.client as m
   from mqtt.auth import _derive_password
   A = "PV25L22BP01R01"
   c = m.Client(m.CallbackAPIVersion.VERSION2, client_id="wss-test", transport="websockets")
   c.ws_set_options(path="/mqtt")
   c.tls_set()
   c.username_pw_set(A, _derive_password(A))
   c.on_connect = lambda c, u, f, rc, p: (print("connect:", rc), c.subscribe(f"vesper/{A}/#"))
   c.on_message = lambda c, u, msg: print("got:", msg.topic)
   c.connect("mqtt.bellsystems.net", 443, 30)
   c.loop_start(); time.sleep(8); c.loop_stop()
   EOF
   

Current VPS broker setup (as of 2026-09-30)

• Container mosquitto, image iegomez/mosquitto-go-auth:latest.

• Config: host /home/bellsystems/stacks/mosquitto/config/mosquitto.conf, mounted at /etc/mosquitto/mosquitto.conf (Docker images usually use /mosquitto/config; this one does not). Passwd file: /home/bellsystems/stacks/mosquitto/passwd.

• Already set correctly: backends files,http, http_host 172.20.0.1 (Docker bridge), http_port 8000, both URIs, http_method post, params_mode form, response_mode status, cache true, cache_reset true.

• allow_anonymous false ✓. auth_cache_seconds 60, acl_cache_seconds 60 ✓. auth_opt_hasher is not set (a harmless "defaulting to PBKDF2" warning at start).

• Listeners:

  • listener 1883 0.0.0.0: plain TCP, boards only (ESP32 can't spare RAM for TLS). Published on the host as 1883:1883.
  • listener 8083 + protocol websockets: plain WebSockets, not published on the host. Only reachable from the Docker network npm_npmnet.

• TLS is terminated by NPM (NPMplus, container npm-npm-1), which also renews the Let's Encrypt certificate. Proxy host: mqtt.bellsystems.net → http://mosquitto:8083, Force SSL + HTTP/2 on. Advanced config:

  proxy_read_timeout 3600s;
  proxy_send_timeout 3600s;
  

  (Without these, NPM drops idle MQTT connections after 60 s.) NPMplus has no "Websockets Support" toggle; WebSockets are always on.

• Compose file: /home/bellsystems/stacks/mosquitto/docker-compose.yml. The service is on default (reaches the backend at 172.20.0.1:8000) and the external network npm_npmnet (so NPM can reach mosquitto:8083). Removing default breaks all board logins.

• Rollback for the TLS change: .bak copies of mosquitto.conf and docker-compose.yml sit next to the originals. Restore them, run docker compose up -d in that folder, then delete the NPM proxy host.

Known gaps / decisions

• Console backend port 8000 is public. It is published as 0.0.0.0:8000 and the VPS has no firewall (Docker-published ports bypass host firewall rules anyway), so anyone can call /mqtt/auth/user and /mqtt/auth/acl. The passwords can't realistically be guessed, so it's not urgent, but these endpoints are meant only for Mosquitto. To be fixed separately, carefully, so nginx/NPM keep reaching the Console.

• Old-firmware boards use a different command topic. Broker logs (2026-09-30) show some boards (e.g. PV26B02BP01R01, BSVSPR-26C20B-STD10R-2KCDPH) subscribing to vesper/{serial}/control, not vesper/{serial}/control/command. The app ACL only allows publishing to control/command, so the app cannot command those boards until they run current firmware. (The Console's own publish_command has the same limitation.) Do not widen the ACL to control without deciding it deliberately.

• FlutterFlow writes to user_list. If the app changes user_list directly in Firestore (e.g. a claim flow), it must also update device_serials the same way (ArrayUnion / ArrayRemove of the serial on the user doc). Otherwise that user cannot reach the device over MQTT. The app owner will handle this in the app.

• The device_users subcollection on devices (older way of recording device users, read first by devices.service.get_device_users) is intentionally ignored by the backfill and the MQTT ACL. Left as is by decision.

• The two Console assignment paths store user_list entries in different formats (path strings vs DocumentReferences). This predates this work; all readers accept both.

• ACL denies are logged at INFO, which is not shown by default. Auth denies and legacy-password logins are WARNING.

Tests

cd backend
pip install pytest    # not in requirements.txt
python -m pytest tests

tests/test_mqtt_auth.py covers the auth and ACL endpoints. tests/test_device_serials_sync.py covers the device PUT sync. Firebase and Firestore are mocked.