bellsystems-cp/docs/mqtt-app-user-auth.md

# MQTT Authentication for Phone-App Users

## Why this exists

The remote phone app (FlutterFlow + Firebase Auth) talks to Vesper boards over MQTT.
We did not want to create an MQTT account per app user, so app users log into
Mosquitto with their **Firebase identity**, and the Console backend decides what
each one may do, based on which devices are assigned to them.

Mosquitto uses **mosquitto-go-auth** with two backends:

- **files** (passwd file on the VPS): `admin`, `bonamin`, `NodeRED`.
- **http** (this Console): `POST /mqtt/auth/user` (on CONNECT) and
  `POST /mqtt/auth/acl` (on SUBSCRIBE, PUBLISH and every message delivery).
  Code: `backend/mqtt/auth.py`, `backend/mqtt/app_users.py`.

The Mosquitto / go-auth config lives on the VPS, **not in this repo**.

## Connection contract for the app (read this if you build the app)

| Setting    | Value                                                                  |
|------------|------------------------------------------------------------------------|
| Username   | `app_<firebase_uid>`                                                   |
| Password   | The user's current **Firebase ID token** (`getIdToken()`)              |
| Client ID  | Must start with `app_<firebase_uid>_`, e.g. `app_<uid>_<installId>`    |
| Transport  | **MQTT over secure WebSockets**: `wss://mqtt.bellsystems.net:443`, path `/mqtt` |
| Wildcards  | Not allowed (`+` / `#` are denied)                                     |

**Never connect the app to port 1883.** It is plain TCP for the boards only. The broker
would still check the login there, but by then the token has already been sent
unencrypted. Keep the MQTT keepalive below 3600 s (e.g. 30–60 s).

Any other client ID is refused on every topic. The prefix rule stops one user from
reusing another user's client ID to kick them off.

Allowed topics, only for serials assigned to that user:

| Action                          | `acc` | Topics                                                                 |
|---------------------------------|-------|------------------------------------------------------------------------|
| Publish                         | 2     | `vesper/{serial}/control/command`                                      |
| Subscribe / receive messages    | 4 / 1 | `vesper/{serial}/control/ack`, `vesper/{serial}/status/heartbeat`, `vesper/{serial}/status/playback` |

Everything else is denied (403).

Token lifetime: Mosquitto checks the password **only at CONNECT**. When the app
reconnects, it must fetch a fresh ID token (`getIdToken()` refreshes an expired one).
A connection stays open past token expiry. Blocking or unassigning still takes effect
on that connection, because every publish or delivery goes through the ACL (within
about 60 s of the go-auth ACL cache).

## Which field is the device serial

The serial in `vesper/{serial}/...` is the device doc's **`serial_number`**: it is
flashed into NVS and the firmware uses it as its MQTT id. Legacy docs only have it in
**`device_id`**. `users.service.device_serial_of()` holds that rule; always use it.

## How access is decided

- `users/{doc}.device_serials: [string]` lists the serials a user may reach.
  It mirrors the devices' `user_list` and is written in the **same atomic Firestore
  batch** by every Console path that edits `user_list`:
  - `users.service.assign_device` / `unassign_device`
  - `POST` / `DELETE /api/devices/{id}/user-list` (device Manage tab)
  - `PUT /api/devices/{id}` when the body contains `user_list`
    (`devices.service.update_device`)
- Users are resolved by the **`uid` field** (a query), **never by doc ID**. Console-
  created users used to get random doc IDs (`.add()`) — fixed 2026-09-30, the Console
  now writes `users/{uid}` like FlutterFlow does — but older docs may still have a
  random ID, so keep resolving by the `uid` field.
- A user is refused when `status == "blocked"`.
- Lookups are cached in-process for 60 s (`mqtt/app_users.py`). Assign/unassign,
  update, block/unblock and delete call `invalidate(uid)`. If you add a new code path
  that changes `user_list`, `device_serials` or `status`, it must do the same.
  The cache is per process: if uvicorn ever runs multiple workers, invalidation only
  reaches one of them (entries still expire after 60 s).
- `app_` usernames never go through the device HMAC or the legacy password.

## Legacy "vesper" password

Boards on pre-HMAC firmware still use the shared password `vesper`.
`MQTT_ALLOW_LEGACY_PASSWORD` (default `true`) controls it. It is only accepted for
device-shaped usernames. Each use is logged at WARNING, once per board per hour:

    MQTT legacy password accepted for <serial> — board still on pre-HMAC firmware

When those lines stop appearing, set `MQTT_ALLOW_LEGACY_PASSWORD=false`.

## Rollout checklist

1. Deploy the backend.
2. **Backfill** `device_serials` for existing assignments (run it right after the deploy):

       cd backend
       python scripts/backfill_user_device_serials.py          # dry run, writes nothing
       python scripts/backfill_user_device_serials.py --apply  # writes

   It only writes the `device_serials` field on `users` docs. It is idempotent: a second
   run finds nothing to change. Until it runs, the Console shows existing users with no
   devices, and app users are denied everything.
3. **go-auth config on the VPS** (R = required, r = recommended):

       auth_opt_backends files, http                  # R  files first, then http
       auth_opt_http_host <host reaching the backend> # R
       auth_opt_http_port 8000                        # R
       auth_opt_http_getuser_uri /mqtt/auth/user      # R
       auth_opt_http_aclcheck_uri /mqtt/auth/acl      # R
       auth_opt_http_params_mode form                 # R  endpoints read form fields; go-auth's default is json
       auth_opt_http_response_mode status             # r
       auth_opt_http_method POST                      # r
       auth_opt_http_timeout 5                        # r
       auth_opt_disable_superuser true                # r  no superuser endpoint exists or is needed
       auth_opt_cache true                            # r
       auth_opt_cache_type go-cache                   # r
       auth_opt_auth_cache_seconds 60                 # r
       auth_opt_acl_cache_seconds 60                  # r
       auth_opt_auth_jitter_seconds 10                # r
       auth_opt_acl_jitter_seconds 10                 # r
       allow_anonymous false                          # R

4. **Files-backend ACL: verified OK (2026-09-30).** The VPS config sets no
   `auth_opt_files_acl_path` (nor `acl_file`). A live test confirmed the files backend
   does **not** grant ACLs to everyone, so every ACL check reaches the Console. If an
   ACL file is ever added, only lines under `user admin` / `user bonamin` /
   `user NodeRED` are safe. A `pattern` line, or a `topic` line placed before the first
   `user` line, applies to every user (app users included) and must not cover `vesper/...`.
   Re-run the isolation test after any broker config change (on the VPS host):

       A=PV25L22BP01R01; B=PV26B02BP01R01
       PW=$(docker exec bellsystems-backend python -c "from mqtt.auth import _derive_password; print(_derive_password('$A'))")
       # own topics -> should print messages
       docker exec mosquitto mosquitto_sub -h localhost -i acl-test-own -u "$A" -P "$PW" -t "vesper/$A/#" -v -W 5
       # foreign topics -> must say "All subscription requests were denied."
       docker exec mosquitto mosquitto_sub -h localhost -i acl-test-foreign -u "$A" -P "$PW" -t "vesper/$B/#" -v -W 5

5. **TLS for the app: done (2026-09-30).** See "Current VPS broker setup" below.
   Test it from the VPS host (should print `connect: Success` and a `got:` line):

       docker exec -i bellsystems-backend python - <<'EOF'
       import time, paho.mqtt.client as m
       from mqtt.auth import _derive_password
       A = "PV25L22BP01R01"
       c = m.Client(m.CallbackAPIVersion.VERSION2, client_id="wss-test", transport="websockets")
       c.ws_set_options(path="/mqtt")
       c.tls_set()
       c.username_pw_set(A, _derive_password(A))
       c.on_connect = lambda c, u, f, rc, p: (print("connect:", rc), c.subscribe(f"vesper/{A}/#"))
       c.on_message = lambda c, u, msg: print("got:", msg.topic)
       c.connect("mqtt.bellsystems.net", 443, 30)
       c.loop_start(); time.sleep(8); c.loop_stop()
       EOF

### Current VPS broker setup (as of 2026-09-30)

- Container `mosquitto`, image `iegomez/mosquitto-go-auth:latest`.
- Config: host `/home/bellsystems/stacks/mosquitto/config/mosquitto.conf`, mounted at
  `/etc/mosquitto/mosquitto.conf` (Docker images usually use `/mosquitto/config`; this
  one does not). Passwd file: `/home/bellsystems/stacks/mosquitto/passwd`.
- Already set correctly: `backends files,http`, `http_host 172.20.0.1` (Docker bridge),
  `http_port 8000`, both URIs, `http_method post`, `params_mode form`,
  `response_mode status`, `cache true`, `cache_reset true`.
- `allow_anonymous false` ✓. `auth_cache_seconds 60`, `acl_cache_seconds 60` ✓.
  `auth_opt_hasher` is not set (a harmless "defaulting to PBKDF2" warning at start).
- Listeners:
  - `listener 1883 0.0.0.0`: plain TCP, **boards only** (ESP32 can't spare RAM for TLS).
    Published on the host as `1883:1883`.
  - `listener 8083` + `protocol websockets`: plain WebSockets, **not published** on the
    host. Only reachable from the Docker network `npm_npmnet`.
- TLS is terminated by **NPM** (NPMplus, container `npm-npm-1`), which also renews the
  Let's Encrypt certificate. Proxy host: `mqtt.bellsystems.net` → `http://mosquitto:8083`,
  Force SSL + HTTP/2 on. Advanced config:

      proxy_read_timeout 3600s;
      proxy_send_timeout 3600s;

  (Without these, NPM drops idle MQTT connections after 60 s.) NPMplus has no
  "Websockets Support" toggle; WebSockets are always on.
- Compose file: `/home/bellsystems/stacks/mosquitto/docker-compose.yml`. The service is
  on `default` (reaches the backend at `172.20.0.1:8000`) **and** the external network
  `npm_npmnet` (so NPM can reach `mosquitto:8083`). Removing `default` breaks all
  board logins.
- Rollback for the TLS change: `.bak` copies of `mosquitto.conf` and `docker-compose.yml`
  sit next to the originals. Restore them, run `docker compose up -d` in that folder,
  then delete the NPM proxy host.

## Known gaps / decisions

- **Console backend port 8000 is public.** It is published as `0.0.0.0:8000` and the VPS
  has no firewall (Docker-published ports bypass host firewall rules anyway), so anyone
  can call `/mqtt/auth/user` and `/mqtt/auth/acl`. The passwords can't realistically be
  guessed, so it's not urgent, but these endpoints are meant only for Mosquitto.
  To be fixed separately, carefully, so nginx/NPM keep reaching the Console.

- **Old-firmware boards use a different command topic.** Broker logs (2026-09-30) show
  some boards (e.g. `PV26B02BP01R01`, `BSVSPR-26C20B-STD10R-2KCDPH`) subscribing to
  `vesper/{serial}/control`, not `vesper/{serial}/control/command`. The app ACL only
  allows publishing to `control/command`, so the app cannot command those boards until
  they run current firmware. (The Console's own `publish_command` has the same limitation.)
  Do not widen the ACL to `control` without deciding it deliberately.

- **FlutterFlow writes to `user_list`.** If the app changes `user_list` directly in
  Firestore (e.g. a claim flow), it must also update `device_serials` the same way
  (ArrayUnion / ArrayRemove of the serial on the user doc). Otherwise that user
  cannot reach the device over MQTT. The app owner will handle this in the app.
- **The `device_users` subcollection** on devices (older way of recording device users,
  read first by `devices.service.get_device_users`) is **intentionally ignored** by
  the backfill and the MQTT ACL. Left as is by decision.
- The two Console assignment paths store `user_list` entries in different formats
  (path strings vs DocumentReferences). This predates this work; all readers accept both.
- ACL denies are logged at INFO, which is not shown by default. Auth denies and
  legacy-password logins are WARNING.

## Tests

    cd backend
    pip install pytest    # not in requirements.txt
    python -m pytest tests

`tests/test_mqtt_auth.py` covers the auth and ACL endpoints.
`tests/test_device_serials_sync.py` covers the device PUT sync.
Firebase and Firestore are mocked.