Appearance
Comms service — runbook
What to do when something is wrong, and how to deploy when nothing is.
- Status: live from phase 7. Phase 6 extends it with DLQ replay and alerting.
- Scope:
comms-serviceatcomms.services.vroum.tech, and the parts ofvroum-appthat talk to it. - Design: comms-service-design.md. This file assumes it and does not repeat it.
1. Where everything is
| Thing | Where |
|---|---|
| Service site | Forge, isolated site comms.services.vroum.tech, own Linux user |
| Service database | Postgres comms on the same server, own database user, no grants on the app's database |
| Broker | RabbitMQ on this same server, bound to 127.0.0.1 — see §8 and comms-service/deploy/rabbitmq.md |
| App | Same server, separate Forge site and Linux user |
| Daemon logs | Forge writes one file per daemon under the site's .forge/ directory; read them from the Forge UI or tail them as the site user |
| Application log | storage/logs/laravel.log under the site root, readable only by the site user |
The three domains, and which is which
Easy to confuse for six months and then impossible to remember. They are deliberately separate.
| Domain | What it is | Who talks to it |
|---|---|---|
| the app's own domain | the customer-facing application | shops, clients, browsers |
email.vroum.solutions | the email sending and receiving domain | Mailgun, and every customer replying to a shop |
comms.services.vroum.tech | the service hostname | Twilio and Mailgun webhooks, and vroum-app over localhost |
A service hostname has no reason to sit under the customer-facing domain, and keeping it apart means a certificate or DNS problem on one cannot take out the others.
2. Deploying
Manual and gated: Actions → Release → Run workflow on comms-service. It refuses unless CI has passed for that exact commit, then calls the Forge deploy webhook. Forge runs deploy/deploy.sh from the repository.
The order in that script is load-bearing:
composer install --no-dev --optimize-autoloaderphp artisan migrate --force— before the restart, so new code never meets an old schemaconfig:cache,route:cache,event:cachephp artisan bus:restart
bus:restart is the entire zero-downtime story. It writes a timestamp the workers compare between messages. Each finishes the message in flight, exits 0, and supervisor starts a fresh process on the new code. Nothing is redelivered and nothing is killed.
Never hard-restart the daemons instead. A supervisor restart sends
SIGTERMand starts astopwaitsecscountdown against a worker that may be mid-Twilio-call. That is precisely theSIGKILLthe lifecycle work exists to avoid.
It works because every daemon reads the same cache store. This service uses the database cache, so on one server with one database they all see the flag. A file store would restart only the process that ran the command and look like it worked.
Rolling back
git revert the commit, push, run Release again. There is no "redeploy the previous release" button, and that is fine: a revert leaves a trail of what was undone and why.
A migration is not rolled back by a revert. If the bad release added a column, the revert removes only the code. Decide explicitly whether to write a compensating migration — a nullable column left behind is harmless and usually the right answer.
3. The daemons
Forge daemons, and both sites have them — the bus has two ends. Each runs as its own isolated site user, never forge: a daemon left as forge cannot read .env or write storage/, and fails in ways that read like application bugs.
comms-service
| Command | Processes | Notes |
|---|---|---|
php8.4 artisan bus:consume sms | scale as needed | competing consumers, prefetch=1 |
php8.4 artisan bus:consume email | scale as needed | its own queue so a slow send never sits in front of SMS |
php8.4 artisan bus:consume numbers | exactly 1 | see below — this one is not a tuning choice |
php8.4 artisan bus:outbox:relay | 1 | publishes this service's own events |
vroum-app
| Command | Processes | Notes |
|---|---|---|
php8.4 artisan bus:consume app.comms | 1–2 | applies status, inbound SMS and email onto threads |
php8.4 artisan bus:outbox:relay | 1 | publishes commands to the service |
php8.4 artisan horizon | 1 | the app's genuine queued work — unrelated to the bus |
php8.4 explicitly rather than php: a server with more than one PHP installed will otherwise pick whichever is first on the daemon's PATH, which is not necessarily the one the site runs under.
Forge daemon settings
Per daemon, in Forge's Daemons tab on the relevant site:
| Field | Value |
|---|---|
| Directory | the site's path, e.g. /home/comms-service/comms.services.vroum.tech |
| User | the site user, never forge |
| Processes | per the tables above |
| Stop Wait Seconds | 60 |
| Stop Signal | SIGTERM (supervisor's default — only change it if Forge exposes something else) |
stopwaitsecs is the one to change. Supervisor defaults to about 10 seconds and then sends SIGKILL. A consumer waiting on a slow Twilio call will exceed 10 comfortably, and SIGKILL mid-send is exactly what the graceful shutdown work exists to avoid. Sixty is above the slowest provider call we have seen with room to spare; too high costs a slow deploy, too low costs a killed worker.
Not daemons
php artisan bus:declare— one-shot, run fromvroum-appafter the broker is set up and after any broker rebuild. It owns the topology; the service declares nothing and fails loudly if the topology is missing.- The scheduler — use Forge's Scheduler tab on
comms-service, which installs a per-minuteschedule:runcron. Notschedule:work, which is the long-running variantcompose.yamluses because containers have no cron. The service's schedule covers domain polling, body pruning and suppression expiry.
Start the broker's topology before the daemons
bus:declare first, daemons second. Started in the other order the comms-service consumers exit on a missing queue and supervisor restarts them in a loop — harmless, and very confusing if you are not expecting it.
Why the numbers daemon is pinned at one process
Not performance. Concurrency safety on that queue rests on an undocumented Twilio behaviour — IncomingPhoneNumbers.create returns 201 with the existing record for a number the account already owns — plus a friendly-name comparison to tell "I bought this" from "someone already had it" (design doc §6.6). Two workers racing the same request_ref do not double-charge today, and only because of that. Provisioning is rare enough that a second process buys nothing.
comms.subaccount.provision rides the same queue on purpose: a shop's subaccount must exist before a number can be bought into it, and one ordered queue buys that ordering for free.
If you scale it, you are betting money on an undocumented provider behaviour. Don't.
4. When the DLQ is not empty
A DLQ you don't watch is a data-loss queue. Depth > 0 is always worth looking at.
- Read one message in the broker's management UI. The envelope carries
type,correlation_id,idandtenant.shop_id. - Grep both services for that
correlation_id. It is on the scope in Sentry and in the structured logs of both sides, which is the whole reason it exists. - Classify it:
| What you see | What it means | What to do |
|---|---|---|
No handler registered for bus type | a contract type this release does not know | deploy the release that handles it, then replay |
| A consumer exception, same message repeatedly | a real bug | fix, deploy, replay |
Refusing a … command whose payload and envelope disagree | a publisher bug or a tampered message | do not replay; investigate the publisher |
| Malformed envelope | a non-platform publisher | do not replay; find what is publishing |
Replay tooling lands with phase 6. Until then, a message can be re-published by hand from its envelope JSON — and that is the honest state of it.
The two sides do not behave alike, and it matters here
vroum-app dead-letters a type it has no handler for: a DLQ message can be inspected and replayed once a handler exists, and DLQ depth is alerted on.
comms-service acks and drops it. The comment on BusConsume::HANDLERS calls that normal during a rollout, and for an event it is. For a command it is not the same thing at all: an unrecognised comms.*.send is a customer's message that no longer exists, with a healthy-looking queue behind it and nothing in any DLQ to find.
BusHandlerCoverageTest now makes that unreachable through the front door — a command with no consumer fails the build. What it cannot cover is a newer publisher against an older consumer, which is exactly the rollout window the comment is about. Worth an explicit decision before launch: deploy the service ahead of the app every time, or make unknown commands dead-letter here too.
5. Re-pointing webhooks after an environment move
TwilioNumbers::buy() writes smsUrl and statusCallback onto a number at purchase time and never again. Changing COMMS_PUBLIC_URL does nothing to numbers that already exist.
The failure is asymmetric and quiet: sending keeps working — the status callback is handed over per message — while receiving silently stops. A shop's customers text it and nothing arrives, with no error on this side.
bash
php artisan comms:number:rewebhook --all --dry-run # see what would change
php artisan comms:number:rewebhook --all # do it
php artisan comms:number:rewebhook --shop=5 # one shop--all includes the platform account even when it has no credential row — where numbers bought before subaccounts existed live, and exactly the ones an iteration over database rows would forget.
Run it after: the first deploy, any COMMS_PUBLIC_URL change, and any restore of the service database into a different environment.
Mailgun is not covered by this command. Its event webhooks and the inbound route are account-level, not per-number, and are re-pointed through the Mailgun API or console. The inbound route is a catch_all() forward; if it points at a dead URL, mail is accepted by Mailgun and forwarded into nothing.
6. APP_KEY — the secret that cannot be lost
provider_credentials.secret holds every shop's Twilio subaccount auth token, cast encrypted. That token is what signs the webhooks the shop's numbers generate.
Lose the key and those tokens are unrecoverable. The symptom is not a decryption error in a log: it is every one of that shop's inbound messages returning 403, and the only remedy is re-provisioning the subaccount and re-pointing its numbers.
- The production key is generated once and stored in 1Password, not only on the server.
- A server rebuild that regenerates
APP_KEYdestroys the tokens. Restore the key before restoring the database.
Rotating it safely
bash
# 1. Old key to the front of APP_PREVIOUS_KEYS, new key as APP_KEY.
# 2. Then, and only then:
php artisan comms:credentials:reencrypt --dry-run
php artisan comms:credentials:reencrypt
# 3. Only once that succeeds, remove the old key from APP_PREVIOUS_KEYS.An unreadable secret fails the command loudly rather than being skipped. That means the key it was written with is gone, and the fix is a different and much larger job — see above.
7. Backups
The service database holds provider credentials, sender inventory, domain verification state and the delivery audit trail. Losing it means re-provisioning every shop.
- Daily dumps, stored off the box.
- The app's database is backed up too — confirm this rather than assuming Forge is doing it.
An untested backup is a belief. Test a restore by loading a dump into a scratch database and confirming a shop's subaccount token still decrypts:
bash
php artisan tinker --execute 'print(filled(App\Models\ProviderCredential::whereNotNull("shop_id")->first()?->secret) ? "decrypts\n" : "FAILED\n");'That one step proves both the backup and the APP_KEY handling, which is why it is the check worth repeating.
8. The broker
RabbitMQ on this server, bound to loopback, shared by both sites. Setup and the config that matters: comms-service/deploy/rabbitmq.md. Why self-hosted rather than managed: design doc §11.
Reading it. The management UI is on loopback, so tunnel to it rather than opening a port:
bash
ssh -N -L 15672:127.0.0.1:15672 forge@<server> # then http://127.0.0.1:15672Or from the box:
bash
sudo rabbitmqctl list_queues -p vroum name messages consumers
sudo rabbitmqctl status | grep -A3 'alarms'| Symptom | Likely cause |
|---|---|
| Consumer count 0 on a queue | that daemon is not running — check supervisor as the site user |
Depth climbing on comms.commands.* | consumers are down or failing; check the daemon log |
Depth climbing on app.comms | the app's bus:consume is down |
| Outbox rows unpublished and ageing | the relay is down, or the broker is blocking publishers — see below |
NOT_FOUND on startup, consumers exiting | topology is missing; re-run php artisan bus:declare from the app |
The failure that makes no noise
A memory or disk alarm does not crash RabbitMQ — it blocks publishing connections. The relay stops publishing, the app degrades to "queued, not sent" exactly as §10 promises, and nothing errors anywhere. No dead letters, no exceptions, no log line.
This is why the alert is on outbox age, not only DLQ depth. A blocked publisher is invisible to every other signal.
bash
sudo rabbitmqctl status | grep -A3 alarms # `{resource_limit,memory,...}` or diskBoth watermarks are set explicitly in rabbitmq.conf precisely so this is rare — the defaults (40% of RAM, 50MB disk) are wrong for a box shared with Postgres.
When the broker is unreachable
The app degrades rather than erroring: Message rows are written, outbox rows accumulate unpublished, the relay retries. Nothing surfaces at the user.
A single-node restart — an upgrade, a reboot — is a bus outage of that shape. Expect connection resets and heartbeat timeouts in the daemon logs rather than a clean refusal; the relay logs Missed server heartbeat on its way to retrying, and recovers on its own.
If the broker is rebuilt
Topology is in no backup. Re-run php artisan bus:declare from the app, then confirm consumers reconnect. Anything that was sitting in a dead-letter queue at the time is gone — that is the one piece of state with no copy elsewhere, and the reason to keep DLQ depth at zero rather than to start backing up /var/lib/rabbitmq.
9. Environment split
Dev and production must never share provider state. See design doc §11.
| Development | Deployed | |
|---|---|---|
| Service URL | cloudflared quick tunnel, rotates | https://comms.services.vroum.tech |
| Twilio | separate subaccounts under the parent, dev-only | shops' own subaccounts |
| Mailgun sending domain | separate from production's | email.vroum.solutions |
| Email driver | smtp → mailpit | mailgun |
| SMS driver | messagepit | twilio |
| Buying numbers | COMMS_NUMBERS_PURCHASE_ENABLED=false | true, deliberately |
The environment matrix that lives only in someone's head is the one that sends a test SMS from a customer's number.
10. First-run checklist for a new environment
None of these has ever run outside dev. In order:
bash
# Broker first — the service fails loudly on a missing topology, by design.
# See comms-service/deploy/rabbitmq.md, then from the vroum-app site user:
php artisan bus:declare
# Then, as the comms-service site user:
php artisan migrate --force
php artisan comms:domain:add email.vroum.solutions --verified # or :register for a new one
php artisan comms:sender:add ... # per shop, or let subaccount provisioning do it
php artisan comms:number:rewebhook --all # point existing numbers hereShops created after this get their email identity and subaccount automatically, from comms.subaccount.provision published inside the shop-creation transaction. Shops that predate it are backfilled by re-publishing that command — the subaccount half short-circuits and the email half still lands.
Then confirm, from outside the box:
bash
curl -sS https://comms.services.vroum.tech/up # 200
curl -si https://comms.services.vroum.tech/query/numbers/available # 403 from nginx
curl -si -X POST https://comms.services.vroum.tech/webhook/twilio/status -d 'MessageSid=SM1' | head -1The last one must be 403: the tunnel reaches the app and the signature check is doing its job. 404 means nginx is not routing, and 200 would mean the middleware is not attached at all.