---
name: staging-mirror
description: Per-instance Staging/Canary mirror — test MoonStack updates on a data-identical copy before applying to live. elmadina's manual staging (s-elmadina) + the generic feature plan.
updated: 2026-06-28
plans:
  - https://moonui.elbaset.com/moonstack-staging-feature-plan.html
  - https://moonui.elbaset.com/elmadina-staging-mirror-plan.html
---

# Staging / Mirror environment

Goal: a **data-identical mirror** of a live install; apply the MoonStack update on the mirror FIRST; if it passes, apply to live. Golden rule: the mirror MUST have its **own DB copy** (not the live DB) — an update runs `migrate`, which would break live if shared.

## ✅ elmadina staging — LIVE (manual, 2026-06-28)
- **Account:** separate cPanel `selmadina`, domain **`s-elmadina.elbaset.com`**, docroot `/home/selmadina/public_html` (same server). DB **`selmadina_stg`** (created via `uapi Mysql create_database`).
- **Built by:** rsync elmadina `public_html` → selmadina (excl. backups/logs) → `chown selmadina:nobody public_html` (cPanel needs group nobody on the docroot, **750**, else 403) → `.env` (DB `selmadina_stg` + `APP_URL=https://s-elmadina.elbaset.com`, **same APP_KEY** as elmadina so copied data decodes) → restore staging `config.json` apiUrl → `rm bootstrap/cache/config.php` (else it uses live DB!) → full `mysqldump elmadina_db | mysql selmadina_stg`.
- **Refresh link (the "button"):** `https://s-elmadina.elbaset.com/staging-refresh.php?key=d25a375db195af630b3ceb89` — PHP, no shell: `CREATE TABLE LIKE` + `INSERT SELECT` server-side per table (staging DB user has SELECT grant on `elmadina_db`), + truncates sessions/jobs/cache. **Copies DATA ONLY — NOT code.** (token-gated; ⚠️ don't paste the URL with the markdown backticks → Forbidden).
- **Red STAGING banner:** a fixed red frame + `STAGING` corner ribbon + bottom tag «نسخة تجريبية (Staging) — s-elmadina» (`pointer-events:none`), so the owner always knows they're on the mirror, not live. It lives **inside `public/app/index.html` only**.
  - **⚠️ Why it keeps disappearing — ROOT CAUSE (confirmed 2026-06-29):** running a **MoonStack UPDATE on staging replaces the whole FE (incl. `index.html`) → the banner is wiped.** And that update is *exactly* the thing you do on staging (test the release), so the banner vanishes precisely when you start testing. Proven: `storage/moonstack/{update.log,version.json}` stamped 10:23:14 (staging updated to `4.0.34`) = the moment the banner disappeared. Survives a DATA refresh (`staging-refresh.php`) but NOT an update. (`hook_deploy_worker.php` cron is a DIFFERENT product's deploy hub — not Moon ERP — red herring.)
  - **✅ Restore it (no shell, idempotent):** hit **`https://s-elmadina.elbaset.com/staging-banner.php?key=<same token as staging-refresh>`** — re-injects the marker-wrapped block before `</body>` (no-op if already present; add `&force=1` to force). NEW file `public/staging-banner.php` (mirrors staging-refresh's token gate; also `ea-php82-cli staging-banner.php <token>`). **Run it AFTER every staging update.** 🔁 Standing workflow: owner releases on main → **updates s-elmadina FIRST + tests there** → hit `staging-banner.php` → verify → only then promote to elmadina live.
  - **✅ Durable fix INSTALLED (owner-authorized 2026-06-29):** a per-minute `selmadina` cron `* * * * * /opt/cpanel/ea-php82/root/usr/bin/php /home/selmadina/public_html/public/staging-banner.php <token>` auto-restores the banner ≤60s after any update (idempotent skip-if-present). Verified (stripped → restored in ~5s). The banner no longer needs manual restoring.

## ⚠️ Gotchas
- **Refresh ≠ code update.** The refresh button syncs the DB; it does NOT bring code fixes. To test a code fix on staging you must deploy the code (or run a real MoonStack update). Symptom: "I refreshed and the fix still isn't there" → staging is on old code.
- After an FE redeploy to staging: restore staging `config.json` (apiUrl) + re-inject the banner. Use `\cp -f` (cp is aliased to `-i`).

## 🐞 First real bug staging caught — price-list create 500 (2026-06-28)
Adding a SECOND price list on `s-elmadina` returned **500** (`POST /api/lis/price-lists`). Log: `1062 Duplicate entry 'PL-001'/'PL-003' for key 'lab_price_lists_code_unique'`. **Root cause (a systemic onboarding gotcha):** `lab_price_lists.code` has a **global unique** index, but onboarding imported the lab's price lists directly (`PL-001..PL-011`) **without advancing the `lis/price_list` sequence**, so the counter sat at 1 → `SequenceService::generateNext` reproduced an existing code → 1062 → 500. **Not staging-specific — elmadina live has the same data + code**, so it would 500 there too.
- **Fix (BE, `store()`):** self-heal on a duplicate-code violation — jump the counter past the highest existing numeric suffix (incl. trashed, computed in PHP so it's driver-agnostic) via `SequenceService::advanceToAtLeast`, regenerate, retry (≤3). Only AUTO codes self-heal; a user-supplied dup stays a clean 422. Mirrors the existing **MRN self-heal** in `LabPatientController`. Ships to elmadina via **MoonStack update** (code self-heals on first add — no manual data heal).
- **✅ SWEEP DONE (BE `f92c6349c`, hazemdev) — generalized.** New reusable trait `Modules/LIS/Concerns/SelfHealsSequencedCode::createWithSequencedCode()` applied to the `store()` of **10 LIS master controllers** (price-lists [refactored onto it], doctors, external-labs, investigations, specimen-types, sections, investigation-categories, machines, insurance-contracts, packages). Data-measured on staging: **doctors (DOC-0001..0006 vs counter 0) + external-labs WOULD 500 today**; rest carry real-format codes (defense-in-depth). **`SPT-WBE` (Mode B) fixed too:** the reference seeders (`LabSpecimenTypeSeeder`/`LabSectionSeeder`) now **skip a code that exists ANYWHERE incl. trashed** (the cleanup-duplicate-specimen-types migration soft-deletes the `SPT-*` shadows; firstOrCreate's live-scoped lookup missed them → 1062 on the global unique). Code-only, NO schema/data change. +3 Pest, 117 LIS green, native review APPROVE (0 crit/high/med). 🔴 **Durable root fix deferred (needs owner OK):** the real flaw is the **GLOBAL `code`/`mrn` unique** on ~10 per-company tables → change to `unique(company_id, code, deleted_at)` via migration on `moonui_dev_be`. Owner explainer: `moonui.elbaset.com/lis-sequence-code-sweep.html`.
- **Staging was hand-unblocked** for immediate testing: deployed the fixed controller to `/home/selmadina/public_html/Modules/LIS/...` (chown selmadina:nobody) + bumped `sequence_counters.last_number`→11 (next code `PL-012`). ⚠️ a refresh-from-live resets the counter to elmadina's (1); the deployed fix then self-heals on demand.
- **Gotcha for tests:** the duplicate-key message differs by driver — MySQL `...key 'lab_price_lists_code_unique'` vs SQLite `UNIQUE constraint failed: lab_price_lists.code`. Guard with `str_contains(msg,'price_lists') && str_contains(msg,'code')`, not the MySQL key name. Also **`SUBSTRING_INDEX` is MySQL-only** (the MRN self-heal uses it → its self-heal path can't be unit-tested on sqlite); compute the max suffix in PHP instead.

## Generic feature (planned, not built)
Make it a MoonStack toggle (`staging.enabled`) any instance can turn on → own subdomain + DB copy + storage. cPanel-only (no shell): provision via cPanel UAPI or guided manual; refresh via pure-PHP PDO copy (batched/resumable); update = the existing updater on staging first. Plan: `moonstack-staging-feature-plan.html`.

## 🐞 Service worker cache — "I updated staging but the FE fix still isn't there" (2026-07-02)
Owner updated s-elmadina to a release WITH a FE fix (LIS-138 portal smart-report gate) but the old UI kept showing. **NOT a bug, NOT a stale deploy.** Verified the origin was correct: staging's `index.html` referenced the NEW build (`main-OFWGGBET.js` = v5.0.5) and the fixed chunk (`chunk-IAT64LGF.js`) was served 200 — the fix WAS deployed. **Root cause: the Angular app ships a service worker** (`/app/ngsw-worker.js` + `/app/ngsw.json` both 200). `ngsw` caches the whole app shell + assets (chunks are `Cache-Control: immutable, max-age=1y`) and **serves the cached OLD version even after the server files change** — a normal refresh doesn't help because the SW intercepts and answers from cache; it only swaps to the new version on a later full navigation / after all tabs close.
- **Diagnose without touching staging:** `curl -sI https://<host>/app/index.html` (should be `no-cache`, fresh `Last-Modified`) + `curl -s .../app/index.html | grep main-` → compare the `main-*.js` hash to the release's; if they match, the server is updated and the client is just SW-cached.
- **Owner fix (fastest → surest):** (1) open the page in an **Incognito/private window** (no SW → shows the new build immediately — the definitive test); (2) **close ALL tabs of the site** and reopen (lets the SW activate the new version); (3) DevTools → **Application → Service Workers → Unregister** + **Clear storage** → reload.
- **Standing rule:** after ANY FE update on staging/a client, the tester must do a hard SW-busting reload (incognito or close-all-tabs), not just a refresh. This is separate from the "refresh ≠ code update" DB-sync note above. Consider (future) an in-app "new version available — reload" prompt via the ngsw `SwUpdate` API so users aren't stuck on a cached build.

## 🐞 BLANK white page — `index.html` references a `main-*.js` from a DIFFERENT build (mixed manual deploy) (2026-07-07)
Client reported **s-elmadina opens to a blank white page**. Root cause = a **mismatched FE deploy**: the served `/app/index.html` referenced `main-4Q7ZUQ5J.js` (a **moonui** build hash) but the file that actually existed on staging's disk was `main-EJB4OL2V.js` (staging's own 22:23 MoonStack-update build). Because there's a **SPA `.htaccess` fallback**, the browser's request for the missing `main-4Q7ZUQ5J.js` was answered with **`index.html` (HTTP 200, `Content-Type: text/html`)** instead of a 404 — so the browser tried to execute HTML as JavaScript → bootstrap fails silently → **blank page**. The `<app-root>` never mounts.
- **How it happened:** the MoonStack update deployed a consistent build at 22:23 (main-EJB4OL2V + polyfills-6ISPNSXF + all chunks, all internally consistent — `main` references those chunks). Then a **manual FE deploy at 22:53** (30 min later) overwrote **only `index.html`** with one from a different (moonui) build, whose `main-*.js` hash doesn't exist here. The chunk/polyfills refs happened to still resolve; **only the `main` entry-point ref was dangling.** (The `staging-banner.php` cron is innocent — it only injects the banner into whatever `index.html` exists; it does not rewrite script refs.)
- **⚠️ FALSE-POSITIVE trap — the SPA fallback makes EVERYTHING look 200.** `curl -sI .../app/<anything>.js` returns **200** even for files that don't exist, because the fallback serves index.html. **So HTTP 200 ≠ file exists.** Two earlier mis-diagnoses came from this: (1) `main-4Q7ZUQ5J.js` "200" was actually the HTML shell; (2) `ngsw.json`/`ngsw-worker.js` "200" made it look like a service worker was present — but **this build ships NO service worker** (index doesn't register one; the files don't exist on disk). Always check **`%{content_type}`**, not just the status code: `curl -s -o /dev/null -w "%{http_code} %{content_type}\n" <url>` — a `.js` that returns `text/html` is a dangling ref hitting the fallback.
- **Diagnose (server-side, read-only):** compare what `index.html` requests vs what's on disk. `grep -oE 'main-[A-Z0-9]+\.js' <docroot>/public/app/index.html` then `ls <docroot>/public/app/main-*.js` — if they differ, that's the bug. Timestamps expose the mixed deploy (`stat -c '%y %n' index.html main-*.js chunk-*.js` → index newer than the build set = overwritten after).
- **✅ Fix (surgical, 1 line):** point `index.html` at the `main-*.js` that actually exists on disk. Backup first, then `perl -pi -e 's/main-4Q7ZUQ5J\.js/main-EJB4OL2V.js/g' <docroot>/public/app/index.html` + `chown selmadina:selmadina index.html`. Verify over HTTP that the entry-point now returns real JS: `curl -s -o /dev/null -w "%{content_type}\n" .../app/main-EJB4OL2V.js` must be `application/javascript` (not `text/html`), and the first lazy chunk it imports resolves 200/JS too. Client then does one hard/incognito reload.
- **Lesson (why we're moving to MoonStack-only for staging FE):** a **manual FE deploy must copy the WHOLE bundle atomically** — `index.html` + `main-*.js` + `polyfills-*.js` + ALL `chunk-*.js` from the **same** build. Copying `index.html` alone (or main alone) leaves a dangling entry-point that the SPA fallback masks as a blank page, not an error. **MoonStack update deploys the FE as one consistent unit → this class of bug can't happen.** Going forward, don't hand-deploy FE onto staging; let the updater do it.