Document Control	Institutional Specification
Document Title	Disaster Recovery and Data Restoration SOP
Document ID	HWB-QMS-9.3
Version	4.1.0
Status	APPROVED
Author	Peter (Recovery Specialist) & George (Systems Architect)
Approved By	Humberto Dominguez, CEO
Effective Date	10/02/2026
ISO Standard	ISO 9001:2015 Clause 7.1.3 & ISO 27001:2022 A.12.1 (Operational Continuity)

---

Standard Operating Procedure: Disaster Recovery and Data Restoration

1.0 Purpose & Continuous Recovery Mandate

This procedure defines the protocol for recovering company operations, databases, application containers, cryptographic keys, and AI agent brains following a hardware failure, data corruption event, or cloud disruption. In strict adherence to our Continuous Recovery Mandate, this SOP is updated immediately following any new software installation, container dependency upgrade, or database migration.

2.0 Scope

Applies to all Linux host machines, Docker containers, PostgreSQL databases, Microsoft 365 integrations, and AI agent state directories across HWB Cleaning Services LLC.

3.0 Peter's 6 Institutional Recovery Directives

Directive	Execution Cadence	Standard & Target
1. Binary DB Snapshot	Hourly	Instant recovery for binary PostgreSQL database states via automated pg_dump and WAL shipping.
2. Environment Freeze	Daily	Secure encrypted archive of .env configurations, secret salts, and Docker environment variables.
3. Ghost Checkpoints	Pre-Restart	Automatic local Git branch checkpoint created before every system restart or container teardown.
4. Institutional Mirror	Weekly	Off-site synchronization of the complete SigmaFidelity™ Brain to sovereign physical storage.
5. Surge Protector	Every 15 Minutes	Disk-usage watchdog check enforcing a 500MB runaway log threshold to prevent disk exhaustion.
6. Vault Guardian	Daily	Incremental backup of all active operational folders and QMS documentation vaults.

4.0 System Dependencies & Cryptographic Vault Configuration

The system infrastructure relies on the following core dependencies (updated October 01, 2026):

• Operating System: Linux / WSL2 Ubuntu 22.04 LTS.
• Container Fleet: Docker 26+ running containers: hwb_web_app (Gunicorn Flask, Port 5000), hwb_postgres_dev (PostgreSQL 13, Port 5432), hwb_compliance_engine (Nginx static QMS, Port 80), hwb_traffic_director (Port 8000), and hwb_agent_worker (Autonomous Worker Container).
• Autonomous Worker Browser Engine: playwright 1.63.0 and Chrome Headless Shell (v1243) installed natively in hwb_agent_worker for portal crawling.
• Statewide Hunter Crawler Fleet: scripts/hunter_portal_crawler.py continuously scouts 35 Texas municipal and ISD procurement portals on an autonomous 6-hour cycle.
• Python Cryptography Core: cryptography 48.0.0 installed in hwb_web_app for deterministic AES-256 Fernet ciphers.
• Environment Secrets:
  • SECRET_KEY & PII_ENCRYPTION_KEY: Cryptographic seed keys for database column encryption. Never committed to Git.
  • DATABASE_URL: PostgreSQL connection string formatted as postgresql://postgres:...@localhost:5432/hwb_crm.
  • AZURE_CLIENT_SECRET: Microsoft Graph API OAuth secret. Institutional Alert: Secret rotation required prior to expiration on 03/02/2027.
• Database Schema State (Migration 036 Baseline): Unified relational schema up to Migration 036 (scripts/migrate_036_client_breadcrumbs.py), providing:
  • ClientBreadcrumbs: Client DOM telemetry with session ID, event type, and target selectors.
  • SecurityAuditLogs: Protected by Write-Once-Read-Many (WORM) trigger prevent_security_audit_mutation() blocking all updates and deletes.
  • GovernmentPrograms, RackTelemetryHistory, GeneralContractors, InstitutionalBids, ConstructionBids, and SCA_WageDeterminations.
  • AES-256 encrypted PII vaulting on Employees.
• Automated Data Retention: Native PostgreSQL stored function purge_expired_client_breadcrumbs(retention_days=30) automatically purges telemetry logs older than 30 days to enforce ISO 27001 Control A.8.10 data minimization.
• Timezone Standard: All system timestamps, database audit columns, and telemetry records are strictly pinned to Texas Central Time (America/Chicago / CDT/CST).
• Session Timeout Standard: All authenticated administrative, backoffice, and executive web sessions automatically invalidate after 30 minutes of inactivity.
• First-Party DOM Sensor: static/js/sigma_breadcrumbs.js (< 3.5KB vanilla JS) capturing page views, button clicks, rage clicks, and JavaScript runtime errors with automatic client-side PII sanitization.
• Dual-Zone Air-Gap: Third-party visual replay scripts (Microsoft Clarity) strictly isolated to anonymous public marketing pages. Authenticated internal backoffice portals utilize 100% sovereign first-party telemetry.
• Automated Quality Gates:
  • scripts/tessa_regression_suite.py: 11-module platform verification battery supervised continuously in hwb_agent_worker.
  • scripts/yamamoto_bid_test_suite.py: 8-module bidding engine verification battery certifying institutional estimating logic.

5.0 Step-by-Step Restoration Procedures

5.1 Cold Server Rebuild (Total Catastrophe Recovery)

1. Clone primary repository: git clone https://github.com/Humbertoed11/gemini_projects.git
2. Restore frozen .env file from the Vault Guardian archive into HWB-COMPANY/HWB-IT/HWB-IT-WEBSITE/.env.
3. Execute master startup sequence: bash scripts/startup_master.sh
4. Verify Docker containers are healthy: docker ps

5.2 Database Restoration & Migration Alignment

1. Restore the latest binary PostgreSQL backup: docker exec -i hwb_postgres_dev psql -U postgres -d hwb_crm < backup_latest.sql
2. Run migration checks inside web container: docker exec hwb_web_app python3 scripts/migrate_036_client_breadcrumbs.py
3. Verify lead count via audit endpoint: curl -s http://localhost:5000/api/v1/db-audit
4. Run automated regression gate: python3 scripts/tessa_regression_suite.py and python3 scripts/yamamoto_bid_test_suite.py

5.3 The Panic Button (Working Tree Clean Reset)

If local uncommitted code disrupts container execution: git reset --hard HEAD && git clean -fd followed by docker restart hwb_web_app.

5.4 Sovereign Weather Catastrophe Recovery Appliance (Drive D:\ Grab-and-Go)

In accordance with CEO Humberto Dominguez's executive directive for catastrophic weather events (tornadoes, hurricanes, freezes, floods, or hardware destruction), physical Drive D:\ operates as an air-gapped, sovereign Grab-and-Go Recovery Appliance:

• Drive Inventory: Contains ubuntu-ext4.vhdx (82 GiB raw virtual disk with full OS, /gemini_projects, /hexgrowth, and .env keys), database_snapshots/hwb_dev_db.dump (47.1 MB), database_snapshots/hex_dev_db.dump (21.2 MB), 1-click launcher restore-system.bat, and emergency documentation (README.html and EMERGENCY_RESTORE_INSTRUCTIONS.txt).
• Foreign Machine Recovery: Plug into ANY Windows 10/11 computer, right-click restore-system.bat, select "Run as administrator", and type RESTORE. In under 5 minutes, both HWB (:5000) and HexGrowth (:5100) are 100% operational with all 27,984 leads and PostGIS GIS layers intact.
• Pre-Storm Refresh Routine: Run D:\run-backup.ps1 on primary workstation before major weather events to capture the latest relational database freeze and sync the virtual disk.

6.0 Verification & Continuity Sign-Off

• Web application responds with HTTP 200 at http://mop.test:5000.
• PostgreSQL connection pool establishes within 1.0 ms latency.
• WORM immutability trigger actively prevents modification of SecurityAuditLogs.
• Telemetry breadcrumb ingestion (POST /api/v1/telemetry/breadcrumbs) records client interactions with 0 errors.
• SQL Brain synchronization (docker exec hwb_web_app python3 /app/scripts/sigma_sync.py) completes with 0 errors.

---

7.0 Revision History

Version	Date	Author	Change Description
4.1.0	10/02/2026	Peter & George	Codified Section 5.4 Sovereign Weather Catastrophe Recovery Appliance (Drive D:\ Grab-and-Go architecture), enabling bare-metal 1-click restoration of both HWB Cleaning Services LLC and HexGrowth on any foreign Windows machine via restore-system.bat/ps1. Automated 68.3 MB dual-database freeze (hwb_dev_db.dump and hex_dev_db.dump) resolving Docker Desktop volume segregation and storage capacity constraints on Drive D. Approved by Humberto Dominguez, CEO.
4.0.0	10/01/2026	Peter & George	Upgraded per Continuous Recovery Mandate following Migration 036 (ClientBreadcrumbs table, sigma_breadcrumbs.js DOM sensor, and purge_expired_client_breadcrumbs(30) automated data lifecycle). Codified SecurityAuditLogs WORM immutability trigger, 30-minute idle session timeout, Texas Central Time standard, Yamamoto Moto AI Estimator test gate, and dual-zone air-gapped observability architecture. Approved by Humberto Dominguez, CEO.
3.4.0	09/25/2026	Peter & George	Upgraded per Continuous Recovery Mandate following installation of PyMuPDF 1.26+ and BeautifulSoup4 in requirements.txt (BUG-093) and integration of Tessa Test (scripts/tessa_regression_suite.py) as continuous 7-module supervisor daemon in worker_launcher.py (ARCH-008). Approved by Humberto Dominguez, CEO.
3.3.0	09/25/2026	Peter & George	Upgraded per Continuous Recovery Mandate following Migration 026 (Provisioned RackTelemetryHistory table in PostgreSQL and integrated 7-rack SPC snapshot persistence) and deployment of Telegram Operations Command Node v3.0. Approved by Humberto Dominguez, CEO.
3.2.0	09/23/2026	Peter & George	Upgraded per Continuous Recovery Mandate following Migration 025 (Master GeneralContractors Directory, 4-Point Vetting Scorecard, and Autonomous Profile Enricher). Approved by Humberto Dominguez, CEO.
3.1.0	09/22/2026	Peter & George	Upgraded per Continuous Recovery Mandate following installation of Playwright 1.63.0 and Chrome Headless Shell in hwb_agent_worker container and Statewide Hunter Contract Crawler. Approved by Humberto Dominguez, CEO.
3.0.0	09/21/2026	Peter & George	Upgraded per Continuous Recovery Mandate following Migration 016 and cryptography 48.0.0 installation. Approved by Humberto Dominguez, CEO.
2.0.0	05/21/2026	George	Modernized to post-May 1st, 2026 baseline. Standardized under Everyday Words.
1.0.0	02/28/2026	Gemini	Initial Release.