Document IDHWB-QMS-7.9

Version2.0.0

Status● APPROVED

Clause8.2 & 8.4 (Market Requirements & External Sourcing)

Document Control
Document Title	Statewide Autonomous Contract Hunter and Portal Crawler SOP
Document ID	HWB-QMS-7.9
Version	2.0.0
Status	APPROVED
Author	George (Systems Architect & mbB)
Approved By	Humberto Dominguez, CEO
Date	09-22-2026
ISO 9001 Clause	8.2.1, 8.2.2, 8.4.1 (Control of Externally Provided Solicitations)

Standard Operating Procedure: Statewide Autonomous Contract Hunter and Portal Crawler

1.0 Purpose

This procedure establishes the rules and operating instructions for the SigmaFidelity™ Autonomous Hunter Engine. The hunter crawler continuously monitors, searches, extracts, and ingests live public janitorial, commercial custodial, and facility maintenance bidding opportunities across all municipal, county, and school district procurement networks in the entire State of Texas. This eliminates manual search effort and gives HWB instant visibility on all newly posted public tenders.

2.0 Scope & Geographic Perimeter

In accordance with the 2026-07-22 Statewide Mandate, the scouting perimeter encompasses all major metropolitan, regional, and school district jurisdictions across Texas:

Dallas-Fort Worth Metroplex: City of Dallas, City of Fort Worth, City of McKinney, City of Frisco, City of Richardson, City of Denton, City of Garland, City of Carrollton, City of Plano, City of Irving, and Collin County Government.
Independent School Districts (ISDs): Dallas ISD, Austin ISD, Mansfield ISD, Lewisville ISD, Denton ISD, Plano ISD, Frisco ISD, McKinney ISD, Richardson ISD, Allen ISD, Coppell ISD, Keller ISD, Arlington ISD, and the Central Texas Purchasing Alliance (CTPA / Midlothian ISD, covering 30+ regional districts).
Central Texas: City of Round Rock, Austin ISD, Travis County, and Williamson County.
Greater Houston: Harris County, Fort Bend County, Brazoria County, and City of Houston.
South Texas: City of San Antonio and Bexar County.
West Texas & Panhandle: City of Midland and Lubbock Power & Light / City of Lubbock.
Statewide Higher Education: University of Texas System and University of Texas at Dallas.

3.0 Monitored Public Portal Fleet (35 Target Hubs)

Portal Network	Monitored Portals	Technical Architecture & Endpoint
Bonfire Hub Network (17 Hubs)	Dallas, Fort Worth, Dallas ISD, McKinney, Frisco, Richardson, Harris County, Fort Bend County, Brazoria County, San Antonio, Bexar County, Round Rock, Austin ISD, UT System, UT Dallas, Midland, Lubbock.	Single-Page Application (Angular/SPA). Solicitations rendered via JavaScript tables. Crawled via Playwright Headless Chromium at `https://[subdomain].bonfirehub.com/portal/?tab=openOpportunities`.
IonWave Electronic Bidding Fleet (18 Portals)	City of Denton, City of Garland, City of Carrollton, Mansfield ISD, CTPA / Midlothian ISD, City of Plano, City of Irving, Plano ISD, Frisco ISD, McKinney ISD, Lewisville ISD, Denton ISD, Richardson ISD, Allen ISD, Coppell ISD, Keller ISD, Arlington ISD, Collin County.	ASP.NET Telerik RadGrid structures. Public sourcing events extracted from `tr.rgRow` and `tr.rgAltRow` table rows at `https://[subdomain].ionwave.net/SourcingEvents.aspx?SourceType=1`.

4.0 Crawling Protocol & Web Security Rules

Headless Browser Deployment: All portal crawling must be executed using Playwright Headless Chromium running inside the hwb_agent_worker container or Linux host.
Network Load & Pacing:
- Bonfire Hubs: Enforce a minimum 1.5-second polite pause between portal requests to prevent rate limiting.
- IonWave Portals: Enforce a minimum 2.0-second delay between agency endpoints to avoid Web Application Firewall (WAF) HTTP 429 warnings.
Page Load Strategy: Use wait_until="domcontentloaded" with a 20-second timeout. Wait for AJAX tables to settle before reading DOM nodes.

5.0 Poka-Yoke False-Positive Filtration System

To preserve 100% empirical database integrity and prevent non-cleaning public works projects from cluttering executive dashboards, the hunter engine enforces a strict two-stage filter:

Stage 1: Industrial Negative Exclusions (Immediate Disqualification)

If any of the following terms appear in the title or project description, the record is immediately discarded: sewer, culvert, grease trap, fleet, vehicle, truck, street sweeper, hvac duct, heavy equipment, parts only, stormwater, catch basin, dry cleaner, laundry service, water reclamation, wastewater, pipe cleaning.

Stage 2: Positive Custodial Keyword Matching

Solicitations are qualified if they match core janitorial terms: janitorial, custodial, clean, day porter, floor care, carpet extraction, sanitization, disinfection, window washing, pressure wash, power wash, facility maintenance, facilities maintenance, post-construction clean, or restroom cleaning.

6.0 Automated Attachment Harvesting & Document Staging

Upon discovering an active public opportunity, the crawler initiates automated attachment harvesting:

Institutional Workspace Provisioning: Allocates an isolated quote directory at HWB-COMPANY/HWB-QUOTES/[AGENCY]_[SOLICITATION]/.
Attachment Ingestion: Downloads primary solicitation specifications, addenda, and bid pricing sheets (.pdf, .docx, .xlsx) via Playwright or direct session streaming.
Staging Manifest Creation: Generates a machine-readable PORTAL_STAGING_MANIFEST.json recording metadata, closing dates, portal URLs, and verification checksums.
Vendor Portal Authentication Handling: Where public portals restrict bid sheet downloads behind supplier credentials, the engine creates the structured directory, records metadata, and flags status as 'Pending Takeoff Analysis' awaiting staged specification documents.

7.0 Autonomous Scope Takeoff Parsing Engine (ARCH-006 Frontier 1)

All harvested solicitation specification documents (.pdf) are processed through the autonomous PyMuPDF Scope Parser (scripts/solicitation_scope_parser.py):

Cleanable Area & Multi-Facility Extraction: Extracts individual building cleanable areas and addresses. Applies normalized key deduplication to prevent double-counting when Exhibit schedules repeat introduction scopes.
Staffing & Porter Calculation: Identifies mandatory on-site day porters and night custodians. Calibrates weekly labor hours based on facility footprint.
Procurement Governance & Key Deadlines: Automatically extracts mandatory pre-bid conference date/time, virtual meeting links (Microsoft Teams / WebEx / Zoom), dial-in numbers, site walk itineraries, question deadlines, and proposal submission closing times.
Wage Standard & Dallas Living Wage Floor: Enforces the mandatory $18.00/hr Dallas Living Wage floor on municipal bids, or DOL Service Contract Act (SCA) / Davis-Bacon wage determinations where applicable.
Takeoff Discrepancy Mandate Compliance: In accordance with the 2026-09-04 Mandate, any heuristic assumption or unlisted room schedule is explicitly labeled with [DATA DISCREPANCY: Heuristic Estimate - Field Verification Required].

8.0 SigmaEstimator Coupling & PostgreSQL Persistence

The extracted parameters pass directly into core/services/estimator.py (calculate_institutional_bid) to generate binding contract models:

Negotiation Triad Generation: Computes Published Submittal Price (21% margin), Authorized Field Close Price (18% margin), and Walk-Away Floor Price (14% margin).
Cost of Goods Sold (COGS) Breakdown: Computes base labor, statutory payroll burden (20.0%), consumable supplies, equipment depreciation, and fixed G&A overhead (5.5%).
Database Synchronicity: Updates PostgreSQL "InstitutionalBids" (setting cleanable_sqft, facilities_count, hwb_bid_total, monthly_base_rate, annual_base_rate, hourly_porter_rate, key event timestamps, and compliance_status = 'Scope Parsed / Proposal Ready').
Cryptographic Document Vaulting: Registers every harvested file in "BidDocuments" with SHA-256 integrity hash, byte size, and MIME type.
Executive Hold Mandate: No emails, inquiries, or open records requests may be transmitted to any agency without explicit sign-off from CEO Humberto Dominguez.

9.0 Autonomous Worker Execution Schedule

The hunter crawler and scope parsing engine are registered in sigma_orchestrator.py inside hwb_agent_worker and execute on a recurring 6-hour cycle (21,600 seconds) alongside the Texas Daycare Registry sync and master contract mining tools.

10.0 Verification & Zero-Defect Check

Executing python3 scripts/hunter_portal_crawler.py --harvest-only returns exit code 0 without unhandled exceptions.
Executing python3 scripts/solicitation_scope_parser.py --sync-all synchronizes all quotes and returns exit code 0 with absolute Linux/Docker parity.
PostgreSQL table "InstitutionalBids" reflects parsed square footages and calibrated proposals with status 'Scope Parsed / Proposal Ready'.
Outbound communications outbox remains 100% frozen until CEO release.

11.0 Revision History

Version	Date	Author	Change Description
1.0.0	09-22-2026	George (Systems Architect & mbB)	Initial release codifying the Statewide Autonomous Contract Hunter Engine across 35 Texas portals (18 IonWave + 17 Bonfire hubs), headless Chromium container deployment, and Poka-Yoke false-positive filtration. Approved by Humberto Dominguez, CEO.
2.0.0	09-22-2026	George (Systems Architect & mbB)	Major upgrade codifying ARCH-006 Frontier 1 (Autonomous Proposal Factory & Scope Parsing Engine). Integrated automated document attachment harvesting, PyMuPDF blueprint and specification takeoff, normalized key multi-facility deduplication, Dallas Living Wage ($18.00/hr) SigmaEstimator integration, BidDocuments cryptographic SHA-256 cataloging, and 6-hour autonomous orchestrator loop execution. Approved by Humberto Dominguez, CEO.

Document Structure