TextNLPClassifierApp Constitution

Core Principles

I. Modularity & CLI-First Interoperability

Every capability (crawling, multi-engine extraction, consensus selection, markdown conversion, NLP/LLM classification, and consolidation runtime) MUST be implemented as a modular, decoupled component under src/ or scripts/. Every core component MUST expose a standard CLI interface supporting both human-readable logging and structured JSON input/output over standard streams, with explicit exit codes for automated pipeline orchestration.

II. Determinism, Atomic Operations & Data Integrity

Text processing, shingle extraction, F_1 consensus calculation, metadata matrix resolution, and formatting transformations MUST be strictly deterministic, reproducible, and idempotent. All filesystem writes MUST execute via atomic transactional write patterns (temporary file staging followed by atomic rename) to eliminate corrupted state. Source data and multi-extractor payloads MUST be preserved non-destructively.

III. Multi-Engine Consensus & Fault-Tolerant Fallback

Data extraction and parsing MUST NOT rely on single points of failure. The architecture enforces multi-engine extraction (Trafilatura, Newspaper4k, Readability) coupled with automated consensus scoring (F_1 5-token shingles) and strict fallback heuristics. Edge cases (paywalls, empty bodies, anti-bot challenges) MUST be handled gracefully with explicit diagnostics.

IV. Test-First & Empirical Validation (TDD & Regressions)

Test-Driven Development (TDD) and empirical test suites are mandatory. Unit tests, integration contracts, and regression suites MUST cover parser algorithms, CLI flags, exit codes, and error conditions before deployment. A 100% passing test baseline MUST be maintained (pytest), accompanied by static linting (ruff) and static type validation (mypy/pyright).

V. Observability, Structured Logging & Traceability

All pipeline phases (crawling, article selection, consolidation, LLM/NLP classification) MUST emit structured, contextual telemetry. Operations MUST log execution metrics, engine scores, token consumption, and decision paths. Tracing integrations (e.g., Langfuse) MUST provide full visibility into prompt performance, latency, cost, and classification rationale.

Technical, Security & Environmental Constraints

Language & Runtime: Python >=3.10 with strict type annotations across all modules.
Code Quality: Linting and formatting governed by ruff (100-character line limit) and type safety verified via mypy.
Security & Secrets: Zero hardcoded credentials or API keys; configuration MUST be loaded from environment variables (.env) with schemas documented in .env.example.
Stealth & Web Automation: Browser automation (Foxcape/Camoufox) MUST enforce appropriate delays, backoff, and fingerprint management without exceeding target service rate limits.

Development Workflow & Quality Gates

Specification First: Features and structural changes MUST be documented in specifications (specs/ or docs/) with clear functional requirements and architecture decision records (ADRs).
Mandatory Quality Gates: Every code contribution MUST pass:
1. ruff check and ruff format --check (clean formatting and linting).
2. mypy / pyright (no type check violations).
3. pytest (full test suite execution with 100% passing rate).
Commit Standards: Atomic commits using Conventional Commits convention (feat:, fix:, docs:, test:, refactor:, chore:).

Governance

This Constitution represents the supreme architectural and development policy for TextNLPClassifierApp. All development, automated subagents, and pull requests MUST comply with the principles and quality gates established herein. Amendments require explicit documentation of rationale, team review, and formal version increments:

MAJOR: Incompatible principle removals or foundational architectural shifts.
MINOR: Addition of new principles, governance rules, or expanded standards.
PATCH: Wording improvements, clarifications, and non-semantic corrections.

Version: 1.0.0 | Ratified: 2026-08-24 | Last Amended: 2026-08-24

4.8 KiB Raw Blame History