feat: add deterministic content extractor selector engine with F1 consensus

2026-08-20 22:09:43 -03:00
parent 6a45368cb0
commit ff7a50e0eb
46 changed files with 18503 additions and 2813 deletions
@@ -0,0 +1,54 @@
+# Deterministic Content Selection Checklist: End-to-End Requirements Quality
+
+**Purpose**: Validate the completeness, clarity, consistency, and measurability of requirements for the deterministic extractor selection pipeline  
+**Created**: 2026-08-20  
+**Feature**: [spec.md](../spec.md)
+
+**Note**: This custom checklist is generated by the `/speckit-checklist` command based on feature context and requirements.  
+**Review Ownership**: This checklist is a reviewer-owned requirements-quality review artifact. Mark an item `[x]` only when the reviewer determines the requirements-quality criterion is satisfied.  
+**Marker Semantics**: `[x]` means the criterion has been reviewed and satisfied for requirements quality. It does not mean implementation work is complete.
+
+---
+
+## Text Normalization & Tokenization Quality
+
+- [x] CHK001 Are Unicode NFKC normalization rules explicitly specified for multilingual text content? [Completeness, Spec §FR-007]
+- [x] CHK002 Is HTML and Markdown tag stripping behavior defined to prevent accidental concatenation of neighboring words? [Clarity, Spec §FR-007]
+- [x] CHK003 Are anchor text extraction rules for Markdown and HTML links documented unambiguously? [Clarity, Spec §FR-007]
+- [x] CHK004 Is the tokenization behavior (Unicode alphanumeric tokens, punctuation exclusion, lowercase) completely specified? [Completeness, Spec §FR-007]
+
+## Shingles & Consensus Metric Formulation
+
+- [x] CHK005 Is the sliding window shingle size (5-tokens) and the fallback rule for short texts (< 5 tokens) explicitly defined? [Clarity, Spec §FR-008]
+- [x] CHK006 Are the mathematical formulas for Coverage, Support, and F1 Score defined with explicit zero-division handling? [Measurability, Spec §FR-010]
+- [x] CHK007 Is the threshold for a shingle to enter the Consensus set (presence in $\ge 2$ active candidates) unambiguously stated? [Clarity, Spec §FR-009]
+
+## Decision & Tie-Breaking Hierarchy
+
+- [x] CHK008 Is the technical tie threshold ($\le 0.03$) quantified with exact comparison semantics? [Clarity, Spec §FR-011]
+- [x] CHK009 Is the tie-breaker preference for the smaller candidate (fewest shingles) explicitly constrained to candidates within the technical tie pool? [Consistency, Spec §FR-011]
+- [x] CHK010 Is the zero-consensus fallback hierarchy (median of 3, maximum of 2, single candidate) completely specified without ambiguous gaps? [Coverage, Spec §FR-012]
+- [x] CHK011 Is the final mandatory priority order (`newspaper4k` > `readability` > `trafilatura`) consistent across all tie scenarios? [Consistency, Spec §FR-011, §FR-012]
+
+## Candidate State Transitions & Resilience
+
+- [x] CHK012 Are the criteria distinguishing Usable, Degraded, and Unavailable candidates defined unambiguously? [Completeness, Spec §FR-005]
+- [x] CHK013 Does the spec define the exact behavior and fallback when all 3 extractors are Unavailable? [Edge Case, Spec §FR-013]
+- [x] CHK014 Does the spec define what occurs when degraded candidates exist but no usable candidates are present? [Coverage, Spec §FR-006]
+
+## JSON Schema Integrity & Atomic I/O
+
+- [x] CHK015 Are requirements explicit that 100% of pre-existing fields, structures, and article order must be preserved unchanged? [Completeness, Spec §FR-003, §FR-010]
+- [x] CHK016 Is the output filename pattern `<original_name_without_extension>_selected.json` specified for default CLI execution? [Clarity, Spec §FR-016]
+- [x] CHK017 Are atomic write requirements (temporary file + atomic replacement) defined to prevent partial or corrupted files on disk? [Non-Functional, Spec §FR-002, §FR-016]
+- [x] CHK018 Is the behavior for recalculating an already present `selected_extractor` key explicitly specified? [Clarity, Spec §FR-014]
+
+---
+
+## Notes
+
+- Mark items `[x]` only after review confirms the requirement-quality criterion is satisfied.
+- Leave items unchecked when they still require clarification, correction, or reviewer evaluation.
+- `/speckit-implement` reads checklist checkbox state as a gate and must not modify markers.
+- `checklists/requirements.md` has a separate built-in lifecycle maintained by `/speckit-specify` and `/speckit-clarify`.
+- Items are numbered sequentially (CHK001 - CHK018) for easy reference.
@@ -0,0 +1,36 @@
+# Specification Quality Checklist: Deterministic Content Selection
+
+**Purpose**: Validate specification completeness and quality before proceeding to planning  
+**Created**: 2026-08-20  
+**Feature**: [spec.md](../spec.md)
+
+## Content Quality
+
+- [x] No implementation details (languages, frameworks, APIs)
+- [x] Focused on user value and business needs
+- [x] Written for non-technical stakeholders
+- [x] All mandatory sections completed
+
+## Requirement Completeness
+
+- [x] No [NEEDS CLARIFICATION] markers remain
+- [x] Requirements are testable and unambiguous
+- [x] Success criteria are measurable
+- [x] Success criteria are technology-agnostic (no implementation details)
+- [x] All acceptance scenarios are defined
+- [x] Edge cases are identified
+- [x] Scope is clearly bounded
+- [x] Dependencies and assumptions identified
+
+## Feature Readiness
+
+- [x] All functional requirements have clear acceptance criteria
+- [x] User scenarios cover primary flows
+- [x] Feature meets measurable outcomes defined in Success Criteria
+- [x] No implementation details leak into specification
+
+## Notes
+
+- All requirements are derived directly from PRD `docs/prd_deterministic_content_selection.md`.
+- No ambiguity remains; all edge cases and tie-breaking hierarchies are fully specified.
+- Ready for `/speckit-plan`.