TextNLPClassifierApp/specs/005-convert-json-markdown/research.md

# Research: Convert Article JSON to Markdown

**Feature**: `005-convert-json-markdown` | **Date**: 2026-08-21

## 1. Executive Summary & Goals

This research addresses the design decisions for converting a single news article JSON with `selected_extractor` (`trafilatura`, `newspaper4k`, or `readability`) into a standardized, clean, human-readable Markdown file (`.md`) with deterministic metadata extraction and fallback rules.

---

## 2. Technical Decisions & Research Findings

### Decision 1: HTML-to-Markdown Engine Selection

- **Decision**: Use [`markdownify`](https://github.com/matthewwithanm/python-markdownify) with ATX heading style (`heading_style=ATX`).
- **Rationale**:
  - `markdownify` is a lightweight, battle-tested Python library focused exclusively on converting HTML trees to clean Markdown.
  - It natively converts `<h1>`–`<h6>` to `#`–`######` (ATX style), parses tables, code blocks, lists, quotes, and inline styles (`<b>`, `<i>`, `<a>`, `<img>`).
  - Unlike broader conversion tools like Microsoft MarkItDown or Pandoc, `markdownify` has zero external non-Python dependencies, low overhead, and avoids unnecessary multi-format abstractions.
- **Alternatives Considered**:
  - *Microsoft MarkItDown*: Evaluated in PRD; rejected because it pulls broader dependencies (PDF, DOCX, audio, Azure AI) that exceed the scope of pure HTML-to-Markdown conversion.
  - *Custom BeautifulSoup converter*: Unnecessary wheel reinvention; maintenance burden for complex HTML elements (nested lists, tables, inline formatting).

---

### Decision 2: Direct Markdown Handling for Trafilatura

- **Decision**: When `selected_extractor == "trafilatura"`, directly use `trafilatura.markdown` (or fallback to `trafilatura.text`) without running HTML-to-Markdown conversion.
- **Rationale**:
  - Trafilatura natively emits high-quality Markdown in its extraction output.
  - Plain text (`trafilatura.text`) is already valid Markdown without special markup.
- **Alternatives Considered**:
  - *Converting Trafilatura's raw HTML*: Inefficient and degrades Trafilatura's native document structural tree.

---

### Decision 3: Metadata Normalization & Priority Resolution Pipeline

- **Decision**: Implement a pure Python deterministic metadata resolver supporting:
  - Strict priority tables matching the PRD specification.
  - Scalar normalization: HTML entity decoding (`html.unescape`), whitespace trimming/collapsing, placeholder discarding (`null`, `none`, `n/a`, `unknown`, `[no-author]`, `no-author`).
  - List normalization: Splitting on `;` if string, trimming elements, filtering out URL-like authors (`http://`, `https://`, `www.`), deduplicating case-insensitively while preserving initial case and original order.
  - First-valid-source selection: Pick the first source in priority order that yields a valid, non-empty candidate list without cross-source merging.
- **Rationale**:
  - Guarantees 100% deterministic and reproducible metadata output.
  - Prevents corrupt or placeholder values from leaking into final editorial documents.

---

### Decision 4: Date Parsing Strategy (ISO 8601 & RFC 2822)

- **Decision**: Use Python's standard library `datetime.fromisoformat` and `email.utils.parsedate_to_datetime` / standard datetime parsing routines without heavy external dependencies.
- **Rationale**:
  - All input dates observed from extractors follow ISO 8601 (e.g. `2026-08-20T00:36:33-03:00` or `2026-08-20T03:36:33Z`) or RFC 2822 (e.g. `Thu, 20 Aug 2026 00:36:33 -0300`).
  - `datetime.fromisoformat()` in Python 3.11+ handles full ISO 8601 with timezone offsets and 'Z'.
  - `email.utils.parsedate_to_datetime()` standard library natively handles RFC 2822 dates.
  - If a date cannot be parsed, the candidate is discarded and resolution advances to the next source in priority order.
- **Alternatives Considered**:
  - *dateparser / python-dateutil*: Adds extra heavy dependency; unnecessary given the standardized datetime formats emitted by upstream extractors.

---

### Decision 5: URL Validation & Media Filtering

- **Decision**:
  - Validate all URLs with `urllib.parse.urlparse`: Scheme must be `http` or `https`, and `netloc` (hostname) must be non-empty.
  - In Markdown body: Filter out images with relative URLs, empty URLs, or `data:` URIs.
  - Deduplicate identical image URLs in the body, keeping only the first occurrence.
- **Rationale**:
  - Prevents broken local references or bloated base64 data URIs in downstream pipelines.

---

### Decision 6: Duplicate Title Heading (H1) Stripping

- **Decision**:
  - Check the first top-level ATX heading (`# ...`) in the converted body.
  - If its text matches the resolved article title (after HTML entity decoding, whitespace collapsing, and case-insensitive comparison), remove that H1 line and preceding/following whitespace.
  - Preserve all subsequent H1/H2/H3 headings in the body.
- **Rationale**:
  - Many news articles embed the title in `<h1>` inside the article HTML. Since our Markdown schema places `# <Resolved Title>` at the top of the document, stripping the redundant body H1 avoids awkward repeated headings.

---

### Decision 7: Atomic File Writing & Error Resilience

- **Decision**:
  - Write Markdown output to a temporary file in the same directory (`.<output_filename>.tmp`) and atomically replace the destination using `os.replace` (or `pathlib.Path.replace`).
  - If any error or validation exception occurs during processing, clean up the temporary file and exit with code `1`, leaving any pre-existing target file untouched.
- **Rationale**:
  - Guarantees transactional file operations in unattended automated batch pipelines.

---

## 3. Technology Stack & Dependencies

- **Runtime**: Python `>=3.10` (tested on 3.10, 3.11, 3.12)
- **New Dependency**: `markdownify>=0.13.0`
- **Standard Library Modules**: `argparse`, `json`, `os`, `sys`, `pathlib`, `re`, `html`, `urllib.parse`, `datetime`, `email.utils`
- **Testing & Quality**: `pytest`, `ruff`, `mypy`