TextNLPClassifierApp/specs/005-convert-json-markdown/tasks.md

# Tasks: Convert Article JSON to Markdown

**Branch**: `005-convert-json-markdown` | **Feature**: Convert Article JSON to Markdown  
**Spec**: [spec.md](spec.md) | **Plan**: [plan.md](plan.md) | **Data Model**: [data-model.md](data-model.md)

---

## Phase 1: Setup (Shared Infrastructure)

**Purpose**: Project initialization, dependency management, and test fixture directory setup

- [X] T001 Add `markdownify>=0.13.0` to `requirements.txt`
- [X] T002 [P] Create fixtures directory structure in `tests/fixtures/markdown_conversion/`

---

## Phase 2: Foundational (Blocking Prerequisites)

**Purpose**: Golden fixtures, negative test fixtures, and baseline test infrastructure that MUST be complete before user stories begin

- [X] T003 [P] Create Golden Test Fixtures for Trafilatura in `tests/fixtures/markdown_conversion/valid_trafilatura.json` and `tests/fixtures/markdown_conversion/valid_trafilatura.md`
- [X] T004 [P] Create Golden Test Fixtures for Newspaper4k in `tests/fixtures/markdown_conversion/valid_newspaper4k.json` and `tests/fixtures/markdown_conversion/valid_newspaper4k.md`
- [X] T005 [P] Create Golden Test Fixtures for Readability in `tests/fixtures/markdown_conversion/valid_readability.json` and `tests/fixtures/markdown_conversion/valid_readability.md`
- [X] T006 [P] Create Negative Test Fixtures (`batch_articles_invalid.json`, `corrupt_json_invalid.json`, `missing_extractor_invalid.json`, `missing_body_invalid.json`, `missing_title_invalid.json`, `invalid_url_invalid.json`) in `tests/fixtures/markdown_conversion/`

**Checkpoint**: Foundational test fixtures ready. User story implementation can begin in strict TDD order.

---

## Phase 3: User Story 1 - Single Article JSON to Clean Markdown Conversion (Priority: P1) 🎯 MVP

**Goal**: Convert a single article JSON into a clean Markdown document using exclusively the selected extractor (`trafilatura`, `newspaper4k`, `readability`), supporting direct Markdown reuse and HTML-to-Markdown conversion with intra-extractor fallback.

**Independent Test**: Provide single-article JSON fixtures for each extractor and verify that the resulting Markdown body text matches the selected extractor's content without cross-extractor leakage.

### Tests for User Story 1 (TDD) ⚠️
> **NOTE: Write these tests FIRST, ensure they FAIL before implementing**

- [X] T007 [P] [US1] Write unit and integration tests for extractor body resolution, strict extractor isolation, and HTML-to-Markdown conversion in `tests/test_convert_article_to_markdown.py`

### Implementation for User Story 1

- [X] T008 [US1] Implement body resolution and strict extractor isolation logic (prohibiting cross-extractor fallback) in `scripts/convert_article_to_markdown.py`
- [X] T009 [US1] Implement HTML-to-Markdown conversion using `markdownify` (ATX headings) and intra-extractor fallback (`markdown`/`html` → `text`) in `scripts/convert_article_to_markdown.py`

**Checkpoint**: At this point, User Story 1 is fully functional and testable independently (MVP ready).

---

## Phase 4: User Story 2 - Deterministic Metadata Resolution and Fallback (Priority: P2)

**Goal**: Resolve mandatory and optional metadata across all extractor candidates and input metadata according to strict priority hierarchies, normalizing strings, lists, dates, and URLs, stripping duplicate H1 headers, and formatting the final Markdown structure.

**Independent Test**: Pass articles with missing metadata in the selected extractor but present in secondary sources; verify that resolved metadata strictly follows priority chains, discards placeholders, sanitizes dates/URLs/images, and formats output accurately.

### Tests for User Story 2 (TDD) ⚠️
> **NOTE: Write these tests FIRST, ensure they FAIL before implementing**

- [X] T010 [P] [US2] Write unit tests for metadata priority chains, scalar normalization, list deduplication, date parsing (ISO 8601 / RFC 2822), URL validation, duplicate H1 heading removal, and body image sanitation in `tests/test_convert_article_to_markdown.py`

### Implementation for User Story 2

- [X] T011 [US2] Implement scalar and list normalizers (HTML unescape, whitespace collapsing, placeholder filtering, author URL filtering, case-insensitive deduplication) in `scripts/convert_article_to_markdown.py`
- [X] T012 [US2] Implement ISO 8601 and RFC 2822 date parser (with timezone preservation and `YYYY-MM-DD` date-only output) and URL validator in `scripts/convert_article_to_markdown.py`
- [X] T013 [US2] Implement deterministic metadata priority resolution matrix in `scripts/convert_article_to_markdown.py`
- [X] T014 [US2] Implement duplicate H1 title heading stripper and body image link sanitizer (removing relative/`data:`/empty URLs and deduplicating repeats) in `scripts/convert_article_to_markdown.py`
- [X] T015 [US2] Implement final Markdown document layout assembler (Header, Subtitle, Metadata key-values, Top Image, Separator `---`, Body) with LF line endings in `scripts/convert_article_to_markdown.py`

**Checkpoint**: At this point, User Stories 1 and 2 work together and pass all unit/integration tests.

---

## Phase 5: User Story 3 - CLI Usability, Validation, and Atomic Output (Priority: P3)

**Goal**: Provide a production-ready CLI interface with `-i`/`--input` and `-o`/`--output` flags, standard exit codes (0, 1, 2), error logging to `stderr`, and atomic file replacement with transactional error cleanup.

**Independent Test**: Execute the CLI against valid JSON, invalid JSON, batch `articles` array JSON, and missing files; verify exit codes, stderr diagnostics, and target file integrity.

### Tests for User Story 3 (TDD) ⚠️
> **NOTE: Write these tests FIRST, ensure they FAIL before implementing**

- [X] T016 [P] [US3] Write CLI integration tests covering `-i`/`-o` flags, exit codes (`0`, `1`, `2`), stderr logging, rejection of batch `articles` JSON, and atomic replacement rollback on failure in `tests/test_convert_article_to_markdown.py`

### Implementation for User Story 3

- [X] T017 [US3] Implement CLI argument parsing, input JSON validation (object root, rejection of `articles` key, `selected_extractor` check), and diagnostic logging to `stderr` in `scripts/convert_article_to_markdown.py`
- [X] T018 [US3] Implement transactional atomic file writing (`.<output>.tmp` + `os.replace`) with complete cleanup on failure in `scripts/convert_article_to_markdown.py`

**Checkpoint**: All user stories are fully implemented, resilient, and verified.

---

## Phase 6: Polish & Cross-Cutting Concerns

**Purpose**: Quality gate verification, Golden Fixture byte-for-byte validation, and documentation

- [X] T019 [P] Run full test suite (`pytest tests/test_convert_article_to_markdown.py -v`) and verify 100% exact Golden Fixtures matching
- [X] T020 [P] Run code quality and type checks (`ruff check scripts/ src/ tests/` and `mypy scripts/convert_article_to_markdown.py`)
- [X] T021 Update `README.md` with conversion CLI documentation, pipeline usage examples, and argument references
- [X] T022 Run quickstart validation scenarios from `specs/005-convert-json-markdown/quickstart.md`

---

## Dependencies & Execution Order

### Phase Dependencies

- **Setup (Phase 1)**: No dependencies — start immediately.
- **Foundational (Phase 2)**: Depends on Setup (Phase 1) — BLOCKS all user stories.
- **User Story 1 (Phase 3)**: Depends on Foundational (Phase 2) — Core MVP.
- **User Story 2 (Phase 4)**: Depends on User Story 1 (Phase 3).
- **User Story 3 (Phase 5)**: Depends on User Story 2 (Phase 4).
- **Polish (Phase 6)**: Depends on all user stories (Phases 3–5) being complete.

---

## Parallel Opportunities

- **Phase 1**: `T002` [P] can run in parallel with `T001`.
- **Phase 2**: `T003` [P], `T004` [P], `T005` [P], and `T006` [P] can all be created in parallel.
- **Phase 3**: `T007` [P] (tests) can be created in parallel with fixture setup.
- **Phase 4**: `T010` [P] (tests) can be written before implementation.
- **Phase 5**: `T016` [P] (CLI tests) can be written before CLI wiring.
- **Phase 6**: `T019` [P] (pytest) and `T020` [P] (ruff/mypy) can run in parallel.

---

## Implementation Strategy

### MVP First (User Story 1 Only)
1. Complete Phase 1 (Setup) and Phase 2 (Foundational Fixtures).
2. Complete Phase 3 (User Story 1: TDD tests `T007` → Implementation `T008`, `T009`).
3. Validate independent execution of User Story 1.

### Incremental Delivery
1. Foundation Ready → MVP (US1: Body Conversion).
2. Deliver US2 (Deterministic Metadata & Sanitization).
3. Deliver US3 (CLI, Error Handling & Atomic Output).
4. Run Polish & Quality Gates (US1 + US2 + US3 verified with 100% tests passing).