Markdown Structure Parser

Project Overview

Markdown Structure Parser is an advanced data extraction system designed to parse and structure technical specifications from markdown documents. The system uses a pipeline approach with AI agents to extract, transform, and load machine specifications into structured formats.

Key Components

Machine Specification Extraction Pipeline

The core of this project is a sophisticated extraction pipeline (machine_spec_extractor_pipeline.py) that:

1. Divides markdown documents into logical sections
2. Filters relevant sections using an AI agent to prioritize sections likely to contain machine specifications
3. Processes each filtered section with the md_parser_v2 agent to extract machine specifications
4. Collects and merges partial specifications from different sections
5. Generates structured CSV output with comprehensive logging

input markdown file → section extraction → AI section filtering → agent processing → specification merging → CSV output

Enhanced Pipeline Features

The current pipeline includes several advanced features:

• Complete Logging System: All terminal output is captured and saved to timestamped log files in the logs/ directory
• Section Filtering Agent: Uses an AI agent to intelligently filter sections most likely to contain machine specifications
• Debug Outputs: Generates intermediate JSON files (*_raw.json, *_merged.json) for debugging and analysis
• Robust Error Handling: Comprehensive error handling with detailed logging and graceful failure recovery
• Rate Limiting: Built-in delays to prevent API rate limiting issues

AI Agent Parser (md_parser_v2)

The intelligent extraction is powered by a specialized agent (md_parser_v2.py) that:

• Uses GPT models to analyze markdown content
• Recognizes machine specifications in various formats
• Handles complex tabular data and unstructured text
• Returns structured JSON outputs with comprehensive error handling

Machine Specification Model

The system uses a comprehensive model that defines 25+ required fields for machine specifications:

• Machine Identification: Machine Name and type
• Technical Parameters: Shot volume, Screw diameter, Injection pressure, Injection rate, Max. injection speed
• Mechanical Specifications: Screw stroke, Screw speed(stepless), Clamping force, Opening stroke
• Physical Dimensions: Space between bars, Max. Daylight, mold thickness (Min. Max)
• Ejection System: Hydraulic ejection stroke, Ejector number, Hydraulic ejection force
• Power Systems: Hydraulic system pressure, Pump motor, Heating capacity
• Control Systems: Number of temp control zones, Dry cycle time
• Physical Properties: Oil tank capacity, Machine dimensions (LxWxH), Machine weight

Project Structure

markdown_struct_parse/
├── data/                # Data files
│   ├── catalogues/      # Original PDF catalogues
│   ├── markdowns/       # Markdown files derived from PDFs
│   ├── structured/      # Structured output (CSV)
│   └── extracted_*.json # Intermediate JSON outputs
├── logs/                # Comprehensive logging system
│   ├── debug/           # Debug files for each section
│   └── extraction_log_*.txt # Complete session logs
├── parsers/             # Parser implementations
│   ├── markdown_specifications_parser_v2.py
│   └── pdf_to_markdown.py
├── pipeline/            # Pipeline components
│   └── machine_spec_extractor_pipeline.py
└── src/                 # Core source code
    ├── agents/          # Agent implementations
    │   └── md_parser_v2.py
    ├── config/          # Configuration
    │   └── prompts/     # Agent prompts
    │       ├── md_parser_v2_agent.md
    │       └── md_section_filter_agent.md
    ├── models/          # Data models
    │   └── machine_spec_model.py
    └── tools/           # Agent tools

How the Pipeline Works

1. Section Extraction: The pipeline takes a markdown file and divides it into relevant sections, prioritizing those likely to contain machine specifications.

2. AI Section Filtering: A specialized agent analyzes all extracted sections and filters them to retain only those most likely to contain machine specifications, significantly reducing processing time.

3. Agent Processing: Each filtered section is sent to the md_parser_v2 agent with its specialized prompt that enables it to:

   • Identify machine names using multiple strategies
   • Extract specifications from tables and text
   • Handle multiple machine types/variants
   • Return structured JSON data with comprehensive error handling

4. Specification Merging: The pipeline collects partial specifications from different sections and merges them by machine name and type, ensuring complete data coverage.

5. CSV Generation: The merged specifications are formatted into a CSV file with all 25+ required fields, with empty values marked as "null".

6. Debug Outputs: The pipeline generates intermediate JSON files for debugging:

   • *_raw.json: All specifications before merging
   • *_merged.json: Specifications after merging
   • Complete session logs in logs/extraction_log_*.txt

Running the Pipeline

python pipeline/machine_spec_extractor_pipeline.py <input_markdown_file> <output_csv_file>

Example:

uv run pipeline/machine_spec_extractor_pipeline.py data/markdowns/example.md data/structured/output.csv

Technical Details

Enhanced Logging System

The pipeline implements a comprehensive logging system (TeeOutput class) that:

• Captures all terminal output (stdout and stderr)
• Saves complete session logs with timestamps
• Creates debug files for each section processed
• Maintains console output while logging to files

Section Filtering Agent

The pipeline now includes an intelligent section filtering step that:

• Analyzes all extracted sections using an AI agent
• Prioritizes sections containing keywords like 'specifications', 'machine', 'dimensions', 'technical', 'parameters'
• Reduces processing time by focusing only on relevant sections
• Uses the MD_SECTION_FILTER_AGENT_PROMPT for consistent filtering

Machine Specification Model

The system defines 25+ required fields including:

• Machine Name and type
• Technical specifications (Shot volume, Screw diameter, etc.)
• Physical dimensions and performance metrics
• Power and control systems
• All specifications are required fields (empty strings marked as "null" when not found)

Error Handling and Debugging

The pipeline includes robust error handling:

• Comprehensive exception handling with detailed logging
• Rate limiting with built-in delays (5-second intervals)
• Multiple JSON parsing strategies for agent responses
• Debug file generation for failed extractions
• Graceful handling of API errors and service limitations

Installation and Setup

Setting up the Environment

1. Create a virtual environment:

   python -m venv .venv

2. Activate the virtual environment:

   • Windows:

     .\.venv\Scripts\Activate.ps1

   • Linux/Mac:

     source .venv/bin/activate

3. Install dependencies using uv:

   # Install uv if not already installed
   curl -LsSf https://astral.sh/uv/install.sh | sh
   # Install dependencies from pyproject.toml
   uv pip install -e .

   Alternatively, using pip:

   # Install from requirements.txt
   pip install -r requirements.txt
   # OR install from pyproject.toml
   pip install -e .

4. Setting the environment prompt (optional): To customize the virtual environment prompt to show "Z-Data":

   # Edit .venv/Scripts/Activate.ps1 and add this line before the prompt setting:
   $Prompt = "Z-Data"

Running the Application

uv run pipeline/machine_spec_extractor_pipeline.py data/markdowns/example.md data/structured/output.csv

Docker Deployment

This project can run in:

1. Development environment (local Docker)
2. Production environment (AWS ECS)

See the deployment scripts in the /scripts directory for details.