Skip to content

configuration.md

Latest commit

 

History

History
161 lines (111 loc) · 3.24 KB

configuration.md

File metadata and controls

161 lines (111 loc) · 3.24 KB
Raw

Configuration Guide

Basic Configuration

from pyset import TokenBoundaryDetector

detector = TokenBoundaryDetector()

Language Support

PySET supports 50+ languages. Set language at initialization:

detector = TokenBoundaryDetector(language='en')   # English (default)
detector = TokenBoundaryDetector(language='de')   # German
detector = TokenBoundaryDetector(language='fr')  # French
detector = TokenBoundaryDetector(language='es')   # Spanish
detector = TokenBoundaryDetector(language='it')  # Italian
detector = TokenBoundaryDetector(language='pt')  # Portuguese
detector = TokenBoundaryDetector(language='nl')  # Dutch
detector = TokenBoundaryDetector(language='ru')  # Russian
detector = TokenBoundaryDetector(language='zh')  # Chinese
detector = TokenBoundaryDetector(language='ja')  # Japanese
detector = TokenBoundaryDetector(language='ko')  # Korean

Abbreviations

Default Abbreviations

PySET includes common abbreviations: Dr, Mr, Mrs, Ms, Jr, Sr, Inc, Ltd, etc.

Add Custom Abbreviations

detector.set_abbreviations({'CEO', 'CFO', 'CTO', 'COO', 'ACME', 'NASA'})

Aggressive Mode

detector = TokenBoundaryDetector(aggressive_abbreviations=True)

This applies stricter rules for abbreviations to avoid false splits.

Sentence Length

Minimum Sentence Length

detector = TokenBoundaryDetector(min_sentence_length=5)

Sentences shorter than 5 words will be merged with the next sentence.

Merge Short Sentences

detector = TokenBoundaryDetector(merge_short_sentences=True)

Automatically merges short sentences.

Rule Configuration

Include Specific Rules

detector = TokenBoundaryDetector(include_rules=[1, 2, 3, 10, 15])

Exclude Specific Rules

detector = TokenBoundaryDetector(exclude_rules=[17, 18])

Custom Rules

from pyset.rules import Rule

class MyRule(Rule):
    priority = 95
    
    def evaluate(self, context):
        # Your logic
        return 1.0 if condition else 0.0

detector = TokenBoundaryDetector(custom_rules=[MyRule()])

Debug Mode

detector = TokenBoundaryDetector(debug=True)

explanations = detector.explain("Your text here.")
# Returns detailed rule-by-rule breakdown

Performance Tuning

Caching

PySET automatically caches Context calls for better performance. No configuration needed.

Early Exit

PySET exits early when confidence is high:

  • Exit if confidence >= 0.75 AND priority >= 85
  • Exit if confidence >= 0.90 for BOUNDARY (1.0)

This provides ~3x speedup on average. Disable if needed:

# No current way to disable - early exit is always on

Recommended Configurations

RAG/Document Chunking

detector = TokenBoundaryDetector(
    min_sentence_length=10,
    merge_short_sentences=True,
    aggressive_abbreviations=True
)

Legal Documents

detector = TokenBoundaryDetector(
    aggressive_abbreviations=True,
    min_sentence_length=5
)

High-Accuracy NLP Pipelines

detector = TokenBoundaryDetector(
    language='en',
    aggressive_abbreviations=True,
    debug=False
)

Debugging/Development

detector = TokenBoundaryDetector(
    debug=True,
    min_sentence_length=1
)