getting-started.md

Getting Started with PySET

A quick guide to get you up and running with PySET.

Installation

pip install pyset

Basic Usage

from pyset import TokenBoundaryDetector

detector = TokenBoundaryDetector()
sentences = detector.split("Hello world. How are you? I'm doing great!")
print(sentences)
# ['Hello world.', 'How are you?', "I'm doing great!"]

Return Options

Get Character Spans

spans = detector.split(text, return_spans=True)
# Returns: [(start, end), ...]

Get Metadata

info = detector.split(text, return_metadata=True)
# Returns: [{'text': str, 'start': int, 'end': int, 'confidence': float}, ...]

Common Patterns

Multiple Paragraphs

text = """
This is the first paragraph.

This is the second paragraph. It has multiple sentences.
The final sentence here.
"""

sentences = detector.split(text)

Handle Abbreviations

# Add custom abbreviations
detector.set_abbreviations({'Dr', 'Prof', 'Inc', 'Ltd'})

# Or use aggressive mode
detector = TokenBoundaryDetector(aggressive_abbreviations=True)

Multi-language Support

detector = TokenBoundaryDetector(language='en')  # English (default)
detector = TokenBoundaryDetector(language='de')  # German
detector = TokenBoundaryDetector(language='fr')  # French
# 50+ languages supported

Debug Mode

detector = TokenBoundaryDetector(debug=True)
explanations = detector.explain("Mr. Smith went to the store.")
# Shows which rules applied and why

Integration with NLP Pipelines

# Document chunking for RAG systems
def chunk_document(text, max_words=500):
    sentences = detector.split(text)
    chunks = []
    current_chunk = []
    current_count = 0
    
    for sentence in sentences:
        word_count = len(sentence.split())
        if current_count + word_count > max_words:
            if current_chunk:
                chunks.append(' '.join(current_chunk))
            current_chunk = [sentence]
            current_count = word_count
        else:
            current_chunk.append(sentence)
            current_count += word_count
    
    if current_chunk:
        chunks.append(' '.join(current_chunk))
    
    return chunks