eBay Product Search Scraper Pipeline

A modular, production-ready Python scraping pipeline designed to extract product search listings and pricing from eBay, bypass connection blocking using session cookies, handle dual-DOM layout systems, and export structured datasets.

---

🛠️ Tech Stack

• Python 3.x
• requests: Network requests, sessions, and headers emulation.
• BeautifulSoup4: HTML syntax tree parsing and DOM navigation.
• re: Regular expressions for text cleaning.
• pandas: Structured data loading, cleaning, and CSV serialization.

---

📐 Architecture & Pipeline Steps

graph TD
    A[Start Pipeline] --> B[Establish requests Session]
    B --> C[Fetch eBay homepage to acquire cookies]
    C --> D[Fetch Search page using Session with Referer]
    D --> E[Locate containers using .s-item & .s-card selectors]
    E --> F[Extract URL & Title, decomposing ad banners]
    F --> G[Clean & convert Price to Float format]
    G --> H[Clean Condition dividers and Shipping text]
    H --> I[Convert to pandas DataFrame]
    I --> J[Export to ebay_iphone_pricing.csv]
    J --> K[End Pipeline]

Loading

1. Bot Bypass Strategy: Utilizes requests.Session() to first fetch the eBay homepage and set initial session cookies, bypassing the 403 Forbidden wall.
2. Dual-Layout Parsing: Handles both traditional list items (.s-item) and modern card grid items (.s-card) container structures automatically depending on user country or device context.
3. DOM De-noising: Decomposes visual junk like screen-reader spans ("Opens in a new window") and search labels ("New Listing") directly from titles.
4. Data Standardization:
   • Price: Cleans out currency symbols, splits price ranges, and formats them into floats.
   • Shipping: Cleans up delivery pricing and maps free shipping strings directly to "Free shipping".
5. Data Export: Generates a clean ebay_iphone_pricing.csv file without row indexes.

---

🚀 How to Run

Clone the project and install requirements:

pip install requests beautifulsoup4 pandas
python scrape_ebay.py

---

📊 Dataset Preview Sample

Title	Price	Condition	Shipping Cost	Product URL
Apple iPhone 13 A2482 (Unlocked) 512GB Midnight (Excellent)	339.99	Excellent - Refurbished - Apple iPhone 13 - 512 GB - Unlocked	+$135.62	Link
Apple iPhone 13 128GB Factory Unlocked Very Good	259.00	12-MONTH WARRANTY • FREE FAST SHIPPING • 30 DAY RETURNS	+$116.99	Link