Let them search - Week 3

[Aleksandergreg]

User Story 2: Scrape, Index, and Search External Web Content

As an application user,
I want to search for content from selected external websites (identified based on logged queries or pre-selection) in addition to the existing application content,
so that I can find a broader range of relevant information directly within the application.

---

Acceptance Criteria

• Content from at least two distinct, pre-selected or log-informed external websites is successfully scraped and indexed.
• A mechanism (e.g., a Rake task, a separate script) exists to perform web scraping on the selected websites.
• Scraped content (e.g., URL, title, relevant text snippet, source website) is stored in a searchable database (can be separate from the user database, e.g., SQLite, Postgres).
• The application's search functionality queries and returns results from both existing internal content AND the newly indexed external web content.
• The search results potentially indicate the source of the information (internal vs. external website).
• The overall design of the scraping, indexing, and search solution demonstrates considerations for scalability (e.g., modular scraper design, potential for background jobs, efficient indexing), even if the current implementation operates at a small scale.
• The team can explain the scraping/indexing process and the scalability aspects of their chosen solution.

---

Tasks

• (Optional/Input) Analyze logs from User Story 1 to identify popular search topics/keywords not covered internally.
• Select 2+ specific external websites suitable for scraping based on analysis or pre-selection (Important: Check robots.txt and Terms of Service for permissions).
• Choose a Ruby library for web scraping (e.g., Nokogiri for parsing HTML, HTTParty/Faraday for fetching pages).
• Design a database schema/structure for storing scraped web page data (e.g., url, title, content_snippet, source_site, scraped_at).
• Set up the database instance for scraped content (e.g., create SQLite file, configure Postgres).
• Develop/implement a web scraper module/script in Ruby to fetch content from target sites and parse out relevant data (title, text).
• Implement an indexing process to save the parsed data into the database, handling potential duplicates or updates.
• Modify the application's search function (in Sinatra) to query both the original data source and the new scraped content database.
• Combine and potentially rank/sort results from both sources.
• Update the search results view (HTML/ERB template) to display combined results, possibly indicating the source.
• Document the scraping process, chosen libraries, database schema, indexing strategy, and how the solution is designed for potential future scaling.
• Test the end-to-end flow: running the scraper -> checking the database -> performing searches -> verifying results.

---

INVEST Principle

I: Independent, N: Negotiable, V: Valuable, E: Estimable, S: Small, T: Testable

• Independent: Yes, largely independent from other features, though it depends on having a basic search function and ideally uses input from the logging story.
• Negotiable: Yes, which websites, what data to scrape/index, the frequency of scraping, the database choice, and the specific scalability techniques considered are negotiable.
• Valuable: Yes, directly expands the application's content offering based on user interest, fulfilling the core exercise goal.
• Estimable: Yes, effort can be estimated based on the complexity of the target websites and the chosen implementation approach.
• Small: Yes, by limiting to 2+ sites and focusing on the design's scalability rather than operational scale, it remains manageable.
• Testable: Yes, via the Acceptance Criteria: successful scraping, indexing, integrated search results, and explanation of the scalable design.

---

Additional Context / Notes (Optional)

• Addresses the "Scrape and index webpages" and "scalability of your solution" requirements.
• Emphasizes respecting website policies (robots.txt, ToS).
• The database can be different from the main application database (e.g., using SQLite via the sqlite3 gem or Postgres via the pg gem is common in Ruby).
• Scalability considerations might include: running scrapers as background jobs (e.g., using Sidekiq/Redis), designing scrapers to handle website structure changes gracefully, efficient database indexing for search, rate limiting calls to external sites.