Skip to content

Add Django Admin action to run blog scraper - #71

Open
Gurudevkini wants to merge 1 commit into
UniCourt:mainfrom
Gurudevkini:workshop-complete
Open

Gurudevkini wants to merge 1 commit into
UniCourt:mainfrom
Gurudevkini:workshop-complete

Conversation

@Gurudevkini

Copy link
Copy Markdown

Summary

This pull request adds a Django Admin action to the existing Blog admin, allowing an administrator to trigger the workshop's existing scraper directly from the Admin UI.

The action calls start_extraction() from myworld/web_scrapper.py and displays a success or error message depending on the result.

Changes

  • Added run_scraper as a Django Admin action in myworld/members/admin.py.
  • Connected the action to the existing BlogAdmin using actions = [run_scraper].
  • The action calls the existing start_extraction() function from web_scrapper.py.
  • Added success feedback when the scraper completes successfully.
  • Added error feedback when the scraper raises an exception.
  • No changes were made to the models, migrations, Docker configuration, or scraper implementation.

Existing Workshop Functionality

The following functionality was already provided by the workshop:

  • Django project with the members app.
  • Students and Blog Django models.
  • Django Admin registration for the models.
  • PostgreSQL database integration.
  • Docker Compose setup with Django and PostgreSQL services.
  • Existing database migrations for the Blog and Students models.
  • Python Insider scraper using Requests and BeautifulSoup.
  • Existing scraper persistence using psycopg2 and SQL.

Data Flow

The new Admin action provides the following flow:

Django Admin

run_scraper()

start_extraction()

web_scrapper.py

Python Insider

BeautifulSoup + Requests

PostgreSQL

The existing scraper clears the members_blog table and inserts the scraped blog records using psycopg2.

Limitations

  • The Admin action does not use the selected Blog queryset; it triggers a full scraper refresh.
  • The scraper currently uses psycopg2 and raw SQL rather than Django ORM.
  • The scraper establishes its PostgreSQL connection when web_scrapper is imported.
  • The Admin action runs the scraper synchronously as part of the Admin request.
  • The success message does not report the number of records inserted.
  • There are currently no tests specifically covering the new Admin action.

Verification

  • The current branch contains the Admin action in myworld/members/admin.py.
  • The action is registered on BlogAdmin.
  • The existing scraper remains unchanged.
  • Django Admin and scraper behavior were manually verified during development.

Scope

This PR adds a Django Admin interface for triggering the existing workshop scraper. It does not modify the existing models, migrations, Docker setup, or scraper implementation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant