Skip to content

TechHub-Extensions/ScoutBot

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

284 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

ScoutBot

An open-source Python bot that automatically finds scholarships, fellowships, and internships for Nigerian students. It updates a shared Google Spreadsheet, sends a weekly email digest, and delivers opportunities directly to WhatsApp campus groups and a Telegram channel.

GitHub Issues GitHub Stars Python 3.11 License: MIT ScoutBot Scrape Weekly Digest Pytest


📬 Three Ways to Receive ScoutBot

1 — Weekly Email Digest (free, no login)

ScoutBot emails a curated digest of the latest student opportunities every Sunday at 10AM Lagos time.

→ Fill the ScoutBot Subscription Form

No app, no login, no fee. Fill the form once and you're on the list.


2 — WhatsApp Campus Delivery V2 (filtered by level)

🆕 V2 released June 2026 — graduate-level filtering added, Nigerian timezone accuracy fixed, and pipeline gap closed (issue #62).

Campus leads can register their WhatsApp group to receive opportunities automatically, filtered by academic level:

Option What you receive
Both All opportunities — Undergrad + Graduate/PhD
Undergraduate & Internships only Entry-level, NYSC-eligible, and internship posts
Graduate, Masters & PhD only Postgraduate scholarships and fellowships

How to register your campus group:

  1. Open the Campus Lead Portal →
  2. Enter your campus / university name
  3. Paste your WhatsApp group invite link (chat.whatsapp.com/...)
  4. Select which type of opportunities your group wants to receive
  5. ScoutBot joins your group automatically and sends a welcome message
  6. Make +234 816 449 9922 (ScoutBot) a Group Admin — see instructions below

⚙️ Making ScoutBot a Group Admin

After ScoutBot joins your group, please make it a group admin. This ensures ScoutBot can always post — some groups have admin-only messaging enabled, and without admin status ScoutBot's broadcasts may be blocked.

Steps (takes less than 30 seconds):

  1. Open your WhatsApp group
  2. Tap the group name at the top to open Group Info
  3. Scroll down to Members and find +234 816 449 9922 (ScoutBot)
  4. Tap ScoutBot's number → tap Make Group Admin
  5. Done ✅ — ScoutBot can now always post to the group

ScoutBot will only ever post opportunities — it will never read your members' messages or interfere with group conversations.


3 — Telegram Channel

ScoutBot also publishes opportunities to a Telegram channel. No registration required — just join and get notified.

→ Join the ScoutBot Telegram Channel


📋 View the live opportunity spreadsheet →


What ScoutBot Does

  • 🔍 Scrapes 21+ direct org pages daily — checks PTDF, NDDC, NNPC, MTN Foundation, Tony Elumelu Foundation, Commonwealth Scholarships, Chevening, Fulbright, World Bank, AfDB, AU, UNDP, UNICEF, British Council, and more
  • 🔗 Direct org application links only — every link goes to the actual organisation's apply page, never a news aggregator or redirect URL
  • 📊 Two separate tabs: Nigeria 🇳🇬 and International 🌍 — never mixed
  • 📱 WhatsApp campus delivery — scraper automatically writes new items to distribution-bridge/ after every run; broadcast.py fans them out to all registered campus groups
  • 📣 Telegram channel — real-time posts as new opportunities are discovered
  • 🧹 Auto-cleans daily — entries removed when closed, past deadline, or older than 23 days
  • 📧 One email per week — Sunday digest with only opportunities added in the last 7 days, sent to 500+ subscribers
  • 🚫 Students only — scholarships, fellowships, internships only
  • ☁️ Runs entirely on GitHub Actions — no server dependency, works 24/7 independently

How It Works

Every day at 07:00 WAT (GitHub Actions — scoutbot.yml):

  1. scrapy crawl opportunities
        │
        ├─ DedupePipeline (100)
        │    └─ drops links already seen this run
        │
        ├─ SheetsPipeline (200)
        │    └─ writes new rows to Nigeria / International tab
        │       skips links already in either tab
        │
        └─ WhatsAppQueuePipeline (300)          ← added in fix #62
             └─ writes distribution-bridge/opportunities.json
                writes distribution-bridge/whatsapp_queue.db
                      (pending_broadcasts table)

  2. python run.py --cleanup
        └─ removes entries older than 23 days or past deadline

  3. broadcast.py --source json                 ← triggered automatically
        └─ reads opportunities.json
        └─ sends each new item to every registered campus WhatsApp group
        └─ skips if SESSION_API_URL is not set or whatsapp.js is not running

Every Sunday 10:00 WAT (GitHub Actions — digest.yml):

  4. python run.py --notify
        └─ sends weekly email digest (last 7 days) to all subscribers

1st of every month 07:30 WAT (GitHub Actions — admin-report.yml):

  5. python admin_report.py
        └─ monthly stats report to project lead

WhatsApp Broadcast Setup

The broadcast pipeline requires the whatsapp.js session manager to be running. Once it is, broadcast.py fires automatically after every scrape with no manual steps.

1. Install Node dependencies

cd distribution-bridge
npm install

2. Start the WhatsApp session manager

node whatsapp.js

On first run, a QR code appears in the terminal. Scan it with the ScoutBot WhatsApp number (+234 816 449 9922). The session is saved locally — you only scan once.

3. Set SESSION_API_URL in your .env

SESSION_API_URL=http://localhost:3001

This is the URL broadcast.py uses to reach whatsapp.js. The default port is 3001.

4. Run the full pipeline

python run.py

The order is: scrape → cleanup → broadcast → email digest. WhatsApp messages go out automatically after every scrape. If SESSION_API_URL is not set or whatsapp.js is not running, the broadcast step is skipped cleanly — nothing else breaks.

5. Make ScoutBot a Group Admin

After ScoutBot joins any group, make it a group admin so it can always post:

  1. Open the WhatsApp group
  2. Tap the group name → Group Info
  3. Scroll to Members → find +234 816 449 9922 (ScoutBot)
  4. Tap ScoutBot → Make Group Admin

This step is critical for groups with admin-only messaging enabled.

Broadcast manually

cd distribution-bridge
python broadcast.py --source json          # from opportunities.json (default)
python broadcast.py --source sheets        # pull directly from Google Sheets
python broadcast.py --dry-run              # print messages without sending
python broadcast.py --preview              # preview first message and exit
python broadcast.py --limit 3             # broadcast only last 3 opportunities

Why ScoutBot Exists

Opportunities for Nigerian students are scattered across dozens of websites with no single reliable source. Most are announced on corporate press offices, government portals, or international org pages that students rarely check. ScoutBot checks all of them automatically, every day, and pushes the results to wherever students already are — their WhatsApp group, their Telegram feed, or their inbox once a week.


Project Structure

ScoutBot/
├── scoutbot/
│   ├── spiders/
│   │   └── opportunities_spider.py  ← Scrapes 21+ org pages + scholars4dev RSS
│   ├── pipelines.py                 ← DedupePipeline → SheetsPipeline → WhatsAppQueuePipeline
│   ├── items.py                     ← Scrapy item definition
│   └── settings.py                  ← Scrapy settings + pipeline order
├── distribution-bridge/             ← WhatsApp delivery system
│   ├── whatsapp.js                  ← whatsapp-web.js session manager (node)
│   ├── broadcast.py                 ← Reads opportunities.json → sends to campus groups
│   ├── import_data.py               ← One-off helper: load JSON into whatsapp_queue.db
│   ├── opportunities.json           ← Written by WhatsAppQueuePipeline after every scrape
│   ├── whatsapp_queue.db            ← SQLite queue (pending_broadcasts table)
│   └── requirements.txt             ← Python deps for broadcast.py
├── notify.py                        ← Weekly email digest sender
├── cleanup.py                       ← Removes expired sheet entries (23-day cap)
├── admin_report.py                  ← Monthly stats email to project lead
├── welcome.py                       ← Welcome email for new subscribers
├── run.py                           ← CLI entry point
├── requirements.txt
├── .env.example                     ← Copy to .env and fill in credentials
├── docs/
│   └── VOLUNTEER_ROLES.md           ← Step-by-step guide for all volunteer roles
├── .github/
│   └── workflows/
│       ├── scoutbot.yml             ← Daily 07:00 WAT scrape
│       ├── digest.yml               ← Sunday 10:00 WAT weekly email
│       ├── admin-report.yml         ← Monthly 07:30 WAT stats report
│       ├── welcome.yml              ← Monthly welcome email to new subscribers
│       └── pytest.yml               ← CI tests on every push/PR
├── CHANGELOG.md
├── CONTRIBUTING.md
└── ENGINEERING.md

Opportunity Sources

ScoutBot checks 21 organisation pages directly every day — no news aggregators, no redirects:

Nigeria International
PTDF (ptdf.gov.ng) Commonwealth Scholarship
NDDC (nddc.gov.ng) Chevening Scholarship
NNPC Group Fulbright Program
Shell/SNEPCo Mastercard Foundation
MTN Foundation World Bank
Tony Elumelu Foundation African Development Bank
Dangote Foundation African Union
Access Bank UNDP Nigeria
YouthHub Africa UNICEF
NDIC (SIWES) UN Fellowship, British Council NG

Plus scholars4dev.com RSS as a supplementary feed when it has qualifying items.


Run Locally

git clone https://github.com/TechHub-Extensions/ScoutBot.git
cd ScoutBot
pip install -r requirements.txt
cp .env.example .env
# Fill in .env with your credentials (see ENGINEERING.md)

python run.py              # Full pipeline: scrape → cleanup → broadcast → email
python run.py --scrape     # Scrape only (no email)
python run.py --cleanup    # Remove expired entries only
python run.py --notify     # Send digest email only
python run.py --dry-run    # Build email preview without sending
python run.py --schedule   # Run on schedule: 07:00 + 19:00 WAT daily

Required .env variables

SENDER_EMAIL=your_gmail@gmail.com
GMAIL_APP_PASSWORD=xxxx xxxx xxxx xxxx
SPREADSHEET_ID=your_google_sheet_id
GOOGLE_SERVICE_ACCOUNT_JSON=service_account.json
RECIPIENT_EMAILS=email1@gmail.com,email2@gmail.com

# WhatsApp broadcast (optional — skipped if not set)
SESSION_API_URL=http://localhost:3001

GitHub Actions Setup (runs independently)

Add these secrets under Settings → Secrets → Actions:

Secret Description
SENDER_EMAIL Gmail address to send from
GMAIL_APP_PASSWORD Gmail App Password (not your main password)
SPREADSHEET_ID ID of the main Google Sheet
FORM_SHEET_ID ID of the subscriber form response sheet
RECIPIENT_EMAILS Comma-separated fallback recipients
GOOGLE_SERVICE_ACCOUNT_JSON_B64 Base64-encoded service account JSON
SESSION_API_URL URL of the whatsapp.js session manager (WhatsApp broadcast only)

Encode your service account: base64 -i service_account.json | tr -d '\n'


Accomplishments We're Proud Of

  • 500+ email subscribers acquired organically through student WhatsApp groups and word-of-mouth — zero paid promotion
  • WhatsApp campus delivery system — full distribution bridge that joins WhatsApp groups, filters opportunities by academic level (undergrad / grad / both), and broadcasts automatically. Pipeline gap fixed June 2026 (issue #62)
  • Subscriber web portal with real-time registration, QR code generation, and live ScoutBot status indicator
  • Telegram integration — built by @tsouk88, extending delivery to a third channel with zero extra infrastructure
  • Zero cost infrastructure — entire stack runs free: GitHub Actions, Gmail SMTP, Google Sheets API
  • All links are direct org URLs — no news.google.com, no redirects

🤝 Volunteer — Help Us Grow ScoutBot

ScoutBot is maintained by a small founding team and open-source contributors. Several roles are open to volunteers — no application required.

→ Full volunteer guide: docs/VOLUNTEER_ROLES.md

Role Time/week Skills needed
Source Hunter 1–3 hrs Browser, no coding
Community Ambassador 1–2 hrs Writing, social media
Issue Triager 30 min Basic GitHub
Data Curator 1–2 hrs Google Sheets
Documentation Writer 2–4 hrs Markdown
Email Designer 2–5 hrs HTML + CSS
PR Reviewer 1–2 hrs Python
QA Tester 2–4 hrs Python, CLI

To apply: open a GitHub issue titled volunteer: interested in [Role Name].


💛 Support ScoutBot

ScoutBot is open source and free for every subscriber. We are Nigerian students building this with zero budget.

→ Support on Paystack (₦1,000) → Support on Ko-fi (international)

Organisations can sponsor a featured placement in the Sunday digest for ₦5,000/placement — reaches 500+ students directly. Email kamsirichard1960@gmail.com to enquire.

📄 ScoutBot Fundraising Brief (Google Doc)


The Founding Team

Name Email Role
Kamsi Richard Ivanna kamsirichard1960@gmail.com Founder & Project Lead
Ibukun Ojo adeojoibukun28@gmail.com Core Team
Success (Olamide) successolamide46@gmail.com Core Team — WhatsApp delivery

Contributing

Quick ways to help:

  • Star this repo (takes 2 seconds)
  • 🐛 Open an Issue — report a broken source, a bug, or a feature idea
  • 🔀 Fork and submit a PR — add sources, fix bugs, improve email design
  • 📣 Share with Nigerian student WhatsApp groups, Discord servers, Twitter/X
  • 🤝 Volunteer — see the volunteer guide

Ready to code? Start with issues labelled good first issue.

See CONTRIBUTING.md for the full guide.


Contributors

Every merged contribution is permanently credited in CONTRIBUTORS.md.


kamsirichard

Founder — architecture,
scraping, delivery,
182 commits

olamidefasogbon

30 PRs — WhatsApp delivery
engine, Campus Lead Portal,
link validation

tsouk88

Telegram channel,
new sources, auto-label

Arnish-val

CI badges,
pytest workflow

Want to see your face here? Open a PR or pick an issue.


License

MIT — see LICENSE.