Channel Scraper

Automatically monitor a Soroush Plus (splus.ir) channel and send notifications to Telegram when a new message about regulations (آیین‌نامه) is published.

── Built with pure PHP, specifically designed for shared hosting and cron jobs ──

---

1) What does this project do?

1. Every few minutes (using a cron job), it fetches channel messages.
2. It compares new messages with your configured keywords (for example: "regulation") and ignores messages containing excluded words (for example: "officer").
3. Every matching message is sent to all users who have started the bot.

Features

• Automatic user subscription with /start and unsubscribe with /stop
• Keyword filtering (include / exclude) through configuration file
• Persian text normalization: "آیین نامه", "آیین‌نامه", and "آئین‌نامه" are treated as the same
• Direct link to each message in notifications
• Duplicate prevention (memory of already processed messages)
• Complete logging system with automatic rotation
• Execution lock (prevents multiple cron executions from running simultaneously)
• Can run both from command line and through a URL with a security key

---

2) How it works (Important)

The page:

https://splus.ir/ershadiiha

is a JavaScript application (React).

The message content does not exist in the initial HTML (the body only contains):

<div id="root"></div>

Therefore, parsing HTML will never work.

This scraper does exactly what the website itself does: it sends a POST request to the JSON channel archive API:

POST https://splus.ir/srvcs-app/v1/json/<TOKEN-A>/<TOKEN-B>/channel/archive

Content-Type: application/json
Origin:  https://splus.ir
Referer: https://splus.ir/ershadiiha

{
  "channel_id":"13013718"
}

The response contains a messages array where index 0 is the newest message.

The scraper uses these fields from each message:

• id
• text
• timestampISO
• message_link

---

3) File Structure

channel-scraper/
├── cron.php              # Entry point — this file is executed by cron
├── config.php.example    # Configuration template — copy as config.php
├── README.md
├── .gitignore
├── .htaccess             # Blocks web access to config.php
├── src/
│   ├── Scraper.php       # Fetch messages from JSON API
│   ├── Telegram.php      # Telegram API communication
│   ├── Logger.php        # Logging system
│   └── Storage.php       # JSON file storage handling
├── storage/              # users.json and state.json are created here
└── logs/                 # scraper.log is stored here

---

4) Requirements

• PHP version 7.2 or higher (PHP 8.x recommended)

• Extensions:

  • curl
  • json
  • mbstring

• Outgoing access from hosting server to:

  • splus.ir
  • api.telegram.org

• Ability to create cron jobs in hosting panel (cPanel / DirectAdmin / etc.)

If your hosting cannot access api.telegram.org, change telegram.base_url in config.php to a valid proxy/worker URL.

---

5) Installation Guide

Step 1 — Create Telegram Bot

1. Open Telegram and message @BotFather.
2. Send /newbot.
3. Enter the bot name and username.
4. Copy the token provided by BotFather.

Example:

123456:ABC-DEF...

---

Step 2 — Upload Files

Upload all files to a directory such as:

public_html/channel-scraper/

Set permissions:

chmod 755 channel-scraper
chmod 775 channel-scraper/storage
chmod 775 channel-scraper/logs

---

Step 3 — Create config.php

Copy:

cp config.php.example config.php

At minimum, configure these values:

'telegram' => [
    'bot_token' => 'YOUR-BOT-TOKEN',
],

'channel' => [
    'url' => 'https://splus.ir/ershadiiha',
],

'api' => [
    'url'        => 'https://splus.ir/srvcs-app/v1/json/.../channel/archive',
    'channel_id' => '13013718',
],

---

Step 4 — Manual Test

Run:

cd ~/public_html/channel-scraper
php cron.php

Expected output on the first execution:

Done. First run baseline saved — no notifications sent.

The first execution intentionally does not send messages. This prevents users from receiving dozens of old messages. Starting from the second execution, only new messages will be sent.

Now open your Telegram bot and send:

/start

You should receive a welcome message.

---

Step 5 — Configure Cron Job

Go to the Cron Jobs section in your hosting panel.

Method A — Direct PHP execution (Recommended)

*/5 * * * * /usr/bin/php /home/USERNAME/public_html/channel-scraper/cron.php >/dev/null 2>&1

Method B — Execute through URL

If your hosting only supports URL-based execution:

Add this to config.php:

'cron_key' => 'a-long-random-secret-string',

Then:

*/5 * * * * wget -q -O /dev/null "https://your-domain.com/channel-scraper/cron.php?key=YOUR_KEY"

Recommended interval:

*/5 * * * *

If your hosting has limitations:

*/15 * * * *

or

*/30 * * * *

is also acceptable.

Find the correct PHP path using which php. Some servers use:

/usr/local/bin/php

or:

/opt/alt/php82/usr/bin/php

---

6) Using Another Channel

To find api.url and channel_id:

1. Open the channel page in Chrome.
2. Press F12.
3. Open the Network tab.
4. Filter by Fetch/XHR.
5. Refresh the page (F5).
6. Find the request containing:

.../channel/archive

7. Open it and check:

Headers tab

Copy:

Request URL

into:

api.url

Payload tab

Copy:

channel_id

into:

api.channel_id

Also update:

channel.url
api.origin
api.referer

according to the new channel.

---

7) Configuration Reference

Key	Description
telegram.bot_token	Telegram bot token from BotFather
telegram.base_url	Telegram API base URL (change for proxy use)
channel.url	Channel page URL
api.url	Channel archive API URL
api.channel_id	Numeric channel ID
api.timeout	Maximum request timeout (seconds)
keywords.include	Messages must contain one of these words
keywords.exclude	Messages containing these words are ignored
keywords.normalize	Persian text normalization (true recommended)
parser.mode	api / auto / html / json / regex
parser.debug_dump	Save raw API response when nothing is found
notification.header	Notification header text (Telegram HTML)
notification.send_channel_link	Add message link
log.level	debug / info / warning / error
log.max_lines	Maximum log lines
cron_key	URL execution security key

---

How Keyword Filtering Works

1. First, exclude words are checked.

   • If found, the message is rejected.
   • This happens even if an include keyword exists.

2. Then include words are checked.

   • At least one must exist.

3. Messages without text (image-only messages) are never sent.

With:

normalize = true

the system normalizes Persian text before comparison:

• Removes half-spaces

• Removes extra spaces

• Removes diacritics

• Converts Arabic characters:

  • ي → ی
  • ك → ک
  • آ → ا

---

8) Logging System

All events are stored in:

logs/scraper.log

Example:

[2026-08-05 15:22:41] [INFO] --- Channel Scraper run started ---
[2026-08-05 15:22:43] [INFO] 30 message(s) fetched from JSON API.
[2026-08-05 15:22:44] [INFO] Notified 12/12 user(s) about message 1713
[2026-08-05 15:22:44] [INFO] Run finished: 2 new, 1 matched, 12 subscriber(s).

Live log monitoring:

tail -f logs/scraper.log

The log file is automatically limited according to:

log.max_lines

---

9) Troubleshooting

Problem	Possible Cause	Solution
config.php not found	Configuration file missing	Run cp config.php.example config.php
api.url or api.channel_id empty	Incomplete API settings	Fill the API section
API returned HTTP 403	Wrong Origin/Referer headers	Match browser headers
API returned HTTP 404	API URL changed	Find the new URL using Network tab
cURL error	Hosting blocks outgoing requests	Ask hosting support to allow access
response does not contain a "messages" array	API response changed	Enable debug dump
No messages sent	Nobody started the bot	Send /start
No messages sent	No new matching messages	Enable debug logs
Regulation messages ignored	Excluded keyword exists	Remove it from exclude list
Cron does not run	Wrong PHP path	Find it with which php
Another instance running	Previous execution not finished	Increase cron interval

---

Full Debug Mode

Enable:

'log' => [
    'level' => 'debug'
],

'parser' => [
    'debug_dump' => true
],

Then run:

php cron.php

Check:

logs/scraper.log
logs/last-api.json
logs/last-api-error.txt

---

10) Security

• config.php contains your bot token — never commit it to Git.

• .htaccess blocks web access to config.php.

• storage/ and logs/ directories are also protected.

• If using URL cron execution, always use a long random cron_key.

• Best practice:

  • Store the project outside public_html
  • Run it only through CLI

---

11) Resetting

To start from zero (remove processed message history):

rm storage/state.json

To remove all subscribers:

rm storage/users.json

---

License

Free to use — provided without any warranty.