This is a universal Reddit scraper that can scrape Subreddits, Redditors, and comments on posts.
Written in Python and utilizes the Reddit API (PRAW).
I provided a requirements.txt for a quick install of both PRAW and argparse.
pip install -r requirements.txt
You will also need your own Reddit account and API credentials. I have included a tutorial on how to do this below.
NOTE: PRAW is currently supported on Python 3.5+. This project was tested with Python 3.6.
- Scraping Reddit
- How to get Reddit API Credentials
- Walkthrough
- Some Linux Tips
- Contributing
- Contributors
- Releases
Scrape speeds will be determined by the speed of your internet connection.
All exported files will be saved to the current working directory.
These attributes will be included in each scrape.
| Subreddits | Redditors | Post Comments |
|---|---|---|
| Title | Name | Parent ID |
| Flair | Fullname | Comment ID |
| Date Created | ID | Author |
| Upvotes | Date Created | Date Created |
| Upvote Ratio | Comment Karma | Upvotes |
| ID | Link Karma | Text |
| Is Locked? | Is Employee? | Edited? |
| NSFW? | Is Friend? | Is Submitter? |
| Is Spoiler? | Is Mod? | Stickied? |
| Stickied? | Is Gold? | |
| URL | Submissions* | |
| Comment Count | Comments* | |
| Text | Hot* | |
| New* | ||
| Controversial* | ||
| Top* | ||
| Upvoted* (may be forbidden) | ||
| Downvoted* (may be forbidden) | ||
| Gilded* | ||
| Gildings* (may be forbidden) | ||
| Hidden* (may be forbidden) | ||
| Saved* (may be forbidden) |
* Includes additional attributes; see Redditors section for more information
$ ./scraper.py -r SUBREDDIT [H|N|C|T|R|S] N_RESULTS_OR_KEYWORDS --FILE_FORMAT
You can specify Subreddits, which category of posts, and how many results are returned from each scrape. I have also added a search option where you can search for keyword(s) within a Subreddit and the scraper will get all posts that are returned from the search.
These are the post category options:
- Hot
- New
- Controversial
- Top
- Rising
- Search
NOTE: All results are returned if you search for something within a Subreddit, so you will not be able to specify how many results to keep.
Once you configure the settings for the scrape, the program will save the results to either a .csv or .json file.
The file names will follow this format: "r-SUBREDDIT-POST_CATEGORY DATE.[FILE_FORMAT]"
If you have searched for keywords in a Subreddit, file names are formatted as such: "r-SUBREDDIT-Search-'KEYWORDS' DATE.[FILE_FORMAT]"
$ ./scraper.py -u USER N_RESULTS --FILE_FORMAT
You can also scrape Redditor profiles and specify how many results are returned.
Of these Redditor attributes, the following will include additional attributes:
| Submissions, Hot, New, Controversial, Top, Upvoted, Downvoted, Gilded, Gildings, Hidden, and Saved | Comments |
|---|---|
| Title | Date Created |
| Date Created | Score |
| Upvotes | Text |
| Upvote Ratio | Parent ID |
| ID | Link ID |
| NSFW? | Edited? |
| Text | Stickied? |
| Replying to (title of post or comment) | |
| In Subreddit (Subreddit name) |
NOTE: If you are not allowed to access a Redditor's lists, PRAW will raise a 403 HTTP Forbidden exception and the program will just append a "FORBIDDEN" underneath that section in the exported file.
NOTE: The number of results returned will be applied to all attributes. I have not implemented code to allow users to specify different number of results returned for individual attributes.
The file names will follow this format: "u-USERNAME DATE.[FILE_FORMAT]"
$ ./scraper.py -c URL N_RESULTS --FILE_FORMAT
These scrapes were designed to be used with JSON only. Exporting to CSV is not recommended, but it will still work.
You can also scrape comments from posts and specify the number of results returned.
Comments scraping can either return structured JSON data down to third-level comment replies, or you can simply return a raw list of all comments with no structure.
To return a raw list of all comments, specify 0 results to be returned from the scrape.
When exporting raw comments, all top-level comments are listed first, followed by second-level, third-level, etc.
NOTE: You cannot specify the number of raw comments returned. The program with scrape all comments from the post, which may take a while depending on the post's popularity.
The file names will follow this format: "c-POST_TITLE DATE.[FILE_FORMAT]"
First, create your own Reddit account and then head over to Reddit's apps page.
Click "are you a developer? create an app...".
Name your app, choose "script" for the type of app, and type "http://localhost:8080" in the redirect URI field since this is a personal use app. You can also add a description and an about URL.
Click "create app", then "edit" to reveal more information.
You should see a string of 14 characters on the top left corner underneath "personal use script." That is your API ID. Further down you will see "secret" and a string of 27 characters; that is your API password. Save this information as it will be used in the program in order to use the Reddit API.
You will also have to provide your app name, Reddit account username and password in the block of credentials found on lines 18-22.
First, you will have to provide your own Reddit credentials in this block of code.
You have to provide valid credentials, otherwise the scraper will not work.
If you choose to use 2FA with your Reddit account, enter your password followed by a colon and then your 2FA token in the passwd field on line 22. For example, if your password is "p4ssw0rd" and your 2FA token is "123456", you will enter "p4ssw0rd:123456" in the passwd field.
2FA is NOT recommended for use with this program. This is because PRAW will raise an OAuthException after one hour, prompting you to refresh your 2FA token and re-enter your credentials. Additionally, this means your 2FA token would be stored alongside your Reddit username and password, which would defeat the purpose of enabling 2FA in the first place. See here for more information.
If you do not want to read the rest of this walkthrough, or forget the args, you can always consult the built-in help message by using -h or --help.
Scraping Subreddits using flags is much faster than the basic scraper.
Use the -r flag to indicate a Subreddit, the post category, and finally, depending on the category selected, either the number of results returned or keyword(s) to search for during the scrape.
Category options are as follows:
- H,h - Hot
- N,n - New
- C,c - Controversial
- T,t - Top
- R,r - Rising
- S,s - Search
Scraping 10 r/AskReddit posts in the Hot category and export to JSON:
The program will then display the type of scrape, check if the Subreddit(s) exist, and display the settings for each Subreddit. It will display a list of invalid Subreddits, if applicable. You can also include -y in your args if you want to skip this confirmation screen and immediately scrape.
JSON Sample:
CSV Sample:
Use the -u flag to indicate a Redditor and the number of results returned. The program will then display the type of scrape and check if the Redditor(s) exist. It will display a list of invalid Redditors, if applicable.
This is currently for JSON ONLY. There is a CSV export bug that needs squashing.
Scraping 5 results for each of u/spez's user attributes and export to JSON:
There are a couple user lists that are typically restricted and will raise an 403 HTTP Forbidden exception. If you are forbidden from accessing a list, the program will display its name and append "FORBIDDEN" to that section in the export file.
JSON Sample:
Use the -c flag to indicate a post and the number of comments returned. The program will then display the type of scrape and check if the post(s) exist. It will display a list of invalid posts, if applicable.
I have designed this functionality to work best with JSON and strongly recommend this export option, however you will still be able to get your results if you choose to export to CSV instead.
There are two ways you can scrape comments with this program. You can indicate a number to return a structured JSON file that includes down to third-level replies. Or you can specify 0 comments to be returned and the program will return an unstructured JSON file of all comments.
Structured Scrape
Scraping 10 comments from this Reddit post and export to JSON:
Structured JSON Sample:
Unstructured Scrape
When exporting raw comments, all top-level comments are listed first, followed by second-level, third-level, etc.
Unstructured JSON Sample:
I kept URS 1.0's functionality after I added CLI support, in case anyone prefers to use it over CLI arguments. It only scrapes Subreddits; Redditor and post comments scraping would require you to use CLI arguments.
You can access the basic scraper by using the -b flag and an export option.
You can just scrape a single Subreddit, or enter a list of Subreddits separated by a space.
After entering the Subreddit(s) you want to scrape, the program will check if the Subreddit exists. It will separate the results into a list of valid and invalid Subreddits.
You will then choose the post category within the Subreddit (Hot, New, Controversial, Top, Rising, Search). After choosing the category, you will also choose how many results you would like to be returned.
If you choose to search for keyword(s) within the Subreddit, you will be greeted with these settings instead.
After you have configured all settings for each Subreddit, you will be greeted with the following screen which displays all of your settings. After confirming, the program will scrape the Subreddits based on your parameters.
- You can further simplify running the program by making the program executable.
sudo chmod +x scraper.py- Make sure the shebang at the top of scraper.py matches the location in which your Python3.6 is installed. You can use
which python3.6to check. The default shebang is#!/usr/bin/python3.6. - Now you will only have to prepend
./to run the scraper../scraper.py ...
- Troubleshooting
- If you run the scraper with
./and are greeted with a bad interpreter error, you will have to set the fileformat to UNIX. I did this using Vim.-
vim scraper.py :set fileformat=unix :wq!
-
- If you run the scraper with
I have decided URS 3.0 will be the last major iteration of this project that I will release. I believe the current features should satisfy users who need to scrape Reddit. However, you can still suggest new features that I can add. If there are good suggestions, and a good reason, for a new feature, I will consider adding it. You are also more than welcome to create a pull request, adding additional features or simply improving runtime or streamlining existing code yourself. If the pull request is approved, I will merge the pull request into the master branch, tag it as a new release, and credit you for contributing to this project.
Make sure you follow the contributing guidelines when creating a pull request. See the Contributing document for more information.
- March 11, 2020: User ThereGoesMySanity created a pull request adding 2FA information to Readme.
- May 25, 2019: Universal Reddit Scraper 1.0. Does not include CLI support.
- July 29, 2019: Universal Reddit Scraper 2.0. Now includes CLI support!
- December 28, 2019: Universal Reddit Scraper 3.0 (Beta).
- New features include:
- Exporting to JSON
- Scraping Redditors
- Comments scraping functionality is still under construction.
- New features include:
- December 31, 2019: Universal Reddit Scraper 3.0 (Official).
- Comments scraping functionality is now working!
- Added additional exception handling for creating filenames
- Minor code reformatting
- Simplified verbose output
- Added an additional Submission attribute when scraping Redditors
- Happy New Year!
- January 15, 2020: Universal Reddit Scraper 3.0 (Final Release).
- Numerous changes to Readme
- Minor code reformatting
- Fulfilled community standards by adding the following docs:
- Contributing guidelines
- Pull request template
- Issue templates (bug report and feature request)
- Code of Conduct
- License