Repository navigation
[BUG REPORT] - "Too many requests" #4
Description
Activity
Thank you for your first issue. To better understand your request or the problem you've encountered, please provide as many details as possible. If the behavior changes or if you have new information about your request, don't hesitate to add it. It will be reviewed ASAP.
Hi there,
Well that happens because you have reached a certain limit from your IP, you can connect to a VPN and continue downloading from there or wait a while. Some solutions would be to implement some “interval” between downloads or to be able to define proxies. Any other ideas?
I got this to and added a interval that paused for a few seconds and then tried again for a couple of times. It works better. But also adding a millisecond wait before each go routine adds a bit more help as well.
Also did a check to only download the file if it is not already downloaded and if the request succeeds.
Could add this as a param to the command if it should apply the wait's.
I got this to and added a interval that paused for a few seconds and then tried again for a couple of times. It works better. But also adding a millisecond wait before each go routine adds a bit more help as well.
Also did a check to only download the file if it is not already downloaded and if the request succeeds.
Could add this as a param to the command if it should apply the wait's.
Yeah, definitely dont download the text “too many request” has to be implemented. And some proxy and interval parameters...
The problem with adding so many flags is that it becomes unmaintainable, and you would have to use the flag package, the problem arises because of how Go's flag package parses command-line arguments. By default, the flag package stops parsing flags when it encounters the first non-flag argument (in my case, the URL) then, we would always have to define --url, and I think it is already breaking little by little the simplicity...
Yeah true. Maybe a config json file that have some default values for wait timer etc and then later a proxy config or such could be added in it if / when it is developed. That maybe is the way to go?
I made this snipped, in theory, if the flags are at the beginning they can be parsed until they reach the string URL
// Manually parse flags and positional arguments var args []string for i := 1; i < len(os.Args); i++ { arg := os.Args[i] if arg == "--" { // All remaining args are positional args = append(args, os.Args[i+1:]...) break } if strings.HasPrefix(arg, "-") { // Flag fs.Parse(os.Args[i:]) break } // Positional argument args = append(args, arg) } // After parsing flags, any remaining arguments are positional args = append(args, fs.Args()...)
I made a branch https://github.com/SegoCode/4cget/tree/develop-flags to test and keep coding this implementation. 888dc23
Restarting development on this 🎉
4cget now reads the status first and skips 403/429 instead of calling
io.Copy. On those statuses it prints a Cloudflare hint: low IP trust score, browser challenge required. 429 also mentions--sleep.Cloudflare scores each GET on
boards.4chan.orgfrom IP, ASN, TLS fingerprint, and User-Agent. Normal browser traffic from the same IP raises that score. Open the thread in a browser, browse a bit, then retry the CLI.--sleep Nserializes downloads with N seconds between files. That is the interval we talked about here.--monitoralready waits between thread polls, so the two flags cannot run together.I dropped proxy support. 429 on
i.4cdn.orgis a rate limit on that IP. Spacing requests is the control we have without another hop.If 429 shows up again, raise
--sleepand retry.
Description
When downloading any thread, most of the files will be downloaded as raw text files containing the text: "Too Many Requests"
Steps to reproduce
I used https://boards.4chan.org/wsg/thread/5731328 as a test.
Expected behavior
All of the images should correctly download.
Actual behavior
Only some images correctly download, and the rest are just raw text files.
OS version
Windows 11
Confirmation