Throttling detection and automatic temporary cooldown

[ebbyb]

On occasion I lock horns with YouTube's throttling algorithm but usually catch it after a week of stalled downloads. It would be nice to detect throttling-based errors, re-test a known successful (low bandwidth) task, and implement a cooldown for 48-72 hours or so if necessary before testing or ramping up again.

(Feature request)

[tcely]

This is already implemented.

tubesync/tubesync/sync/tasks.py

Lines 316 to 363 in 8c41617

	def contains_http429(q, task_id, /):
	from huey.exceptions import TaskException
	try:
	q.result(preserve=True, id=task_id)
	except TaskException as e:
	return True if 'HTTPError 429: Too Many Requests' in str(e) else False
	return False
	def wait_for_errors(model, /, *, queue_name=None, task_name=None):
	if task_name is None:
	task_name=tuple((
	'sync.tasks.download_media_file',
	'sync.tasks.download_media_metadata',
	))
	elif isinstance(task_name, str):
	task_name = tuple((task_name,))
	tasks = list()
	for tn in task_name:
	ft = get_first_task(tn, instance=model)
	if ft:
	tasks.append(ft)
	for task in tasks:
	update_task_status(task, 'paused (429)')
	total_count = int()
	if queue_name:
	from django_huey import get_queue
	q = get_queue(queue_name)
	total_count += sum([ 1 if contains_http429(q, k) else 0 for k in q.all_results() ])
	delay = 10 * total_count
	time_str = seconds_to_timestr(delay)
	db_down_path = Path('/run/service/huey-database/down')
	fs_down_path = Path('/run/service/huey-filesystem/down')
	log.info(f'waiting for errors: 429 ({time_str}): {model}')
	while delay > 0:
	# this happenes when the container is shutting down
	# do not prevent that while we are delaying a task
	if db_down_path.exists() and fs_down_path.exists():
	break
	time.sleep(5)
	delay -= 5
	for task in tasks:
	update_task_status(task, None)
	if delay > 0:
	raise HueyConsumerError(_('queue consumer stopped'))

Search your logs for waiting for errors: 429 to see if this was triggered.

If you have another idea, please describe it fully.

[ebbyb]

My apologies for the redundancy, I tried searching before posting; my skills failed me.

I do see that error in my logs. However, the soft ban lasted from June 3rd to today, July 2nd only after shutting down the docker container for 3 days.

I also see a LOT of "Sign in to confirm you're not a bot" throughout the month perhaps indicating TubeSync isn't as stealthy as it should be during cooldown.

Nonetheless, thank you and the tubeSync teams for all your work.

EDIT: It took about 4 minutes before Error 429 popped back up in the logs and a bunch of "Sign in to confirm you're not a bot" messages to return. I don't think TubeSync is waiting.

[tcely]

I won't claim it is working perfectly. There is always room for improvement.

Are you using the Proof-of-Origin Token support?

What version of yt-dlp is being used?

Are you seeing delays with the waiting for errors: 429 log messages?