|
| 1 | +# Tasks Delegated to the Queue |
| 2 | + |
| 3 | +Normally, the tasks delegated to the queue: |
| 4 | + |
| 5 | +- Take extended periods of time to execute and may be interrupted by PHP limitations (like the PHP `max_execution_time` parameter). |
| 6 | +- Are external which introduces delays from authentication, awaiting responses from potentially multiple interrogations, or may even fail completely because the server is offline. |
| 7 | +- Are not part of the PHP response (sending emails, processing videos). |
| 8 | + |
| 9 | +Here are some example tasks meant for the queue: |
| 10 | + |
| 11 | +- **Data Processing** like big data analytics, scientific simulations, or mathematical computations. |
| 12 | +- **File Handling & Media Processing** like video and image processing, or compression/decompression of large files. |
| 13 | +- **Networking** like uploading/downloading large files. |
| 14 | +- **Database Operations** like imports/exports or migrations. |
| 15 | +- **System & Infrastructure Tasks** like OS updates, software compilation, CI pipelines. |
| 16 | + |
| 17 | +The long execution times are caused by: |
| 18 | + |
| 19 | +- Data size - gigabytes, terabytes etc. |
| 20 | +- Complex algorithms. |
| 21 | +- Hardware limitations - CPU speed, memory, storage, network bandwidth. |
| 22 | +- External dependencies - waiting on APIs or human input. |
| 23 | + |
| 24 | +# How the Queue Works |
| 25 | + |
| 26 | +- The queue system has an active **daemon** that listens for TPC connections on a specific port and **stores incoming messages** into Redis. |
| 27 | +This method supports a **large number of requests per second** without overloading. |
| 28 | +- Operations are then **scheduled for execution** when resources are available. |
| 29 | +The order of the execution uses the **FIFO** (First-In, First-Out) method where the oldest request is processed first, followed by newer requests. |
| 30 | + |
| 31 | +# Main Features |
| 32 | + |
| 33 | +- **Logging** - Ensures maintainability and allows debugging. |
| 34 | +- **Security** - The **firewall** allows requests only from whitelisted IPs for fast response times. |
| 35 | +- **Retry Mechanism** - Retries failing tasks a certain number of times before removing a task from the queue. |
| 36 | +- **Reporting** - The logs enable developers to investigate various metrics via console commands like: |
| 37 | + - Queue length - How many jobs are in the to-do list. |
| 38 | + - Processing time per job. |
| 39 | + - Error rates - How many messages failed. |
| 40 | + - Throughput - Jobs/sec processed. |
0 commit comments