Skip to content

Commit 564bea0

Browse files
committed
added page - what is queue
Signed-off-by: bidi <bidi@apidemia.com>
1 parent dba3b93 commit 564bea0

2 files changed

Lines changed: 46 additions & 1 deletion

File tree

docs/book/v1/overview.md

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,11 @@
11
# Overview
22

3-
> Dotkernel component used to queue tasks to be processed asynchronously based on [netglue/laminas-messenger](https://github.com/netglue/laminas-messenger)
3+
> [Dotkernel Queue](https://github.com/dotkernel/dot-queue) is a component based on [Symfony Messenger](https://github.com/symfony/messenger) that is used to queue asynchronous tasks.
4+
[netglue/laminas-messenger](https://github.com/netglue/laminas-messenger) is a layer that turns Symfony Messenger into a middleware compatible with Mezzio/Laminas applications.
5+
6+
Some everyday **operations are time-consuming and resource-intensive**, so it's best if they run on separate machines, decoupled from the regular request-response cycle.
7+
**Asynchronous execution** performed by background workers ensures that these operations won't overload the main platform.
8+
It allows the main platform to return a response and remain responsive for new requests, while tasks with long execution times are scheduled to run later.
49

510
![Queue process](https://docs.dotkernel.org/img/queue/schema.png)
611

docs/book/v1/what-is-queue.md

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# Tasks Delegated to the Queue
2+
3+
Normally, the tasks delegated to the queue:
4+
5+
- Take extended periods of time to execute and may be interrupted by PHP limitations (like the PHP `max_execution_time` parameter).
6+
- Are external which introduces delays from authentication, awaiting responses from potentially multiple interrogations, or may even fail completely because the server is offline.
7+
- Are not part of the PHP response (sending emails, processing videos).
8+
9+
Here are some example tasks meant for the queue:
10+
11+
- **Data Processing** like big data analytics, scientific simulations, or mathematical computations.
12+
- **File Handling & Media Processing** like video and image processing, or compression/decompression of large files.
13+
- **Networking** like uploading/downloading large files.
14+
- **Database Operations** like imports/exports or migrations.
15+
- **System & Infrastructure Tasks** like OS updates, software compilation, CI pipelines.
16+
17+
The long execution times are caused by:
18+
19+
- Data size - gigabytes, terabytes etc.
20+
- Complex algorithms.
21+
- Hardware limitations - CPU speed, memory, storage, network bandwidth.
22+
- External dependencies - waiting on APIs or human input.
23+
24+
# How the Queue Works
25+
26+
- The queue system has an active **daemon** that listens for TPC connections on a specific port and **stores incoming messages** into Redis.
27+
This method supports a **large number of requests per second** without overloading.
28+
- Operations are then **scheduled for execution** when resources are available.
29+
The order of the execution uses the **FIFO** (First-In, First-Out) method where the oldest request is processed first, followed by newer requests.
30+
31+
# Main Features
32+
33+
- **Logging** - Ensures maintainability and allows debugging.
34+
- **Security** - The **firewall** allows requests only from whitelisted IPs for fast response times.
35+
- **Retry Mechanism** - Retries failing tasks a certain number of times before removing a task from the queue.
36+
- **Reporting** - The logs enable developers to investigate various metrics via console commands like:
37+
- Queue length - How many jobs are in the to-do list.
38+
- Processing time per job.
39+
- Error rates - How many messages failed.
40+
- Throughput - Jobs/sec processed.

0 commit comments

Comments
 (0)