Repository navigation
Extend driver API and add subqueuettl driver - #259
Conversation
Support driver-specific task selection through take() options. Some drivers partition tasks by driver-specific attributes and must select tasks only from the partition requested by a consumer. For example, a driver may use opts.subqueue to take tasks from a particular subqueue. Drivers can derive a consumer group from take() options and tasks. This lets the wake-up mechanism notify only consumers eligible to take a newly ready task, rather than waking an arbitrary consumer from the same tube that cannot process it.
Support driver-specific statistics. Some drivers maintain additional state or partition tasks using driver-specific attributes. They need to expose this data through the standard tube statistics API, alongside the common queue metrics. This allows each driver to report meaningful operational metrics without extending the generic statistics implementation for every driver-specific use case.
1cfcd23 to
c2947d0
Compare
49d0bea to
6871e86
Compare
6871e86 to
d8dcd2f
Compare
d8dcd2f to
89476fc
Compare
|
I added registry cleanup in case there are no tasks left in the subqueue, which may mean that the subqueue is no longer in use. |
|
Please, remove trailing whitespaces from |
Some workloads create logical queues with arbitrary names. Creating a separate tube for each one requires converting names to valid space names and granting access whenever a tube is created. Add a TTL-aware driver that stores named subqueues in one space. Require a subqueue name for put() and take(), so consumers can take tasks from a specific subqueue. Support TTL, TTR, priority, and delay.
89476fc to
bc4d835
Compare
|
The project doesn't have a CONTRIBUTE.md file with commit message guidelines. I fixed the message by trying to follow the tarantool/tarantool guidelines. |
|
I think we should restart the failed jobs. Tarantool could not be loaded due to a network error. Second failed with a strange error:
I also created the same PR in the fork. The workflow completed successfully. |

Extend driver API:
Added the optional consumer_group driver API so a driver can wake a consumer waiting for the same logical group instead of waking an unrelated consumer.
Added custom statistics for drivers to support showing custom statistics by them.
Subqueuettl:
Some workloads need to create queues dynamically with arbitrary names. Creating a separate tube for each queue requires converting those names to valid Tarantool space names and granting access whenever a new tube is created. This adds operational overhead and makes access management harder.
The utube model is otherwise well suited for this use case: it stores multiple logical queues in one space and guarantees sequential processing inside each queue. However, it does not support taking a task from a specific utube, so it cannot be used when consumers must be assigned to particular dynamically named queues.
Added subqueuettl as a TTL-aware alternative. It stores multiple dynamically named subqueues in one space, and both put() and take() require a subqueue name. This keeps task selection isolated while retaining TTL, TTR, priority, delay, and ready-buffer support.
If you don’t think it’s necessary to add this driver to the general library, I can remove it from this PR, but for my purposes, I need support for changes in the “abstract.lua” in order to use this driver.