3rd October 2021
Redshift supports multiple concurrent queries.
The mechanism used to control and configure concurrent queries, both system-issued and user-issued, is that of queues and slots.
For users-issued queries, there can be from one to seven queues, each of which has one to fifty slots, and there is a system-wide maximum of fifty slots. The number of queues and how many slots are in each queue is configured by a system administrator.
Each slot can run one query (actually, a single slot can in the right circumstances run multiple queries, but that’s for a later white paper).
When a query is issued, a queue is selected (by means which are irrelevant to the investigation) and the query sent to that queue. If the queue has empty slots, one of them will be used to execute the query immediately. If all slots are occupied, the query goes into a FIFO queue for that queue, and will be executed once a slot becomes available.
There is then with Redshift a limit of fifty concurrent user-issued queries; fifty slots would be configured and fifty queries issued. At this point any additional user-issued queries would necessarily be queued.
This white paper investigates whether Redshift does in fact allow fifty user-issued queries to be executed concurrently.
A Redshift cluster is configured to have one queue of 50 slots.
Two simple test tables are created, and then 50 identical queries are
simultaneously issued, which select the count(*) of the join of the two
tables. The tables contain a single raw encoded int8
column, with one exactly full block per slice of randomly generated
int8
, so the count is typically zero. Where there are so
many slots, each slot has very little memory and even with only a single
block per slice, the queries take a long time, about 30 minutes.
Once all 50 queries have been issued, we immediately issue a two queries into the superuser queue, the first being leader-node only, the second using worker-nodes.
We then note when every query was queued, when it began executing and when it finished executing.
We then examine the results to see what we find.
The same as proof #1, but now only 40 queries are issued.
Based on knowledge from the first proof, we now create two queues, of 30 slots and 10 slots (memory allocated pro rata).
Then as before 30 identical queries (still counting the join of the tables) are simultaneously issued, this time into the 30 slot queue, and then once they are all issued, 10 more identical queries as issued into the ten slot queue.
As before, we then immediately issue a two queries into the superuser queue, the first being leader-node only, the second using worker-nodes, and we then note when every query was queued, when it began executing and when it finished executing and examine the results to see what we find.
The same as proof #3, but now with 20 queries in the second queue, which has 20 queries issued.
See Appendix A for the Python
pprint
dump of the results dictionary.
The Python script used to generate the results is available from the White Pages page at the Redshift Research Project. The script is user-friendly, self-contained, stand-alone and exists so that the results can simply and easily be reproduced.
Total test time was 8,095 seconds.
Queue 6 is the single user-configured and default queue. All other queues are system queues.
Queue # | Slots |
---|---|
1 | 1 |
2 | 1 |
3 | 2 |
4 | 1 |
5 | 1 |
6 | 50 |
15 | 0 |
All queries are reported as being queued and beginning to execute at the same time, but note the first 30 queries after about 33 minutes complete together, and then after another 20 minutes, the remaining 20 queries complete together (examine the execution end times for queries 30 and 31).
Query | Queue | Queue Start | Exec Start | Exec End |
---|---|---|---|---|
1 | 6 | 15:49:58.127771 | 15:49:58.127780 | 16:21:50.613780 |
2 | 6 | 15:49:58.070566 | 15:49:58.070609 | 16:21:50.615748 |
3 | 6 | 15:49:58.120498 | 15:49:58.120510 | 16:21:50.617066 |
4 | 6 | 15:49:58.210868 | 15:49:58.210878 | 16:21:50.642211 |
5 | 6 | 15:49:58.160839 | 15:49:58.160847 | 16:21:50.671250 |
6 | 6 | 15:49:58.062473 | 15:49:58.062483 | 16:21:50.710787 |
7 | 6 | 15:49:58.089557 | 15:49:58.089566 | 16:21:50.755797 |
8 | 6 | 15:49:57.835326 | 15:49:57.835334 | 16:21:50.770947 |
9 | 6 | 15:49:58.232241 | 15:49:58.232252 | 16:21:50.823570 |
10 | 6 | 15:49:58.132214 | 15:49:58.132222 | 16:21:50.855381 |
11 | 6 | 15:49:58.087055 | 15:49:58.087063 | 16:21:50.887268 |
12 | 6 | 15:49:58.051477 | 15:49:58.051487 | 16:21:53.884859 |
13 | 6 | 15:49:58.259194 | 15:49:58.259203 | 16:21:54.227551 |
14 | 6 | 15:49:58.117327 | 15:49:58.117336 | 16:21:54.585056 |
15 | 6 | 15:49:58.263676 | 15:49:58.263683 | 16:21:54.848468 |
16 | 6 | 15:49:58.122405 | 15:49:58.158070 | 16:21:55.096476 |
17 | 6 | 15:49:58.071046 | 15:49:58.071056 | 16:21:56.054904 |
18 | 6 | 15:49:57.829600 | 15:49:57.829612 | 16:21:56.435633 |
19 | 6 | 15:49:58.184733 | 15:49:58.184741 | 16:21:58.094943 |
20 | 6 | 15:49:58.274706 | 15:49:58.274714 | 16:21:58.710918 |
21 | 6 | 15:49:58.086042 | 15:49:58.086050 | 16:21:59.336118 |
22 | 6 | 15:49:58.183266 | 15:49:58.183278 | 16:21:59.395178 |
23 | 6 | 15:49:58.181344 | 15:49:58.181354 | 16:21:59.635844 |
24 | 6 | 15:49:58.045031 | 15:49:58.045051 | 16:22:00.307921 |
25 | 6 | 15:49:58.132691 | 15:49:58.132699 | 16:22:00.427576 |
26 | 6 | 15:49:58.227170 | 15:49:58.227183 | 16:22:00.658998 |
27 | 6 | 15:49:58.219830 | 15:49:58.219837 | 16:22:01.937114 |
28 | 6 | 15:49:58.190592 | 15:49:58.190610 | 16:22:03.198462 |
29 | 6 | 15:49:58.078237 | 15:49:58.078245 | 16:22:03.220855 |
30 | 6 | 15:49:58.200588 | 15:49:58.200595 | 16:22:04.820606 |
31 | 6 | 15:49:58.243559 | 15:49:58.243568 | 16:42:11.351617 |
32 | 6 | 15:49:58.196226 | 15:49:58.196232 | 16:42:11.936405 |
33 | 6 | 15:49:58.215345 | 15:49:58.215358 | 16:42:18.099604 |
34 | 6 | 15:49:58.213584 | 15:49:58.213594 | 16:42:18.491249 |
35 | 6 | 15:49:58.217188 | 15:49:58.217199 | 16:42:18.510718 |
36 | 6 | 15:49:58.294468 | 15:49:58.294476 | 16:42:20.603540 |
37 | 6 | 15:49:58.282827 | 15:49:58.282834 | 16:42:20.604723 |
38 | 6 | 15:49:58.278099 | 15:49:58.278107 | 16:42:21.234185 |
39 | 6 | 15:49:58.215968 | 15:49:58.215977 | 16:42:21.551156 |
40 | 6 | 15:49:58.154419 | 15:49:58.154426 | 16:42:21.603266 |
41 | 6 | 15:49:58.224768 | 15:49:58.224776 | 16:42:21.651115 |
42 | 6 | 15:49:58.287855 | 15:49:58.287862 | 16:42:23.474200 |
43 | 6 | 15:49:58.425462 | 15:49:58.425469 | 16:42:23.685237 |
44 | 6 | 15:49:58.262150 | 15:49:58.262157 | 16:42:23.934793 |
45 | 6 | 15:49:58.278275 | 15:49:58.278281 | 16:42:24.114984 |
46 | 6 | 15:49:58.262670 | 15:49:58.262681 | 16:42:24.856536 |
47 | 6 | 15:49:58.354781 | 15:49:58.354789 | 16:42:26.005504 |
48 | 6 | 15:49:58.416414 | 15:49:58.416423 | 16:42:26.258593 |
49 | 6 | 15:49:58.419910 | 15:49:58.419917 | 16:42:26.276310 |
50 | 6 | 15:49:58.416569 | 15:49:58.416576 | 16:42:26.483855 |
The following single row table is the superuser queue leader-node query issued immediately after the 50 queries were issued.
It appears currently no logging of completed leader-node
queries is occurring (it used to) and so all we have is the timestamp
from the timeofday()
function, which was issued in the
query. This is however adequate to show when execution occurred.
timeofday() |
---|
Mon Oct 04 15:50:02.432721 2021 UTC |
The following single row table is the superuser queue worker-node query issued immediately after leader-node query completed. We see it only finally executed once one or more of the initial batch of 30 queries completed.
Queue # | Queue Start | Exec Start | timeofday() | Exec End |
---|---|---|---|---|
5 | 15:50:02.481307 | 15:50:02.481321 | Mon Oct 04 16:21:50.898244 2021 UTC | 16:21:50.898769 |
Queue # | Slots |
---|---|
1 | 1 |
2 | 1 |
3 | 2 |
4 | 1 |
5 | 1 |
6 | 50 |
15 | 0 |
Here we issued only 40 queries. As before, we see a first batch of 30 completing together, and then the final batch of 10 completing together.
Query # | Queue # | Queue Start | Exec Start | Exec End |
---|---|---|---|---|
1 | 6 | 16:43:56.785943 | 16:43:56.785951 | 17:14:54.546982 |
2 | 6 | 16:43:56.990794 | 16:43:56.990802 | 17:15:13.130696 |
3 | 6 | 16:43:56.932712 | 16:43:56.932726 | 17:15:14.538681 |
4 | 6 | 16:43:57.101137 | 16:43:57.101143 | 17:15:42.708895 |
5 | 6 | 16:43:56.792195 | 16:43:56.792205 | 17:15:43.101332 |
6 | 6 | 16:43:57.087052 | 16:43:57.087060 | 17:15:44.954837 |
7 | 6 | 16:43:56.643655 | 16:43:56.643979 | 17:15:46.211009 |
8 | 6 | 16:43:56.990043 | 16:43:56.990052 | 17:15:46.500821 |
9 | 6 | 16:43:56.997153 | 16:43:56.997821 | 17:15:46.946331 |
10 | 6 | 16:43:56.783937 | 16:43:56.783947 | 17:15:47.163977 |
11 | 6 | 16:43:57.099557 | 16:43:57.099564 | 17:15:47.438160 |
12 | 6 | 16:43:57.090594 | 16:43:57.090603 | 17:15:47.732970 |
13 | 6 | 16:43:56.943640 | 16:43:56.943649 | 17:15:47.737157 |
14 | 6 | 16:43:57.065514 | 16:43:57.065521 | 17:15:47.739972 |
15 | 6 | 16:43:57.077606 | 16:43:57.077625 | 17:15:47.856719 |
16 | 6 | 16:43:56.890170 | 16:43:56.890181 | 17:15:48.003832 |
17 | 6 | 16:43:57.049426 | 16:43:57.049434 | 17:15:48.910747 |
18 | 6 | 16:43:56.641216 | 16:43:56.641229 | 17:15:49.269827 |
19 | 6 | 16:43:56.867961 | 16:43:56.867989 | 17:15:49.288471 |
20 | 6 | 16:43:57.036741 | 16:43:57.036749 | 17:15:51.362420 |
21 | 6 | 16:43:56.631125 | 16:43:56.631135 | 17:15:51.563885 |
22 | 6 | 16:43:57.079126 | 16:43:57.079139 | 17:15:51.840308 |
23 | 6 | 16:43:57.041135 | 16:43:57.041149 | 17:15:51.958705 |
24 | 6 | 16:43:57.022074 | 16:43:57.022085 | 17:15:51.998679 |
25 | 6 | 16:43:56.988643 | 16:43:56.988656 | 17:15:52.021174 |
26 | 6 | 16:43:56.998370 | 16:43:56.998380 | 17:15:52.077485 |
27 | 6 | 16:43:57.025380 | 16:43:57.025387 | 17:15:52.114683 |
28 | 6 | 16:43:56.990945 | 16:43:56.990951 | 17:15:54.020131 |
29 | 6 | 16:43:57.067723 | 16:43:57.067731 | 17:15:54.785593 |
30 | 6 | 16:43:57.063241 | 16:43:57.063250 | 17:15:55.953745 |
31 | 6 | 16:43:57.118359 | 16:43:57.118365 | 17:25:36.035945 |
32 | 6 | 16:43:57.155690 | 16:43:57.155698 | 17:25:45.103205 |
33 | 6 | 16:43:57.121212 | 16:43:57.121219 | 17:25:45.825897 |
34 | 6 | 16:43:57.167407 | 16:43:57.167415 | 17:25:46.036079 |
35 | 6 | 16:43:57.121998 | 16:43:57.125695 | 17:25:48.425447 |
36 | 6 | 16:43:57.146333 | 16:43:57.146338 | 17:25:48.902028 |
37 | 6 | 16:43:57.160894 | 16:43:57.160901 | 17:25:49.069143 |
38 | 6 | 16:43:57.146171 | 16:43:57.146180 | 17:25:49.130059 |
39 | 6 | 16:43:57.161057 | 16:43:57.161063 | 17:25:49.499375 |
40 | 6 | 16:43:57.157965 | 16:43:57.157972 | 17:25:49.911973 |
The following single row table is the superuser queue leader-node query issued immediately after the 50 queries were issued.
It appears currently no logging of completed leader-node
queries is occurring (it used to) and so all we have is the timestamp
from the timeofday()
function, which was issued in the
query. This is however adequate to show when execution occurred.
timeofday() |
---|
Mon Oct 04 16:44:01.546710 2021 UTC |
The following single row table is the superuser queue worker-node query issued immediately after leader-node query completed. Here we see the query as before only executed once queries from the first 30 queries completed.
Queue # | Queue Start | Exec Start | timeofday() | Exec End |
---|---|---|---|---|
5 | 16:44:01.594764 | 16:44:01.594776 | Mon Oct 04 16:21:50.898244 2021 UTC | 17:14:54.548745 |
Queue # | Slots |
---|---|
1 | 1 |
2 | 1 |
3 | 2 |
4 | 1 |
5 | 1 |
6 | 10 |
7 | 30 |
15 | 0 |
The 30 queries to queue #7 were issued first (examine the Queue Start timestamps), followed by the 10 queries into queue #6.
Here we see all queries actually executed in parallel; they all begin at about the same time and all ended at about the same time; they ran in parallel.
Query # | Queue # | Queue Start | Exec Start | Exec End |
---|---|---|---|---|
1 | 7 | 17:29:10.116657 | 17:29:10.116666 | 17:41:57.179292 |
2 | 7 | 17:29:10.152597 | 17:29:10.152608 | 17:41:49.509474 |
3 | 7 | 17:29:10.159927 | 17:29:10.159936 | 17:41:48.999495 |
4 | 7 | 17:29:10.210375 | 17:29:10.210384 | 17:41:49.683588 |
5 | 7 | 17:29:10.213187 | 17:29:10.213194 | 17:41:58.276473 |
6 | 7 | 17:29:10.235929 | 17:29:10.235939 | 17:41:58.284404 |
7 | 7 | 17:29:10.254439 | 17:29:10.254448 | 17:41:50.098457 |
8 | 7 | 17:29:10.296682 | 17:29:10.296692 | 17:41:49.919376 |
9 | 7 | 17:29:10.324581 | 17:29:10.324590 | 17:41:58.246735 |
10 | 7 | 17:29:10.328897 | 17:29:10.328904 | 17:41:58.341320 |
11 | 7 | 17:29:10.358083 | 17:29:10.358092 | 17:41:50.165056 |
12 | 7 | 17:29:10.363645 | 17:29:10.363653 | 17:41:50.077641 |
13 | 7 | 17:29:10.365459 | 17:29:10.365467 | 17:41:58.239877 |
14 | 7 | 17:29:10.368713 | 17:29:10.368720 | 17:41:50.164706 |
15 | 7 | 17:29:10.411387 | 17:29:10.411397 | 17:41:50.118964 |
16 | 7 | 17:29:10.411742 | 17:29:10.411748 | 17:41:58.312306 |
17 | 7 | 17:29:10.417405 | 17:29:10.417412 | 17:41:52.788602 |
18 | 7 | 17:29:10.448896 | 17:29:10.448909 | 17:41:58.340954 |
19 | 7 | 17:29:10.449213 | 17:29:10.449220 | 17:41:58.366137 |
20 | 7 | 17:29:10.451035 | 17:29:10.451046 | 17:41:58.266653 |
21 | 7 | 17:29:10.452813 | 17:29:10.452823 | 17:41:58.129337 |
22 | 7 | 17:29:10.461381 | 17:29:10.461389 | 17:41:58.257869 |
23 | 7 | 17:29:10.461546 | 17:29:10.461554 | 17:41:50.103795 |
24 | 7 | 17:29:10.462643 | 17:29:10.462651 | 17:41:58.252911 |
25 | 7 | 17:29:10.489043 | 17:29:10.489051 | 17:41:55.676683 |
26 | 7 | 17:29:10.508213 | 17:29:10.508222 | 17:41:52.912922 |
27 | 7 | 17:29:10.524630 | 17:29:10.524637 | 17:41:55.929408 |
28 | 7 | 17:29:10.525336 | 17:29:10.525354 | 17:41:52.015737 |
29 | 7 | 17:29:10.584906 | 17:29:10.584913 | 17:41:50.928450 |
30 | 7 | 17:29:10.590228 | 17:29:10.590234 | 17:41:51.585367 |
31 | 6 | 17:29:15.139446 | 17:29:15.139459 | 17:41:58.368530 |
32 | 6 | 17:29:15.146818 | 17:29:15.146829 | 17:41:56.844642 |
33 | 6 | 17:29:15.149278 | 17:29:15.149286 | 17:41:58.521654 |
34 | 6 | 17:29:15.151224 | 17:29:15.151233 | 17:41:58.817737 |
35 | 6 | 17:29:15.151487 | 17:29:15.151497 | 17:41:58.370361 |
36 | 6 | 17:29:15.155440 | 17:29:15.156011 | 17:41:55.992180 |
37 | 6 | 17:29:15.187292 | 17:29:15.187304 | 17:41:58.621754 |
38 | 6 | 17:29:15.187804 | 17:29:15.187814 | 17:41:55.308719 |
39 | 6 | 17:29:15.189784 | 17:29:15.189792 | 17:41:54.085506 |
40 | 6 | 17:29:15.193096 | 17:29:15.193103 | 17:41:56.972871 |
The following single row table is the superuser queue leader-node query issued immediately after the 50 queries were issued.
It appears currently no logging of completed leader-node
queries is occurring (it used to) and so all we have is the timestamp
from the timeofday()
function, which was issued in the
query. This is however adequate to show when execution occurred.
timeofday() |
---|
Mon Oct 04 17:29:15.046693 2021 UTC |
The following single row table is the superuser queue worker-node query issued immediately after leader-node query completed. Here we see this query executed immediately; it was not blocked.
Queue # | Queue Start | Exec Start | timeofday() | Exec End |
---|---|---|---|---|
5 | 17:29:15.181314 | 17:29:15.181327 | Mon Oct 04 17:29:15.223939 2021 UTC | 17:29:15.231921 |
Queue # | Slots |
---|---|
1 | 1 |
2 | 1 |
3 | 2 |
4 | 1 |
5 | 1 |
6 | 20 |
7 | 30 |
15 | 0 |
Finally, to check that even with 50 queries, as long as they’re spread over multiple queues, execution is fully parallel, this 30/20 proof. We see here that all queries began at about the same time and ended at about the same time; they really did run in parallel.
Query # | Queue # | Queue Start | Exec Start | Exec End |
---|---|---|---|---|
1 | 7 | 17:45:40.508312 | 17:45:40.508322 | 18:01:41.512858 |
2 | 7 | 17:45:40.565982 | 17:45:40.565995 | 18:01:23.100758 |
3 | 7 | 17:45:40.608091 | 17:45:40.608103 | 18:01:23.720642 |
4 | 7 | 17:45:40.663260 | 17:45:40.663291 | 18:01:42.685471 |
5 | 7 | 17:45:40.674131 | 17:45:40.674140 | 18:01:42.632185 |
6 | 7 | 17:45:40.681154 | 17:45:40.681164 | 18:01:42.484303 |
7 | 7 | 17:45:40.717998 | 17:45:40.718013 | 18:01:42.171824 |
8 | 7 | 17:45:40.726040 | 17:45:40.726051 | 18:01:42.272628 |
9 | 7 | 17:45:40.726193 | 17:45:40.726200 | 18:01:24.175341 |
10 | 7 | 17:45:40.746223 | 17:45:40.746232 | 18:01:42.579424 |
11 | 7 | 17:45:40.753393 | 17:45:40.753400 | 18:01:42.687405 |
12 | 7 | 17:45:40.756592 | 17:45:40.756601 | 18:01:26.416616 |
13 | 7 | 17:45:40.760058 | 17:45:40.760067 | 18:01:24.048386 |
14 | 7 | 17:45:40.761097 | 17:45:40.761106 | 18:01:42.580756 |
15 | 7 | 17:45:40.761158 | 17:45:40.761321 | 18:01:24.080262 |
16 | 7 | 17:45:40.747651 | 17:45:40.762795 | 18:01:24.112432 |
17 | 7 | 17:45:40.760882 | 17:45:40.764710 | 18:01:42.648259 |
18 | 7 | 17:45:40.764762 | 17:45:40.764771 | 18:01:23.920977 |
19 | 7 | 17:45:40.768682 | 17:45:40.768691 | 18:01:42.471272 |
20 | 7 | 17:45:40.778725 | 17:45:40.778736 | 18:01:42.664954 |
21 | 7 | 17:45:40.778944 | 17:45:40.778952 | 18:01:23.964037 |
22 | 7 | 17:45:40.783861 | 17:45:40.783869 | 18:01:23.816765 |
23 | 7 | 17:45:40.791753 | 17:45:40.791760 | 18:01:40.880951 |
24 | 7 | 17:45:40.792762 | 17:45:40.792769 | 18:01:41.372720 |
25 | 7 | 17:45:40.795472 | 17:45:40.795479 | 18:01:42.455657 |
26 | 7 | 17:45:40.894647 | 17:45:40.894656 | 18:01:23.948445 |
27 | 7 | 17:45:40.895070 | 17:45:40.895079 | 18:01:34.244923 |
28 | 7 | 17:45:40.911679 | 17:45:40.911690 | 18:01:42.376488 |
29 | 7 | 17:45:40.911803 | 17:45:40.911812 | 18:01:42.705763 |
30 | 7 | 17:45:40.917010 | 17:45:40.917016 | 18:01:24.320836 |
31 | 6 | 17:45:45.596045 | 17:45:45.596060 | 18:01:41.684372 |
32 | 6 | 17:45:45.606666 | 17:45:45.606677 | 18:01:40.760743 |
33 | 6 | 17:45:45.608494 | 17:45:45.608501 | 18:01:43.120154 |
34 | 6 | 17:45:45.627556 | 17:45:45.627565 | 18:01:42.940785 |
35 | 6 | 17:45:45.632816 | 17:45:45.632827 | 18:01:39.921029 |
36 | 6 | 17:45:45.647367 | 17:45:45.647376 | 18:01:43.102451 |
37 | 6 | 17:45:45.652132 | 17:45:45.652139 | 18:01:29.509487 |
38 | 6 | 17:45:45.665223 | 17:45:45.665234 | 18:01:30.024999 |
39 | 6 | 17:45:45.671703 | 17:45:45.671710 | 18:01:42.976345 |
40 | 6 | 17:45:45.671790 | 17:45:45.671797 | 18:01:29.632491 |
41 | 6 | 17:45:45.675316 | 17:45:45.675324 | 18:01:29.544447 |
42 | 6 | 17:45:45.683275 | 17:45:45.683283 | 18:01:42.579789 |
43 | 6 | 17:45:45.683556 | 17:45:45.683562 | 18:01:29.850047 |
44 | 6 | 17:45:45.711906 | 17:45:45.711915 | 18:01:30.019516 |
45 | 6 | 17:45:45.712075 | 17:45:45.712082 | 18:01:29.967331 |
46 | 6 | 17:45:45.715886 | 17:45:45.715894 | 18:01:40.505223 |
47 | 6 | 17:45:45.721388 | 17:45:45.721397 | 18:01:34.643008 |
48 | 6 | 17:45:45.722392 | 17:45:45.722399 | 18:01:29.714263 |
49 | 6 | 17:45:45.728650 | 17:45:45.728658 | 18:01:35.295424 |
50 | 6 | 17:45:45.729012 | 17:45:45.729022 | 18:01:37.573317 |
The following single row table is the superuser queue leader-node query issued immediately after the 50 queries were issued.
It appears currently no logging of completed leader-node
queries is occurring (it used to) and so all we have is the timestamp
from the timeofday()
function, which was issued in the
query. This is however adequate to show when execution occurred.
timeofday() |
---|
Mon Oct 04 17:45:45.476594 2021 UTC |
The following single row table is the superuser queue worker-node query issued immediately after leader-node query completed. The query was not blocked.
Queue # | Queue Start | Exec Start | timeofday() | Exec End |
---|---|---|---|---|
5 | 17:45:45.736648 | 17:45:45.736655 | Mon Oct 04 17:45:46.532921 2021 UTC | 17:45:46.533540 |
With a single 50 slot queue, we can see all 50 queries queued and began execution at about the same time, 15:49:58, but that they did not in fact actually all execute in parallel.
The first 30 queries ran and completed at about 16:21:50, as can be seen in their “Exec End” timestamps, taking about 32 minutes to run.
The outstanding 20 queries however completed at about 16:42:20 - 21 minutes later.
The explanation here is that the first 30 queries were running while the second 20 queries were not, and then when the first 30 finished, the second 20 queries than actually began to ran, and finished about 20 minutes later, which fits - if 30 queries take about 30 minutes to run, then 20 queries will take about 2/3rds of the time, which is 20 minutes).
This means the start execution time listed in the system tables is basically bogus. It does not mean a query is actually executing.
Returning to the test, once all 50 queries has been issued, a leader-node only query was issued to the superuser queue. This completed immediately.
Then a worker-node query was issued the superuser queue, this at 15:50:02.
This query was blocked until the first batch of 30 queries began to complete, and then it finally was able to complete at 16:21:50 (the time the second batch of 20 queries were able also to begin executing).
This proof is identical to proof #1, except 40 queries are issued rather than 50.
We expect then to see the first 30 execute and when they have completed, the final 10 queries execute, and this what is seen.
As before, the superuser queue leader-only query executed immediately, but the superuser queue work-node-using query was blocked.
This proof has two queues, one of 30 slots and one of 10.
The first queue of 30 slots has 30 identical queries issued to it; once this has been done, 10 identical queries are issued to the second queue.
What we find now is that all these queries actually really do execute in parallel, and that the superuser queries are not blocked.
Identical to proof #3, except now with 20 slots and queries in the second queue, to reach the 50 slot limit.
However, this does not change the results : all queries really did execute in parallel, and the superuser queries were not blocked.
So what do we make of this?
First, it looks like there is for any single queue an actual limit of 30 concurrent queries, regardless of how many slots are configured, and the reporting in the system tables of the timestamp of when a query begins execution is bogus (the systems tables are a disaster area, and this is more of the same).
Regarding this limit, it’s always a good idea to keep queue configuration simple; any fancy queue configuration in my view is never going to be doing what the admin thinks it’s doing. This mitigates in favour of few or even single queues, with lots of slots. Queries which use merge joins, which is to say, well designed queries with matching well designed tables, require practically no memory and so in a situation where these are the queries being issued, you can have lots of slots - you don’t need the usual buckets of memory to make hash joins performant. In this case this 30 query limit could be a problem, which seems a bit cruel - it’s hitting the people who are operating Redshift correctly.
In my experience though, every client I’ve worked with has been operating Redshift incorrectly, and so needs plenty of slot memory to handle their hash joins, but at the same time have no awareness of this need at all, and so they tend to have some sort of semi-randomly selected queue configuration which has never been examined for actual impact or performance, because queries are executing and it all seems to work (despite end-users complaining that performance is poor and erratic).
Second, it looks like (and this is pretty odd) that if there is a single queue which has reached the 30 concurrent query limit, the superuser queue - but not other normal user queues, as we’ve seen in proof #3 - can no longer execute queries which use worker nodes.
I think leader-node only queries continue to execute as normal
because they do not in fact use WLM queues at all, and so do not in fact
pay attention to query_group
.
The superuser queue is there at least in part to allow admin to always be able to issue commands, to recover a cluster, no matter what state the cluster is in, so if this is true, it’s a significant and unexpected issue.
I’ve not yet experimented to find out, but my expectation is that anything relating to queries - enumerating queries, cancelling processes, etc, would be leader-node only, but anything which touches normal Redshift tables in any way, such as dropping tables or truncating them, would not be.
There is a limit of 30 concurrent queries per queue, regardless of the number of slots configured in a queue.
Once any one queue reaches its limit of 30 concurrent queries, the superuser queue is no longer able to execute queries.
However, other user queues are not affected, and continue to accept and execute queries normally.
(Note leader-node only queries continue to execute as normal, I think because they do not use WLM queues and so cannot be blocked by anything going on in WLM.)
When you investigate Redshift, there are always unexpected findings.
STL_QUERY
.Note these results are completely unprocessed; they are a raw dump of the results, so the original, wholly unprocessed data, is available.
{'proofs': {'dc2.large': {2: {'1_queue_40_slots': [[6,
datetime.datetime(2021, 10, 4, 16, 43, 56, 785862),
datetime.datetime(2021, 10, 4, 16, 43, 56, 785943),
datetime.datetime(2021, 10, 4, 16, 43, 56, 785943),
0,
datetime.datetime(2021, 10, 4, 16, 43, 56, 785951),
datetime.datetime(2021, 10, 4, 17, 14, 54, 546982),
1857761031,
datetime.datetime(2021, 10, 4, 17, 14, 54, 546982)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 56, 990706),
datetime.datetime(2021, 10, 4, 16, 43, 56, 990794),
datetime.datetime(2021, 10, 4, 16, 43, 56, 990794),
0,
datetime.datetime(2021, 10, 4, 16, 43, 56, 990802),
datetime.datetime(2021, 10, 4, 17, 15, 13, 130696),
1876139894,
datetime.datetime(2021, 10, 4, 17, 15, 13, 130696)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 56, 932612),
datetime.datetime(2021, 10, 4, 16, 43, 56, 932712),
datetime.datetime(2021, 10, 4, 16, 43, 56, 932712),
0,
datetime.datetime(2021, 10, 4, 16, 43, 56, 932726),
datetime.datetime(2021, 10, 4, 17, 15, 14, 538681),
1877605955,
datetime.datetime(2021, 10, 4, 17, 15, 14, 538681)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 57, 101059),
datetime.datetime(2021, 10, 4, 16, 43, 57, 101137),
datetime.datetime(2021, 10, 4, 16, 43, 57, 101137),
0,
datetime.datetime(2021, 10, 4, 16, 43, 57, 101143),
datetime.datetime(2021, 10, 4, 17, 15, 42, 708895),
1905607752,
datetime.datetime(2021, 10, 4, 17, 15, 42, 708895)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 56, 792098),
datetime.datetime(2021, 10, 4, 16, 43, 56, 792195),
datetime.datetime(2021, 10, 4, 16, 43, 56, 792195),
0,
datetime.datetime(2021, 10, 4, 16, 43, 56, 792205),
datetime.datetime(2021, 10, 4, 17, 15, 43, 101332),
1906309127,
datetime.datetime(2021, 10, 4, 17, 15, 43, 101332)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 57, 85036),
datetime.datetime(2021, 10, 4, 16, 43, 57, 87052),
datetime.datetime(2021, 10, 4, 16, 43, 57, 87052),
0,
datetime.datetime(2021, 10, 4, 16, 43, 57, 87060),
datetime.datetime(2021, 10, 4, 17, 15, 44, 954837),
1907867777,
datetime.datetime(2021, 10, 4, 17, 15, 44, 954837)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 56, 643556),
datetime.datetime(2021, 10, 4, 16, 43, 56, 643655),
datetime.datetime(2021, 10, 4, 16, 43, 56, 643655),
0,
datetime.datetime(2021, 10, 4, 16, 43, 56, 643979),
datetime.datetime(2021, 10, 4, 17, 15, 46, 211009),
1909567030,
datetime.datetime(2021, 10, 4, 17, 15, 46, 211009)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 56, 989952),
datetime.datetime(2021, 10, 4, 16, 43, 56, 990043),
datetime.datetime(2021, 10, 4, 16, 43, 56, 990043),
0,
datetime.datetime(2021, 10, 4, 16, 43, 56, 990052),
datetime.datetime(2021, 10, 4, 17, 15, 46, 500821),
1909510769,
datetime.datetime(2021, 10, 4, 17, 15, 46, 500821)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 56, 997071),
datetime.datetime(2021, 10, 4, 16, 43, 56, 997153),
datetime.datetime(2021, 10, 4, 16, 43, 56, 997153),
0,
datetime.datetime(2021, 10, 4, 16, 43, 56, 997821),
datetime.datetime(2021, 10, 4, 17, 15, 46, 946331),
1909948510,
datetime.datetime(2021, 10, 4, 17, 15, 46, 946331)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 56, 783848),
datetime.datetime(2021, 10, 4, 16, 43, 56, 783937),
datetime.datetime(2021, 10, 4, 16, 43, 56, 783937),
0,
datetime.datetime(2021, 10, 4, 16, 43, 56, 783947),
datetime.datetime(2021, 10, 4, 17, 15, 47, 163977),
1910380030,
datetime.datetime(2021, 10, 4, 17, 15, 47, 163977)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 57, 99476),
datetime.datetime(2021, 10, 4, 16, 43, 57, 99557),
datetime.datetime(2021, 10, 4, 16, 43, 57, 99557),
0,
datetime.datetime(2021, 10, 4, 16, 43, 57, 99564),
datetime.datetime(2021, 10, 4, 17, 15, 47, 438160),
1910338596,
datetime.datetime(2021, 10, 4, 17, 15, 47, 438160)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 57, 90506),
datetime.datetime(2021, 10, 4, 16, 43, 57, 90594),
datetime.datetime(2021, 10, 4, 16, 43, 57, 90594),
0,
datetime.datetime(2021, 10, 4, 16, 43, 57, 90603),
datetime.datetime(2021, 10, 4, 17, 15, 47, 732970),
1910642367,
datetime.datetime(2021, 10, 4, 17, 15, 47, 732970)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 56, 943555),
datetime.datetime(2021, 10, 4, 16, 43, 56, 943640),
datetime.datetime(2021, 10, 4, 16, 43, 56, 943640),
0,
datetime.datetime(2021, 10, 4, 16, 43, 56, 943649),
datetime.datetime(2021, 10, 4, 17, 15, 47, 737157),
1910793508,
datetime.datetime(2021, 10, 4, 17, 15, 47, 737157)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 57, 65441),
datetime.datetime(2021, 10, 4, 16, 43, 57, 65514),
datetime.datetime(2021, 10, 4, 16, 43, 57, 65514),
0,
datetime.datetime(2021, 10, 4, 16, 43, 57, 65521),
datetime.datetime(2021, 10, 4, 17, 15, 47, 739972),
1910674451,
datetime.datetime(2021, 10, 4, 17, 15, 47, 739972)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 57, 77476),
datetime.datetime(2021, 10, 4, 16, 43, 57, 77606),
datetime.datetime(2021, 10, 4, 16, 43, 57, 77606),
0,
datetime.datetime(2021, 10, 4, 16, 43, 57, 77625),
datetime.datetime(2021, 10, 4, 17, 15, 47, 856719),
1910779094,
datetime.datetime(2021, 10, 4, 17, 15, 47, 856719)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 56, 890044),
datetime.datetime(2021, 10, 4, 16, 43, 56, 890170),
datetime.datetime(2021, 10, 4, 16, 43, 56, 890170),
0,
datetime.datetime(2021, 10, 4, 16, 43, 56, 890181),
datetime.datetime(2021, 10, 4, 17, 15, 48, 3832),
1911113651,
datetime.datetime(2021, 10, 4, 17, 15, 48, 3832)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 57, 49344),
datetime.datetime(2021, 10, 4, 16, 43, 57, 49426),
datetime.datetime(2021, 10, 4, 16, 43, 57, 49426),
0,
datetime.datetime(2021, 10, 4, 16, 43, 57, 49434),
datetime.datetime(2021, 10, 4, 17, 15, 48, 910747),
1911861313,
datetime.datetime(2021, 10, 4, 17, 15, 48, 910747)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 56, 641115),
datetime.datetime(2021, 10, 4, 16, 43, 56, 641216),
datetime.datetime(2021, 10, 4, 16, 43, 56, 641216),
0,
datetime.datetime(2021, 10, 4, 16, 43, 56, 641229),
datetime.datetime(2021, 10, 4, 17, 15, 49, 269827),
1912628598,
datetime.datetime(2021, 10, 4, 17, 15, 49, 269827)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 56, 867851),
datetime.datetime(2021, 10, 4, 16, 43, 56, 867961),
datetime.datetime(2021, 10, 4, 16, 43, 56, 867961),
0,
datetime.datetime(2021, 10, 4, 16, 43, 56, 867989),
datetime.datetime(2021, 10, 4, 17, 15, 49, 288471),
1912420482,
datetime.datetime(2021, 10, 4, 17, 15, 49, 288471)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 57, 36486),
datetime.datetime(2021, 10, 4, 16, 43, 57, 36741),
datetime.datetime(2021, 10, 4, 16, 43, 57, 36741),
0,
datetime.datetime(2021, 10, 4, 16, 43, 57, 36749),
datetime.datetime(2021, 10, 4, 17, 15, 51, 362420),
1914325671,
datetime.datetime(2021, 10, 4, 17, 15, 51, 362420)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 56, 631031),
datetime.datetime(2021, 10, 4, 16, 43, 56, 631125),
datetime.datetime(2021, 10, 4, 16, 43, 56, 631125),
0,
datetime.datetime(2021, 10, 4, 16, 43, 56, 631135),
datetime.datetime(2021, 10, 4, 17, 15, 51, 563885),
1914932750,
datetime.datetime(2021, 10, 4, 17, 15, 51, 563885)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 57, 79023),
datetime.datetime(2021, 10, 4, 16, 43, 57, 79126),
datetime.datetime(2021, 10, 4, 16, 43, 57, 79126),
0,
datetime.datetime(2021, 10, 4, 16, 43, 57, 79139),
datetime.datetime(2021, 10, 4, 17, 15, 51, 840308),
1914761169,
datetime.datetime(2021, 10, 4, 17, 15, 51, 840308)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 57, 41024),
datetime.datetime(2021, 10, 4, 16, 43, 57, 41135),
datetime.datetime(2021, 10, 4, 16, 43, 57, 41135),
0,
datetime.datetime(2021, 10, 4, 16, 43, 57, 41149),
datetime.datetime(2021, 10, 4, 17, 15, 51, 958705),
1914917556,
datetime.datetime(2021, 10, 4, 17, 15, 51, 958705)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 57, 21992),
datetime.datetime(2021, 10, 4, 16, 43, 57, 22074),
datetime.datetime(2021, 10, 4, 16, 43, 57, 22074),
0,
datetime.datetime(2021, 10, 4, 16, 43, 57, 22085),
datetime.datetime(2021, 10, 4, 17, 15, 51, 998679),
1914976594,
datetime.datetime(2021, 10, 4, 17, 15, 51, 998679)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 56, 988507),
datetime.datetime(2021, 10, 4, 16, 43, 56, 988643),
datetime.datetime(2021, 10, 4, 16, 43, 56, 988643),
0,
datetime.datetime(2021, 10, 4, 16, 43, 56, 988656),
datetime.datetime(2021, 10, 4, 17, 15, 52, 21174),
1915032518,
datetime.datetime(2021, 10, 4, 17, 15, 52, 21174)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 56, 998276),
datetime.datetime(2021, 10, 4, 16, 43, 56, 998370),
datetime.datetime(2021, 10, 4, 16, 43, 56, 998370),
0,
datetime.datetime(2021, 10, 4, 16, 43, 56, 998380),
datetime.datetime(2021, 10, 4, 17, 15, 52, 77485),
1915079105,
datetime.datetime(2021, 10, 4, 17, 15, 52, 77485)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 57, 25284),
datetime.datetime(2021, 10, 4, 16, 43, 57, 25380),
datetime.datetime(2021, 10, 4, 16, 43, 57, 25380),
0,
datetime.datetime(2021, 10, 4, 16, 43, 57, 25387),
datetime.datetime(2021, 10, 4, 17, 15, 52, 114683),
1915089296,
datetime.datetime(2021, 10, 4, 17, 15, 52, 114683)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 56, 990876),
datetime.datetime(2021, 10, 4, 16, 43, 56, 990945),
datetime.datetime(2021, 10, 4, 16, 43, 56, 990945),
0,
datetime.datetime(2021, 10, 4, 16, 43, 56, 990951),
datetime.datetime(2021, 10, 4, 17, 15, 54, 20131),
1917029180,
datetime.datetime(2021, 10, 4, 17, 15, 54, 20131)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 57, 67647),
datetime.datetime(2021, 10, 4, 16, 43, 57, 67723),
datetime.datetime(2021, 10, 4, 16, 43, 57, 67723),
0,
datetime.datetime(2021, 10, 4, 16, 43, 57, 67731),
datetime.datetime(2021, 10, 4, 17, 15, 54, 785593),
1917717862,
datetime.datetime(2021, 10, 4, 17, 15, 54, 785593)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 57, 63156),
datetime.datetime(2021, 10, 4, 16, 43, 57, 63241),
datetime.datetime(2021, 10, 4, 16, 43, 57, 63241),
0,
datetime.datetime(2021, 10, 4, 16, 43, 57, 63250),
datetime.datetime(2021, 10, 4, 17, 15, 55, 953745),
1918890495,
datetime.datetime(2021, 10, 4, 17, 15, 55, 953745)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 57, 118295),
datetime.datetime(2021, 10, 4, 16, 43, 57, 118359),
datetime.datetime(2021, 10, 4, 16, 43, 57, 118359),
0,
datetime.datetime(2021, 10, 4, 16, 43, 57, 118365),
datetime.datetime(2021, 10, 4, 17, 25, 36, 35945),
2498917580,
datetime.datetime(2021, 10, 4, 17, 25, 36, 35945)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 57, 155608),
datetime.datetime(2021, 10, 4, 16, 43, 57, 155690),
datetime.datetime(2021, 10, 4, 16, 43, 57, 155690),
0,
datetime.datetime(2021, 10, 4, 16, 43, 57, 155698),
datetime.datetime(2021, 10, 4, 17, 25, 45, 103205),
2507947507,
datetime.datetime(2021, 10, 4, 17, 25, 45, 103205)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 57, 121136),
datetime.datetime(2021, 10, 4, 16, 43, 57, 121212),
datetime.datetime(2021, 10, 4, 16, 43, 57, 121212),
0,
datetime.datetime(2021, 10, 4, 16, 43, 57, 121219),
datetime.datetime(2021, 10, 4, 17, 25, 45, 825897),
2508704678,
datetime.datetime(2021, 10, 4, 17, 25, 45, 825897)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 57, 167325),
datetime.datetime(2021, 10, 4, 16, 43, 57, 167407),
datetime.datetime(2021, 10, 4, 16, 43, 57, 167407),
0,
datetime.datetime(2021, 10, 4, 16, 43, 57, 167415),
datetime.datetime(2021, 10, 4, 17, 25, 46, 36079),
2508868664,
datetime.datetime(2021, 10, 4, 17, 25, 46, 36079)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 57, 121896),
datetime.datetime(2021, 10, 4, 16, 43, 57, 121998),
datetime.datetime(2021, 10, 4, 16, 43, 57, 121998),
0,
datetime.datetime(2021, 10, 4, 16, 43, 57, 125695),
datetime.datetime(2021, 10, 4, 17, 25, 48, 425447),
2511299752,
datetime.datetime(2021, 10, 4, 17, 25, 48, 425447)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 57, 146262),
datetime.datetime(2021, 10, 4, 16, 43, 57, 146333),
datetime.datetime(2021, 10, 4, 16, 43, 57, 146333),
0,
datetime.datetime(2021, 10, 4, 16, 43, 57, 146338),
datetime.datetime(2021, 10, 4, 17, 25, 48, 902028),
2511755690,
datetime.datetime(2021, 10, 4, 17, 25, 48, 902028)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 57, 160809),
datetime.datetime(2021, 10, 4, 16, 43, 57, 160894),
datetime.datetime(2021, 10, 4, 16, 43, 57, 160894),
0,
datetime.datetime(2021, 10, 4, 16, 43, 57, 160901),
datetime.datetime(2021, 10, 4, 17, 25, 49, 69143),
2511908242,
datetime.datetime(2021, 10, 4, 17, 25, 49, 69143)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 57, 146086),
datetime.datetime(2021, 10, 4, 16, 43, 57, 146171),
datetime.datetime(2021, 10, 4, 16, 43, 57, 146171),
0,
datetime.datetime(2021, 10, 4, 16, 43, 57, 146180),
datetime.datetime(2021, 10, 4, 17, 25, 49, 130059),
2511983879,
datetime.datetime(2021, 10, 4, 17, 25, 49, 130059)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 57, 160989),
datetime.datetime(2021, 10, 4, 16, 43, 57, 161057),
datetime.datetime(2021, 10, 4, 16, 43, 57, 161057),
0,
datetime.datetime(2021, 10, 4, 16, 43, 57, 161063),
datetime.datetime(2021, 10, 4, 17, 25, 49, 499375),
2512338312,
datetime.datetime(2021, 10, 4, 17, 25, 49, 499375)],
[6,
datetime.datetime(2021, 10, 4, 16, 43, 57, 157894),
datetime.datetime(2021, 10, 4, 16, 43, 57, 157965),
datetime.datetime(2021, 10, 4, 16, 43, 57, 157965),
0,
datetime.datetime(2021, 10, 4, 16, 43, 57, 157972),
datetime.datetime(2021, 10, 4, 17, 25, 49, 911973),
2512754001,
datetime.datetime(2021, 10, 4, 17, 25, 49, 911973)]],
'1_queue_40_slots_superuser_leader_node': [['Mon '
'Oct '
'04 '
'16:44:01.546710 '
'2021 '
'UTC',
1711]],
'1_queue_40_slots_superuser_worker_node': [['Mon '
'Oct '
'04 '
'17:14:54.548186 '
'2021 '
'UTC',
523976]],
'1_queue_40_slots_superuser_worker_node_wlm_query': [[5,
datetime.datetime(2021, 10, 4, 16, 44, 1, 594699),
datetime.datetime(2021, 10, 4, 16, 44, 1, 594764),
datetime.datetime(2021, 10, 4, 16, 44, 1, 594764),
0,
datetime.datetime(2021, 10, 4, 16, 44, 1, 594776),
datetime.datetime(2021, 10, 4, 17, 14, 54, 548745),
1852953969,
datetime.datetime(2021, 10, 4, 17, 14, 54, 548745)]],
'1_queue_40_slots_wlm_config': [[1, 1],
[2, 1],
[3, 2],
[4, 1],
[5, 1],
[6, 50],
[15, 0]],
'1_queue_50_slots': [[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 127689),
datetime.datetime(2021, 10, 4, 15, 49, 58, 127771),
datetime.datetime(2021, 10, 4, 15, 49, 58, 127771),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 127780),
datetime.datetime(2021, 10, 4, 16, 21, 50, 613780),
1912486000,
datetime.datetime(2021, 10, 4, 16, 21, 50, 613780)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 62374),
datetime.datetime(2021, 10, 4, 15, 49, 58, 70566),
datetime.datetime(2021, 10, 4, 15, 49, 58, 70566),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 70609),
datetime.datetime(2021, 10, 4, 16, 21, 50, 615748),
1912545139,
datetime.datetime(2021, 10, 4, 16, 21, 50, 615748)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 120387),
datetime.datetime(2021, 10, 4, 15, 49, 58, 120498),
datetime.datetime(2021, 10, 4, 15, 49, 58, 120498),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 120510),
datetime.datetime(2021, 10, 4, 16, 21, 50, 617066),
1912496556,
datetime.datetime(2021, 10, 4, 16, 21, 50, 617066)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 173375),
datetime.datetime(2021, 10, 4, 15, 49, 58, 210868),
datetime.datetime(2021, 10, 4, 15, 49, 58, 210868),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 210878),
datetime.datetime(2021, 10, 4, 16, 21, 50, 642211),
1912431333,
datetime.datetime(2021, 10, 4, 16, 21, 50, 642211)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 160567),
datetime.datetime(2021, 10, 4, 15, 49, 58, 160839),
datetime.datetime(2021, 10, 4, 15, 49, 58, 160839),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 160847),
datetime.datetime(2021, 10, 4, 16, 21, 50, 671250),
1912510403,
datetime.datetime(2021, 10, 4, 16, 21, 50, 671250)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 62376),
datetime.datetime(2021, 10, 4, 15, 49, 58, 62473),
datetime.datetime(2021, 10, 4, 15, 49, 58, 62473),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 62483),
datetime.datetime(2021, 10, 4, 16, 21, 50, 710787),
1912648304,
datetime.datetime(2021, 10, 4, 16, 21, 50, 710787)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 89462),
datetime.datetime(2021, 10, 4, 15, 49, 58, 89557),
datetime.datetime(2021, 10, 4, 15, 49, 58, 89557),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 89566),
datetime.datetime(2021, 10, 4, 16, 21, 50, 755797),
1912666231,
datetime.datetime(2021, 10, 4, 16, 21, 50, 755797)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 57, 835241),
datetime.datetime(2021, 10, 4, 15, 49, 57, 835326),
datetime.datetime(2021, 10, 4, 15, 49, 57, 835326),
0,
datetime.datetime(2021, 10, 4, 15, 49, 57, 835334),
datetime.datetime(2021, 10, 4, 16, 21, 50, 770947),
1912935613,
datetime.datetime(2021, 10, 4, 16, 21, 50, 770947)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 232141),
datetime.datetime(2021, 10, 4, 15, 49, 58, 232241),
datetime.datetime(2021, 10, 4, 15, 49, 58, 232241),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 232252),
datetime.datetime(2021, 10, 4, 16, 21, 50, 823570),
1912591318,
datetime.datetime(2021, 10, 4, 16, 21, 50, 823570)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 132139),
datetime.datetime(2021, 10, 4, 15, 49, 58, 132214),
datetime.datetime(2021, 10, 4, 15, 49, 58, 132214),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 132222),
datetime.datetime(2021, 10, 4, 16, 21, 50, 855381),
1912723159,
datetime.datetime(2021, 10, 4, 16, 21, 50, 855381)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 86969),
datetime.datetime(2021, 10, 4, 15, 49, 58, 87055),
datetime.datetime(2021, 10, 4, 15, 49, 58, 87055),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 87063),
datetime.datetime(2021, 10, 4, 16, 21, 50, 887268),
1912800205,
datetime.datetime(2021, 10, 4, 16, 21, 50, 887268)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 51377),
datetime.datetime(2021, 10, 4, 15, 49, 58, 51477),
datetime.datetime(2021, 10, 4, 15, 49, 58, 51477),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 51487),
datetime.datetime(2021, 10, 4, 16, 21, 53, 884859),
1915833372,
datetime.datetime(2021, 10, 4, 16, 21, 53, 884859)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 259105),
datetime.datetime(2021, 10, 4, 15, 49, 58, 259194),
datetime.datetime(2021, 10, 4, 15, 49, 58, 259194),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 259203),
datetime.datetime(2021, 10, 4, 16, 21, 54, 227551),
1915968348,
datetime.datetime(2021, 10, 4, 16, 21, 54, 227551)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 117237),
datetime.datetime(2021, 10, 4, 15, 49, 58, 117327),
datetime.datetime(2021, 10, 4, 15, 49, 58, 117327),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 117336),
datetime.datetime(2021, 10, 4, 16, 21, 54, 585056),
1916467720,
datetime.datetime(2021, 10, 4, 16, 21, 54, 585056)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 263603),
datetime.datetime(2021, 10, 4, 15, 49, 58, 263676),
datetime.datetime(2021, 10, 4, 15, 49, 58, 263676),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 263683),
datetime.datetime(2021, 10, 4, 16, 21, 54, 848468),
1916584785,
datetime.datetime(2021, 10, 4, 16, 21, 54, 848468)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 122296),
datetime.datetime(2021, 10, 4, 15, 49, 58, 122405),
datetime.datetime(2021, 10, 4, 15, 49, 58, 122405),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 158070),
datetime.datetime(2021, 10, 4, 16, 21, 55, 96476),
1916938406,
datetime.datetime(2021, 10, 4, 16, 21, 55, 96476)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 70968),
datetime.datetime(2021, 10, 4, 15, 49, 58, 71046),
datetime.datetime(2021, 10, 4, 15, 49, 58, 71046),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 71056),
datetime.datetime(2021, 10, 4, 16, 21, 56, 54904),
1917983848,
datetime.datetime(2021, 10, 4, 16, 21, 56, 54904)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 57, 829505),
datetime.datetime(2021, 10, 4, 15, 49, 57, 829600),
datetime.datetime(2021, 10, 4, 15, 49, 57, 829600),
0,
datetime.datetime(2021, 10, 4, 15, 49, 57, 829612),
datetime.datetime(2021, 10, 4, 16, 21, 56, 435633),
1918606021,
datetime.datetime(2021, 10, 4, 16, 21, 56, 435633)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 184640),
datetime.datetime(2021, 10, 4, 15, 49, 58, 184733),
datetime.datetime(2021, 10, 4, 15, 49, 58, 184733),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 184741),
datetime.datetime(2021, 10, 4, 16, 21, 58, 94943),
1919910202,
datetime.datetime(2021, 10, 4, 16, 21, 58, 94943)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 274622),
datetime.datetime(2021, 10, 4, 15, 49, 58, 274706),
datetime.datetime(2021, 10, 4, 15, 49, 58, 274706),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 274714),
datetime.datetime(2021, 10, 4, 16, 21, 58, 710918),
1920436204,
datetime.datetime(2021, 10, 4, 16, 21, 58, 710918)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 85958),
datetime.datetime(2021, 10, 4, 15, 49, 58, 86042),
datetime.datetime(2021, 10, 4, 15, 49, 58, 86042),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 86050),
datetime.datetime(2021, 10, 4, 16, 21, 59, 336118),
1921250068,
datetime.datetime(2021, 10, 4, 16, 21, 59, 336118)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 183170),
datetime.datetime(2021, 10, 4, 15, 49, 58, 183266),
datetime.datetime(2021, 10, 4, 15, 49, 58, 183266),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 183278),
datetime.datetime(2021, 10, 4, 16, 21, 59, 395178),
1921211900,
datetime.datetime(2021, 10, 4, 16, 21, 59, 395178)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 181252),
datetime.datetime(2021, 10, 4, 15, 49, 58, 181344),
datetime.datetime(2021, 10, 4, 15, 49, 58, 181344),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 181354),
datetime.datetime(2021, 10, 4, 16, 21, 59, 635844),
1921454490,
datetime.datetime(2021, 10, 4, 16, 21, 59, 635844)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 44908),
datetime.datetime(2021, 10, 4, 15, 49, 58, 45031),
datetime.datetime(2021, 10, 4, 15, 49, 58, 45031),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 45051),
datetime.datetime(2021, 10, 4, 16, 22, 0, 307921),
1922262870,
datetime.datetime(2021, 10, 4, 16, 22, 0, 307921)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 132611),
datetime.datetime(2021, 10, 4, 15, 49, 58, 132691),
datetime.datetime(2021, 10, 4, 15, 49, 58, 132691),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 132699),
datetime.datetime(2021, 10, 4, 16, 22, 0, 427576),
1922294877,
datetime.datetime(2021, 10, 4, 16, 22, 0, 427576)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 227089),
datetime.datetime(2021, 10, 4, 15, 49, 58, 227170),
datetime.datetime(2021, 10, 4, 15, 49, 58, 227170),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 227183),
datetime.datetime(2021, 10, 4, 16, 22, 0, 658998),
1922431815,
datetime.datetime(2021, 10, 4, 16, 22, 0, 658998)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 219748),
datetime.datetime(2021, 10, 4, 15, 49, 58, 219830),
datetime.datetime(2021, 10, 4, 15, 49, 58, 219830),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 219837),
datetime.datetime(2021, 10, 4, 16, 22, 1, 937114),
1923717277,
datetime.datetime(2021, 10, 4, 16, 22, 1, 937114)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 144992),
datetime.datetime(2021, 10, 4, 15, 49, 58, 190592),
datetime.datetime(2021, 10, 4, 15, 49, 58, 190592),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 190610),
datetime.datetime(2021, 10, 4, 16, 22, 3, 198462),
1925007852,
datetime.datetime(2021, 10, 4, 16, 22, 3, 198462)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 78144),
datetime.datetime(2021, 10, 4, 15, 49, 58, 78237),
datetime.datetime(2021, 10, 4, 15, 49, 58, 78237),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 78245),
datetime.datetime(2021, 10, 4, 16, 22, 3, 220855),
1925142610,
datetime.datetime(2021, 10, 4, 16, 22, 3, 220855)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 200513),
datetime.datetime(2021, 10, 4, 15, 49, 58, 200588),
datetime.datetime(2021, 10, 4, 15, 49, 58, 200588),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 200595),
datetime.datetime(2021, 10, 4, 16, 22, 4, 820606),
1926620011,
datetime.datetime(2021, 10, 4, 16, 22, 4, 820606)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 172546),
datetime.datetime(2021, 10, 4, 15, 49, 58, 243559),
datetime.datetime(2021, 10, 4, 15, 49, 58, 243559),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 243568),
datetime.datetime(2021, 10, 4, 16, 42, 11, 351617),
3133108049,
datetime.datetime(2021, 10, 4, 16, 42, 11, 351617)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 196150),
datetime.datetime(2021, 10, 4, 15, 49, 58, 196226),
datetime.datetime(2021, 10, 4, 15, 49, 58, 196226),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 196232),
datetime.datetime(2021, 10, 4, 16, 42, 11, 936405),
3133740173,
datetime.datetime(2021, 10, 4, 16, 42, 11, 936405)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 215248),
datetime.datetime(2021, 10, 4, 15, 49, 58, 215345),
datetime.datetime(2021, 10, 4, 15, 49, 58, 215345),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 215358),
datetime.datetime(2021, 10, 4, 16, 42, 18, 99604),
3139884246,
datetime.datetime(2021, 10, 4, 16, 42, 18, 99604)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 213473),
datetime.datetime(2021, 10, 4, 15, 49, 58, 213584),
datetime.datetime(2021, 10, 4, 15, 49, 58, 213584),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 213594),
datetime.datetime(2021, 10, 4, 16, 42, 18, 491249),
3140277655,
datetime.datetime(2021, 10, 4, 16, 42, 18, 491249)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 217097),
datetime.datetime(2021, 10, 4, 15, 49, 58, 217188),
datetime.datetime(2021, 10, 4, 15, 49, 58, 217188),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 217199),
datetime.datetime(2021, 10, 4, 16, 42, 18, 510718),
3140293519,
datetime.datetime(2021, 10, 4, 16, 42, 18, 510718)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 294375),
datetime.datetime(2021, 10, 4, 15, 49, 58, 294468),
datetime.datetime(2021, 10, 4, 15, 49, 58, 294468),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 294476),
datetime.datetime(2021, 10, 4, 16, 42, 20, 603540),
3142309064,
datetime.datetime(2021, 10, 4, 16, 42, 20, 603540)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 282751),
datetime.datetime(2021, 10, 4, 15, 49, 58, 282827),
datetime.datetime(2021, 10, 4, 15, 49, 58, 282827),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 282834),
datetime.datetime(2021, 10, 4, 16, 42, 20, 604723),
3142321889,
datetime.datetime(2021, 10, 4, 16, 42, 20, 604723)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 278022),
datetime.datetime(2021, 10, 4, 15, 49, 58, 278099),
datetime.datetime(2021, 10, 4, 15, 49, 58, 278099),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 278107),
datetime.datetime(2021, 10, 4, 16, 42, 21, 234185),
3142956078,
datetime.datetime(2021, 10, 4, 16, 42, 21, 234185)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 215880),
datetime.datetime(2021, 10, 4, 15, 49, 58, 215968),
datetime.datetime(2021, 10, 4, 15, 49, 58, 215968),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 215977),
datetime.datetime(2021, 10, 4, 16, 42, 21, 551156),
3143335179,
datetime.datetime(2021, 10, 4, 16, 42, 21, 551156)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 154342),
datetime.datetime(2021, 10, 4, 15, 49, 58, 154419),
datetime.datetime(2021, 10, 4, 15, 49, 58, 154419),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 154426),
datetime.datetime(2021, 10, 4, 16, 42, 21, 603266),
3143448840,
datetime.datetime(2021, 10, 4, 16, 42, 21, 603266)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 224690),
datetime.datetime(2021, 10, 4, 15, 49, 58, 224768),
datetime.datetime(2021, 10, 4, 15, 49, 58, 224768),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 224776),
datetime.datetime(2021, 10, 4, 16, 42, 21, 651115),
3143426339,
datetime.datetime(2021, 10, 4, 16, 42, 21, 651115)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 287779),
datetime.datetime(2021, 10, 4, 15, 49, 58, 287855),
datetime.datetime(2021, 10, 4, 15, 49, 58, 287855),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 287862),
datetime.datetime(2021, 10, 4, 16, 42, 23, 474200),
3145186338,
datetime.datetime(2021, 10, 4, 16, 42, 23, 474200)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 425385),
datetime.datetime(2021, 10, 4, 15, 49, 58, 425462),
datetime.datetime(2021, 10, 4, 15, 49, 58, 425462),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 425469),
datetime.datetime(2021, 10, 4, 16, 42, 23, 685237),
3145259768,
datetime.datetime(2021, 10, 4, 16, 42, 23, 685237)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 261909),
datetime.datetime(2021, 10, 4, 15, 49, 58, 262150),
datetime.datetime(2021, 10, 4, 15, 49, 58, 262150),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 262157),
datetime.datetime(2021, 10, 4, 16, 42, 23, 934793),
3145672636,
datetime.datetime(2021, 10, 4, 16, 42, 23, 934793)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 278199),
datetime.datetime(2021, 10, 4, 15, 49, 58, 278275),
datetime.datetime(2021, 10, 4, 15, 49, 58, 278275),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 278281),
datetime.datetime(2021, 10, 4, 16, 42, 24, 114984),
3145836703,
datetime.datetime(2021, 10, 4, 16, 42, 24, 114984)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 262581),
datetime.datetime(2021, 10, 4, 15, 49, 58, 262670),
datetime.datetime(2021, 10, 4, 15, 49, 58, 262670),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 262681),
datetime.datetime(2021, 10, 4, 16, 42, 24, 856536),
3146593855,
datetime.datetime(2021, 10, 4, 16, 42, 24, 856536)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 354715),
datetime.datetime(2021, 10, 4, 15, 49, 58, 354781),
datetime.datetime(2021, 10, 4, 15, 49, 58, 354781),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 354789),
datetime.datetime(2021, 10, 4, 16, 42, 26, 5504),
3147650715,
datetime.datetime(2021, 10, 4, 16, 42, 26, 5504)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 416325),
datetime.datetime(2021, 10, 4, 15, 49, 58, 416414),
datetime.datetime(2021, 10, 4, 15, 49, 58, 416414),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 416423),
datetime.datetime(2021, 10, 4, 16, 42, 26, 258593),
3147842170,
datetime.datetime(2021, 10, 4, 16, 42, 26, 258593)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 419837),
datetime.datetime(2021, 10, 4, 15, 49, 58, 419910),
datetime.datetime(2021, 10, 4, 15, 49, 58, 419910),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 419917),
datetime.datetime(2021, 10, 4, 16, 42, 26, 276310),
3147856393,
datetime.datetime(2021, 10, 4, 16, 42, 26, 276310)],
[6,
datetime.datetime(2021, 10, 4, 15, 49, 58, 416499),
datetime.datetime(2021, 10, 4, 15, 49, 58, 416569),
datetime.datetime(2021, 10, 4, 15, 49, 58, 416569),
0,
datetime.datetime(2021, 10, 4, 15, 49, 58, 416576),
datetime.datetime(2021, 10, 4, 16, 42, 26, 483855),
3148067279,
datetime.datetime(2021, 10, 4, 16, 42, 26, 483855)]],
'1_queue_50_slots_superuser_leader_node': [['Mon '
'Oct '
'04 '
'15:50:02.432721 '
'2021 '
'UTC',
1711]],
'1_queue_50_slots_superuser_worker_node': [['Mon '
'Oct '
'04 '
'16:21:50.898244 '
'2021 '
'UTC',
523976]],
'1_queue_50_slots_superuser_worker_node_wlm_query': [[5,
datetime.datetime(2021, 10, 4, 15, 50, 2, 481213),
datetime.datetime(2021, 10, 4, 15, 50, 2, 481307),
datetime.datetime(2021, 10, 4, 15, 50, 2, 481307),
0,
datetime.datetime(2021, 10, 4, 15, 50, 2, 481321),
datetime.datetime(2021, 10, 4, 16, 21, 50, 898769),
1908417448,
datetime.datetime(2021, 10, 4, 16, 21, 50, 898769)]],
'1_queue_50_slots_wlm_config': [[1, 1],
[2, 1],
[3, 2],
[4, 1],
[5, 1],
[6, 50],
[15, 0]],
'2_queues_30/10_slots': [[7,
datetime.datetime(2021, 10, 4, 17, 29, 10, 116574),
datetime.datetime(2021, 10, 4, 17, 29, 10, 116657),
datetime.datetime(2021, 10, 4, 17, 29, 10, 116657),
0,
datetime.datetime(2021, 10, 4, 17, 29, 10, 116666),
datetime.datetime(2021, 10, 4, 17, 41, 57, 179292),
767062626,
datetime.datetime(2021, 10, 4, 17, 41, 57, 179292)],
[7,
datetime.datetime(2021, 10, 4, 17, 29, 10, 152508),
datetime.datetime(2021, 10, 4, 17, 29, 10, 152597),
datetime.datetime(2021, 10, 4, 17, 29, 10, 152597),
0,
datetime.datetime(2021, 10, 4, 17, 29, 10, 152608),
datetime.datetime(2021, 10, 4, 17, 41, 49, 509474),
759356866,
datetime.datetime(2021, 10, 4, 17, 41, 49, 509474)],
[7,
datetime.datetime(2021, 10, 4, 17, 29, 10, 159847),
datetime.datetime(2021, 10, 4, 17, 29, 10, 159927),
datetime.datetime(2021, 10, 4, 17, 29, 10, 159927),
0,
datetime.datetime(2021, 10, 4, 17, 29, 10, 159936),
datetime.datetime(2021, 10, 4, 17, 41, 48, 999495),
758839559,
datetime.datetime(2021, 10, 4, 17, 41, 48, 999495)],
[7,
datetime.datetime(2021, 10, 4, 17, 29, 10, 210285),
datetime.datetime(2021, 10, 4, 17, 29, 10, 210375),
datetime.datetime(2021, 10, 4, 17, 29, 10, 210375),
0,
datetime.datetime(2021, 10, 4, 17, 29, 10, 210384),
datetime.datetime(2021, 10, 4, 17, 41, 49, 683588),
759473204,
datetime.datetime(2021, 10, 4, 17, 41, 49, 683588)],
[7,
datetime.datetime(2021, 10, 4, 17, 29, 10, 213110),
datetime.datetime(2021, 10, 4, 17, 29, 10, 213187),
datetime.datetime(2021, 10, 4, 17, 29, 10, 213187),
0,
datetime.datetime(2021, 10, 4, 17, 29, 10, 213194),
datetime.datetime(2021, 10, 4, 17, 41, 58, 276473),
768063279,
datetime.datetime(2021, 10, 4, 17, 41, 58, 276473)],
[7,
datetime.datetime(2021, 10, 4, 17, 29, 10, 235837),
datetime.datetime(2021, 10, 4, 17, 29, 10, 235929),
datetime.datetime(2021, 10, 4, 17, 29, 10, 235929),
0,
datetime.datetime(2021, 10, 4, 17, 29, 10, 235939),
datetime.datetime(2021, 10, 4, 17, 41, 58, 284404),
768048465,
datetime.datetime(2021, 10, 4, 17, 41, 58, 284404)],
[7,
datetime.datetime(2021, 10, 4, 17, 29, 10, 254358),
datetime.datetime(2021, 10, 4, 17, 29, 10, 254439),
datetime.datetime(2021, 10, 4, 17, 29, 10, 254439),
0,
datetime.datetime(2021, 10, 4, 17, 29, 10, 254448),
datetime.datetime(2021, 10, 4, 17, 41, 50, 98457),
759844009,
datetime.datetime(2021, 10, 4, 17, 41, 50, 98457)],
[7,
datetime.datetime(2021, 10, 4, 17, 29, 10, 296593),
datetime.datetime(2021, 10, 4, 17, 29, 10, 296682),
datetime.datetime(2021, 10, 4, 17, 29, 10, 296682),
0,
datetime.datetime(2021, 10, 4, 17, 29, 10, 296692),
datetime.datetime(2021, 10, 4, 17, 41, 49, 919376),
759622684,
datetime.datetime(2021, 10, 4, 17, 41, 49, 919376)],
[7,
datetime.datetime(2021, 10, 4, 17, 29, 10, 324485),
datetime.datetime(2021, 10, 4, 17, 29, 10, 324581),
datetime.datetime(2021, 10, 4, 17, 29, 10, 324581),
0,
datetime.datetime(2021, 10, 4, 17, 29, 10, 324590),
datetime.datetime(2021, 10, 4, 17, 41, 58, 246735),
767922145,
datetime.datetime(2021, 10, 4, 17, 41, 58, 246735)],
[7,
datetime.datetime(2021, 10, 4, 17, 29, 10, 328823),
datetime.datetime(2021, 10, 4, 17, 29, 10, 328897),
datetime.datetime(2021, 10, 4, 17, 29, 10, 328897),
0,
datetime.datetime(2021, 10, 4, 17, 29, 10, 328904),
datetime.datetime(2021, 10, 4, 17, 41, 58, 341320),
768012416,
datetime.datetime(2021, 10, 4, 17, 41, 58, 341320)],
[7,
datetime.datetime(2021, 10, 4, 17, 29, 10, 358000),
datetime.datetime(2021, 10, 4, 17, 29, 10, 358083),
datetime.datetime(2021, 10, 4, 17, 29, 10, 358083),
0,
datetime.datetime(2021, 10, 4, 17, 29, 10, 358092),
datetime.datetime(2021, 10, 4, 17, 41, 50, 165056),
759806964,
datetime.datetime(2021, 10, 4, 17, 41, 50, 165056)],
[7,
datetime.datetime(2021, 10, 4, 17, 29, 10, 363567),
datetime.datetime(2021, 10, 4, 17, 29, 10, 363645),
datetime.datetime(2021, 10, 4, 17, 29, 10, 363645),
0,
datetime.datetime(2021, 10, 4, 17, 29, 10, 363653),
datetime.datetime(2021, 10, 4, 17, 41, 50, 77641),
759713988,
datetime.datetime(2021, 10, 4, 17, 41, 50, 77641)],
[7,
datetime.datetime(2021, 10, 4, 17, 29, 10, 365384),
datetime.datetime(2021, 10, 4, 17, 29, 10, 365459),
datetime.datetime(2021, 10, 4, 17, 29, 10, 365459),
0,
datetime.datetime(2021, 10, 4, 17, 29, 10, 365467),
datetime.datetime(2021, 10, 4, 17, 41, 58, 239877),
767874410,
datetime.datetime(2021, 10, 4, 17, 41, 58, 239877)],
[7,
datetime.datetime(2021, 10, 4, 17, 29, 10, 368639),
datetime.datetime(2021, 10, 4, 17, 29, 10, 368713),
datetime.datetime(2021, 10, 4, 17, 29, 10, 368713),
0,
datetime.datetime(2021, 10, 4, 17, 29, 10, 368720),
datetime.datetime(2021, 10, 4, 17, 41, 50, 164706),
759795986,
datetime.datetime(2021, 10, 4, 17, 41, 50, 164706)],
[7,
datetime.datetime(2021, 10, 4, 17, 29, 10, 411289),
datetime.datetime(2021, 10, 4, 17, 29, 10, 411387),
datetime.datetime(2021, 10, 4, 17, 29, 10, 411387),
0,
datetime.datetime(2021, 10, 4, 17, 29, 10, 411397),
datetime.datetime(2021, 10, 4, 17, 41, 50, 118964),
759707567,
datetime.datetime(2021, 10, 4, 17, 41, 50, 118964)],
[7,
datetime.datetime(2021, 10, 4, 17, 29, 10, 411669),
datetime.datetime(2021, 10, 4, 17, 29, 10, 411742),
datetime.datetime(2021, 10, 4, 17, 29, 10, 411742),
0,
datetime.datetime(2021, 10, 4, 17, 29, 10, 411748),
datetime.datetime(2021, 10, 4, 17, 41, 58, 312306),
767900558,
datetime.datetime(2021, 10, 4, 17, 41, 58, 312306)],
[7,
datetime.datetime(2021, 10, 4, 17, 29, 10, 417331),
datetime.datetime(2021, 10, 4, 17, 29, 10, 417405),
datetime.datetime(2021, 10, 4, 17, 29, 10, 417405),
0,
datetime.datetime(2021, 10, 4, 17, 29, 10, 417412),
datetime.datetime(2021, 10, 4, 17, 41, 52, 788602),
762371190,
datetime.datetime(2021, 10, 4, 17, 41, 52, 788602)],
[7,
datetime.datetime(2021, 10, 4, 17, 29, 10, 448809),
datetime.datetime(2021, 10, 4, 17, 29, 10, 448896),
datetime.datetime(2021, 10, 4, 17, 29, 10, 448896),
0,
datetime.datetime(2021, 10, 4, 17, 29, 10, 448909),
datetime.datetime(2021, 10, 4, 17, 41, 58, 340954),
767892045,
datetime.datetime(2021, 10, 4, 17, 41, 58, 340954)],
[7,
datetime.datetime(2021, 10, 4, 17, 29, 10, 449139),
datetime.datetime(2021, 10, 4, 17, 29, 10, 449213),
datetime.datetime(2021, 10, 4, 17, 29, 10, 449213),
0,
datetime.datetime(2021, 10, 4, 17, 29, 10, 449220),
datetime.datetime(2021, 10, 4, 17, 41, 58, 366137),
767916917,
datetime.datetime(2021, 10, 4, 17, 41, 58, 366137)],
[7,
datetime.datetime(2021, 10, 4, 17, 29, 10, 450950),
datetime.datetime(2021, 10, 4, 17, 29, 10, 451035),
datetime.datetime(2021, 10, 4, 17, 29, 10, 451035),
0,
datetime.datetime(2021, 10, 4, 17, 29, 10, 451046),
datetime.datetime(2021, 10, 4, 17, 41, 58, 266653),
767815607,
datetime.datetime(2021, 10, 4, 17, 41, 58, 266653)],
[7,
datetime.datetime(2021, 10, 4, 17, 29, 10, 452734),
datetime.datetime(2021, 10, 4, 17, 29, 10, 452813),
datetime.datetime(2021, 10, 4, 17, 29, 10, 452813),
0,
datetime.datetime(2021, 10, 4, 17, 29, 10, 452823),
datetime.datetime(2021, 10, 4, 17, 41, 58, 129337),
767676514,
datetime.datetime(2021, 10, 4, 17, 41, 58, 129337)],
[7,
datetime.datetime(2021, 10, 4, 17, 29, 10, 461287),
datetime.datetime(2021, 10, 4, 17, 29, 10, 461381),
datetime.datetime(2021, 10, 4, 17, 29, 10, 461381),
0,
datetime.datetime(2021, 10, 4, 17, 29, 10, 461389),
datetime.datetime(2021, 10, 4, 17, 41, 58, 257869),
767796480,
datetime.datetime(2021, 10, 4, 17, 41, 58, 257869)],
[7,
datetime.datetime(2021, 10, 4, 17, 29, 10, 461464),
datetime.datetime(2021, 10, 4, 17, 29, 10, 461546),
datetime.datetime(2021, 10, 4, 17, 29, 10, 461546),
0,
datetime.datetime(2021, 10, 4, 17, 29, 10, 461554),
datetime.datetime(2021, 10, 4, 17, 41, 50, 103795),
759642241,
datetime.datetime(2021, 10, 4, 17, 41, 50, 103795)],
[7,
datetime.datetime(2021, 10, 4, 17, 29, 10, 462557),
datetime.datetime(2021, 10, 4, 17, 29, 10, 462643),
datetime.datetime(2021, 10, 4, 17, 29, 10, 462643),
0,
datetime.datetime(2021, 10, 4, 17, 29, 10, 462651),
datetime.datetime(2021, 10, 4, 17, 41, 58, 252911),
767790260,
datetime.datetime(2021, 10, 4, 17, 41, 58, 252911)],
[7,
datetime.datetime(2021, 10, 4, 17, 29, 10, 488961),
datetime.datetime(2021, 10, 4, 17, 29, 10, 489043),
datetime.datetime(2021, 10, 4, 17, 29, 10, 489043),
0,
datetime.datetime(2021, 10, 4, 17, 29, 10, 489051),
datetime.datetime(2021, 10, 4, 17, 41, 55, 676683),
765187632,
datetime.datetime(2021, 10, 4, 17, 41, 55, 676683)],
[7,
datetime.datetime(2021, 10, 4, 17, 29, 10, 508157),
datetime.datetime(2021, 10, 4, 17, 29, 10, 508213),
datetime.datetime(2021, 10, 4, 17, 29, 10, 508213),
0,
datetime.datetime(2021, 10, 4, 17, 29, 10, 508222),
datetime.datetime(2021, 10, 4, 17, 41, 52, 912922),
762404700,
datetime.datetime(2021, 10, 4, 17, 41, 52, 912922)],
[7,
datetime.datetime(2021, 10, 4, 17, 29, 10, 524551),
datetime.datetime(2021, 10, 4, 17, 29, 10, 524630),
datetime.datetime(2021, 10, 4, 17, 29, 10, 524630),
0,
datetime.datetime(2021, 10, 4, 17, 29, 10, 524637),
datetime.datetime(2021, 10, 4, 17, 41, 55, 929408),
765404771,
datetime.datetime(2021, 10, 4, 17, 41, 55, 929408)],
[7,
datetime.datetime(2021, 10, 4, 17, 29, 10, 525250),
datetime.datetime(2021, 10, 4, 17, 29, 10, 525336),
datetime.datetime(2021, 10, 4, 17, 29, 10, 525336),
0,
datetime.datetime(2021, 10, 4, 17, 29, 10, 525354),
datetime.datetime(2021, 10, 4, 17, 41, 52, 15737),
761490383,
datetime.datetime(2021, 10, 4, 17, 41, 52, 15737)],
[7,
datetime.datetime(2021, 10, 4, 17, 29, 10, 584839),
datetime.datetime(2021, 10, 4, 17, 29, 10, 584906),
datetime.datetime(2021, 10, 4, 17, 29, 10, 584906),
0,
datetime.datetime(2021, 10, 4, 17, 29, 10, 584913),
datetime.datetime(2021, 10, 4, 17, 41, 50, 928450),
760343537,
datetime.datetime(2021, 10, 4, 17, 41, 50, 928450)],
[7,
datetime.datetime(2021, 10, 4, 17, 29, 10, 590172),
datetime.datetime(2021, 10, 4, 17, 29, 10, 590228),
datetime.datetime(2021, 10, 4, 17, 29, 10, 590228),
0,
datetime.datetime(2021, 10, 4, 17, 29, 10, 590234),
datetime.datetime(2021, 10, 4, 17, 41, 51, 585367),
760995133,
datetime.datetime(2021, 10, 4, 17, 41, 51, 585367)],
[6,
datetime.datetime(2021, 10, 4, 17, 29, 15, 137374),
datetime.datetime(2021, 10, 4, 17, 29, 15, 139446),
datetime.datetime(2021, 10, 4, 17, 29, 15, 139446),
0,
datetime.datetime(2021, 10, 4, 17, 29, 15, 139459),
datetime.datetime(2021, 10, 4, 17, 41, 58, 368530),
763229071,
datetime.datetime(2021, 10, 4, 17, 41, 58, 368530)],
[6,
datetime.datetime(2021, 10, 4, 17, 29, 15, 140697),
datetime.datetime(2021, 10, 4, 17, 29, 15, 146818),
datetime.datetime(2021, 10, 4, 17, 29, 15, 146818),
0,
datetime.datetime(2021, 10, 4, 17, 29, 15, 146829),
datetime.datetime(2021, 10, 4, 17, 41, 56, 844642),
761697813,
datetime.datetime(2021, 10, 4, 17, 41, 56, 844642)],
[6,
datetime.datetime(2021, 10, 4, 17, 29, 15, 144178),
datetime.datetime(2021, 10, 4, 17, 29, 15, 149278),
datetime.datetime(2021, 10, 4, 17, 29, 15, 149278),
0,
datetime.datetime(2021, 10, 4, 17, 29, 15, 149286),
datetime.datetime(2021, 10, 4, 17, 41, 58, 521654),
763372368,
datetime.datetime(2021, 10, 4, 17, 41, 58, 521654)],
[6,
datetime.datetime(2021, 10, 4, 17, 29, 15, 146748),
datetime.datetime(2021, 10, 4, 17, 29, 15, 151224),
datetime.datetime(2021, 10, 4, 17, 29, 15, 151224),
0,
datetime.datetime(2021, 10, 4, 17, 29, 15, 151233),
datetime.datetime(2021, 10, 4, 17, 41, 58, 817737),
763666504,
datetime.datetime(2021, 10, 4, 17, 41, 58, 817737)],
[6,
datetime.datetime(2021, 10, 4, 17, 29, 15, 146527),
datetime.datetime(2021, 10, 4, 17, 29, 15, 151487),
datetime.datetime(2021, 10, 4, 17, 29, 15, 151487),
0,
datetime.datetime(2021, 10, 4, 17, 29, 15, 151497),
datetime.datetime(2021, 10, 4, 17, 41, 58, 370361),
763218864,
datetime.datetime(2021, 10, 4, 17, 41, 58, 370361)],
[6,
datetime.datetime(2021, 10, 4, 17, 29, 15, 155341),
datetime.datetime(2021, 10, 4, 17, 29, 15, 155440),
datetime.datetime(2021, 10, 4, 17, 29, 15, 155440),
0,
datetime.datetime(2021, 10, 4, 17, 29, 15, 156011),
datetime.datetime(2021, 10, 4, 17, 41, 55, 992180),
760836169,
datetime.datetime(2021, 10, 4, 17, 41, 55, 992180)],
[6,
datetime.datetime(2021, 10, 4, 17, 29, 15, 187213),
datetime.datetime(2021, 10, 4, 17, 29, 15, 187292),
datetime.datetime(2021, 10, 4, 17, 29, 15, 187292),
0,
datetime.datetime(2021, 10, 4, 17, 29, 15, 187304),
datetime.datetime(2021, 10, 4, 17, 41, 58, 621754),
763434450,
datetime.datetime(2021, 10, 4, 17, 41, 58, 621754)],
[6,
datetime.datetime(2021, 10, 4, 17, 29, 15, 187721),
datetime.datetime(2021, 10, 4, 17, 29, 15, 187804),
datetime.datetime(2021, 10, 4, 17, 29, 15, 187804),
0,
datetime.datetime(2021, 10, 4, 17, 29, 15, 187814),
datetime.datetime(2021, 10, 4, 17, 41, 55, 308719),
760120905,
datetime.datetime(2021, 10, 4, 17, 41, 55, 308719)],
[6,
datetime.datetime(2021, 10, 4, 17, 29, 15, 189701),
datetime.datetime(2021, 10, 4, 17, 29, 15, 189784),
datetime.datetime(2021, 10, 4, 17, 29, 15, 189784),
0,
datetime.datetime(2021, 10, 4, 17, 29, 15, 189792),
datetime.datetime(2021, 10, 4, 17, 41, 54, 85506),
758895714,
datetime.datetime(2021, 10, 4, 17, 41, 54, 85506)],
[6,
datetime.datetime(2021, 10, 4, 17, 29, 15, 192921),
datetime.datetime(2021, 10, 4, 17, 29, 15, 193096),
datetime.datetime(2021, 10, 4, 17, 29, 15, 193096),
0,
datetime.datetime(2021, 10, 4, 17, 29, 15, 193103),
datetime.datetime(2021, 10, 4, 17, 41, 56, 972871),
761779768,
datetime.datetime(2021, 10, 4, 17, 41, 56, 972871)]],
'2_queues_30/10_slots_superuser_leader_node': [['Mon '
'Oct '
'04 '
'17:29:15.046693 '
'2021 '
'UTC',
1711]],
'2_queues_30/10_slots_superuser_worker_node': [['Mon '
'Oct '
'04 '
'17:29:15.223939 '
'2021 '
'UTC',
523976]],
'2_queues_30/10_slots_superuser_worker_node_wlm_query': [[5,
datetime.datetime(2021, 10, 4, 17, 29, 15, 181230),
datetime.datetime(2021, 10, 4, 17, 29, 15, 181314),
datetime.datetime(2021, 10, 4, 17, 29, 15, 181314),
0,
datetime.datetime(2021, 10, 4, 17, 29, 15, 181327),
datetime.datetime(2021, 10, 4, 17, 29, 15, 231921),
50594,
datetime.datetime(2021, 10, 4, 17, 29, 15, 231921)]],
'2_queues_30/10_slots_wlm_config': [[1, 1],
[2, 1],
[3, 2],
[4, 1],
[5, 1],
[6, 10],
[7, 30],
[15, 0]],
'2_queues_30/20_slots': [[7,
datetime.datetime(2021, 10, 4, 17, 45, 40, 508220),
datetime.datetime(2021, 10, 4, 17, 45, 40, 508312),
datetime.datetime(2021, 10, 4, 17, 45, 40, 508312),
0,
datetime.datetime(2021, 10, 4, 17, 45, 40, 508322),
datetime.datetime(2021, 10, 4, 18, 1, 41, 512858),
961004536,
datetime.datetime(2021, 10, 4, 18, 1, 41, 512858)],
[7,
datetime.datetime(2021, 10, 4, 17, 45, 40, 555006),
datetime.datetime(2021, 10, 4, 17, 45, 40, 565982),
datetime.datetime(2021, 10, 4, 17, 45, 40, 565982),
0,
datetime.datetime(2021, 10, 4, 17, 45, 40, 565995),
datetime.datetime(2021, 10, 4, 18, 1, 23, 100758),
942534763,
datetime.datetime(2021, 10, 4, 18, 1, 23, 100758)],
[7,
datetime.datetime(2021, 10, 4, 17, 45, 40, 607984),
datetime.datetime(2021, 10, 4, 17, 45, 40, 608091),
datetime.datetime(2021, 10, 4, 17, 45, 40, 608091),
0,
datetime.datetime(2021, 10, 4, 17, 45, 40, 608103),
datetime.datetime(2021, 10, 4, 18, 1, 23, 720642),
943112539,
datetime.datetime(2021, 10, 4, 18, 1, 23, 720642)],
[7,
datetime.datetime(2021, 10, 4, 17, 45, 40, 663162),
datetime.datetime(2021, 10, 4, 17, 45, 40, 663260),
datetime.datetime(2021, 10, 4, 17, 45, 40, 663260),
0,
datetime.datetime(2021, 10, 4, 17, 45, 40, 663291),
datetime.datetime(2021, 10, 4, 18, 1, 42, 685471),
962022180,
datetime.datetime(2021, 10, 4, 18, 1, 42, 685471)],
[7,
datetime.datetime(2021, 10, 4, 17, 45, 40, 674044),
datetime.datetime(2021, 10, 4, 17, 45, 40, 674131),
datetime.datetime(2021, 10, 4, 17, 45, 40, 674131),
0,
datetime.datetime(2021, 10, 4, 17, 45, 40, 674140),
datetime.datetime(2021, 10, 4, 18, 1, 42, 632185),
961958045,
datetime.datetime(2021, 10, 4, 18, 1, 42, 632185)],
[7,
datetime.datetime(2021, 10, 4, 17, 45, 40, 681071),
datetime.datetime(2021, 10, 4, 17, 45, 40, 681154),
datetime.datetime(2021, 10, 4, 17, 45, 40, 681154),
0,
datetime.datetime(2021, 10, 4, 17, 45, 40, 681164),
datetime.datetime(2021, 10, 4, 18, 1, 42, 484303),
961803139,
datetime.datetime(2021, 10, 4, 18, 1, 42, 484303)],
[7,
datetime.datetime(2021, 10, 4, 17, 45, 40, 717895),
datetime.datetime(2021, 10, 4, 17, 45, 40, 717998),
datetime.datetime(2021, 10, 4, 17, 45, 40, 717998),
0,
datetime.datetime(2021, 10, 4, 17, 45, 40, 718013),
datetime.datetime(2021, 10, 4, 18, 1, 42, 171824),
961453811,
datetime.datetime(2021, 10, 4, 18, 1, 42, 171824)],
[7,
datetime.datetime(2021, 10, 4, 17, 45, 40, 725940),
datetime.datetime(2021, 10, 4, 17, 45, 40, 726040),
datetime.datetime(2021, 10, 4, 17, 45, 40, 726040),
0,
datetime.datetime(2021, 10, 4, 17, 45, 40, 726051),
datetime.datetime(2021, 10, 4, 18, 1, 42, 272628),
961546577,
datetime.datetime(2021, 10, 4, 18, 1, 42, 272628)],
[7,
datetime.datetime(2021, 10, 4, 17, 45, 40, 726110),
datetime.datetime(2021, 10, 4, 17, 45, 40, 726193),
datetime.datetime(2021, 10, 4, 17, 45, 40, 726193),
0,
datetime.datetime(2021, 10, 4, 17, 45, 40, 726200),
datetime.datetime(2021, 10, 4, 18, 1, 24, 175341),
943449141,
datetime.datetime(2021, 10, 4, 18, 1, 24, 175341)],
[7,
datetime.datetime(2021, 10, 4, 17, 45, 40, 746136),
datetime.datetime(2021, 10, 4, 17, 45, 40, 746223),
datetime.datetime(2021, 10, 4, 17, 45, 40, 746223),
0,
datetime.datetime(2021, 10, 4, 17, 45, 40, 746232),
datetime.datetime(2021, 10, 4, 18, 1, 42, 579424),
961833192,
datetime.datetime(2021, 10, 4, 18, 1, 42, 579424)],
[7,
datetime.datetime(2021, 10, 4, 17, 45, 40, 753320),
datetime.datetime(2021, 10, 4, 17, 45, 40, 753393),
datetime.datetime(2021, 10, 4, 17, 45, 40, 753393),
0,
datetime.datetime(2021, 10, 4, 17, 45, 40, 753400),
datetime.datetime(2021, 10, 4, 18, 1, 42, 687405),
961934005,
datetime.datetime(2021, 10, 4, 18, 1, 42, 687405)],
[7,
datetime.datetime(2021, 10, 4, 17, 45, 40, 756506),
datetime.datetime(2021, 10, 4, 17, 45, 40, 756592),
datetime.datetime(2021, 10, 4, 17, 45, 40, 756592),
0,
datetime.datetime(2021, 10, 4, 17, 45, 40, 756601),
datetime.datetime(2021, 10, 4, 18, 1, 26, 416616),
945660015,
datetime.datetime(2021, 10, 4, 18, 1, 26, 416616)],
[7,
datetime.datetime(2021, 10, 4, 17, 45, 40, 759971),
datetime.datetime(2021, 10, 4, 17, 45, 40, 760058),
datetime.datetime(2021, 10, 4, 17, 45, 40, 760058),
0,
datetime.datetime(2021, 10, 4, 17, 45, 40, 760067),
datetime.datetime(2021, 10, 4, 18, 1, 24, 48386),
943288319,
datetime.datetime(2021, 10, 4, 18, 1, 24, 48386)],
[7,
datetime.datetime(2021, 10, 4, 17, 45, 40, 760900),
datetime.datetime(2021, 10, 4, 17, 45, 40, 761097),
datetime.datetime(2021, 10, 4, 17, 45, 40, 761097),
0,
datetime.datetime(2021, 10, 4, 17, 45, 40, 761106),
datetime.datetime(2021, 10, 4, 18, 1, 42, 580756),
961819650,
datetime.datetime(2021, 10, 4, 18, 1, 42, 580756)],
[7,
datetime.datetime(2021, 10, 4, 17, 45, 40, 761080),
datetime.datetime(2021, 10, 4, 17, 45, 40, 761158),
datetime.datetime(2021, 10, 4, 17, 45, 40, 761158),
0,
datetime.datetime(2021, 10, 4, 17, 45, 40, 761321),
datetime.datetime(2021, 10, 4, 18, 1, 24, 80262),
943318941,
datetime.datetime(2021, 10, 4, 18, 1, 24, 80262)],
[7,
datetime.datetime(2021, 10, 4, 17, 45, 40, 747546),
datetime.datetime(2021, 10, 4, 17, 45, 40, 747651),
datetime.datetime(2021, 10, 4, 17, 45, 40, 747651),
0,
datetime.datetime(2021, 10, 4, 17, 45, 40, 762795),
datetime.datetime(2021, 10, 4, 18, 1, 24, 112432),
943349637,
datetime.datetime(2021, 10, 4, 18, 1, 24, 112432)],
[7,
datetime.datetime(2021, 10, 4, 17, 45, 40, 744996),
datetime.datetime(2021, 10, 4, 17, 45, 40, 760882),
datetime.datetime(2021, 10, 4, 17, 45, 40, 760882),
0,
datetime.datetime(2021, 10, 4, 17, 45, 40, 764710),
datetime.datetime(2021, 10, 4, 18, 1, 42, 648259),
961883549,
datetime.datetime(2021, 10, 4, 18, 1, 42, 648259)],
[7,
datetime.datetime(2021, 10, 4, 17, 45, 40, 764641),
datetime.datetime(2021, 10, 4, 17, 45, 40, 764762),
datetime.datetime(2021, 10, 4, 17, 45, 40, 764762),
0,
datetime.datetime(2021, 10, 4, 17, 45, 40, 764771),
datetime.datetime(2021, 10, 4, 18, 1, 23, 920977),
943156206,
datetime.datetime(2021, 10, 4, 18, 1, 23, 920977)],
[7,
datetime.datetime(2021, 10, 4, 17, 45, 40, 768601),
datetime.datetime(2021, 10, 4, 17, 45, 40, 768682),
datetime.datetime(2021, 10, 4, 17, 45, 40, 768682),
0,
datetime.datetime(2021, 10, 4, 17, 45, 40, 768691),
datetime.datetime(2021, 10, 4, 18, 1, 42, 471272),
961702581,
datetime.datetime(2021, 10, 4, 18, 1, 42, 471272)],
[7,
datetime.datetime(2021, 10, 4, 17, 45, 40, 778640),
datetime.datetime(2021, 10, 4, 17, 45, 40, 778725),
datetime.datetime(2021, 10, 4, 17, 45, 40, 778725),
0,
datetime.datetime(2021, 10, 4, 17, 45, 40, 778736),
datetime.datetime(2021, 10, 4, 18, 1, 42, 664954),
961886218,
datetime.datetime(2021, 10, 4, 18, 1, 42, 664954)],
[7,
datetime.datetime(2021, 10, 4, 17, 45, 40, 778869),
datetime.datetime(2021, 10, 4, 17, 45, 40, 778944),
datetime.datetime(2021, 10, 4, 17, 45, 40, 778944),
0,
datetime.datetime(2021, 10, 4, 17, 45, 40, 778952),
datetime.datetime(2021, 10, 4, 18, 1, 23, 964037),
943185085,
datetime.datetime(2021, 10, 4, 18, 1, 23, 964037)],
[7,
datetime.datetime(2021, 10, 4, 17, 45, 40, 783777),
datetime.datetime(2021, 10, 4, 17, 45, 40, 783861),
datetime.datetime(2021, 10, 4, 17, 45, 40, 783861),
0,
datetime.datetime(2021, 10, 4, 17, 45, 40, 783869),
datetime.datetime(2021, 10, 4, 18, 1, 23, 816765),
943032896,
datetime.datetime(2021, 10, 4, 18, 1, 23, 816765)],
[7,
datetime.datetime(2021, 10, 4, 17, 45, 40, 791671),
datetime.datetime(2021, 10, 4, 17, 45, 40, 791753),
datetime.datetime(2021, 10, 4, 17, 45, 40, 791753),
0,
datetime.datetime(2021, 10, 4, 17, 45, 40, 791760),
datetime.datetime(2021, 10, 4, 18, 1, 40, 880951),
960089191,
datetime.datetime(2021, 10, 4, 18, 1, 40, 880951)],
[7,
datetime.datetime(2021, 10, 4, 17, 45, 40, 792688),
datetime.datetime(2021, 10, 4, 17, 45, 40, 792762),
datetime.datetime(2021, 10, 4, 17, 45, 40, 792762),
0,
datetime.datetime(2021, 10, 4, 17, 45, 40, 792769),
datetime.datetime(2021, 10, 4, 18, 1, 41, 372720),
960579951,
datetime.datetime(2021, 10, 4, 18, 1, 41, 372720)],
[7,
datetime.datetime(2021, 10, 4, 17, 45, 40, 795398),
datetime.datetime(2021, 10, 4, 17, 45, 40, 795472),
datetime.datetime(2021, 10, 4, 17, 45, 40, 795472),
0,
datetime.datetime(2021, 10, 4, 17, 45, 40, 795479),
datetime.datetime(2021, 10, 4, 18, 1, 42, 455657),
961660178,
datetime.datetime(2021, 10, 4, 18, 1, 42, 455657)],
[7,
datetime.datetime(2021, 10, 4, 17, 45, 40, 894555),
datetime.datetime(2021, 10, 4, 17, 45, 40, 894647),
datetime.datetime(2021, 10, 4, 17, 45, 40, 894647),
0,
datetime.datetime(2021, 10, 4, 17, 45, 40, 894656),
datetime.datetime(2021, 10, 4, 18, 1, 23, 948445),
943053789,
datetime.datetime(2021, 10, 4, 18, 1, 23, 948445)],
[7,
datetime.datetime(2021, 10, 4, 17, 45, 40, 894995),
datetime.datetime(2021, 10, 4, 17, 45, 40, 895070),
datetime.datetime(2021, 10, 4, 17, 45, 40, 895070),
0,
datetime.datetime(2021, 10, 4, 17, 45, 40, 895079),
datetime.datetime(2021, 10, 4, 18, 1, 34, 244923),
953349844,
datetime.datetime(2021, 10, 4, 18, 1, 34, 244923)],
[7,
datetime.datetime(2021, 10, 4, 17, 45, 40, 911593),
datetime.datetime(2021, 10, 4, 17, 45, 40, 911679),
datetime.datetime(2021, 10, 4, 17, 45, 40, 911679),
0,
datetime.datetime(2021, 10, 4, 17, 45, 40, 911690),
datetime.datetime(2021, 10, 4, 18, 1, 42, 376488),
961464798,
datetime.datetime(2021, 10, 4, 18, 1, 42, 376488)],
[7,
datetime.datetime(2021, 10, 4, 17, 45, 40, 911727),
datetime.datetime(2021, 10, 4, 17, 45, 40, 911803),
datetime.datetime(2021, 10, 4, 17, 45, 40, 911803),
0,
datetime.datetime(2021, 10, 4, 17, 45, 40, 911812),
datetime.datetime(2021, 10, 4, 18, 1, 42, 705763),
961793951,
datetime.datetime(2021, 10, 4, 18, 1, 42, 705763)],
[7,
datetime.datetime(2021, 10, 4, 17, 45, 40, 916942),
datetime.datetime(2021, 10, 4, 17, 45, 40, 917010),
datetime.datetime(2021, 10, 4, 17, 45, 40, 917010),
0,
datetime.datetime(2021, 10, 4, 17, 45, 40, 917016),
datetime.datetime(2021, 10, 4, 18, 1, 24, 320836),
943403820,
datetime.datetime(2021, 10, 4, 18, 1, 24, 320836)],
[6,
datetime.datetime(2021, 10, 4, 17, 45, 45, 595939),
datetime.datetime(2021, 10, 4, 17, 45, 45, 596045),
datetime.datetime(2021, 10, 4, 17, 45, 45, 596045),
0,
datetime.datetime(2021, 10, 4, 17, 45, 45, 596060),
datetime.datetime(2021, 10, 4, 18, 1, 41, 684372),
956088312,
datetime.datetime(2021, 10, 4, 18, 1, 41, 684372)],
[6,
datetime.datetime(2021, 10, 4, 17, 45, 45, 606559),
datetime.datetime(2021, 10, 4, 17, 45, 45, 606666),
datetime.datetime(2021, 10, 4, 17, 45, 45, 606666),
0,
datetime.datetime(2021, 10, 4, 17, 45, 45, 606677),
datetime.datetime(2021, 10, 4, 18, 1, 40, 760743),
955154066,
datetime.datetime(2021, 10, 4, 18, 1, 40, 760743)],
[6,
datetime.datetime(2021, 10, 4, 17, 45, 45, 608422),
datetime.datetime(2021, 10, 4, 17, 45, 45, 608494),
datetime.datetime(2021, 10, 4, 17, 45, 45, 608494),
0,
datetime.datetime(2021, 10, 4, 17, 45, 45, 608501),
datetime.datetime(2021, 10, 4, 18, 1, 43, 120154),
957511653,
datetime.datetime(2021, 10, 4, 18, 1, 43, 120154)],
[6,
datetime.datetime(2021, 10, 4, 17, 45, 45, 627463),
datetime.datetime(2021, 10, 4, 17, 45, 45, 627556),
datetime.datetime(2021, 10, 4, 17, 45, 45, 627556),
0,
datetime.datetime(2021, 10, 4, 17, 45, 45, 627565),
datetime.datetime(2021, 10, 4, 18, 1, 42, 940785),
957313220,
datetime.datetime(2021, 10, 4, 18, 1, 42, 940785)],
[6,
datetime.datetime(2021, 10, 4, 17, 45, 45, 632719),
datetime.datetime(2021, 10, 4, 17, 45, 45, 632816),
datetime.datetime(2021, 10, 4, 17, 45, 45, 632816),
0,
datetime.datetime(2021, 10, 4, 17, 45, 45, 632827),
datetime.datetime(2021, 10, 4, 18, 1, 39, 921029),
954288202,
datetime.datetime(2021, 10, 4, 18, 1, 39, 921029)],
[6,
datetime.datetime(2021, 10, 4, 17, 45, 45, 647278),
datetime.datetime(2021, 10, 4, 17, 45, 45, 647367),
datetime.datetime(2021, 10, 4, 17, 45, 45, 647367),
0,
datetime.datetime(2021, 10, 4, 17, 45, 45, 647376),
datetime.datetime(2021, 10, 4, 18, 1, 43, 102451),
957455075,
datetime.datetime(2021, 10, 4, 18, 1, 43, 102451)],
[6,
datetime.datetime(2021, 10, 4, 17, 45, 45, 652053),
datetime.datetime(2021, 10, 4, 17, 45, 45, 652132),
datetime.datetime(2021, 10, 4, 17, 45, 45, 652132),
0,
datetime.datetime(2021, 10, 4, 17, 45, 45, 652139),
datetime.datetime(2021, 10, 4, 18, 1, 29, 509487),
943857348,
datetime.datetime(2021, 10, 4, 18, 1, 29, 509487)],
[6,
datetime.datetime(2021, 10, 4, 17, 45, 45, 665137),
datetime.datetime(2021, 10, 4, 17, 45, 45, 665223),
datetime.datetime(2021, 10, 4, 17, 45, 45, 665223),
0,
datetime.datetime(2021, 10, 4, 17, 45, 45, 665234),
datetime.datetime(2021, 10, 4, 18, 1, 30, 24999),
944359765,
datetime.datetime(2021, 10, 4, 18, 1, 30, 24999)],
[6,
datetime.datetime(2021, 10, 4, 17, 45, 45, 671613),
datetime.datetime(2021, 10, 4, 17, 45, 45, 671703),
datetime.datetime(2021, 10, 4, 17, 45, 45, 671703),
0,
datetime.datetime(2021, 10, 4, 17, 45, 45, 671710),
datetime.datetime(2021, 10, 4, 18, 1, 42, 976345),
957304635,
datetime.datetime(2021, 10, 4, 18, 1, 42, 976345)],
[6,
datetime.datetime(2021, 10, 4, 17, 45, 45, 671717),
datetime.datetime(2021, 10, 4, 17, 45, 45, 671790),
datetime.datetime(2021, 10, 4, 17, 45, 45, 671790),
0,
datetime.datetime(2021, 10, 4, 17, 45, 45, 671797),
datetime.datetime(2021, 10, 4, 18, 1, 29, 632491),
943960694,
datetime.datetime(2021, 10, 4, 18, 1, 29, 632491)],
[6,
datetime.datetime(2021, 10, 4, 17, 45, 45, 675236),
datetime.datetime(2021, 10, 4, 17, 45, 45, 675316),
datetime.datetime(2021, 10, 4, 17, 45, 45, 675316),
0,
datetime.datetime(2021, 10, 4, 17, 45, 45, 675324),
datetime.datetime(2021, 10, 4, 18, 1, 29, 544447),
943869123,
datetime.datetime(2021, 10, 4, 18, 1, 29, 544447)],
[6,
datetime.datetime(2021, 10, 4, 17, 45, 45, 683193),
datetime.datetime(2021, 10, 4, 17, 45, 45, 683275),
datetime.datetime(2021, 10, 4, 17, 45, 45, 683275),
0,
datetime.datetime(2021, 10, 4, 17, 45, 45, 683283),
datetime.datetime(2021, 10, 4, 18, 1, 42, 579789),
956896506,
datetime.datetime(2021, 10, 4, 18, 1, 42, 579789)],
[6,
datetime.datetime(2021, 10, 4, 17, 45, 45, 683483),
datetime.datetime(2021, 10, 4, 17, 45, 45, 683556),
datetime.datetime(2021, 10, 4, 17, 45, 45, 683556),
0,
datetime.datetime(2021, 10, 4, 17, 45, 45, 683562),
datetime.datetime(2021, 10, 4, 18, 1, 29, 850047),
944166485,
datetime.datetime(2021, 10, 4, 18, 1, 29, 850047)],
[6,
datetime.datetime(2021, 10, 4, 17, 45, 45, 711807),
datetime.datetime(2021, 10, 4, 17, 45, 45, 711906),
datetime.datetime(2021, 10, 4, 17, 45, 45, 711906),
0,
datetime.datetime(2021, 10, 4, 17, 45, 45, 711915),
datetime.datetime(2021, 10, 4, 18, 1, 30, 19516),
944307601,
datetime.datetime(2021, 10, 4, 18, 1, 30, 19516)],
[6,
datetime.datetime(2021, 10, 4, 17, 45, 45, 712008),
datetime.datetime(2021, 10, 4, 17, 45, 45, 712075),
datetime.datetime(2021, 10, 4, 17, 45, 45, 712075),
0,
datetime.datetime(2021, 10, 4, 17, 45, 45, 712082),
datetime.datetime(2021, 10, 4, 18, 1, 29, 967331),
944255249,
datetime.datetime(2021, 10, 4, 18, 1, 29, 967331)],
[6,
datetime.datetime(2021, 10, 4, 17, 45, 45, 715758),
datetime.datetime(2021, 10, 4, 17, 45, 45, 715886),
datetime.datetime(2021, 10, 4, 17, 45, 45, 715886),
0,
datetime.datetime(2021, 10, 4, 17, 45, 45, 715894),
datetime.datetime(2021, 10, 4, 18, 1, 40, 505223),
954789329,
datetime.datetime(2021, 10, 4, 18, 1, 40, 505223)],
[6,
datetime.datetime(2021, 10, 4, 17, 45, 45, 721301),
datetime.datetime(2021, 10, 4, 17, 45, 45, 721388),
datetime.datetime(2021, 10, 4, 17, 45, 45, 721388),
0,
datetime.datetime(2021, 10, 4, 17, 45, 45, 721397),
datetime.datetime(2021, 10, 4, 18, 1, 34, 643008),
948921611,
datetime.datetime(2021, 10, 4, 18, 1, 34, 643008)],
[6,
datetime.datetime(2021, 10, 4, 17, 45, 45, 721476),
datetime.datetime(2021, 10, 4, 17, 45, 45, 722392),
datetime.datetime(2021, 10, 4, 17, 45, 45, 722392),
0,
datetime.datetime(2021, 10, 4, 17, 45, 45, 722399),
datetime.datetime(2021, 10, 4, 18, 1, 29, 714263),
943991864,
datetime.datetime(2021, 10, 4, 18, 1, 29, 714263)],
[6,
datetime.datetime(2021, 10, 4, 17, 45, 45, 728572),
datetime.datetime(2021, 10, 4, 17, 45, 45, 728650),
datetime.datetime(2021, 10, 4, 17, 45, 45, 728650),
0,
datetime.datetime(2021, 10, 4, 17, 45, 45, 728658),
datetime.datetime(2021, 10, 4, 18, 1, 35, 295424),
949566766,
datetime.datetime(2021, 10, 4, 18, 1, 35, 295424)],
[6,
datetime.datetime(2021, 10, 4, 17, 45, 45, 728912),
datetime.datetime(2021, 10, 4, 17, 45, 45, 729012),
datetime.datetime(2021, 10, 4, 17, 45, 45, 729012),
0,
datetime.datetime(2021, 10, 4, 17, 45, 45, 729022),
datetime.datetime(2021, 10, 4, 18, 1, 37, 573317),
951844295,
datetime.datetime(2021, 10, 4, 18, 1, 37, 573317)]],
'2_queues_30/20_slots_superuser_leader_node': [['Mon '
'Oct '
'04 '
'17:45:45.476594 '
'2021 '
'UTC',
1711]],
'2_queues_30/20_slots_superuser_worker_node': [['Mon '
'Oct '
'04 '
'17:45:46.532921 '
'2021 '
'UTC',
523976]],
'2_queues_30/20_slots_superuser_worker_node_wlm_query': [[5,
datetime.datetime(2021, 10, 4, 17, 45, 45, 736588),
datetime.datetime(2021, 10, 4, 17, 45, 45, 736648),
datetime.datetime(2021, 10, 4, 17, 45, 45, 736648),
0,
datetime.datetime(2021, 10, 4, 17, 45, 45, 736655),
datetime.datetime(2021, 10, 4, 17, 45, 46, 533540),
796885,
datetime.datetime(2021, 10, 4, 17, 45, 46, 533540)]],
'2_queues_30/20_slots_wlm_config': [[1, 1],
[2, 1],
[3, 2],
[4, 1],
[5, 1],
[6, 20],
[7, 30],
[15, 0]]}}},
'tests': {'dc2.large': {2: {}}},
'versions': {'dc2.large': {2: 'PostgreSQL 8.0.2 on i686-pc-linux-gnu, '
'compiled by GCC gcc (GCC) 3.4.2 20041017 (Red '
'Hat 3.4.2-6.fc3), Redshift 1.0.30840'}}}
I’ve started up a Redshift Slack.
Join URL is;
https://join.slack.com/t/redshiftobservatory/shared_invite/zt-2vm3deqis-hc6h4GMDcG6Gs7~IECQNuQ