Spark Connect server - Details for Query 21

Submitted Time: 2026/05/11 04:36:30
Duration: 1.0 s
Succeeded Jobs: 12 13

Download

Plan Visualization

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [id="node0" labelType="html" label=" AdaptiveSparkPlan " tooltip="AdaptiveSparkPlan isFinalPlan=true"]; subgraph cluster1 { isCluster="true"; id="cluster1"; label="WholeStageCodegen (2)\n \nduration: 41 ms"; tooltip="WholeStageCodegen (2)"; 2 [id="node2" labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build: 8 ms peak memory: 64.2 MiB number of output rows: 2 number of sort fallback tasks: 0 avg hash probes per key: 1" tooltip="HashAggregate(keys=[status#230], functions=[count(1)])"]; } 3 [id="node3" labelType="html" label="AQEShuffleRead number of partitions: 1 number of empty partitions: 198 partition data size: 320.0 B number of coalesced partitions: 1" tooltip="AQEShuffleRead coalesced"]; 4 [id="node4" labelType="html" label="Exchange shuffle records written: 4 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 4 local bytes read: 292.0 B merged fetch fallback count: 0 local blocks read: 2 remote merged chunks fetched: 0 remote blocks read: 0 data size total (min, med, max (stageId: taskId)) 128.0 B (64.0 B, 64.0 B, 64.0 B (stage 12.0: task 55)) local merged bytes read: 0.0 B local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 22 ms (11 ms, 11 ms, 11 ms (stage 12.0: task 55)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 fetch wait time: 3 ms remote bytes read: 0.0 B number of partitions: 200 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 292.0 B (146.0 B, 146.0 B, 146.0 B (stage 12.0: task 55))" tooltip="Exchange hashpartitioning(status#230, 200), ENSURE_REQUIREMENTS, [plan_id=322]"]; subgraph cluster5 { isCluster="true"; id="cluster5"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n769 ms (374 ms, 395 ms, 395 ms (stage 12.0: task 54))"; tooltip="WholeStageCodegen (1)"; 6 [id="node6" labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build total (min, med, max (stageId: taskId)) 568 ms (279 ms, 289 ms, 289 ms (stage 12.0: task 54)) peak memory total (min, med, max (stageId: taskId)) 512.0 KiB (256.0 KiB, 256.0 KiB, 256.0 KiB (stage 12.0: task 55)) number of output rows: 4 number of sort fallback tasks: 0 avg hash probes per key: 0" tooltip="HashAggregate(keys=[status#230], functions=[partial_count(1)])"]; 7 [id="node7" labelType="html" label="ColumnarToRow number of output rows: 1,000,000 number of input batches: 200" tooltip="ColumnarToRow"]; } 8 [id="node8" labelType="html" label="BatchScan bronze.test.test number of row deletes applied: 0 number of skipped delete files: 0 number of skipped data files: 0 total data manifests: 1 total planning duration (ms): 31 number of scanned delete manifests: 0 number of file splits read: 8 number of output rows: 1,000,000 number of skipped delete manifest: 0 total delete manifests: 0 number of equality delete files: 0 total data file size (bytes): 335748 number of indexed delete files: 0 number of positional delete files: 0 number of result delete files: 0 total delete file size (bytes): 0 number of scanned data manifests: 1 number of result data files: 8 number of skipped data manifests: 0" tooltip="BatchScan bronze.test.test[status#230] bronze.test.test (branch=null) [filters=, groupedBy=] RuntimeFilters: []"]; 2->0; 3->2; 4->3; 6->4; 7->6; 8->7; }

Plan Details

== Physical Plan ==
AdaptiveSparkPlan (13)
+- == Final Plan ==
   ResultQueryStage (8), Statistics(sizeInBytes=8.0 EiB)
   +- * HashAggregate (7)
      +- AQEShuffleRead (6)
         +- ShuffleQueryStage (5), Statistics(sizeInBytes=128.0 B, rowCount=4)
            +- Exchange (4)
               +- * HashAggregate (3)
                  +- * ColumnarToRow (2)
                     +- BatchScan bronze.test.test (1)
+- == Initial Plan ==
   CollectLimit (12)
   +- HashAggregate (11)
      +- Exchange (10)
         +- HashAggregate (9)
            +- BatchScan bronze.test.test (1)


(1) BatchScan bronze.test.test
Output [1]: [status#230]
bronze.test.test (branch=null) [filters=, groupedBy=]

(2) ColumnarToRow [codegen id : 1]
Input [1]: [status#230]

(3) HashAggregate [codegen id : 1]
Input [1]: [status#230]
Keys [1]: [status#230]
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#243L]
Results [2]: [status#230, count#244L]

(4) Exchange
Input [2]: [status#230, count#244L]
Arguments: hashpartitioning(status#230, 200), ENSURE_REQUIREMENTS, [plan_id=322]

(5) ShuffleQueryStage
Output [2]: [status#230, count#244L]
Arguments: 0

(6) AQEShuffleRead
Input [2]: [status#230, count#244L]
Arguments: coalesced

(7) HashAggregate [codegen id : 2]
Input [2]: [status#230, count#244L]
Keys [1]: [status#230]
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#233L]
Results [2]: [toprettystring(count(1)#233L, Some(Etc/UTC)) AS count(1)#240, toprettystring(status#230, Some(Etc/UTC)) AS status#241]

(8) ResultQueryStage
Output [2]: [count(1)#240, status#241]
Arguments: 1

(9) HashAggregate
Input [1]: [status#230]
Keys [1]: [status#230]
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#243L]
Results [2]: [status#230, count#244L]

(10) Exchange
Input [2]: [status#230, count#244L]
Arguments: hashpartitioning(status#230, 200), ENSURE_REQUIREMENTS, [plan_id=308]

(11) HashAggregate
Input [2]: [status#230, count#244L]
Keys [1]: [status#230]
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#233L]
Results [2]: [toprettystring(count(1)#233L, Some(Etc/UTC)) AS count(1)#240, toprettystring(status#230, Some(Etc/UTC)) AS status#241]

(12) CollectLimit
Input [2]: [count(1)#240, status#241]
Arguments: 21

(13) AdaptiveSparkPlan
Output [2]: [count(1)#240, status#241]
Arguments: isFinalPlan=true