-
Notifications
You must be signed in to change notification settings - Fork 0
/
syslog_withLocalAggregation
499 lines (498 loc) · 28.9 KB
/
syslog_withLocalAggregation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
2021-01-04 14:12:16,185 INFO org.apache.hadoop.yarn.client.RMProxy (main): Connecting to ResourceManager at ip-172-31-7-2.ec2.internal/172.31.7.2:8032
2021-01-04 14:12:16,393 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-172-31-7-2.ec2.internal/172.31.7.2:10200
2021-01-04 14:12:18,613 WARN org.apache.hadoop.mapreduce.JobResourceUploader (main): Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2021-01-04 14:12:19,326 INFO org.apache.hadoop.mapreduce.JobResourceUploader (main): Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1609769451940_0001
2021-01-04 14:12:22,022 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat (main): Total input files to process : 1
2021-01-04 14:12:22,094 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): number of splits:24
2021-01-04 14:12:22,793 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Submitting tokens for job: job_1609769451940_0001
2021-01-04 14:12:22,795 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Executing with tokens: []
2021-01-04 14:12:22,980 INFO org.apache.hadoop.conf.Configuration (main): resource-types.xml not found
2021-01-04 14:12:22,980 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils (main): Unable to find 'resource-types.xml'.
2021-01-04 14:12:23,498 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl (main): Submitted application application_1609769451940_0001
2021-01-04 14:12:23,571 INFO org.apache.hadoop.mapreduce.Job (main): The url to track the job: http://ip-172-31-7-2.ec2.internal:20888/proxy/application_1609769451940_0001/
2021-01-04 14:12:23,572 INFO org.apache.hadoop.mapreduce.Job (main): Running job: job_1609769451940_0001
2021-01-04 14:12:40,884 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1609769451940_0001 running in uber mode : false
2021-01-04 14:12:40,886 INFO org.apache.hadoop.mapreduce.Job (main): map 0% reduce 0%
2021-01-04 14:13:03,046 INFO org.apache.hadoop.mapreduce.Job (main): map 1% reduce 0%
2021-01-04 14:13:08,073 INFO org.apache.hadoop.mapreduce.Job (main): map 2% reduce 0%
2021-01-04 14:13:09,080 INFO org.apache.hadoop.mapreduce.Job (main): map 3% reduce 0%
2021-01-04 14:13:14,108 INFO org.apache.hadoop.mapreduce.Job (main): map 4% reduce 0%
2021-01-04 14:13:15,114 INFO org.apache.hadoop.mapreduce.Job (main): map 5% reduce 0%
2021-01-04 14:13:21,139 INFO org.apache.hadoop.mapreduce.Job (main): map 6% reduce 0%
2021-01-04 14:13:27,183 INFO org.apache.hadoop.mapreduce.Job (main): map 8% reduce 0%
2021-01-04 14:13:47,311 INFO org.apache.hadoop.mapreduce.Job (main): map 10% reduce 0%
2021-01-04 14:13:53,347 INFO org.apache.hadoop.mapreduce.Job (main): map 11% reduce 0%
2021-01-04 14:13:59,371 INFO org.apache.hadoop.mapreduce.Job (main): map 13% reduce 0%
2021-01-04 14:14:05,398 INFO org.apache.hadoop.mapreduce.Job (main): map 14% reduce 0%
2021-01-04 14:14:10,419 INFO org.apache.hadoop.mapreduce.Job (main): map 15% reduce 0%
2021-01-04 14:14:12,428 INFO org.apache.hadoop.mapreduce.Job (main): map 17% reduce 0%
2021-01-04 14:14:32,520 INFO org.apache.hadoop.mapreduce.Job (main): map 18% reduce 0%
2021-01-04 14:14:35,530 INFO org.apache.hadoop.mapreduce.Job (main): map 19% reduce 0%
2021-01-04 14:14:38,545 INFO org.apache.hadoop.mapreduce.Job (main): map 20% reduce 0%
2021-01-04 14:14:41,560 INFO org.apache.hadoop.mapreduce.Job (main): map 21% reduce 0%
2021-01-04 14:14:44,573 INFO org.apache.hadoop.mapreduce.Job (main): map 22% reduce 0%
2021-01-04 14:14:53,606 INFO org.apache.hadoop.mapreduce.Job (main): map 24% reduce 0%
2021-01-04 14:14:54,610 INFO org.apache.hadoop.mapreduce.Job (main): map 25% reduce 0%
2021-01-04 14:15:12,693 INFO org.apache.hadoop.mapreduce.Job (main): map 26% reduce 0%
2021-01-04 14:15:18,713 INFO org.apache.hadoop.mapreduce.Job (main): map 27% reduce 0%
2021-01-04 14:15:20,720 INFO org.apache.hadoop.mapreduce.Job (main): map 28% reduce 0%
2021-01-04 14:15:24,734 INFO org.apache.hadoop.mapreduce.Job (main): map 29% reduce 0%
2021-01-04 14:15:26,741 INFO org.apache.hadoop.mapreduce.Job (main): map 30% reduce 0%
2021-01-04 14:15:32,767 INFO org.apache.hadoop.mapreduce.Job (main): map 31% reduce 0%
2021-01-04 14:15:37,782 INFO org.apache.hadoop.mapreduce.Job (main): map 33% reduce 0%
2021-01-04 14:15:57,862 INFO org.apache.hadoop.mapreduce.Job (main): map 35% reduce 0%
2021-01-04 14:16:03,886 INFO org.apache.hadoop.mapreduce.Job (main): map 36% reduce 0%
2021-01-04 14:16:09,908 INFO org.apache.hadoop.mapreduce.Job (main): map 38% reduce 0%
2021-01-04 14:16:15,930 INFO org.apache.hadoop.mapreduce.Job (main): map 39% reduce 0%
2021-01-04 14:16:20,947 INFO org.apache.hadoop.mapreduce.Job (main): map 40% reduce 0%
2021-01-04 14:16:21,950 INFO org.apache.hadoop.mapreduce.Job (main): map 42% reduce 0%
2021-01-04 14:16:46,031 INFO org.apache.hadoop.mapreduce.Job (main): map 43% reduce 0%
2021-01-04 14:16:48,037 INFO org.apache.hadoop.mapreduce.Job (main): map 45% reduce 0%
2021-01-04 14:16:52,048 INFO org.apache.hadoop.mapreduce.Job (main): map 46% reduce 0%
2021-01-04 14:16:54,054 INFO org.apache.hadoop.mapreduce.Job (main): map 47% reduce 0%
2021-01-04 14:17:02,086 INFO org.apache.hadoop.mapreduce.Job (main): map 49% reduce 0%
2021-01-04 14:17:06,096 INFO org.apache.hadoop.mapreduce.Job (main): map 50% reduce 0%
2021-01-04 14:17:26,163 INFO org.apache.hadoop.mapreduce.Job (main): map 52% reduce 0%
2021-01-04 14:17:32,178 INFO org.apache.hadoop.mapreduce.Job (main): map 54% reduce 0%
2021-01-04 14:17:38,196 INFO org.apache.hadoop.mapreduce.Job (main): map 55% reduce 0%
2021-01-04 14:17:44,215 INFO org.apache.hadoop.mapreduce.Job (main): map 56% reduce 0%
2021-01-04 14:17:46,221 INFO org.apache.hadoop.mapreduce.Job (main): map 57% reduce 0%
2021-01-04 14:17:48,226 INFO org.apache.hadoop.mapreduce.Job (main): map 58% reduce 0%
2021-01-04 14:18:05,285 INFO org.apache.hadoop.mapreduce.Job (main): map 59% reduce 0%
2021-01-04 14:18:08,294 INFO org.apache.hadoop.mapreduce.Job (main): map 60% reduce 0%
2021-01-04 14:18:11,306 INFO org.apache.hadoop.mapreduce.Job (main): map 61% reduce 0%
2021-01-04 14:18:14,313 INFO org.apache.hadoop.mapreduce.Job (main): map 62% reduce 0%
2021-01-04 14:18:17,321 INFO org.apache.hadoop.mapreduce.Job (main): map 63% reduce 0%
2021-01-04 14:18:20,329 INFO org.apache.hadoop.mapreduce.Job (main): map 64% reduce 0%
2021-01-04 14:18:30,357 INFO org.apache.hadoop.mapreduce.Job (main): map 67% reduce 0%
2021-01-04 14:18:56,432 INFO org.apache.hadoop.mapreduce.Job (main): map 69% reduce 0%
2021-01-04 14:19:02,448 INFO org.apache.hadoop.mapreduce.Job (main): map 71% reduce 0%
2021-01-04 14:19:08,473 INFO org.apache.hadoop.mapreduce.Job (main): map 72% reduce 0%
2021-01-04 14:19:15,493 INFO org.apache.hadoop.mapreduce.Job (main): map 75% reduce 0%
2021-01-04 14:19:35,564 INFO org.apache.hadoop.mapreduce.Job (main): map 76% reduce 0%
2021-01-04 14:19:41,579 INFO org.apache.hadoop.mapreduce.Job (main): map 78% reduce 0%
2021-01-04 14:19:47,595 INFO org.apache.hadoop.mapreduce.Job (main): map 80% reduce 0%
2021-01-04 14:19:53,610 INFO org.apache.hadoop.mapreduce.Job (main): map 81% reduce 0%
2021-01-04 14:19:58,624 INFO org.apache.hadoop.mapreduce.Job (main): map 83% reduce 0%
2021-01-04 14:20:18,684 INFO org.apache.hadoop.mapreduce.Job (main): map 85% reduce 0%
2021-01-04 14:20:24,700 INFO org.apache.hadoop.mapreduce.Job (main): map 87% reduce 0%
2021-01-04 14:20:30,720 INFO org.apache.hadoop.mapreduce.Job (main): map 89% reduce 0%
2021-01-04 14:20:41,751 INFO org.apache.hadoop.mapreduce.Job (main): map 92% reduce 0%
2021-01-04 14:21:01,813 INFO org.apache.hadoop.mapreduce.Job (main): map 93% reduce 0%
2021-01-04 14:21:07,827 INFO org.apache.hadoop.mapreduce.Job (main): map 95% reduce 0%
2021-01-04 14:21:13,844 INFO org.apache.hadoop.mapreduce.Job (main): map 97% reduce 0%
2021-01-04 14:21:20,864 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2021-01-04 14:21:23,871 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2021-01-04 14:21:39,919 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 100%
2021-01-04 14:21:41,930 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1609769451940_0001 completed successfully
2021-01-04 14:21:42,050 INFO org.apache.hadoop.mapreduce.Job (main): Counters: 61
File System Counters
FILE: Number of bytes read=34728206
FILE: Number of bytes written=81388667
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=3288
HDFS: Number of bytes written=0
HDFS: Number of read operations=24
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
HDFS: Number of bytes read erasure-coded=0
S3N: Number of bytes read=1604672212
S3N: Number of bytes written=53012247
S3N: Number of read operations=0
S3N: Number of large read operations=0
S3N: Number of write operations=0
Job Counters
Killed map tasks=1
Launched map tasks=24
Launched reduce tasks=1
Data-local map tasks=24
Total time spent by all maps in occupied slots (ms)=48610560
Total time spent by all reduces in occupied slots (ms)=1506144
Total time spent by all map tasks (ms)=1012720
Total time spent by all reduce tasks (ms)=15689
Total vcore-milliseconds taken by all map tasks=1012720
Total vcore-milliseconds taken by all reduce tasks=15689
Total megabyte-milliseconds taken by all map tasks=1555537920
Total megabyte-milliseconds taken by all reduce tasks=48196608
Map-Reduce Framework
Map input records=163471963
Map output records=71119513
Map output bytes=2400202188
Map output materialized bytes=40707502
Input split bytes=3288
Combine input records=71119513
Combine output records=3372258
Reduce input groups=1686118
Reduce shuffle bytes=40707502
Reduce input records=3372258
Reduce output records=1686118
Spilled Records=6744516
Shuffled Maps =24
Failed Shuffles=0
Merged Map outputs=24
GC time elapsed (ms)=16832
CPU time spent (ms)=759840
Physical memory (bytes) snapshot=20901978112
Virtual memory (bytes) snapshot=78785126400
Total committed heap usage (bytes)=18010865664
Peak Map Physical memory (bytes)=855044096
Peak Map Virtual memory (bytes)=3111997440
Peak Reduce Physical memory (bytes)=816365568
Peak Reduce Virtual memory (bytes)=4455940096
FileSpliter$ReducerClass$Counter
N_COUNTER=245382239
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=1604672212
File Output Format Counters
Bytes Written=53012247
2021-01-04 14:21:42,092 INFO org.apache.hadoop.yarn.client.RMProxy (main): Connecting to ResourceManager at ip-172-31-7-2.ec2.internal/172.31.7.2:8032
2021-01-04 14:21:42,093 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-172-31-7-2.ec2.internal/172.31.7.2:10200
2021-01-04 14:21:42,197 WARN org.apache.hadoop.mapreduce.JobResourceUploader (main): Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2021-01-04 14:21:42,203 INFO org.apache.hadoop.mapreduce.JobResourceUploader (main): Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1609769451940_0002
2021-01-04 14:21:43,391 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat (main): Total input files to process : 1
2021-01-04 14:21:43,403 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader (main): Loaded native gpl library
2021-01-04 14:21:43,405 INFO com.hadoop.compression.lzo.LzoCodec (main): Successfully loaded & initialized native-lzo library [hadoop-lzo rev 049362b7cf53ff5f739d6b1532457f2c6cd495e8]
2021-01-04 14:21:43,443 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): number of splits:1
2021-01-04 14:21:43,486 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Submitting tokens for job: job_1609769451940_0002
2021-01-04 14:21:43,486 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Executing with tokens: []
2021-01-04 14:21:43,536 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl (main): Submitted application application_1609769451940_0002
2021-01-04 14:21:43,545 INFO org.apache.hadoop.mapreduce.Job (main): The url to track the job: http://ip-172-31-7-2.ec2.internal:20888/proxy/application_1609769451940_0002/
2021-01-04 14:21:43,545 INFO org.apache.hadoop.mapreduce.Job (main): Running job: job_1609769451940_0002
2021-01-04 14:22:02,667 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1609769451940_0002 running in uber mode : false
2021-01-04 14:22:02,667 INFO org.apache.hadoop.mapreduce.Job (main): map 0% reduce 0%
2021-01-04 14:22:18,748 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2021-01-04 14:22:30,800 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 100%
2021-01-04 14:22:31,810 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1609769451940_0002 completed successfully
2021-01-04 14:22:31,846 INFO org.apache.hadoop.mapreduce.Job (main): Counters: 59
File System Counters
FILE: Number of bytes read=16143278
FILE: Number of bytes written=32762175
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=95
HDFS: Number of bytes written=0
HDFS: Number of read operations=1
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
HDFS: Number of bytes read erasure-coded=0
S3N: Number of bytes read=53012247
S3N: Number of bytes written=121603
S3N: Number of read operations=0
S3N: Number of large read operations=0
S3N: Number of write operations=0
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=691872
Total time spent by all reduces in occupied slots (ms)=858240
Total time spent by all map tasks (ms)=14414
Total time spent by all reduce tasks (ms)=8940
Total vcore-milliseconds taken by all map tasks=14414
Total vcore-milliseconds taken by all reduce tasks=8940
Total megabyte-milliseconds taken by all map tasks=22139904
Total megabyte-milliseconds taken by all reduce tasks=27463680
Map-Reduce Framework
Map input records=1686118
Map output records=3372236
Map output bytes=94422608
Map output materialized bytes=16143274
Input split bytes=95
Combine input records=0
Combine output records=0
Reduce input groups=4382
Reduce shuffle bytes=16143274
Reduce input records=3372236
Reduce output records=8764
Spilled Records=6744472
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=509
CPU time spent (ms)=17170
Physical memory (bytes) snapshot=1303216128
Virtual memory (bytes) snapshot=7536336896
Total committed heap usage (bytes)=1124073472
Peak Map Physical memory (bytes)=812306432
Peak Map Virtual memory (bytes)=3086270464
Peak Reduce Physical memory (bytes)=490909696
Peak Reduce Virtual memory (bytes)=4450066432
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=53012247
File Output Format Counters
Bytes Written=121603
2021-01-04 14:22:31,877 INFO org.apache.hadoop.yarn.client.RMProxy (main): Connecting to ResourceManager at ip-172-31-7-2.ec2.internal/172.31.7.2:8032
2021-01-04 14:22:31,877 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-172-31-7-2.ec2.internal/172.31.7.2:10200
2021-01-04 14:22:32,008 WARN org.apache.hadoop.mapreduce.JobResourceUploader (main): Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2021-01-04 14:22:32,028 INFO org.apache.hadoop.mapreduce.JobResourceUploader (main): Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1609769451940_0003
2021-01-04 14:22:33,735 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat (main): Total input files to process : 2
2021-01-04 14:22:34,565 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): number of splits:2
2021-01-04 14:22:34,597 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Submitting tokens for job: job_1609769451940_0003
2021-01-04 14:22:34,597 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Executing with tokens: []
2021-01-04 14:22:34,823 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl (main): Submitted application application_1609769451940_0003
2021-01-04 14:22:34,829 INFO org.apache.hadoop.mapreduce.Job (main): The url to track the job: http://ip-172-31-7-2.ec2.internal:20888/proxy/application_1609769451940_0003/
2021-01-04 14:22:34,829 INFO org.apache.hadoop.mapreduce.Job (main): Running job: job_1609769451940_0003
2021-01-04 14:22:52,029 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1609769451940_0003 running in uber mode : false
2021-01-04 14:22:52,029 INFO org.apache.hadoop.mapreduce.Job (main): map 0% reduce 0%
2021-01-04 14:23:07,110 INFO org.apache.hadoop.mapreduce.Job (main): map 50% reduce 0%
2021-01-04 14:23:13,140 INFO org.apache.hadoop.mapreduce.Job (main): map 83% reduce 0%
2021-01-04 14:23:15,152 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2021-01-04 14:23:33,230 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 89%
2021-01-04 14:23:38,259 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 100%
2021-01-04 14:23:39,268 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1609769451940_0003 completed successfully
2021-01-04 14:23:39,307 INFO org.apache.hadoop.mapreduce.Job (main): Counters: 59
File System Counters
FILE: Number of bytes read=50166310
FILE: Number of bytes written=101011376
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=462
HDFS: Number of bytes written=0
HDFS: Number of read operations=2
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
HDFS: Number of bytes read erasure-coded=0
S3N: Number of bytes read=53133850
S3N: Number of bytes written=128898155
S3N: Number of read operations=0
S3N: Number of large read operations=0
S3N: Number of write operations=0
Job Counters
Launched map tasks=2
Launched reduce tasks=1
Data-local map tasks=2
Total time spent by all maps in occupied slots (ms)=1630224
Total time spent by all reduces in occupied slots (ms)=2016960
Total time spent by all map tasks (ms)=33963
Total time spent by all reduce tasks (ms)=21010
Total vcore-milliseconds taken by all map tasks=33963
Total vcore-milliseconds taken by all reduce tasks=21010
Total megabyte-milliseconds taken by all map tasks=52167168
Total megabyte-milliseconds taken by all reduce tasks=64542720
Map-Reduce Framework
Map input records=1694882
Map output records=3381000
Map output bytes=109054514
Map output materialized bytes=50130692
Input split bytes=462
Combine input records=0
Combine output records=0
Reduce input groups=8764
Reduce shuffle bytes=50130692
Reduce input records=3381000
Reduce output records=3372236
Spilled Records=6762000
Shuffled Maps =2
Failed Shuffles=0
Merged Map outputs=2
GC time elapsed (ms)=1430
CPU time spent (ms)=33880
Physical memory (bytes) snapshot=2379767808
Virtual memory (bytes) snapshot=10638434304
Total committed heap usage (bytes)=2153250816
Peak Map Physical memory (bytes)=815538176
Peak Map Virtual memory (bytes)=3101827072
Peak Reduce Physical memory (bytes)=915193856
Peak Reduce Virtual memory (bytes)=4449042432
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=128898155
2021-01-04 14:23:39,343 INFO org.apache.hadoop.yarn.client.RMProxy (main): Connecting to ResourceManager at ip-172-31-7-2.ec2.internal/172.31.7.2:8032
2021-01-04 14:23:39,344 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-172-31-7-2.ec2.internal/172.31.7.2:10200
2021-01-04 14:23:39,442 WARN org.apache.hadoop.mapreduce.JobResourceUploader (main): Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2021-01-04 14:23:39,446 INFO org.apache.hadoop.mapreduce.JobResourceUploader (main): Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1609769451940_0004
2021-01-04 14:23:40,630 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat (main): Total input files to process : 1
2021-01-04 14:23:41,056 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): number of splits:2
2021-01-04 14:23:41,081 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Submitting tokens for job: job_1609769451940_0004
2021-01-04 14:23:41,081 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Executing with tokens: []
2021-01-04 14:23:41,118 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl (main): Submitted application application_1609769451940_0004
2021-01-04 14:23:41,121 INFO org.apache.hadoop.mapreduce.Job (main): The url to track the job: http://ip-172-31-7-2.ec2.internal:20888/proxy/application_1609769451940_0004/
2021-01-04 14:23:41,121 INFO org.apache.hadoop.mapreduce.Job (main): Running job: job_1609769451940_0004
2021-01-04 14:24:00,250 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1609769451940_0004 running in uber mode : false
2021-01-04 14:24:00,250 INFO org.apache.hadoop.mapreduce.Job (main): map 0% reduce 0%
2021-01-04 14:24:21,361 INFO org.apache.hadoop.mapreduce.Job (main): map 33% reduce 0%
2021-01-04 14:24:27,384 INFO org.apache.hadoop.mapreduce.Job (main): map 67% reduce 0%
2021-01-04 14:24:31,405 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2021-01-04 14:24:48,482 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 89%
2021-01-04 14:24:53,510 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 100%
2021-01-04 14:24:54,518 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1609769451940_0004 completed successfully
2021-01-04 14:24:54,565 INFO org.apache.hadoop.mapreduce.Job (main): Counters: 59
File System Counters
FILE: Number of bytes read=42955237
FILE: Number of bytes written=86681160
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=468
HDFS: Number of bytes written=0
HDFS: Number of read operations=2
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
HDFS: Number of bytes read erasure-coded=0
S3N: Number of bytes read=128899879
S3N: Number of bytes written=78906794
S3N: Number of read operations=0
S3N: Number of large read operations=0
S3N: Number of write operations=0
Job Counters
Launched map tasks=2
Launched reduce tasks=1
Data-local map tasks=2
Total time spent by all maps in occupied slots (ms)=2766768
Total time spent by all reduces in occupied slots (ms)=1942176
Total time spent by all map tasks (ms)=57641
Total time spent by all reduce tasks (ms)=20231
Total vcore-milliseconds taken by all map tasks=57641
Total vcore-milliseconds taken by all reduce tasks=20231
Total megabyte-milliseconds taken by all map tasks=88536576
Total megabyte-milliseconds taken by all reduce tasks=62149632
Map-Reduce Framework
Map input records=3372236
Map output records=3372236
Map output bytes=128898155
Map output materialized bytes=43009017
Input split bytes=468
Combine input records=0
Combine output records=0
Reduce input groups=1686118
Reduce shuffle bytes=43009017
Reduce input records=3372236
Reduce output records=0
Spilled Records=6744472
Shuffled Maps =2
Failed Shuffles=0
Merged Map outputs=2
GC time elapsed (ms)=1741
CPU time spent (ms)=55070
Physical memory (bytes) snapshot=2539683840
Virtual memory (bytes) snapshot=10668036096
Total committed heap usage (bytes)=2148532224
Peak Map Physical memory (bytes)=839770112
Peak Map Virtual memory (bytes)=3109928960
Peak Reduce Physical memory (bytes)=863473664
Peak Reduce Virtual memory (bytes)=4451872768
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=0
2021-01-04 14:24:54,597 INFO org.apache.hadoop.yarn.client.RMProxy (main): Connecting to ResourceManager at ip-172-31-7-2.ec2.internal/172.31.7.2:8032
2021-01-04 14:24:54,598 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-172-31-7-2.ec2.internal/172.31.7.2:10200
2021-01-04 14:24:54,704 WARN org.apache.hadoop.mapreduce.JobResourceUploader (main): Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2021-01-04 14:24:54,707 INFO org.apache.hadoop.mapreduce.JobResourceUploader (main): Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1609769451940_0005
2021-01-04 14:24:55,882 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat (main): Total input files to process : 2
2021-01-04 14:24:56,712 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): number of splits:3
2021-01-04 14:24:57,150 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Submitting tokens for job: job_1609769451940_0005
2021-01-04 14:24:57,150 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Executing with tokens: []
2021-01-04 14:24:57,374 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl (main): Submitted application application_1609769451940_0005
2021-01-04 14:24:57,377 INFO org.apache.hadoop.mapreduce.Job (main): The url to track the job: http://ip-172-31-7-2.ec2.internal:20888/proxy/application_1609769451940_0005/
2021-01-04 14:24:57,377 INFO org.apache.hadoop.mapreduce.Job (main): Running job: job_1609769451940_0005
2021-01-04 14:25:14,671 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1609769451940_0005 running in uber mode : false
2021-01-04 14:25:14,671 INFO org.apache.hadoop.mapreduce.Job (main): map 0% reduce 0%
2021-01-04 14:25:35,771 INFO org.apache.hadoop.mapreduce.Job (main): map 34% reduce 0%
2021-01-04 14:25:37,781 INFO org.apache.hadoop.mapreduce.Job (main): map 45% reduce 0%
2021-01-04 14:25:41,797 INFO org.apache.hadoop.mapreduce.Job (main): map 56% reduce 0%
2021-01-04 14:25:48,828 INFO org.apache.hadoop.mapreduce.Job (main): map 89% reduce 0%
2021-01-04 14:25:58,864 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2021-01-04 14:26:15,929 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 100%
2021-01-04 14:26:17,942 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1609769451940_0005 completed successfully
2021-01-04 14:26:17,981 INFO org.apache.hadoop.mapreduce.Job (main): Counters: 61
File System Counters
FILE: Number of bytes read=29746193
FILE: Number of bytes written=60447494
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=707
HDFS: Number of bytes written=0
HDFS: Number of read operations=3
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
HDFS: Number of bytes read erasure-coded=0
S3N: Number of bytes read=78921783
S3N: Number of bytes written=78906794
S3N: Number of read operations=0
S3N: Number of large read operations=0
S3N: Number of write operations=0
Job Counters
Killed map tasks=1
Launched map tasks=3
Launched reduce tasks=1
Other local map tasks=1
Data-local map tasks=2
Total time spent by all maps in occupied slots (ms)=3451008
Total time spent by all reduces in occupied slots (ms)=1582464
Total time spent by all map tasks (ms)=71896
Total time spent by all reduce tasks (ms)=16484
Total vcore-milliseconds taken by all map tasks=71896
Total vcore-milliseconds taken by all reduce tasks=16484
Total megabyte-milliseconds taken by all map tasks=110432256
Total megabyte-milliseconds taken by all reduce tasks=50638848
Map-Reduce Framework
Map input records=1686118
Map output records=1686118
Map output bytes=59275372
Map output materialized bytes=29746104
Input split bytes=707
Combine input records=0
Combine output records=0
Reduce input groups=1686118
Reduce shuffle bytes=29746104
Reduce input records=1686118
Reduce output records=0
Spilled Records=3372236
Shuffled Maps =3
Failed Shuffles=0
Merged Map outputs=3
GC time elapsed (ms)=1975
CPU time spent (ms)=56230
Physical memory (bytes) snapshot=2928979968
Virtual memory (bytes) snapshot=13705859072
Total committed heap usage (bytes)=2637692928
Peak Map Physical memory (bytes)=835387392
Peak Map Virtual memory (bytes)=3095736320
Peak Reduce Physical memory (bytes)=723664896
Peak Reduce Virtual memory (bytes)=4438732800
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=0