Map-Reduce Framework Counters


2 Answer(s)


If I understand the question correctly..Then in a distributed system several mappers/reducers run parallel ,no one knows which one will start or end first, so its difficult to serialize anything.But you can easily visualize the things per mapper/reducer level from UI.
If you really want to see the difference, how counters are increasing , push a huge set of data. Then run your job from command prompt and go to jobTracker URL in browser and open maps and see how counters increases before map/reducer finishes.

Hi David,

Have you take this snapshot at the end of a MR job or in between?
If it is in-between, then what ratnakar says is correct but if it is taken at then end, then that should not happen.

Concept behind Counters:
Each task-tracker executes one or more map-reduce job and capture the counters and report them back to JobTracker. It is JobTracker, that keeps track of all the counters of a job and displays it in the end of a job, after successful completion.

So if snapshot is taken in between, when map-reduce tasks were running, arbitrary partial counts are shown based on which task, a TT is running.
But if it is taken at the successful completion, then Jt provides all the counters in the order that you have specified.

If that is not the case, please re-try the job and share the snap-shot.
Also mention the Cloudera version, you are using for this job.

Vote-up, if it helps.
Happy Learning!!