IO Operations in Hadoop

Consider the below diagram

Take the sample Wordcount example, where the most of the words has been repeated for half a million or more times.

In that case after the Mapper phase, each mapper output will have words in the range of half a million.

While transferring the data from Mapper to S & S, due to network bandwidth / any connection issues, one or more mapper output (Mapper 2 output in above case) did not reach to Sort and Shuffle, then the complete MR job will be failed.

To avoid this situation, Mapper output will always be stored in Local File System(LFS) till the MR Job completion. If because of any of the issues mentioned above, mapper output did not reach to Sort and Shuffle phase, then stored output in LFS will be taken and resent to the S & S phase.

Points to be remember

• This phase is only for Mapper in Map Reduce.
• It is NOT available for storing Sort and Shuffle output or Reducer output.
• Life of the Mapper output is till the end of the job completion i.e., as the job completion success or failure, the Local copies of mapper output will be automatically be revoked by Mapper only.

Certification Note: Mapper output copies will always be stored in LFS.