Fault Tolerance in Map Reduce

Before understanding the Fault Tolerance, lets understand the heart beat mechanism in Hadoop.

Heart Beat mechanism

Consider the scenario where
We have Name node where Job tracker is running and we have multiple data nodes, where Task trackers are running.
Task tracker 1 is running on Data node 1, Task tracker 2 is running on Data node 2 and so on..

1. Data node always sends the signal via RPC to Name node at regular interval. This is to indicate that the Data node is alive and working fine. 

2. If Data node does not give any signal to Name node at regular interval, then Name node assumes that Data node is failed.
3. In that case, Name node will divert the task to some other active data node, which is available and having copy of replicated data.
4. The default interval for which Name node waits for data node to respond is 10 minutes.

The value can be set in hdfs-site.xml

dfs.heartbeat.interval

3

Determines datanode heartbeat interval in seconds.

For rest of the tutorial, we will consider heartbeat interval as 10 minutes.

Fault Tolerance

1. When Name node allots the task to Data node, Data node provides the status of the task completion at regular interval(as per Heart beat mechanism) to Job tracker, 
2. Depending upon the Task tracker's response only, Job tracker will proceed to next Job.

Now if Job tracker assigned a task to Task tracker 1

• Name node will wait for 10 minutes for the Data node 1 to respond.
• If Data node 1 is slow or due to any network issue, Data node 1 did not respond back in 10 minutes, then Name node will assume that Data node 1 is failed and will assign the task to next available Data node 2 in the cluster, where the data is available
• Now suppose, Data node 2 started processing the task, but in meantime Data node 1 completed its task and respond to Name node at 12th minute. 
• In that scenario, Name node will abort the Data node 2 task and take on completed task from Data node 1.

This process of scheduling the task to other data node in case any data node failure is called as Fault Tolerance in Hadoop.

Consider the case, where Data Node 1 job respond as failed within 2 minutes (i.e., within the Heart beat interval time)

In such situation, Name node will not assign the task to another Data Node, as the Data node responded within the stipulated time (heartbeat interval) only.
In this case, Name node will retry the task on same data node for second time. If the Data node responded as failed again, Name node will retry the task on same data node third and fourth time as well.
If in all 4 attempts, Data node responds as failed only, in such scenario, Name node will assume that either the input data provided is not correct or the logic put in Map OR Reduce is not working properly and inform the client, about the failed job, with details like programming errors or any other error.