datanode not able to communicate with namenode after upgrade



0
datanodes and namenode,nodemangers,resourcemanger sare starting fine.

datanode logs:
-----------------
2016-03-25 15:49:01,561 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time to add replicas to map for block pool BP-1598670878-192.168.56.81-1458871848204 on volume /hdfs_storage/data/current: 163ms
2016-03-25 15:49:01,561 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Total time to add all replicas to map: 165ms
2016-03-25 15:49:01,563 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-1598670878-192.168.56.81-1458871848204 (Datanode Uuid null) service to hadoop-vm1/192.168.56.81:9000 beginning handshake with NN
2016-03-25 15:50:01,661 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: hadoop-vm1/192.168.56.81:9000
2016-03-25 15:51:02,732 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: hadoop-vm1/192.168.56.81:9000
2016-03-25 15:52:03,791 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: hadoop-vm1/192.168.56.81:9000

namnode logs:
------------------
2016-03-25 15:48:44,911 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under construction: 0
2016-03-25 15:48:44,911 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under construction: 0
2016-03-25 15:48:44,911 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode ON.
The reported blocks 0 needs additional 335 blocks to reach the threshold 0.9990 of total blocks 335.
The number of live datanodes 0 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
2016-03-25 15:48:44,950 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2016-03-25 15:48:44,951 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9000: starting
2016-03-25 15:48:44,952 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode RPC up at: hadoop-vm1/192.168.56.81:9000
2016-03-25 15:48:44,952 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services required for active state
2016-03-25 15:48:44,961 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Starting CacheReplicationMonitor with interval 30000 milliseconds
2016-03-25 15:48:44,961 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning because of pending operations
2016-03-25 15:48:45,328 INFO org.apache.hadoop.fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 30 minutes, Emptier interval = 0 minutes.
2016-03-25 15:48:45,334 INFO org.apache.hadoop.fs.TrashPolicyDefault: The configured checkpoint interval is 0 minutes. Using an interval of 30 minutes that is used for deletion instead
2016-03-25 15:48:45,339 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 378 millisecond(s).
2016-03-25 15:49:00,497 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* registerDatanode: from DatanodeRegistration(192.168.56.83, datanodeUuid=423ea673-d9ef-41f7-a5bb-964b35d02cf4, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-cfd64e1a-c749-49bb-8bce-8a855dabe10c;nsid=526961048;c=1458871848513) storage 423ea673-d9ef-41f7-a5bb-964b35d02cf4
2016-03-25 15:49:14,962 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds

2 Answer(s)


0

Hi Ramesh,

Once restart your datanode (hadoop-daemon.sh start datanode) and send last 300 lines of datanode logs from $HADOOP_HOME/logs.

You also need to confirm whether your cluster IDs (/name/current/VERSION) for datanode and namenode is same.

Regards,
Sakhuja

0

Hello,

I also have same problem. My Datanode is not connecting to Namenode though my namenode and datanode is running fine. I have 3 node cluster (1 Namenode 2 Datanode). Datanode daemon start successfully but it is not showing in livenodes list in namenode webUI. Datanode logs continue showing these lines:

org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: master/172.16.30.159:9000

I have cross checked all configurations everytingh is fine.

If you find any solution to this please share...