Hadoop DistCp

DistCp (distributed copy) is a tool for large inter/intra-cluster copying.

Run the DistCp command for inter-cluster.

$ hadoop distcp hdfs://nn1:8020/basePath hdfs://nn2:8020/basePath

$ hadoop distcp hdfs://nn1:8020/basePath1 hdfs://nn1:8020/basePath2 hdfs://nn2:8020/basePath

$ hadoop distcp hdfs://nn1:8020/srclist hdfs://nn2:8020/basePath

Where srclist contains hdfs://nn1:8020/basePath1 and hdfs://nn1:8020/basePath2.

This will expand the namespace under /basepath on nn1 into a temporary file, partition its contents among a map tasks, and start a copy on each TaskTracker from nn1 to nn2.

Run the DistCp command on the destination cluster only for copying between different versions of Hadoop.

$ hadoop distcp hftp://cdh3-namenode:50070/ hdfs:/cdh4-namenode/

For specific path such as /hbase

$ hadoop distcp hftp://cdh3-namenode:50070/basePath hdfs:/cdh4-namenode/basePath

Where cdh3-namenode refers to the source NameNode hostname as defined by the config  fs.default.name and 50070 is the NameNode port as defined by the config dfs.http.address, and cdh4-namenode is destination NameNode as defined by the config fs.defaultFS. You can also use destination nameservice-id. basePath in both above URIs is the directory to copy.