HDFS Commands I

Hadoop HDFS Basic Commands

Hadoop is a software framework used for distributed storage and processing of very large datasets. The core of Hadoop consists of a storage part, known as Hadoop Distributed File System (HDFS), and a processing part which is a MapReduce programming model. Hadoop services provide for data storage, data processing, data access, data governance, security, and operations.
* Hadoop is "write once read many" access.
* Modification of file is not possible in HDFS.

To check Hadoop Version

$ hdfs version

$ hdfs -version

To check the running processes (Java Processes)

$ jps

To start Storage Daemons

$ start-dfs.sh

To start Processing Daemons

$ start-yarn.sh

To list the files and directories in the HDFS

              Syntax: hdfs dfs -ls <dirname>

$ hdfs dfs -ls /

To list all the files and dirs of HDFS dir and sub-dirs(recursive listing)

               Syntax: hdfs dfs -ls -R <dirname>

$ hdfs  dfs  -ls  -R  /

To create file in HDFS

               Syntax: hdfs dfs -touchz <filename>

$ hdfs dfs -touchz  /samp.log

To create directory in HDFS

               Syntax: hdfs dfs -mkdir <dirname>

$ hdfs dfs -mkdir /sampleDir

To remove a file from HDFS

                Syntax: hdfs dfs -rm <filename>

$ hdfs dfs -rm /samp.log

To change the directory, check the current path and files in the current directory

                                            $ cd HIVE

                                            $ pwd

                                            $ ls

To copy all file from LFS (local path) to HDFS 

             Syntax: hdfs dfs -put <LFSsourcepath> <HDFSdestpath>

                   Syntax: hdfs dfs -copyFromLocal <LFSsrcpath><HDFSdestpath>

$ hdfs dfs -put  /home/hduser/HIVE/*  /sampleDir

OR

$ hdfs dfs -copyFromLocal  /home/hduser/HIVE/*  /sampleDir

To copy a file from LFS (local path) to HDFS 

                   Syntax: hdfs dfs -put <LFSsourcepath> <HDFSdestpath>

                   Syntax: hdfs dfs -copyFromLocal <LFSsrcpath><HDFSdestpath>

$ hdfs dfs -put  CiscoLogData.log  /sampDir

To move a file from LFS (local path) to HDFS 

                      Syntax: hdfs dfs -moveFromLocal <LFSsrcpath> <HDFSdestpath>

$ hdfs  dfs  -moveFromLocal  retailData.log  /sampDir

To copy a file from HDFS  to LFS (local path) 

                 Syntax: hdfs fs -get <HDFSsrcpath> <LFSdestpath>

Syntax: hdfs fs -copyToLocal <HDFSsrcpath> <LFSdestpath>

$ hdfs dfs -get /sampDir/retailData.log .

$ hdfs dfs -copyToLocal /sampDir/retailData.log .

Here "." (Dot) represents the current directory

To remove the directory in HDFS

Syntax: hdfs dfs -rm -R <dirname>

$ hdfs dfs -rm -R /sampDir

To copy the file from one HDFS directory to another HDFS directory

Syntax: hdfs dfs -cp <HDFSsrcpath> <HDFSdestpath>

$ hdfs dfs -cp /sampleDir/updatedData.log  /testDir

To rename the file in HDFS

Syntax: hdfs dfs -mv <HDFSsrcpath> <HDFSdestpath>

$ hdfs dfs -mv /testDir/updatedData.log /testDir/LogFiles.log

To move the file from one HDFS directory to another HDFS directory

Syntax: hdfs dfs -mv <HDFSsrcpath> <HDFSdestpath>

$ hdfs dfs -mv /testDir/LogFiles.log  /sampDir

To see the contents in the file present in HDFS

Syntax: hdfs dfs -cat <filename>

$ hdfs dfs -cat /sampDir/LogFiles.log

To stop Processing Daemons

$ stop-yarn.sh

To stop Storage Daemons

$ stop-dfs.sh