HDFS commands II

HDFS Commands

                  Default Hadoop Ports are as follows:-

Hadoop Web UI

          https://localhost:50070

Set the replication of a file

                  Syntax: hdfs dfs -setrep -w 5 <filename>

$ hdfs dfs -setrep /sampDir/LogFile.log

In UI, you can see Replication of LogFiles.log is set to 5.

To see the number of blocks file is using

                Syntax: hdfs fsck -blocks <filename>

$ hdfs fsck -blocks /sampleDir/olympic_data.csv

To check the disk usage

                Syntax: hdfs dfs -du <dirname>/<filename>

$ hdfs dfs -du /sampleDir

To count the total number of lines, words and characters

             Syntax: hdfs dfs -cat <filename> | wc

$ hdfs dfs -cat /sampleDir/olympic_data.csv | wc

To count the total number of lines

           Syntax: hdfs dfs -cat <filename> | wc -l

                 $ hdfs dfs -cat /sampleDir/olympic_data.csv | wc -l

To count the total number of words

             Syntax: hdfs dfs -cat <filename> | wc -w

 $ hdfs dfs -cat /sampleDir/olympic_data.csv | wc -w

To count the total number of characters

             Syntax: hdfs dfs -cat <filename> | wc -c

 $ hdfs dfs -cat /sampleDir/olympic_data.csv | wc -c

To print statistics about the file/directory
             Syntax: hdfs dfs -stat <filename>/<directory>
             Syntax: hdfs dfs -stat "%F %u:%g %b %y %n" <filename>

$ hdfs dfs -stat /sampleDir/olympix_data.csv

$ hdfs dfs -stat "%F %u:%g  %b %y %n" /sampleDir/olympix_data.csv

file size in blocks (%b), 
file type (%F), 
group name of owner (%g), 
name (%n), 
block size (%o), 
replication (%r), 
user name of owner(%u),
modification date (%y, %Y). %y shows UTC date as “yyyy-MM-dd HH: mm: ss” 

Count the number of directories, files and bytes

             Syntax: hdfs dfs -count <filename>/<directory>

$ hdfs dfs -count /sampleDir/olympix_data.csv

$ hdfs dfs -count /sampleDir                             

To change the group of file/directory to root

              Syntax: hdfs dfs -chgrp root <filename>/<directory>

$ hdfs dfs -chgrp root /home

To change the owner of file/directory to root                                
        Syntax: hdfs dfs -chown root <filename>/<directory>

$ hdfs dfs -chown root /home

To change the permissions of files/directory

      There are three Classes – Owner, Group, Others.

• The Owner is usually the creator of the files/folders.
• The Group contains a group of users who share the same permissions and user privileges.
• Others mean the general public.

       As for permissions, there are 3 types of actions that you can perform on a file/folder. You can either read, write or execute.

• Read – You can only view the file, but cannot modify the content of the file. 
• Write – You can edit and modify the file. For Folders, you can delete and add the files into the folder.
• Execute – Execute is mainly used when you need to run the file.

                                       

           Syntax : hdfs dfs -chmod [-R] <filename>/<directory>

With -R, make the change recursively through the directory. The user must be the owner of the file, or a super-user.

$ hdfs dfs -chmod 777 /home

"chmod 777” means making the file readable, writable and executable by everyone.

To get the cluster summary

$ hdfs dfsadmin -report

To know the status,change the status (safemode)