Hadoop集群-HDFS集群中大数据运维常用的命令总结

Hadoop集群-HDFS集群中大数据运维常用的命令总结

Hadoop集群-HDFS集群中大数据运维常用的命令总结

本篇博客会简单涉及到滚动编辑，融合镜像文件，目录的空间配额等运维操作简介。话不多少，直接上命令便于以后查看。

一.查看hadf的帮助信息

[hdfs@node101.yinzhengjie.org.cn ~]$ hdfsUsage: hdfs [--config confdir] COMMAND where COMMAND is one of: dfs run a filesystem command on the file systems supported in Hadoop. namenode -format format the DFS filesystem secondarynamenode run the DFS secondary namenode namenode run the DFS namenode journalnode run the DFS journalnode zkfc run the ZK Failover Controller daemon datanode run a DFS datanode dfsadmin run a DFS admin client diskbalancer Distributes data evenly among disks on a given node haadmin run a DFS HA admin client fsck run a DFS filesystem checking utility balancer run a cluster balancing utility jmxget get JMX exported values from NameNode or DataNode. mover run a utility to move block replicas across storage types oiv apply the offline fsimage viewer to an fsimage oiv_legacy apply the offline fsimage viewer to an legacy fsimage oev apply the offline edits viewer to an edits file fetchdt fetch a delegation token from the NameNode getconf get config values from configuration groups get the groups which users belong to snapshotDiff diff two snapshots of a directory or diff the current directory contents with a snapshot lsSnapshottableDir list all snapshottable dirs owned by the current user Use -help to see options portmap run a portmap service nfs3 run an NFS version 3 gateway cacheadmin configure the HDFS cache crypto configure HDFS encryption zones storagepolicies list/get/set block storage policies version print the versionMost commands print help when invoked w/o parameters.[hdfs@node101.yinzhengjie.org.cn ~]$

综上所述，hdfs有多个子选项，作为一枚新手建议从dfs入手，dfs子选项意思是在hdfs文件系统上运行当前系统的命令，而这些命令跟咱们学习的Linux命令长得几乎一样，接下来我们一起来看看如果使用它们吧。

二.hdfs与dfs结合使用的案例

其实hdfs 和dfs 结合使用的话实际上调用的是hadoop fs这个命令。不信你自己看帮助信息如下：

1>.查看hdfs子命令的帮助信息

2>.查看hdfs文件系统中已经存在的文件

3>.在hdfs文件系统中创建文件

4>.上传文件至根目录(在上传的过程中会产生一个以"*.Copying"字样的临时文件)

5>.在hdfs文件系统中下载文件

6>.在hdfs文件系统中删除文件

7>.在hdfs文件系统中查看文件内容

8>.在hdfs文件系统中创建目录

9>.在hdfs文件系统中修改文件名称（当然你可以可以用来移动文件到目录哟）

10>.在hdfs问系统中拷贝文件到目录

11>.递归删除目录

12>.列出本地文件的内容（默认是hdfs文件系统哟）

13>.追加文件内容到hdfs文件系统中的文件

14>.格式化名称节点

16>.重命名快照

17>.删除快照

18>.查看hadoop的Sequencefile文件内容

19>.使用df命令查看可用空间

20>.降低复制因子

21>.使用du命令查看已用空间

三.hdfs与getconf结合使用的案例

1>.获取NameNode的节点名称(可能包含多个)

2>.获取hdfs最小块信息（默认大小为1M,即1048576字节，如果想要修改的话必须为512的倍数，因为HDFS底层传输数据是每512字节进行校验）

3>.查找hdfs的NameNode的RPC地址

四.hdfs与dfsadmin结合使用的案例

1>.查看hdfs dfsadmin的帮助信息

2>.查看指定命令的帮助信息

4>.查看当前的模式

5>.进入安全模式

6>.离开安全模式

7>.安全模式的wait状态

8>.检查HDFS集群的状态

9>.目录配额（计算目录下的所有文件的总个数，如果为1，表示目录下不能放文件，即空目录！）

10>.空间配额（计算目录下所有文件的总大小，包括副本数，因此空间配最小的值可以得到一个等式："空间配最小的值  >= 需要上传文件的实际大小 * 副本数"）

11>.清空配额管理

12>.对某个目录启用快照功能（快照功能默认为禁用状态）

13>.对某个目录禁用快照功能

14>.获取某个namenode的节点状态

15>.使用dfsadmin -metasave命令提供的信息比dfsadmin -report命令提供的更多。使用此命令可以获取各种的块相关的信息（比如：块总数，正在等待复制的块，当前正在复制的块）

五.hdfs与fsck结合使用的案例

1>.查看hdfs文件系统信息

2>.fsck指令显示HDFS块信息

3>.

六.hdfs与oiv结合我使用案例

1>.查看hdfs oiv的帮助信息

[yinzhengjie@s101 ~]$ hdfs oivUsage: bin/hdfs oiv [OPTIONS] -i INPUTFILE -o OUTPUTFILEOffline Image ViewerView a Hadoop fsimage INPUTFILE using the specified PROCESSOR,saving the results in OUTPUTFILE.The oiv utility will attempt to parse correctly formed image filesand will abort fail with mal-formed image files.The tool works offline and does not require a running cluster inorder to process an image file.The following image processors are available: * XML: This processor creates an XML document with all elements of the fsimage enumerated, suitable for further analysis by XML tools. * FileDistribution: This processor analyzes the file size distribution in the image. -maxSize specifies the range [0, maxSize] of file sizes to be analyzed (128GB by default). -step defines the granularity of the distribution. (2MB by default) * Web: Run a viewer to expose read-only WebHDFS API. -addr specifies the address to listen. (localhost:5978 by default) * Delimited (experimental): Generate a text file with all of the elements common to both inodes and inodes-under-construction, separated by a delimiter. The default delimiter is \t, though this may be changed via the -delimiter argument.Required command line arguments:-i,--inputFile FSImage file to process.Optional command line arguments:-o,--outputFile Name of output file. If the specified file exists, it will be overwritten. (output to stdout by default)-p,--processor Select which type of processor to apply against image file. (XML|FileDistribution|Web|Delimited) (Web by default)-delimiter Delimiting string to use with Delimited processor. -t,--temp Use temporary dir to cache intermediate result to generate Delimited outputs. If not set, Delimited processor constructs the namespace in memory before outputting text.-h,--help Display usage information and exit[yinzhengjie@s101 ~]$

2>.使用oiv命令查询hadoop镜像文件

3>.

七.hdfs与oev结合我使用案例

1>.查看hdfs oev的帮助信息

[yinzhengjie@s101 ~]$ hdfs oevUsage: bin/hdfs oev [OPTIONS] -i INPUT_FILE -o OUTPUT_FILEOffline edits viewerParse a Hadoop edits log file INPUT_FILE and save resultsin OUTPUT_FILE.Required command line arguments:-i,--inputFile edits file to process, xml (case insensitive) extension means XML format, any other filename means binary format-o,--outputFile Name of output file. If the specified file exists, it will be overwritten, format of the file is determined by -p optionOptional command line arguments:-p,--processor Select which type of processor to apply against image file, currently supported processors are: binary (native binary format that Hadoop uses), xml (default, XML format), stats (prints statistics about edits file)-h,--help Display usage information and exit-f,--fix-txids Renumber the transaction IDs in the input, so that there are no gaps or invalid transaction IDs.-r,--recover When reading binary edit logs, use recovery mode. This will give you the chance to skip corrupt parts of the edit log.-v,--verbose More verbose output, prints the input and output filenames, for processors that write to a file, also output to screen. On large image files this will dramatically increase processing time (default is false).Generic options supported are-conf specify an application configuration file-D use value for given property-fs specify a namenode-jt specify a ResourceManager-files specify comma separated files to be copied to the map reduce cluster-libjars specify comma separated jar files to include in the classpath.-archives specify comma separated archives to be unarchived on the compute machines.The general command line syntax isbin/hadoop command [genericOptions] [commandOptions][yinzhengjie@s101 ~]$

2>.使用oev命令查询hadoop的编辑日志文件

3>.

八.hadoop命令介绍

在上面我们以及提到过，"hadoop fs"其实就等价于“hdfs dfs”,但是hadoop有些命令是hdfs 命令所不支持的，我们举几个例子：

1>.检查压缩库本地安装情况

2>.格式化名称节点

3>.执行自定义jar包