实现一个全链路监控平台的方案，如何选择应用性能监控平台？
1035
2023-03-11
用Shell脚本分析Nginx日志
本站部分文章、图片属于网络上可搜索到的公开信息，均用于学习和交流用途，不能代表睿象云的观点、立场或意见。我们接受网民的监督，如发现任何违法内容或侵犯了您的权益，请第一时间联系小编邮箱jiasou666@gmail.com 处理。
用Shell脚本分析Nginx日志
本节将介绍用Shell脚本来分析Nginx负载均衡器的日志，这样可以快速得出排名靠前的网站和IP等，推荐大家使用线上环境下的Shell脚本。本节中的Shell脚本又分为两种情况，第一种情况是Nginx作为最前端的负载均衡器，其集群架构为Nginx+Keepalived时，脚本内容如下所示：
[root@tiejiang ~]# vim log-nginx.sh #!/bin/bash if ［$# -eq 0 ］; then echo "Error: please specify logfile." exit 0 else LOG=$1 fi if ［ ! -f$1 ］; then echo "Sorry, sir, I can't find this apache log file, pls try again!" exit 0 fi #################################################### echo "Most of the ip:" echo "-------------------------------------------" awk '{ print$1 }'$LOG| sort| uniq -c| sort -nr| head -10 echo echo #################################################### echo "Most of the time:" echo "--------------------------------------------" awk '{ print$4 }'$LOG| cut -c 14-18| sort| uniq -c| sort -nr| head -10 echo echo #################################################### echo "Most of the page:" echo "--------------------------------------------" awk '{print$11}'$LOG| sed 's/^.*\\(.cn*\\)\"/\\1/g'| sort| uniq -c| sort -rn| head -10 echo echo #################################################### echo "Most of the time / Most of the ip:" echo "--------------------------------------------" awk '{ print$4 }'$LOG| cut -c 14-18| sort -n| uniq -c| sort -nr| head -10 > timelog for i in 'awk '{ print$2 }' timelog' do num='grep$i timelog| awk '{ print$1 }'' echo "$i$num" ip='grep$i$LOG| awk '{ print$1}'| sort -n| uniq -c| sort -nr| head -10' echo "$ip" echo done rm -f timelog[root@ ~]# vim log-nginx.sh #!/bin/bash if ［$# -eq 0 ］; then echo "Error: please specify logfile." exit 0 else LOG=$1 fi if ［ ! -f$1 ］; then echo "Sorry, sir, I can't find this apache log file, pls try again!" exit 0 fi #################################################### echo "Most of the ip:" echo "-------------------------------------------" awk '{ print$1 }'$LOG| sort| uniq -c| sort -nr| head -10 echo echo #################################################### echo "Most of the time:" echo "--------------------------------------------" awk '{ print$4 }'$LOG| cut -c 14-18| sort| uniq -c| sort -nr| head -10 echo echo #################################################### echo "Most of the page:" echo "--------------------------------------------" awk '{print$11}'$LOG| sed 's/^.*\\(.cn*\\)\"/\\1/g'| sort| uniq -c| sort -rn| head -10 echo echo #################################################### echo "Most of the time / Most of the ip:" echo "--------------------------------------------" awk '{ print$4 }'$LOG| cut -c 14-18| sort -n| uniq -c| sort -nr| head -10 > timelog for i in 'awk '{ print$2 }' timelog' do num='grep$i timelog| awk '{ print$1 }'' echo "$i$num" ip='grep$i$LOG| awk '{ print$1}'| sort -n| uniq -c| sort -nr| head -10' echo "$ip" echo done rm -f timelog
#!/bin/bash if ［$# -eq 0 ］; then echo "Error: please specify logfile." exit 0 else LOG=$1 fi if ［ ! -f$1 ］; then echo "Sorry, sir, I can't find this apache log file, pls try again!" exit 0 fi #################################################### echo "Most of the ip:" echo "-------------------------------------------" awk '{ print$1 }'$LOG| sort| uniq -c| sort -nr| head -10 echo echo #################################################### echo "Most of the time:" echo "--------------------------------------------" awk '{ print$4 }'$LOG| cut -c 14-18| sort| uniq -c| sort -nr| head -10 echo echo #################################################### echo "Most of the page:" echo "--------------------------------------------" awk '{print$11}'$LOG| sed 's/^.*\\(.cn*\\)\"/\\1/g'| sort| uniq -c| sort -rn| head -10 echo echo #################################################### echo "Most of the time / Most of the ip:" echo "--------------------------------------------" awk '{ print$4 }'$LOG| cut -c 14-18| sort -n| uniq -c| sort -nr| head -10 > timelog for i in 'awk '{ print$2 }' timelog' do num='grep$i timelog| awk '{ print$1 }'' echo "$i$num" ip='grep$i$LOG| awk '{ print$1}'| sort -n| uniq -c| sort -nr| head -10' echo "$ip" echo done rm -f timelog#!/bin/bash if ［$# -eq 0 ］; then echo "Error: please specify logfile." exit 0 else LOG=$1 fi if ［ ! -f$1 ］; then echo "Sorry, sir, I can't find this apache log file, pls try again!" exit 0 fi #################################################### echo "Most of the ip:" echo "-------------------------------------------" awk '{ print$1 }'$LOG| sort| uniq -c| sort -nr| head -10 echo echo #################################################### echo "Most of the time:" echo "--------------------------------------------" awk '{ print$4 }'$LOG| cut -c 14-18| sort| uniq -c| sort -nr| head -10 echo echo #################################################### echo "Most of the page:" echo "--------------------------------------------" awk '{print$11}'$LOG| sed 's/^.*\\(.cn*\\)\"/\\1/g'| sort| uniq -c| sort -rn| head -10 echo echo #################################################### echo "Most of the time / Most of the ip:" echo "--------------------------------------------" awk '{ print$4 }'$LOG| cut -c 14-18| sort -n| uniq -c| sort -nr| head -10 > timelog for i in 'awk '{ print$2 }' timelog' do num='grep$i timelog| awk '{ print$1 }'' echo "$i$num" ip='grep$i$LOG| awk '{ print$1}'| sort -n| uniq -c| sort -nr| head -10' echo "$ip" echo done rm -f timelog
#!/bin/bash if ［$# -eq 0 ］; then echo "Error: please specify logfile." exit 0 else cat$1| egrep -v '203.93.236.141|145' > LOG fi if ［ ! -f$1 ］; then echo "Sorry, sir, I can't find this apache log file, pls try again!" exit 0 fi #################################################### echo "Most of the ip:" echo "-------------------------------------------" awk '{ print$1 }' LOG| sort| uniq -c| sort -nr| head -10 echo echo #################################################### echo "Most of the time:" echo "--------------------------------------------" awk '{ print$4 }' LOG| cut -c 14-18| sort| uniq -c| sort -nr| head -10 echo echo #################################################### echo "Most of the page:" echo "--------------------------------------------" awk '{print$11}' LOG| sed 's/^.*\(.cn*\)\"/\1/g'| sort| uniq -c| sort -rn| head -10 echo echo #################################################### echo "Most of the time / Most of the ip:" echo "--------------------------------------------" awk '{ print$4 }' LOG| cut -c 14-18| sort -n| uniq -c| sort -nr| head -10 > timelog for i in 'awk '{ print$2 }' timelog' do num='grep$i timelog| awk '{ print$1 }'' echo "$i$num" ip='grep$i LOG| awk '{ print$1}'| sort -n| uniq -c| sort -nr| head -10' echo "$ip" echo done rm -f timelog#!/bin/bash if ［$# -eq 0 ］; then echo "Error: please specify logfile." exit 0 else cat$1| egrep -v '203.93.236.141|145' > LOG fi if ［ ! -f$1 ］; then echo "Sorry, sir, I can't find this apache log file, pls try again!" exit 0 fi #################################################### echo "Most of the ip:" echo "-------------------------------------------" awk '{ print$1 }' LOG| sort| uniq -c| sort -nr| head -10 echo echo #################################################### echo "Most of the time:" echo "--------------------------------------------" awk '{ print$4 }' LOG| cut -c 14-18| sort| uniq -c| sort -nr| head -10 echo echo #################################################### echo "Most of the page:" echo "--------------------------------------------" awk '{print$11}' LOG| sed 's/^.*\(.cn*\)\"/\1/g'| sort| uniq -c| sort -rn| head -10 echo echo #################################################### echo "Most of the time / Most of the ip:" echo "--------------------------------------------" awk '{ print$4 }' LOG| cut -c 14-18| sort -n| uniq -c| sort -nr| head -10 > timelog for i in 'awk '{ print$2 }' timelog' do num='grep$i timelog| awk '{ print$1 }'' echo "$i$num" ip='grep$i LOG| awk '{ print$1}'| sort -n| uniq -c| sort -nr| head -10' echo "$ip" echo done rm -f timelog
第二种情况是以Nginx作为Web端，置于LVS后面，这时要剔除掉LVS的IP地址，比如LVS服务器的公网IP地址(如203.93.236.141、203.93.236.145等)。这样可以将第一种情况的脚本略微调整一下，如下所示：
#!/bin/bash if ［$# -eq 0 ］; then echo "Error: please specify logfile." exit 0 else cat$1| egrep -v '203.93.236.141|145' > LOG fi if ［ ! -f$1 ］; then echo "Sorry, sir, I can't find this apache log file, pls try again!" exit 0 fi #################################################### echo "Most of the ip:" echo "-------------------------------------------" awk '{ print$1 }' LOG| sort| uniq -c| sort -nr| head -10 echo echo #################################################### echo "Most of the time:" echo "--------------------------------------------" awk '{ print$4 }' LOG| cut -c 14-18| sort| uniq -c| sort -nr| head -10 echo echo #################################################### echo "Most of the page:" echo "--------------------------------------------" awk '{print$11}' LOG| sed 's/^.*\(.cn*\)\"/\1/g'| sort| uniq -c| sort -rn| head -10 echo echo #################################################### echo "Most of the time / Most of the ip:" echo "--------------------------------------------" awk '{ print$4 }' LOG| cut -c 14-18| sort -n| uniq -c| sort -nr| head -10 > timelog for i in 'awk '{ print$2 }' timelog' do num='grep$i timelog| awk '{ print$1 }'' echo "$i$num" ip='grep$i LOG| awk '{ print$1}'| sort -n| uniq -c| sort -nr| head -10' echo "$ip" echo done rm -f timelog#!/bin/bash if ［$# -eq 0 ］; then echo "Error: please specify logfile." exit 0 else cat$1| egrep -v '203.93.236.141|145' > LOG fi if ［ ! -f$1 ］; then echo "Sorry, sir, I can't find this apache log file, pls try again!" exit 0 fi #################################################### echo "Most of the ip:" echo "-------------------------------------------" awk '{ print$1 }' LOG| sort| uniq -c| sort -nr| head -10 echo echo #################################################### echo "Most of the time:" echo "--------------------------------------------" awk '{ print$4 }' LOG| cut -c 14-18| sort| uniq -c| sort -nr| head -10 echo echo #################################################### echo "Most of the page:" echo "--------------------------------------------" awk '{print$11}' LOG| sed 's/^.*\(.cn*\)\"/\1/g'| sort| uniq -c| sort -rn| head -10 echo echo #################################################### echo "Most of the time / Most of the ip:" echo "--------------------------------------------" awk '{ print$4 }' LOG| cut -c 14-18| sort -n| uniq -c| sort -nr| head -10 > timelog for i in 'awk '{ print$2 }' timelog' do num='grep$i timelog| awk '{ print$1 }'' echo "$i$num" ip='grep$i LOG| awk '{ print$1}'| sort -n| uniq -c| sort -nr| head -10' echo "$ip" echo done rm -f timelog
如果对日志的要求不高，我们可以直接通过Awk和Sed来分析Linux日志(如果对Perl熟练也可以用它来操作)，还可以通过Awstats来进行详细分析，后者尤其适合Web服务器和邮件服务器。另外，如果对日志有特殊需求的话，还可以架设专用的日志服务器来收集Linux服务器日志。总之一句话：一切看需求而定。
相关文章
发表评论
暂时没有评论，来抢沙发吧~