Added grep examples
This commit is contained in:
parent
5ebf471c7c
commit
8a88a96e9f
@ -1,2 +1,116 @@
|
|||||||
|
test
|
||||||
# Grep examples
|
# Grep examples
|
||||||
|
|
||||||
|
`grep` is one of the most useful commands that you can use to filter through files or command output
|
||||||
|
|
||||||
|
#### Basic examples
|
||||||
|
|
||||||
|
If you needed to find something in a log file
|
||||||
|
|
||||||
|
```bash
|
||||||
|
10.6.9.67 - - [19/Feb/2019:09:48:30 -0500] "GET /favicon.ico HTTP/1.1" 502 575 "http://10.6.9.73/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36" "-"
|
||||||
|
10.6.9.67 - - [19/Feb/2019:09:48:30 -0500] "GET / HTTP/1.1" 502 575 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36" "-"
|
||||||
|
10.6.9.67 - - [19/Feb/2019:09:48:30 -0500] "GET /favicon.ico HTTP/1.1" 502 575 "http://10.6.9.73/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36" "-"
|
||||||
|
10.6.9.67 - - [19/Feb/2019:09:48:30 -0500] "GET / HTTP/1.1" 502 575 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36" "-"
|
||||||
|
10.6.9.67 - - [19/Feb/2019:09:48:30 -0500] "GET /favicon.ico HTTP/1.1" 502 575 "http://10.6.9.73/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36" "-"
|
||||||
|
10.6.9.67 - - [19/Feb/2019:09:48:51 -0500] "GET / HTTP/1.1" 200 1377 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36" "-"
|
||||||
|
10.6.9.67 - - [19/Feb/2019:09:48:51 -0500] "GET /favicon.ico HTTP/1.1" 200 1252 "http://10.6.9.73/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36" "-"
|
||||||
|
10.6.9.67 - - [19/Feb/2019:09:49:48 -0500] "GET / HTTP/1.1" 200 1377 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36" "-"
|
||||||
|
```
|
||||||
|
|
||||||
|
If we need to look for say just the lines with the 502 error code we can use something like
|
||||||
|
|
||||||
|
```bash
|
||||||
|
[root@gunicorn nginx]# grep " 502 " access.log-20190220
|
||||||
|
10.6.9.67 - - [19/Feb/2019:09:46:08 -0500] "GET / HTTP/1.1" 502 575 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36" "-"
|
||||||
|
10.6.9.67 - - [19/Feb/2019:09:46:08 -0500] "GET /favicon.ico HTTP/1.1" 502 575 "http://10.6.9.73/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36" "-"
|
||||||
|
10.6.9.67 - - [19/Feb/2019:09:47:55 -0500] "GET / HTTP/1.1" 502 575 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36" "-"
|
||||||
|
10.6.9.67 - - [19/Feb/2019:09:47:55 -0500] "GET /favicon.ico HTTP/1.1" 502 575 "http://10.6.9.73/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36" "-"
|
||||||
|
10.6.9.67 - - [19/Feb/2019:09:47:56 -0500] "GET / HTTP/1.1" 502 575 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36" "-"
|
||||||
|
10.6.9.67 - - [19/Feb/2019:09:47:56 -0500] "GET /favicon.ico HTTP/1.1" 502 575 "http://10.6.9.73/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36" "-"
|
||||||
|
10.6.9.67 - - [19/Feb/2019:09:47:56 -0500] "GET / HTTP/1.1" 502 575 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36" "-"
|
||||||
|
10.6.9.67 - - [19/Feb/2019:09:47:56 -0500] "GET /favicon.ico HTTP/1.1" 502 575 "http://10.6.9.73/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36" "-"
|
||||||
|
```
|
||||||
|
This will only return lines with ' 502 ' in them (The spaces are included so we can match as much information that will be static to avoid false positives)
|
||||||
|
|
||||||
|
If we wanted to filter this again so we only get hits on the favicon.ico file we can use the pipe symbol '|' to run grep again
|
||||||
|
|
||||||
|
```bash
|
||||||
|
[root@gunicorn nginx]# grep " 502 " access.log-20190220 | grep favicon
|
||||||
|
10.6.9.67 - - [19/Feb/2019:09:46:08 -0500] "GET /favicon.ico HTTP/1.1" 502 575 "http://10.6.9.73/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36" "-"
|
||||||
|
10.6.9.67 - - [19/Feb/2019:09:47:55 -0500] "GET /favicon.ico HTTP/1.1" 502 575 "http://10.6.9.73/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36" "-"
|
||||||
|
10.6.9.67 - - [19/Feb/2019:09:47:56 -0500] "GET /favicon.ico HTTP/1.1" 502 575 "http://10.6.9.73/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36" "-"
|
||||||
|
10.6.9.67 - - [19/Feb/2019:09:47:56 -0500] "GET /favicon.ico HTTP/1.1" 502 575 "http://10.6.9.73/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36" "-"
|
||||||
|
```
|
||||||
|
|
||||||
|
As you can see we did not specify a file in the second grep command as the input is being provided by the first grep command
|
||||||
|
|
||||||
|
#### Regex
|
||||||
|
|
||||||
|
If you only need a certain part of the output of grep instead of the full line you can use the '-o' flag, this will return only what you have searched.
|
||||||
|
|
||||||
|
This is quite limited as it will just return what we have put in as the search term, we can extend this using regex in our search's this means that we can pull data we do not know and display it or more often then not push it through to another application
|
||||||
|
|
||||||
|
```bash
|
||||||
|
[root@gunicorn nginx]# grep " 502 " access.log-20190220 | grep -oE "\"http.*?\/\""
|
||||||
|
"http://10.6.9.73/"
|
||||||
|
"http://10.6.9.73/"
|
||||||
|
"http://10.6.9.73/"
|
||||||
|
"http://10.6.9.73/"
|
||||||
|
```
|
||||||
|
This may look confusing but once you figure it out it can make the output of many commands usable by others, in this case from the above output I have pulles the HOST that the client was connecting to
|
||||||
|
|
||||||
|
We can modify the regex and the type slightly (instead of -oE we can use -oP which uses perl regex)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
[root@gunicorn nginx]# grep " 502 " access.log-20190220 | grep -oP "\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}"
|
||||||
|
10.6.9.67
|
||||||
|
10.6.9.67
|
||||||
|
10.6.9.73
|
||||||
|
10.6.9.67
|
||||||
|
10.6.9.67
|
||||||
|
10.6.9.73
|
||||||
|
```
|
||||||
|
|
||||||
|
This gets the connecting IP address from the log file, this can be used along with something like `sort -hr | uniq -c` to count how many times the IP address has connected
|
||||||
|
|
||||||
|
```bash
|
||||||
|
[root@gunicorn nginx]# grep " 502 " access.log-20190220 | grep -oP "\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}" | sort -hr | uniq -c
|
||||||
|
14 10.6.9.73
|
||||||
|
28 10.6.9.67
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Other tips and tricks
|
||||||
|
|
||||||
|
You can also use grep with something like tail to monitor a log file for some specific information, this can be useful if you are waiting for a specific connection, page to be hit or a specific IP
|
||||||
|
|
||||||
|
```bash
|
||||||
|
tail -f /path/to/log | grep DATA
|
||||||
|
10.6.9.67 - - [19/Feb/2019:09:46:08 -0500] "GET /DATA HTTP/1.1" 502 575 "http://10.6.9.73/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36" "-"
|
||||||
|
```
|
||||||
|
|
||||||
|
This would only show the output that has DATA and it will continue to update as the log is written to
|
||||||
|
|
||||||
|
You can also list all files that contain the search text in a folder using the `-irl` flags
|
||||||
|
|
||||||
|
```bash
|
||||||
|
[root@gunicorn ~]# grep -irl import *
|
||||||
|
paste/paste/.git/hooks/pre-rebase.sample
|
||||||
|
paste/paste/paste.py
|
||||||
|
paste/paste/wsgi.py
|
||||||
|
```
|
||||||
|
|
||||||
|
As you can see all the files that have the string 'import' are listed in the output
|
||||||
|
|
||||||
|
You can then use something like
|
||||||
|
|
||||||
|
```bash
|
||||||
|
[root@gunicorn ~]# grep -irl import * | xargs ls -alh
|
||||||
|
-rw-r--r--. 1 root nginx 1.5K Feb 19 09:33 paste/env/bin/activate_this.py
|
||||||
|
-rw-r--r--. 1 root nginx 991 Feb 19 09:33 paste/env/bin/activate.xsh
|
||||||
|
-rwxr-xr-x. 1 root nginx 246 Feb 19 09:33 paste/env/bin/easy_install
|
||||||
|
-rwxr-xr-x. 1 root nginx 246 Feb 19 09:33 paste/env/bin/easy_install-3.6
|
||||||
|
-rwxr-xr-x. 1 root nginx 224 Feb 19 09:33 paste/env/bin/flask
|
||||||
|
```
|
||||||
|
|
||||||
|
This will run each line throug `ls -alh` which gives more information on the files, of course you can also use something like sed to then make a change to the files
|
||||||
Loading…
x
Reference in New Issue
Block a user