#  Shell Scripting for DevOps Engineers — Log Management & Analysis

In this article, we will learn how Shell Scripting can help us **read, search, filter, monitor, and analyze log files**.

Log management is an important DevOps skill because logs help us understand **application failures, server issues, deployment problems, authentication failures, and system events**.

* * *

## 1\. What is a Log File?

A log file contains information about events happening inside a system or application.

For example:

```text
Application
     ↓
    Logs
     ↓
Log File
     ↓
Analysis
     ↓
Troubleshooting
```

Common Linux log locations include:

```bash
/var/log/
```

Examples:

```text
/var/log/syslog
/var/log/auth.log
/var/log/messages
```

The exact files depend on the Linux distribution and logging configuration.

* * *

## 2\. Why Are Logs Important for DevOps?

Logs help us:

*   Troubleshoot application failures
    
*   Find errors
    
*   Monitor applications
    
*   Investigate failed deployments
    
*   Identify authentication problems
    
*   Analyze server issues
    
*   Monitor services
    
*   Automate alerting and health checks
    

* * *

## 3\. List Log Files

To view the `/var/log` directory:

```bash
ls -lh /var/log/
```

To see only files:

```bash
find /var/log -type f
```

* * *

## 4\. `cat` Command

`cat` displays file contents.

```bash
cat app.log
```

For a small log file, this is useful.

However, very large log files can produce a lot of output.

* * *

## 5\. `less` Command

For large files, use:

```bash
less app.log
```

Useful keys:

```text
Space → Next page
b     → Previous page
/word → Search
q     → Quit
```

Example:

```bash
less /var/log/syslog
```

* * *

## 6\. `head` Command

`head` displays the beginning of a file.

```bash
head app.log
```

Display the first 20 lines:

```bash
head -n 20 app.log
```

Useful when you want to quickly inspect the beginning of a log.

* * *

## 7\. `tail` Command

`tail` displays the last lines of a file.

```bash
tail app.log
```

Display the last 20 lines:

```bash
tail -n 20 app.log
```

* * *

## 8\. `tail -f` — Real-Time Log Monitoring

One of the most useful commands for DevOps engineers is:

```bash
tail -f app.log
```

It continuously displays new lines added to the file.

Example:

```text
2026-09-30 16:10:01 INFO Application started
2026-09-30 16:10:10 INFO Request received
2026-09-30 16:10:15 ERROR Database connection failed
```

This is useful while troubleshooting an application or deployment.

To stop:

```text
Ctrl + C
```

* * *

## 9\. `grep` Command

`grep` searches for text inside files.

Find errors:

```bash
grep "ERROR" app.log
```

Find warnings:

```bash
grep "WARN" app.log
```

Find information:

```bash
grep "INFO" app.log
```

* * *

## 10\. Case-Insensitive Search

Use `-i`:

```bash
grep -i "error" app.log
```

This can match:

```text
ERROR
Error
error
```

* * *

## 11\. Show Line Numbers

Use `-n`:

```bash
grep -n "ERROR" app.log
```

Example:

```text
25:ERROR Database connection failed
47:ERROR Payment service unavailable
```

This makes it easier to locate the matching lines.

* * *

## 12\. Count Errors

Use `-c`:

```bash
grep -c "ERROR" app.log
```

Example:

```text
15
```

This means 15 matching lines were found.

* * *

## 13\. Search Multiple Patterns

Use:

```bash
grep -E "ERROR|WARN" app.log
```

This searches for both:

```text
ERROR
WARN
```

* * *

## 14\. Exclude Text

Use `-v`:

```bash
grep -v "INFO" app.log
```

This displays lines that don't contain `INFO`.

* * *

## 15\. Search Recursively

To search through multiple files/directories:

```bash
grep -R "ERROR" /var/log/
```

For a case-insensitive recursive search:

```bash
grep -Ri "error" /var/log/
```

You may need appropriate permissions to read some system logs.

* * *

## 16\. Combine `tail` and `grep`

This is very useful for real-time monitoring:

```bash
tail -f app.log | grep "ERROR"
```

Now you see only new lines containing `ERROR`.

You can monitor multiple patterns:

```bash
tail -f app.log | grep -E "ERROR|WARN"
```

* * *

## 17\. Using `awk` for Log Analysis

`awk` is useful for processing structured text.

Suppose the log contains:

```text
2026-09-30 INFO UserLogin Success
2026-09-30 ERROR Database Failed
2026-09-30 INFO Payment Success
```

Display the first field:

```bash
awk '{print $1}' app.log
```

Display the second field:

```bash
awk '{print $2}' app.log
```

Display the first and third fields:

```bash
awk '{print $1, $3}' app.log
```

* * *

## 18\. Count Specific Log Entries with `awk`

For example:

```bash
awk '/ERROR/ {count++} END {print count}' app.log
```

This counts lines containing `ERROR`.

* * *

## 19\. Using `sed`

`sed` is useful for searching and transforming text.

Display lines containing `ERROR`:

```bash
sed -n '/ERROR/p' app.log
```

Replace text in output:

```bash
sed 's/ERROR/FAILURE/g' app.log
```

For log analysis, `sed` is commonly combined with other commands.

* * *

## 20\. Using Pipes for Log Analysis

Pipes allow one command's output to become another command's input.

Example:

```bash
cat app.log | grep "ERROR"
```

Another example:

```bash
grep "ERROR" app.log | wc -l
```

This means:

```text
Log File
   ↓
grep ERROR
   ↓
Matching Errors
   ↓
wc -l
   ↓
Error Count
```

* * *

## 21\. `wc` Command

`wc` counts lines, words, and characters.

Count lines:

```bash
wc -l app.log
```

Count words:

```bash
wc -w app.log
```

Count characters:

```bash
wc -m app.log
```

Count errors:

```bash
grep -i "error" app.log | wc -l
```

* * *

## 22\. Find the Most Common Errors

You can combine `grep`, `sort`, and `uniq`.

For example:

```bash
grep "ERROR" app.log | sort | uniq -c | sort -nr
```

Conceptually:

```text
grep
 ↓
Extract errors
 ↓
sort
 ↓
Group similar lines
 ↓
uniq -c
 ↓
Count occurrences
 ↓
sort -nr
 ↓
Show frequent entries first
```

* * *

## 23\. `journalctl`

On systems using `systemd`, `journalctl` can be used to query the system journal.

View logs:

```bash
journalctl
```

View recent logs:

```bash
journalctl -n 50
```

Follow logs in real time:

```bash
journalctl -f
```

View logs for a service:

```bash
journalctl -u nginx
```

Follow a service's logs:

```bash
journalctl -u nginx -f
```

* * *

## 24\. Check Logs for a Specific Time

For example:

```bash
journalctl --since "1 hour ago"
```

Or:

```bash
journalctl --since today
```

This is useful during incident troubleshooting.

* * *

## 25\. Shell Script for Error Monitoring

We can automate error detection:

```bash
#!/bin/bash

LOG_FILE="app.log"

ERROR_COUNT=$(grep -ic "error" "$LOG_FILE")

echo "Error count: $ERROR_COUNT"

if [ "$ERROR_COUNT" -gt 0 ]
then
    echo "Errors found in the log"
else
    echo "No errors found"
fi
```

* * *

## 26\. Shell Script for Log Monitoring

```bash
#!/bin/bash

LOG_FILE="app.log"

if [ ! -f "$LOG_FILE" ]
then
    echo "Log file not found: $LOG_FILE"
    exit 1
fi

echo "Monitoring $LOG_FILE..."

tail -f "$LOG_FILE" | while read -r line
do
    if echo "$line" | grep -qi "ERROR"
    then
        echo "ERROR DETECTED: $line"
    fi
done
```

This continuously watches the log and prints lines containing `ERROR`.

* * *

## 27\. Real-Time DevOps Example — Deployment Logs

Suppose Jenkins deploys an application.

During deployment:

```text
Jenkins
   ↓
Build
   ↓
Deploy
   ↓
Application Start
   ↓
Application Logs
```

We can monitor:

```bash
tail -f /var/log/application.log
```

If we see:

```text
ERROR Database connection failed
```

we can investigate:

*   Database connectivity
    
*   Credentials
    
*   Security rules
    
*   DNS
    
*   Application configuration
    
*   Database availability
    

* * *

## 28\. Log Monitoring with CI/CD

A shell script can check deployment logs:

```bash
#!/bin/bash

LOG_FILE="deployment.log"

if grep -qi "FAILED" "$LOG_FILE"
then
    echo "Deployment failed"
    exit 1
fi

if grep -qi "ERROR" "$LOG_FILE"
then
    echo "Errors detected during deployment"
    exit 1
fi

echo "Deployment log check passed"
```

This can be used as a CI/CD pipeline step.

* * *

## 29\. Log Analysis with Docker

Docker provides container logs using:

```bash
docker logs <container-name>
```

Example:

```bash
docker logs myapp
```

Follow logs:

```bash
docker logs -f myapp
```

Show the last 100 lines:

```bash
docker logs --tail 100 myapp
```

Search errors:

```bash
docker logs myapp 2>&1 | grep -i "error"
```

* * *

## 30\. Log Analysis with Kubernetes

Kubernetes provides:

```bash
kubectl logs <pod-name>
```

Example:

```bash
kubectl logs myapp-pod
```

Follow logs:

```bash
kubectl logs -f myapp-pod
```

For a specific container:

```bash
kubectl logs myapp-pod -c app
```

For a previous container instance:

```bash
kubectl logs myapp-pod --previous
```

This is particularly useful when a container has restarted or crashed.

* * *

## 31\. Real-Time Kubernetes Troubleshooting

Suppose a Pod is restarting.

Start with:

```bash
kubectl get pods
```

Then:

```bash
kubectl describe pod <pod-name>
```

Then:

```bash
kubectl logs <pod-name>
```

If the container restarted:

```bash
kubectl logs <pod-name> --previous
```

Basic flow:

```text
Pod Problem
    ↓
kubectl get pods
    ↓
kubectl describe pod
    ↓
kubectl logs
    ↓
--previous if needed
    ↓
Find Error
    ↓
Troubleshoot
```

* * *

## 32\. Log Analysis Script

Here is a simple reusable script:

```bash
#!/bin/bash

LOG_FILE="$1"

if [ -z "$LOG_FILE" ]
then
    echo "Usage: $0 <log-file>"
    exit 1
fi

if [ ! -f "$LOG_FILE" ]
then
    echo "File not found: $LOG_FILE"
    exit 1
fi

echo "=============================="
echo "Log Analysis Report"
echo "=============================="

echo "Total Lines:"
wc -l < "$LOG_FILE"

echo "Errors:"
grep -ic "error" "$LOG_FILE"

echo "Warnings:"
grep -ic "warn" "$LOG_FILE"

echo "=============================="
```

Run:

```bash
./log-analysis.sh app.log
```

Example output:

```text
==============================
Log Analysis Report
==============================
Total Lines:
1500

Errors:
12

Warnings:
35
==============================
```

* * *

## 33\. Important Log Commands Cheat Sheet

| Command | Purpose |
| --- | --- |
| `cat` | Display file |
| `less` | Read large files |
| `head` | First lines |
| `tail` | Last lines |
| `tail -f` | Real-time monitoring |
| `grep` | Search text |
| `grep -i` | Case-insensitive search |
| `grep -n` | Show line numbers |
| `grep -c` | Count matches |
| `awk` | Process structured text |
| `sed` | Transform/filter text |
| `sort` | Sort output |
| `uniq` | Remove/count duplicates |
| `wc` | Count lines/words/chars |
| `journalctl` | Query systemd journal |

* * *

## 34\. Best Practices

✅ Use `less` for large log files.

✅ Use `grep` to quickly search errors.

✅ Use `tail -f` for real-time monitoring.

✅ Use `awk` for structured log analysis.

✅ Use `journalctl` for systemd services.

✅ Validate that log files exist before processing them.

✅ Avoid storing passwords, API keys, or other secrets in logs.

✅ Use proper log rotation for continuously growing log files.

✅ Use centralized logging solutions for large production environments.

* * *

## 🎯 Key Takeaways

```text
              LOG MANAGEMENT
                    ↓
       ┌────────────┴────────────┐
       ↓                         ↓
    Read Logs               Monitor Logs
       ↓                         ↓
cat / less                 tail -f
       ↓                         ↓
    Search                    Filter
       ↓                         ↓
   grep / awk                sed
       ↓                         ↓
    Analyze                    Count
       ↓                         ↓
 sort / uniq / wc          Error Detection
       ↓                         ↓
       └──────────→ DevOps Automation
```

You should now understand:

✅ Linux log files ✅ `cat` and `less` ✅ `head` and `tail` ✅ `tail -f` ✅ `grep` ✅ `awk` ✅ `sed` ✅ `sort` and `uniq` ✅ `wc` ✅ `journalctl` ✅ Docker logs ✅ Kubernetes logs ✅ Real-time monitoring ✅ Automated error detection ✅ CI/CD log analysis

> **Logs are one of the most important sources of information during DevOps troubleshooting. Shell scripting allows us to automate log searching, filtering, monitoring, and error detection.**

![](https://cdn.hashnode.com/uploads/covers/6878a472fb2990c4c07d4019/58c5c518-a291-44b1-9b08-386ac3b847fc.png align="center")