Skip to content

Latest commit

 

History

History
382 lines (289 loc) · 9.66 KB

logging_quickref.md

File metadata and controls

382 lines (289 loc) · 9.66 KB

Logging Quick Reference

Using the AWF CLI

The firewall includes a built-in awf logs command for viewing and analyzing Squid proxy logs:

View Logs in Real-Time

# Follow logs in real-time (like tail -f)
awf logs -f

# Follow with PID tracking to identify which process made each request
awf logs -f --with-pid

# Follow with JSON output
awf logs -f --format json

View Historical Logs

# View logs from last run (pretty format, default)
awf logs

# View logs from specific directory
awf logs --source /tmp/squid-logs-1234567890

# Raw format (no colorization)
awf logs --format raw

# JSON format for scripting
awf logs --format json

List Available Log Sources

# Find all preserved log directories
awf logs --list

Generate Statistics

# Show aggregated stats (pretty terminal output)
awf logs stats

# JSON format for scripting
awf logs stats --format json

# Markdown format
awf logs stats --format markdown

Generate Summary for GitHub Actions

# Add summary to GitHub Actions step summary
awf logs summary >> $GITHUB_STEP_SUMMARY

# Also available in JSON and pretty formats
awf logs summary --format json

CLI Options:

Flag Description
-f, --follow Follow log output in real-time (like tail -f)
--format <format> Output format: raw, pretty, json
--source <path> Path to log directory or "running" for live container
--list List available log sources
--with-pid Enrich logs with PID/process info (requires -f)

Statistics Commands:

Command Description
awf logs stats Show aggregated statistics (total requests, allowed/denied counts, per-domain breakdown)
awf logs summary Generate markdown summary (optimized for GitHub Actions)

Manual Docker Commands

For advanced use cases or when the CLI is not available, you can access logs directly via Docker:

HTTP/HTTPS Traffic (Squid)

# View all logs
docker exec awf-squid cat /var/log/squid/access.log

# Follow in real-time
docker exec awf-squid tail -f /var/log/squid/access.log

# Show only blocked requests
docker exec awf-squid grep "TCP_DENIED" /var/log/squid/access.log

# Show only allowed requests
docker exec awf-squid grep "TCP_TUNNEL\|TCP_MISS" /var/log/squid/access.log

Non-HTTP Traffic (iptables)

# From host (requires sudo)
sudo dmesg | grep FW_BLOCKED

# From agent container
docker exec awf-agent dmesg | grep FW_BLOCKED

# Using journalctl (systemd)
sudo journalctl -k | grep FW_BLOCKED

DNS Query Logging (Audit Trail)

# View all DNS queries made by containers
sudo dmesg | grep FW_DNS_QUERY

# Using journalctl (systemd)
sudo journalctl -k | grep FW_DNS_QUERY

# Real-time DNS query monitoring
sudo dmesg -w | grep FW_DNS_QUERY

# Count DNS queries by destination
sudo dmesg | grep FW_DNS_QUERY | grep -oP 'DST=\K[^ ]+' | sort | uniq -c | sort -rn

# Show DNS queries to specific resolver (e.g., 8.8.8.8)
sudo dmesg | grep FW_DNS_QUERY | grep 'DST=8.8.8.8'

Note: DNS queries are logged for audit trail purposes. This helps detect potential DNS tunneling attempts or unusual DNS activity. The log prefix [FW_DNS_QUERY] is used to identify DNS traffic.

Log Format

Squid Log Entry

timestamp client_ip:port domain dest_ip:port proto method status decision url user_agent

Example (blocked):

1760987995.318 172.20.98.20:55960 example.com:443 -:- 1.1 CONNECT 403 TCP_DENIED:HIER_NONE example.com:443 "curl/7.81.0"

Example (allowed):

1760987995.312 172.20.98.20:55952 github.com:443 140.82.116.3:443 1.1 CONNECT 200 TCP_TUNNEL:HIER_DIRECT github.com:443 "curl/7.81.0"

iptables Log Entry

[kernel_time] [PREFIX] IN= OUT=interface SRC=source_ip DST=dest_ip PROTO=protocol SPT=src_port DPT=dst_port UID=uid

Example (blocked UDP):

[1234567.890] [FW_BLOCKED_UDP] IN= OUT=eth0 SRC=172.20.98.20 DST=1.1.1.1 PROTO=UDP SPT=12345 DPT=443 UID=0

Common Queries

Find Blocked Domains

docker exec awf-squid grep "TCP_DENIED" /var/log/squid/access.log | awk '{print $3}' | sort -u

Count Blocked Attempts by Domain

docker exec awf-squid grep "TCP_DENIED" /var/log/squid/access.log | awk '{print $3}' | sort | uniq -c | sort -rn

Find All Unique Accessed Domains

docker exec awf-squid awk '{print $3}' /var/log/squid/access.log | sort -u

Show Last 20 Blocked Requests

docker exec awf-squid grep "TCP_DENIED" /var/log/squid/access.log | tail -20

Find QUIC/HTTP3 Block Attempts

sudo dmesg | grep "FW_BLOCKED_UDP" | grep "DPT=443"

Convert Timestamps to Human-Readable

docker exec awf-squid cat /var/log/squid/access.log | \
  while IFS= read -r line; do
    ts=$(echo "$line" | awk '{print $1}')
    rest=$(echo "$line" | cut -d' ' -f2-)
    echo "$(date -d @${ts} '+%Y-%m-%d %H:%M:%S') $rest"
  done

Export Blocked Domains to File

docker exec awf-squid grep "TCP_DENIED" /var/log/squid/access.log | \
  awk '{print $3}' | sort -u > blocked_domains.txt

PID/Process Tracking

The awf logs command supports real-time PID tracking using the --with-pid flag (see "Using the AWF CLI" section above for examples).

When enabled, logs include:

Field Description
pid Process ID that made the request
cmdline Full command line of the process
comm Short command name (from /proc/[pid]/comm)
inode Socket inode for advanced correlation

Limitations

  • Real-time only: PID tracking requires -f (follow mode)
  • Linux only: Requires /proc filesystem access
  • Ephemeral: Process must still be running; historical logs cannot be enriched

Use Cases

  • Identify which MCP server or tool made a specific request
  • Trace data exfiltration attempts to specific commands
  • Audit agent network behavior for compliance

Decision Codes

Code Meaning Action
TCP_DENIED:HIER_NONE Blocked Domain not in allowlist
TCP_TUNNEL:HIER_DIRECT Allowed HTTPS tunneled successfully
TCP_MISS:HIER_DIRECT Allowed HTTP request forwarded

Status Codes

Code Meaning
200 Request allowed and successful
403 Request blocked (domain not in allowlist)
502 Bad Gateway (destination unreachable)
503 Service Unavailable (DNS resolution failed)

Real-Time Monitoring

Watch Blocked Attempts

watch -n 1 'docker exec awf-squid grep TCP_DENIED /var/log/squid/access.log | tail -20'

Count by Domain (Updates Every 5 Seconds)

watch -n 5 'docker exec awf-squid grep TCP_DENIED /var/log/squid/access.log | awk "{print \$3}" | sort | uniq -c | sort -rn | head -10'

Follow New Entries

docker exec awf-squid tail -f /var/log/squid/access.log | \
  grep --line-buffered "TCP_DENIED" | \
  while read -r line; do
    echo "[BLOCKED] $line"
  done

Filtering by Field

By Client IP

docker exec awf-squid awk '$2 ~ /^172\.20\.98\.20:/' /var/log/squid/access.log

By Domain Pattern

docker exec awf-squid grep "github\.com" /var/log/squid/access.log

By Status Code

docker exec awf-squid awk '$7 == 403' /var/log/squid/access.log

By Time Range (Last Hour)

NOW=$(date +%s)
HOUR_AGO=$((NOW - 3600))
docker exec awf-squid awk -v start=$HOUR_AGO '$1 >= start' /var/log/squid/access.log

By User Agent

docker exec awf-squid grep "curl" /var/log/squid/access.log

Statistics

Manual Statistics Queries

Total Requests

docker exec awf-squid wc -l /var/log/squid/access.log

Blocked vs Allowed Count

echo "Blocked: $(docker exec awf-squid grep -c TCP_DENIED /var/log/squid/access.log)"
echo "Allowed: $(docker exec awf-squid grep -cE 'TCP_TUNNEL|TCP_MISS' /var/log/squid/access.log)"

Top 10 Accessed Domains

docker exec awf-squid awk '{print $3}' /var/log/squid/access.log | \
  sort | uniq -c | sort -rn | head -10

Unique Client IPs

docker exec awf-squid awk '{split($2,a,":"); print a[1]}' /var/log/squid/access.log | sort -u

Integration Examples

Export to CSV

docker exec awf-squid cat /var/log/squid/access.log | \
  awk 'BEGIN{OFS=","} {print $1,$2,$3,$4,$5,$6,$7,$8,$9,$10}' > access.csv

Send to Syslog

docker exec awf-squid tail -f /var/log/squid/access.log | \
  logger -t awf -n syslog.example.com -P 514

Alert on Blocked Attempt

docker exec awf-squid tail -f /var/log/squid/access.log | \
  grep --line-buffered "TCP_DENIED" | \
  while read -r line; do
    # Send alert (email, Slack, etc.)
    echo "ALERT: Blocked access detected: $line" | mail -s "Firewall Alert" admin@example.com
  done

Troubleshooting

No Logs Appearing

# Check container is running
docker ps | grep awf-squid

# Check volume mount
docker inspect awf-squid | grep -A5 Mounts

# Check Squid is logging
docker exec awf-squid grep access_log /etc/squid/squid.conf

Log File Too Large

# Check size
docker exec awf-squid ls -lh /var/log/squid/access.log

# Rotate manually
docker exec awf-squid squid -k rotate

# Clear logs (use with caution)
docker exec awf-squid sh -c "> /var/log/squid/access.log"

Parse Errors

# Validate log format
docker exec awf-squid grep logformat /etc/squid/squid.conf

# Check for corrupted lines
docker exec awf-squid awk 'NF != 10' /var/log/squid/access.log

See Also