Log-Munger

Extracts structured fields from log lines, akin to grok for Logstash, but as a standalone Perl distribution and a log_munger CLI, with no Elasticsearch/Logstash stack required.

Feed it a decoded log record (a hash) or a raw log line, and it runs the record through a set of YAML rule files. The first rule that matches returns the named captures from its regexp, optionally broken down further (decompose), coerced (convert), and enriched with GeoIP (geoip).

$ echo '{"PROGRAM":"sshd","MESSAGE":"Accepted publickey for neti from 192.0.2.5 port 54321 ssh2: RSA SHA256:AbCd"}' \
    | log_munger munge --rules sshd
---
ssh_key_fingerprint: SHA256:AbCd
ssh_key_type: RSA
ssh_method: publickey
ssh_src_ip: 192.0.2.5
ssh_src_port: 54321
ssh_user: neti

Why

Logstash is a dumpster fire to deal with and this allows easy parsing of log stuff in a reusable manner for Perl.

Install

From source

perl Makefile.PL
make
make test
make install

FreeBSD

pkg install p5-App-cpanminus p5-YAML-LibYAML p5-JSON p5-File-ShareDir p5-File-Slurp \
    p5-Template-Toolkit p5-Hash-Merge p5-App-Cmd p5-Algorithm-Dependency
cpanm Log::Munger

Debian

apt-get install cpanminus libyaml-libyaml-perl libjson-perl libfile-sharedir-perl \
    libfile-slurp-perl libtemplate-perl libhash-merge-perl libapp-cmd-perl \
    libalgorithm-dependency-perl
cpanm Log::Munger

GeoIP

GeoIP enrichment additionally needs IP::Geolocation::MMDB. It is only recommended rather than required, and is loaded only when you actually pass a database.

Quick start

As a CLI

# what rule files are available?
log_munger list

# run one item through a rule file and dump the fields
log_munger munge --rules sshd \
    --string '{"PROGRAM":"sshd","MESSAGE":"Failed password for root from 203.0.113.7 port 44444 ssh2"}'

# a gateless rule (like http_access_logs) works on a raw line with --raw
log_munger munge --rules http_access_logs --raw \
    --string '127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /a.gif HTTP/1.0" 200 2326'

# see WHICH rule/pattern fired
log_munger explain --rules sshd --string '{"PROGRAM":"sshd","MESSAGE":"..."}'

# stream NDJSON in, enriched NDJSON out
cat events.ndjson | log_munger enrich --rules sshd --rules postfix > enriched.ndjson

As a library

use Log::Munger;

my $munger = Log::Munger->new( rules => [ 'sshd', 'postfix' ] );

# a decoded record (e.g. from journald / syslog-ng JSON output)
my $fields = $munger->process_item( item => {
    PROGRAM => 'sshd',
    MESSAGE => 'Failed password for root from 203.0.113.7 port 44444 ssh2',
} );
# $fields = { ssh_method => 'password', ssh_user => 'root', ssh_src_ip => '203.0.113.7', ... }

# a bare string is matched as the MESSAGE field
my $access = $munger->process_item( item => $raw_apache_line );

Bundled rule files

The distribution ships a primitive library plus ready-to-use rule files (installed into the dist share dir):

Rule file	Matches
base	The primitive library (IP, WORD, TIMESTAMP_ISO8601, etc). No rules of its own; included by the others

Authentication and privilege

Rule file	Matches
sshd	OpenSSH auth events, plus the scan traffic that never reaches auth
dropbear	Dropbear SSH server auth events
pam / su / sudo / login	PAM, su, sudo, and console login authentication
nslcd	The LDAP name-service daemon behind libnss_ldap / pam_ldap
luci	LuCI (the OpenWrt web interface) authentication
xscreensaver	XScreenSaver lock-screen unlock attempts
polkit	polkit authorization decisions — the third way to gain privilege
auditd	Linux audit daemon records, including SELinux AVC and AppArmor
slapd	OpenLDAP binds, searches, and result codes
openvpn	OpenVPN handshakes, certificate verification, and auth failures
strongswan	strongSwan/charon IPsec tunnels, IKE, and EAP
freeradius	RADIUS logins — the auth behind 802.1X and many VPNs
samba	Samba (smbd/nmbd/winbindd) auth audit and share access
nfs	rpc.mountd mount requests, allowed and refused

Mail

Rule file	Matches
postfix	Postfix mail log (smtpd, qmgr, delivery, etc)
exim	Exim mail log, including SMTP AUTH and TLS failures
dovecot	Dovecot IMAP/POP3
sendmail	Sendmail transactions, AUTH failures, and rejections
rspamd	Rspamd scan results — score, action, and symbols
spamd	SpamAssassin scan results
clamav	ClamAV detections and signature-database freshness
opendkim / opendmarc	DKIM and DMARC results, joined to the MTA by queue id
ssmtp	sSMTP, the send-only MTA
sympa	The Sympa mailing list manager's daemons

Web, proxy, and network services

Rule file	Matches
http_access_logs	Apache/nginx Common, Combined, and the vhost-prefixed variants
http_error_logs	Apache/nginx error logs
haproxy	HAProxy HTTP and TCP traffic, plus health-check state changes
squid	Squid access.log (all three shipped logformats) and cache.log
vsftpd / proftpd	FTP logins and transfers
php_fpm	PHP-FPM pool health: dying children, slow requests, exhaustion
cups	CUPS printing — both error_log and access_log
named / unbound	DNS server logs
resolved	systemd-resolved upstream health and DNSSEC failures
dnsmasq	dnsmasq's DNS, DHCP, and TFTP logging
dhcpd	ISC DHCP server leases
hostapd	hostapd wireless association events (the access-point side)
wpa_supplicant	Wireless client associations and auth failures
networkmanager / networkd	Device state machines, carrier, and DHCP leases
chrony / ntpd / timesyncd	Time synchronization daemons
xinetd	Superserver dispatch: who reached which service, and refusals
snmpd	Net-SNMP connections — who is querying the agent
asterisk	Asterisk PBX
avahi	Avahi mDNS/DNS-SD responder
lldpd	lldpd/lldpcli link-layer neighbor discovery
netifd / odhcpd	OpenWrt's network interface and DHCPv6/router-advertisement daemons
huawei	Huawei VRP devices — S-series switches, AR routers, USG firewalls
tor	Tor daemon

Firewalls

Rule file	Matches
netfilter	iptables/nftables/UFW kernel firewall logs
ipfw	FreeBSD ipfw firewall logs
pf	OpenBSD/FreeBSD pf, read from tcpdump -r /var/log/pflog
fortinet	FortiGate/FortiOS key=value logs
sonicwall	SonicWall/SonicOS key=value logs
fail2ban	fail2ban ban/unban actions
kur / ereshkigal	Rules for ereshkigal and it's related backend bit, kur.
galla / baphomet	Rules for baphomet and it's related backend bit, galla.
daemonlogger	daemonlogger's rolling packet capture
virani	Virani, which carves per-request pcaps out of daemonlogger's capture set
mojo_cape_submit	mojo_cape_submit/nergal rules for CAPEv2, submission endpoint receiving for CAPE::Utils
suricata_extract_submit	suricata_extract_submit rules for CAPEv2, sample suricata extract shipper for CAPE::Utils

Databases, storage, and the host itself

Rule file	Matches
mysql	MySQL/MariaDB access denials and aborted connections
postgresql	PostgreSQL authentication and connection logging
mongodb	MongoDB structured (JSON) logging
kernel	Linux and FreeBSD kernel ring buffer — OOM, filesystem, I/O, SYN floods
smartd	Disk health: failing attributes, bad sectors, temperature
zed	ZFS Event Daemon — checksum errors, vdev states, resilvers
docker	Docker/containerd logfmt output
libvirt	libvirt daemons, monolithic and modular
systemd / logind	systemd unit lifecycle and systemd-logind sessions
dbus	D-Bus message bus (dbus-daemon and dbus-broker)
cron / atd	Scheduled job execution
shutdown	shutdown / reboot / halt
fwupd	fwupd firmware updates, daemon and clients
pkg	FreeBSD pkg(8) package changes
rc	FreeBSD rc(8) service-script warnings
syslog_daemon	rsyslog and syslog-ng internals — rate limiting and stalled outputs

Documentation

Full documentation lives in docs/:

• Getting started :: Install, first munge, the log-record model.
• CLI reference :: Every log_munger subcommand and option.
• Rule-file format :: The YAML schema — vars, rules, gate, decompose, convert, geoip, and tests.
• Writing a rule file :: A step-by-step tutorial.
• Primitive library :: The named patterns in base.yaml.
• Perl API :: Log::Munger and the supporting modules.
• Architecture :: How loading, templating, and matching fit together.
• GeoIP enrichment :: Enriching captured addresses.
• Grok migration :: Converting existing grok patterns.

Authors and license

Copyright (c) 2026 Zane C. Bowers-Hadley <vvelox at vvelox.net>.

This is free software, licensed under the GNU Lesser General Public License, Version 3. See LICENSE.

Bugs and feature requests: GitHub issues or bug-log-munger at rt.cpan.org.