Lumberjack v2 (Embedded & Flash-Optimized Fork)

An enhanced, high-performance Go rolling logger based on natefinch/lumberjack, specifically engineered for embedded systems, telemetry units, industrial controllers, and flash storage (eMMC, SD cards, SPI-NAND/NOR).

---

🎯 Why This Fork Exists

Standard logging packages and upstream Lumberjack are designed for server environments with hard disks or SSDs. In embedded Linux systems and IoT devices, standard logging causes critical problems:

1. Severe Flash-Memory Wear (Write Amplification):
   • Flash storage cannot write byte-by-byte; it writes in Pages (4 KB – 16 KB) and erases in Blocks (128 KB – 4 MB).
   • Writing small log lines (50–200 bytes) frequently triggers continuous flash page rewrites, garbage collection in the Flash Translation Layer (FTL), and heavy filesystem metadata churn (mtime, inode updates).
2. Double I/O Overhead with Post-Rotation Compression:
   • Upstream Lumberjack writes uncompressed log files to disk, and only after rotation reads the entire file back from flash to compress it and rewrite it as .gz. On flash storage, this causes double wear and high CPU spikes.
3. Missing Hardware fsync & Buffered Control:
   • Standard loggers lack integrated, thread-safe user-space write buffering and hardware-sync (fsync) guarantees during graceful system shutdown.

---

🚀 Key Advantages & Features

Feature	Upstream (natefinch/lumberjack)	This Fork (themulle/lumberjack/v2)
Go Module v2 Support	Partial / gopkg.in	Native github.com/themulle/lumberjack/v2
Dependencies	0 external dependencies	0 external dependencies (100% Go StdLib)
Direct Gzip Compression	❌ (Post-rotation only)	✅ CompressImmediately (Streams into *gzip.Writer)
Flash-Wear Reduction	❌ (Writes directly to disk)	✅ BufferedLogger (In-Memory RAM buffer up to 99% less I/O)
Periodic Auto-Flush	❌ None	✅ WithFlushInterval (Guaranteed bounded latency)
Emergency Sync Write	❌ None	✅ WriteSync(p) for immediate FATAL/Panic flush
Hardware fsync on Close	❌ Standard file.Close()	✅ os.File.Sync() (Hardware-level flush before close)
Zap / Logger Compatibility	Partial (io.WriteCloser)	✅ Full (io.WriteCloser, Sync(), Flush())

---

📦 Installation

go get github.com/themulle/lumberjack/v2

Drop-in Replacement for Existing Projects

If your project or dependencies already import gopkg.in/natefinch/lumberjack.v2 and you want to use this fork without refactoring existing imports:

// in your go.mod
replace gopkg.in/natefinch/lumberjack.v2 => github.com/themulle/lumberjack/v2 v2.4.0

---

💡 Usage Examples

1. Flash-Safe Embedded Logging with BufferedLogger (Recommended)

Combines in-memory write buffering with direct gzip streaming:

package main

import (
    "log"
    "time"

    "github.com/themulle/lumberjack/v2"
)

func main() {
    // 1. Configure underlying rolling logger
    rawLogger := &lumberjack.Logger{
        Filename:            "/data/logs/telemetry.log",
        MaxSize:             10,   // megabytes before rotation
        MaxBackups:          5,    // keep up to 5 archived files
        MaxAge:              30,   // days
        CompressImmediately: true, // stream directly into gzip (70-90% space reduction)
    }

    // 2. Wrap with BufferedLogger for flash-wear reduction
    flashLogger := lumberjack.NewBufferedLogger(
        rawLogger,
        lumberjack.WithBufferSize(64*1024),          // 64 KB RAM buffer (matches flash blocks)
        lumberjack.WithFlushInterval(30*time.Second), // auto-flush every 30s
    )
    defer flashLogger.Close() // flushes remaining buffer & executes hardware fsync

    log.SetOutput(flashLogger)
    log.Println("Embedded telemetry started with flash-safe logging")
}

---

2. Emergency Flush for Critical Errors (WriteSync)

For FATAL, PANIC, or critical event logs where data must hit persistent storage immediately:

// Writes and immediately flushes through user-space buffers to the underlying target
flashLogger.WriteSync([]byte("CRITICAL: Watchdog threshold exceeded, initiating safe stop\n"))

---

3. Integration with Uber Zap Logger

BufferedLogger and Logger implement Sync() error, making them directly compatible with uber-go/zap:

package main

import (
    "time"

    "github.com/themulle/lumberjack/v2"
    "go.uber.org/zap"
    "go.uber.org/zap/zapcore"
)

func NewZapLogger() (*zap.Logger, func(), error) {
    rawLogger := &lumberjack.Logger{
        Filename:            "/var/log/app.log",
        MaxSize:             5,
        MaxBackups:          3,
        CompressImmediately: true,
    }

    buffered := lumberjack.NewBufferedLogger(
        rawLogger,
        lumberjack.WithBufferSize(64*1024),
        lumberjack.WithFlushInterval(15*time.Second),
    )

    writeSyncer := zapcore.AddSync(buffered)
    encoderConfig := zap.NewProductionEncoderConfig()
    core := zapcore.NewCore(
        zapcore.NewJSONEncoder(encoderConfig),
        writeSyncer,
        zap.InfoLevel,
    )

    logger := zap.New(core)
    cleanup := func() {
        _ = logger.Sync()
        _ = buffered.Close()
    }

    return logger, cleanup, nil
}

---

4. Graceful Shutdown & Signal Handling Guarantee

package main

import (
    "context"
    "log"
    "os"
    "os/signal"
    "syscall"

    "github.com/themulle/lumberjack/v2"
)

func main() {
    logger := lumberjack.NewBufferedLogger(&lumberjack.Logger{
        Filename:            "/data/logs/app.log",
        MaxSize:             10,
        CompressImmediately: true,
    })
    defer logger.Close() // Guaranteed fsync on exit

    log.SetOutput(logger)

    // Catch SIGINT and SIGTERM for graceful systemd / container shutdown
    ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
    defer stop()

    <-ctx.Done()
    log.Println("Shutting down: all buffers will be flushed and synced to flash storage")
}

---

⚙️ Configuration Reference

Logger Struct Options

Field	Type	Default	Description
Filename	string	<procname>-lumberjack.log	Target file path for the active log file.
MaxSize	int	100	Maximum uncompressed payload size in megabytes before rotation.
MaxAge	int	0 (no limit)	Maximum days to retain old backup log files.
MaxBackups	int	0 (all)	Maximum number of rotated backup files to retain.
LocalTime	bool	false (UTC)	If true, timestamps in backup filenames use local time.
Compress	bool	false	Post-rotation compression (used when CompressImmediately is false).
CompressImmediately	bool	false	New: Writes directly through *gzip.Writer into the file for 70–90% storage savings.

BufferedLogger Options

Functional Option	Default	Description
WithBufferSize(size int)	64 KB (64 * 1024)	In-memory RAM buffer size. Batches small writes into flash-block aligned chunks.
WithFlushInterval(d time.Duration)	30s	Auto-flush interval. Flushes in-memory data if no activity occurs within this window.

---

🛡️ Flash Memory Protection Architecture

flowchart TD
    subgraph Application ["Application Goroutines"]
        W1[Log Write 1] --> BL[BufferedLogger]
        W2[Log Write 2] --> BL
        W3[Critical Log] -->|WriteSync| BL
    end

    subgraph RAM ["RAM (0 Flash Writes)"]
        BL --> Buf["In-Memory Buffer (bufio.Writer, e.g. 64 KB)"]
        Ticker["Auto-Flush Ticker (e.g. 30s)"] -.->|Trigger| Buf
    end

    subgraph FlashStorage ["Persistent Flash Storage"]
        Buf -->|1. Buffer Full / 2. Ticker / 3. Close| GZ["gzip.Writer (Direct Stream)"]
        GZ --> FSYNC["os.File.Sync() (Hardware fsync)"]
        FSYNC --> FILE["/data/logs/app.log"]
    end

Loading

1. User Space (RAM): Writes are held in an in-memory buffer protected by sync.Mutex.
2. Flash Page Alignment: When the buffer reaches 64 KB (or during auto-flush), a single sequential block is written.
3. Zero Double-Wear: CompressImmediately streams gzip directly, avoiding the need to read and rewrite files from flash during rotation.
4. Clean Power-Down: Calling Close() flushes user-space buffers, finalizes the gzip stream, and executes os.File.Sync() (fsync syscall) to guarantee physical persistence.

---

📜 License

MIT License – see the LICENSE file for details.