Automating Mass ERP Audits with Distributed Queues

When managing a school network of over 12,000 students, the end-of-month audit cycle is the ultimate stress test for any ERP. At SLS, our administrative team needed to generate bulk financial challenges, report cards, and audit trails for every single student simultaneously.

The "White Screen of Death"

The initial implementation used a standard synchronous PDF generation flow. When a user clicked "Generate Monthly Audits," the server would attempt to query thousands of ledger entries, calculate balances, and render a PDF using DOMpdf.

The result? 504 Gateway Timeouts and the dreaded PHP memory exhaustion errors. Processing 12,000 documents consumes gigabytes of RAM that no single web request should ever touch.

The Distributed Solution

I architected a background processing pipeline using Laravel Queues backed by Redis and Horizon. Instead of processing the batch in-request, we dispatched a "GenerateAudit" job for every student into a high-priority queue.

// Dispatching distributed audit jobs with chunking
public function generateBatch()
{
    Student::active()->chunkById(500, function ($students) {
        foreach ($students as $student) {
            GenerateAuditPdf::dispatch($student)
                ->onQueue('audits')
                ->delay(now()->addSeconds(2)); // Throttling for IO stability
        }
    });

    return response()->json(['message' => 'Batch generation initiated.']);
}

Supervisor and Horizon Monitoring

We deployed 8 dedicated Laravel Supervisor workers on an independent EC2 instance. This decoupled the PDF generation (CPU intensive) from the main web application (IO intensive). Using Horizon, we could monitor throughput in real-time, observing as the queue cleared thousands of documents in minutes without a single dropped packet.

Financial Data Integrity

Automation is dangerous without verification. Every generated PDF was checksummed and its metadata stored in a hash-table. This allowed accountants to immediately identify any missing or corrupted files in a 12,000-deep batch, ensuring that the double-entry books always matched the physical records.

By moving from synchronous blocks to distributed queues, we turned a week-long manual accounting headache into a 30-minute automated background task. Scaling isn't always about more power; it's about better choreography.