Queue-driven processing
The upload request starts the workflow, while Laravel workers handle validation, transformation, database processing, and report generation in the background.
I rebuilt a memory-heavy Excel import as a background pipeline that processes large datasets without blocking the website or requiring users to keep their browser open.
Processing an entire workbook inside a web request caused high memory usage, timeouts, and poor visibility. A failure could stop the full process, and restarting it repeated work that had already succeeded.
The upload request starts the workflow, while Laravel workers handle validation, transformation, database processing, and report generation in the background.
Files are split into bounded chunks and grouped into chains and batches so dependent stages run in order while individual units remain retryable.
XML streaming reads only the header and sample rows needed for a preview instead of parsing an entire million-row workbook.
Upload states and WebSocket updates show progress, while failed stages are recorded separately and validation reports are stored in S3.
The workflow contained dependent stages: validation had to finish before processing, and post-processing had to wait until the main records were ready. I combined Laravel job chains and batches to preserve that order while keeping work inside each stage small and recoverable.
Large data workflows become easier to scale and support when they are split into small, observable, retryable jobs instead of one long-running process.
PHP, Laravel, Laravel Queues, Laravel Batch Jobs, Redis, Amazon S3, Laravel Vapor, Excel and XML streaming, React, TypeScript, and WebSockets.