Laravel Lazy Collections

Processing large datasets crashes applications. Laravel 5.8 introduces lazy collections, streaming records without loading everything into memory. At ZIRA Software, lazy collections enabled processing millions of records on standard servers.

The Memory Problem

Traditional approach (loads all into memory):

// Bad for large datasets
$users = User::all();  // Loads 100,000 users into memory

foreach ($users as $user) {
    processUser($user);
}
// Memory exhausted!

Chunk approach (better but still loads chunks):

// Better but loads 1000 records at a time
User::chunk(1000, function ($users) {
    foreach ($users as $user) {
        processUser($user);
    }
});

Lazy Collections

Stream records one at a time:

use Illuminate\Support\LazyCollection;

// Loads records one by one using cursor
User::cursor()->each(function ($user) {
    processUser($user);
});

Memory comparison:

// Regular collection: ~100MB for 10,000 records
$users = User::all();

// Lazy collection: ~10MB constant memory
$users = User::cursor();

Creating Lazy Collections

From database cursor:

$users = User::cursor();  // Returns LazyCollection

$activeUsers = $users->filter(function ($user) {
    return $user->isActive();
});

$activeUsers->each(function ($user) {
    echo $user->name;
});

From generator:

$collection = LazyCollection::make(function () {
    for ($i = 1; $i <= 1000000; $i++) {
        yield $i;
    }
});

$collection->filter(function ($number) {
    return $number % 2 === 0;
})->take(10)->each(function ($number) {
    echo $number;
});
// Only processes what's needed!

From file:

// Process huge CSV file line by line
$collection = LazyCollection::make(function () {
    $handle = fopen('huge-file.csv', 'r');

    while (($line = fgets($handle)) !== false) {
        yield $line;
    }

    fclose($handle);
});

$collection->skip(1)  // Skip header
    ->map(function ($line) {
        return str_getcsv($line);
    })
    ->filter(function ($row) {
        return isset($row[2]) && $row[2] > 100;
    })
    ->each(function ($row) {
        // Process row
    });

Lazy Collection Methods

All collection methods work lazily:

$users = User::cursor();

// Filter, map, reduce - all lazy
$result = $users
    ->filter(fn($user) => $user->isActive())
    ->map(fn($user) => $user->email)
    ->unique()
    ->take(100)
    ->values();

// Only processes until 100 unique emails found

Chunking with lazy collections:

$users = User::cursor();

$users->chunk(100)->each(function ($chunk) {
    // Process batch of 100
    sendBulkEmail($chunk);
});

Real-World Examples

Export large dataset to CSV:

public function exportUsers()
{
    $headers = ['Name', 'Email', 'Created'];

    return response()->streamDownload(function () use ($headers) {
        $file = fopen('php://output', 'w');
        fputcsv($file, $headers);

        User::cursor()->each(function ($user) use ($file) {
            fputcsv($file, [
                $user->name,
                $user->email,
                $user->created_at->toDateString(),
            ]);
        });

        fclose($file);
    }, 'users.csv');
}

Process orders with related data:

Order::with('items', 'customer')
    ->whereDate('created_at', today())
    ->cursor()
    ->each(function ($order) {
        generateInvoice($order);
        sendInvoiceEmail($order);
    });

Generate reports from logs:

$stats = LazyCollection::make(function () {
    $handle = fopen('storage/logs/laravel.log', 'r');

    while (($line = fgets($handle)) !== false) {
        yield $line;
    }

    fclose($handle);
})
->filter(function ($line) {
    return strpos($line, 'ERROR') !== false;
})
->map(function ($line) {
    // Extract error details
    return parseErrorLine($line);
})
->groupBy('type')
->map(function ($errors) {
    return $errors->count();
});

Cursor Improvements

Better cursor implementation:

// Before 5.8: Multiple queries
User::chunk(1000, function ($users) {
    // Runs SELECT * FROM users LIMIT 1000 OFFSET 0
    // Then SELECT * FROM users LIMIT 1000 OFFSET 1000
    // etc...
});

// After 5.8: Single streaming query
User::cursor()->each(function ($user) {
    // Single query with cursor
    // Fetches records one at a time
});

When to Use Lazy Collections

Use lazy collections when:

• Processing thousands+ records
• Memory is constrained
• Exporting large datasets
• Processing files line by line
• Need to stop early (with take/filter)

Use regular collections when:

• Small datasets (less than 1000 records)
• Need count or statistics upfront
• Multiple iterations needed
• Random access required

Performance Tips

1. Eager load relationships:

// Bad: N+1 queries
User::cursor()->each(function ($user) {
    echo $user->posts->count();  // Query per user
});

// Good: Single query
User::with('posts')->cursor()->each(function ($user) {
    echo $user->posts->count();
});

2. Use take() to limit processing:

// Stop after finding 10 matches
$users = User::cursor()
    ->filter(fn($user) => $user->score > 90)
    ->take(10);

3. Combine with queued jobs:

User::cursor()
    ->chunk(100)
    ->each(function ($chunk) {
        ProcessUsersJob::dispatch($chunk);
    });

Other 5.8 Eloquent Improvements

hasOneThrough relationship:

class Country extends Model
{
    public function postOwner()
    {
        return $this->hasOneThrough(User::class, Post::class);
    }
}

New where methods:

// whereDate, whereMonth, whereDay, whereYear, whereTime
User::whereDate('created_at', '2019-01-15')->get();
User::whereMonth('created_at', 1)->get();

Conclusion

Lazy collections revolutionize large dataset processing. Stream millions of records with constant memory usage. Essential for exports, reports, and batch processing.

Processing large datasets? Contact ZIRA Software for performance optimization consultation.