Handling Large Data with Pagination in Node.js

When building web applications, one common challenge is handling large datasets. Fetching thousands of records at once can slow down your server and make your application unresponsive. This is where pagination comes in handy. Pagination allows you to split your data into smaller, manageable chunks and display them page by page. In this blog post, we’ll explore how to implement pagination in Node.js with simple examples.

What is Pagination?

Pagination is a technique to divide large datasets into smaller pages. For example, if you have 1,000 users in your database, instead of showing all users at once, you can display 10 users per page. Users can then navigate through pages to see more records.

Benefits of pagination:

• Improves server performance
• Reduces response time
• Enhances user experience

How Pagination Works?
Pagination usually requires two pieces of information from the client (front-end):

1. Page number – Which page of data the user wants to see.
2. Page size – How many records should appear on each page.

For example:
GET /users?page=2&limit=10

• page=2 means the user wants the second page.
• limit=10 means each page shows 10 records.

Pagination Example in Node.js
Let’s create a simple Node.js application with Express and MongoDB to demonstrate pagination.

Step 1: Setup Node.js and Express
npm init -y
npm install express mongoose

Create server.js:

const express = require('express');
    const mongoose = require('mongoose');
   
    const app = express();
   
    // Connect to MongoDB
    mongoose.connect('mongodb://localhost:27017/pagination_example', {
        useNewUrlParser: true,
        useUnifiedTopology: true
    });
   
    // User schema
    const userSchema = new mongoose.Schema({
        name: String,
        email: String
    });
   
    const User = mongoose.model('User', userSchema);
   
    // Pagination API
    app.get('/users', async (req, res) => {
        const page = parseInt(req.query.page) || 1;
        const limit = parseInt(req.query.limit) || 10;
   
        const skip = (page - 1) * limit;
   
        try {
            const users = await User.find().skip(skip).limit(limit);
            const total = await User.countDocuments();
   
            res.json({
                page,
                limit,
                totalPages: Math.ceil(total / limit),
                totalUsers: total,
                users
            });
        } catch (err) {
            res.status(500).json({ message: err.message });
        }
    });
   
    app.listen(3000, () => console.log('Server running on port 3000'));

Step 2: Test Your API
You can test the API using a browser or Postman:
http://localhost:3000/users?page=1&limit=5

Response example:

{
    "page": 1,
    "limit": 5,
    "totalPages": 20,
    "totalUsers": 100,
    "users": [
        { "_id": "1", "name": "Alice", "email": "alice@example.com" },
        { "_id": "2", "name": "Bob", "email": "bob@example.com" }
    ]
 }

Tips for Efficient Pagination

• Use indexing in the database: This makes queries faster for large datasets.
• Limit the maximum page size: Avoid fetching too many records at once.
• Use cursor-based pagination for very large datasets: This is more efficient than skipping many records.

Summary
Pagination is a simple but powerful way to handle large datasets in Node.js applications. By splitting your data into smaller chunks, you can improve server performance and create a better experience for your users. Using MongoDB or any other database, you can implement pagination easily with skip and limit, or advanced techniques like cursor-based pagination for very large data.