How to Optimize Node.js App with API Caching

7 min readMay 9, 2023

Caching in a Node.js application using the Redis database

Introduction

Data is a crucial component for most applications, which can be obtained either from a database or an API. When data is fetched from an API, a network request is made to the API server, which responds with the data. However, this process can be time-consuming and can lead to slower application response times for users. Additionally, APIs often have a restriction on the number of requests they can fulfill for an application within a set time period, which is referred to as rate limiting.

To overcome these challenges, we can implement data caching, where the application initially sends a request to the API to retrieve the data, and then stores the data in a cache. Subsequent data requests are then retrieved from the cache instead of sending additional requests to the API.

In this article, we are going to implement caching in a node js application using Redis, but, before we delve into the implementation details, let’s explore what caching is and how it can help the performance of our application.

What is caching?

Caching refers to the practice of storing copies of data in temporary storage or cache, allowing for faster access to the data.

Why do we cache?

The primary goal of caching is to enhance the speed of data access beyond what a database or remote server can provide, particularly for operations that are time-consuming.

Prerequisites

Node.js and NPM — (Download link
Postman/Web Browser — (Download link)
Code Editor (VsCode) — (Download link)
Redis — (Download link)

Getting Started

In this section, we will start building our Node.js application

Step 1 — Setting Up the Project

In this step, we will install the required dependencies for this project and start an Express server. we will create a wiki containing information about Harry Potter characters. We will name the project as hogwarts_wiki.

First, create the directory for the project using the mkdir command:

mkdir hogwarts_wiki

Move into the directory:

cd hogwarts_wiki

Initialize the package.json file using the npm command:

npm init -y

When you run the npm init command, it will create the package.json file with the following content:

Next, we will install express, redis and axios by using the following command:

npm install express redis axios

Open the hogwarts_wiki directory in your code editor. Your folder structure should now look like the one below:

After installing the dependencies, create an Express server.

create a new file index.js and enter the following code:

express-server

save and run the following command:

node index.js

the console will show the following message:

Step 2 — Retrieving Data From a RESTful API Without Caching

Next, you will retrieve data from the API. The Harry Potter Characters API returns information about different available characters

calling api

now we will create a routes /hogwarts/characters and /hogwarts/characters/:id which calls fetchDataFromApi function.

now start the server by running node index.js and open Postman to make a request to the /hogwarts/characters endpoint.

We can observe that the request took 885ms to fetch data, which is a quite long amount of time to retrieve data that doesn’t change frequently. To enhance this, we will implement caching with Redis.

Step 3— Caching API’s using Redis

here we will cache data from the API so that only the initial visit to your app endpoint will request data from an External API, and all the following requests will fetch data from the cache.

To implement caching, we are importing redis module in our project.

const redis = require("redis");

then connecting to Redis by adding the following code:

let redisClient;

(async () => {
  redisClient = redis.createClient();
  redisClient.on("error", (error) => console.error(`Error : ${error}`));
  await redisClient.connect();
})();

and then we have updated our API handler to set and get results from the cache.

we are fetching data from the cache using redisClient.get() method, if data is present, we are setting isCached flag to true and assigning cachedResult to result variable, if cachedResult is empty, then we are calling the external API to fetch data and then setting into the cache using redisClient.set() method

const redisKey = "hogwarts-characters";
let results;
let isCached = false;
const cachedResult = await redisClient.get(redisKey);
if (cachedResult) {
  isCached = true;
  results = JSON.parse(cachedResult);
} else {
  results = await fetchDataFromApi();
  if (!results.length) {
    throw new Error("Data unavailable");
  }
  await redisClient.set(redisKey, JSON.stringify(results));
}

The complete file will now look like this:

Save and exit your file, and run the index.js using the node command:

node index.js

Once the server has started, send API request to http://localhost:3000/hogwarts/characters route and notice that fromCache is still set to false.

Now resend the api to see that this time fromCache is set to true:

To further confirm that the data is stored in Redis, stop your server using CTRL+C. Connect to the Redis server client with the following command:

redis-cli

fetch the data under the key hogwarts-characters

get hogwarts-characters

to exit Redis client run the following command:

exit

Till now we can get/set cached data from an API, we can also set the cache validity.

Step 4— Implementing Cache Validity

When caching data, we don’t know how often the data changes. To set a suitable expiry we should check API Documentation.

Once we have cache duration, convert it into seconds. In this tutorial, we will set the cache duration to 2 minutes or 120 seconds.

To add the cache validity duration, open the index.js file and add the following code:

await redisClient.set(redisKey, JSON.stringify(results), {
  EX: 120,
  NX: true,
});

EX: accepts a value with the cache duration in seconds.
NX: when set to true, it ensures that the set() method should only set a key that doesn’t already exist in Redis.

Step 5— Caching data in middleware

remove the code from APIs which fetches data from cache. after removing the file will now look like this:

now create a new middleware that fetches data from the cache. to create a middleware add the following code:

To make the cacheData() middleware passes control to the API handler when next() is invoked, update the express module’s get() method accordingly:

// fetch character by id
app.get("/hogwarts/characters/:id", cacheData, async (req, res) => {
  try {
    const redisKey = `hogwarts-character-${req.params.id}`;
    results = await fetchDataFromApi(req.params.id);
    if (!results.length) {
      throw new Error("Data unavailable");
    }
    await redisClient.set(redisKey, JSON.stringify(results), {
      EX: 120,
      NX: true,
    });

    return res.status(200).send({
      fromCache: false,
      data: results,
    });
  } catch (error) {
    console.log(error);
    res.status(404).send("Data unavailable");
  }
});

// fetch all characters
app.get("/hogwarts/characters", cacheData, async (req, res) => {
  try {
    const redisKey = "hogwarts-characters";
    results = await fetchDataFromApi();
    if (!results.length) {
      throw new Error("Data unavailable");
    }
    await redisClient.set(redisKey, JSON.stringify(results), {
      EX: 120,
      NX: true,
    });

    return res.status(200).send({
      fromCache: false,
      data: results,
    });
  } catch (error) {
    console.log(error);
    res.status(404).send("Data unavailable");
  }
});

Now, when you visit the /hogwarts/characters endpoint, cacheData() executes first. If data is cached, it will return the response. However, if no data is found in the cache, the handler will be called to retrieve data from API, store it in the cache, and return a response.

The complete file will now look like this:

now run the server using node index.js command to check the API response. The behavior is consistent with how the application worked in the previous section and api response time is reduced drastically to 15ms only as we are getting data from cache once data is cached:

We can now cache data in Redis using middleware.

Conclusion

In this article, we built a Node.js application that retrieves data from an external API and implemented a cache using Redis . Initially, we retrieve the data from the API and store it in the cache. Subsequent requests then retrieve the cached data until it expires.

The complete code for this project can be found in this GitHub repository: https://github.com/kumaranil3921/node-redis-cache-hogwarts-api-tutorial

I hope this article has been helpful and enjoyable for you. If you have any questions or comments, please don’t hesitate to contact me. I would also appreciate 👏 if you like the post, so others can find this too.

If you enjoy reading this post, got help, knowledge, inspiration, and motivation through it, and you want to support me — you can “Buy me A Coffee.” Your support really makes a difference.

Thanks, and see you next time!