Guide to Redis on GCP : Memorystore

Until now I was happy inside my sweet little bubble of GCP where i used BigQuery for providing data warehouse and analytics services and if need be, switch to Cloud SQL for handling transactional services. I was having a happy life, until I came across a use case where time complexity was of utmost importance. Basically, the use case to expose data present in GCP through an API which responds back in under 100 ms. Which is like 1/10th of a second. And, to be frank in the start it sounded insane to me.

I had discussions with a lot of my colleagues regarding this use case and even whether designing such an API was possible or not. After having a lot of these discussions, it was at least clear to me that i will have to leave BigQuery aside for this use case and explore a faster data service which can help me achieve the needed performance. BigQuery is meant for storing large data and through it gives you query performance and everything, it will not give you the response in 100 ms. Also, another thing about BigQuery is that it might take variable amount of times to run the same query depending upon the current slots/compute being utilized inside your project.

Finally, someone suggested me to explore MemoryStore (Redis on GCP) which is a no-sql, in-memory database. And i thought that it made sense, since Redis stores all the data in its cache and is able to serve requests quickly through that. So let’s start with our journey of setting up Memory Store and getting started with it. You can always learn more about MemoryStore in their official documentation.

P.S : After implementation, i was able to fetch the data in under 8 ms. I know right, its super-fast. (for about 10k records of total 10 MB size)

Note : All interactions with Redis in this article will be Python oriented

Step 1 : Pre-requisites

Before we get in to MemoryStore we first need to create a VPC in our GCP project. You will not be able to create a Redis instance until you have at least 1 VPC in your project.

If you already know how to create a VPC, GREAT !!! Go ahead and create one.
If you are someone like, please follow this link to learn how to create a VPC Network. It’s fairly simple and hardly takes 5 minutes.