Building a Threads or Instagram-like Feed System with Redis

I’m exploring how to implement a Threads- or Instagram-like feed system, and I quickly realized that caching is going to be a key part of it. I know very little about Redis to start with, but it seemed like it might be suitable for this kind of problem. I wasn’t exactly sure how it should be used, so I spent some time figuring that out. Here’s what I learned.

Understanding the problem

A feed system like Threads or Instagram has a few characteristics that make it tricky:

• Users expect the feed to load almost instantly.
• There are far more reads than writes.
• One post can end up in hundreds or thousands of feeds if a user has many followers.
• The feed needs to be ordered, usually by time or some kind of ranking.

Because of this, the naive approach of querying the database every time someone scrolls won’t scale well. That’s where Redis comes in.

Redis basics

At a high level, Redis is an in-memory data store. It keeps data in RAM, which makes reads and writes extremely fast. Unlike a simple key-value cache, Redis supports data structures like lists, sets, hashes, and sorted sets.

This is useful for feed systems. For example, you can store feeds as ordered sets of post IDs, store post content in hashes, and maintain counters for likes or comments.

Redis also supports key expiration for automatic cleanup and pub/sub for real-time updates. For this use case, though, the focus is on storing and serving feeds efficiently.

How Redis fits in

The core idea is to keep a precomputed feed for each user in Redis. When a user creates a post, it gets pushed into the feeds of their followers. Then when someone opens the app, their feed can be read directly from Redis instead of querying the database.

Redis works well for this because:

• It’s fast (in-memory reads).
• It supports sorted sets for maintaining order by timestamp or score.
• It provides atomic operations for safe updates.
• It allows trimming feeds to control memory usage.

Data model in Redis

A reasonable structure might look like this:

• feed:user:{user_id} → sorted set of post IDs for that user’s feed (score = timestamp or ranking score)
• feed:public:{topic} → sorted sets of public posts grouped by topic (for discovery)
• post:{post_id} → hash storing post content, author, timestamp, media URL, etc.
• followers:{user_id} → set of follower user IDs
• likes:{post_id} → counter or hash of users who liked the post

Posting workflow

1. A user creates a post.
2. Store it in post:{post_id}.
3. Push the post ID into feed:user:{follower_id} for each follower.
4. Optionally trim each feed to the last N posts to control memory.

Reading a feed then becomes straightforward:

• Fetch post IDs from feed:user:{me}.
• Fetch post content from the corresponding hashes.

Adding relevant public posts

In addition to posts from followed users, most feed systems surface content the user hasn’t discovered yet. This is the “Explore” or discovery part of the feed. The goal is to include relevant content without making the feed feel random.

A simple approach:

1. Maintain per-topic public feeds

   • feed:public:{topic} → sorted set of post IDs scored by recency and engagement

2. Track user interests

   • user:topics:{user_id} → set of topics the user engages with most

3. Sample relevant posts

   • For each topic, sample top posts from the corresponding public feed
   • Optionally include a few posts from other topics for diversity

4. Merge with the user’s feed

   • Combine sampled posts with the followed feed, optionally with light shuffling

5. Avoid duplicates

   • Track already shown posts using a temporary Redis set or Bloom filter

Scoring example:

User interest: tech

• Compute a weighted score: score = timestamp + likes*10 + shares*20
• Push posts into feed:public:tech
• Sample top posts for the user to ensure relevance without frequent database queries

Likes, comments, and engagement

For engagement:

• Use counters or hashes per post, updated atomically
• Optionally adjust feed scores based on engagement

HINCRBY likes:999 1
ZINCRBY feed:user:456 1 post_999

This allows engagement to influence ranking if needed.

Trimming and memory management

• Keep each feed bounded (e.g., 100–500 posts) to control memory usage
• Use ZREMRANGEBYRANK to remove older entries
• Optionally set expiration on less critical keys