eHarmony Engineering logo

Redis at eHarmony as a Store and Cache

June 26, 2015

There are few options when evaluating low latency in-memory Cache or Store solutions, and Redis is at the top of the list. Redis is an open source, advanced key-value Cache and Store; it is often referred to as a data structure server. In this blog post, I’ll discuss how we use Redis at eHarmony as both a Store and a Cache… Brent Hohn

Redis as Store

As we started migrating eHarmony’s legacy monolithic web applications to a new microservices architecture, we explored various stores that could maintain the tokens for our authentication service and secure all our public API endpoints. The key objective was low latency.

Redis was at the top of the list as far as latency was concerned, as well as providing a simple and stable solution. The Redis cluster solution was only in beta when we first set up Redis, so the only available option was master-slave mode. Single point of failure and availability then became considerations, however (unlike memcached) Redis supports snapshotting and replication, which slightly reduce the risks.

Low latency was our prime requirement (as all our requests are intercepted at a gateway layer for auth verification) – that made Redis the obvious choice. With near realtime replication  and an auto fail over mechanism with sentinel, we could reduce an outage to a few seconds should any major issue arise. Our current production auth store response time is less than a millisecond.

Redis Configuration at eHarmony

Due to immediate consistency requirements, all reads and writes go through the master server.
Our setup includes 2 slaves; either one can become master during failover. In addition to their role in the failover configuration, the slave servers run analytics and diagnostics.

An auth service generates tokens upon user login and saves the tokens in the Redis Store. These tokens are then fetched via the public API with every subsequent API call. Both the auth service and the public API connect to the master server. Data is saved to disk and replicated to the slave servers (see Redis Configuration Properties below).

Redis Configuration Properties

Redis configuration is very simple. All the properties are in a single configuration file, which is well documented with very decent defaults.

You can find the configuration file at:

{redis.home}/etc/redis.conf[/]

Here are some of the more important properties…

Snapshotting/Backup

save 60 10

Saves the data to disk every minute, if at least 10 keys are changed.

  • This is very aggressive setting, implemented to reduce the impact on failover. With this setting in place, we could lose a minute’s worth of data in a worst case scenario.
  • In case of failure, this snapshot will be reloaded on node restart. The snapshot loading is actually pretty rapid, depending on data size. Our storage size is relatively minimal (about 12GB), so it’s fairly fast for us.
  • We are running the nodes on Virtual Machines. For a larger data set, consider using SSD to reduce backup and restore time.

Replication

slaveof masterip masterport
repl-ping-slave-period 10

Data is replicated to slaves in every 10 secs.

  • This is an aggressive setting, but was configured to reduce the impact on failover. Slaves are configured using the slaveof command.
  • Redis doesn’t need to restart for configuration changes. Just execute the commands using redis-cli, and they will be affective immediately without restart.
  • Failover happens in 2 ways: manual failover or automatic failover using Redis Sentinel.

Security

masterauth
requirepass 

Consider disabling certain dangerous commands, which can potentially case major issues. One such command is FLUSHDB.

rename-command CONFIG ""
rename-command FLUSHALL ""
rename-command FLUSHDB ""
rename-command SHUTDOWN "XXXXX"
  • Make sure that you set up sufficient file handlers on the box where you install Redis.
  • Set up the maximum memory parameter. Redis recommends that you leave some memory for copy-on-write replication.
  • Choose an appropriate eviction algorithm. We are using LRU for the auth store.

Monitoring and Visualization Tools

At eHarmony, we use both Graphite and AppDynamics for monitoring. Redis provides good extensions for both of these tools.
redis_dashboard

References

Redis as a Data Structure Service – Internals

One of the strengths of the Redis Store is its rich set of data structures. Redis supports Lists, Sets, Hashes, and various other data structures (along with Pub/Sub support). It also supports accurate counters on these data structures, as it stores all data in-memory. You can easily call Redis as an in-memory data structure server.

Redis Clients

At eHarmony, Spring Data projects are used heavily, so we chose Spring Data Redis for our clients. Spring Data Redis implements the Jedis client internally. We also used Redis as a Hibernate Second Level Cache, as discussed in the next section.

Redis as Cache

Redis can be used as a distributed Cache at the service layer, and also as a Hibernate Second Level Cache. At eHarmony, we are considering both options.

So far we have configured Redis as a Hibernate Second Level Cache in one of our services, which serves configurations from a database. It has operated seamlessly and provided a good performance boost.

The only caveat I found was the Jedis pool configuration. As of now it is hard-coded at 250 max connections, however you can customize this pretty easily by extending the SingletonRedisRegionFactory class. Follow these steps:

  1. Here’s the configuration snippet for reference:
    com.github.debop
    hibernate-redis
    1.6.1
  2. In hibernate-context.xml, include Cache provider information:
    <property name="hibernateProperties">
    			<map>
    				<entry key="hibernate.dialect" value="org.hibernate.dialect.Oracle9Dialect" />
    				<entry key="hibernate.cache.use_query_cache" value="true" />
    				<entry key="hibernate.cache.use_second_level_cache" value="true" />
    				<entry key="hibernate.show_sql" value="true" />
    				<entry key="hibernate.generate_statistics" value="false" />
    				<entry key="hibernate.cache.region.factory_class" value="org.hibernate.cache.redis.SingletonRedisRegionFactory" />
    				<entry key="hibernate.cache.region_prefix" value="hibernate" />
    				<entry key="hibernate.cache.provider_configuration_file_resource_path" value="file://${hibernate.cache.provider_configuration_file_resource_path}" />
    			</map>
    		</property>
  3. Provide the below properties in the properties file:
    redis.host=redis host name
    redis.port=6379
    redis.expiryInSeconds=900 //default value is 120 seconds

What’s next?

We are planning to configure Redis as a distributed Cache in various service layer components, and we are currently exploring a Redis cluster solution to replace the current master-slave model. Basically, we plan to run Redis in multi-tenancy mode, with multiple Stores in a cluster solution.