Auto Refreshing Cache in .NET
Caching is hard
On a bright sunny day while you’re working on your awesome project you catch a glimpse of something. You bring yourself closer to the monitor and begin meticulously examining your code. What you find brings you feelings of disgust and shame! You’ve been calling an API retrieving data you need over and over again even though said data hardly changes!
Being the smart and responsible engineer that you are, you decide to rectify your mistake immediately. You think to yourself
“OK, I’ll just implement caching around that piece of data and I’m done!”
Not so fast! Little do you know that you’re opening the gates of your own small personal hell!
Caching in .NET
The little story above is a slight exaggeration, but caching is indeed a touchy subject. It’s a nice performance boost when done right, but it can also lead you to a dark path filled with scary monsters and unforeseen problems.
There are many articles written on caching in .NET, you should read the official docs and perhaps other articles written by the good people on the Internet.
This article however will deal with a very specific caching problem. How can you implement a cache that is refreshing itself after a specific amount of time with the following characteristics:
- Thread safety
- Not getting any cache misses (even if that means we’re returning stale data)
If that’s what you need, keep on reading!
Implementation with MemoryCache in .NET Framework
Let’s start with the simple stuff. We want to perform some basic CRUD operations on our cache - saving data to it, getting our cached objects and potentially deleting our data. The best way to do it is to define an interface.
using System.Runtime.Caching;
namespace FooBar.Caching
{
public interface IMemoryAutoRefreshCache
{
T GetById<T>(string id);
void Save<T>(string id, T cacheObj, CacheItemPolicy refreshPolicy);
void Remove(string id);
}
}
These are the methods I’m going to use to illustrate my example. Depending on your use case you can have a lot more of them doing various things to your cache, but for my case those are enough. The point I want you to focus on here is the CacheItemPolicy we can set during the Save.
Now that we have our interface we can also implement it in a new class
using System.Runtime.Caching;
namespace FooBar.Caching
{
public class MemoryAutoRefreshCache : IMemoryAutoRefreshCache
{
private readonly MemoryCache memoryCache;
public MemoryAutoRefreshCache(MemoryCache memoryCache = null) => this.memoryCache = memoryCache ?? MemoryCache.Default;
public T GetById<T>(string id)
{
var value = (T)memoryCache.Get(id);
if (value == null)
{
return default(T);
}
return value;
}
public void Save<T>(string id, T result, CacheItemPolicy refreshPolicy)
{
memoryCache.Set(id, result, refreshPolicy);
}
public void Remove(string id)
{
memoryCache.Remove(id);
}
}
}
Nothing out of the ordinary here. I want to bring your attention to our Save method one more time and the CacheItemPolicy. The summary of CacheItemPolicy states:
Represents a set of eviction and expiration details for a specific cache entry.
Inside the CacheItemPolicy we have two callbacks which will give us everything we need to accomplish our task.
RemovedCallback- Occurs after the item has been removedUpdateCallback- Occurs before the item is removed
As you can imagine we’ll use the UpdateCallback in our case.
So let’s see how an implementation of this auto refreshing cache will look like!
using System;
using System.Runtime.Caching;
using System.Threading;
using FooBar.Caching;
namespace FooBar.Services.Configuration
{
public class ConfigurationProvider
{
private readonly IMemoryAutoRefreshCache localCache;
private const string cacheKey = "myAwesomeCacheKey";
private static readonly object _lock = new object();
public ConfigurationProvider(ILocalAutoRefreshCache localCache) => this.localCache = localCache;
public Configuration GetConfiguration()
{
var cachedConfig = localCache.GetById<Configuration>(cacheKey);
if (cachedConfig != null)
{
return cachedConfig;
}
lock (_lock)
{
// Check to see if anyone wrote in the cache while we're waiting for our turn
cachedConfig = localCache.GetById<Configuration>(cacheKey);
if (cachedConfig != null)
{
return cachedConfig;
}
// This method is not implemented because it can be anything. The main part is that you want to cache an object.
cachedConfig = GetConfigurationFromRemoteLocation(provider);
int refreshTimeInSeconds = 3600; // 1 hour
localCache.Save(cacheKey, cachedConfig, GetPolicy(refreshTimeInSeconds));
return cachedConfig;
}
}
private CacheItemPolicy GetPolicy(int refreshInterval)
{
return new CacheItemPolicy
{
// This is where the magic happens
// The UpdateCallback will be called before our object is removed from the cache
UpdateCallback = (CacheEntryUpdateArguments args) =>
{
if (args.RemovedReason == CacheEntryRemovedReason.Expired)
{
var cacheKey = args.Key;
// Get current cached value
var currentCachedEntity = args.Source[cacheKey] as Configuration;
// Get the potentially new data
var newEntity = GetConfigurationFromRemoteLocation(provider);
// If new is not null - update, otherwise just refresh the old value
// The condition by which you decide to update or refresh the data depends entirely on you
if (newEntity != null)
{
var updatedEntity = newEntity;
args.UpdatedCacheItem = new CacheItem(cacheKey, updatedEntity);
args.UpdatedCacheItemPolicy = GetPolicy(refreshInterval);
}
else
{
var updatedEntity = currentCachedEntity;
args.UpdatedCacheItem = new CacheItem(cacheKey, updatedEntity);
args.UpdatedCacheItemPolicy = GetPolicy(refreshInterval);
}
}
},
AbsoluteExpiration = DateTime.UtcNow.AddSeconds(refreshInterval),
};
}
}
}
So there you go! Now you can refresh, update or invalidate your cache every hour depending on the behavior you desire!
Bonus
As a bonus you can read on the interesting thing that happens when you put AbsoluteExpiration below 20 seconds here.
Also it’s important to note that items can expire some time after the AbsoluteExpiration, there’s no guarantee that they’ll expire right on the spot. Here are two tests confirming that (I know we shouldn’t test code that isn’t ours, but it’s fun!)
using FooBar.Caching;
using NUnit.Framework;
using System;
using System.Runtime.Caching;
using System.Threading;
using System.Threading.Tasks;
namespace FooBar.Caching.UnitTests
{
[TestFixture]
public class MemoryAutoRefreshCacheTests
{
private MemoryAutoRefreshCache subject;
private int updateItemCounter = 0;
[SetUp]
public void Setup() => subject = new MemoryAutoRefreshCache();
[Test]
public async Task VerifyUpdateCallback_LocalMemoryCache()
{
updateItemCounter = 0;
subject.Save($"key", "value", GetPolicy(1));
// The item can be expired between 0 and 30 seconds after expiration time
await Task.Delay(TimeSpan.FromSeconds(32));
// Test that the update callback was invoked at least once
Assert.IsTrue(updateItemCounter >= 1);
}
[Test]
public void VerifyUpdateCallbackReturnsDuringRefresh_LocalMemoryCache()
{
updateItemCounter = 0;
string key = "key";
string value = "value";
subject.Save(key, value, GetPolicyWithDelay(1));
// The item can be expired between 0 and 30 seconds after expiration time
var start = DateTime.UtcNow;
do
{
var cachedItem = subject.GetById<string>("key");
Assert.AreEqual(value, cachedItem);
} while (start.AddSeconds(40) > DateTime.UtcNow);
Assert.IsTrue(updateItemCounter >= 1);
}
private string AddKeys(string key)
{
updateItemCounter++;
return key + key;
}
private void UpdateCounter()
{
updateItemCounter++;
}
private CacheItemPolicy GetPolicyWithDelay(double refreshInterval)
{
return new CacheItemPolicy
{
UpdateCallback = (CacheEntryUpdateArguments args) =>
{
if (args.RemovedReason == CacheEntryRemovedReason.Expired)
{
Thread.Sleep(2000);
var id = args.Key;
// Get current cached value
var currentCachedEntity = args.Source[id] as string;
UpdateCounter();
var newEntity = args.Source[id];
// If new is not null - update, otherwise just refresh the old value
if (newEntity != null)
{
var updatedEntity = newEntity;
args.UpdatedCacheItem = new CacheItem(id, updatedEntity);
args.UpdatedCacheItemPolicy = GetPolicy(refreshInterval);
}
else
{
var updatedEntity = currentCachedEntity;
args.UpdatedCacheItem = new CacheItem(id, updatedEntity);
args.UpdatedCacheItemPolicy = GetPolicy(refreshInterval);
}
}
},
AbsoluteExpiration = DateTimeOffset.UtcNow.AddSeconds(refreshInterval)
};
}
private CacheItemPolicy GetPolicy(double refreshInterval)
{
return new CacheItemPolicy
{
UpdateCallback = (CacheEntryUpdateArguments args) =>
{
if (args.RemovedReason == CacheEntryRemovedReason.Expired)
{
var id = args.Key;
// Get current cached value
var currentCachedEntity = args.Source[id] as string;
var newEntity = AddKeys(id);
// If new is not null - update, otherwise just refresh the old value
if (newEntity != null)
{
var updatedEntity = newEntity;
args.UpdatedCacheItem = new CacheItem(id, updatedEntity);
args.UpdatedCacheItemPolicy = GetPolicy(refreshInterval);
}
else
{
var updatedEntity = currentCachedEntity;
args.UpdatedCacheItem = new CacheItem(id, updatedEntity);
args.UpdatedCacheItemPolicy = GetPolicy(refreshInterval);
}
}
},
SlidingExpiration = TimeSpan.FromSeconds(refreshInterval),
};
}
}
}
Implementation with IMemoryCache in .NET Core
I have a strong feeling a lot of people will come to read this article for the part that follows… Who cares about .NET Framework, amirite?! It’s all about .NET Core nowadays, son!
I’d be lying if I didn’t say that I prefer the .NET Core implementation. But I digress. Let’s stay on the issue at hand - implementing an auto refreshing cache.
We still have access to MemoryCache in .NET Core. However it only provides a callback which will execute after an item has been removed. That doesn’t matter for our use case. Remember what we said at the start. We don’t care if we get stale data as long as there are no cache misses! To achieve the same thing we did in the previous section we are going to use a shiny new interface called IMemoryCache. And guess what! It’s provided by Microsoft too!
The code is more or less the same with some differences when it comes to the locking. This time we’re performing operations inside an async method so we can’t use a regular lock. For the sake of completeness I’ll provide the full implementation + tests and gotchas after.
using System;
using System.Collections.Concurrent;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.Caching.Memory;
namespace FooBar.Providers
{
public class SettingsStore
{
private const string cacheKey = "FooBarPrivateKey";
private const int refreshTimeInSeconds = 3600;
private readonly IMemoryCache memoryCache;
private readonly ConcurrentDictionary<object, SemaphoreSlim> locks = new ConcurrentDictionary<object, SemaphoreSlim>();
public SettingsStore(IMemoryCache memoryCache) => this.memoryCache = memoryCache;
public async Task<Settings> GetSettingsAsync()
{
// Normal lock doesn't work in async code
if (!memoryCache.TryGetValue(cacheKey, out Settings settings))
{
SemaphoreSlim certLock = locks.GetOrAdd(cacheKey, k => new SemaphoreSlim(1, 1));
await certLock.WaitAsync();
try
{
if (!memoryCache.TryGetValue(cacheKey, out settings))
{
// This method is not implemented because it can be anything.
// The main part is that you want to cache an object.
settings = await GetSettingsFromRemoteLocation();
memoryCache.Set(cacheKey, settings, GetMemoryCacheEntryOptions(refreshTimeInSeconds)); // 1 hour
}
}
finally
{
certLock.Release();
}
}
return settings;
}
private MemoryCacheEntryOptions GetMemoryCacheEntryOptions(int expireInSeconds = 3600)
{
var expirationTime = DateTime.Now.AddSeconds(expireInSeconds);
var expirationToken = new CancellationChangeToken(
new CancellationTokenSource(TimeSpan.FromSeconds(expireInSeconds + .01)).Token);
var options = new MemoryCacheEntryOptions();
options.SetAbsoluteExpiration(expirationTime);
options.AddExpirationToken(expirationToken);
options.PostEvictionCallbacks.Add(new PostEvictionCallbackRegistration()
{
EvictionCallback = (key, value, reason, state) =>
{
if (reason == EvictionReason.TokenExpired || reason == EvictionReason.Expired)
{
// If newValue is not null - update, otherwise just refresh the old value
// The condition by which you decide to update or refresh the data depends entirely on you
// If you want a cache object that will never expire you can just make the following call:
// memoryCache.Set(key, value, GetMemoryCacheEntryOptions(expireInSeconds));
var newValue = await GetSettingsFromRemoteLocation();
if (newValue != null)
{
memoryCache.Set(key, newValue, GetMemoryCacheEntryOptions(expireInSeconds)); // 1 hour
}
else
{
memoryCache.Set(key, value, GetMemoryCacheEntryOptions(expireInSeconds)); // 1 hour
}
}
}
});
return options;
}
}
}
Here we have 2 interesting behaviors we should take into consideration. Why do we need to use a cancellation token, and why won’t the EvictionCallback execute after 1 hour without explicitly setting said token.
-
No background thread that actively scans the expiration. This means we have to trigger the eviction/expiration ourselves.
-
We can trigger an expiration with the help of a CancellationToken
To prove this to you I’ll provide another pair of tests
using System;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Primitives;
using Microsoft.VisualStudio.TestTools.UnitTesting;
namespace FooBar.Tests
{
[TestClass]
public class MemoryCacheTests
{
private IMemoryCache CreateCache()
{
return new MemoryCache(new MemoryCacheOptions()
{
ExpirationScanFrequency = new TimeSpan(0, 0, 1)
});
}
[TestMethod]
public async Task ExpireAndReAddFromCallbackWorks()
{
var cache = CreateCache();
var initialValue = "I'm a value";
var newValue = "I'm the refreshed value";
string key = "myKey";
int refreshCounter = 0;
var options = new MemoryCacheEntryOptions();
options.SetAbsoluteExpiration(new TimeSpan(0, 0, 1));
options.PostEvictionCallbacks.Add(new PostEvictionCallbackRegistration()
{
EvictionCallback = (subkey, subValue, reason, state) =>
{
if (reason == EvictionReason.Expired)
{
cache.Set(key, newValue);
refreshCounter++;
}
}
});
cache.Set(key, initialValue, options);
await Task.Delay(TimeSpan.FromSeconds(6));
// Any activity on the cache (Get, Set, Remove) can trigger a background scan for expired items.
// There's no background thread that scans the cache for expired times
var result = cache.Get(key);
await Task.Delay(TimeSpan.FromSeconds(1));
Assert.IsTrue(refreshCounter >= 1);
Assert.AreEqual(newValue, cache.Get(key));
}
[TestMethod]
public async Task TokenExpireAndReAddFromCallbackWorks()
{
var cache = CreateCache();
var initialValue = "I'm a value";
var newValue = "I'm the refreshed value";
string key = "myKey";
int refreshCounter = 0;
int expirationSeconds = 1;
var expirationTime = DateTime.Now.AddSeconds(expirationSeconds);
var expirationToken = new CancellationChangeToken(
new CancellationTokenSource(TimeSpan.FromSeconds(expirationSeconds + .01)).Token);
var options = new MemoryCacheEntryOptions();
options.SetAbsoluteExpiration(expirationTime);
options.AddExpirationToken(expirationToken);
options.PostEvictionCallbacks.Add(new PostEvictionCallbackRegistration()
{
EvictionCallback = (subkey, subValue, reason, state) =>
{
if (reason == EvictionReason.TokenExpired)
{
cache.Set(key, newValue);
refreshCounter++;
}
}
});
cache.Set(key, initialValue, options);
await Task.Delay(TimeSpan.FromSeconds(6));
Assert.IsTrue(refreshCounter >= 1);
Assert.AreEqual(newValue, cache.Get(key));
}
}
}
So that’s basically it! Now we have our auto refreshing cache! There’s a lot that we can improve. You have to be careful who has access to the keys for your specific object and a million other considerations. Caching is a steep hill to climb, but I hope you found this blog post informative in some capacity!
I’ve used both implementations for some heavy objects that don’t change often in production environments and they’ve been doing their job pretty well so far!