File Watchers and Inotify Instances Has Been Reached

During a recent production deployment of one of our applications we had an issue where containers started rolling, this was very odd as the same container had just deployed successfully to our canary environment. After the initial panic, the deployment was cancelled, and old containers started up again, panic over. Now what the heck happened?

Checking the logs, we had started getting this error.

1
2
3
4
5
6
7
8
9
10
System.IO.IOException: The configured user limit (128) on the number of inotify instances has been reached, or the per-process limit on the number of open file descriptors has been reached.
at System.IO.FileSystemWatcher.StartRaisingEvents()
at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter)
at Microsoft.Extensions.Primitives.ChangeToken.OnChange(Func`1 changeTokenProducer, Action changeTokenConsumer)
at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(FileConfigurationSource source)
at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(IConfigurationBuilder builder)
at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
at Microsoft.Extensions.Configuration.SystemsManagerExtensions.GetAwsOptions(IConfigurationBuilder builder)
at Microsoft.Extensions.Configuration.SystemsManagerExtensions.AddSystemsManager(IConfigurationBuilder builder, Action`1 configureSource)

The other thing that had changed was that due to having to encrypt the clusters EBS volumes we had used this as an opportunity to upgrade the version of Linux the cluster was using.

After a little bit of Googling this appeared to be due to the reloadOnChange flag being set to true when adding our Json config files. Since we are running this in a container and containers are immutable it seemed to be an easy fix, just set reloadOnChange to false.

When we deployed the updated version, we ended up with the same exception, from the call stack it appeared that .Net was ignoring the value of the flag. Very confusing 😕.

I ended up debugging into the Microsoft code to see what was going on and found that we were using Host.CreateDefaultBuilder adds the config files itself with reload on change set to true as you can see in the code snippet below.

1
2
3
4
5
6
7
8
builder.ConfigureAppConfiguration((hostingContext, config) =>
{
IHostEnvironment env = hostingContext.HostingEnvironment;

bool reloadOnChange = hostingContext.Configuration.GetValue("hostBuilder:reloadConfigOnChange", defaultValue: true);

config.AddJsonFile("appsettings.json", optional: true, reloadOnChange: reloadOnChange)
.AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true, reloadOnChange: reloadOnChange);

Link to this code on MS GitHub

Microsoft was setting up the Json config files as well as our code, and theirs was defaulting to reloadOnChange. This call is pointless in our solution as we are then doing all this set up ourselves. If you read the rest of the CreateDefaultBuilder file there is stuff in there we don’t use and stuff we end up replacing, so why are we using it?

The fix was easy, change Host.CreateDefaultBuilder to new HostBuilder. Digging into the history of Host.CreateDefaultBuilder (thanks MS for putting it on GitHub), it turns out this function has worked the same way since at least .Net core 2.0. In .Net 5 they have added a flag so you can set reloadOnChange to false, but as all our applications do this type of setup themselves it is pointless to us.

The learnings from this are to check what sample code does, given that all the source for .Net core/.Net 5 is one GitHub check what the MS calls are doing. From what found it appears that we have been doing our set up wrong for a while as we end up duplicating a lot of what is in Host.CreateDefaultBuilder. Going forward I’ll be using new HostBuilder, unless I’m doing a simple app that only needs what is in Host.CreateDefaultBuilder.