ImageSharp.Formats.Jpg.DecodedBlock ArrayPool is using up WAY too much memory

[DeCarabas]

Prerequisites

• I have written a descriptive issue title
• I have verified that I am running the latest version of ImageSharp
• I have verified if the problem exist in both DEBUG and RELEASE mode
• I have searched open and closed issues to ensure it has not already been reported

Description

I'm basically reporting issue #123 again, but with a slightly different spin. I'm running my little web app on a VM with 1GB of RAM, and I'm constantly getting killed with out of memory because of the ArrayPool being used for ImageSharp.Formats.Jpg.DecodedBlock[].

My app concurrently fetches RSS feeds, then for each entry in each RSS feed concurrently fetches all the images and evaluates all the images for a given entry to see which one will make the best thumbnail. I don't need to get into the large number of feeds before I'm decoding a lot of images in parallel.

After a single fetch session, I dumped the process and examined the heap; I discovered that ImageSharp.Formats.Jpg.DecodedBlockArray.ArrayPool was holding on to 337MB all on its own; the largest entry was 262,144 elements long for a size of 70,254,616. (Obviously that's just the current state of the pool; as currently configured by ImageSharp it can grow way past that.)

I need some way to get a handle on this memory usage; I'm looking at other constraints but some kind of configuration to let me limit the pool would be really nice.

Steps to Reproduce

N/A

System Configuration

• ImageSharp version: 1.0.0 alpha 4
• Environment (Operating system, version and so on): linux
• .NET Framework version: .net core 1.1

[antonfirsov]

@DeCarabas Thank you for your detailed report!

@JimBobSquarePants based on these issues it seems obvious to me, that we need to make our pooling configurable. My proposal:
https://gist.github.com/antonfirsov/25301496604015fbfe7f2b90d608a190

[tocsoft]

totally agree with @antonfirsov here. Lets our users limit the max pool size to use... the difficulty might be around the LimitInBytes as it looks like the total size will be limited not a limit by Type... might have to see if there's a way we can solve that and have it be a shared size (maybe internally we use a common byte[] pool and use Unsafe.As<> to cast them to the correct type thus the array pool will share a common memory limit.)

[DeCarabas]

I'd take something as simple as the ability to reach in and tweak the individual pool settings. An overall memory usage policy sounds appealing but I think would be (a) hard to implement and (b) hard for me to really tune properly. Whereas I have a bunch of heap dumps and statistics around pool sizes that I can reason about for various pools...

[antonfirsov]

Individual pools could be customized by implementing custom IArrayPoolProvider -s. There will be no such thing as Configuration.DecodedJpegBlockArrayPoolLimit because it's an implementation detail we want to hide.

@DeCarabas why do you feel a byte-measured limit is hard to tune?

@tocsoft I would be much happier to operate on "typeless" generic byte[] buffers but I'm afraid Unsafe.As<> is just too unsafe, and it would lead to bugs like #146.
The problem is that Unsafe.As<int[]>(new byte[1]).GetType() == typeof(byte[]).

[tocsoft]

ah well that's unfortunate 😞

[JimBobSquarePants]

We can certainly make our pool configurable but we have to look at a use case issue here also.

Using 1GB ram for any serious image processing is simply a no-go. You'll run out of contiguous memory in the LOH pretty quickly if you work with multiple images. We're pooling yes but the pool is smart and will not grossly over-deliver on array requests. Making the pool more configurable will still lead to very similar memory consumption as it will end up allocating a new array anyway.

I would like to double check our codebase though to ensure we're not missing something in Jpeg. I'm sure we're good but maybe we can make it leaner than it is.

[DeCarabas]

@antonfirsov I appreciate the desire to hide the pools from the caller, but then you're left with several unappealing options.

1. You can try to have a single capped pool that dynamically trades off against other pools.
2. You can try to have a single limit, and somehow divide up that limit amongst other pools.
3. You can try to have a single per-pool limit.

The problem with option 1 is that picking a good policy is hard-- how do you trade DecodedBlock off vs. Pixel? What if I consume the entire pool with DecodedBlock at the beginning of my program and then never it again, since I spend the rest of the time doing image manipulation on uncompressed pixels?

The problem with option 2 is similar-- if you try to protect the pool for pixels by dividing the limit up "statically", then you waste by reserving memory even when you don't know the pool will ever be used. Why is the array pool for pixels smaller even if I never end up deocoding JPEGs?

The problem with option 3 is that you haven't really hidden the number of pools from me-- you've just made it harder for me to figure out how I should set the limit, since the real limit is now the per-pool-cap times the number of pools. I can do this; I have the memory statistics, I've dumped the heap, I know where to tune, but you haven't hidden anything from me, really, just made it harder to figure out.

[DeCarabas]

@JimBobSquarePants Yeah I'm not entirely sure what I was trying to load that allocated 70MB of DecodedBlock; I probably need to do something to protect myself from that anyway.

But the pool is not that smart-- it doesn't have any kind of idle shrinking capability, it doesn't shrink on memory pressure, and it has this annoying power-of-two thing that can end up wildly over-allocating in order to hit the bucket.

The power-of-two thing is annoying because I was not necessarily using that 300MB concurrently-- if I needed an array of 131,073 DecodedBlocks I would have ended up allocating 262,144 of them. In the dump I'm looking at now, I have one such array at that size, for 70MB, and three at 131,072, at 35MB a piece. I only needed to get asked for an array of 65,537 blocks to allocate one of those... at those sizes I would much rather stop using the array pool and ask the allocator for an exact fit instead.

[antonfirsov]

@DeCarabas our main difficulties:

• Jpeg decoder implementation is changing rapidly, so it's really dangerous to expose this kind of super-specific fine-tuning option on a public API.
• Not only DecodedBlock-s are pooled. There are dozens of different places using array pools in different codec implementations. We have to think about all the use cases, not just yours :)

We can only solve these by thinking in a general way.

What could be a fair compromise for me: we can introduce IDecoderOptions.ArrayPoolProviderOverride , so you can tune the pooling for a specific decoding process. It will limit down, or disable pooling for DecodedBlock and temporal jpeg image buffers (JpegPixelArea.Pixels) as well.

Also totally agree with @JimBobSquarePants: a fragmented LOH could also lead to OutOfMemoryException-s.

[DeCarabas]

@antonfirsov Totally understand on both points; I expect tuning parameters to be something weird and fiddly and likely to break at a moment's notice.

This is always the difficulty with adding caches to libraries like this-- you can't meaningfully add a cache without being in complete control of the resources you're using for your cache, and your library is not in control over the memory in my app. Alas.

Another couple of options:

• A big boolean flag saying "DISABLE ALLOCATION POOLS"
• An API I can call that says "FLUSH ALLOCATION POOLS NOW"

Either of these would fix my problems without exposing the internals. (I'd personally take the hit of just relying on the GC more heavily over accidentally permanently consuming large blocks of memory.(

[tocsoft]

In that case If/when we add the IArrayPoolProvider you would be able to make one that fakes out the ArrayPool<T> abstract class with one that just allocates arrays and ignores the Return(...) that will just allow the GC to do its work on the arrays.

[antonfirsov]

I think we would provide an easy way to disable pooling by setting Configuration.ArrayPoolProvider to null or a similar construct. By implementing your own ArrayPool you can take even more control, eg. add flushing. (There is no such builtin feature in the default System.Buffers implementation ).

[dlemstra]

If we are going to create an ArrayPoolProvider (sounds like a good idea) we should offer two implementations. One that reuses the pool (our default) and one that just creates a new one all the time. Otherwise we need null checks everywhere we use pools.

[BrianThomson]

Same problem here. Using a 13MB JPEG image resulted in 2.4GB private memory allocation. The second upload was even higher. This brings any server down.