Memory Leak/ Garbage collection issue Version 2.4.2

[matthewbassett]

Hi All,
I am experiencing an issue with memory growing out of control. I am running RC on Ubuntu 16.04 using SNAPS. I'm having a hard time tracking down what is causing it, but the best I can guess/tell from observations is that it has to do with file uploads that users are doing. I can watch the RAM amount increase as file transfers come in monitoring network with iftop. This grows until the RAM is totally full and things start to hang. The amount of RAM has no bearing, it will grow and consume whatever is available. I increased the RAM available from 16GB to 24GB to test this theory, and all it does is delay the inevitable. Both node and mongod seem to grow at roughly the same rate in RAM usage.

Hopefully this is helpful information and not a duplicate of #15624 #13602 #12880 #6588 #6552
If it is a duplicate feel free to merge mine in with someone else, I just didn't want to muddy the waters of their ticket, and I feel that mine provides potentially more information. If you need/want specific logs etc and I am happy to oblige.

Thank you for the fantastic App (: I hope you all are able to get enough people on board to start tackling the open issues log Some of those issues I referenced above are open from 2017 :/

[sampaiodiego]

hi @matthewbassett .. thanks for the infos.. I have some other questions:

• how long it takes to eat all RAM available?
• do you have any charts to share showing the RAM increase that co-relates to file uploading?
• how many users are using your server?

thanks for your understanding as well.. it's always hard for us decide whether focus on github issues or developing new features.. we do our best.

[sampaiodiego]

another important question, what storage type are you using?

[matthewbassett]

Thanks for getting back, my apologies for the delay in my response. I work for an international school in China, so life is a little crazy right now as I'm sure you can imagine! q:

• I just had RC use up all of the 24GB of RAM and need to be restarted again. The amount of time it takes to eat up all the RAM varies greatly. overnight it won't add anything, then throughout the day it just steadily climbs. Increasing to 24GB gives me maybe 3-5 days of uptime at this point before I need to restart? I will try to keep better track of the timing so we have more accurate data.

• We have ~1400 users, generally between 100-300 online at any given time. Many of the users rarely use the service, a good group use it a great deal.

• Storage type: GridFS. See attached for further settings

[matthewbassett]

I actually came up with some ideas for charting RAM usage and network usage over time to see if I can come up with some correlations. I'll start working on that and post results here.

[UAnton]

I monitored a memory leak and found out that this is a nodejs.

[matthewbassett]

For me both nodejs and mongod increase at a pretty similar rate. @UAnton are you seeing similar things? or is your nodeJS growing faster than mongo?

[matthewbassett]

@sampaiodiego any further thoughts with the information provided?

[matthewbassett]

Here is some data from the graphing I setup. I wouldn't say its exactly conclusive, but certainly the memory drain starts when traffic picks up.

Interestingly enough you can see some things requiring memory, but then it springs back. So it must be something very specific that is causing the drain.

[matthewbassett]

Sorry, had my graph set to last 6 hours. here is a more comprehensive shot of the whole time I have been monitoring:

I wouldn't call it conclusive by any means, but I certainly think it supports my theory a bit. At least proves that high usage times cause the available RAM to drop and it does not recover during low use times, but it also doesn't drop further during low usage times.

[matthewbassett]

Hi wanted to bump this again. Any movement on looking into this or any further data needed to troubleshoot. I am now on 2.4.11 and I am still seeing this issue regularly. I have had to restart the server several times in the past week.

[rsjr]

Hi @matthewbassett . We made some improvements on RC version 3.0 (specially regarding the file upload, that also 'helps' to create this memory leak). Maybe, if you are interested, you can install another server of RC 3.0 and compare both behaviors.

We will analyze/investigate this memory leak situation as whole, but we cannotgive an ETA right now. Thanks for the detailed analysis, so far.

[rodrigok]

@matthewbassett do you have Admin > Logs > Prometheus enabled?

[matthewbassett]

@rsjr Thanks for the thoughts! Glad there has been some progress, and I really appreciate you all looking into this. I will work on getting a 3.0 instance up and running (:

[matthewbassett]

@rodrigok I do not have Prometheus enabled. Should I?