Possible memory leak when clearing oembed cache

[oliver-bowman]

Rocket.Chat Version: 0.53.0
Running Instances: 4
DB Replicaset OpLog: Yes 7 days
Node Version: v4.5.0

Over the course of a many months I have had several problems with RocketChat running out of ram and swap on several of my VM's each VM has 2.5gb of ram and 2GB of swap. This is a gradual increase in ram, but looking at its memory usage it keep rising till the point the server dies.

[geekgonecrazy]

@oliver-bowman how quickly does this seem to happen? And do you notice any pattern? Like more users connected and it starts using more memory? Any pattern you can identify would greatly help us track down this potential problem.

[oliver-bowman]

This happens around once a month, every month. Users connected stays around the same + - a few users. I will continue to find more information when I can.

[geekgonecrazy]

@oliver-bowman please do! Anything that could help us reproduce or at least identify where the leak might be would be very useful.

[Rohlik]

@oliver-bowman I think that 2GB swap with 2,5GB RAM is too much. Best practises is use 50% of RAM for swap.

[geekgonecrazy]

@Rohlik its not best practice, but in this case other then anytime swap is used causing the read / writes to be slower it shouldn't be effecting his issue.

I think I understood correctly that each of these rocket.chat instances is running on its own VM?

[oliver-bowman]

This was not the original intention for resources of VM's, I had to increase them based on this particular issue. I needed to try to decrease down time. Yes each instance is on its own VM.

[BirknerAlex]

We ran into the same issue, our instance is crashing each day. We have ~ 50 online users around the clock. That should be fine.

Rocket.Chat Version: 0.55.1
Running Instances: 1
DB Replicaset OpLog: Disabled
Node Version: v4.5.0

The latest crash report:

Apr 24 11:31:47 chat node[30215]: FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
Apr 24 11:31:47 chat node[30215]: <--- Last few GCs --->
Apr 24 11:31:47 chat node[30215]: 86452953 ms: Mark-sweep 1317.2 (1457.7) -> 1317.4 (1457.7) MB, 404.6 / 0 ms [allocation failure] [scavenge might not succeed].
Apr 24 11:31:47 chat node[30215]: 86453388 ms: Mark-sweep 1317.4 (1457.7) -> 1317.4 (1457.7) MB, 435.1 / 0 ms [allocation failure] [scavenge might not succeed].
Apr 24 11:31:47 chat node[30215]: 86453815 ms: Mark-sweep 1317.4 (1457.7) -> 1315.4 (1457.7) MB, 427.2 / 0 ms [last resort gc].
Apr 24 11:31:47 chat node[30215]: 86454241 ms: Mark-sweep 1315.4 (1457.7) -> 1316.8 (1457.7) MB, 425.9 / 0 ms [last resort gc].
Apr 24 11:31:47 chat node[30215]: <--- JS stacktrace --->
Apr 24 11:31:47 chat node[30215]: ==== JS stack trace =========================================
Apr 24 11:31:47 chat node[30215]: Security context: 0x20cc65ab4629 <JS Object>
Apr 24 11:31:47 chat node[30215]:     1: slowToString [buffer.js:405] [pc=0x321c91a06695] (this=0xf4ac73e9b41 <an Uint8Array with map 0x18d5f1ac8c09>,encoding=0x20cc65abd761 <String[4]: utf8>,start=2322,end=236296)
Apr 24 11:31:47 chat node[30215]:     2: toString [buffer.js:~432] [pc=0x321c91ca8fe5] (this=0xf4ac73e9b41 <an Uint8Array with map 0x18d5f1ac8c09>)
Apr 24 11:31:47 chat node[30215]:     3: arguments adaptor frame: 3->0
Apr 24 11:31:47 chat node[30215]:     4: /* anonymous */(aka /* anonymous */) [/opt/Rocket.Chat-0.5...
Apr 24 11:31:48 chat systemd[1]: rocketchat.service: Main process exited, code=killed, status=6/ABRT
Apr 24 11:31:48 chat systemd[1]: rocketchat.service: Unit entered failed state.
Apr 24 11:31:48 chat systemd[1]: rocketchat.service: Failed with result 'signal'.
Apr 24 11:35:31 chat systemd[1]: Stopped Rocket Chat server.

Our custom systemd unit does a great job on restarting the crashed instance but this always results in a unexpected downtime.

I've seen that node has a memory limit by default at 1.76 GB on a 64 bit machine. Is this maybe to less for our instance? Or is it really a memory leak? I'm really not sure.

The memory usage of the system itself seems okay, the host has 4 GB of ram.

[geekgonecrazy]

@BirknerAlex ah! You are spot on! Does appear nodejs limits its self to 1.76GB of memory. Adding the flag --max_old_space_size=4096 should allow it to use all available memory.

So switching the command to something like: node --max_old_space_size=4096 main.js of course swapping the number to how ever many Megabytes of memory you wish to allow.

@oliver-bowman this might also solve the problem for you.

This likely needs added to our documentation. Let me know if this does the trick for you guys

[BirknerAlex]

@geekgonecrazy So do you think it's legit that our instances are consuming more memory then 1.76GB? If yes, sure I will adjust our memory limit 👍 If it's really a memory leak we will just increase the time until the instance is crashing again.

//EDIT: Okay I will test this parameter change and let you know guys if this fixes the issue for me.

[geekgonecrazy]

@BirknerAlex its definitely possible. How many users / connected users do you have?

[BirknerAlex]

Between 30-50 users at the same time each day. I've changed the setting on our production instance and I've restarted it. I will monitor the process for the next days and let you know if I expect further crashes.