Freeze/crash in About menu since 9f15fd1

[nlfx]

I've had frequent crashes since yesterday merges. Here is how to reliably reproduce the issue:

• Open and close About menu twice
• Open About menu a third time
• Swipe up to try to access screen 2
• Watch freezes and is reset by watchdog a few seconds later

infinitime-about-menu-crash.mp4

I've bisected the problem to commit 9f15fd1. However given its content, I don't understand how it could create this crash. Maybe it's a problem with some completely unrelated part of the code which only becomes visible after this commit due to some timing or memory position constraint?

Could someone with a dev kit and SWD access please see if they manage to get more info about what is creating the above crash?

In case this bug is memory location dependent and strings with file path in the firmware change the behavior, here is my version of the above commit.

Thanks a lot for your help!

[ObiKeahloa]

What exactly have you changed in the commit?
My first response too was "memory allocation maybe?" but there seems to be enough memory for doing a simple vibration.
It might be something else triggering this maybe the driver?

[nlfx]

What exactly have you changed in the commit?

No change, I just did a git checkout --recurse-submodules 9f15fd1820e620e42fb717fb5d692d44a3f0f70d, compiled it and flashed it.

My first response too was "memory allocation maybe?" but there seems to be enough memory for doing a simple vibration.

What is weird, is that the the crash happens on the About menu (SystemInfo.cpp) which has nothing to do with the motor controller change in this commit.

So indeed, I also initially though that the problem might be that the memory for the stack was at its limit and the additional motorController argument passed down the call stack was enough to start overwriting other memory region. But the fact that the crash happen only when accessing the About menu for the 3rd time would make no sense if that was the real problem. Unless I'm missing something?

So then I though that it might be a memory leak or memory corruption in the LVGL, and that's why opening the same menu multiple times resulted in a crash. But then it makes no sense that I cannot reproduce this crash on the previous commit (d141888) which contains exactly the same LVGL version and same menus.

So without SWD access, I really don't think I can solve this issue. If no one has time to test this, I'll see if I can re-order a dev kit when they'll be back in stock.

[Avamander]

@JF002

But the fact that the crash happen only when accessing the About menu for the 3rd time would make no sense if that was the real problem.

Potential memory leak?

On another note: Something like Valgrind but for remotely debugged MCUs would be so nice. If anyone knows tools like that that work over a gdb remote, please let me know.

[ObiKeahloa]

Is the vibration motor causing some kind of limit? Not really sure as it happening only around the third time as @nlfx told is weird.

Is it maybe a glitch in the Watchdog?

[Avamander]

Vibration motor could cause a brownout event when the battery has a very high internal resistance... but someone with that specific watch would have to actually measure and test out this theory. It's really just predicting on tea leaves when the issue isn't reproducible.

[nlfx]

Is the vibration motor causing some kind of limit?

The crash happens without ever activating the motor. As explained / shown in the above bug description, just opening / closing the About menu 3 time and trying to go to its 2nd screen is enough to crash the watch.

It's really just predicting on tea leaves when the issue isn't reproducible.

Is it not? Did anyone try to reproduce the steps I've described in my bug report with commit 9f15fd1 or any commit more recent? On my watch it reliably crashes 100% of the time if I follow the steps I've described.

[ObiKeahloa]

Is the vibration motor causing some kind of limit?

The crash happens without ever activating the motor. As explained / shown in the above bug description, just opening / closing the About menu 3 time and trying to go to its 2nd screen is enough to crash the watch.

It's really just predicting on tea leaves when the issue isn't reproducible.

Is it not? Did anyone try to reproduce the steps I've described in my bug report with commit 9f15fd1 or any commit more recent? On my watch it reliably crashes 100% of the time if I follow the steps I've described.

Oh thanks for explaining , this seems really weird and the fact that it only happens after three tries (On the third one) , can you try this again but try to access the second menu on the fourth try instead of the third?

[nlfx]

Oh thanks for explaining , this seems really weird and the fact that it only happens after three tries (On the third one) , can you try this again but try to access the second menu on the fourth try instead of the third?

It also happens on the 4th, 5th, etc.
If the Screen1 of the About menu has been opened 3 times or more, the watch crashes when trying to access the Screen2.

[ObiKeahloa]

Oh thanks for explaining , this seems really weird and the fact that it only happens after three tries (On the third one) , can you try this again but try to access the second menu on the fourth try instead of the third?

It also happens on the 4th, 5th, etc.
If the Screen1 of the About menu has been opened 3 times or more, the watch crashes when trying to access the Screen2.

What intrigued me was that it only happens at the third time.

[ObiKeahloa]

I am not sure if this will fix the error or not , can you try replacing the "components/motor/MotorController.cpp" file with this one(Modified it a bit , not sure if it will fix it or not):

MotorController.cpp.txt

Do rename it back to MotorController.cpp from MotorController.cpp.txt

[JF002]

The behavior you're describing makes me think of a memory fragmentation issue : after a few executions, the memory (from lvgl, freertos,...) is too fragmented and it cannot find a memory segment big enough for its needs.

However, I've just tried to reproduce the issue on the latest commit on develop with no success for now. It doesn't mean that the issue was auto-magically fixed, but it probably "moved" elsewhere in the code and execution paths :/

EDIT : it happened to me too on this last commit : just need to open/close the app multiple time, and then to scroll down to display the next page, it crashes.

[JF002]

I can easily reproduce this issue : open/close SystemInfo 2-3 times, then, scroll to the 2nd screen of displayInfo aaand... crash!

I spent far too much hours trying to figure out the cause of this crash, with no luck :/ It looks like the stack of the Display task gets corrupted when it crashes and falls into HardFault_handler:

Here's the stacktrace when the screen is displayed correctly:

I've tried many things : analyze heap, stack and freertos heap/stack usage, disable all the code inside CreateScreen1() and CreateScreen2(), logging, breakpoints,.. but I cannot understand what's happening!

[ObiKeahloa]

This sounds really weird , what happens of we were to start removing info from the screen?