ci: MemBrowse Integration

[michael-membrowse]

Summary

Integrates MemBrowse memory footprint tracking into NuttX CI to automatically report firmware memory footprint across a representative set of boards on
every PR and push to master.

• Adds .github/membrowse-targets.json: tracks 9 boards spanning all supported architectures
  • ARM Cortex-M/A
  • AVR
  • MIPS
  • RISC-V
  • RX
  • SPARC
  • x86_64
  • Xtensa
• Adds membrowse-report.yml: builds Nuttx on the above targets and runs membrowse-action to generate and upload the reports
• Adds membrowse-comment.yml: posts a memory-diff comment on PRs after the report workflow finishes.
• Adds membrowse-onboard.yml: manual workflow to backfill the last N commits per target.
• Adds a MemBrowse badge to README.md to easily access the MemBrowse Nuttx dashboard

Setup

To view the memory dashboard and receive PR comments:

1. Create an account and a project at https://membrowse.com.
2. Copy the project's API key.
3. In the MemBrowse project settings set the dashboard as publicly available
4. In the GitHub repo, add it as a secret named MEMBROWSE_API_KEY

Testing

The integration has been running on a fork. See the live dashboard here:
https://membrowse.com/public/michael-membrowse/nuttx

[lupyuen]

I'm very sorry Michael, we have a very small team of 2 part-time volunteers taking care of everything in NuttX CI. I'm afraid we can't deploy / monitor / maintain this right now. Thanks!

[xiaoxiang781216]

I'm very sorry Michael, we have a very small team of 2 part-time volunteers taking care of everything in NuttX CI. I'm afraid we can't deploy / monitor / maintain this right now. Thanks!

why not give a chance to try it? it's very important to track the firmware size change over time.

[hartmannathan]

@xiaoxiang781216 yes, it is important to track the firmware growth over time. As @lupyuen points out, there is a shortage of volunteers to maintain and administer the CI. We really need more volunteers for this, so that the burden won't be on just two people. They have been struggling to keep the CI usage under control. It might be helpful if some tests could run once per 24 hours (nightly tests) but again, just two people who are volunteers... We might need to ask on the mailing list for more volunteers.

[xiaoxiang781216]

@xiaoxiang781216 yes, it is important to track the firmware growth over time. As @lupyuen points out, there is a shortage of volunteers to maintain and administer the CI. We really need more volunteers for this, so that the burden won't be on just two people. They have been struggling to keep the CI usage under control. It might be helpful if some tests could run once per 24 hours (nightly tests) but again, just two people who are volunteers... We might need to ask on the mailing list for more volunteers.

I suppose @michael-membrowse will commit to maintain the integration quality of membrowse. Since membrowse is an elf based tool, the ci usage should be small. But, anyway we should measure how much ci usage is increased before we decide to close the PR.

[lupyuen]

@xiaoxiang781216 Could Xiaomi offer an Operations Person to help monitor the CI Usage of this Ci Enhancement? We already have difficulty monitoring our Current CI Jobs, as I explained in this video: https://youtu.be/dT8VppS4qR8

Moving Ahead: We need to identify a CI Lead who will be responsible for all decisions on NuttX CI (and accept any consequences). I'm suffering from hypertension, I'm not sure how much more stress I can handle here. Thanks!

[michael-membrowse]

@lupyuen @xiaoxiang781216 @hartmannathan
I measured the CI impact of this integration:

MemBrowse adds ~37 billable minutes per PR.
The current average PR in NuttX is ~1,742 billable minutes.
That's roughly a 2% increase.

I think that's a reasonable tradeoff for per-PR memory footprint tracking, but if you'd prefer a smaller footprint I can reduce the number of tracked targets. Either way, I'm committed to maintaining this integration going forward.

[simbit18]

It’s certainly a useful tool, but personally I’m not keen on using something over which we have no control.
@lupyuen @jerpelea I see it uses membrowse-action and is licensed under GPL-3.0 – is this compatible with Apache 2.0?

[michael-membrowse]

@simbit18 I would like to address your concerns:

On the license: the Action runs in CI, reads the built ELF, and produces a report. It isn't linked into NuttX binaries or distributed with the source, so from a licensing standpoint it sits closer to a build tool than to a dependency, similar to how GPL-3.0 toolchains like GCC are used to compile NuttX without affecting its Apache 2.0 licensing.

On the external dependency concern: NuttX CI already relies on a number of external components that aren't under the project's control, including GitHub Actions itself, the Ubuntu base image, and the toolchains fetched from external URLs in your Docker setup. MemBrowse would sit in the same category, an external tool used at CI time, with no runtime or distribution impact on NuttX.

[simbit18]

@michael-membrowse Of course, I wasn’t referring to GitHub, Inc. or Docker, Inc. – otherwise NuttX wouldn’t be able to stay here! :)

An example of an unfortunate situation that occurred recently

#18359

Of course, I am just a humble committer and this is merely my personal opinion. I will abide by whatever decision NuttX community makes.

[hartmannathan]

@lupyuen @xiaoxiang781216 @hartmannathan

I measured the CI impact of this integration:

MemBrowse adds ~37 billable minutes per PR.

The current average PR in NuttX is ~1,742 billable minutes.

That's roughly a 2% increase.

I think that's a reasonable tradeoff for per-PR memory footprint tracking, but if you'd prefer a smaller footprint I can reduce the number of tracked targets. Either way, I'm committed to maintaining this integration going forward.

@michael-membrowse a 2% increase doesn't sound bad at all. The fact that we're currently using a whopping 1,742 billable minutes per CI run means that there should be plenty of opportunities to gain back those 2% and maybe much, much more. I think that if Someone (tm) could analyze the NuttX tree and how the CI determines which builds to execute, the usage could probably be reduced significantly. For example, if the change touches a particular arch, builds of unrelated archs that do not use that code should not be done at all. If the change touches a driver, configurations that do not utilize that driver should not be built. OTOH when a change touches platform independent code that affects all archs and boards, then yes, let's do 1,742 billable minutes worth of CI because those changes are likely to be high impact. There has already been improvement in this area but I believe the usage could be cut down much more without reducing the quality and utility of the CI. Unfortunately, although these are all wonderful ideas, there's a reason I wrote Someone (tm): we need significant help to make this happen!!

[hartmannathan]

It’s certainly a useful tool, but personally I’m not keen on using something over which we have no control.

@lupyuen @jerpelea I see it uses membrowse-action and is licensed under GPL-3.0 – is this compatible with Apache 2.0?

@simbit18 I've been burnt by that before too!! Though, I would suggest that we take two approaches: for things that are critical to our success, our future, the longevity of NuttX as a project, we should use things that we can control: free, open-source software, etc. For ancillary things that are nice to have but not critical, we can be more relaxed, because in the worst case scenario, if any non-critical thing stops working, we can simply remove it or replace it with an alternative. I think that seeing the memory impact of a proposed change is important yet still falls under the "not critical" category, so we don't need to worry too much.

[michael-membrowse]

Thanks for the thoughtful framing @hartmannathan. So, willing to give it a shot? :)