net: limit TCP and UDP send/recv buffer usage with throttled IOB

[zhhyu7]

Summary

The main content of this submission is to limit both the TX/RX buffers of TCP/UDP to throttled IOBs, avoiding impacts on the sending and receiving of control-type messages.
and optimized TX for TCP buffered mode.

Impact

• Prevents TCP/UDP data traffic from exhausting IOB pool and blocking control messages;
• Improves system stability under high network load or with limited IOB resources ;
• Maintains responsiveness of ICMP/ICMPv6 and other control protocols;
• Avoid the impact on TCP receiving TCP_ACK when UDP data accumulates in readahead for a long time.

Testing

sim:matter with very small amount of IOB configuration

CONFIG_MM_IOB=y
CONFIG_IOB_NBUFFERS=40
CONFIG_IOB_BUFSIZE=196
CONFIG_IOB_ALIGNMENT=4
CONFIG_IOB_SECTION=""
CONFIG_IOB_NCHAINS=40
CONFIG_IOB_THROTTLE=10

NuttX test log:

nsh> iperf -c 10.0.1.1 -B 10.0.1.2
     IP: 10.0.1.2

 mode=tcp-client sip=10.0.1.2:5001,dip=10.0.1.1:5001, interval=3, time=30

           Interval         Transfer         Bandwidth

   0.00-   3.01 sec  148127744 Bytes  393.70 Mbits/sec
   3.01-   6.02 sec  147177472 Bytes  391.17 Mbits/sec
   6.02-   9.03 sec  132677632 Bytes  352.63 Mbits/sec
   9.03-  12.04 sec  143818752 Bytes  382.24 Mbits/sec
  12.04-  15.05 sec  148914176 Bytes  395.79 Mbits/sec
  15.05-  18.06 sec  158990336 Bytes  422.57 Mbits/sec
  18.06-  21.07 sec  140148736 Bytes  372.49 Mbits/sec
  21.07-  24.08 sec  127320064 Bytes  338.39 Mbits/sec
  24.08-  27.09 sec  129712128 Bytes  344.75 Mbits/sec
  27.09-  30.10 sec  133414912 Bytes  354.59 Mbits/sec
   0.00-  30.10 sec 1410301952 Bytes  374.83 Mbits/sec
iperf exit
nsh> 

esp32c3-devkit:wifi board
NuttX test log:

nsh> iperf -c 10.212.241.242 &
iperf [13:100]
nsh>      IP: 10.212.241.66

 mode=tcp-client sip=10.212.241.66:5001,dip=10.212.241.242:5001, interval=3, time=30

           Interval         Transfer         Bandwidth
   0.00-   3.01 sec    1359872 Bytes    3.61 Mbits/sec
   3.01-   6.02 sec    1343488 Bytes    3.57 Mbits/sec
   6.02-   9.03 sec    1458176 Bytes    3.88 Mbits/sec
   9.03-  12.04 sec    1179648 Bytes    3.14 Mbits/sec
  12.04-  15.05 sec    1425408 Bytes    3.79 Mbits/sec
  15.05-  18.06 sec    1474560 Bytes    3.92 Mbits/sec
  18.06-  21.07 sec    1376256 Bytes    3.66 Mbits/sec
  21.07-  24.08 sec    1277952 Bytes    3.40 Mbits/sec
  24.08-  27.09 sec    1261568 Bytes    3.35 Mbits/sec
  27.09-  30.10 sec    1409024 Bytes    3.74 Mbits/sec
   0.00-  30.10 sec   13565952 Bytes    3.61 Mbits/sec
iperf exit
iperf -c 10.212.241.242 &
iperf [22:100]
nsh>      IP: 10.212.241.66

 mode=tcp-client sip=10.212.241.66:5001,dip=10.212.241.242:5001, interval=3, time=30

           Interval         Transfer         Bandwidth

   0.00-   3.01 sec     180224 Bytes    0.48 Mbits/sec

nsh> cat /proc/iobinfo
    ntotal     nfree     nwait nthrottle
       100        40         0         0
nsh>    3.01-   6.02 sec     376832 Bytes    1.00 Mbits/sec
   6.02-   9.03 sec     491520 Bytes    1.31 Mbits/sec
   9.03-  12.04 sec    1228800 Bytes    3.27 Mbits/sec
  12.04-  15.05 sec    1048576 Bytes    2.79 Mbits/sec
  15.05-  18.06 sec    1392640 Bytes    3.70 Mbits/sec
  18.06-  21.07 sec    1327104 Bytes    3.53 Mbits/sec
  21.07-  24.08 sec    1277952 Bytes    3.40 Mbits/sec
  24.08-  27.09 sec     278528 Bytes    0.74 Mbits/sec
  27.09-  30.10 sec    1130496 Bytes    3.00 Mbits/sec
   0.00-  30.10 sec    8732672 Bytes    2.32 Mbits/sec
iperf exit

nsh> cat /proc/iobinfo
    ntotal     nfree     nwait nthrottle
       100       100         0        60
nsh> 
nsh> 

My local environment has a relatively high packet loss rate, so the throughput is relatively low.

[zhhyu7]

@fdcavalcanti Could you please help conduct a detailed test with this patch when you have time? It should be able to fix any known stuck issues, and by the way, you can also compare if there are any changes in throughput. Thanks a lot.

[fdcavalcanti]

I will run some tests on this patch, let's wait a few days before merging please.

[zhhyu7]

I will run some tests on this patch, let's wait a few days before merging please.

OK, thanks.

[linguini1]

Can you please fill out the impact section?

[zhhyu7]

Can you please fill out the impact section?

Done.

[fdcavalcanti]

@zhhyu7 I have questions regarding this change. Bear in mind I'm not too familiar with the network stack.

Those questions revolve around my tests using Espressif devices and WiFi.

• From this change, it seems I need to set CONFIG_NET_RECV_BUFSIZE and CONFIG_NET_SEND_BUFSIZE. If I keep them at the default value of 0, it does no work anymore. Why is that? Should every device be modified now and what does the 0 value mean?
• Do you have any suggestions on how to orchestrate the IOB options to NET_RECV_BUFSIZE and NET_SEND_BUFSIZE?
  My current approach is to calculate the number of IOBs to match NET_RECV_BUFSIZE + NET_SEND_BUFSIZE, given that each IOB has a fixed size (128 for instance, or 196 on many other devices). Does it make sense?
  Example:

CONFIG_IOB_BUFSIZE=128
CONFIG_IOB_NBUFFERS=256
CONFIG_IOB_NCHAINS=32
CONFIG_IOB_THROTTLE=24
CONFIG_NET_RECV_BUFSIZE=16384
CONFIG_NET_SEND_BUFSIZE=16384

If I take those values: 16384 + 16384 = 128 * 256. So that is how I am matching them and seems to work for ESP32-C3, so far.

[zhhyu7]

@zhhyu7 I have questions regarding this change. Bear in mind I'm not too familiar with the network stack.

Those questions revolve around my tests using Espressif devices and WiFi.

• From this change, it seems I need to set CONFIG_NET_RECV_BUFSIZE and CONFIG_NET_SEND_BUFSIZE. If I keep them at the default value of 0, it does no work anymore. Why is that? Should every device be modified now and what does the 0 value mean?
• Do you have any suggestions on how to orchestrate the IOB options to NET_RECV_BUFSIZE and NET_SEND_BUFSIZE?
  My current approach is to calculate the number of IOBs to match NET_RECV_BUFSIZE + NET_SEND_BUFSIZE, given that each IOB has a fixed size (128 for instance, or 196 on many other devices). Does it make sense?
  Example:

CONFIG_IOB_BUFSIZE=128
CONFIG_IOB_NBUFFERS=256
CONFIG_IOB_NCHAINS=32
CONFIG_IOB_THROTTLE=24
CONFIG_NET_RECV_BUFSIZE=16384
CONFIG_NET_SEND_BUFSIZE=16384

If I take those values: 16384 + 16384 = 128 * 256. So that is how I am matching them and seems to work for ESP32-C3, so far.

@fdcavalcanti Get! The latest version has reverted a modification that might have caused the sent message to be too small, and the previous issue was caused by this reason. In the TCP sending process, if our CONFIG_NET_SEND_BUFSIZE is set to 0, we need to ensure that CONFIG_IOB_THROTTLE * CONFIG_IOB_BUFSIZE > TCP_MSS (or CONFIG_NET_ETH_PKTSIZE - 54), otherwise the protocol stack will freeze.

[fdcavalcanti]

@zhhyu7 thanks for the change.
It works without having to modify the NET_SEND_BUFSIZE.

I have tested the following devices before and on this commit and got very similar throughput in each.

• ESP32: 12 MBits/s (before: same)
• ESP32S3: 4 MBits/s (before: same) (even with SMP)
• ESP32C3: 6 MBits/s (before: 9 MBits/s)
• ESP32C6: 1 MBits/s (before: same)

While some devices are very slow, it still connects and performs IPERF.
So on my side, no changes required.

Current issue noticed:
IPERF still gets stuck on the first ~6 seconds, and this is happening on all devices even before this PR:

nsh> iperf -c 192.168.0.125 -i 3 -t 30
     IP: 192.168.0.8

 mode=tcp-client sip=192.168.0.8:5001,dip=192.168.0.125:5001, interval=3, time=30

           Interval         Transfer         Bandwidth

   0.00-   3.01 sec      49152 Bytes    0.13 Mbits/sec
   3.01-   6.02 sec          0 Bytes    0.00 Mbits/sec
   6.02-   9.03 sec    2064384 Bytes    5.49 Mbits/sec
   9.03-  12.04 sec    2080768 Bytes    5.53 Mbits/sec
  12.04-  15.05 sec    2113536 Bytes    5.62 Mbits/sec