ARROW-9201: [Archery] More user-friendly console output for benchmark diffs, add repetitions argument, don't build unit tests

[wesm]

This uses pandas to generate a sorted text table when using archery benchmark diff. Example:

#7506 (comment)

There's some other incidental changes

• pandas is required for archery benchmark diff. I don't think there's value in reimplementing the stuff that pandas can do in a few lines of code (read JSON, create a sorted table and print it nicely for us).
• The default # of benchmarks repetitions has been changed from 10 to 1 (see ARROW-9155 for context). IMHO more interactive benchmark results is more useful than higher precision. If you need higher precision you can pass --repetitions=10 on the command line
• archery benchmark was building the unit tests unnecessarily. This also occluded a bug ARROW-9209, which is fixed here

[wesm]

@kszucs can you assist me with adapting ursabot for these changes? I think we can use pandas's DataFrame.to_html to create a colorized table for GitHub, too https://pandas.pydata.org/pandas-docs/stable/user_guide/style.html

Changes that would be good to have in ursabot benchmark:

• Pass through --cc and --cxx options
• Pass through --repetitions

[github-actions]

https://issues.apache.org/jira/browse/ARROW-9201

[pitrou]

Just a small question: why are m and b used for millions and billions, respectively? (I would probably expect M and G)

[kszucs]

@kszucs can you assist me with adapting ursabot for these changes?

Sure.

I think we can use pandas's DataFrame.to_html to create a colorized table for GitHub, too https://pandas.pydata.org/pandas-docs/stable/user_guide/style.html

I'm afraid this is not going to work, because we can't embed any CSS into the comment, this is why we generate the ursabot responses as diffs.

Changes that would be good to have in ursabot benchmark:

• Pass through --cc and --cxx options
• Pass through --repetitions

[fsaintjacques]

ursabot uses tabulate which I think is smaller dependencies.

[wesm]

I’m sort of -1 on using anything but pandas for data munging and data presentation in our tooling. It’s not a very large dependency and has everything we need. FWIW, the current Ursabot output doesn't even sort the results, which is really needed to easily make sense of what got faster or slower at a glance.

[kszucs]

Using pandas is not a problem, but the results cannot be improved much other than sorting the table.

[wesm]

I improved the output to show the state.counters stuff

                  benchmark         baseline        contender  change %                                    counters
40            UniqueInt64/5    6.442 GiB/sec   18.346 GiB/sec   184.782  {'iterations': 145, 'null_percent': 100.0}
0            UniqueInt64/11    6.500 GiB/sec   18.364 GiB/sec   182.522  {'iterations': 145, 'null_percent': 100.0}
11            UniqueUInt8/5  812.047 MiB/sec    1.755 GiB/sec   121.298  {'iterations': 142, 'null_percent': 100.0}
7             UniqueUInt8/1  683.943 MiB/sec    1.253 GiB/sec    87.593    {'iterations': 117, 'null_percent': 0.1}
38            UniqueUInt8/4  762.983 MiB/sec  950.521 MiB/sec    24.580   {'iterations': 133, 'null_percent': 99.0}
29            UniqueUInt8/2  659.082 MiB/sec  820.410 MiB/sec    24.478    {'iterations': 114, 'null_percent': 1.0}
5             UniqueInt64/1    2.656 GiB/sec    3.300 GiB/sec    24.223     {'iterations': 60, 'null_percent': 0.1}
32            UniqueInt64/4    5.627 GiB/sec    6.772 GiB/sec    20.349   {'iterations': 119, 'null_percent': 99.0}
25           UniqueInt64/10    5.234 GiB/sec    6.294 GiB/sec    20.254   {'iterations': 110, 'null_percent': 99.0}
39  UniqueString100bytes/11   26.815 GiB/sec   31.122 GiB/sec    16.061   {'iterations': 48, 'null_percent': 100.0}
23    UniqueString10bytes/5    2.691 GiB/sec    3.113 GiB/sec    15.667   {'iterations': 48, 'null_percent': 100.0}
34   UniqueString100bytes/5   26.944 GiB/sec   31.015 GiB/sec    15.108   {'iterations': 48, 'null_percent': 100.0}
6    UniqueString10bytes/11    2.699 GiB/sec    3.096 GiB/sec    14.721   {'iterations': 49, 'null_percent': 100.0}
21   UniqueString100bytes/7    1.947 GiB/sec    2.217 GiB/sec    13.866      {'iterations': 3, 'null_percent': 0.1}
28            UniqueInt64/2    2.622 GiB/sec    2.904 GiB/sec    10.770     {'iterations': 59, 'null_percent': 1.0}
13            UniqueInt64/3    2.157 GiB/sec    2.343 GiB/sec     8.644    {'iterations': 48, 'null_percent': 10.0}
33   UniqueString100bytes/4   24.286 GiB/sec   26.030 GiB/sec     7.181    {'iterations': 43, 'null_percent': 99.0}
22            UniqueInt64/7    2.542 GiB/sec    2.707 GiB/sec     6.497     {'iterations': 56, 'null_percent': 0.1}
20  UniqueString100bytes/10   22.536 GiB/sec   23.985 GiB/sec     6.432    {'iterations': 40, 'null_percent': 99.0}
35    UniqueString10bytes/1  788.817 MiB/sec  836.008 MiB/sec     5.983     {'iterations': 14, 'null_percent': 0.1}
17    UniqueString10bytes/7  592.671 MiB/sec  628.054 MiB/sec     5.970     {'iterations': 10, 'null_percent': 0.1}
3     UniqueString10bytes/4    2.515 GiB/sec    2.658 GiB/sec     5.687    {'iterations': 45, 'null_percent': 99.0}
19   UniqueString10bytes/10    2.402 GiB/sec    2.529 GiB/sec     5.269    {'iterations': 42, 'null_percent': 99.0}
9    UniqueString100bytes/1    3.929 GiB/sec    4.077 GiB/sec     3.762      {'iterations': 7, 'null_percent': 0.1}
30    UniqueString10bytes/8  593.560 MiB/sec  610.253 MiB/sec     2.812     {'iterations': 10, 'null_percent': 1.0}
12    UniqueString10bytes/2  788.505 MiB/sec  808.396 MiB/sec     2.523     {'iterations': 14, 'null_percent': 1.0}
37   UniqueString100bytes/8    1.965 GiB/sec    1.998 GiB/sec     1.697      {'iterations': 3, 'null_percent': 1.0}
1    UniqueString100bytes/2    3.984 GiB/sec    4.025 GiB/sec     1.028      {'iterations': 7, 'null_percent': 1.0}
36   UniqueString100bytes/3    4.262 GiB/sec    4.293 GiB/sec     0.725     {'iterations': 8, 'null_percent': 10.0}
8     BuildStringDictionary   85.507 MiB/sec   85.687 MiB/sec     0.211                         {'iterations': 198}
16   UniqueString100bytes/9    2.121 GiB/sec    2.111 GiB/sec    -0.469     {'iterations': 4, 'null_percent': 10.0}
4    UniqueString100bytes/6    2.056 GiB/sec    2.043 GiB/sec    -0.626      {'iterations': 4, 'null_percent': 0.0}
10            UniqueUInt8/3  453.281 MiB/sec  448.407 MiB/sec    -1.075    {'iterations': 79, 'null_percent': 10.0}
14   UniqueString100bytes/0    4.100 GiB/sec    4.055 GiB/sec    -1.089      {'iterations': 7, 'null_percent': 0.0}
24            UniqueInt64/8    2.473 GiB/sec    2.443 GiB/sec    -1.202     {'iterations': 55, 'null_percent': 1.0}
26    UniqueString10bytes/9  615.880 MiB/sec  608.453 MiB/sec    -1.206    {'iterations': 11, 'null_percent': 10.0}
42    UniqueString10bytes/6  651.430 MiB/sec  640.128 MiB/sec    -1.735     {'iterations': 11, 'null_percent': 0.0}
27            UniqueUInt8/0    1.775 GiB/sec    1.738 GiB/sec    -2.063    {'iterations': 318, 'null_percent': 0.0}
31            UniqueInt64/9    2.076 GiB/sec    2.033 GiB/sec    -2.067    {'iterations': 46, 'null_percent': 10.0}
15          BuildDictionary    1.535 GiB/sec    1.503 GiB/sec    -2.079                         {'iterations': 277}
41            UniqueInt64/0    3.915 GiB/sec    3.827 GiB/sec    -2.262     {'iterations': 87, 'null_percent': 0.0}
43    UniqueString10bytes/3  802.729 MiB/sec  784.279 MiB/sec    -2.298    {'iterations': 14, 'null_percent': 10.0}
18            UniqueInt64/6    3.284 GiB/sec    3.178 GiB/sec    -3.229     {'iterations': 72, 'null_percent': 0.0}
2     UniqueString10bytes/0  895.983 MiB/sec  849.150 MiB/sec    -5.227     {'iterations': 16, 'null_percent': 0.0}

[wesm]

+1. The bot changes can't be done here so going to go ahead and merge this so I can use it more easily without having to switch branches (to use this branch) before running benchmarks

[kszucs]

I’m going to update the bot tomorrow.