Document the resource requirements for stacks/demos

[Jimvin]

The stacks used for demos specify certain CPU and RAM requirements for each container, it would be useful to have then sum of these appear in the demo documentation to guide users as to the resources required to run them.

In addition to the resources requested by the Stackable components, we also need to account for resources required by non-stackable components (databases and such). The best way to do it would be to install the demos and note all the resources that get used.

resources include: disk space, RAM, CPU

Step 1: Gather the numbers of demos

• airflow-scheduled-job
  • CPU: 2401 - 4500
  • RAM: 9010 - 8854 Mi
  • 24GB (8GB Postgres, 2x 8GB Redis)
• data-lakehouse-iceberg-trino-spark: already has requirements: 80 cores, 200GB RAM, 1TB PVCs
• hbase-hdfs-cycling-data
  • CPU: 3000 - 4400
  • RAM: 5638 - 5738 Mi
  • 16GB (3x5GB HDFS, 1GB ZK)
• jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data
  • CPU: 3350 - 5800
  • RAM: 7174 - 7274 Mi
  • 22GB
• logging
  • CPU: 6500 - 5200
  • RAM: 5098 - 4895 Mi
  • 27GB
• nifi-kafka-druid-earthquake-data
  • CPU: 8700 - 17900
  • RAM: 27746 - 25286 Mi
  • 75GB (kafka log dir 30GB)
• nifi-kafka-druid-water-level-data
  • CPU: 8900 - 18500
  • RAM: 28042 - 25582 Mi
  • 75GB RAM
• spark-k8s-anomaly-detection-taxi-data
  • CPU: 6400 - 8600
  • RAM: 12622 - 9906 Mi
  • 35GB
• trino-iceberg
  • CPU: 8550 - 12000
  • RAM: 26662 - 22410 Mi
  • 110GB (2x 50GB Minio)
• trino-taxi-data
  • CPU: 6800 - 11500
  • RAM: 15822 - 13362 Mi
  • 28GB

Step 2: Gather the numbers of stacks

• monitoring
  • CPU: 1750
  • RAM: 1714 Mi
  • 110GB
• logging
  • CPU: 5150
  • RAM: 3147 Mi
  • 24GB
• airflow
  • CPU: 3400
  • RAM: 9010 Mi
  • 24GB
• data-lakehouse-airflow-trino-spark
  • 71 cores, 160GB RAM, 1TB pvcs
• hdfs-hbase
  • CPU: 4200
  • RAM: 10758 Mi
  • 21GB
• nifi-kafka-druid-superset-s3
  • CPU: 8900
  • RAM: 28042 Mi
  • 75GB
• spark-trino-superset-s3
  • CPU: 7100
  • RAM: 15910 Mi
  • 36GB
• trino-superset-s3
  • CPU: 6800
  • RAM: 15822 Mi
  • 28GB
• trino-iceberg
  • CPU: 8550
  • RAM: 26662 Mi
  • 110GB
• jupyterhub-pyspark-hdfs
  • CPU: 3350
  • RAM: 7174 Mi
  • 22GB
• dual-hive-hdfs-s3
  • CPU: 7750
  • RAM: 19974
  • 44GB
• tutorial-openldap & openldap
  • CPU: 1950
  • RAM: 2182
  • no pvcs

Step 3: Put the numbers in the docs

#280

[sbernauer]

When we work on this, I would like to add these numbers (cpu + ram + pvc) to stacks.yaml and demos.yaml.
When stackablectl install a stack or demo, it should simply sum up the allocatable resources of all k8s nodes and print a warning when they are less than whatever is written in stacks.yaml or demos.yaml

[Techassi]

I can take a stab at this!