[Merged by Bors] - Implement demo command

[sbernauer]

Description

Proudly presenting the first Stackable demo of the world 🥳
Works best with stackabletech/superset-operator#247.
If you don't increasing the Superset timeout the charts may time out. In that case simply refresh them one by one while there is no other query running on Trino (check via Trino webui).

To invoke run (from feature branch)

sbernauer@sbernauer:~/stackabletech/stackablectl$ cargo r -- --additional-demos-file demos/demos-v1.yaml demo install trino-taxi-data
    Finished dev [unoptimized + debuginfo] target(s) in 0.06s
     Running `target/debug/stackablectl --additional-demos-file demos/demos-v1.yaml demo install trino-taxi-data`
[INFO ] Installing demo trino-taxi-data
[WARN ] Failed to parse demo list from https://github.com/stackabletech/stackablectl/main/demos/demos-v1.yaml: missing field `demos` at line 1 column 4
[INFO ] Installing stack trino-superset-s3
[INFO ] Installing release 22.06
[INFO ] Installing airflow operator in version 0.4.0
[INFO ] The release airflow-operator in version 0.4.0 is already installed, not installing it
[INFO ] Installing commons operator in version 0.2.0
[INFO ] The release commons-operator in version 0.2.0 is already installed, not installing it
[INFO ] Installing druid operator in version 0.6.0
[INFO ] The release druid-operator in version 0.6.0 is already installed, not installing it
[INFO ] Installing hbase operator in version 0.3.0
[INFO ] The release hbase-operator in version 0.3.0 is already installed, not installing it
[INFO ] Installing hdfs operator in version 0.4.0
[INFO ] The release hdfs-operator in version 0.4.0 is already installed, not installing it
[INFO ] Installing hive operator in version 0.6.0
[INFO ] The release hive-operator in version 0.6.0 is already installed, not installing it
[INFO ] Installing kafka operator in version 0.6.0
[INFO ] The release kafka-operator in version 0.6.0 is already installed, not installing it
[INFO ] Installing nifi operator in version 0.6.0
[INFO ] The release nifi-operator in version 0.6.0 is already installed, not installing it
[INFO ] Installing opa operator in version 0.9.0
[INFO ] The release opa-operator in version 0.9.0 is already installed, not installing it
[INFO ] Installing secret operator in version 0.5.0
[INFO ] The release secret-operator in version 0.5.0 is already installed, not installing it
[INFO ] Installing spark-k8s operator in version 0.3.0
[INFO ] The release spark-k8s-operator in version 0.3.0 is already installed, not installing it
[INFO ] Installing superset operator in version 0.5.0
[INFO ] The release superset-operator in version 0.5.0 is already installed, not installing it
[INFO ] Installing trino operator in version 0.4.0
[INFO ] The release trino-operator in version 0.4.0 is already installed, not installing it
[INFO ] Installing zookeeper operator in version 0.10.0
[INFO ] The release zookeeper-operator in version 0.10.0 is already installed, not installing it
[INFO ] Installing components of stack trino-superset-s3
[INFO ] The release minio-trino in version 4.0.5 is already installed, not installing it
[INFO ] The release postgresql-hive in version 10.16.2 is already installed, not installing it
[INFO ] The release postgresql-superset in version 11.0.0 is already installed, not installing it
[INFO ] Installed stack trino-superset-s3
[INFO ] Installing components of demo trino-taxi-data
[INFO ] Installed demo trino-taxi-data. Use "stackablectl services list" to list the installed services




sbernauer@sbernauer:~/stackabletech/stackablectl$ stackablectl services list
 PRODUCT   NAME         NAMESPACE  ENDPOINTS                                     EXTRA INFOS                          
                                                                                                                      
 hive      hive         default    hive                172.18.0.4:32658                                               
                                   metrics             172.18.0.4:30745                                               
                                                                                                                      
 opa       opa          default    http                http://172.18.0.2:31324                                        
                                                                                                                      
 superset  superset     default    external-superset   http://172.18.0.2:32716   Admin user: admin, password: admin   
                                                                                                                      
 trino     trino        default    coordinator-http    http://172.18.0.5:32128                                        
                                   coordinator-metrics 172.18.0.5:31199                                               
                                   coordinator-https   https://172.18.0.5:32721                                       
                                                                                                                      
 minio     minio-trino  default    http                http://172.18.0.4:31026   Third party service                  
                                   console-http        http://172.18.0.4:30354   Admin user: root, password: rootroot

Review Checklist

• Code contains useful comments
• (Integration-)Test cases added (or not applicable)
• Documentation added (or not applicable)
• Changelog updated (or not applicable)
• Cargo.toml only contains references to git tags (not specific commits or branches)

Once the review is done, comment bors r+ (or bors merge) to merge. Further information

[maltesander]

Just one typo. Works great!

Just ran in the superset timeout, but nothing we can do about that here.

malte@mdesktop ~/d/w/stackablectl (demos) [2]> target/debug/stackablectl demo list
[WARN ] Failed to parse demo list from https://github.com/stackabletech/stackablectl/main/demos/demos-v1.yaml: missing field `demos` at line 1 column 4
DEMO                                STACKABLE STACK           DESCRIPTION
malte@mdesktop ~/d/w/stackablectl (demos)> target/debug/stackablectl demo describe trino-taxi-data
[WARN ] Failed to parse demo list from https://github.com/stackabletech/stackablectl/main/demos/demos-v1.yaml: missing field `demos` at line 1 column 4
[ERROR] Demo trino-taxi-data not found. Use `stackablectl demo list` to list the available demos.
malte@mdesktop ~/d/w/stackablectl (demos) [255]> 

The list and describe do not work for me?

[sbernauer]

As this PR is not merged you have to specify the additional demos file like
cargo r -- --additional-demos-file demos/demos-v1.yaml demo list
cargo r -- --additional-demos-file demos/demos-v1.yaml demo describe trino-taxi-data

[maltesander]

LGTM.

[sbernauer]

@maltesander could you please also review the new docs?
I think it's a good idea to document any demo, so i created a sub-folder for them.

[maltesander]

I like it but im not sure about the detail of the description. It feels a lot of it could e.g. be in the superset docs (the story about how many passengers, invalid records, views etc.)

[sbernauer]

I moved the documentation to a separate (optional) field so it's machine-readable. I don't think we want to print the docs link when listing the demos, what do you think?

$ cargo r -- --additional-demos-file demos/demos-v1.yaml demo list
DEMO                                STACKABLE STACK           DESCRIPTION
trino-taxi-data                     trino-superset-s3         Demo loading 2 years of New York taxi data into S3 bucket, creating Trino table and Superset dashboard
$ cargo r -- --additional-demos-file demos/demos-v1.yaml demo describe trino-taxi-data
Demo:               trino-taxi-data
Description:        Demo loading 2 years of New York taxi data into S3 bucket, creating Trino table and Superset dashboard
Documentation:      https://docs.stackable.tech/stackablectl/stable/demos/trino-taxi-data.html
Stackable stack:    trino-superset-s3
Labels:             trino, superset, minio, s3, ny-taxi-data

[sbernauer]

Thanks @maltesander, comments should be addressed.
Regarding the level of detail: I plan to document all the demos this way as i don't assume any(or not much) knowledge of the users. The (the story about how many passengers, invalid records, views etc.) is for me bound to this specific datasets and in turn to this specific demo. I'm planning on adding a similar demo with other data, maybe real-time data with druid. Then the analysis in Superset will be a different story.
Anyway it might be a good idea to mention and link to the demos out there using Superset. Something like "If you'd like to see Superset i action check out this demos: [.,..]". But I'm hoping that demos will be more present in the future and some sort of an entry-point for the users ;)

[maltesander]

Thanks @maltesander, comments should be addressed. Regarding the level of detail: I plan to document all the demos this way as i don't assume any(or not much) knowledge of the users. The (the story about how many passengers, invalid records, views etc.) is for me bound to this specific datasets and in turn to this specific demo. I'm planning on adding a similar demo with other data, maybe real-time data with druid. Then the analysis in Superset will be a different story. Anyway it might be a good idea to mention and link to the demos out there using Superset. Something like "If you'd like to see Superset i action check out this demos: [.,..]". But I'm hoping that demos will be more present in the future and some sort of an entry-point for the users ;)

Ok yeah related to the demo data that makes sense then for me. One more thing:
In the stackablectl demo describe examples in the docs the "documentation" field does not show up yet?

[sbernauer]

Oh noes, missed that. Added now

[maltesander]

Nice! LGTM.

[sbernauer]

Thanks a lot for your very fast review!

[sbernauer]

bors r+

[bors]

Pull request successfully merged into main.

Build succeeded:

• Run Rustfmt