Discover dataset names when uploading using Dataset Uploader app

[rudokemper]

Goal

Closes #252.

Note to the reviewer: you may want to review this code on our staging Windmill in addition to code review, or forego code review entirely in lieu of a UI/UX focused review. Especially as most of the changes are to app.yaml which is not reviewable by a human.

Screenshots

What I changed and why

• Fetch warehouse tables on load (like Local Contexts)
• Replaced “type name + Check Availability” with dropdown (existing datasets) or new name field
• Removed "Check Availability" button and bottom status messages (“Appending…”, etc.). Instead, auto-resolve dataset name in background (normalize + check if table exists)
• Stepper validation: dropdown or name must be set

How I convinced myself this is right

We use the same logic in the Local Contexts app to manage dropdowns there.

LLM use disclosure

None

[nicopace]

Thanks Rudo for the implementation.

I tested it out and it looks clear to me.

One usability change that I might request is that, instead of a drop down it would be good to have a list. The rationale here is that the drop down disappears if you click away and you're looking at the screen, and you not always know what you are looking for... also, would be good to eventually see the rest of the metadata that was added on dataset creation (like description, source, etc), and a list or table might be easier for that.

Also, it seems that when you add a database name and press next, before there was a check that showed you that the name was available or it already existed and it was going to append to that dataset, and now it seems there is none.
Or at least it doesn't show for some reason.
Also, if I use the name of an existing dataset It doesn't tell me that exists.

[rudokemper]

Also, it seems that when you add a database name and press next, before there was a check that showed you that the name was available or it already existed and it was going to append to that dataset, and now it seems there is none.
Or at least it doesn't show for some reason.
Also, if I use the name of an existing dataset It doesn't tell me that exists.

I took it out as it was previously triggered by a "Check Availability" button, which is now no longer necessary. But you are right that it is still helpful to show these messages, in particular to indicate that the given name is of an existing dataset and we will be appending data to it. I have made this change:

[rudokemper]

One usability change that I might request is that, instead of a drop down it would be good to have a list. The rationale here is that the drop down disappears if you click away and you're looking at the screen, and you not always know what you are looking for... also, would be good to eventually see the rest of the metadata that was added on dataset creation (like description, source, etc), and a list or table might be easier for that.

I'm not sure I follow. By "list" do you mean radio buttons with each table spelled out as an option? I'm not sure how I would fit that into the small UI of the dataset uploader form stepper. And for GC instances with >20 tables this could look very unwieldy.

I also don't think a component exists in Windmill's component library that matches that expectation. Notably, the select component is not radio buttons, but rather also a dropdown with some typeahead behavior.

Lastly I just don't think this is in scope, similar to my comment about seeing metadata here. I think trying to allow the users to inspect datasets in this simple application is not fit-for-purpose and I'd rather lean on existing tools for that. To me your requests are about a data catalog, or at least a lightweight way to inspect database tables like NocoDB, and I think I'd rather point the user to a tool like that to build an understanding of their data before they come to this uploader app.

[nicopace]

I'm not sure I follow. By "list" do you mean radio buttons with each table spelled out as an option? I'm not sure how I would fit that into the small UI of the dataset uploader form stepper. And for GC instances with >20 tables this could look very unwieldy.

I was thinking more like a styled "ul" , or a table...

I also don't think a component exists in Windmill's component library that matches that expectation. Notably, the select component is not radio buttons, but rather also a dropdown with some typeahead behavior.

but I see there are no components like that one.
I am happy with what there is.
If we ever want to expose the metadata (and as you said it might be out of scope), we can always share it appending that information to the table name.

To me your requests are about a data catalog,

My line of thinking is to provide the obvious (and existing) ways of identifying a dataset... and that they are already providing, so it might be easy for them to relate with that information (in case they are appending to a dataset that they themselves uploaded).

I am happy with the way things are still.

[rudokemper]

As a point of formality @nicopace if you approve of the current work, then submit a review approving the PR, even if you don't actually review the code itself.

[rudokemper]

Future work might propose adding a Modal component triggered by a "Discover Datasets" button on the first page of the GC Dataset Uploader stepper. The modal itself could contain a Database Studio component (display-only) that renders a future warehouse_metadata table that regularly records and updates some of the things you are bringing up. (I don't think it makes sense to compute COUNT * for each table on the fly, among other things.)

If you feel strongly about it, please add it as a Discussion item. I don't think we can support this at this time.