Consolidate Setup Tool, Configuration Tool, and Preloaded Tree Creation

[CarolineDenis]

Fixes #7673
See issue description

(Cloned from #6671)

[acwhite211]

I pushed a fix to fix the error on the stats page query. I not seeing any errors on the Accounts page.

Changed the query in get_percent_georeferenced from

SELECT CASE WHEN
  (SELECT COUNT(*)
   FROM locality
   JOIN discipline ON locality.DisciplineID = discipline.DisciplineID
   WHERE discipline.DisciplineID = 3) = 0
THEN 0
ELSE ((COUNT(localityid) * 1.0) /
      ((SELECT COUNT(*) FROM locality) * 1.0)) * 100.0
END AS PercentGeoReferencedLocalities
FROM locality
WHERE latitude1 IS NOT NULL;

to

SELECT CASE WHEN totals.total_count = 0 THEN 0
ELSE (totals.georeferenced_count * 100.0) / totals.total_count
END AS PercentGeoReferencedLocalities
FROM (
  SELECT
    COUNT(*) AS total_count,
    SUM(CASE WHEN Latitude1 IS NOT NULL THEN 1 ELSE 0 END) AS georeferenced_count
  FROM locality
  WHERE DisciplineID = 3
) AS totals;

[emenslin]

Summary of existing issues:

• Sometimes missing notifications and stuck on "default tree creation started" dialog when uploading a prepopulated tree. (==> ok for this release)
• Fungi and Minerology tree not loading properly after initial upload (==> ok for this release)
• Chronostrat tree is now populated after creating a new discipline, however, it is very slow and takes several minutes to load, even if not prepopulating geography or taxon trees. (==> ok for this release)
• Possibly fixed incorrectly displaying the "Tree is loading, please wait..." message.
• Possibly fixed being able to import into an existing tree multiple times. It should refresh after closing the tree progress dialog, but I haven't fully tested it yet. (==> Fix with preventing user to access bd/collection before every task is finished)
• When creating a new discipline "Collection Object" is misnamed "collectionobject", I imagine this might be related to the misnamed tree issue mentioned here (although in this case it was a different discipline not division), but I think users should not be able to open up a collection if the schema config hasn't been created yet, I feel like this could lead to even more problems. (==> Fix with preventing user to access bd/collection before every task is finished)

• We need to consider having some sort of indication that various processes are happening, whether that be notifications or different wording. Right now if you open up a new discipline the schema config might still be in the process of being created (see previous point), however, if you pre-loaded a geography and taxon tree and look at the each of them, they will both say they're being created even though they haven't even started yet. I feel like this could be very confusing to users, especially when certain trees take a really long time to upload which can be very misleading.

[grantfitzsimmons]

Responding to @emenslin's last comment

We need to consider having some sort of indication that various processes are happening, whether that be notifications or different wording. Right now if you open up a new discipline the schema config might still be in the process of being created (see previous point), however, if you pre-loaded a geography and taxon tree and look at the each of them, they will both say they're being created even though they haven't even started yet. I feel like this could be very confusing to users, especially when certain trees take a really long time to upload which can be very misleading.

I wonder why users can start using the database before all resources are created. Is there a benefit to this now that tree creation is much faster? It seems they will encounter unfinished sections of the collection/discipline, such as incomplete trees and/or unfinished schema config generation. The advantage of using the app early seems to be outweighed by the downsides.

[Iwantexpresso]

HI Here is a comment expanding on a smaller issue I have been able to find, This is related to the populate tree with default records button and initialize empty tree button.

when the populate button is first clicked on, you are allowed to click the create empty tree button as well.
 Here is a video better showing the actions that can be taken:
https://github.com/user-attachments/assets/1d8e884e-3589-4611-aa1c-51f42d4d7562
Resulting in the following tree setup below having both a life and a root node at its origin.

As I have briefly mentioned before I don't believe it to be a critical issue and functionality wise nothing is broken, but it is still worth mentioning to keep a record of its existence. I have briefly discussed this with Alejandro and a possible solution could be to make the page reload after clicking the close button on the default tree creation, the one shown below, since this would properly set the please wait until default tree is created option.


[CarolineDenis]

Decision to fix most of mentioned issues:

• At first run (Guided Setup), we should not let the user work in the app until all necessary resources are created.

• When using System Config and adding a discipline, we don't make it possible to log into the new collection until all resources are ready.
  It shouldn't even appear in the collection list until it is done.

Once this is implemented, we can remove the guard in the tree viewer with the warning letting the user know that a tree is being created.
All codes on frontend and backend that were added for this feature can also be removed.

[acwhite211]

Sounds good. I think the simplest way to determine that it is all done with the setup process is to check when the worker task is completed.

[acwhite211]

I pushed a commit that helps block users from using unfinished records from setup and config tools.

For the setup tool, extra checks on all the background tasks were added to the setup_progress API function, so that the loading page will not complete until all background tasks associated with the setup are finished.

When creating a new collection, I added a filter to the choose_collection API request, so that is will filter out newly created collections that are not finished being created yet. This is done by checking if any background tasks are still running, and if so, then filter out recently created collections as options for logging into for the user.

I've just made back-end changes so far. We might want to add a similar loading screen when creating a new collection in the config tool, like we do in the setup tool. I created a new API config_progress that acts like the setup_progress API. If we want, we can use this API in the config tool, just like we do in the setup tool to create a loading screen. So currently, creating a collection in the config tool looks the same, it's just that the user can't login to the new collection until all background tasks are finished.

I also added a new API for getting all running background tasks. This might be an easy way in the future for front-end to make decisions on how to act without needing to go through notifications.

[grantfitzsimmons]

When new disciplines are created and added (including at first run), there is no discipline app resource directory created.

That itself is OK, as this mirrors what Specify 6 does. The problem is that when you try to add an app resource to a discipline, it simply puts it under the collection.

Screen.Recording.2026-02-25.at.9.12.24.AM.mov

This makes it impossible to add a new app resource under a discipline. In Specify 6, if you added a new app resource to a discipline, it would simply create a new row in teh spappresourcedir table to give the resource a new home:

SpAppResourceDirID	TimestampCreated	TimestampModified	Version	DisciplineType	IsPersonal	UserType	CollectionID	SpecifyUserID	ModifiedByAgentID	DisciplineID	CreatedByAgentID
4	2026-02-25 08:46:52.000	2026-02-25 08:46:52.000	0	Botanique	0			1		3	1

We need to implement that logic here as well, else users will be unable to establish discipline resources using newly created disciplines/databases.

[acwhite211]

Made a new api call /context/all_system_data.json for getting all collections for the config tool view.
The /context/all_system_data.json api will filter out new collections when the background workers are still running.

The config tool is using a separate api now for getting all the collections, so they will see the new collection created immediately. The collection switching dialog box will not show the new collection until all the background tasks are done. The only thing that's not ideal, is that the front-end only calls the all_system_data.json once, so the user will have to refresh the page to be able to switch to the new collection.

[grantfitzsimmons]

@acwhite211 For users who are currently missing their discipline app resource dir, does this solve it for them?

[bhumikaguptaa]

1. Setup Flow
Setup Forms

• All setup forms must display correctly in both light mode and dark mode (based on system settings).
• Users can only progress to the next form once all required fields are completed.
• User selections must be reflected in the Overview sidebar in real time.
• The setup flow must be submittable at the end.
• Users must be able to log into the database after setup completes.

Default Data & Configuration

• Default schema configuration captions and descriptions must be populated (not blank).
• Default picklists must be created correctly.
  • Note: The Agent Type picklist does not exist in the schema config; this is expected.
• Default preparation types must be created correctly.
• All tree viewer pages must load without errors.
• The admin user’s agent must be created correctly and must always include a last name.
• An empty Global Preferences file must be created in App Resources.

2. Tree Configuration During Setup
Storage Tree

• Users must be able to modify the storage tree configuration during setup.
• Changes must be correctly reflected in the application after setup.

Geography Tree

• Users must be able to modify the geography tree configuration during setup.
• Both options must work:
  Preloaded tree
  Empty tree

Taxon Tree

• Users must be able to select a preloaded default tree when one exists.
• The selected tree must be correctly preloaded and visible when the app opens.

Geology Discipline

• Relevant tectonic and chronostratigraphic (chrono) trees must be created automatically.

3. Tree Viewer: Preloaded & Empty Trees

• Users can create an empty tree from the tree viewer.
• Users can modify rank configuration in the tree viewer.
• Users can create a preloaded tree from the tree viewer.
• Empty trees now have an upload button. The Upload Tree icon under root creation must work as expected.
• Users can delete a rank with no associated nodes.
• If ranks are deleted and a tree is then loaded, the tree configuration must be respected.
• Users receive notifications for:
  Tree creation starts
  Successful completion
  Errors or cancellation

4. Configuration Tool
Institution Tree Viewer

• Users cannot access config tool if they are not an institution admin.
• Users can click and drag to view all nodes in the institution tree.
• Clicking a resource node opens the resource form.
• Users cannot create a discipline with a discipline name that is already in use.

I was able to create a discipline with a discipline name that is already in use in a different division (not sure about that intended behavior)
I also get an error message if there are two disciplines with the same name, with/without a trailing space, like 'Ichthyology' and 'Ichthyology '. There shouldn't be an error message; the system should just not let it save.

Specify 7 Crash Report - 2026-02-25T21_34_51.304Z.txt

• Users can:
  • Edit a resource
  • Create new resources
  • Delete collections (created in 7) (the ones they are not logged into)
• Users cannot create collections through the subview in a discipline form.
• Users can collapse and expand sections in the left panel.
• Users can switch between vertical and horizontal views in the institution tree viewer.

5. App resources

• User can create new app resources under collections
• User can create new app resources under any disciplines

---

Once the preloaded tree is created, the screen refreshes, but doesn't open the tree; it is on the default 'taxon' tree. The same goes for empty trees. Also, for empty trees, there are no notifications (unsure if only preloaded trees are supposed to have them). Everything else works as expected.
Link: https://blankdbsetup2b320260225-issue-2931-2.test.specifysystems.org/specify/system-configuration/
(Expected)

[acwhite211]

@grantfitzsimmons The solution applies the fix when you create the new discipline. So disciplines that were created in the config tool before the fix might still have the issue.

[acwhite211]

@grantfitzsimmons I added a task to the key migrations that makes sure that all existing disciplines have a correctly setup app resource directory. The fix is in setup_tool 0001_ensure_discipline_resource_dirs migration, so anyone upgrading will have their disciplines fixed.

[kwhuber]

1. Setup Flow
Setup Forms

• All setup forms must display correctly in both light mode and dark mode (based on system settings).
• Users can only progress to the next form once all required fields are completed.
• User selections must be reflected in the Overview sidebar in real time.
• The setup flow must be submittable at the end.
• Users must be able to log into the database after setup completes.

Default Data & Configuration

• Default schema configuration captions and descriptions must be populated (not blank).
• Default picklists must be created correctly.
  • Note: The Agent Type picklist does not exist in the schema config, this is expected.
• Default preparation types must be created correctly.
• All tree viewer pages must load without errors.
• The admin user’s agent must be created correctly and must always include a last name.
• An empty Global Preferences file must be created in App Resources.

2. Tree Configuration During Setup
Storage Tree

• Users must be able to modify the storage tree configuration during setup.
• Changes must be correctly reflected in the application after setup.

Geography Tree

• Users must be able to modify the geography tree configuration during setup.
• Both options must work:
  Preloaded tree
  Empty tree

Taxon Tree

• Users must be able to select a preloaded default tree when one exists.
• The selected tree must be correctly preloaded and visible when the app opens.

Geology Discipline

• Relevant tectonic and chronostratigraphic (chrono) trees must be created automatically.

• Tested with Invertebrate Paleontology

3. Tree Viewer: Preloaded & Empty Trees

• Users can create an empty tree from the tree viewer.
• Users can modify rank configuration in the tree viewer.
• Users can create a preloaded tree from the tree viewer.

• Minerals looks good (although it takes 30-40 seconds to load...even when switching between different Taxon trees after it had been created)! Still, I don't get transferred to the new tree I created when the creation process is completed...I am taken back to the tree I initially clicked the add button.

• Empty trees now have an upload button. The Upload Tree icon under root creation must work as expected.

• Upon creating an empty Botany tree and then importing a populated tree, Botany (Bryophyta), I was given the choice to add the missing Phylum rank which I said yes to. After creation, this rank was not included.

• Looking at taxon_botany_tree.json, there are even other ranks included in the screenshot that are not defined here (Division and Subdivision). Is Subphylum also left out?

• I noticed other disciplines don't strictly follow their json definition either so perhaps you are correct with your implementation, but wanted to ask.

• Users can delete a rank with no associated nodes.
• If ranks are deleted and a tree is then loaded, the tree configuration must be respected.
• Users receive notifications for:
  Tree creation start
  Successful completion
  Errors or cancellation

4. Configuration Tool
Institution Tree Viewer

• Users cannot access config tool if they are not an institution admin.
• Users can click and drag to view all nodes in the institution tree.
• Clicking a resource node opens the resource form.
  • Users cannot create a discipline with a discipline name that is already in use.

• @bhumikaguptaa In your most recent review, I think you may have attempted this at the Division level! Still, important to note as the same name can be saved by adding a space. This behavior is also present at the collection level.

• Users can:
  • Edit a resource
  • Create new resources
  • Delete collections (created in 7) (the ones they are not logged into)
• Users cannot create collections through the subview in a discipline form.
• Users can collapse and expand sections in the left panel.
• Users can switch between vertical and horizontal views in the institution tree viewer.

5. App resources

• User can create new app resources under collections
• User can create new app resources under any disciplines