Add valid chronostrat filter to geo_time relativeage and paleocontext queries

[acwhite211]

Fixes #5410

Add a filter to the 'specify_row' API so that the age names in the extended age query's picklist will not include invalid Chronostrat records.

Also, Add valid Chronostrat filter to RelativeAge and PaleoContext queries. This done to filter Chronostrat names in the extended age filter to remove all names that have a start period smaller than the end period.

Address comment #5272 (review)

Checklist

• Self-review the PR after opening it to make sure the changes look good
  and self-explanatory (or properly documented)
• Add automated tests
• Add relevant issue to release milestone

Testing instructions

The first part of testing this PR is to make sure that all the values in the age name picklist are valid Choronostrats. A valid Chronostrat/GeologicTimePeriod needs to have a StartPeriod and EndPeriod that are not null, also the StartPeriod value needs to be larger than the EndPeriod.

Use any database on the test-panel that has values in the GeologicTimePeriod table.

1. Open a new QB on the Chronostratigraphy table. Select '(any rank)', and then add the following fields: ID, Name, Start Period, End Period. Run the query. In the query results note down some of the Name values that are invalid due to either the start period or end period being null. Also, if there are any rows where the End Time is larger than the start time, then note down that name as well for being invalid. We'll check some of those names in the next step.

2. Open a new QB on the CollectionObject table. Select the Age field. Instead of 'Any', select 'Name' in the options. In the picklist that appears next to 'Name', verify that each of the names present are valid Chronostratigraphy records. Review the query results from the Chronostratigraphy query in the previous step to verify that the names in the picklist are valid. Verify that the names noted down in the previous step do not appear in this picklist.

This next part of the testing will need to show that COs associated with an invalid Chronostrat, which is where the StartPeriod is less than the EndPeriod, does not show up in an extended age query.

Use the ciscollections_age_qb database on the test-panel.

1. Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Set the 'start_time' to 1000 and set the 'end_time' to 0. Make sure that for this query, that the value 40064 is NOT present in the query results of the 'Age' column. This is because the CO with ID 40046 points to a GeologicTimePeriod with a start period of 10 and an end period of 90, this is invalid and thus shouldn't appear in the results.

[CarolineDenis]

@grantfitzsimmons, which db were you using when you discovered this issue? Do you have some ages references to provide?

[pashiav]

Testing instructions

The testing will need to show that COs associated with an invalid Chronostrat, which is where the StartPeriod is less than the EndPeriod, does not show up in an extended age query.

Use the ciscollections_age_qb database on the test-panel.

1. Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Set the 'start_time' to 0 and set the 'end_time' to 1000. Make sure that for this query, that the value 40064 is NOT present in the query results of the 'Age' column. This is because the CO with ID 40046 points to a GeologicTimePeriod with a start period of 10 and an end period of 90, this is invalid and thus shouldn't appear in the results.

---

Error pops up upon querying with start time 0 and end time 1000, so I could not check the results.

Specify 7 Crash Report - 2024-11-25T15_07_12.088Z.txt

[grantfitzsimmons]

Use the ciscollections_age_qb database on the test-panel.

These testing instructions are too narrow, in my opinion. We need better testing coverage. We cannot expect testers to go beyond the bounds of the instructions as they are written. Therefore, we should evaluate this under multiple circumstances.

You can iterate through all databases on the test panel and run this script (seen here):

/* 
This finds all disciplines in a database that are paleo-based, meaning
they can use PaleoContext, GeologicTimePeriod, and Lithostratigraphy 
*/

select Name from discipline where Type IN ('vertpaleo','invertpaleo','paleobotany');

And from here you can find a variety of databases that have disciplines which include a valid geologictimeperiod table and tree. I think we should test a few!

KUInvertPaleo_2024_06_06, KUPaleoBot_2024_06_06, morpaleo_6_4_24_10_25_2024, uw_geo_collections_08_09_2024, etc.

[grantfitzsimmons]

Update: Here you go. You can do this with any script, see how here:
https://github.com/specify/data-management/blob/main/test-panel-databases/runQueryOnAllDatabases.py

---

Database Results Summary

ChadronTest

• Name: Vertebrate Paleontology

ChadronTest2

• Name: Vertebrate Paleontology

ChadronTest_fix

• Name: Vertebrate Paleontology

Elizabeth_geo_swiss_forms

• Name: Invertebrate Paleontology

GeoSpecifyCOG

• Name:
• Invertebrate Paleontology
• Vertebrate Paleontology
• Geology

KUEntomology_2024_06_06

• Name: Invertebrate Paleontology

KUInvertPaleo_2024_06_06

• Name: Invertebrate Paleontology

KUInvertPaleo_2024_06_06_migration5333

• Name: Invertebrate Paleontology

KUPaleoBot_2024_06_06

• Name: Paleobotany

KUPaleoBot_2024_06_06_migration5345

• Name: Paleobotany

KUVertPaleo_1_11_23

• Name: Vertebrate Paleontology

KUVertPaleo_2024_06_06

• Name: Vertebrate Paleontology

KUVertPaleo_2024_06_06_5324

• Name: Vertebrate Paleontology

KUZooarchaeology_2024_06_06

• Name: Zooarchaeology

PriPaleo_SchemaUpdated_CM_202304211540

• Name: Invertebrate Paleontology

UOPD_1_18_23

• Name:
• Vertebrate Paleontology
• Invertebrate Paleontology
• Paleobotany

all_discipline_defaults

• Name:
• Vertebrate Paleontology
• Invertebrate Paleontology

beaty_2024_07_03

• Name: Paleontology

bulgaria_nmnhs_4_11_24

• Name:
• Paleozoology
• Vertebrate Paleontology
• Paleobotany

calvertmarinemuseum_2024_11_14

• Name:
• Invertebrate Paleontology
• Vertebrate Paleontology
• Casts

ciscollections_2_15_24

• Name:
• Invertebrate Paleontology
• Vertebrate Paleontology
• Rocks and Minerals

ciscollections_age_qb

• Name:
• Invertebrate Paleontology
• Vertebrate Paleontology
• Rocks and Minerals

ciscollections_age_qb_issue_5180

• Name:
• Invertebrate Paleontology
• Vertebrate Paleontology
• Rocks and Minerals

ciscollections_age_qb_migration5272

• Name:
• Invertebrate Paleontology
• Vertebrate Paleontology
• Rocks and Minerals

ciscollections_age_qb_migration5272_2

• Name:
• Invertebrate Paleontology
• Vertebrate Paleontology
• Rocks and Minerals

geo_defaults

• Name: Geology

geo_swiss_2024_08_05

• Name: Invertebrate Paleontology

geology

• Name: Invertebrate Paleontology

geology_blank_tree

• Name: Invertebrate Paleontology

gnm_2024_10_10

• Name: Geology

gothenburg_9_1_23

• Name: Geology

hujinnhc_103024_migration

• Name: Vertebrata

hujinnhc_2024_07_01

• Name: Vertebrata

hujinnhc_9_6_23

• Name: Vertebrata

hujinnhc_geology

• Name: Vertebrata

jerusalem_with_changes_2_19

• Name: Vertebrata

lexi_2024_11_14_16_23_09

• Name: Micropaleontology

lexi_NCForms_2

• Name: Vertebrate Paleontology

lexi_NCforms_8_12

• Name: Naturalist Center

lexi_wyomingforms

• Name: Paleobotany

lexi_wyomingforms2

• Name: Paleobotany

morpaleo_6_4_24_10_25_2024

• Name: Vertebrate Paleontology

mpabi_2024_06_04

• Name: MPABI

naturkundemuseum_11_20_2024_latest

• Name:
• Invertebrate Paleontology Ehrenberg
• Invertebrate Paleontology
• Vertebrate Paleontology
• Paleobotany
• Paleontology

naturkundemuseum_2024_11_07

• Name:
• Invertebrate Paleontology Ehrenberg
• Invertebrate Paleontology
• Vertebrate Paleontology
• Paleobotany
• Paleontology

ndGeoBackup_2023

• Name: Vertebrate Paleontology

ojsmnh_1_10_24

• Name: Paleontology

ojsmnh_2024_10_03_WB

• Name: Paleontology

oldChadronTest2_migration

• Name: Vertebrate Paleontology

testpaleo_2024_11_21

• Name: Invertebrate Paleontology

ummp_9_19_23

• Name:
• UMMP Invertebrate Paleontology
• Paleobotany

uw_geo_collections_08_09_2024

• Name:
• Vertebrate Paleontology
• PaleoBotany

[acwhite211]

@pashiav sorry, got the the start_time and end_time numbers mixed up for the query. It should be "Set the 'start_time' to 1000 and set the 'end_time' to 0." That way you shouldn't get the error.

Thanks @grantfitzsimmons, I'll create more testing instructions that involve other databases.

[acwhite211]

The latest commit adds filtering to the 'specify_rows' api so that only valid GeologicTimePeriod records are returned. This will prevent invalid age names from being available in the extend age query picklist.

[acwhite211]

@specify/testing @specify/dev-testing The testing instructions have been updated and can be tested again. The ciscollections_age_qb needs to be used in the second part of the test. The first part of the test can be any database that has records in the GeologicTimePeriod table.

[pashiav]

Testing instructions

The first part of testing this PR is to make sure that all the values in the age name picklist are valid Choronostrats. A valid Chronostrat/GeologicTimePeriod needs to have a StartPeriod and EndPeriod that are not null, also the StartPeriod value needs to be larger than the EndPeriod.

Use any database on the test-panel that has values in the GeologicTimePeriod table.

1. Open a new QB on the Chronostratigraphy table. Select '(any rank)', and then add the following fields: ID, Name, Start Period, End Period. Run the query. In the query results note down some of the Name values that are invalid due to either the start period or end period being null. Also, if there are any rows where the End Time is larger than the start time, then note down that name as well for being invalid. We'll check some of those names in the next step.

2. Open a new QB on the CollectionObject table. Select the Age field. Instead of 'Any', select 'Name' in the options. In the picklist that appears next to 'Name', verify that each of the names present are valid Chronostratigraphy records. Review the query results from the Chronostratigraphy query in the previous step to verify that the names in the picklist are valid. Verify that the names noted down in the previous step do not appear in this picklist.

This next part of the testing will need to show that COs associated with an invalid Chronostrat, which is where the StartPeriod is less than the EndPeriod, does not show up in an extended age query.

Use the ciscollections_age_qb database on the test-panel.

1. Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Set the 'start_time' to 1000 and set the 'end_time' to 0. Make sure that for this query, that the value 40064 is NOT present in the query results of the 'Age' column. This is because the CO with ID 40046 points to a GeologicTimePeriod with a start period of 10 and an end period of 90, this is invalid and thus shouldn't appear in the results.

---

Looks good! Everything is working as expected.

*tested on: KUInvertPaleo_2024_06_06, KUPaleoBot_2024_06_06, morpaleo_6_4_24_10_25_2024, uw_geo_collections_08_09_2024, and ciscollections_age_qb

[Areyes42]

Testing instructions

The first part of testing this PR is to make sure that all the values in the age name picklist are valid Choronostrats. A valid Chronostrat/GeologicTimePeriod needs to have a StartPeriod and EndPeriod that are not null, also the StartPeriod value needs to be larger than the EndPeriod.

Use any database on the test-panel that has values in the GeologicTimePeriod table.

• Open a new QB on the Chronostratigraphy table. Select '(any rank)', and then add the following fields: ID, Name, Start Period, End Period. Run the query. In the query results note down some of the Name values that are invalid due to either the start period or end period being null. Also, if there are any rows where the End Time is larger than the start time, then note down that name as well for being invalid. We'll check some of those names in the next step.

• Open a new QB on the CollectionObject table. Select the Age field. Instead of 'Any', select 'Name' in the options. In the picklist that appears next to 'Name', verify that each of the names present are valid Chronostratigraphy records. Review the query results from the Chronostratigraphy query in the previous step to verify that the names in the picklist are valid. Verify that the names noted down in the previous step do not appear in this picklist.

This next part of the testing will need to show that COs associated with an invalid Chronostrat, which is where the StartPeriod is less than the EndPeriod, does not show up in an extended age query.

Use the ciscollections_age_qb database on the test-panel.

• Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Set the 'start_time' to 1000 and set the 'end_time' to 0. Make sure that for this query, that the value 40064 is NOT present in the query results of the 'Age' column. This is because the CO with ID 40046 points to a GeologicTimePeriod with a start period of 10 and an end period of 90, this is invalid and thus shouldn't appear in the results.

---

Everything looks good, besides the last part. 40064 shows up for me when querying on CO using those start and end times.

[acwhite211]

@Areyes42 for some reason that collection object's start and end period got changed. I edited it back, so the COID 40064 shouldn't show up. Let me know if it works now, thanks.

[pashiav]

Testing instructions

The first part of testing this PR is to make sure that all the values in the age name picklist are valid Choronostrats. A valid Chronostrat/GeologicTimePeriod needs to have a StartPeriod and EndPeriod that are not null, also the StartPeriod value needs to be larger than the EndPeriod.

Use any database on the test-panel that has values in the GeologicTimePeriod table.

1. Open a new QB on the Chronostratigraphy table. Select '(any rank)', and then add the following fields: ID, Name, Start Period, End Period. Run the query. In the query results note down some of the Name values that are invalid due to either the start period or end period being null. Also, if there are any rows where the End Time is larger than the start time, then note down that name as well for being invalid. We'll check some of those names in the next step.

2. Open a new QB on the CollectionObject table. Select the Age field. Instead of 'Any', select 'Name' in the options. In the picklist that appears next to 'Name', verify that each of the names present are valid Chronostratigraphy records. Review the query results from the Chronostratigraphy query in the previous step to verify that the names in the picklist are valid. Verify that the names noted down in the previous step do not appear in this picklist.

This next part of the testing will need to show that COs associated with an invalid Chronostrat, which is where the StartPeriod is less than the EndPeriod, does not show up in an extended age query.

Use the ciscollections_age_qb database on the test-panel.

1. Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Set the 'start_time' to 1000 and set the 'end_time' to 0. Make sure that for this query, that the value 40064 is NOT present in the query results of the 'Age' column. This is because the CO with ID 40046 points to a GeologicTimePeriod with a start period of 10 and an end period of 90, this is invalid and thus shouldn't appear in the results.

---

The value 40064 is not present in the query results. Looks good!

[alesan99]

Testing instructions

Use any database on the test-panel that has values in the GeologicTimePeriod table.

1. Open a new QB on the Chronostratigraphy table. Select '(any rank)', and then add the following fields: ID, Name, Start Period, End Period. Run the query. In the query results note down some of the Name values that are invalid due to either the start period or end period being null. Also, if there are any rows where the End Time is larger than the start time, then note down that name as well for being invalid. We'll check some of those names in the next step.

2. Open a new QB on the CollectionObject table. Select the Age field. Instead of 'Any', select 'Name' in the options. In the picklist that appears next to 'Name', verify that each of the names present are valid Chronostratigraphy records. Review the query results from the Chronostratigraphy query in the previous step to verify that the names in the picklist are valid. Verify that the names noted down in the previous step do not appear in this picklist.

Use the ciscollections_age_qb database on the test-panel.

1. Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Set the 'start_time' to 1000 and set the 'end_time' to 0. Make sure that for this query, that the value 40064 is NOT present in the query results of the 'Age' column. This is because the CO with ID 40046 points to a GeologicTimePeriod with a start period of 10 and an end period of 90, this is invalid and thus shouldn't appear in the results.

Working as expected 👍👍👍👍
Tested 1 and 2 on kuinvertpaleo20240606