Add new age filter in QB - #5272
Conversation
|
Here is the QB request body to the {
"name": "New Query",
"contextname": "CollectionObject",
"contexttableid": 1,
"selectdistinct": false,
"countonly": false,
"formatauditrecids": false,
"specifyuser": "/api/specify/specifyuser/1/",
"isfavorite": true,
"ordinal": 32767,
"fields": [
{
"tablelist": "1",
"stringid": "1.collectionobject.age",
"fieldname": "age",
"isrelfld": false,
"sorttype": 0,
"position": 2,
"isdisplay": true,
"operstart": 16,
"startvalue": "10,100",
"isnot": false
},
{
"tablelist": "1",
"stringid": "1.collectionobject.catalogNumber",
"fieldname": "catalogNumber",
"isrelfld": false,
"sorttype": 0,
"position": 0,
"isdisplay": true,
"operstart": 8,
"startvalue": "",
"isnot": false
}
],
"_tablename": "SpQuery",
"remarks": null,
"searchsynonymy": null,
"smushed": null,
"sqlstr": null,
"timestampcreated": "2024-08-05",
"timestampmodified": null,
"version": 1,
"createdbyagent": null,
"modifiedbyagent": null,
"offset": 0,
"limit": 40
}TODO: Decide what we want to return in the age column. Right now the operation does does the filtering. |
Triggered by b4a65a1 on branch refs/heads/issue-5180
combs-a
left a comment
There was a problem hiding this comment.
Testing instructions
This PR affects database migrations. See migration testing instructions (here)
Use the ciscollections_age_qb database in the test panel. Use the 'Rocks' collection.
Go to the QB and select Collection Object, then verify the following queries. Note the the value returned in the 'Age' column is the CollectionObjectID, so no need to add an ID column in the query.
-
Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Try setting the 'start_time' to a value that is less than the 'end_time' value. When the query runs, it should through an error "Start time must be greater than or equal to end time".
-
Failed because: see end of review.
-
Create a Collection Object Query. Select the 'Age' field. Select the 'Any' filter. Run the query and make sure no errors occur.
-
Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Use the following table to make multiple queries with differing values for the 'start_time', 'end_time', and 'strict' in the 'Range' filter. Make sure that for all these queries, that the value
40057is present in the query results of the 'Age' column. -
Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Use the following table to make multiple queries with differing values for the 'start_time', 'end_time', and 'strict' in the 'Range' filter. Make sure that for all these queries, that the value
40057is NOT present in the query results of the 'Age' column. -
Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Use the following table to make multiple queries with differing values for the 'start_time', 'end_time', and 'strict' in the 'Range' filter. Make sure that for all these queries, that the value
40058is present in the query results of the 'Age' column. -
Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Use the following table to make multiple queries with differing values for the 'start_time', 'end_time', and 'strict' in the 'Range' filter. Make sure that for all these queries, that the value
40058is NOT present in the query results of the 'Age' column. -
Failed because: 180-160-true, 180-100-true, 300-160-true tests returned
40058when they should not have (Same as Pashia) -
Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Use the following table to make multiple queries with differing values for the 'start_time', 'end_time', and 'strict' in the 'Range' filter. Make sure that for all these queries, that the value
40059is present in the query results of the 'Age' column. -
Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Set the 'start_time' to 170 and 'end_time' to 130. Run the query and verify that the Collection Object
40060is returned for a query results. -
Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Set the 'start_time' to 550 and 'end_time' to 500. Run the query and verify that the Collection Object
40061is returned for a query results. -
Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Set the 'start_time' to 200 and 'end_time' to 140. Run the query and verify that the Collection Object
40063is returned for a query results. -
Test the age query with the 'Name'. Here are a defined subset of CollectionObjectIDs that should be included in the returned query results with a given age period name:
-
Failed because: No Late Jurassic or Franconian ages were in the list of names. Other tests passed.
-
Create a Collection Object Query. Select the 'Age' field. Select the 'Name' filter. Set the 'start_time' to 170 and 'end_time' to 130. Run the query and verify that the Collection Object
40060is returned for a query results. -
Is this test correct? You cannot set start and end time if you have Name selected.
-
Do age queries with 'Any' and explore including the follow query field paths and make sure that column in the query results have at least one row with data populated. Might be good to just set each field to 'Not Empty' one at a time, to make sure rows are returned:
- collectionobject->absoluteage
- collectionobject->relativeage->chronostrat
- collectionobject->paleocontext->chronostrat
- collectionobject->collectionevent->paleocontext->chronostrat
- Collectionobject->collectionevent->locality->paleocontext->chronostrat
Tester comments:
So for the start and end time, it does produce an error when start is less than end--however, it seems that the same error is being thrown twice. This should be a single error.
It also incorrectly throws errors like Pashia ran into--I did some range testing, and it seems like when the end time is more than a tenth of the start time and less than 100 it gives the start value error, if that helps narrow down what's causing the issue.
agefilter1.mp4
Also, I'm wondering about returning the CO ID in that column. Oftentimes fields are added to simply filter the results, and I don't think the current behavior makes sense looking at the rest of Specify.
I'm also eh about the Name field being a picklist, since that list can get so big.
@pashiav : I also think that it should be cleared. Unless we're going to add support for having a range of names, strings cause errors.
Good work so far! This filter is pretty neat.
pashiav
left a comment
There was a problem hiding this comment.
Testing instructions
This PR affects database migrations. See migration testing instructions (here)
Use the ciscollections_age_qb database in the test panel. Use the 'Rocks' collection.
Go to the QB and select Collection Object, then verify the following queries. Note the the value returned in the 'Age' column is the CollectionObjectID, so no need to add an ID column in the query.
- Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Try setting the 'start_time' to a value that is less than the 'end_time' value. When the query runs, it should through an error "Start time must be greater than or equal to end time".
- Create a Collection Object Query. Select the 'Age' field. Select the 'Any' filter. Run the query and make sure no errors occur.
- Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Use the following table to make multiple queries with differing values for the 'start_time', 'end_time', and 'strict' in the 'Range' filter. Make sure that for all these queries, that the value
40057is present in the query results of the 'Age' column.
| start_time | end_time | strict |
|---|---|---|
| 300 | 100 | false |
| 300 | 200 | false |
| 200 | 100 | false |
| 300 | 201 | false |
| 199 | 100 | false |
| 300 | 199 | false |
| 201 | 100 | false |
| 300 | 100 | true |
| 201 | 199 | true |
- Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Use the following table to make multiple queries with differing values for the 'start_time', 'end_time', and 'strict' in the 'Range' filter. Make sure that for all these queries, that the value
40057is NOT present in the query results of the 'Age' column.
| start_time | end_time | strict |
|---|---|---|
| 300 | 250 | false |
| 150 | 100 | false |
| 300 | 250 | true |
| 150 | 100 | true |
- Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Use the following table to make multiple queries with differing values for the 'start_time', 'end_time', and 'strict' in the 'Range' filter. Make sure that for all these queries, that the value
40058is present in the query results of the 'Age' column.
| start_time | end_time | strict |
|---|---|---|
| 220 | 140 | false |
| 180 | 160 | false |
| 220 | 100 | false |
| 300 | 140 | false |
| 180 | 100 | false |
| 300 | 160 | false |
| 300 | 100 | false |
| 300 | 100 | true |
| 195 | 145 | true |
- Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Use the following table to make multiple queries with differing values for the 'start_time', 'end_time', and 'strict' in the 'Range' filter. Make sure that for all these queries, that the value
40058is NOT present in the query results of the 'Age' column.
| start_time | end_time | strict |
|---|---|---|
| 300 | 250 | false |
| 120 | 100 | false |
| 300 | 250 | true |
| 120 | 100 | true |
| 180 | 160 | true |
| 180 | 100 | true |
| 300 | 160 | true |
- Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Use the following table to make multiple queries with differing values for the 'start_time', 'end_time', and 'strict' in the 'Range' filter. Make sure that for all these queries, that the value
40059is present in the query results of the 'Age' column.
| start_time | end_time | strict |
|---|---|---|
| 200 | 100 | false |
| 200 | 10 | false |
- Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Set the 'start_time' to 170 and 'end_time' to 130. Run the query and verify that the Collection Object
40060is returned for a query results. - Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Set the 'start_time' to 550 and 'end_time' to 500. Run the query and verify that the Collection Object
40061is returned for a query results. - Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Set the 'start_time' to 200 and 'end_time' to 140. Run the query and verify that the Collection Object
40063is returned for a query results. - Test the age query with the 'Name'. Here are a defined subset of CollectionObjectIDs that should be included in the returned query results with a given age period name:
| name | co_id |
|---|---|
| Oxfordian | 40063 |
| Jurassic | 40060 |
| Jurassic | 40058 |
- Do age queries with 'Any' and explore including the follow query field paths and make sure that column in the query results have at least one row with data populated. Might be good to just set each field to 'Not Empty' one at a time, to make sure rows are returned:
- collectionobject->absoluteage
- ollectionobject->relativeage->chronostrat
- collectionobject->paleocontext->chronostrat
- collectionobject->collectionevent->paleocontext->chronostrat
- Collectionobject->collectionevent->locality->paleocontext->chronostrat
Issues
- Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Use the following table to make multiple queries with differing values for the 'start_time', 'end_time', and 'strict' in the 'Range' filter. Make sure that for all these queries, that the value
40058is NOT present in the query results of the 'Age' column.
Start and end times in this table still have 40058 present in the query results:
| start_time | end_time | strict |
|---|---|---|
| 180 | 160 | true |
| 180 | 100 | true |
| 300 | 160 | true |
Everything else looks good! Great job!!
Note for future testers:
From #5272 (review):
If a "Name" option is selected, then the filter is changed to "Range", the option selected for Name is the input for the start time of Range.
This will be fixed in a later pr.
There was a problem hiding this comment.
I am still doing a full review, but I wanted to surface this concern before we proceed;
Currently, in a tree with the default Chronostratigraphy tree (no changes from the user, the one we ship in Specify 6), I get the following error:
AssertionError at /stored_query/ephemeral/ Start time must be greater than or equal to end time.
select count(*) from geologictimeperiod
where EndPeriod > StartPeriod; # 114 records returned
select count(*) from geologictimeperiod
where EndPeriod < StartPeriod; # 48 records returned
select count(*) from geologictimeperiod
where EndPeriod IS NULL or StartPeriod IS NULL; # 3 records returnedThis means that 114 chronostratigraphy tree records cannot use the 'Age' function, while only 48 can be used effectively. Can we handle this error more gracefully by preventing the search in these cases? It seems excessive for the system to crash when a user constructs an otherwise valid query.
emenslin
left a comment
There was a problem hiding this comment.
Testing instructions
This PR affects database migrations. See migration testing instructions (here)
Use the ciscollections_age_qb database in the test panel. Use the 'Rocks' collection.
Go to the QB and select Collection Object, then verify the following queries. Note the the value returned in the 'Age' column is the CollectionObjectID, so no need to add an ID column in the query.
- Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Try setting the 'start_time' to a value that is less than the 'end_time' value. When the query runs, it should through an error "Start time must be greater than or equal to end time".
- Create a Collection Object Query. Select the 'Age' field. Select the 'Any' filter. Run the query and make sure no errors occur.
- Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Use the following table to make multiple queries with differing values for the 'start_time', 'end_time', and 'strict' in the 'Range' filter. Make sure that for all these queries, that the value
40057is present in the query results of the 'Age' column.
| start_time | end_time | strict |
|---|---|---|
| 300 | 100 | false |
| 300 | 200 | false |
| 200 | 100 | false |
| 300 | 201 | false |
| 199 | 100 | false |
| 300 | 199 | false |
| 201 | 100 | false |
| 300 | 100 | true |
| 201 | 199 | true |
- Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Use the following table to make multiple queries with differing values for the 'start_time', 'end_time', and 'strict' in the 'Range' filter. Make sure that for all these queries, that the value
40057is NOT present in the query results of the 'Age' column.
| start_time | end_time | strict |
|---|---|---|
| 300 | 250 | false |
| 150 | 100 | false |
| 300 | 250 | true |
| 150 | 100 | true |
- Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Use the following table to make multiple queries with differing values for the 'start_time', 'end_time', and 'strict' in the 'Range' filter. Make sure that for all these queries, that the value
40058is present in the query results of the 'Age' column.
| start_time | end_time | strict |
|---|---|---|
| 220 | 140 | false |
| 180 | 160 | false |
| 220 | 100 | false |
| 300 | 140 | false |
| 180 | 100 | false |
| 300 | 160 | false |
| 300 | 100 | false |
| 300 | 100 | true |
| 195 | 145 | true |
- Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Use the following table to make multiple queries with differing values for the 'start_time', 'end_time', and 'strict' in the 'Range' filter. Make sure that for all these queries, that the value
40058is NOT present in the query results of the 'Age' column.
| start_time | end_time | strict |
|---|---|---|
| 300 | 250 | false |
| 120 | 100 | false |
| 300 | 250 | true |
| 120 | 100 | true |
| 180 | 160 | true |
| 180 | 100 | true |
| 300 | 160 | true |
- Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Use the following table to make multiple queries with differing values for the 'start_time', 'end_time', and 'strict' in the 'Range' filter. Make sure that for all these queries, that the value
40059is present in the query results of the 'Age' column.
| start_time | end_time | strict |
|---|---|---|
| 200 | 100 | false |
| 200 | 10 | false |
- Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Set the 'start_time' to 170 and 'end_time' to 130. Run the query and verify that the Collection Object
40060is returned for a query results. - Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Set the 'start_time' to 550 and 'end_time' to 500. Run the query and verify that the Collection Object
40061is returned for a query results. - Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Set the 'start_time' to 200 and 'end_time' to 140. Run the query and verify that the Collection Object
40063is returned for a query results. - Test the age query with the 'Name'. Here are a defined subset of CollectionObjectIDs that should be included in the returned query results with a given age period name:
| name | co_id |
|---|---|
| Oxfordian | 40063 |
| Jurassic | 40060 |
| Jurassic | 40058 |
- Do age queries with 'Any' and explore including the follow query field paths and make sure that column in the query results have at least one row with data populated. Might be good to just set each field to 'Not Empty' one at a time, to make sure rows are returned:
- collectionobject->absoluteage
- ollectionobject->relativeage->chronostrat
- collectionobject->paleocontext->chronostrat
- collectionobject->collectionevent->paleocontext->chronostrat
- Collectionobject->collectionevent->locality->paleocontext->chronostrat
- Error is thrown twice when start time is less than the end time
chrome_ZvriPwGet7.mp4
Specify 7 Crash Report - 2024-11-19T15_46_25.771Z.txt
- 40058 is present in the query results for the last three of that table
| start_time | end_time | strict |
|---|---|---|
| 180 | 160 | true |
| 180 | 100 | true |
| 300 | 160 | true |
- For the name filter, I have some concerns about how the list of names shows up, does it have to be this way or could it be a drop down like picklists are? It looks a little out of place and is hard to navigate (at least on a small screen, I'm not sure if it looks better on a bigger screen). Some other concerns are if you select an item and then click out of it, it is highlighted white and you can't read it, the background of the list is the same as the background of the query when highlighted so it blends in, and if you save a query with a name selected and reopen it the item selected shows up at the bottom instead of more centered. None of these are major, I just wanted to point them out since I could see some visual issues with these.
chrome_mpmYXOqz4U.mp4
This should be fixed now. Seemed to be a problem with the strict mode setting being passed to the age query logic. |
pashiav
left a comment
There was a problem hiding this comment.
Testing instructions
This PR affects database migrations. See migration testing instructions (here)
Use the ciscollections_age_qb database in the test panel. Use the 'Rocks' collection.
Go to the QB and select Collection Object, then verify the following queries. Note the the value returned in the 'Age' column is the CollectionObjectID, so no need to add an ID column in the query.
- Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Try setting the 'start_time' to a value that is less than the 'end_time' value. When the query runs, it should through an error "Start time must be greater than or equal to end time".
- Create a Collection Object Query. Select the 'Age' field. Select the 'Any' filter. Run the query and make sure no errors occur.
- Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Use the following table to make multiple queries with differing values for the 'start_time', 'end_time', and 'strict' in the 'Range' filter. Make sure that for all these queries, that the value
40057is present in the query results of the 'Age' column.
| start_time | end_time | strict |
|---|---|---|
| 300 | 100 | false |
| 300 | 200 | false |
| 200 | 100 | false |
| 300 | 201 | false |
| 199 | 100 | false |
| 300 | 199 | false |
| 201 | 100 | false |
| 300 | 100 | true |
| 201 | 199 | true |
- Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Use the following table to make multiple queries with differing values for the 'start_time', 'end_time', and 'strict' in the 'Range' filter. Make sure that for all these queries, that the value
40057is NOT present in the query results of the 'Age' column.
| start_time | end_time | strict |
|---|---|---|
| 300 | 250 | false |
| 150 | 100 | false |
| 300 | 250 | true |
| 150 | 100 | true |
- Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Use the following table to make multiple queries with differing values for the 'start_time', 'end_time', and 'strict' in the 'Range' filter. Make sure that for all these queries, that the value
40058is present in the query results of the 'Age' column.
| start_time | end_time | strict |
|---|---|---|
| 220 | 140 | false |
| 180 | 160 | false |
| 220 | 100 | false |
| 300 | 140 | false |
| 180 | 100 | false |
| 300 | 160 | false |
| 300 | 100 | false |
| 300 | 100 | true |
| 195 | 145 | true |
- Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Use the following table to make multiple queries with differing values for the 'start_time', 'end_time', and 'strict' in the 'Range' filter. Make sure that for all these queries, that the value
40058is NOT present in the query results of the 'Age' column.
| start_time | end_time | strict |
|---|---|---|
| 300 | 250 | false |
| 120 | 100 | false |
| 300 | 250 | true |
| 120 | 100 | true |
| 180 | 160 | true |
| 180 | 100 | true |
| 300 | 160 | true |
- Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Use the following table to make multiple queries with differing values for the 'start_time', 'end_time', and 'strict' in the 'Range' filter. Make sure that for all these queries, that the value
40059is present in the query results of the 'Age' column.
| start_time | end_time | strict |
|---|---|---|
| 200 | 100 | false |
| 200 | 10 | false |
- Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Set the 'start_time' to 170 and 'end_time' to 130. Run the query and verify that the Collection Object
40060is returned for a query results. - Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Set the 'start_time' to 550 and 'end_time' to 500. Run the query and verify that the Collection Object
40061is returned for a query results. - Create a Collection Object Query. Select the 'Age' field. Select the 'Range' filter. Set the 'start_time' to 200 and 'end_time' to 140. Run the query and verify that the Collection Object
40063is returned for a query results. - Test the age query with the 'Name'. Here are a defined subset of CollectionObjectIDs that should be included in the returned query results with a given age period name:
| name | co_id |
|---|---|
| Oxfordian | 40063 |
| Jurassic | 40060 |
| Jurassic | 40058 |
- Do age queries with 'Any' and explore including the follow query field paths and make sure that column in the query results have at least one row with data populated. Might be good to just set each field to 'Not Empty' one at a time, to make sure rows are returned:
- collectionobject->absoluteage
- ollectionobject->relativeage->chronostrat
- collectionobject->paleocontext->chronostrat
- collectionobject->collectionevent->paleocontext->chronostrat
- Collectionobject->collectionevent->locality->paleocontext->chronostrat
Looks great!!
Note for future testers:
UI for the list will not be changed in this pr because it is a reused component.
UI for V2
|
So @CarolineDenis @acwhite211 this was not addressed? Are we still crashing if the user builds a seemingly OK query? Does not have to do with UI
|







Fixes #5180
Checklist
and self-explanatory (or properly documented)
Testing instructions
This PR affects database migrations. See migration testing instructions (here)
https://ciscollectionsageqbmigration52722-issue-5180.test.specifysystems.org/specify/query/new/collectionobject/
Use the
ciscollections_age_qbdatabase in the test panel. Use the 'Rocks' collection.Go to the QB and select Collection Object, then verify the following queries. Note the the value returned in the 'Age' column is the CollectionObjectID, so no need to add an ID column in the query.
40057is present in the query results of the 'Age' column.40057is NOT present in the query results of the 'Age' column.40058is present in the query results of the 'Age' column.40058is NOT present in the query results of the 'Age' column.40059is present in the query results of the 'Age' column.40060is returned for a query results.40061is returned for a query results.40063is returned for a query results.