Add institutional preferences

[maxpatiiuk]

Similar to collection preferences added by stats page.

The tricky part to this is the migration:

• Need to create a Django migration that gets the original defaults from remote prefs and global prefs
• Also, as part of this, should migrate away from moment and dayjs libraries and move to a more modern library (date-fns was great option last time I checked, but more research is needed) - the current way of specifying a date format is different that what's expected by other libraries so migration would have to do string replacement, rather than simply using the same value
• Add a pref for ability to add institution prefs and enforce that in the app resource editor

[realVinayak]

@maxpatiiuk @grantfitzsimmons should changes done in specify6 to the files be seen in specify7 after this? I think they 7 should maintain it's own

[grantfitzsimmons]

I'm okay with migrating existing Specify 6 preferences to 7 and breaking away. I don't see us supporting cross compatible preferences as 6 deletes any preferences it doesn't recognize

[maxpatiiuk]

100% agree with @grantfitzsimmons

Migrate all as best as we can one time on specify update, and then keep them as separate as possible

BUT, take this migration as a time to:

• rethink and standardize the naming (the naming used in remote/global prefs right now is totally inconsistent) - probably would make most sense to reuse the naming convention used in user preferences, and to follow as similar structure to that as possible
• improve existing prefs during the transition (improve names, make values clearer, etc - whatever you see can be improved, now is the best time)
• think through all the other prefs we could introduce so that we can introduce a lot at once (I did that with user preferences) - to discover the edge cases early on and to figure out which categories/subcategories you should have (it's easier to see what categories you should have when you have 30 prefs rather than 3) so that you don't have to add additional migrations in the future to change that
• DATES - The date format is specified in the prefs right now. This is the moment to migrate from moment.js to a newer date library. luxon and date-fns seem to be the two most popular at the moment - evaluate these to see what works best for us. The syntax for specifying the date format would change. Fortunately, there is only a limited number of date formats we currently support (probably the same as sp6) -

  specify7/specifyweb/stored_queries/format.py

  Lines 273 to 352 in 6bc3785

  	MYSQL_TO_YEAR = {
  	"%m %d %y": "%y",
  	"%m %d %Y": "%Y",
  	"%m-%d-%y": "%y",
  	"%m-%d-%Y": "%Y",
  	"%m.%d.%y": "%y",
  	"%m.%d.%Y": "%Y",
  	"%m/%d/%y": "%y",
  	"%m/%d/%Y": "%Y",
  	"%d %m %y": "%y",
  	"%d %m %Y": "%Y",
  	"%d %b %Y": "%Y",
  	"%d-%m-%y": "%y",
  	"%d-%m-%Y": "%Y",
  	"%d-%b-%Y": "%Y",
  	"%d.%m.%y": "%y",
  	"%d.%m.%Y": "%Y",
  	"%d.%b.%Y": "%Y",
  	"%d/%m/%y": "%y",
  	"%d/%m/%Y": "%Y",
  	"%d/%b/%Y": "%Y",
  	"%Y %m %d": "%Y",
  	"%Y-%m-%d": "%Y",
  	"%Y.%m.%d": "%Y",
  	"%Y/%m/%d": "%Y",
  	}
  	MYSQL_TO_MONTH = {
  	"%m %d %y": "%m %y",
  	"%m %d %Y": "%m %Y",
  	"%m-%d-%y": "%m-%y",
  	"%m-%d-%Y": "%m-%Y",
  	"%m.%d.%y": "%m.%y",
  	"%m.%d.%Y": "%m.%Y",
  	"%m/%d/%y": "%m/%y",
  	"%m/%d/%Y": "%m/%Y",
  	"%d %m %y": "%m %y",
  	"%d %m %Y": "%m %Y",
  	"%d %b %Y": "%b %Y",
  	"%d-%m-%y": "%m-%y",
  	"%d-%m-%Y": "%m-%Y",
  	"%d-%b-%Y": "%b-%Y",
  	"%d.%m.%y": "%m.%y",
  	"%d.%m.%Y": "%m.%Y",
  	"%d.%b.%Y": "%b.%Y",
  	"%d/%m/%y": "%m/%y",
  	"%d/%m/%Y": "%m/%Y",
  	"%d/%b/%Y": "%b/%Y",
  	"%Y %m %d": "%Y %m",
  	"%Y-%m-%d": "%Y-%m",
  	"%Y.%m.%d": "%Y.%m",
  	"%Y/%m/%d": "%Y/%m",
  	}
  	LDLM_TO_MYSQL = {
  	"MM dd yy": "%m %d %y",
  	"MM dd yyyy": "%m %d %Y",
  	"MM-dd-yy": "%m-%d-%y",
  	"MM-dd-yyyy": "%m-%d-%Y",
  	"MM.dd.yy": "%m.%d.%y",
  	"MM.dd.yyyy": "%m.%d.%Y",
  	"MM/dd/yy": "%m/%d/%y",
  	"MM/dd/yyyy": "%m/%d/%Y",
  	"dd MM yy": "%d %m %y",
  	"dd MM yyyy": "%d %m %Y",
  	"dd MMM yyyy": "%d %b %Y",
  	"dd-MM-yy": "%d-%m-%y",
  	"dd-MM-yyyy": "%d-%m-%Y",
  	"dd-MMM-yyyy": "%d-%b-%Y",
  	"dd.MM.yy": "%d.%m.%y",
  	"dd.MM.yyyy": "%d.%m.%Y",
  	"dd.MMM.yyyy": "%d.%b.%Y",
  	"dd/MM/yy": "%d/%m/%y",
  	"dd/MM/yyyy": "%d/%m/%Y",
  	"dd/MMM/yyy": "%d/%b/%Y",
  	"yyyy MM dd": "%Y %m %d",
  	"yyyy-MM-dd": "%Y-%m-%d",
  	"yyyy.MM.dd": "%Y.%m.%d",
  	"yyyy/MM/dd": "%Y/%m/%d",
  	}

  - that makes migration easier (just create a dictionary of mappings from old date to new equivalent date syntax in the new library)
  • Also, in the UI also do a better job of pointing out that the date pickers will use the date format based on current system date format, but that will still show correct date in the database, and query results and reports and workbench. If they do want to see equivalent date formats, they should disable the "Use Accessible Date Picker" setting. I.e put the "Use Accessible Date Picker" pref right below the date format, and add a handy description/discourse page link.

side note

we are using day.js right now instead of moment.js, but day.js was built to be backwards compatible with moment.js to serve as a drop-in replacement. But, I have been quite unhappy with day.js so far:

• First, they advertise themself as being much leaner than moment.js - but they do that by packaging some features into a separate bundle - we happen to need those features (date parsing and date building from a date format string) - as a result, our bundle size increased after migrating from moment.js to day.js.
• Secondly, and much more importantly, day.js has bugs - I had to add clumsy workarounds for their bugs -

  specify7/specifyweb/frontend/js_src/lib/components/FormPlugins/PartialDateUi.tsx

  Lines 70 to 112 in 6bc3785

  	// TODO: [REFACTOR] migrate from day.js to date-fns
  	/**
  	* Several bugs have been discovered in day.js in the process of testing
  	* Specify 7. The bugs are present in the latest stable version as of this
  	* writing (1.11.4).
  	* Day.js 2.0 is currently in beta. Need to wait for 2.0 to get released or
  	* migrate to date-fns.
  	*/
  	function fixDayJsBugs(
  	precision: PartialDatePrecision,
  	value: string
  	): ReturnType<typeof dayjs> | undefined {
  	if (precision === 'month-year') return unsafeParseMonthYear(value);
  	else if (precision === 'full') return unsafeParseFullDate(value);
  	else return undefined;
  	}
  	/**
  	* An ugly workaround for a bug in day.js where any date in the MM/YYYY format is
  	* parsed as an invalid date.
  	*/
  	function unsafeParseMonthYear(
  	value: string
  	): ReturnType<typeof dayjs> | undefined {
  	const parsed = /(\d{2})\D(\d{4})/.exec(value)?.slice(1);
  	if (parsed === undefined) return undefined;
  	const [month, year] = parsed.map(f.unary(Number.parseInt));
  	return dayjs(new Date(year, month - 1));
  	}
  	/**
  	* An ugly workaround for a bug in day.js where any date in the DD/MM/YYY format
  	* is parsed as an invalid date.
  	*/
  	function unsafeParseFullDate(
  	value: string
  	): ReturnType<typeof dayjs> | undefined {
  	if (fullDateFormat().toUpperCase() !== 'DD/MM/YYYY') return;
  	const parsed = /(\d{2})\D(\d{2})\D(\d{4})/.exec(value)?.slice(1);
  	if (parsed === undefined) return undefined;
  	const [day, month, year] = parsed.map(f.unary(Number.parseInt));
  	return dayjs(new Date(year, month - 1, day));
  	}

  . At the time, even the latest version of the library had these bugs. The biggest one is the fact that the European date format is not supported!!! I could not belive it when I saw it, but that seems to be the case (see the ticket for that - [Sp.77] - when using European date format, aggregated dates are broken for any day over the 12th of the month #1908)

cc @melton-jason @CarolineDenis

[melton-jason]

The reasoning (and that decision was originally made 10 or 20 years ago in sp6 and carried over into sp7) is that people would assign their own GUIDs OR they would get data from some source and use that source's GUIDs.

i.e specify is not a Taxa management system, so we are not in the business of assigning GUIDs for Taxa (same for Geography) we are in the business of Collection Object management, so we are assigning GUIDs for that

though that is a bit idealistic as we don't provide easy enough ways to get taxa authority data into specify, and we definitely don't have an easy way to update the taxa with updates in the taxa authority or to even keep them in sync.

so I suppose this decision was part of us being a bit more opinionated/trying to enforce the practices we belive are good - but in this case we didn't go all the way to actually make those good practices easy on people.

so yeah, I would be ok with adding a global pref for that (AND WE NEED INSTITUTIONAL PREFERENCES - pretty urgently - so many features are held back because of lack of that)

From #4243 (comment)