pants: Add system_user detection to pants-plugins/uses_services (+ mongo detection improvements)

[cognifloyd]

Some of the tests require the system_user.user as an actual local user on the machine running pytest. That is now configurable with the ST2TESTS_SYSTEM_USER env var added in #6242. If someone forgot to set--or did not know they could set--the ST2TESTS_SYSTEM_USER env var, they will get really odd errors that are difficult to interpret and debug. So, this PR adds a pants-plugin feature to fail with an error message about setting that env var before pytest even runs tests that require the system_user.

This improves on the pants-plugins/uses_services plugin, which was added in these PRs (the PR descriptions can be helpful in understanding the uses_services plugin):

• Add pants-plugins/uses_services to check before running tests for required services (like mongo) #5864
• Update pants-plugins/uses_services to support checking for rabbitmq #5884
• Update pants-plugins/uses_services to support checking for redis #5893
• Pants: Misc pants-plugins/uses_services improvements #5898

has_system_user.py script and the rule that runs it

has_system_user.py is the part that checks to see if the system_user is present on the system.

The has_system_user.py script gets opened and run in a pex with the similar rule logic to the mongo, redis, and rabbitmq detection rules.

Only system_user has to be passed to the has_system_user.py script to check for that user:

st2/pants-plugins/uses_services/system_user_rules.py

Lines 46 to 55 in 1413e11

	@dataclass(frozen=True)
	class UsesSystemUserRequest:
	"""One or more targets need the system_user (like stanley) using these settings.
	The system_user attributes represent the system_user.user settings from st2.conf.
	In st2 code, they come from:
	oslo_config.cfg.CONF.system_user.user
	"""
	system_user: str = "stanley"

Here is the definition of the rule that runs has_system_user.py:

st2/pants-plugins/uses_services/system_user_rules.py

Lines 92 to 94 in 1413e11

	async def has_system_user(
	request: UsesSystemUserRequest, platform: Platform
	) -> HasSystemUser:

So, when another rule Gets HasSystemUser with a UsesSystemUserRequest, pants will also run the rule that generates Platform (described in #5864), and then it will run this has_system_user rule.

The has_system_user rule either returns HasSystemUser() if the user is present, or raises ServiceMissingError if it is not (the same error that the other uses_services rules raise). By raising an error, this actually breaks a convention for pants rules. Exceptions stop everything and get shown to the user right away, and for most goals, pants wants to see some dataclass returned that has something like a succeeded boolean instead. But, we want to error as early as possible, so this breaks that convention on purpose.

wiring up the test goal so it runs has_system_user when pytest runs on a target with the uses field.

The last piece that ties this all together is a rule that makes sure the has_system_user rule runs before pytest runs (if pants is running it on a target with the uses field). Here is the definition of the has_system_user_for_pytest rule:

st2/pants-plugins/uses_services/system_user_rules.py

Lines 76 to 79 in 1413e11

	async def has_system_user_for_pytest(
	request: PytestUsesSystemUserRequest,
	test_extra_env: TestExtraEnv,
	) -> PytestPluginSetup:

This rule is fairly simple. It looks up the system_user based on the [test].extra_env_vars setting defined here:

st2/pants.toml

Lines 242 to 247 in 1413e11

	[test]
	extra_env_vars = [
	# Use this so that the test system does not require the stanley user.
	# For example: export ST2TESTS_SYSTEM_USER=${USER}
	"ST2TESTS_SYSTEM_USER",
	]

Then it just triggers running the has_system_user rule, passing in the system_user:

st2/pants-plugins/uses_services/system_user_rules.py

Lines 80 to 85 in 1413e11

	system_user = test_extra_env.env.get("ST2TESTS_SYSTEM_USER", "stanley")
	# this will raise an error if system_user is not present
	_ = await Get(HasSystemUser, UsesSystemUserRequest(system_user=system_user))
	return PytestPluginSetup()

This rule needs the PytestUsesSystemUserRequest which selects targets that have the uses field.

st2/pants-plugins/uses_services/system_user_rules.py

Lines 63 to 69 in 1413e11

	class PytestUsesSystemUserRequest(PytestPluginSetupRequest):
	@classmethod
	def is_applicable(cls, target: Target) -> bool:
	if not target.has_field(UsesServicesField):
	return False
	uses = target.get(UsesServicesField).value
	return uses is not None and "system_user" in uses

This request will be made by the pants-internal pytest rules thanks to this UnionRule, which is a way to declare that our class "implements" the PytestPluginSetupRequest:

st2/pants-plugins/uses_services/system_user_rules.py

Line 156 in 1413e11

UnionRule(PytestPluginSetupRequest, PytestUsesSystemUserRequest),

If we need to add uses support to other targets, we will need to find similar UnionRules where we can inject our rule logic. For now, I've only wired this up so our uses rules will run for pytest.

update mongo bits to use pants idiom

This PR also updates the mongo detection rules to use pants to grab the ST2TESTS_PARALLEL_SLOT env var. I learned about how to do this recently, so I included that update here.

Pants is configured to use ST2TESTS_PARALLEL_SLOT here:

st2/pants.toml

Lines 229 to 234 in 1413e11

	[pytest]
	install_from_resolve = "pytest"
	args = [
	"--no-header", # don't print pytest version for every tested file
	]
	execution_slot_var = "ST2TESTS_PARALLEL_SLOT"

So, the mongo_is_running_for_pytest rule passes that var name here:

st2/pants-plugins/uses_services/mongo_rules.py

Lines 110 to 113 in 1413e11

	_ = await Get(
	MongoIsRunning,
	UsesMongoRequest(execution_slot_var=pytest.execution_slot_var or ""),
	)

And this line tells pants to include the var in the env used when running is_mongo_running.py here:

st2/pants-plugins/uses_services/mongo_rules.py

Line 157 in 1413e11

execution_slot_variable=request.execution_slot_var,

Finally the db name calculation is moved into is_mongo_running.py here:

st2/pants-plugins/uses_services/scripts/is_mongo_running.py

Lines 52 to 56 in 1413e11

	db_name = args.get(3, "st2-test{}")
	connection_timeout_ms = args.get(4, 3000)
	slot_var = args.get(5, "ST2TESTS_PARALLEL_SLOT")
	db_name = db_name.format(os.environ.get(slot_var) or "")