Pack dependency management #4769

jinpingh · 2019-08-19T19:24:25Z

Fixes https://github.com/StackStorm/discussions/issues/354
Resolves #3336
Closes #3698

packs.install workflow changed from achtion-chain to orquesta for workflow complexity.
Download all packs and dependency packs before install pack requirements and register pack.
Register packs in reverse order, it means the last dependency pack will be installed first.
Workflow fails on any time if conflict pack is recognized, downloaded packs will be removed.
New option --skip (default value is False) for skipping dependencies check: st2 install pack --skip
Only case that will check conflict is that if pack was installed and dependency pack has specified pack version.
Nested level of dependencies to prevent infinite or long download loops is 3.
Support StackStorm Exchange packs, the private repository ULR and local file system for conflict pack checking.

Fixes https://github.com/StackStorm/discussions/issues/354 1. `packs.install` workflow changed from `achtion-chain` to `orquesta` for workflow complexity. 2. Download all packs and dependency packs, check for conflict packs before `install pack requirements` and `register pack`. 3. Workflow fails on any time if conflict pack is recognized, downloaded packs will be removed. 4. Register packs in reverse order, it means the last dependency pack will be installed first. 5. New option `--skip` (default value is False) for skipping dependencies check: `st2 install pack --skip` 6. Only case that will check conflict is that if pack was installed and dependency pack has specified pack version. 7. Nested level of dependencies to prevent infinite or long download loops is 3. 8. Support StackStorm Exchange packs, the private repo ULR and local file system for conflict pack checking.

contrib/packs/actions/workflows/install.yaml

Two new test files for testing get_pack_dependencies.py and virtualenv_setup_prerun.py New test cases for download dependencies packs. New test cases for `skip` install dependency flag. Modified install workflow for more simplicity.

m4dcoder

Got some questions.

CHANGELOG.rst

contrib/packs/actions/delete.yaml

contrib/packs/actions/pack_mgmt/get_pack_dependencies.py

contrib/packs/actions/workflows/install.yaml

When install/remove list of packs, if the pack `ref` is not same as pack `name`, the pack information is not printed. See file: https://github.com/StackStorm/st2/blob/master/st2tests/st2tests/fixtures/packs/dummy_pack_19/pack.yaml For example: `st2 pack install files:///dummy_pack_19 csv` Actually result: +---------------+-------------------------------+---------+--------------+ | name | description | version | author | +---------------+-------------------------------+---------+--------------+ | email | E-Mail Actions/Sensors for | 1.1.3 | James Fryman | | | StackStorm | | | +---------------+-------------------------------+---------+--------------+ Expected result: +-------------------+---------------+-------------------------------+---------+--------------+ | ref | name | description | version | author | +-------------------+---------------+-------------------------------+---------+--------------+ | dummy_pack_19_ref | dummy_pack_19 | dummy pack | 0.1.0 | st2-dev | | email | email | E-Mail Actions/Sensors for | 1.1.3 | James Fryman | | | | StackStorm | | | +-------------------+---------------+-------------------------------+---------+--------------+ Root Cause: Pack install/remove is not query `ref` as print out parameter, and only pack `name` is checked to find out pack. When pack `name` and `ref` is not the same, pack information is not printed.

m4dcoder · 2019-08-22T22:39:16Z

Also, how are you doing e2e tests? The unit tests here are not sufficient.

jinpingh · 2019-08-22T22:53:57Z

Also, how are you doing e2e tests? The unit tests here are not sufficient.

I am working on it. Have't found good way to do it. Suggestions are welcomed.

m4dcoder · 2019-08-22T23:12:19Z

@jinpingh You can try to do that as integration tests like make orquesta-itests and run it during CI. You can set it up to run it under st2_e2e_tests using bats. In any case, you will need to setup different packs to test various scenarios. I would like to see a list of test cases that will be covered.

jinpingh · 2019-08-22T23:52:03Z

@jinpingh You can try to do that as integration tests like make orquesta-itests and run it during CI. You can set it up to run it under st2_e2e_tests using bats. In any case, you will need to setup different packs to test various scenarios. I would like to see a list of test cases that will be covered.

@m4dcoder
Mostly likely will modify https://github.com/StackStorm-Exchange/stackstorm-test/pack.yaml to add dependency list and run it under st2_e2e_tests using bats.
List of test cases that can think about for now:

Test one level of dependencies.
Test st2 pack install with --skip flag.
Test conflict case to make sure to fail workflow, the exiting packs are untouched, and newly downloaded packs are removed.

m4dcoder · 2019-08-23T00:09:55Z

@jinpingh You have to spend some more thoughts on the list of test cases. At the minimum, you did not include what I asked for above with handling conflict against a pack that is already installed on the system. Also, if we support x level of nested dependencies then testing 1 level is not sufficient.

jinpingh · 2019-08-23T01:11:27Z

@jinpingh You have to spend some more thoughts on the list of test cases. At the minimum, you did not include what I asked for above with handling conflict against a pack that is already installed on the system. Also, if we support x level of nested dependencies then testing 1 level is not sufficient.

From my above list 3. Test conflict case to make sure to fail workflow, **the exiting packs are untouched**, and newly downloaded packs are removed. is the case that will test conflict against a pack that is already installed on the system.

Did manually tested 4 level (supported level is 3) nested dependencies and last level of dependencies are not installed. I can go with same with e2e test by modifying dummy_packs or add new StackStorm exchange test packs. At least 4 packs are needed to do this test.

I am still working on e2e test and will see what can be tested from bats.

Thanks!

m4dcoder · 2019-08-23T01:15:51Z

Please separate 3 into different test cases and be precise. There could be different types of conflicts.

contrib/packs/actions/install.meta.yaml

contrib/packs/actions/get_pack_dependencies.yaml

arm4b

Looks promising!
I'll start running this branch and trying new pack install with dependencies next week to see how it behaves and provide more detailed feedback.

Talking about code. There are a lot of questionable assumptions made and code duplication. Instead of duplicating logic and implementation between pack components we should be reusing common functions and make sure logic is shared between all the pack actions. This will ensure we don't end in situation when pack download accepts one format while pack dependency follows another.

However it's a great starting prototype for a quick demo phase so everyone can try and provide early feedback 👍

contrib/packs/actions/pack_mgmt/get_pack_dependencies.py

…gement

blag · 2019-09-20T00:29:49Z

@armab @m4dcoder This and StackStorm/st2tests#179 are ready for review.

Manually tested with packages build by CircleCI and StackStorm/st2tests#179 on Ubuntu 16.04.

arm4b · 2019-09-20T12:35:29Z

@blag Awesome! I'll battle-test the feature locally.

arm4b · 2019-09-20T12:41:48Z

First thing, looks like in order to use new st2 pack install, one needs to re-register content (st2ctl reload).
Please add that into st2docs upgrade notes for upcoming 3.2.0 release.

For the "github" pack, the following content will be registered:

rules     |  1
sensors   |  1
triggers  |  0
actions   |  23
aliases   |  7

Installation may take a while for packs with many items.
id: 5d84c8446cb8de08ab1574b8
action.ref: packs.install
parameters: 
  packs:
  - github
  python3: false
status: failed
error: Additional properties are not allowed ('output', 'tasks', 'description', 'version', 'input' were unexpected)

Failed validating 'additionalProperties' in schema:
    {'additionalProperties': False,
     'description': 'A chain of sequentially executed actions.',
     'properties': {'chain': {'description': 'The chain.',
                              'items': [{'additionalProperties': False,
                                         'description': 'Node of an ActionChain.',
                                         'properties': {'name': {'description': 'The name of this node.',
                                                                 'required': True,
                                                                 'type': 'string'},
                                                        'notify': {'additionalProperties': False,
                                                                   'description': 'Notification settings for action.',
                                                                   'properties': {'on-complete': {'additionalProperties': False,
                                                                                                  'properties': {'channels': {'description': 'Channels to post notifications to.',
                                                                                                                              'type': 'array'},
                                                                                                                 'data': {'description': 'Data to be sent as part of notification',
                                                                                                                          'type': 'object'},
                                                                                                                 'message': {'description': 'Message to use for notification',
                                                                                                                             'type': 'string'},
                                                                                                                 'routes': {'description': 'Channels to post notifications to.',
                                                                                                                            'type': 'array'}},
                                                                                                  'type': 'object'},
                                                                                  'on-failure': {'additionalProperties': False,
                                                                                                 'properties': {'channels': {'description': 'Channels to post notifications to.',
                                                                                                                             'type': 'array'},
                                                                                                                'data': {'description': 'Data to be sent as part of notification',
                                                                                                                         'type': 'object'},
                                                                                                                'message': {'description': 'Message to use for notification',
                                                                                                                            'type': 'string'},
                                                                                                                'routes': {'description': 'Channels to post notifications to.',
                                                                                                                           'type': 'array'}},
                                                                                                 'type': 'object'},
                                                                                  'on-success': {'additionalProperties': False,
                                                                                                 'properties': {'channels': {'description': 'Channels to post notifications to.',
                                                                                                                             'type': 'array'},
                                                                                                                'data': {'description': 'Data to be sent as part of notification',
                                                                                                                         'type': 'object'},
                                                                                                                'message': {'description': 'Message to use for notification',
                                                                                                                            'type': 'string'},
                                                                                                                'routes': {'description': 'Channels to post notifications to.',
                                                                                                                           'type': 'array'}},
                                                                                                 'type': 'object'}},
                                                                   'type': 'object'},
                                                        'on-failure': {'default': '',
                                                                       'description': 'Name of the node to invoke on failure of action executed for this node.',
                                                                       'type': 'string'},
                                                        'on-success': {'default': '',
                                                                       'description': 'Name of the node to invoke on successful completion of action executed for this node.',
                                                                       'type': 'string'},
                                                        'parameters': {'default': {},
                                                                       'description': 'Parameter for the execution.',
                                                                       'type': 'object'},
                                                        'params': {'default': {},
                                                                   'description': 'Parameter for the execution (old name, here for backward compatibility reasons).',
                                                                   'type': 'object'},
                                                        'publish': {'description': 'The variables to publish from the result. Should be of the form name.foo. o1: {{node_name.foo}} will result in creation of a variable o1 which is now available for reference through remainder of the chain as a global variable.',
                                                                    'patternProperties': {'^\w+$': {}},
                                                                    'type': 'object'},
                                                        'ref': {'description': 'Ref of the action to be executed.',
                                                                'required': True,
                                                                'type': 'string'}},
                                         'title': 'Node',
                                         'type': 'object'}],
                              'required': True,
                              'type': 'array'},
                    'default': {'description': 'name of the action to be executed.',
                                'type': 'string'},
                    'vars': {'description': '',
                             'patternProperties': {'^\w+$': {}},
                             'type': 'object'}},
     'title': 'ActionChain',
     'type': 'object'}

On instance:
    {'description': 'A orquesta workflow to install packs.',
     'input': ['packs',
               'register',
               'env',
               'force',
               'python3',
               'skip_dependencies'],
     'output': [{'packs_list': '<% ctx().packs_list %>'}],
     'tasks': {'check_dependency_and_conflict_list': {'action': 'core.noop',
                                                      'next': [{'do': 'stop_installation_and_cleanup_because_conflict',
                                                                'when': '<% ctx().conflict_list %>'},
                                                               {'do': 'download_pack',
                                                                'when': '<% not ctx().conflict_list and ctx().dependency_list %>'},
                                                               {'do': 'install_pack_requirements',
                                                                'when': '<% not ctx().conflict_list and not ctx().dependency_list %>'}]},
               'download_pack': {'action': 'packs.download',
                                 'input': {'dependency_list': '<% ctx().dependency_list %>',
                                           'force': '<% ctx().force %>',
                                           'packs': '<% ctx().packs %>',
                                           'python3': '<% ctx().python3 %>'},
                                 'next': [{'do': 'make_a_prerun',
                                           'when': '<% succeeded() %>'}]},
               'get_pack_dependencies': {'action': 'packs.get_pack_dependencies',
                                         'input': {'nested': '<% ctx().nested%>',
                                                   'packs_status': '<% task(download_pack).result.result %>'},
                                         'next': [{'do': 'check_dependency_and_conflict_list',
                                                   'publish': [{'dependency_list': '<% result().result.dependency_list %>'},
                                                               {'conflict_list': '<% result().result.conflict_list %>'},
                                                               {'nested': '<% result().result.nested %>'}],
                                                   'when': '<% succeeded() %>'}]},
               'init_task': {'action': 'core.noop',
                             'next': [{'do': 'download_pack'}]},
               'install_pack_requirements': {'action': 'packs.setup_virtualenv',
                                             'input': {'env': '<% ctx().env %>',
                                                       'packs': '<% ctx().packs_list %>',
                                                       'python3': '<% ctx().python3 %>'},
                                             'next': [{'do': 'register_pack',
                                                       'when': '<% succeeded() %>'}]},
               'make_a_prerun': {'action': 'packs.virtualenv_prerun',
                                 'input': {'packs_list': '<% ctx().packs_list %>',
                                           'packs_status': '<% task(download_pack).result.result %>'},
                                 'next': [{'do': 'install_pack_requirements',
                                           'publish': [{'packs_list': '<% task(make_a_prerun).result.result %>'}],
                                           'when': '<% succeeded() and (ctx().skip_dependencies or ctx().nested = 0) %>'},
                                          {'do': 'get_pack_dependencies',
                                           'publish': [{'packs_list': '<% task(make_a_prerun).result.result %>'}],
                                           'when': '<% succeeded() and (ctx().nested > 0 and not ctx().skip_dependencies) %>'}]},
               'register_pack': {'action': 'packs.load',
                                 'input': {'packs': '<% ctx().packs_list %>',
                                           'register': '<% ctx().register %>'}},
               'stop_installation_and_cleanup_because_conflict': {'action': 'packs.delete',
                                                                  'input': {'delete_env': False,
                                                                            'packs': '<% ctx().packs_list %>'},
                                                                  'next': [{'do': 'fail'}]}},
     'vars': [{'packs_list': None},
              {'dependency_list': None},
              {'conflict_list': None},
              {'nested': 10}],
     'version': 1.0}
traceback:   File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2actions/container/base.py", line 113, in _do_run
    runner.pre_run()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/action_chain_runner/action_chain_runner.py", line 282, in pre_run
    raise runner_exc.ActionRunnerPreRunError(message)

start_timestamp: Fri, 20 Sep 2019 12:38:28 UTC
end_timestamp: Fri, 20 Sep 2019 12:38:29 UTC
result: See error and traceback.

arm4b

packs metadata https://github.com/StackStorm/st2/blob/fb2932166408d2572dae9ee57c8e64c212b2d1a3/contrib/packs/pack.yaml needs a major version increment per new functionality/changes made in this PR.

arm4b

Second run of st2 pack install https://github.com/StackStorm/stackstorm-ms fails, after first successful run. Looks like the workflow mistakenly assumes dependency conflict for the 2nd run.

This will result in broken re-installations of the same packs for users. Needs fixing.

arm4b

Failure in processing dependencies should output helpful error message to the user, where was the problem and how to act to resolve the issue:

$ st2 pack install https://github.com/StackStorm/stackstorm-ms

	[ succeeded ] init_task
	[ succeeded ] download_pack
	[ succeeded ] make_a_prerun
	[ succeeded ] get_pack_dependencies
	[ succeeded ] check_dependency_and_conflict_list
	[ succeeded ] stop_installation_and_cleanup_because_conflict

id: 5d84cec86cb8de08ab157533
action.ref: packs.install
parameters: 
  packs:
  - https://github.com/StackStorm/stackstorm-ms
  python3: false
status: failed
start_timestamp: Fri, 20 Sep 2019 13:06:16 UTC
end_timestamp: Fri, 20 Sep 2019 13:06:21 UTC
result: 
  errors:
  - message: Execution failed. See result for details.
    task_id: fail
    type: error
  output:
    packs_list:
    - microsoft_test
+--------------------------+------------------------+-----------------------------------+-----------------------------+-------------------------------+
| id                       | status                 | task                              | action                      | start_timestamp               |
+--------------------------+------------------------+-----------------------------------+-----------------------------+-------------------------------+
| 5d84cec86cb8de033487d866 | succeeded (0s elapsed) | init_task                         | core.noop                   | Fri, 20 Sep 2019 13:06:16 UTC |
| 5d84cec96cb8de033487d869 | succeeded (1s elapsed) | download_pack                     | packs.download              | Fri, 20 Sep 2019 13:06:17 UTC |
| 5d84cecb6cb8de033487d86c | succeeded (0s elapsed) | make_a_prerun                     | packs.virtualenv_prerun     | Fri, 20 Sep 2019 13:06:19 UTC |
| 5d84cecb6cb8de033487d86f | succeeded (1s elapsed) | get_pack_dependencies             | packs.get_pack_dependencies | Fri, 20 Sep 2019 13:06:19 UTC |
| 5d84cecc6cb8de033487d872 | succeeded (0s elapsed) | check_dependency_and_conflict_lis | core.noop                   | Fri, 20 Sep 2019 13:06:20 UTC |
|                          |                        | t                                 |                             |                               |
| 5d84cecd6cb8de033487d875 | succeeded (0s elapsed) | stop_installation_and_cleanup_bec | packs.delete                | Fri, 20 Sep 2019 13:06:21 UTC |
|                          |                        | ause_conflict                     |                             |                               |
+--------------------------+------------------------+-----------------------------------+-----------------------------+-------------------------------+

As a user, I have no idea what's wrong behind the:

 - message: Execution failed. See result for details.

arm4b

~~--skip-dependencies param makes no difference and tries to check and install pack dependencies anyway~~

Update: Confirming --skip-dependencies works well 👍 as API restart was required per code changes.

arm4b

Besides of the above issues, looks like workflow is running and first install with dependencies is happening, so we're close.

Please also prepare the respective st2docs documentation change for this new and highly requested functionality.

arm4b · 2019-09-24T20:29:37Z

CHANGELOG.rst

  ``url_hosts_blacklist`` and ``url_hosts_whitelist`` runner attribute. (new feature)
  #4757
 * Add ``user`` parameter to ``re_run`` method of st2client. #4785
+* Install pack dependencies automatically. #4769


Changelog message could be improved for this new functionality.

arm4b · 2019-09-24T20:33:07Z

contrib/packs/actions/pack_mgmt/get_pack_dependencies.py

+
+                # Try one more time to get existing pack version by name if 'stackstorm-' is in
+                # pack name
+                if not existing_pack_version and 'stackstorm-' in pack_name.lower():


Bad assumption thinking that stackstorm- is in pack name and take actions based on that.

OK about this for now, could be improved in future.

arm4b

LGTM. Thanks @jinpingh for initial effort and @blag for additional polishing and changes made 👍

I would expect to get a review from @m4dcoder as well before merging this.

m4dcoder

LGTM. Thank you for your diligence in completing and closing this feature.

jinpingh requested review from arm4b and m4dcoder August 19, 2019 19:24

jinpingh added feature packs WIP labels Aug 19, 2019

jinpingh removed request for arm4b and m4dcoder August 19, 2019 19:31

jinpingh force-pushed the issue-354/pack_management branch 2 times, most recently from 49e1c8f to e1afd79 Compare August 20, 2019 20:37

jinpingh requested review from arm4b and m4dcoder August 20, 2019 21:18

jinpingh force-pushed the issue-354/pack_management branch from e1afd79 to 139709c Compare August 21, 2019 00:52

jinpingh force-pushed the issue-354/pack_management branch from 139709c to d47e267 Compare August 22, 2019 01:26

jinpingh commented Aug 22, 2019

View reviewed changes

contrib/packs/actions/workflows/install.yaml Outdated Show resolved Hide resolved

Add unit test cases for Pack dependency management

754f632

Two new test files for testing get_pack_dependencies.py and virtualenv_setup_prerun.py New test cases for download dependencies packs. New test cases for `skip` install dependency flag. Modified install workflow for more simplicity.

m4dcoder reviewed Aug 22, 2019

View reviewed changes

jinpingh force-pushed the issue-354/pack_management branch from 7216010 to 6bb6512 Compare August 22, 2019 21:33

arm4b reviewed Aug 23, 2019

View reviewed changes

contrib/packs/actions/install.meta.yaml Show resolved Hide resolved

arm4b reviewed Aug 23, 2019

View reviewed changes

contrib/packs/actions/get_pack_dependencies.yaml Show resolved Hide resolved

arm4b suggested changes Aug 23, 2019

View reviewed changes

arm4b added this to the 3.2.0 milestone Aug 23, 2019

blag added 2 commits September 19, 2019 02:19

Merge remote-tracking branch 'origin/master' into issue-354/pack_mana…

9b13162

…gement

Fix overzealous simplification

fb29321

arm4b self-requested a review September 20, 2019 12:04

arm4b suggested changes Sep 20, 2019

View reviewed changes

arm4b reviewed Sep 20, 2019

View reviewed changes

blag added 4 commits September 20, 2019 14:16

Fix version comparisons

b91a05a

Bump packs pack version

1c3b243

Handle pack versions that are None

b8b480f

Add dependencies list example to hello_st2/pack.yaml

105c588

blag force-pushed the issue-354/pack_management branch from 24e35cd to 105c588 Compare September 20, 2019 23:24

blag mentioned this pull request Sep 20, 2019

Add section on pack dependencies StackStorm/st2docs#934

Merged

Betterize error messages and add a core.error action

32ad61f

arm4b reviewed Sep 24, 2019

View reviewed changes

arm4b approved these changes Sep 24, 2019

View reviewed changes

arm4b requested a review from m4dcoder September 24, 2019 20:34

m4dcoder changed the title ~~WIP: Pack dependency management~~ Pack dependency management Sep 27, 2019

m4dcoder removed the WIP label Sep 27, 2019

pull-request-size bot added the size/XL PR that changes 500-999 lines. Consider splitting work into several ones that easier to review. label Sep 27, 2019

m4dcoder approved these changes Sep 27, 2019

View reviewed changes

blag merged commit 17418f6 into master Sep 27, 2019

blag deleted the issue-354/pack_management branch September 27, 2019 20:04

danielburrell mentioned this pull request Feb 13, 2020

st2 Pack Dependencies Not Resolved via st2-pack-install tool #4867

Open

arm4b mentioned this pull request Apr 17, 2020

StackStorm v3.2dev pre-release Testing StackStorm/community#6

Closed

Uh oh!

Pack dependency management #4769

Pack dependency management #4769

Uh oh!

Conversation

jinpingh commented Aug 19, 2019 • edited by arm4b Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

m4dcoder left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

m4dcoder commented Aug 22, 2019

Uh oh!

jinpingh commented Aug 22, 2019

Uh oh!

m4dcoder commented Aug 22, 2019

Uh oh!

jinpingh commented Aug 22, 2019

Uh oh!

m4dcoder commented Aug 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jinpingh commented Aug 23, 2019

Uh oh!

m4dcoder commented Aug 23, 2019

Uh oh!

Uh oh!

Uh oh!

arm4b left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

blag commented Sep 20, 2019

Uh oh!

arm4b commented Sep 20, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arm4b commented Sep 20, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arm4b left a comment

Choose a reason for hiding this comment

Uh oh!

arm4b left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arm4b left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arm4b left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arm4b left a comment

Choose a reason for hiding this comment

Uh oh!

arm4b Sep 24, 2019

Choose a reason for hiding this comment

Uh oh!

arm4b Sep 24, 2019

Choose a reason for hiding this comment

Uh oh!

arm4b left a comment

Choose a reason for hiding this comment

Uh oh!

m4dcoder left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

jinpingh commented Aug 19, 2019 •

edited by arm4b

Loading

m4dcoder commented Aug 23, 2019 •

edited

Loading

arm4b commented Sep 20, 2019 •

edited

Loading

arm4b commented Sep 20, 2019 •

edited

Loading

arm4b left a comment •

edited

Loading

arm4b left a comment •

edited

Loading

arm4b left a comment •

edited

Loading