[Subnet Mgr] When create and delete subnet concurrently, VPC will still see deleted subnet ID

[yanmo96]

• When run rally test create_and_delete_subnets, create and delete 50 times with 5 concurrent. During final clean up will give "subnets are not empty exception".
• Below are the logs

2021-07-08 16:41:02.444 2160 RALLYDEBUG rally_openstack.task.cleanup.manager [-] Deleting neutron.network object c_rally_423936b7_KF12a1rx (e79c49e0-3cf1-4ae3-a4ee-db41ecff01eb)
2021-07-08 16:41:02.995 2160 DEBUG neutronclient.v2_0.client [-] Error message: {"timestamp":"2021-07-08T23:41:02.979+0000","status":500,"error":"Internal Server Error","message":"subnets are not empty exception","path":"/project/4a2b28454d124295a6781be312360adc/vpcs/e79c49e0-3cf1-4ae3-a4ee-db41ecff01eb"} _handle_fault_response /root/rally_venv/lib/python3.6/site-packages/neutronclient/v2_0/client.py:262
2021-07-08 16:41:03.052 2160 DEBUG neutronclient.v2_0.client [-] Error message: {"timestamp":"2021-07-08T23:41:03.032+0000","status":500,"error":"Internal Server Error","message":"subnets are not empty exception","path":"/project/4a2b28454d124295a6781be312360adc/vpcs/e79c49e0-3cf1-4ae3-a4ee-db41ecff01eb"} _handle_fault_response /root/rally_venv/lib/python3.6/site-packages/neutronclient/v2_0/client.py:262
2021-07-08 16:41:03.065 2160 DEBUG neutronclient.v2_0.client [-] Error message: {"timestamp":"2021-07-08T23:41:03.063+0000","status":500,"error":"Internal Server Error","message":"subnets are not empty exception","path":"/project/4a2b28454d124295a6781be312360adc/vpcs/e79c49e0-3cf1-4ae3-a4ee-db41ecff01eb"} _handle_fault_response /root/rally_venv/lib/python3.6/site-packages/neutronclient/v2_0/client.py:262
2021-07-08 16:41:03.066 2160 ERROR rally_openstack.task.cleanup.manager [-] Resource deletion failed, max retries exceeded for neutron.network: e79c49e0-3cf1-4ae3-a4ee-db41ecff01eb.: neutronclient.common.exceptions.InternalServerError: subnets are not empty exception
2021-07-08 16:41:03.066 2160 ERROR rally_openstack.task.cleanup.manager Traceback (most recent call last):
2021-07-08 16:41:03.066 2160 ERROR rally_openstack.task.cleanup.manager   File "/root/rally_venv/lib/python3.6/site-packages/rally_openstack/task/cleanup/manager.py", line 83, in _delete_single_resource
2021-07-08 16:41:03.066 2160 ERROR rally_openstack.task.cleanup.manager     rutils.retry(resource._max_attempts, resource.delete)
2021-07-08 16:41:03.066 2160 ERROR rally_openstack.task.cleanup.manager   File "/root/rally_venv/lib/python3.6/site-packages/rally/common/utils.py", line 211, in retry
2021-07-08 16:41:03.066 2160 ERROR rally_openstack.task.cleanup.manager     return func(*args, **kwargs)
2021-07-08 16:41:03.066 2160 ERROR rally_openstack.task.cleanup.manager   File "/root/rally_venv/lib/python3.6/site-packages/rally_openstack/task/cleanup/resources.py", line 226, in delete
2021-07-08 16:41:03.066 2160 ERROR rally_openstack.task.cleanup.manager     delete_method(self.id())
2021-07-08 16:41:03.066 2160 ERROR rally_openstack.task.cleanup.manager   File "/root/rally_venv/lib/python3.6/site-packages/rally/task/service.py", line 114, in wrapper
2021-07-08 16:41:03.066 2160 ERROR rally_openstack.task.cleanup.manager     return func(instance, *args, **kwargs)
2021-07-08 16:41:03.066 2160 ERROR rally_openstack.task.cleanup.manager   File "/root/rally_venv/lib/python3.6/site-packages/rally/task/atomic.py", line 91, in func_atomic_actions
2021-07-08 16:41:03.066 2160 ERROR rally_openstack.task.cleanup.manager     f = func(self, *args, **kwargs)
2021-07-08 16:41:03.066 2160 ERROR rally_openstack.task.cleanup.manager   File "/root/rally_venv/lib/python3.6/site-packages/rally_openstack/common/services/network/neutron.py", line 383, in delete_network
2021-07-08 16:41:03.066 2160 ERROR rally_openstack.task.cleanup.manager     self.client.delete_network(network_id)
2021-07-08 16:41:03.066 2160 ERROR rally_openstack.task.cleanup.manager   File "/root/rally_venv/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 866, in delete_network
2021-07-08 16:41:03.066 2160 ERROR rally_openstack.task.cleanup.manager     return self.delete(self.network_path % (network))
2021-07-08 16:41:03.066 2160 ERROR rally_openstack.task.cleanup.manager   File "/root/rally_venv/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 353, in delete
2021-07-08 16:41:03.066 2160 ERROR rally_openstack.task.cleanup.manager     headers=headers, params=params)
2021-07-08 16:41:03.066 2160 ERROR rally_openstack.task.cleanup.manager   File "/root/rally_venv/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 334, in retry_request
2021-07-08 16:41:03.066 2160 ERROR rally_openstack.task.cleanup.manager     headers=headers, params=params)
2021-07-08 16:41:03.066 2160 ERROR rally_openstack.task.cleanup.manager   File "/root/rally_venv/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 297, in do_request
2021-07-08 16:41:03.066 2160 ERROR rally_openstack.task.cleanup.manager     self._handle_fault_response(status_code, replybody, resp)
2021-07-08 16:41:03.066 2160 ERROR rally_openstack.task.cleanup.manager   File "/root/rally_venv/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 272, in _handle_fault_response
2021-07-08 16:41:03.066 2160 ERROR rally_openstack.task.cleanup.manager     exception_handler_v20(status_code, error_body)
2021-07-08 16:41:03.066 2160 ERROR rally_openstack.task.cleanup.manager   File "/root/rally_venv/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 92, in exception_handler_v20
2021-07-08 16:41:03.066 2160 ERROR rally_openstack.task.cleanup.manager     request_ids=request_ids)
2021-07-08 16:41:03.066 2160 ERROR rally_openstack.task.cleanup.manager neutronclient.common.exceptions.InternalServerError: subnets are not empty exception
2021-07-08 16:41:03.066 2160 ERROR rally_openstack.task.cleanup.manager 

[yanmo96]

Update more detail on this issue:
With running "create_and_delete_subnets" 10 concurrency. Will have the following error:
2021-09-02 23:17:13.593 26032 DEBUG neutronclient.v2_0.client [-] Error message: {"timestamp":"2021-09-03T06:17:13.587+0000","status":500,"error":"Internal Server Error","message":"subnets are not empty exception","path":"/project/6e02f875344145d49def4eaa36e3ff74/vpcs/76b3781b-73e1-48a2-8850-8c7f10a72546"} _handle_fault_response /root/rally_venv/lib/python3.6/site-packages/neutronclient/v2_0/client.py:262

• From VPC swagger UI shows the network have 6 subnet in it.

    "subnets": [
      "4ea40b03-9741-4c22-b5bd-ea549d5b39ee",
      "b44adc00-caab-493a-bdff-f5bd044135ae",
      "4e9b0ed8-21d4-434d-9a0c-f77ad310e687",
      "980ed39f-5e07-42e7-8d78-86b61ce7c930",
      "d95d6f3f-9c94-44ff-b354-4f9881849f36",
      "2be85bfa-11a8-4c5c-940c-49f33ca217fd"
    ]

• Then using the subnet id from above in subnet swagger UI will show the following:

{
  "subnet": null
}

• With the project id and subnet id in port_manager swagger UI will show:
  • they have 0 normal port from /project/{project_id}/subnet-port-count/{subnet_id}
  • and 0 gateway port from /project/{project_id}/ports