retry GPU memory allocation if fragmented

[szha]

Description

retry GPU memory allocation if allocation failure occurs due to fragmentation.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage:
• To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

• retry GPU memory allocation if allocation failure occurs due to fragmentation.

Comments

• Memory fragmentation happens at runtime when total free memory is larger than the requested memory chunk, but no consecutive space can be found to accommodate the memory request.
• I'm changing the memory pool allocation logic when cudaErrorMemoryAllocation occurrs:
  1. release all cached unused memory chunks and retry
  2. retry memory allocation