PersistentDataset tempfile issue

[wyli]

Describe the bug
issue detected in v0.5.2:

MONAI/monai/data/dataset.py

Line 253 in feb3a33

temp_hash_file.rename(hashfile)

  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 202, in _worker_loop
    data = fetcher.fetch(index)
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/opt/monai/monai/data/dataset.py", line 93, in __getitem__
    return self._transform(index)
  File "/opt/monai/monai/data/dataset.py", line 259, in _transform
    pre_random_item = self._cachecheck(self.data[index])
  File "/opt/monai/monai/data/dataset.py", line 253, in _cachecheck
    temp_hash_file.rename(hashfile)
  File "/opt/conda/lib/python3.8/pathlib.py", line 1358, in rename
    self._accessor.rename(self, target)
OSError: [Errno 18] Invalid cross-device link: '/tmp/tmpc9cd7k66/0e709abea045c1af2a5a94d166f81d65.pt' -> 'val_cache/0e709abea045c1af2a5a94d166f81d65.pt'

root cause is that the default tempfile location /tmp is on a different filesystem than val_cache.
a potential workaround would be changing the default temp location of python:
export TMPDIR=/path/to/val_cache

[YuanTingHsieh]

The documentation of os.rename (https://docs.python.org/3/library/os.html#os.rename) states "The operation may fail on some Unix flavors if src and dst are on different filesystems".

Maybe we can try to use shutil.move (https://docs.python.org/3/library/shutil.html#shutil.move)?