local data backend double-traversal fix

[bghira]

This pull request refactors the list_files method in LocalDataBackend to improve file discovery, especially with symlinks and forbidden directories, and adds comprehensive unit tests to ensure correct behavior. The new implementation uses os.walk to traverse directories, prevents duplicate file listings due to symlink cycles, and excludes files in forbidden directories.

Refactor and bug fixes in file listing:

• Rewrote the file discovery logic in list_files to use os.walk with symlink support, preventing duplicate files from symlink cycles and ensuring forbidden directories are pruned from traversal. [1] [2]

Testing improvements:

• Added a new test suite TestLocalDataBackendListFiles in test_local_files.py to verify that:
  • Nested files are not duplicated.
  • Symlinked directories are followed without causing cycles.
  • Forbidden directories (like .ipynb_checkpoints) are properly excluded from results.