Sandcastle Embeddings Search

[tomdicarlo]

Description

This PR demonstrates an alternative searching approach for sandcastles that uses a vector embedding search instead of the existing pagefind search. It updates the process of building the gallery to create a vector embedding of each sandcastle, and then compares those embeddings against the user query.

To allow everything to run locally, we have selected a small MIT licensed model from Huggingface. One consideration for this PR is concerns over model download size and hardware performance limitations. Memory usage from the model appeared in testing to not rise above what is consumed for several of the existing sandcastles, however the download size is approximately 30MB which is several times larger than the largest file download I have been able to produce from existing sandcastles.

I will open this PR as a draft for now, as modifications will likely need to be made to support a hybrid approach between Pagefind and Vector embedding search, which we can discuss further within the PR.

Issue number and link

https://github.com/iTwin/platform-bentley-community/issues/306

Testing plan

My changes were tested manually against a small subset of search queries that I found to be gaps in the current Cesium Sandcastle search.

I did not include any automated testing of the functionality as it doesn't seem testing is configured in the application, but I would be glad to add automated testing if that is requested before moving the PR out of draft.

Author checklist

• I have submitted a Contributor License Agreement
• I have added my name to CONTRIBUTORS.md
• I have updated CHANGES.md with a short summary of my change
• I have added or updated unit tests to ensure consistent code coverage
  • No, as noted in testing plan
• I have updated the inline documentation, and included code examples where relevant
• I have performed a self-review of my code

[github-actions]

Thank you for the pull request, @tomdicarlo!

✅ We can confirm we have a CLA on file for you.