Skip to content

feat: allow multithreaded fmindex construction #208

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
sarahet merged 3 commits into from
May 9, 2023
Merged

feat: allow multithreaded fmindex construction #208

sarahet merged 3 commits into from
May 9, 2023

Conversation

@SGSSGene
Copy link
Collaborator

@SGSSGene SGSSGene commented Apr 25, 2023

  • updates to newest fmindex
  • uses fmindex_collection now uses libsais instead of libdivsufsort
  • support for openmp
  • adds option to control the thread count for suffix array construction

h-2
h-2 reviewed Apr 25, 2023
View reviewed changes
Copy link
Member

@h-2 h-2 left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks for the PR!

I think you still need to add libsais to the your library's cmake (or ours).

@h-2
Copy link
Member

h-2 commented Apr 25, 2023

Do you know if memory usage changes between libdivsufsort and libsais? Currently, the memory requirements of lambda3's index construction are almost twice as high as lambda2's 😑

@SGSSGene SGSSGene force-pushed the feat/threaded_fmindex_construction branch from cfaf5e3 to eddb7c3 Compare April 26, 2023 14:59
@SGSSGene
Copy link
Collaborator Author

SGSSGene commented Apr 26, 2023

Do you know if memory usage changes between libdivsufsort and libsais? Currently, the memory requirements of lambda3's index construction are almost twice as high as lambda2's expressionless

No, I don't have any clue. I also don't know what influence the thread number will have on the memory consumption.

@SGSSGene SGSSGene force-pushed the feat/threaded_fmindex_construction branch 5 times, most recently from 31fbc60 to 35c9bf9 Compare April 27, 2023 14:31
@SGSSGene SGSSGene marked this pull request as ready for review April 27, 2023 14:51
@SGSSGene SGSSGene requested a review from h-2 April 27, 2023 17:53
SGSSGene and others added 3 commits May 3, 2023 16:41
@SGSSGene
fix: update to newest fmindex_collection
7c85a9b 
- This also changes the way that a search with an reversed index is
working, so no reversing of the text is required anymore.

- Requires to set the numbers of thread used for suffix array
  construction.
@SGSSGene @h-2
Update src/mkindex_algo.hpp
ed7a5a2 
Co-authored-by: Hannes Hauswedell <[email protected]>
@SGSSGene
fix: update fmindex_collection submodule
7b08976
@SGSSGene SGSSGene force-pushed the feat/threaded_fmindex_construction branch from ffccda7 to 7b08976 Compare May 3, 2023 14:41
@SGSSGene SGSSGene mentioned this pull request May 9, 2023
Merged
@sarahet sarahet merged commit 7b08976 into seqan:lambda3 May 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@h-2 h-2 Awaiting requested review from h-2

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@SGSSGene @h-2 @sarahet