Crash with --replace-text input file longer than ~1550 lines (StackOverFlowError) #528
Description
Seems that program crashes with input file longer than 1563 lines when using --replace-text option.
System info:
java --version
openjdk 17.0.13 2024-10-15
OpenJDK Runtime Environment (build 17.0.13+11-Debian-2deb12u1)
OpenJDK 64-Bit Server VM (build 17.0.13+11-Debian-2deb12u1, mixed mode, sharing)
java -jar ./bfg-1.15.0.jar --version
bfg 1.15.0
Reproduction:
git clone https://github.com/rtyley/bfg-repo-cleaner.git
cd bfg-repo-cleaner
wget https://repo1.maven.org/maven2/com/madgag/bfg/1.15.0/bfg-1.15.0.jar
# make random numbers in file
< /dev/urandom tr -dc '0-9' | fold -w 13 | head -n 10000 > 10000_random_numbers.txt
java -jar ./bfg-1.15.0.jar --replace-text 10000_random_numbers.txt .
Crashes with:
$ java -jar ./bfg-1.15.0.jar --replace-text 10000_random_numbers.txt .
Using repo : /home/user/tmp/bfg-repo-cleaner/./.git
This repo has been processed by The BFG before! Will prune repo before proceeding - to avoid unnecessary cleaning work on unused objects...
Completed prune of old objects - will now proceed with the main job!
Found 122 objects to protect
Found 57 commit-pointing refs : HEAD, refs/heads/main, refs/remotes/origin/HEAD, ...
Found 58 tag-pointing refs : refs/tags/v0.10.0, refs/tags/v0.11.0, refs/tags/v0.12.0, ...
Protected commits
-----------------
These are your protected commits, and so their contents will NOT be altered:
java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at use.newer.java.Util.invokeMainMethod(Util.java:89)
at use.newer.java.Util.executeMainProgram(Util.java:28)
at use.newer.java.Util.checkJavaVersionAndExecuteMainProgram(Util.java:18)
at use.newer.java.Version8.main(Version8.java:5)
Caused by: com.google.common.util.concurrent.ExecutionError: com.google.common.util.concurrent.ExecutionError: java.lang.StackOverflowError
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2084)
at com.google.common.cache.LocalCache.get(LocalCache.java:4017)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4040)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4989)
at com.madgag.git.bfg.MemoUtil$$anon$2.apply(memo.scala:60)
at com.madgag.git.bfg.GitUtil$$anon$1.apply(GitUtil.scala:71)
at com.madgag.git.bfg.CleaningMapper.replacement(GitUtil.scala:46)
at com.madgag.git.bfg.CleaningMapper.replacement$(GitUtil.scala:45)
at com.madgag.git.bfg.GitUtil$$anon$1.replacement(GitUtil.scala:70)
at com.madgag.git.bfg.cleaner.protection.ProtectedObjectDirtReport$.$anonfun$reportsFor$3(ProtectedObjectDirtReport.scala:45)
at scala.util.Either.fold(Either.scala:197)
at com.madgag.git.bfg.cleaner.protection.ProtectedObjectDirtReport$.$anonfun$reportsFor$1(ProtectedObjectDirtReport.scala:45)
at scala.collection.Iterator$$anon$9.next(Iterator.scala:584)
at scala.collection.mutable.Growable.addAll(Growable.scala:62)
at scala.collection.mutable.Growable.addAll$(Growable.scala:57)
at scala.collection.immutable.SetBuilderImpl.addAll(Set.scala:405)
at scala.collection.immutable.Set$.from(Set.scala:362)
at scala.collection.immutable.Set$.from(Set.scala:93)
at scala.collection.IterableOps.map(Iterable.scala:684)
at scala.collection.IterableOps.map$(Iterable.scala:684)
at scala.collection.AbstractIterable.map(Iterable.scala:935)
at com.madgag.git.bfg.cleaner.protection.ProtectedObjectDirtReport$.reportsFor(ProtectedObjectDirtReport.scala:43)
at com.madgag.git.bfg.cleaner.CLIReporter.reportProtectedCommitsAndTheirDirt(Reporter.scala:133)
at com.madgag.git.bfg.cleaner.CLIReporter.reportObjectProtection(Reporter.scala:106)
at com.madgag.git.bfg.cleaner.RepoRewriter$.rewrite(RepoRewriter.scala:96)
at com.madgag.git.bfg.cli.Main$.$anonfun$new$1(Main.scala:59)
at com.madgag.git.bfg.cli.Main$.$anonfun$new$1$adapted(Main.scala:34)
at scala.Option.map(Option.scala:242)
at com.madgag.git.bfg.cli.Main$.delayedEndpoint$com$madgag$git$bfg$cli$Main$1(Main.scala:34)
at com.madgag.git.bfg.cli.Main$delayedInit$body.apply(Main.scala:27)
at scala.Function0.apply$mcV$sp(Function0.scala:42)
at scala.Function0.apply$mcV$sp$(Function0.scala:42)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
at scala.App.$anonfun$main$1(App.scala:98)
at scala.App.$anonfun$main$1$adapted(App.scala:98)
at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:619)
at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:617)
at scala.collection.AbstractIterable.foreach(Iterable.scala:935)
at scala.App.main(App.scala:98)
at scala.App.main$(App.scala:96)
at com.madgag.git.bfg.cli.Main$.main(Main.scala:27)
at com.madgag.git.bfg.cli.Main.main(Main.scala)
... 8 more
Caused by: com.google.common.util.concurrent.ExecutionError: java.lang.StackOverflowError
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2084)
at com.google.common.cache.LocalCache.get(LocalCache.java:4017)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4040)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4989)
at com.madgag.git.bfg.MemoUtil$$anon$2.apply(memo.scala:60)
at scala.collection.immutable.List.map(List.scala:247)
at scala.collection.immutable.List.map(List.scala:79)
at com.madgag.git.bfg.cleaner.TreeBlobModifier.apply(TreeBlobModifier.scala:37)
at com.madgag.git.bfg.cleaner.TreeBlobModifier.apply$(TreeBlobModifier.scala:37)
at com.madgag.git.bfg.cli.CLIConfig$$anon$2.apply(CLIConfig.scala:174)
at com.madgag.git.bfg.cli.CLIConfig$$anon$2.apply(CLIConfig.scala:174)
at scala.Function$.$anonfun$chain$2(Function.scala:23)
at scala.collection.LinearSeqOps.foldLeft(LinearSeq.scala:183)
at scala.collection.LinearSeqOps.foldLeft$(LinearSeq.scala:179)
at scala.collection.immutable.List.foldLeft(List.scala:79)
at scala.Function$.$anonfun$chain$1(Function.scala:23)
at com.madgag.git.bfg.cleaner.ObjectIdCleaner.$anonfun$cleanTree$1(ObjectIdCleaner.scala:125)
at com.madgag.git.bfg.MemoUtil$$anon$3.load(memo.scala:74)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3574)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2316)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2190)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2080)
... 49 more
Caused by: java.lang.StackOverflowError
at scala.Function$.$anonfun$chain$2(Function.scala:23)
at scala.collection.LinearSeqOps.foldLeft(LinearSeq.scala:183)
at scala.collection.LinearSeqOps.foldLeft$(LinearSeq.scala:179)
at scala.collection.immutable.List.foldLeft(List.scala:79)
at scala.Function$.$anonfun$chain$1(Function.scala:23)
at scala.Function$.$anonfun$chain$2(Function.scala:23)
End of stack trace repeats itself, full version: https://pastebin.com/AKYcSNU2
As workaround, splitting to parts of 1000 works:
split -l 1000 10000_random_numbers.txt chunk_of_1000_ --additional-suffix=.txt
for file in chunk_of_1000_*.txt; do
java -jar ./bfg-1.15.0.jar --replace-text "$file" .
done
This evaluates without crash, looping over each chunk of 1000 in bash. Traced down that the error starts happening at around 1564 lines or more while 1563 is still okay.
Not familiar with Scala but could it be recursion limit issue or something?