[MRESOLVER-7] Download dependency POMs in parallel#142
[MRESOLVER-7] Download dependency POMs in parallel#142ibabiankou wants to merge 6 commits intoapache:masterfrom ibabiankou:MRESOLVER-7/master
Conversation
Ensures DefaultVersionFilterContext instances are not shared between threads
DefaultDependencyCollectionContext will be shared among the threads, so make it immutable to avoid side effects.
NodeStack is used by multiple threads handling dependencies on the same level of the tree, making it immutable to avoid side effects.
All child dependencies resolved asynchronously in the ExecutorService.
|
I am very certain that the stack trace is a result of a race condition. |
|
@michael-o I'm absolutely certain of that too. I'm not sure however what is the best way to solve it, especially because I don't have a reliable way to reproduce it 😞 Looking at the implementation of the To fix it I can think of modifying getArtifactDescriptorResult to ensure it allows single call for the same artifact at a time (something like this). But that feels a bit wrong place to solve this. Ideally I would put that kind of logic in the What do you think @michael-o? Does this approach make sense to you? |
|
@ibabiankou I need to think about this because from the top of my head I don't understand why we need advisory file lock at all. We have now decent @cstamas Please share your opinion when you are back to normal. |
|
Does this intersect with https://issues.apache.org/jira/browse/MRESOLVER-133? |
@michael-o Only about the switching to BFS strategy, however it’s not the biggest part of the change. In fact the DFS > BFS change is irrelevant for that ticket as far as I can tell. The goal there is to change which version out of the version range is selected. Now
Implied
I'd say the ticket title and description is a bit misleading, but the improvement itself does make sense to me. |
|
🤯 There are several improvements related to the same functionality that seems to be active at the moment:
Lets have a look if we can reconcile them :) |
|
@ibabiankou Thank you for looking into this. I see three mostly independent issues to be solved:
Does this make sense? Therefore, I'd like to discuss the first two and then the parallel approach. I don't want to review all of them in parallel (no pun intended) |
|
@michael-o They are quite independent indeed. They all related to the same area in the code, so even if considered and implemented separately I doubt it will be possible to revert them individually.
Are you talking here about the entire https://issues.apache.org/jira/browse/MRESOLVER-133 (DFS > BFS and changes to the version selection logic) or DFS to BFS change only? I believe it would be easier to consider them separately as version selection order can be changed separately.
Agree. I'll try to help with discussions and review, because I've already spent some time on this and would love to see these three changes to be implemented and released sooner rather than later 😅 |
Yes, they are related, but we can work on them serially.
Clearly, I want them separately. pure DFS to BFS change only w/o any behavior change. Then discuss selection order which intersects with other issues as well.
Yes, I already thought so. I am happy to review and merge upstream as long as:
|
|
Not that it's solid proof, but got a very similar exception today while using the stock maven 3.8.4 🤔 Stack Trace
[ERROR] Failed to execute goal on project dummy-artifact: Could not resolve dependencies for project dummy-group:dummy-artifact:jar:63.0-SNAPSHOT: Could not transfer artifact io.sentry:sentry:jar:5.5.3 from/to nexus-server (https://nexus.at-our-company.com/repository/maven-public): /Users/user/.m2/repository/io/sentry/sentry/5.5.3/sentry-5.5.3.jar.part (No such file or directory) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project dummy-artifact: Could not resolve dependencies for project dummy-group:dummy-artifact:jar:63.0-SNAPSHOT: Could not transfer artifact io.sentry:sentry:jar:5.5.3 from/to nexus-server (https://nexus.at-our-company.com/repository/maven-public): /Users/user/.m2/repository/io/sentry/sentry/5.5.3/sentry-5.5.3.jar.part (No such file or directory)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies (LifecycleDependencyResolver.java:269)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies (LifecycleDependencyResolver.java:147)
at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved (MojoExecutor.java:248)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:202)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:196)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:186)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:515)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628)
at java.lang.Thread.run (Thread.java:829)
Caused by: org.apache.maven.project.DependencyResolutionException: Could not resolve dependencies for project dummy-group:dummy-artifact:jar:63.0-SNAPSHOT: Could not transfer artifact io.sentry:sentry:jar:5.5.3 from/to nexus-server (https://nexus.at-our-company.com/repository/maven-public): /Users/user/.m2/repository/io/sentry/sentry/5.5.3/sentry-5.5.3.jar.part (No such file or directory)
at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve (DefaultProjectDependenciesResolver.java:198)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies (LifecycleDependencyResolver.java:243)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies (LifecycleDependencyResolver.java:147)
at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved (MojoExecutor.java:248)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:202)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:196)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:186)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:515)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628)
at java.lang.Thread.run (Thread.java:829)
Caused by: org.eclipse.aether.resolution.DependencyResolutionException: Could not transfer artifact io.sentry:sentry:jar:5.5.3 from/to nexus-server (https://nexus.at-our-company.com/repository/maven-public): /Users/user/.m2/repository/io/sentry/sentry/5.5.3/sentry-5.5.3.jar.part (No such file or directory)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies (DefaultRepositorySystem.java:357)
at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve (DefaultProjectDependenciesResolver.java:191)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies (LifecycleDependencyResolver.java:243)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies (LifecycleDependencyResolver.java:147)
at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved (MojoExecutor.java:248)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:202)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:196)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:186)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:515)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628)
at java.lang.Thread.run (Thread.java:829)
Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact io.sentry:sentry:jar:5.5.3 from/to nexus-server (https://nexus.at-our-company.com/repository/maven-public): /Users/user/.m2/repository/io/sentry/sentry/5.5.3/sentry-5.5.3.jar.part (No such file or directory)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve (DefaultArtifactResolver.java:425)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts (DefaultArtifactResolver.java:229)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies (DefaultRepositorySystem.java:340)
at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve (DefaultProjectDependenciesResolver.java:191)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies (LifecycleDependencyResolver.java:243)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies (LifecycleDependencyResolver.java:147)
at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved (MojoExecutor.java:248)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:202)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:196)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:186)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:515)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628)
at java.lang.Thread.run (Thread.java:829)
Caused by: org.eclipse.aether.transfer.ArtifactTransferException: Could not transfer artifact io.sentry:sentry:jar:5.5.3 from/to nexus-server (https://nexus.at-our-company.com/repository/maven-public): /Users/user/.m2/repository/io/sentry/sentry/5.5.3/sentry-5.5.3.jar.part (No such file or directory)
at org.eclipse.aether.connector.basic.ArtifactTransportListener.transferFailed (ArtifactTransportListener.java:52)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run (BasicRepositoryConnector.java:369)
at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run (RunnableErrorForwarder.java:75)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628)
at java.lang.Thread.run (Thread.java:829)
Caused by: java.io.FileNotFoundException: /Users/user/.m2/repository/io/sentry/sentry/5.5.3/sentry-5.5.3.jar.part (No such file or directory)
at java.io.FileInputStream.open0 (Native Method)
at java.io.FileInputStream.open (FileInputStream.java:219)
at java.io.FileInputStream.<init> (FileInputStream.java:157)
at org.eclipse.aether.internal.impl.DefaultFileProcessor.copy (DefaultFileProcessor.java:163)
at org.eclipse.aether.internal.impl.DefaultFileProcessor.copy (DefaultFileProcessor.java:151)
at org.eclipse.aether.internal.impl.DefaultFileProcessor.move (DefaultFileProcessor.java:252)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$GetTaskRunner.runTask (BasicRepositoryConnector.java:482)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run (BasicRepositoryConnector.java:364)
at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run (RunnableErrorForwarder.java:75)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628)
at java.lang.Thread.run (Thread.java:829)As discussed earlier, I will revive this PR once #144 and updated version of #136 are merged. |
|
Issue isn't related. This is a vanilla concurrency issue which will go away with Maven 3.9.0 by default. |
|
Don't use the old Java Utils model. |
|
pls incorporate latest master changes (whether rebase+squash+force push or just merge) |
|
@Tibor17 Thanks for pointing this out, however, I'm a bit puzzled as I don't see a good alternative here 🤔 Do you have any ideas from the top of your head? @cstamas The plan is to first focus on reviewing and merging #144, then work on either this one or #136. I'll close this one for now and re-open or open another one later. |
|
Resolve #817 |
1 similar comment
|
Resolve #817 |
Updates
DefaultDependencyCollectorto process descriptors concurrently.Updated implementation does breadth-first traversal of the dependency graph and processes each dependency in a separate thread.
Order of the children nodes then restored to match order in descriptor of parent.
By default uses 5 threads, which can be changed using either
maven.descriptor.threadsormaven.artifact.threadsproperty.All the unit and integration tests are green. The change was also tested locally on a reasonable codebase (~1100 modules with lots dependencies) with 24 modules processed concurrently (-T1C).
Important note: a few times out of dosens of tests I observed an exception that seems to be related to the change (stack trace attached), however I couldn't really piece it together how it's possible, so ideas are welcome.
Stack Trace