gvfs-helper: add config to incrementally replace cache servers #836
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Before modifying the config documentation more, fill in these blanks. Signed-off-by: Derrick Stolee <[email protected]>
3c79ddd to
c83a879
Compare
In anticipation of tests for multiple cache-servers, update the existing logic that sets up and tears down cache-servers to allow multiple instances on different ports. Signed-off-by: Derrick Stolee <[email protected]>
This extension of the gvfs.cache-server config now allows a new key, gvfs.prefetch.cache-server, to override the cache-server URL for only the prefetch endpoint. The purpose of this config is to allow for incremental testing and deployment of new cache-server infrastructure. Hypothetically, we could have special-purpose cache-servers that are glorified bundle servers and other servers that focus on the object and size endpoints. More realistically, this will allow us to test cache servers that have only the prefetch endpoint ready to go. This allows some incremental rollout that is more controlled than a flag day replacing the entire infrastructure. Signed-off-by: Derrick Stolee <[email protected]>
This extension of the gvfs.cache-server config now allows a new key, gvfs.get.cache-server, to override the cache-server URL for only the prefetch endpoint. The purpose of this config is to allow for incremental testing and deployment of new cache-server infrastructure. Signed-off-by: Derrick Stolee <[email protected]>
This extension of the gvfs.cache-server config now allows a new key, gvfs.post.cache-server, to overrid the cache-server URL for only the batched objects endpoint. The purpose of this config is to allow for incremental testing and deployment of new cache-server infrastructure. Signed-off-by: Derrick Stolee <[email protected]>
Test that all three verb-specific cache-server configs can be used simultaneously, each directing requests to a different server. This verifies that prefetch, get, and post verbs each respect their own override and don't interfere with each other.
The t5799-gvfs-helper.sh script is long and takes forever. This slows down PR merges and the local development inner loop is a pain. Before distributing the tests into a set of new test scripts by topic, extract important helper methods that can be imported by the new scripts. Signed-off-by: Derrick Stolee <[email protected]>
c83a879 to
7222e1a
Compare
Move the tests from t5799-gvfs-helper.sh into multiple scripts that can run in parallel. To ensure that the ports do not overlap, add a large multiplier on the instance when needing multiple ports within the same test (currently limited to the verb-specific cache servers). Signed-off-by: Derrick Stolee <[email protected]>
7222e1a to
77a05f6
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I'm exploring ideas around a new cache-server infrastructure. One of the trickiest parts of deploying new infrastructure is the need to have all endpoints ready to go at launch without incremental learning.
This change adds new
gvfs.<verb>.cache-serverconfig keys that allow for verb-by-verb overrides of the typicalgvfs.cache-serverconfig key. These are loaded on a per-verb basis and then reset after the request. Further, if there is a failure then the request is retried with the base cache-server URL.This would allow us to, for example, deploy a service that serves only the
gvfs/prefetchendpoint and see if that is improving on latency and throughput expectations before moving on to the GET and POST verbs for single and batched object downloads.As I was adding tests, I realized that we should split this test script into distinct parts so we can have a faster inner loop when testing specific areas. I know that this script is frequently the longest script running in our PR and CI builds, so the parallel split should help significantly.
Use commit-by-commit review. I tried to keep the last two commits as obviously "copy-and-paste only" except for a small change to the port calculation to avoid overlap when using multiple ports in parallel tests.
GVFS Protocol.