Skip to content

Slice plugin: self heal when asset etag/last-modified block mismatches #6691

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
traeak merged 1 commit into from
Jun 4, 2020
Merged

Slice plugin: self heal when asset etag/last-modified block mismatches #6691

traeak merged 1 commit into from
Jun 4, 2020

Conversation

@traeak
Copy link
Contributor

@traeak traeak commented Apr 20, 2020 •
edited
Loading

This tries to make the slice plugin smarter when origins do bad things to assets without changing their path including:

  • etag change
  • last modified change
  • content length change
  • change no nocache
  • asset removed

Process:

  1. Reference slice fetched (either slice0 or first slice in request).
  2. Second slice fetched -- mismatch detected
  3. Refetch second slice (cache_range_requests X-Crr-Ims header)
    a. Match reference, continue to next slice
    a. Mismatch reference, abort client, refetch reference slice (X-Crr-Ims header)

@traeak traeak force-pushed the slice_selfhealing branch 3 times, most recently from f0882b0 to 791ad9d Compare April 24, 2020 15:58
@traeak traeak force-pushed the slice_selfhealing branch 6 times, most recently from c30687b to 789ce99 Compare April 30, 2020 20:41
@traeak traeak force-pushed the slice_selfhealing branch 9 times, most recently from 74ebe07 to d886207 Compare May 6, 2020 17:15
@traeak traeak force-pushed the slice_selfhealing branch 3 times, most recently from 1d8aa17 to f775374 Compare May 12, 2020 15:17
@traeak traeak marked this pull request as ready for review May 12, 2020 15:17
randall
randall reviewed May 14, 2020
View reviewed changes
Copy link
Contributor

@randall randall left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a handful of unimportant nits

@traeak traeak force-pushed the slice_selfhealing branch from f775374 to 7382700 Compare May 14, 2020 17:40
@zwoop zwoop added this to the 10.0.0 milestone May 14, 2020
@zwoop
Copy link
Contributor

zwoop commented May 14, 2020

Is this something you need for 9.0.x? Or 9.1.x ?

@zwoop zwoop added the Plugins label May 14, 2020
@traeak
Copy link
Contributor Author

traeak commented May 14, 2020

9.0.x would be nice.

@traeak traeak force-pushed the slice_selfhealing branch from 7382700 to 53b95cd Compare May 15, 2020 15:09
@traeak
Copy link
Contributor Author

traeak commented May 15, 2020

Update removed some of the error logs and also trimmed more off the autests (traffic_manager -> traffic_server)

@traeak
Slice plugin: recover out of sync slices, better handling of non 206s
6dd31eb 
The very first slice is used as the reference slice to make recovery easiest.
Recover asset where slices differ in  content-length, etag, last-modified.
autests included to cover basic functionality.

Includes additions to include/ts/experimental.h:

tsapi time_t TSMimeParseDate(char const *const value_str, int const value_len);
tsapi TSReturnCode TSMimeFormatDate(time_t const value_time, char *const value_str, int *const value_len);

These are derived from TSMimeHdrFieldValueDateSet/TSMimeHdrFieldValueDateGet.
@traeak traeak force-pushed the slice_selfhealing branch from 53b95cd to 6dd31eb Compare May 27, 2020 19:44
@randall
Copy link
Contributor

randall commented May 28, 2020

[approve ci autest]

@traeak traeak merged commit 3683411 into apache:master Jun 4, 2020
traeak added a commit to traeak/trafficserver that referenced this pull request Jan 19, 2021
@traeak
Slice plugin: recover out of sync slices, better handling of non 206s (…
40f7e72 
…apache#6691)

The very first slice is used as the reference slice to make recovery easiest.
Recover asset where slices differ in  content-length, etag, last-modified.
autests included to cover basic functionality.

Includes additions to include/ts/experimental.h:

tsapi time_t TSMimeParseDate(char const *const value_str, int const value_len);
tsapi TSReturnCode TSMimeFormatDate(time_t const value_time, char *const value_str, int *const value_len);

These are derived from TSMimeHdrFieldValueDateSet/TSMimeHdrFieldValueDateGet.

(cherry picked from commit 6dd31eb)
@traeak traeak mentioned this pull request Jan 19, 2021
Merged
zwoop pushed a commit that referenced this pull request Jan 20, 2021
@traeak @maskit
Slice: 9.0.x back port of self healing and throttle by default (#7435)
794f2c3 
* Slice plugin: recover out of sync slices, better handling of non 206s (#6691)

The very first slice is used as the reference slice to make recovery easiest.
Recover asset where slices differ in  content-length, etag, last-modified.
autests included to cover basic functionality.

Includes additions to include/ts/experimental.h:

tsapi time_t TSMimeParseDate(char const *const value_str, int const value_len);
tsapi TSReturnCode TSMimeFormatDate(time_t const value_time, char *const value_str, int *const value_len);

These are derived from TSMimeHdrFieldValueDateSet/TSMimeHdrFieldValueDateGet.

(cherry picked from commit 6dd31eb)

* Make format specifier for time_t portable (#6860)

(cherry picked from commit dde3897)

* autest whitespace

* slice: clean up of created 502 response header (#6919)

Change header debug to use TSHttpHdrPrint.

Co-authored-by: Brian Olsen <bolsen149@ipcdn-cache-58.cdnlab.comcast.net>
(cherry picked from commit 9d23ab6)

* default to throttling and subsequently simplify the transfer code (#7257)

slice: default to throttling, 416 changes and bug fix for downstream closing.
(cherry picked from commit 4462484)

Co-authored-by: Masakazu Kitajo <maskit@apache.org>
@zwoop
Copy link
Contributor

zwoop commented Jan 20, 2021

Backported to 9.0.1 in #7435

@zwoop zwoop modified the milestones: 10.0.0, 9.0.1 Jan 20, 2021
traeak added a commit to traeak/trafficserver that referenced this pull request Apr 23, 2021
@traeak
slice port from master as of c8c9540
46c1714 
better handling of TSVIO calls and TSVConnAbort (apache#6239)
Slice plugin: recover out of sync slices, better handling of non 206s (apache#6691)
slice: clean up of created 502 response header (apache#6919)
default to throttling and subsequently simplify the transfer code (apache#7257)

(cherry picked from commit a622cbe)

slice/handleFirstServerHeader: return sooner on requested range errors (apache#7486)

(cherry picked from commit dfd4c7a)
@traeak traeak mentioned this pull request Apr 23, 2021
Merged
ezelkow1 pushed a commit that referenced this pull request Apr 27, 2021
@traeak
slice port from master to 8.1.x -- self healing and throttle by defau…
2f8941e 
…lt. (#7740)

* slice port from master as of c8c9540

better handling of TSVIO calls and TSVConnAbort (#6239)
Slice plugin: recover out of sync slices, better handling of non 206s (#6691)
slice: clean up of created 502 response header (#6919)
default to throttling and subsequently simplify the transfer code (#7257)

(cherry picked from commit a622cbe)

slice/handleFirstServerHeader: return sooner on requested range errors (#7486)

(cherry picked from commit dfd4c7a)

* cache_range_requests plugin optional support for special IMS header (#6217)

(cherry picked from commit fbe11ae)
@traeak traeak deleted the slice_selfhealing branch January 13, 2023 18:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@randall randall randall approved these changes

Assignees

@traeak traeak

Labels

Plugins

Projects

None yet

Milestone

9.0.1

Development

Successfully merging this pull request may close these issues.

3 participants

@traeak @zwoop @randall