[SPARK-18669][SS][DOCS] Update Apache docs for Structured Streaming regarding watermarking and status by tdas · Pull Request #16294 · apache/spark

tdas · 2016-12-15T07:46:33Z

What changes were proposed in this pull request?

Extended the Window operation section with code snippet and explanation of watermarking
Extended the Output Mode section with a table showing the compatibility between query type and output mode
Rewrote the Monitoring section with updated jsons generated by StreamingQuery.progress/status
Updated API changes in the StreamingQueryListener example

TODO

Figure showing the watermarking

How was this patch tested?

N/A

Screenshots

Section: Windowed Aggregation with Event Time

Section: Output Modes

Section: Monitoring

SparkQA · 2016-12-15T07:47:37Z

Test build #70182 has started for PR 16294 at commit ed8d9e0.

SparkQA · 2016-12-16T01:26:10Z

Test build #70218 has finished for PR 16294 at commit 1566433.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangmiao1981 · 2016-12-16T06:13:52Z

+<div data-lang="scala"  markdown="1">
+
+{% highlight scala %}
+import spark.implicits._


Just curious, will there be a complete example in the examples folder? In documents like ML, SQL, the code is cited from the example file instead of hard code in the document. Thanks!

I wasnt planning to adding examples in this PR to keep this just about docs. I am happy to see simple examples contributed as PRs from the community.

dongjoon-hyun · 2016-12-16T17:42:38Z

Hi, @tdas .

Is it possible for you to update this statement for Apache Spark 2.1?

Spark 2.0 is the ALPHA RELEASE of Structured Streaming and the APIs are still experimental. In this guide, we are going to walk you through~

https://github.com/apache/spark/blame/master/docs/structured-streaming-programming-guide.md#L13

tdas · 2016-12-21T23:45:18Z

will do. i am rewriting parts of this PR right now.

SparkQA · 2016-12-22T04:08:40Z

Test build #70500 has finished for PR 16294 at commit 8f8c11d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-12-22T21:50:43Z

Test build #70525 has finished for PR 16294 at commit b96351c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-12-22T22:43:24Z

Test build #70524 has finished for PR 16294 at commit a4a93aa.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

tdas · 2016-12-22T23:28:36Z

^^ The last failure on build 70524 is for a older commit. I reverted the culprit change and build 70525 passed.

zsxwing

Looks good overall. Just some nits.

zsxwing · 2016-12-23T00:08:58Z

    }
    @Overrides void onQueryProgress(QueryProgressEvent queryProgress) {
-        System.out.println("Query made progress: " + queryProgress.queryStatus);
+        System.out.println("Query made progress: " + queryProgress.progress);


nit: queryProgress.progress()

zsxwing · 2016-12-23T00:09:13Z

    }
    @Overrides void onQueryTerminated(QueryTerminatedEvent queryTerminated) {
-        System.out.println("Query terminated: " + queryTerminated.queryStatus.name);
+        System.out.println("Query terminated: " + queryTerminated.id);


nit: queryTerminated.id()

zsxwing · 2016-12-23T00:09:27Z

-
    @Overrides void onQueryStarted(QueryStartedEvent queryStarted) {
-        System.out.println("Query started: " + queryTerminated.queryStatus.name);
+        System.out.println("Query started: " + queryTerminated.id);


nit: queryStarted.id()

zsxwing · 2016-12-23T00:10:04Z

-
    override def onQueryStarted(queryStarted: QueryStartedEvent): Unit = {
-        println("Query started: " + queryTerminated.queryStatus.name)
+        println("Query started: " + queryTerminated.id)


nit: queryTerminated -> queryStarted

zsxwing · 2016-12-23T00:11:24Z

 {% highlight python %}
-query = ...  // a StreamingQuery
+query = ...  # a StreamingQuery
+print(query.progress)


nit: lastProgress

zsxwing · 2016-12-23T00:11:36Z

-                latency.getBatch.source: 20
-    Sink status - MySink
-        Committed offsets: [1, -]
+System.out.println(query.status);


nit: query.status()

zsxwing · 2016-12-23T00:12:13Z

 StreamingQuery query = ...

-System.out.println(query.status);
+System.out.println(query.progress);


nit: query.lastProgress()

zsxwing · 2016-12-23T00:12:34Z

 {% highlight scala %}
 val query: StreamingQuery = ...

+println(query.progress)


nit: query.lastProgress

zsxwing · 2016-12-23T00:14:30Z

+        preserves all data in the Result Table.
+    </td>  
+  </tr>
+  <tr>


same reason as the other place.

zsxwing · 2016-12-23T00:14:38Z

-  </tr> 
+  </tr>
+  <tr>
+    <td></td>


empty row to make sure that there is line after the table when rendered. otherwise looks odd.

zsxwing · 2016-12-23T01:15:08Z

    }
    @Overrides void onQueryProgress(QueryProgressEvent queryProgress) {
-        System.out.println("Query made progress: " + queryProgress.progress);
+        System.out.println("Query made progress: " + queryProgress.lastProgress());


nit: the method is QueryProgressEvent.progress

yep. replace fail.

SparkQA · 2016-12-23T01:32:01Z

Test build #70529 has finished for PR 16294 at commit 0cc1dd6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

zsxwing · 2016-12-23T01:37:56Z

LGTM pending tests

SparkQA · 2016-12-23T01:53:40Z

Test build #70530 has finished for PR 16294 at commit 576b432.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

…egarding watermarking and status ## What changes were proposed in this pull request? - Extended the Window operation section with code snippet and explanation of watermarking - Extended the Output Mode section with a table showing the compatibility between query type and output mode - Rewrote the Monitoring section with updated jsons generated by StreamingQuery.progress/status - Updated API changes in the StreamingQueryListener example TODO - [x] Figure showing the watermarking ## How was this patch tested? N/A ## Screenshots ### Section: Windowed Aggregation with Event Time <img width="927" alt="screen shot 2016-12-15 at 3 33 10 pm" src="https://cloud.githubusercontent.com/assets/663212/21246197/0e02cb1a-c2dc-11e6-8816-0cd28d8201d7.png"> ![image](https://cloud.githubusercontent.com/assets/663212/21246241/45b0f87a-c2dc-11e6-9c29-d0a89e07bf8d.png) <img width="929" alt="screen shot 2016-12-15 at 3 33 46 pm" src="https://cloud.githubusercontent.com/assets/663212/21246202/1652cefa-c2dc-11e6-8c64-3c05977fb3fc.png"> ---------------------------- ### Section: Output Modes ![image](https://cloud.githubusercontent.com/assets/663212/21246276/8ee44948-c2dc-11e6-9fa2-30502fcf9a55.png) ---------------------------- ### Section: Monitoring ![image](https://cloud.githubusercontent.com/assets/663212/21246535/3c5baeb2-c2de-11e6-88cd-ca71db7c5cf9.png) ![image](https://cloud.githubusercontent.com/assets/663212/21246574/789492c2-c2de-11e6-8471-7bef884e1837.png) Author: Tathagata Das <tathagata.das1565@gmail.com> Closes #16294 from tdas/SPARK-18669. (cherry picked from commit 092c672) Signed-off-by: Shixiong Zhu <shixiong@databricks.com>

This version is built from the docs source code generated by applying apache/spark#16294 to v2.1.0 (so, other changes in branch 2.1 will not affect the doc).

…egarding watermarking and status ## What changes were proposed in this pull request? - Extended the Window operation section with code snippet and explanation of watermarking - Extended the Output Mode section with a table showing the compatibility between query type and output mode - Rewrote the Monitoring section with updated jsons generated by StreamingQuery.progress/status - Updated API changes in the StreamingQueryListener example TODO - [x] Figure showing the watermarking ## How was this patch tested? N/A ## Screenshots ### Section: Windowed Aggregation with Event Time <img width="927" alt="screen shot 2016-12-15 at 3 33 10 pm" src="https://cloud.githubusercontent.com/assets/663212/21246197/0e02cb1a-c2dc-11e6-8816-0cd28d8201d7.png"> ![image](https://cloud.githubusercontent.com/assets/663212/21246241/45b0f87a-c2dc-11e6-9c29-d0a89e07bf8d.png) <img width="929" alt="screen shot 2016-12-15 at 3 33 46 pm" src="https://cloud.githubusercontent.com/assets/663212/21246202/1652cefa-c2dc-11e6-8c64-3c05977fb3fc.png"> ---------------------------- ### Section: Output Modes ![image](https://cloud.githubusercontent.com/assets/663212/21246276/8ee44948-c2dc-11e6-9fa2-30502fcf9a55.png) ---------------------------- ### Section: Monitoring ![image](https://cloud.githubusercontent.com/assets/663212/21246535/3c5baeb2-c2de-11e6-88cd-ca71db7c5cf9.png) ![image](https://cloud.githubusercontent.com/assets/663212/21246574/789492c2-c2de-11e6-8471-7bef884e1837.png) Author: Tathagata Das <tathagata.das1565@gmail.com> Closes apache#16294 from tdas/SPARK-18669.

tdas added 2 commits December 13, 2016 18:52

Added watermarking

a31c861

Update metrics

ed8d9e0

Added new figure

1566433

tdas changed the title ~~[WIP][SPARK-18669][SS][DOCS] Update Apache docs for Structured Streaming regarding watermarking and status~~ [SPARK-18669][SS][DOCS] Update Apache docs for Structured Streaming regarding watermarking and status Dec 15, 2016

wangmiao1981 reviewed Dec 16, 2016

View reviewed changes

updated

8f8c11d

tdas added 2 commits December 22, 2016 13:15

Updates

a4a93aa

Reverted UnsupportedOperationChecke

b96351c

zsxwing requested changes Dec 23, 2016

View reviewed changes

Addressed comments

0cc1dd6

zsxwing reviewed Dec 23, 2016

View reviewed changes

Fix wrong change

576b432

asfgit closed this in 092c672 Dec 28, 2016

Conversation

tdas commented Dec 15, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Screenshots

Section: Windowed Aggregation with Event Time

Section: Output Modes

Section: Monitoring

Uh oh!

SparkQA commented Dec 15, 2016

Uh oh!

SparkQA commented Dec 16, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Dec 16, 2016

Uh oh!

tdas commented Dec 21, 2016

Uh oh!

SparkQA commented Dec 22, 2016

Uh oh!

SparkQA commented Dec 22, 2016

Uh oh!

SparkQA commented Dec 22, 2016

Uh oh!

tdas commented Dec 22, 2016

Uh oh!

zsxwing left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 23, 2016

Uh oh!

zsxwing commented Dec 23, 2016

Uh oh!

SparkQA commented Dec 23, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

tdas commented Dec 15, 2016 •

edited

Loading