[SPARK-19041][SS] Fix code snippet compilation issues in Structured Streaming Programming Guide#16442
Closed
lw-lin wants to merge 3 commits into
Closed
[SPARK-19041][SS] Fix code snippet compilation issues in Structured Streaming Programming Guide#16442lw-lin wants to merge 3 commits into
lw-lin wants to merge 3 commits into
Conversation
lw-lin
commented
Dec 31, 2016
Contributor
Author
There was a problem hiding this comment.
type won't compile in scala since it is a keyword, hence here change it to deviceType
lw-lin
commented
Dec 31, 2016
Contributor
Author
There was a problem hiding this comment.
file format requires checkpointLocation and path to be explicitly specified
lw-lin
commented
Dec 31, 2016
Contributor
Author
There was a problem hiding this comment.
currently we don't have parquet(...)/csv(...)/text(...)/etc. methods
c029e25 to
96532ae
Compare
lw-lin
commented
Dec 31, 2016
| query.recentProgress // an array of the most recent process updates for this query | ||
|
|
||
| query.sinkStatus() // progress information about data written to the output sink | ||
| query.lastProgress // the most recent process update of this streaming query |
Contributor
Author
There was a problem hiding this comment.
we've wrapped source/sink statuses into recentProcesses/lastProcess
lw-lin
commented
Dec 31, 2016
| } | ||
| @Overrides void onQueryProgress(QueryProgressEvent queryProgress) { | ||
| @Override | ||
| public void onQueryProgress(QueryProgressEvent queryProgress) { |
Contributor
Author
There was a problem hiding this comment.
It's @Override in java, and these methods must be public
Contributor
Author
|
@srowen would you take a look, thanks! |
lw-lin
commented
Dec 31, 2016
| // Running average signal for each device type | ||
| import org.apache.spark.sql.expressions.scalalang.typed._ | ||
| ds.groupByKey(_.type).agg(typed.avg(_.signal)) // using typed API | ||
| import org.apache.spark.sql.expressions.scalalang.typed |
Contributor
Author
There was a problem hiding this comment.
this import should be ….scalalang.typed rather than ….scalalang.typed._
|
Test build #70763 has finished for PR 16442 at commit
|
|
Test build #70764 has finished for PR 16442 at commit
|
srowen
requested changes
Dec 31, 2016
Member
srowen
left a comment
There was a problem hiding this comment.
I take your word for it on the recentProgress()/lastProgress() change but looks OK except for one comment
| noAggDF | ||
| .writeStream() | ||
| .parquet("path/to/destination/directory") | ||
| .writeStream |
Member
There was a problem hiding this comment.
This is a method call? it needs parens then.
|
Test build #70769 has finished for PR 16442 at commit
|
srowen
approved these changes
Jan 1, 2017
Member
|
Merged to master |
Member
|
Thanks! LGTM. I'm going to also cherry-pick to branch 2.1 in case we may want to push new docs to the website.. |
asfgit
pushed a commit
that referenced
this pull request
Jan 2, 2017
…treaming Programming Guide ## What changes were proposed in this pull request? Currently some code snippets in the programming guide just do not compile. We should fix them. ## How was this patch tested? ``` SKIP_API=1 jekyll build ``` ## Screenshot from part of the change:  Author: Liwei Lin <lwlin7@gmail.com> Closes #16442 from lw-lin/ss-pro-guide-.
Contributor
|
@lw-lin thanks for catching these! :) |
cmonkey
pushed a commit
to cmonkey/spark
that referenced
this pull request
Jan 4, 2017
…treaming Programming Guide ## What changes were proposed in this pull request? Currently some code snippets in the programming guide just do not compile. We should fix them. ## How was this patch tested? ``` SKIP_API=1 jekyll build ``` ## Screenshot from part of the change:  Author: Liwei Lin <lwlin7@gmail.com> Closes apache#16442 from lw-lin/ss-pro-guide-.
uzadude
pushed a commit
to uzadude/spark
that referenced
this pull request
Jan 27, 2017
…treaming Programming Guide ## What changes were proposed in this pull request? Currently some code snippets in the programming guide just do not compile. We should fix them. ## How was this patch tested? ``` SKIP_API=1 jekyll build ``` ## Screenshot from part of the change:  Author: Liwei Lin <lwlin7@gmail.com> Closes apache#16442 from lw-lin/ss-pro-guide-.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Currently some code snippets in the programming guide just do not compile. We should fix them.
How was this patch tested?
Screenshot from part of the change: