Skip to content

[SPARK-21180][SQL] Remove conf from stats functions since now we have conf in LogicalPlan#18391

Closed
wzhfy wants to merge 3 commits into
apache:masterfrom
wzhfy:removeConf
Closed

[SPARK-21180][SQL] Remove conf from stats functions since now we have conf in LogicalPlan#18391
wzhfy wants to merge 3 commits into
apache:masterfrom
wzhfy:removeConf

Conversation

@wzhfy
Copy link
Copy Markdown
Contributor

@wzhfy wzhfy commented Jun 22, 2017 •
edited
Loading

What changes were proposed in this pull request?

After wiring SQLConf in logical plan (PR 18299), we can remove the need of passing conf into def stats and def computeStats.

How was this patch tested?

Covered by existing tests, plus some modified existing tests.

@wzhfy
Copy link
Copy Markdown
Contributor Author

wzhfy commented Jun 22, 2017

cc @rxin

kiszk
kiszk reviewed Jun 22, 2017
View reviewed changes
Comment thread sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LocalRelation.scala
}

override def computeStats(conf: SQLConf): Statistics =
override def computeStats: Statistics =
Copy link
Copy Markdown
Member

@kiszk kiszk Jun 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we remove import org.apache.spark.sql.internal.SQLConf, too?

Copy link
Copy Markdown
Contributor Author

@wzhfy wzhfy Jun 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked all modified files and removed unused imports. Thanks!

kiszk
kiszk reviewed Jun 22, 2017
View reviewed changes
Comment thread sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
))

override def computeStats(conf: SQLConf): Statistics = {
override def computeStats: Statistics = {
Copy link
Copy Markdown
Member

@kiszk kiszk Jun 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we remove import org.apache.spark.sql.internal.SQLConf, too?

kiszk
kiszk reviewed Jun 22, 2017
View reviewed changes
Comment thread sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala
*/
final def stats(conf: SQLConf): Statistics = statsCache.getOrElse {
statsCache = Some(computeStats(conf))
final def stats: Statistics = statsCache.getOrElse {
Copy link
Copy Markdown
Member

@kiszk kiszk Jun 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

kiszk
kiszk reviewed Jun 22, 2017
View reviewed changes
Comment thread sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/hints.scala

override def computeStats(conf: SQLConf): Statistics = {
val stats = child.stats(conf)
override def computeStats: Statistics = {
Copy link
Copy Markdown
Member

@kiszk kiszk Jun 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

@SparkQA
Copy link
Copy Markdown

SparkQA commented Jun 22, 2017

Test build #78459 has finished for PR 18391 at commit 5999539.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • case class FilterEstimation(plan: Filter) extends Logging
  • case class InnerOuterEstimation(join: Join) extends Logging
  • case class LeftSemiAntiEstimation(join: Join)

@kiszk
Copy link
Copy Markdown
Member

kiszk commented Jun 23, 2017

thanks, SGTM

@SparkQA
Copy link
Copy Markdown

SparkQA commented Jun 23, 2017

Test build #78496 has finished for PR 18391 at commit b6025b4.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

gatorsmile
gatorsmile reviewed Jun 23, 2017
View reviewed changes
Comment thread sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
))

override def computeStats(conf: SQLConf): Statistics = {
override def computeStats: Statistics = {
Copy link
Copy Markdown
Member

@gatorsmile gatorsmile Jun 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this file's import org.apache.spark.sql.internal.SQLConf need to be removed

Copy link
Copy Markdown
Member

@kiszk kiszk Jun 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I overlooked.

Copy link
Copy Markdown
Contributor Author

@wzhfy wzhfy Jun 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I checked the other interface file, there are several files with this name...

gatorsmile
gatorsmile reviewed Jun 23, 2017
View reviewed changes
Comment thread ...src/test/scala/org/apache/spark/sql/catalyst/statsEstimation/BasicStatsEstimationSuite.scala Outdated
SQLConf.get.setConf(SQLConf.CBO_ENABLED, false)
assert(plan.stats == expectedStatsCboOff)
} finally {
SQLConf.get.unsetConf(SQLConf.CBO_ENABLED)
Copy link
Copy Markdown
Member

@gatorsmile gatorsmile Jun 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

first, get the original and then recover the original one in the finally block. You can check how we did it in SQLConfSuite

gatorsmile
gatorsmile reviewed Jun 23, 2017
View reviewed changes
Comment thread .../src/test/scala/org/apache/spark/sql/catalyst/statsEstimation/AggregateEstimationSuite.scala Outdated
// From UnaryNode.computeStats, childSize * outputRowSize / childRowSize
Statistics(sizeInBytes = 48 * (8 + 4 + 8) / (8 + 4)))
} finally {
SQLConf.get.unsetConf(SQLConf.CBO_ENABLED)
Copy link
Copy Markdown
Member

@gatorsmile gatorsmile Jun 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same here.

gatorsmile
gatorsmile reviewed Jun 23, 2017
View reviewed changes
Comment thread ...t/src/test/scala/org/apache/spark/sql/catalyst/statsEstimation/StatsEstimationTestBase.scala Outdated
}

override def afterAll(): Unit = {
SQLConf.get.unsetConf(SQLConf.CBO_ENABLED)
Copy link
Copy Markdown
Member

@gatorsmile gatorsmile Jun 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same here.

@SparkQA
Copy link
Copy Markdown

SparkQA commented Jun 23, 2017

Test build #78521 has finished for PR 18391 at commit cd9f424.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@gatorsmile
Copy link
Copy Markdown
Member

gatorsmile commented Jun 23, 2017

LGTM

@gatorsmile
Copy link
Copy Markdown
Member

gatorsmile commented Jun 23, 2017

Thanks! Merging to master.

@asfgit asfgit closed this in b803b66 Jun 23, 2017
robert3005 pushed a commit to palantir/spark that referenced this pull request Jun 29, 2017
[SPARK-21180][SQL] Remove conf from stats functions since now we have…
a6296bf 
… conf in LogicalPlan

## What changes were proposed in this pull request?

After wiring `SQLConf` in logical plan ([PR 18299](apache#18299)), we can remove the need of passing `conf` into `def stats` and `def computeStats`.

## How was this patch tested?

Covered by existing tests, plus some modified existing tests.

Author: wangzhenhua <wangzhenhua@huawei.com>
Author: Zhenhua Wang <wzh_zju@163.com>

Closes apache#18391 from wzhfy/removeConf.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kiszk kiszk kiszk left review comments

@gatorsmile gatorsmile gatorsmile left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@wzhfy @SparkQA @kiszk @gatorsmile