Skip to content

Adding property files option in the Spark Submit command #34894

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ghostp13409 wants to merge 6 commits into from
Closed

Adding property files option in the Spark Submit command #34894

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
4f115d0
property-files command added to SparkSubmitOperator
ghostp13409 Oct 12, 2023
99f31cc
34838- property-files command added to SparkSubmitOperator
ghostp13409 Oct 12, 2023
7960932
Merge branch 'apache:main' into 34838-Adding-propertyfiles-option
ghostp13409 Oct 12, 2023
1b3641a
Merge branch 'apache:main' into 34838-Adding-propertyfiles-option
ghostp13409 Oct 13, 2023
1ddae96
reverted previous changes and added property files option in the Spar…
ghostp13409 Oct 13, 2023
b53a091
Merge branch 'apache:main' into 34838-Adding-propertyfiles-option
ghostp13409 Oct 14, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions airflow/providers/apache/spark/hooks/spark_submit.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,7 @@ def get_ui_field_behaviour() -> dict[str, Any]:
def __init__(
self,
conf: dict[str, Any] | None = None,
properties_file: str | None = None,
conn_id: str = "spark_default",
files: str | None = None,
py_files: str | None = None,
Expand Down Expand Up @@ -123,6 +124,7 @@ def __init__(
) -> None:
super().__init__()
self._conf = conf or {}
self._properties_file = properties_file
self._conn_id = conn_id
self._files = files
self._py_files = py_files
Expand Down Expand Up @@ -287,6 +289,8 @@ def _build_spark_submit_command(self, application: str) -> list[str]:
"--conf",
f"spark.kubernetes.namespace={self._connection['namespace']}",
]
if self._properties_file:
connection_cmd += ["--properties-file", self._properties_file]
if self._files:
connection_cmd += ["--files", self._files]
if self._py_files:
Expand Down
4 changes: 4 additions & 0 deletions airflow/providers/apache/spark/operators/spark_submit.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ class SparkSubmitOperator(BaseOperator):
template_fields: Sequence[str] = (
"_application",
"_conf",
"_properties-file"
"_files",
"_py_files",
"_jars",
Expand All @@ -94,6 +95,7 @@ def __init__(
*,
application: str = "",
conf: dict[str, Any] | None = None,
properties_file: str | None = None,
conn_id: str = "spark_default",
files: str | None = None,
py_files: str | None = None,
Expand Down Expand Up @@ -123,6 +125,7 @@ def __init__(
super().__init__(**kwargs)
self._application = application
self._conf = conf
self._properties_file = properties_file
self._files = files
self._py_files = py_files
self._archives = archives
Expand Down Expand Up @@ -163,6 +166,7 @@ def on_kill(self) -> None:
def _get_hook(self) -> SparkSubmitHook:

@Bowrna Bowrna Oct 12, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you have to accept the property_files param value as one of the parameters that can be used when creating SparkSubmitOperator. This code doesn't involve those changes.

@ghostp13409 ghostp13409 Oct 13, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay will try to add that as well

@ghostp13409 ghostp13409 Oct 13, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added properties-file param to Operator and Hook files. However, I haven't added the default file path handling logic as it seems to be already handled by spark-submit command as per the docs. I will follow up with the test case changes in the next commit. let me know if there is any feedback or concern.

you have to accept the property_files param value as one of the parameters that can be used when creating SparkSubmitOperator. This code doesn't involve those changes.

return SparkSubmitHook(
conf=self._conf,
properties_file=self._properties_file,
conn_id=self._conn_id,
files=self._files,
py_files=self._py_files,
Expand Down