[AIRFLOW-1571] Add AWS Lambda Hook

[sid88in]

Dear Airflow maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

• My PR addresses the following Airflow JIRA issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
  • https://issues.apache.org/jira/browse/AIRFLOW-1571

Description

• Here are some details about my PR, no UI changes

Details of this PR are in the Jira above.

Tests

• My PR adds the following unit tests

1. test_aws_lambda_hook
   This unit test uses moto to mock aws lambda. Firstly, we create a lambda function (assuming this function is already created by aws cloudformation template in production). Secondly, we trigger lambda function using invoke function in the hook. Lastly, we parse the response and assert statements.

Commits

• My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

[codecov-io]

Codecov Report

Merging #2718 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #2718   +/-   ##
=======================================
  Coverage   72.38%   72.38%           
=======================================
  Files         154      154           
  Lines       11815    11815           
=======================================
  Hits         8552     8552           
  Misses       3263     3263

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1ef2b6c...7feefcf. Read the comment docs.

[andyxhadji]

@ reviewer - As part of a larger design decision, how should we think about naming hooks extending the AwsHook base class?

e.g. here we name the file "AwsLambda", but elsewhere we call it by the service name without the "Aws" prefix - "S3Hook" instead of "AwsS3Hook". This is also relevant for the RedshiftHook PR (#2717): "AwsRedshiftHook" vs. "RedshiftHook".

Just want to make sure our naming is consistent!

[sid88in]

@andyxhadji I concur with that. For aws related hooks/operators we should have consistent naming convention. For instance, google hooks start with gcp_* filename and GoogleCloudHookname For the same reason for dynamodb I merged aws_dynamodb_hook and for Kinesis #2586 its aws_kinesis_hook filenames and AwsHookname.

[andyxhadji]

Sounds good to me, I'll make a JIRA issue to address naming inconsistencies with current Aws hook extensions in order to follow that practice.

[sid88in]

@bolkedebruin would be great if you could please help review this?

[bolkedebruin]

Merged. Another idea is to merge all Aws hooks into one file, it is getting a bit messy now. Operators maybe as well.

[sid88in]

yay thanks @bolkedebruin !! Yes I agree that we should combine aws related hooks in 1 file. Will create a new PR to add aws related hooks (EMR, DynamoDB and Lambda) in aws_hook.py.. once thats done, I am going to decline my PR #2586 and submit a fresh PR with the hook. Once we start adding more operators we can take similar approach. thanks again

[andyxhadji]

Would having them all in one file be too lengthy? It also would be inconsistent with the importing structure we have now (e.g. from airflow.....S3_hook import S3Hook). Perhaps it makes sense to put them in the same package? Is this general file organization something we'd want to do with gcp as well (there seems to be more hooks for gcp than aws)?

…

On Thu, Nov 2, 2017 at 8:11 AM, Siddharth Gupta ***@***.***> wrote: yay thanks @bolkedebruin <https://github.com/bolkedebruin> !! Yes I agree that we should combine aws related hooks in 1 file. Will create a new PR to add aws related hooks (EMR, DynamoDB and Lambda) in aws_hook.py.. once thats done, I am going to decline my PR #2586 <#2586> and submit a fresh PR with the hook. Once we start adding more operators we can take similar approach. thanks again — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2718 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADcMX5JXWHfIrPrv_fnV6GZJJPtVjDyeks5sybFdgaJpZM4QCToP> .

[sid88in]

@andyxhadji you raise some good points. I think we should think about restructuring with @bolkedebruin .