[FEAT-Lambda] Integrate Lifecycle Generation Lambda Function with S3

[martyheynbah]

The Lifecycle Generation function is deployed on the Lambda dev stack; two things need to be done next:

• 1) Add files that the LCGen lambda needs to run to S3

This lambda needs two .csv files (rel_prob_file and storm_id_prob_file) in order to run.

These files will eventually be added to S3 via an ETL process by FEATAPI, but since FEATAPI is not provisioned yet (see #621), we'll simply stick them in the S3 bucket for now. The diagram below outlines the process that should eventually be done by FEATAPI (process A):

• 2) Write the output of the lambda (Lifecycle Event Date Overview) to S3

[trietmnj]

I updated the LCGen process to write to S3.

Usage is from the base stormsim repo root:

uv run implementation-scripts/lc_generator_main.py --config data/lcgen/config_s3.json

https://github.com/trietmnj/stormsim/blob/07f95c4846040d373d4a1c251238f5aff956f01f/implementation-scripts/lc_generator_main.py

https://github.com/trietmnj/stormsim/blob/07f95c4846040d373d4a1c251238f5aff956f01f/data/lcgen/config_s3.json

EDIT:

[trietmnj]

Not sure how to review this - maybe we'll see what happens when integrating into either Lambda or the container?

[trietmnj]

local working now - instructions:

https://github.com/trietmnj/stormsim/blob/fd949bb154a3e0b55e3562aead5e872df179f1ea/lambda/lcgen/README.md

Attaching to stormsim-lcgen-lambda, stormsim-minio, stormsim-minio-setup
stormsim-lcgen-lambda  | 25 Feb 2026 16:54:16,601 [INFO] (rapid) exec '/var/runtime/bootstrap' (cwd=/var/task, handler=)
stormsim-minio         | INFO: Formatting 1st pool, 1 set(s), 1 drives per set.
stormsim-minio         | INFO: WARNING: Host local has more than 0 drives of set. A host failure will result in data becoming unavailable.
stormsim-minio         | MinIO Object Storage Server
stormsim-minio         | Copyright: 2015-2026 MinIO, Inc.
stormsim-minio         | License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html
stormsim-minio         | Version: RELEASE.2025-09-07T16-13-09Z (go1.24.6 linux/amd64)
stormsim-minio         |
stormsim-minio         | API: http://172.21.0.2:9000  http://127.0.0.1:9000
stormsim-minio         | WebUI: http://172.21.0.2:9001 http://127.0.0.1:9001
stormsim-minio         |
stormsim-minio         | Docs: https://docs.min.io
stormsim-minio         | WARN: Detected default credentials 'minioadmin:minioadmin', we recommend that you change these values with 'MINIO_ROOT_USER' and 'MINIO_ROOT_PASSWORD' environment variables
stormsim-minio-setup   | Added `local_minio` successfully.
stormsim-minio-setup   | Bucket created successfully `local_minio/stormsim`.
stormsim-minio-setup   | `/local-data/.DS_Store` -> `local_minio/stormsim/data/.DS_Store`
stormsim-minio-setup   | `/local-data/chs-files/regional-files/Atlantic_Basin_SRR_and_CRLs.csv` -> `local_minio/stormsim/data/chs-files/regional-files/Atlantic_Basin_SRR_and_CRLs.csv`
stormsim-minio-setup   | `/local-data/chs-files/regional-files/CHS-NA_Master_Track_Table.csv` -> `local_minio/stormsim/data/chs-files/regional-files/CHS-NA_Master_Track_Table.csv`
stormsim-minio-setup   | `/local-data/eurotop/pse_geometry.json` -> `local_minio/stormsim/data/eurotop/pse_geometry.json`
stormsim-minio-setup   | `/local-data/eurotop/dummy_stage_vol.csv` -> `local_minio/stormsim/data/eurotop/dummy_stage_vol.csv`
stormsim-minio-setup   | `/local-data/intermediate/.DS_Store` -> `local_minio/stormsim/data/intermediate/.DS_Store`
stormsim-minio-setup   | `/local-data/lcgen/Relative_probability_bins_Atlantic 4.csv` -> `local_minio/stormsim/data/lcgen/Relative_probability_bins_Atlantic 4.csv`
stormsim-minio-setup   | `/local-data/lcgen/config_local.json` -> `local_minio/stormsim/data/lcgen/config_local.json`
stormsim-minio-setup   | `/local-data/lcgen/config_s3.json` -> `local_minio/stormsim/data/lcgen/config_s3.json`
stormsim-minio-setup   | `/local-data/raw/.DS_Store` -> `local_minio/stormsim/data/raw/.DS_Store`
stormsim-minio-setup   | `/local-data/raw/HydroManipulator_example_Fabian/.DS_Store` -> `local_minio/stormsim/data/raw/HydroManipulator_example_Fabian/.DS_Store`
stormsim-minio-setup   | `/local-data/raw/HydroManipulator_example_Fabian/Manipulated_LCs/.DS_Store` -> `local_minio/stormsim/data/raw/HydroManipulator_example_Fabian/Manipulated_LCs/.DS_Store`
stormsim-minio-setup   | `/local-data/raw/conversion-lifecycle-generation/.DS_Store` -> `local_minio/stormsim/data/raw/conversion-lifecycle-generation/.DS_Store`
stormsim-minio-setup   | `/local-data/raw/eurotop/Deer_Island_Alternative_A_Performance_LCS_project_forcing.mat` -> `local_minio/stormsim/data/raw/eurotop/Deer_Island_Alternative_A_Performance_LCS_project_forcing.mat`
stormsim-minio-setup   | ┌────────────┬─────────────┬──────────┬──────────────┐
stormsim-minio-setup   | │ Total      │ Transferred │ Duration │ Speed        │
stormsim-minio-setup   | │ 175.40 MiB │ 175.40 MiB  │ 00m00s   │ 186.89 MiB/s │
stormsim-minio-setup   | └────────────┴─────────────┴──────────┴──────────────┘
stormsim-minio-setup exited with code 0
stormsim-lcgen-lambda  | START RequestId: 116b43b9-1342-4ea9-a6e9-916051e1dd1f Version: $LATEST
stormsim-lcgen-lambda  | 25 Feb 2026 16:54:28,297 [INFO] (rapid) INIT START(type: on-demand, phase: init)
stormsim-lcgen-lambda  | 25 Feb 2026 16:54:28,297 [INFO] (rapid) The extension's directory "/opt/extensions" does not exist, assuming no extensions to be loaded.
stormsim-lcgen-lambda  | 25 Feb 2026 16:54:28,297 [INFO] (rapid) Starting runtime without AWS_SESSION_TOKEN , Expected?: false
stormsim-lcgen-lambda  | 25 Feb 2026 16:54:28,856 [INFO] (rapid) INIT RTDONE(status: success)
stormsim-lcgen-lambda  | 25 Feb 2026 16:54:28,856 [INFO] (rapid) INIT REPORT(durationMs: 559.576000)
stormsim-lcgen-lambda  | 25 Feb 2026 16:54:28,856 [INFO] (rapid) INVOKE START(requestId: 116b43b9-1342-4ea9-a6e9-916051e1dd1f)
stormsim-lcgen-lambda  | Lambda received event: {"simulation_params": {"initialize_year": 2025, "lifecycle_duration": 5, "num_lcs": 2, "lam_target": 1.5, "min_arrival_trop_days": 7.0}, "inputs": {"use_duckdb": true, "use_s3": true, "rel_prob_file": "s3://stormsim/data/lcgen/Relative_probability_bins_Atlantic 4.csv", "storm_id_prob_file": "s3://stormsim/data/chs-files/regional-files/CHS-NA_Master_Track_Table.csv"}, "outputs": {"storage_type": "s3", "filename": "lambda_s3_results.csv", "s3_bucket": "stormsim", "s3_prefix": "outputs/lcgen"}, "s3_config": {"endpoint": "http://minio:9000", "access_key": "minioadmin", "secret_key": "minioadmin"}, "runtime": {"validate_lambda": true}}
stormsim-lcgen-lambda  | Running LC Generator...
stormsim-lcgen-lambda  | Writing output to s3://stormsim/outputs/lcgen/lambda_s3_results.csv...
stormsim-lcgen-lambda  | /var/task/fsspec/registry.py:301: UserWarning: Your installed version of s3fs is very old and known to cause
stormsim-lcgen-lambda  | severe performance issues, see also https://github.com/dask/dask/issues/10276
stormsim-lcgen-lambda  |
stormsim-lcgen-lambda  | To fix, you should specify a lower version bound on s3fs, or
stormsim-lcgen-lambda  | update the current installation.
stormsim-lcgen-lambda  |
stormsim-lcgen-lambda  |   warnings.warn(s3_msg)
stormsim-lcgen-lambda  |
stormsim-lcgen-lambda  | --- Storm Count Verification ---
stormsim-lcgen-lambda  | Target lambda:       1.5000
stormsim-lcgen-lambda  | Empirical mean:      1.3750
stormsim-lcgen-lambda  | Empirical variance:  0.2344
stormsim-lcgen-lambda  |
stormsim-lcgen-lambda  | k | empirical P(N=k)
stormsim-lcgen-lambda  |  1 | 0.6250
stormsim-lcgen-lambda  |  2 | 0.3750
stormsim-lcgen-lambda  | 25 Feb 2026 16:54:30,932 [INFO] (rapid) INVOKE RTDONE(status: success, produced bytes: 0, duration: 2075.974000ms)
stormsim-lcgen-lambda  | END RequestId: 116b43b9-1342-4ea9-a6e9-916051e1dd1f
stormsim-lcgen-lambda  | REPORT RequestId: 116b43b9-1342-4ea9-a6e9-916051e1dd1f Init Duration: 0.04 msDuration: 2635.71 ms    Billed Duration: 2636 ms        Memory Size: 3008 MB    Max Memory Used: 3008 MB

[noarevalo-bah]

@martyheynbah Repurposing old task for the new lambda function equivalent.