From 4a647e20097357b3db6df972cd0c950d62565d29 Mon Sep 17 00:00:00 2001 From: Raymond Jacobson Date: Fri, 19 Apr 2024 00:12:01 -0700 Subject: [PATCH 1/5] Add aws config option to DDEX README --- packages/ddex/README.md | 63 +++++++++++++++++++++------- packages/ddex/ingester/README.md | 2 +- packages/ddex/publisher/Dockerfile | 2 +- packages/ddex/webapp/Dockerfile.fast | 55 ------------------------ 4 files changed, 50 insertions(+), 72 deletions(-) delete mode 100644 packages/ddex/webapp/Dockerfile.fast diff --git a/packages/ddex/README.md b/packages/ddex/README.md index 6759761223b..b4a22968a04 100644 --- a/packages/ddex/README.md +++ b/packages/ddex/README.md @@ -5,12 +5,11 @@ Processes and uploads DDEX releases to Audius. ## Production setup Use audius-docker-compose to run a production DDEX instance. After you've installed audius-docker-compose, set the following required environment variables in override.env (in the audius-docker-compose repository, not here). -### Glossary +### Creating a bucket in S3 * `env` refers to `dev`, `staging`, or `prod` * `provider` refers to the name of the label/distributor/provider -### Creating a bucket in S3 1. Create a new bucket in the S3 console with the name `ddex---raw`. Use all the defaults, including "ACLs disabled" 2. Do the same for a bucket named `ddex---crawled`. Use all the defaults, including "ACLs disabled" @@ -125,37 +124,71 @@ For docker compose to work: `cat packages/ddex/.env >> dev-tools/compose/.env` ### One-time setup 1. `audius-compose connect` to update your `/etc/hosts` -2. Install the AWS cli and configure it for local dev: - ```sh - pip install awscli && \ - aws configure set aws_access_key_id test && \ - aws configure set aws_secret_access_key test && \ - aws configure set region us-west-2 - ``` +2. Install the AWS cli and configure a profile for local dev: + +```bash +pip install awscli + +aws configure --profile local +# enter these details +# AWS Access Key ID [None]: test +# AWS Secret Access Key [None]: test +# Default region name [None]: us-west-2 +# Default output format [None]: json +``` + +edit `~/.aws/config` and add +``` +[profile local] +region = us-west-2 +endpoint_url = http://ingress:4566 +``` + +To use the created profile, run: +```bash +export AWS_PROFILE=local +``` +You may also pass `--profile local` to all aws commands instead. + 3. To use the DDEX webapp as an admin, add your decoded staging user ID to `extra-env.DDEX_ADMIN_ALLOWLIST` in `../../dev-tools/config.json` - Find your user ID by going to `https://discoveryprovider.staging.audius.co/v1/full/users/handle/`, searching for `id`, and then decoding it by pasting it into the "Encoded" textbox [here](https://healthz.audius.co/#/utils/id) and copying the "Integer" value - - Note that this requires a restart if the app is already running (`audius-compose down && audius-compose up -ddex-[release-by-release|batched]`) + - Note that this requires a restart if the app is already running (`audius-compose down && audius-compose up --ddex-[release-by-release|batched]`) ### Bring up the ddex stack locally Run `audius-compose up --ddex-release-by-release` (or `audius-compose up --ddex-batched` -- see "Choreography" in Glossary below), and navigate to `http://localhost:9000` to view the DDEX webapp To upload a delivery to be processed: - 1. Create buckets: `aws --endpoint=http://ingress:4566 s3 mb s3://audius-test-raw && aws --endpoint=http://ingress:4566 s3 mb s3://audius-test-crawled` - 2. Upload your file: `aws --endpoint=http://ingress:4566 s3 cp s3://audius-test-raw`. Example: `aws --endpoint=http://ingress:4566 s3 cp ./ingester/e2e_test/fixtures/release_by_release/ern381/sony1.zip s3://audius-test-raw` + 1. Create buckets +```bash +aws s3 mb s3://audius-test-raw +aws s3 mb s3://audius-test-crawled +``` + + 2. Upload your file +```bash +aws s3 cp s3://audius-test-raw +# e.g. +# aws s3 cp ./ingester/e2e_test/fixtures/release_by_release/ern381/sony1.zip s3://audius-test-raw +``` + 3. Watch the UI (localhost:9000) for the delivery to be crawled in a few seconds -To access the ddex db via the mongo shell: `docker exec -it ddex-mongo mongosh -u mongo -p mongo --authenticationDatabase admin`, and then `use ddex`. +To access the ddex db via the mongo shell: +```bash +docker exec -it ddex-mongo mongosh -u mongo -p mongo --authenticationDatabase admin +> use ddex +``` ### Develop with hot reloading Each service can be run independently as long as `ddex-mongo` is up (from `audius-compose up --ddex-[release-by-release|batched]` and then optionally stopping individual services). See the respective subdirectories' READMEs. ### Running / debugging the e2e test * Run `audius-compose test down && audius-compose test run ddex-e2e-release-by-release` to start the ddex stack and run the e2e test for the Release-By-Release choreography. Or run `audius-compose test run ddex-e2e-batched` to run the e2e test for the Batched choreography. -* To debug S3, follow the onte-time setup instructions above to update your `/etc/hosts` and install the AWS cli. Then you can run `aws --endpoint=http://localhost:4566 s3 ls` and other commands to debug the S3 state. +* To debug S3, follow the onte-time setup instructions above to update your `/etc/hosts` and install the AWS cli. Then you can run `aws s3 ls` and other commands to debug the S3 state. ## App architecture and flows -1. A distributor uploads a ZIP file to the "raw" AWS S3 bucket. +1. A distributor either uploads a ZIP file to the "raw" AWS S3 bucket or flat files directly. 2. The Crawler periodically checks this bucket for new uploads. It downloads+unzips the file and crawls it for one or more "releases" (ie, metadata and assets for a track -- or collection of tracks -- to upload to Audius). The assets are uploaded to the "crawled" AWS S3 bucket, and metadata is stored in MongoDB. 3. The Parser app watches for new releases and processes each one into a format that the Publisher app can use to upload to Audius. 4. When the release date is reached for a release, the Publisher app uploads the release to Audius. diff --git a/packages/ddex/ingester/README.md b/packages/ddex/ingester/README.md index d337092d5a9..8e058783260 100644 --- a/packages/ddex/ingester/README.md +++ b/packages/ddex/ingester/README.md @@ -9,4 +9,4 @@ The easiest way to test DDEX locally is via `audius-compose up --ddex-[release-b 1. Make sure the DDEX stack is running. See `packages/ddex/README.md` for instructions on how to bring up the DDEX stack locally. 2. `docker stop ddex-crawler` or `docker stop ddex-parser` (assuming it's running as part of the whole DDEX stack) -3. `IS_DEV=true AWS_ENDPOINT=http://ingress:4566 DDEX_CHOREOGRAPHY=ERNReleaseByRelease air -c .air.toml -- --service [crawler|parser]` +3. `IS_DEV=true DDEX_CHOREOGRAPHY=ERNReleaseByRelease air -c .air.toml -- --service [crawler|parser]` diff --git a/packages/ddex/publisher/Dockerfile b/packages/ddex/publisher/Dockerfile index f9ebd5f66ff..268d23880df 100644 --- a/packages/ddex/publisher/Dockerfile +++ b/packages/ddex/publisher/Dockerfile @@ -37,7 +37,7 @@ RUN CI=true npm i --maxsockets 1 COPY --from=turbo-builder /app/out/full/ . COPY svgr-template.js svgr-template.js COPY turbo.json turbo.json -RUN npx turbo run build --filter=@audius/ddex-publisher --filter=@audius/ddex-publisher +RUN npx turbo run build --filter=@audius/ddex-publisher # Make a smaller image by removing all src directories (except for in node_modules) RUN find packages -path '*/node_modules/*' -prune -o -name 'src' -type d -exec rm -rf {} + diff --git a/packages/ddex/webapp/Dockerfile.fast b/packages/ddex/webapp/Dockerfile.fast deleted file mode 100644 index 56ea78dd5ba..00000000000 --- a/packages/ddex/webapp/Dockerfile.fast +++ /dev/null @@ -1,55 +0,0 @@ -# Attempting to build+push locally for a Linux (stage or prod) target may hang. Use this file as a workaround: -# 1. in packages/ddex/webapp/server: npm run build -# 2. in packages/ddex/webapp/client: npm run build -# 3. in docker-compose.ddex.yml, change the value of dockerfile from Dockerfile to Dockerfile.fast -# 4. run DOCKER_DEFAULT_PLATFORM=linux/amd64 audius-compose push --prod "ddex" -# 5. ssh into a stage node and edit ~/audius-docker-compose/discovery-provider/docker-compose.yml: change the ddex image to the SHA you just pushed - -FROM node:18-alpine AS base - -ARG TURBO_TEAM -ENV TURBO_TEAM=$TURBO_TEAM - -ARG TURBO_TOKEN -ENV TURBO_TOKEN=$TURBO_TOKEN - -FROM base as builder - -RUN apk add --no-cache libc6-compat -RUN apk update - -WORKDIR /app -RUN npm install turbo --global - -COPY . . -RUN turbo prune --scope=@audius/ddex-webapp-server --docker - -# Install and build backend dist and run app -FROM base AS runner - -WORKDIR /app - -RUN apk add --no-cache python3 py3-pip make g++ curl bash libc6-compat -RUN apk update - -# First install dependencies (as they change less often) -COPY .gitignore .gitignore -COPY --from=builder /app/out/json/ . -COPY --from=builder /app/out/package-lock.json ./package-lock.json -COPY --from=builder /app/scripts ./scripts - -RUN CI=true npm i - -# Build the app and its dependencies -COPY --from=builder /app/out/full/ . -COPY svgr-template.js svgr-template.js -COPY turbo.json turbo.json -RUN npx turbo run build --filter=@audius/ddex-webapp-server - -# Copy the client dist from local filesystem instead of building it in Docker, which hangs -COPY packages/ddex/webapp/server/dist /app/packages/ddex/webapp/server/public - -WORKDIR /app/packages/ddex/webapp/server - -EXPOSE 9000 -CMD ["npm", "run", "start:prod"] From cef699b2dbae4759ff44c6906d7c1dc3c9ad2602 Mon Sep 17 00:00:00 2001 From: Raymond Jacobson Date: Fri, 19 Apr 2024 12:59:47 -0700 Subject: [PATCH 2/5] Improve git secrets and ignore ddex readme --- dev-tools/config.json | 2 +- package.json | 1 + scripts/check_secrets.sh | 21 --------------------- scripts/install-git-secrets.sh | 22 ++++++++++++++++++++++ scripts/postinstall.sh | 5 +++++ 5 files changed, 29 insertions(+), 22 deletions(-) create mode 100644 scripts/install-git-secrets.sh diff --git a/dev-tools/config.json b/dev-tools/config.json index 34bfec8de76..8555b731cbf 100644 --- a/dev-tools/config.json +++ b/dev-tools/config.json @@ -138,7 +138,7 @@ "DDEX_KEY": "49d5e13d355709b615b7cce7369174fb240b6b39", "DDEX_SECRET": "2b2c2b90d9a489234ae629a5284de84fb0633306257f17667aaebf2345d92152", "SESSION_SECRET": "something random", - "DDEX_ADMIN_ALLOWLIST": "127559427", + "DDEX_ADMIN_ALLOWLIST": "469591815", "NODE_ENV": "stage" } } diff --git a/package.json b/package.json index 860bf5f1eaa..f061dc5e8a9 100644 --- a/package.json +++ b/package.json @@ -19,6 +19,7 @@ "check-secrets": "sh ./scripts/check_secrets.sh", "clean:modules": "rm -rf node_modules && rm -rf packages/*/node_modules", "clean:dist": "rm -rf packages/*/dist", + "install-git-secrets": "sh ./scripts/install-git-secrets.sh", "install-hooks": "mookme init --only-hook --skip-types-selection", "lint:fix": "turbo run lint:fix", "lint": "turbo run lint", diff --git a/scripts/check_secrets.sh b/scripts/check_secrets.sh index db2bb25f452..1c01a490328 100644 --- a/scripts/check_secrets.sh +++ b/scripts/check_secrets.sh @@ -1,23 +1,2 @@ -# used to exercise the `if command is not git command` logic below. change this to an invalid value to test printing out install message -COMMAND_PREFIX='secrets' - -if ! git secrets > /dev/null 2>&1; then - echo "Please install 'git-secrets' from https://github.com/awslabs/git-secrets and run 'git secrets --install' in the audius-protocol/ repo" - exit 1 -fi - -# git secrets --install # this needs to be run once right after install -git secrets --register-aws - -# these are the match rules to add. by default the aws rules only match AWS access id's and keys -# add any additional match strings here -# rds urls -git secrets --add '.*[a-z0-9]*.rds.amazonaws.com:[0-9]*\/.*' - -# match any postgres db with an IP hostname -git secrets --add 'postgres:\/\/.*\:.*@([0-9]*\.?)*:[0-9]{4}\/.*' -git secrets --add --allowed 'postgres:postgres@1\.2\.3\.4:[0-9]{4}\/.*' - # scan the staged files git secrets --scan --cached $(git diff --cached --name-only) - diff --git a/scripts/install-git-secrets.sh b/scripts/install-git-secrets.sh new file mode 100644 index 00000000000..a4ab3b919d9 --- /dev/null +++ b/scripts/install-git-secrets.sh @@ -0,0 +1,22 @@ +# used to exercise the `if command is not git command` logic below. change this to an invalid value to test printing out install message +COMMAND_PREFIX='secrets' + +if ! git secrets > /dev/null 2>&1; then + echo "Please install 'git-secrets' from https://github.com/awslabs/git-secrets and run 'git secrets --install' in the audius-protocol/ repo" + exit 1 +fi + +git secrets --add --allowed 'dev-tools/config.json:.*' +git secrets --add --allowed 'scripts/install-git-secrets.sh' +git secrets --add --allowed 'package.json' + +git secrets --register-aws +# these are the match rules to add. by default the aws rules only match AWS access id's and keys +# add any additional match strings here +# rds urls +git secrets --add '.*[a-z0-9]*.rds.amazonaws.com:[0-9]*\/.*' +git secrets --add --allowed 'packages/ddex/.*/README.md:.*' + +# match any postgres db with an IP hostname +git secrets --add 'postgres:\/\/.*\:.*@([0-9]*\.?)*:[0-9]{4}\/.*' +git secrets --add --allowed 'postgres:postgres@1\.2\.3\.4:[0-9]{4}\/.*' diff --git a/scripts/postinstall.sh b/scripts/postinstall.sh index 97b760a2516..e193c1eb333 100755 --- a/scripts/postinstall.sh +++ b/scripts/postinstall.sh @@ -10,6 +10,11 @@ if [[ -z "${CI}" ]]; then npm run install-hooks > /dev/null fi +if [[ -z "${CI}" ]]; then + printf "${GREEN}Installing git secrets...\n${NC}" + npm run install-git-secrets > /dev/null +fi + printf "${GREEN}Applying patches...\n${NC}" npm run patch-package > /dev/null From 56038c761a2a1edb3ce69c1c2852e2a1fc3fae80 Mon Sep 17 00:00:00 2001 From: Raymond Jacobson Date: Fri, 19 Apr 2024 13:00:38 -0700 Subject: [PATCH 3/5] Revert config change --- dev-tools/config.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dev-tools/config.json b/dev-tools/config.json index 8555b731cbf..34bfec8de76 100644 --- a/dev-tools/config.json +++ b/dev-tools/config.json @@ -138,7 +138,7 @@ "DDEX_KEY": "49d5e13d355709b615b7cce7369174fb240b6b39", "DDEX_SECRET": "2b2c2b90d9a489234ae629a5284de84fb0633306257f17667aaebf2345d92152", "SESSION_SECRET": "something random", - "DDEX_ADMIN_ALLOWLIST": "469591815", + "DDEX_ADMIN_ALLOWLIST": "127559427", "NODE_ENV": "stage" } } From 438889705db701f8006adf3e35e28970bb453d91 Mon Sep 17 00:00:00 2001 From: Raymond Jacobson Date: Fri, 19 Apr 2024 13:02:10 -0700 Subject: [PATCH 4/5] Clean up --- scripts/install-git-secrets.sh | 2 +- scripts/postinstall.sh | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/scripts/install-git-secrets.sh b/scripts/install-git-secrets.sh index a4ab3b919d9..a514025b09e 100644 --- a/scripts/install-git-secrets.sh +++ b/scripts/install-git-secrets.sh @@ -6,7 +6,7 @@ if ! git secrets > /dev/null 2>&1; then exit 1 fi -git secrets --add --allowed 'dev-tools/config.json:.*' +git secrets --add --allowed 'dev-tools/config.json' git secrets --add --allowed 'scripts/install-git-secrets.sh' git secrets --add --allowed 'package.json' diff --git a/scripts/postinstall.sh b/scripts/postinstall.sh index e193c1eb333..7cb6776d914 100755 --- a/scripts/postinstall.sh +++ b/scripts/postinstall.sh @@ -6,12 +6,12 @@ NC='\033[0m' if [[ -z "${CI}" ]]; then - printf "${GREEN}Installing git hooks...\n${NC}" + printf "${GREEN}Updating git hooks...\n${NC}" npm run install-hooks > /dev/null fi if [[ -z "${CI}" ]]; then - printf "${GREEN}Installing git secrets...\n${NC}" + printf "${GREEN}Updating git secrets...\n${NC}" npm run install-git-secrets > /dev/null fi From 43708e2100da328e8210826c3cb83e929b9783da Mon Sep 17 00:00:00 2001 From: Raymond Jacobson Date: Fri, 19 Apr 2024 17:33:37 -0700 Subject: [PATCH 5/5] Revert ingester readme --- packages/ddex/ingester/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/ddex/ingester/README.md b/packages/ddex/ingester/README.md index 8e058783260..d337092d5a9 100644 --- a/packages/ddex/ingester/README.md +++ b/packages/ddex/ingester/README.md @@ -9,4 +9,4 @@ The easiest way to test DDEX locally is via `audius-compose up --ddex-[release-b 1. Make sure the DDEX stack is running. See `packages/ddex/README.md` for instructions on how to bring up the DDEX stack locally. 2. `docker stop ddex-crawler` or `docker stop ddex-parser` (assuming it's running as part of the whole DDEX stack) -3. `IS_DEV=true DDEX_CHOREOGRAPHY=ERNReleaseByRelease air -c .air.toml -- --service [crawler|parser]` +3. `IS_DEV=true AWS_ENDPOINT=http://ingress:4566 DDEX_CHOREOGRAPHY=ERNReleaseByRelease air -c .air.toml -- --service [crawler|parser]`