Skip to content

Commit 27a201b

Browse files
authored
Merge pull request #65 from ONS-Innovation/KEH-1220
KEH-1220 - Add config file support
2 parents aea9ff2 + 6614e5c commit 27a201b

File tree

7 files changed

+461
-148
lines changed

7 files changed

+461
-148
lines changed

.gitignore

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -65,4 +65,8 @@ terraform.rc
6565
.vscode/
6666

6767
# Ignore MegaLinter reports
68-
megalinter-reports/
68+
megalinter-reports/
69+
70+
# Ignore locally written data
71+
output/
72+
debug.log

Makefile

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,9 @@ clean: ## Clean the temporary files.
1212
rm -rf .pytest_cache
1313
rm -rf tests/__pycache__
1414
rm -rf .coverage
15+
rm -rf megalinter-reports/
16+
rm -rf output
17+
rm -rf debug.log
1518

1619
.PHONY: black-check
1720
black-check: ## Run black for code formatting, without fixing.

README.md

Lines changed: 114 additions & 114 deletions
Original file line numberDiff line numberDiff line change
@@ -14,16 +14,6 @@ The Copilot dashboard can be found on the Copilot tab within the Digital Landsca
1414
- [Table of Contents](#table-of-contents)
1515
- [Prerequisites](#prerequisites)
1616
- [Makefile](#makefile)
17-
- [Documentation](#documentation)
18-
- [Testing](#testing)
19-
- [Linting](#linting)
20-
- [Python](#python)
21-
- [Markdown](#markdown)
22-
- [Markdown Configuration](#markdown-configuration)
23-
- [Markdown GitHub Action](#markdown-github-action)
24-
- [Megalinter](#megalinter)
25-
- [Megalinter Configuration](#megalinter-configuration)
26-
- [Megalinter GitHub Action](#megalinter-github-action)
2717
- [AWS Lambda Scripts](#aws-lambda-scripts)
2818
- [Setup - Running in a container](#setup---running-in-a-container)
2919
- [Setup - running outside of a Container (Development only)](#setup---running-outside-of-a-container-development-only)
@@ -40,6 +30,16 @@ The Copilot dashboard can be found on the Copilot tab within the Digital Landsca
4030
- [Allowlisting your IP](#allowlisting-your-ip)
4131
- [Setting up a pipeline](#setting-up-a-pipeline)
4232
- [Triggering a pipeline](#triggering-a-pipeline)
33+
- [Documentation](#documentation)
34+
- [Testing](#testing)
35+
- [Linting](#linting)
36+
- [Python](#python)
37+
- [Markdown](#markdown)
38+
- [Markdown Configuration](#markdown-configuration)
39+
- [Markdown GitHub Action](#markdown-github-action)
40+
- [Megalinter](#megalinter)
41+
- [Megalinter Configuration](#megalinter-configuration)
42+
- [Megalinter GitHub Action](#megalinter-github-action)
4343

4444
## Prerequisites
4545

@@ -62,110 +62,6 @@ This repository has a Makefile for executing common commands. To view all comman
6262
make all
6363
```
6464

65-
## Documentation
66-
67-
This project uses MkDocs for documentation which gets deployed to GitHub Pages at a repository level.
68-
69-
For more information about MkDocs, see the below documentation.
70-
71-
[Getting Started with MkDocs](https://www.mkdocs.org/getting-started/)
72-
73-
There is a guide to getting started on this repository's GitHub Pages site.
74-
75-
## Testing
76-
77-
This project uses Pytest for testing. The tests can be found in the `tests` folder.
78-
79-
To run all tests, use `make test`.
80-
81-
On pull request or push to the `main` branch, the tests will automatically run. The workflow will fail if any tests fail, or if test coverage is below 95%.
82-
83-
The related workflow can be found in `.github/workflows/ci.yml`.
84-
85-
## Linting
86-
87-
### Python
88-
89-
This project uses Black, Ruff, and Pylint for linting and code formatting on Python files in `src`. Configurations for each are located in `pyproject.toml`.
90-
91-
The following Makefile commands can be used to run linting and optionally apply fixes or run a specific linter:
92-
93-
```bash
94-
black-check ## Run black for code formatting, without fixing.
95-
96-
black-apply ## Run black and fix code formatting.
97-
98-
ruff-check ## Run ruff for linting and code formatting, without fixing.
99-
100-
ruff-apply ## Run ruff and fix linting and code formatting.
101-
102-
pylint ## Run pylint for code analysis.
103-
104-
lint ## Run Python linters without fixing.
105-
106-
lint-apply ## Run black and ruff with auto-fix, and Pylint.
107-
```
108-
109-
On pull request or push to the `main` branch, `make lint-check` will automatically run to check code quality, failing if there are any issues. It is up to the developer to apply fixes.
110-
111-
The related workflow can be found in `.github/workflows/ci.yml`.
112-
113-
### Markdown
114-
115-
Markdown linting runs in a docker image, so docker must be running before attempting to lint.
116-
117-
To lint all markdown files, run the following command:
118-
119-
```bash
120-
make md-check
121-
```
122-
123-
To fix all markdown files, run the following command:
124-
125-
```bash
126-
make md-apply
127-
```
128-
129-
#### Markdown Configuration
130-
131-
The `.markdownlint.json` file in the root of the repository contains the configuration for markdownlint. This file is used to set the rules and settings for linting markdown files.
132-
133-
Currently, MD013 (line length) is disabled. This is because the default line length of 80 characters is too restrictive.
134-
135-
For a full list of rules, see [Markdownlint Rules / Aliases](https://github.com/DavidAnson/markdownlint?tab=readme-ov-file#rules--aliases)
136-
137-
The `.markdownlintignore` file in the root of the repository is also used to prevent markdownlint running on unnecessary files such as `venv`.
138-
139-
#### Markdown GitHub Action
140-
141-
On pull request or push to the `main` branch, `make md-check` will automatically run to check for linting errors, failing if there are any issues. It is up to the developer to apply fixes.
142-
143-
The related workflow can be found in `.github/workflows/ci.yml`.
144-
145-
### Megalinter
146-
147-
In addition to Python and Markdown-specific linting, this project uses Megalinter to catch all other types of linting errors across the project.
148-
149-
Megalinter runs in a docker image, so docker must be running before attempting to lint.
150-
151-
To lint with Megalinter, run:
152-
153-
```bash
154-
make megalint
155-
```
156-
157-
After running, Megalinter will create a folder named `megalinter-reports` containing detailed logs on linting.
158-
159-
#### Megalinter Configuration
160-
161-
The configuration file for Megalinter can be found in the root of the repository, named `.mega-linter.yml`.
162-
163-
#### Megalinter GitHub Action
164-
165-
On pull request or push to the `main` branch, Megalinter will automatically run to check project-wide linting, failing if there are any issues.
166-
167-
The related workflow can be found in `.github/workflows/megalinter.yml`.
168-
16965
## AWS Lambda Scripts
17066

17167
This script:
@@ -485,3 +381,107 @@ Once the pipeline has been set, you can manually trigger a build on the Concours
485381
```bash
486382
fly -t aws-sdp trigger-job -j github-copilot-usage-lambda-<branch-name>/build-and-push
487383
```
384+
385+
## Documentation
386+
387+
This project uses MkDocs for documentation which gets deployed to GitHub Pages at a repository level.
388+
389+
For more information about MkDocs, see the below documentation.
390+
391+
[Getting Started with MkDocs](https://www.mkdocs.org/getting-started/)
392+
393+
There is a guide to getting started on this repository's GitHub Pages site.
394+
395+
## Testing
396+
397+
This project uses Pytest for testing. The tests can be found in the `tests` folder.
398+
399+
To run all tests, use `make test`.
400+
401+
On pull request or push to the `main` branch, the tests will automatically run. The workflow will fail if any tests fail, or if test coverage is below 95%.
402+
403+
The related workflow can be found in `.github/workflows/ci.yml`.
404+
405+
## Linting
406+
407+
### Python
408+
409+
This project uses Black, Ruff, and Pylint for linting and code formatting on Python files in `src`. Configurations for each are located in `pyproject.toml`.
410+
411+
The following Makefile commands can be used to run linting and optionally apply fixes or run a specific linter:
412+
413+
```bash
414+
black-check ## Run black for code formatting, without fixing.
415+
416+
black-apply ## Run black and fix code formatting.
417+
418+
ruff-check ## Run ruff for linting and code formatting, without fixing.
419+
420+
ruff-apply ## Run ruff and fix linting and code formatting.
421+
422+
pylint ## Run pylint for code analysis.
423+
424+
lint ## Run Python linters without fixing.
425+
426+
lint-apply ## Run black and ruff with auto-fix, and Pylint.
427+
```
428+
429+
On pull request or push to the `main` branch, `make lint-check` will automatically run to check code quality, failing if there are any issues. It is up to the developer to apply fixes.
430+
431+
The related workflow can be found in `.github/workflows/ci.yml`.
432+
433+
### Markdown
434+
435+
Markdown linting runs in a docker image, so docker must be running before attempting to lint.
436+
437+
To lint all markdown files, run the following command:
438+
439+
```bash
440+
make md-check
441+
```
442+
443+
To fix all markdown files, run the following command:
444+
445+
```bash
446+
make md-apply
447+
```
448+
449+
#### Markdown Configuration
450+
451+
The `.markdownlint.json` file in the root of the repository contains the configuration for markdownlint. This file is used to set the rules and settings for linting markdown files.
452+
453+
Currently, MD013 (line length) is disabled. This is because the default line length of 80 characters is too restrictive.
454+
455+
For a full list of rules, see [Markdownlint Rules / Aliases](https://github.com/DavidAnson/markdownlint?tab=readme-ov-file#rules--aliases)
456+
457+
The `.markdownlintignore` file in the root of the repository is also used to prevent markdownlint running on unnecessary files such as `venv`.
458+
459+
#### Markdown GitHub Action
460+
461+
On pull request or push to the `main` branch, `make md-check` will automatically run to check for linting errors, failing if there are any issues. It is up to the developer to apply fixes.
462+
463+
The related workflow can be found in `.github/workflows/ci.yml`.
464+
465+
### Megalinter
466+
467+
In addition to Python and Markdown-specific linting, this project uses Megalinter to catch all other types of linting errors across the project.
468+
469+
Megalinter runs in a docker image, so docker must be running before attempting to lint.
470+
471+
To lint with Megalinter, run:
472+
473+
```bash
474+
make megalint
475+
```
476+
477+
After running, Megalinter will create a folder named `megalinter-reports` containing detailed logs on linting.
478+
479+
#### Megalinter Configuration
480+
481+
The configuration file for Megalinter can be found in the root of the repository, named `.mega-linter.yml`.
482+
483+
#### Megalinter GitHub Action
484+
485+
On pull request or push to the `main` branch, Megalinter will automatically run to check project-wide linting, failing if there are any issues.
486+
487+
The related workflow can be found in `.github/workflows/megalinter.yml`.

config/config.json

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"features": {
3+
"show_log_locally": false,
4+
"write_data_locally": false
5+
}
6+
}

docs/configuration.md

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
# Configuration
2+
3+
The Copilot lambda uses a local configuration file to manage its settings, located within `./config/config.json`.
4+
5+
## `config.json`
6+
7+
The `config.json` file contains the following:
8+
9+
```json
10+
{
11+
"features": {
12+
"show_log_locally": false,
13+
"write_data_locally": false
14+
},
15+
}
16+
```
17+
18+
### `features` Section
19+
20+
This section contains feature flags that control which of the tool's features are enabled or disabled.
21+
22+
#### `show_log_locally`
23+
24+
If set to `true`, the tool will output logs to the terminal.
25+
26+
When deploying to AWS, this should be set to `false` to avoid files being written to the local filesystem.
27+
28+
#### `write_data_locally`
29+
30+
If set to `true`, the tool will skip writing to the appropriate AWS S3 bucket and instead write data for copilot teams, historic usage, and teams history to `local_data`.
31+
32+
**When deploying to AWS, this must be set to `false` to ensure the tool writes to AWS.**
33+
34+
When debugging locally, you can set this to `true` to use the local configuration file. This is useful if you need to see the logs locally, without affecting the cloud deployment.
35+
36+
### Example During Local Testing
37+
38+
When testing locally, you might set the `config.json` file as follows:
39+
40+
```json
41+
{
42+
"features": {
43+
"show_log_locally": true,
44+
"write_data_locally": true
45+
},
46+
}
47+
```
48+
49+
### Example On AWS
50+
51+
When deploying to AWS, the `config.json` file should be set as follows:
52+
53+
```json
54+
{
55+
"features": {
56+
"show_log_locally": false,
57+
"write_data_locally": false
58+
},
59+
}
60+
```
61+
62+
This configuration ensures that the tool does not log or write data locally.
63+
64+
**It is essential that `write_data_locally` is set to `false` when deploying to AWS.**

0 commit comments

Comments
 (0)