Skip to content

Commit 0352b52

Browse files
mnonnenmachermnonnenmacher
committed
docs: Replace the template docs with the README contents
Replace the Docusaurus template docs with the content of the README.md file. This commit just copies the content as-is, it will be further refined in the following commits. Signed-off-by: Martin Nonnenmacher <[email protected]>
1 parent 8c80e7a commit 0352b52

27 files changed

+836
-539
lines changed

‎docusaurus/docs/development.md‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
---
2+
sidebar_position: 4
3+
---
4+
5+
# Development
6+
7+
ORT is written in [Kotlin](https://kotlinlang.org/) and uses [Gradle](https://gradle.org/) as the build system, with
8+
[Kotlin script](https://docs.gradle.org/current/userguide/kotlin_dsl.html) instead of Groovy as the DSL. Please ensure
9+
to have Gradle's incubating
10+
[configuration on demand](https://docs.gradle.org/current/userguide/multi_project_configuration_and_execution.html#sec:configuration_on_demand)
11+
feature disabled as it is currently [incompatible with ORT](https://github.com/gradle/gradle/issues/4823).
12+
13+
When developing on the command line, use the committed
14+
[Gradle wrapper](https://docs.gradle.org/current/userguide/gradle_wrapper.html) to bootstrap Gradle in the configured
15+
version and execute any given tasks. The most important tasks for this project are:
16+
17+
| Task | Purpose |
18+
|-------------|-------------------------------------------------------------------|
19+
| assemble | Build the JAR artifacts for all projects |
20+
| detekt | Run static code analysis on all projects |
21+
| test | Run unit tests for all projects |
22+
| funTest | Run functional tests for all projects |
23+
| installDist | Build all projects and install the start scripts for distribution |
24+
25+
All contributions need to pass the `detekt`, `test` and `funTest` checks before they can be merged.
26+
27+
For IDE development we recommend the [IntelliJ IDEA Community Edition](https://www.jetbrains.com/idea/download/) which
28+
can directly import the Gradle build files. After cloning the project's source code recursively, simply run IDEA and use
29+
the following steps to import the project.
30+
31+
1. From the *Welcome* dialog: Select `Open`.
32+
33+
From a running IDEA instance: Select `File` > `New` > `Project from Existing Sources...`
34+
35+
2. Browse to ORT's source code directory and select either the `build.gradle.kts` or the `settings.gradle.kts` file.
36+
37+
3. In the *Open Project* dialog select `Open as Project`.
38+
39+
## Debugging
40+
41+
To set up a basic run configuration for debugging, navigate to `OrtMain.kt` in the `cli` module and look for the
42+
`fun main(args: Array<String>)` function. In the gutter next to it, a green "Play" icon should be displayed. Click on it
43+
and select `Run 'OrtMainKt'` to run the entry point, which implicitly creates a run configuration. Double-check that
44+
running ORT without any arguments will simply show the command line help in IDEA's *Run* tool window. Finally, edit the
45+
created run configuration to your needs, e.g. by adding an argument and options to run a specific ORT sub-command.
46+
47+
## Testing
48+
49+
ORT uses [Kotest](https://github.com/kotest/kotest) as the test framework. For running tests and individual test cases
50+
from the IDE, the [Kotest plugin](https://plugins.jetbrains.com/plugin/14080-kotest) needs to be installed. Afterwards,
51+
tests can be run via the green "Play" icon from the gutter as described above.
52+
53+
When running functional tests (for package managers) from the command line, ORT supports the special value "unified" for
54+
Kotest's `kotest.assertions.multi-line-diff` system property. When set like
55+
56+
```shell
57+
./gradlew -Dkotest.assertions.multi-line-diff=unified -p plugins/package-managers funTest
58+
```
59+
60+
any failing tests will show the deviation from the expected result in a unified diff format that is compatible with
61+
`git apply`. If the actual result should be taken as the new expected result, simply copy the diff from the console to
62+
the clipboard and run
63+
64+
* `wl-paste | patch -p1` (Linux with Wayland)
65+
* `xsel -b | patch -p1` (Linux with X)
66+
* `cat /dev/clipboard | patch -p1` (Windows with Git Bash)
67+
* `pbpaste | patch -p1` (macOS)
68+
69+
to apply the diff to the local Git working tree (this does not create a commit yet). After reviewing the changes, create
70+
a commit to accept the new expected result.
71+
72+
## Want to Help or have Questions?
73+
74+
All contributions are welcome. If you are interested in contributing, please read our
75+
[contributing guide](https://github.com/oss-review-toolkit/.github/blob/main/CONTRIBUTING.md), and to get quick answers
76+
to any of your questions we recommend you
77+
[join our Slack community][2].

‎docusaurus/docs/getting-started/_category_.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
position: 2
2+
label: "Getting Started"
3+
link:
4+
type: "generated-index"
5+
title: "Getting Started"
6+
description: "How to get started with using ORT."

‎docusaurus/docs/getting-started/installation.md‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
# Installation
2+
3+
## From binaries
4+
5+
Preliminary binary artifacts for ORT are currently available via
6+
[JitPack](https://jitpack.io/#oss-review-toolkit/ort). Please note that due to limitations with the JitPack build
7+
environment, the reporter is not able to create the Web App report.
8+
9+
## From sources
10+
11+
Install the following basic prerequisites:
12+
13+
* Git (any recent version will do).
14+
15+
Then clone this repository.
16+
17+
```shell
18+
git clone https://github.com/oss-review-toolkit/ort
19+
# If you intend to run tests, you have to clone the submodules too.
20+
cd ort
21+
git submodule update --init --recursive
22+
```
23+
24+
### Build using Docker
25+
26+
Install the following basic prerequisites:
27+
28+
* Docker 18.09 or later (and ensure its daemon is running).
29+
* Enable [BuildKit](https://docs.docker.com/develop/develop-images/build_enhancements/#to-enable-buildkit-builds) for
30+
Docker.
31+
32+
Change into the directory with ORT's source code and run `docker build -t ort .`. Alternatively, use the script at
33+
`scripts/docker_build.sh` which also sets the ORT version from the Git revision.
34+
35+
### Build natively
36+
37+
Install these additional prerequisites:
38+
39+
* Java Development Kit (JDK) version 11 or later; also remember to set the `JAVA_HOME` environment variable accordingly.
40+
41+
Change into the directory with ORT's source code and run `./gradlew installDist` (on the first run this will bootstrap
42+
Gradle and download all required dependencies).
43+
44+
## Basic usage
45+
46+
Depending on how ORT was installed, it can be run in the following ways:
47+
48+
* If the Docker image was built, use
49+
50+
```shell
51+
docker run ort --help
52+
```
53+
54+
You can find further hints for using ORT with Docker in the [documentation](./docs/hints-for-use-with-docker.md).
55+
56+
* If the ORT distribution was built from sources, use
57+
58+
```shell
59+
./cli/build/install/ort/bin/ort --help
60+
```
61+
62+
* If running directly from sources via Gradle, use
63+
64+
```shell
65+
./gradlew cli:run --args="--help"
66+
```
67+
68+
Note that in this case the working directory used by ORT is that of the `cli` project, not the directory `gradlew` is
69+
located in (see https://github.com/gradle/gradle/issues/6074).
70+
71+
For simplicity of the following usage examples, the above ORT invocations are unified to just `ort --help`.

‎docusaurus/docs/getting-started/system-requirements.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
# System requirements
2+
3+
ORT is being continuously used on Linux, Windows and macOS by the
4+
[core development team](https://github.com/orgs/oss-review-toolkit/people), so these operating systems are
5+
considered to be well-supported.
6+
7+
To run the ORT binaries (also see [Installation from binaries](#from-binaries)) at least Java 11 is required. Memory and
8+
CPU requirements vary depending on the size and type of project(s) to analyze / scan, but the general recommendation is
9+
to configure Java with 8 GiB of memory and to use a CPU with at least 4 cores.
10+
11+
```shell
12+
# This will give the Java Virtual Machine 8GB Memory.
13+
export JAVA_OPTS="$JAVA_OPTS -Xmx8g"
14+
```
15+
16+
If ORT requires external tools in order to analyze a project, these tools are listed by the `ort requirements` command.
17+
If a package manager is not list listed there, support for it is integrated directly into ORT and does not require any
18+
external tools to be installed.

‎docusaurus/docs/getting-started/usage.md‎

Lines changed: 210 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,210 @@
1+
# Usage
2+
3+
## Running the Tools
4+
5+
First, make sure that the locale of your system is set to `en_US.UTF-8` as using other locales might lead to issues with
6+
parsing the output of some external tools.
7+
8+
Then, let ORT check whether all required external tools are available by running
9+
10+
```shell
11+
ort requirements
12+
```
13+
14+
and install any missing tools or add compatible versions as indicated.
15+
16+
Finally, ORT tools like the *analyzer* can be run like
17+
18+
```shell
19+
ort --info analyze -f JSON -i /project -o /project/ort/analyzer
20+
```
21+
22+
Just the like top-level `ort` command, the subcommands for all tools provide a `--help` option for detailed usage help.
23+
Use it like `ort analyze --help`.
24+
25+
Please see [Getting Started](./docs/getting-started.md) for an introduction to the individual tools.
26+
27+
## Running on CI
28+
29+
A basic ORT pipeline (using the *analyzer*, *scanner* and *reporter*) can easily be run on
30+
[Jenkins CI](https://jenkins.io/) by using the [Jenkinsfile](./integrations/jenkins/Jenkinsfile) in a (declarative)
31+
[pipeline](https://jenkins.io/doc/book/pipeline/) job. Please see the [Jenkinsfile](./integrations/jenkins/Jenkinsfile)
32+
itself for documentation of the required Jenkins plugins. The job accepts various parameters that are translated to ORT
33+
command line arguments. Additionally, one can trigger a downstream job which e.g. further processes scan results. Note
34+
that it is the downstream job's responsibility to copy any artifacts it needs from the upstream job.
35+
36+
## Configuration
37+
38+
### Environment variables
39+
40+
ORT supports several environment variables that influence its behavior:
41+
42+
| Name | Default value | Purpose |
43+
|-------------------|------------------------|----------------------------------------------------------|
44+
| ORT_DATA_DIR | `~/.ort` | All data, like caches, archives, storages (read & write) |
45+
| ORT_CONFIG_DIR | `$ORT_DATA_DIR/config` | Configuration files, see below (read only) |
46+
| ORT_HTTP_USERNAME | Empty (n/a) | Generic username to use for HTTP(S) downloads |
47+
| ORT_HTTP_PASSWORD | Empty (n/a) | Generic password to use for HTTP(S) downloads |
48+
| http_proxy | Empty (n/a) | Proxy to use for HTTP downloads |
49+
| https_proxy | Empty (n/a) | Proxy to use for HTTPS downloads |
50+
51+
### Configuration files
52+
53+
ORT looks for its configuration files in the directory pointed to by the `ORT_CONFIG_DIR` environment variable. If this
54+
variable is not set, it defaults to the `config` directory below the directory pointed to by the `ORT_DATA_DIR`
55+
environment variable, which in turn defaults to the `.ort` directory below the current user's home directory.
56+
57+
The following provides an overview of the various configuration files that can be used to customize ORT behavior:
58+
59+
#### [ORT configuration file](./model/src/main/resources/reference.yml)
60+
61+
The main configuration file for the operation of ORT. This configuration is maintained by an administrator who manages
62+
the ORT instance. In contrast to the configuration files in the following, this file rarely changes once ORT is
63+
operational.
64+
65+
| Format | Scope | Default location |
66+
|--------|--------|------------------------------|
67+
| YAML | Global | `$ORT_CONFIG_DIR/config.yml` |
68+
69+
The [reference configuration file](./model/src/main/resources/reference.yml) gives a good impression about the content
70+
of the main ORT configuration file. It consists of sections related to different subcomponents of ORT. The meaning
71+
of these sections and the properties they can contain is described together with the corresponding subcomponents.
72+
73+
While the file is rather static, there are means to override configuration options for a specific run of ORT or to
74+
customize the configuration to a specific environment. The following options are supported, in order of precedence:
75+
76+
* Properties can be defined via environment variables by using the full property path as the variable name.
77+
For instance, one can override the Postgres schema by setting
78+
`ort.scanner.storages.postgres.connection.schema=test_schema`. The variable's name is case-sensitive.
79+
Some programs like Bash do not support dots in variable names. For this case, the dots can be
80+
replaced by double underscores, i.e., the above example is turned into
81+
`ort__scanner__storages__postgres__connection__schema=test_schema`.
82+
* In addition to that, one can override the values of properties on the command line using the `-P` option. The option
83+
expects a key-value pair. Again, the key must define the full path to the property to be overridden, e.g.
84+
`-P ort.scanner.storages.postgres.connection.schema=test_schema`. The `-P` option can be repeated on the command
85+
line to override multiple properties.
86+
* Properties in the configuration file can reference environment variables using the syntax `${VAR}`.
87+
This is especially useful to reference dynamic or sensitive data. As an example, the credentials for the
88+
Postgres database used as scan results storage could be defined in the `POSTGRES_USERNAME` and `POSTGRES_PASSWORD`
89+
environment variables. The configuration file can then reference these values as follows:
90+
91+
```yaml
92+
postgres:
93+
connection:
94+
url: "jdbc:postgresql://your-postgresql-server:5444/your-database"
95+
username: ${POSTGRES_USERNAME}
96+
password: ${POSTGRES_PASSWORD}
97+
```
98+
99+
To print the active configuration use:
100+
101+
```shell
102+
ort config --show-active
103+
```
104+
105+
#### [Copyright garbage file](./docs/config-file-copyright-garbage-yml.md)
106+
107+
A list of copyright statements that are considered garbage, for example statements that were incorrectly classified as
108+
copyrights by the scanner.
109+
110+
| Format | Scope | Default location |
111+
|-------------|--------|-----------------------------------------|
112+
| YAML / JSON | Global | `$ORT_CONFIG_DIR/copyright-garbage.yml` |
113+
114+
#### [Curations file](./docs/config-file-curations-yml.md)
115+
116+
A file to correct invalid or missing package metadata, and to set the concluded license for packages.
117+
118+
| Format | Scope | Default location |
119+
|-------------|--------|---------------------------------|
120+
| YAML / JSON | Global | `$ORT_CONFIG_DIR/curations.yml` |
121+
122+
#### [Custom license texts dir](./docs/dir-custom-license-texts.md)
123+
124+
A directory that contains license texts which are not provided by ORT.
125+
126+
| Format | Scope | Default location |
127+
|--------|--------|-----------------------------------------|
128+
| Text | Global | `$ORT_CONFIG_DIR/custom-license-texts/` |
129+
130+
#### [How to fix text provider script](./docs/scripts/how-to-fix-text-provider-kts.md)
131+
132+
A Kotlin script that enables the injection of how-to-fix texts in Markdown format for ORT issues into the reports.
133+
134+
| Format | Scope | Default location |
135+
|---------------|--------|---------------------------------------------------------|
136+
| Kotlin script | Global | `$ORT_CONFIG_DIR/reporter.how-to-fix-text-provider.kts` |
137+
138+
#### [License classifications file](docs/config-file-license-classifications-yml.md)
139+
140+
A file that contains user-defined categorization of licenses.
141+
142+
| Format | Scope | Default location |
143+
|-------------|--------|-----------------------------------------------|
144+
| YAML / JSON | Global | `$ORT_CONFIG_DIR/license-classifications.yml` |
145+
146+
#### [Resolution file](./docs/config-file-resolutions-yml.md)
147+
148+
Configurations to resolve any issues or rule violations by providing a mandatory reason, and an optional comment to
149+
justify the resolution on a global scale.
150+
151+
| Format | Scope | Default location |
152+
|-------------|--------|-----------------------------------|
153+
| YAML / JSON | Global | `$ORT_CONFIG_DIR/resolutions.yml` |
154+
155+
#### [Repository configuration file](./docs/config-file-ort-yml.md)
156+
157+
A configuration file, usually stored in the project's repository, for license finding curations, exclusions, and issues
158+
or rule violations resolutions in the context of the repository.
159+
160+
| Format | Scope | Default location |
161+
|-------------|----------------------|---------------------------------|
162+
| YAML / JSON | Repository (project) | `[analyzer-input-dir]/.ort.yml` |
163+
164+
#### [Package configuration file / directory](./docs/config-file-package-configuration-yml.md)
165+
166+
A single file or a directory with multiple files containing configurations to set provenance-specific path excludes and
167+
license finding curations for dependency packages to address issues found within a scan result. `helper-cli`'s
168+
[`package-config create` command](./helper-cli/src/main/kotlin/commands/packageconfig/CreateCommand.kt)
169+
can be used to populate a directory with template package configuration files.
170+
171+
| Format | Scope | Default location |
172+
|-------------|----------------------|-------------------------------------------|
173+
| YAML / JSON | Package (dependency) | `$ORT_CONFIG_DIR/package-configurations/` |
174+
175+
#### [Policy rules file](./docs/scripts/rules-kts.md)
176+
177+
The file containing any policy rule implementations to be used with the *evaluator*.
178+
179+
| Format | Scope | Default location |
180+
|---------------------|-----------|---------------------------------------|
181+
| Kotlin script (DSL) | Evaluator | `$ORT_CONFIG_DIR/evaluator.rules.kts` |
182+
183+
### Protecting environment variables
184+
185+
In order to do its analysis, ORT invokes a number of external tools, such as package managers or scanners. Especially
186+
when interacting with package managers to obtain the dependencies of the analyzed project, this can lead to the
187+
execution of code in build scripts from potentially unknown sources. A possible risk in this constellation is that
188+
untrusted code could read sensitive information from environment variables used for the ORT configuration, such as
189+
database connection strings or service credentials. This is because the environment variables of a process are by
190+
default propagated to the child processes spawned by it.
191+
192+
To reduce this risk, ORT filters out certain environment variables when it runs external tools in child processes.
193+
This filter mechanism can be configured via the following properties in the
194+
[ORT configuration file](./model/src/main/resources/reference.yml):
195+
196+
| Property | Description |
197+
|----------|-------------|
198+
| deniedProcessEnvironmentVariablesSubstrings | A list of substrings that identify variables containing sensitive information. All variables that contain at least one of these strings (ignoring case) are not propagated to child processes. The default for this property contains strings like "PASS", "PWD", or "TOKEN", which are typically used to reference credentials. |
199+
| allowedProcessEnvironmentVariableNames | This is a list of variable names that are explicitly allowed to be passed to child processes - even if they contain a substring listed in `deniedProcessEnvironmentVariablesSubstrings`. Via this property variables required by external tools, e.g. credentials for repositories needed by package managers, can be passed through. Here, entries must match variables names exactly and case-sensitively. |
200+
201+
This mechanism offers a certain level of security without enforcing an excessive amount of configuration, which would
202+
be needed for instance to define an explicit allow list. With the two configuration properties even corner cases can be
203+
defined:
204+
205+
* In order to disable filtering of environment variables completely, set the
206+
`deniedProcessEnvironmentVariablesSubstrings` property to a single string that is certainly not contained in any
207+
environment variable, such as "This is for sure not contained in a variable name".
208+
* To prevent that any environment variable is passed to a child process, substrings can be configured in
209+
`deniedProcessEnvironmentVariablesSubstrings` that match all variables, for instance one string for each letter of the
210+
alphabet.

0 commit comments

Comments
 (0)