-
Notifications
You must be signed in to change notification settings - Fork 121
Add component detectors for Docker Compose, Helm, and Kubernetes #1759
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
1a3f651
077d1d0
fd74cb5
d5c91ed
a111856
70776f8
a099d56
0bbc3db
5c4e487
b5d3fbf
5158b75
a26f434
0873316
ea3b0a4
5984b1c
9d56f38
8bce22b
ad592b2
9ae1dd4
70762f1
d220257
dff7d5f
569babe
f651b13
7f08f43
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -54,3 +54,4 @@ launchSettings.json | |
|
|
||
| # Dev nupkgs | ||
| dev-source | ||
| .nuget/ | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| # Docker Compose Detection | ||
|
|
||
| ## Requirements | ||
|
|
||
| Docker Compose detection depends on the following to successfully run: | ||
|
|
||
| - One or more Docker Compose files matching the patterns: `docker-compose.yml`, `docker-compose.yaml`, `docker-compose.*.yml`, `docker-compose.*.yaml`, `compose.yml`, `compose.yaml`, `compose.*.yml`, `compose.*.yaml` | ||
|
|
||
| The `DockerComposeComponentDetector` is a **DefaultOff** detector and must be explicitly enabled via the `--DetectorArgs` parameter. | ||
|
|
||
| ## Detection strategy | ||
|
|
||
| The Docker Compose detector parses YAML compose files to extract Docker image references from service definitions. | ||
|
|
||
| ### Service Image Detection | ||
|
|
||
| The detector looks for the `services` section and extracts the `image` field from each service: | ||
|
|
||
| ```yaml | ||
| services: | ||
| web: | ||
| image: nginx:1.21 | ||
| db: | ||
| image: postgres:14 | ||
| ``` | ||
|
|
||
| Services that only define a `build` directive without an `image` field are skipped, as they do not reference external Docker images. | ||
|
|
||
| ### Full Registry References | ||
|
|
||
| The detector supports full registry image references: | ||
|
|
||
| ```yaml | ||
| services: | ||
| app: | ||
| image: ghcr.io/myorg/myapp:v2.0 | ||
| ``` | ||
|
|
||
| ### Variable Resolution | ||
|
|
||
| Images containing unresolved variables (e.g., `${TAG}` or `{{ .Values.tag }}`) are skipped to avoid reporting incomplete or incorrect references. The detector checks for `$`, `{`, or `}` characters in image references. | ||
|
|
||
| ## Known limitations | ||
|
|
||
| - **DefaultOff Status**: This detector must be explicitly enabled using `--DetectorArgs DockerCompose=EnableIfDefaultOff` | ||
| - **Variable Resolution**: Image references containing unresolved environment variables or template expressions are not reported, which may lead to under-reporting in compose files that heavily use variable substitution | ||
| - **Build-Only Services**: Services that only specify a `build` directive without an `image` field are not reported | ||
| - **No Dependency Graph**: All detected images are registered as independent components without parent-child relationships | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| # Helm Detection | ||
|
|
||
| ## Requirements | ||
|
|
||
| Helm detection depends on the following to successfully run: | ||
|
|
||
| - One or more Helm values files matching the patterns: `*values*.yaml`, `*values*.yml` | ||
| - Chart metadata files (`Chart.yaml`, `Chart.yml`, `chart.yaml`, `chart.yml`) are matched for file discovery but only values files are parsed for image references | ||
jpinz marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| The `HelmComponentDetector` is a **DefaultOff** detector and must be explicitly enabled via the `--DetectorArgs` parameter. | ||
|
|
||
| ## Detection strategy | ||
|
|
||
| The Helm detector parses Helm values YAML files to extract Docker image references. It recursively walks the YAML tree looking for `image` keys. | ||
|
|
||
| ### Direct Image Strings | ||
|
|
||
| The detector recognizes image references specified as simple strings: | ||
|
|
||
| ```yaml | ||
| image: nginx:1.21 | ||
| ``` | ||
|
|
||
| ### Structured Image Objects | ||
|
|
||
| The detector also supports the common Helm chart pattern of structured image definitions: | ||
|
|
||
| ```yaml | ||
| image: | ||
| registry: ghcr.io | ||
| repository: org/myimage | ||
| tag: v1.0 | ||
| ``` | ||
|
|
||
| The `registry` and `tag` fields are optional. When present, the detector reconstructs the full image reference. The `digest` field is also supported. | ||
|
|
||
| ### Recursive Search | ||
|
|
||
| The detector recursively traverses all nested mappings and sequences in the values file, detecting image references at any depth in the YAML structure. | ||
|
|
||
| ### Variable Resolution | ||
|
|
||
| Images containing unresolved variables (e.g., `{{ .Values.tag }}`) are skipped to avoid reporting incomplete or incorrect references. The detector checks for `$`, `{`, or `}` characters in image references. | ||
|
|
||
| ## Known limitations | ||
|
|
||
| - **DefaultOff Status**: This detector must be explicitly enabled using `--DetectorArgs Helm=EnableIfDefaultOff` | ||
| - **Values Files Only**: Only files with `values` in the name are parsed for image references. Chart.yaml files are matched but not processed | ||
| - **Variable Resolution**: Image references containing unresolved Helm template expressions are not reported | ||
| - **No Dependency Graph**: All detected images are registered as independent components without parent-child relationships | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| # Kubernetes Detection | ||
|
|
||
| ## Requirements | ||
|
|
||
| Kubernetes detection depends on the following to successfully run: | ||
|
|
||
| - One or more Kubernetes manifest files matching the patterns: `*.yaml`, `*.yml` | ||
| - Manifests must contain both `apiVersion` and `kind` fields to be recognized as Kubernetes resources | ||
|
|
||
| The `KubernetesComponentDetector` is a **DefaultOff** detector and must be explicitly enabled via the `--DetectorArgs` parameter. | ||
|
|
||
| ## Detection strategy | ||
|
|
||
| The Kubernetes detector parses Kubernetes manifest YAML files to extract Docker image references from container specifications. | ||
|
|
||
| ### Supported Resource Kinds | ||
|
|
||
| The detector recognizes the following Kubernetes resource kinds: | ||
|
|
||
| - `Pod` | ||
| - `Deployment` | ||
| - `StatefulSet` | ||
| - `DaemonSet` | ||
| - `ReplicaSet` | ||
| - `Job` | ||
| - `CronJob` | ||
| - `ReplicationController` | ||
|
|
||
| Files with an unrecognized `kind` or missing `apiVersion`/`kind` fields are skipped. | ||
|
|
||
| ### Container Image Detection | ||
|
|
||
| The detector extracts image references from all container types within pod specifications: | ||
|
|
||
| - **containers**: Main application containers | ||
| - **initContainers**: Initialization containers that run before app containers | ||
| - **ephemeralContainers**: Ephemeral debugging containers | ||
|
|
||
| ### Pod Spec Locations | ||
|
|
||
| The detector handles different pod spec locations depending on the resource kind: | ||
|
|
||
| - **Pod**: `spec.containers` | ||
| - **Deployment, StatefulSet, DaemonSet, ReplicaSet, ReplicationController**: `spec.template.spec.containers` | ||
| - **Job**: `spec.template.spec.containers` | ||
| - **CronJob**: `spec.jobTemplate.spec.template.spec.containers` | ||
|
|
||
| ### Variable Resolution | ||
|
|
||
| Images containing unresolved variables (e.g., `${TAG}`) are skipped to avoid reporting incomplete or incorrect references. The detector checks for `$`, `{`, or `}` characters in image references. | ||
|
|
||
| ## Known limitations | ||
|
|
||
| - **DefaultOff Status**: This detector must be explicitly enabled using `--DetectorArgs Kubernetes=EnableIfDefaultOff` | ||
| - **Broad File Matching**: The `*.yaml` and `*.yml` search patterns match all YAML files, so the detector relies on content-based filtering (`apiVersion` and `kind` fields) to identify Kubernetes manifests | ||
| - **Variable Resolution**: Image references containing unresolved template variables are not reported | ||
| - **Limited Resource Kinds**: Only the eight resource kinds listed above are supported. Custom resources (CRDs) or other workload types are not detected | ||
| - **No Dependency Graph**: All detected images are registered as independent components without parent-child relationships |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -38,6 +38,15 @@ public static class DockerReferenceUtility | |
| private const string LEGACYDEFAULTDOMAIN = "index.docker.io"; | ||
| private const string OFFICIALREPOSITORYNAME = "library"; | ||
|
|
||
| /// <summary> | ||
| /// Returns true if the reference contains unresolved variable placeholders (e.g., ${VAR}, {{ .Values.tag }}). | ||
| /// Such references should be skipped before calling <see cref="ParseFamiliarName"/> or <see cref="ParseQualifiedName"/>. | ||
| /// </summary> | ||
| /// <param name="reference">The image reference string to check.</param> | ||
|
Comment on lines
+41
to
+45
|
||
| /// <returns><c>true</c> if the reference contains variable placeholder characters; otherwise <c>false</c>.</returns> | ||
| public static bool HasUnresolvedVariables(string reference) => | ||
| reference.IndexOfAny(['$', '{', '}']) >= 0; | ||
|
Comment on lines
+41
to
+48
|
||
|
|
||
| public static DockerReference ParseQualifiedName(string qualifiedName) | ||
| { | ||
| var regexp = DockerRegex.ReferenceRegexp; | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,145 @@ | ||
| #nullable disable | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Enable nullability for new files please |
||
| namespace Microsoft.ComponentDetection.Detectors.DockerCompose; | ||
|
|
||
| using System; | ||
| using System.Collections.Generic; | ||
| using System.IO; | ||
| using System.Threading; | ||
| using System.Threading.Tasks; | ||
| using Microsoft.ComponentDetection.Common; | ||
| using Microsoft.ComponentDetection.Contracts; | ||
| using Microsoft.ComponentDetection.Contracts.Internal; | ||
| using Microsoft.ComponentDetection.Contracts.TypedComponent; | ||
| using Microsoft.Extensions.Logging; | ||
| using YamlDotNet.RepresentationModel; | ||
|
|
||
| public class DockerComposeComponentDetector : FileComponentDetector, IDefaultOffComponentDetector | ||
| { | ||
| public DockerComposeComponentDetector( | ||
| IComponentStreamEnumerableFactory componentStreamEnumerableFactory, | ||
| IObservableDirectoryWalkerFactory walkerFactory, | ||
| ILogger<DockerComposeComponentDetector> logger) | ||
| { | ||
| this.ComponentStreamEnumerableFactory = componentStreamEnumerableFactory; | ||
| this.Scanner = walkerFactory; | ||
| this.Logger = logger; | ||
| } | ||
|
|
||
| public override string Id => "DockerCompose"; | ||
|
|
||
| public override IList<string> SearchPatterns { get; } = | ||
| [ | ||
| "docker-compose.yml", "docker-compose.yaml", | ||
| "docker-compose.*.yml", "docker-compose.*.yaml", | ||
| "compose.yml", "compose.yaml", | ||
jpinz marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| "compose.*.yml", "compose.*.yaml", | ||
jpinz marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| ]; | ||
|
Comment on lines
+30
to
+36
|
||
|
|
||
| public override IEnumerable<ComponentType> SupportedComponentTypes => [ComponentType.DockerReference]; | ||
|
|
||
| public override int Version => 1; | ||
|
|
||
| public override IEnumerable<string> Categories => [Enum.GetName(typeof(DetectorClass), DetectorClass.DockerCompose)]; | ||
|
|
||
| protected override async Task OnFileFoundAsync(ProcessRequest processRequest, IDictionary<string, string> detectorArgs, CancellationToken cancellationToken = default) | ||
| { | ||
| var singleFileComponentRecorder = processRequest.SingleFileComponentRecorder; | ||
| var file = processRequest.ComponentStream; | ||
|
|
||
| try | ||
| { | ||
| this.Logger.LogInformation("Discovered Docker Compose file: {Location}", file.Location); | ||
|
|
||
| string contents; | ||
| using (var reader = new StreamReader(file.Stream)) | ||
| { | ||
| contents = await reader.ReadToEndAsync(cancellationToken); | ||
| } | ||
|
|
||
| var yaml = new YamlStream(); | ||
| yaml.Load(new StringReader(contents)); | ||
|
|
||
| if (yaml.Documents.Count == 0) | ||
| { | ||
| return; | ||
| } | ||
|
|
||
| foreach (var document in yaml.Documents) | ||
| { | ||
| if (document.RootNode is YamlMappingNode rootMapping) | ||
| { | ||
| this.ExtractImageReferences(rootMapping, singleFileComponentRecorder, file.Location); | ||
| } | ||
| } | ||
| } | ||
| catch (Exception e) | ||
| { | ||
| this.Logger.LogError(e, "Failed to parse Docker Compose file: {Location}", file.Location); | ||
| } | ||
| } | ||
|
|
||
| private static YamlMappingNode GetMappingChild(YamlMappingNode parent, string key) | ||
| { | ||
| foreach (var entry in parent.Children) | ||
| { | ||
| if (entry.Key is YamlScalarNode scalarKey && string.Equals(scalarKey.Value, key, StringComparison.OrdinalIgnoreCase)) | ||
| { | ||
| return entry.Value as YamlMappingNode; | ||
| } | ||
| } | ||
|
|
||
| return null; | ||
| } | ||
|
|
||
| private void ExtractImageReferences(YamlMappingNode rootMapping, ISingleFileComponentRecorder recorder, string fileLocation) | ||
| { | ||
| var services = GetMappingChild(rootMapping, "services"); | ||
| if (services == null) | ||
| { | ||
| return; | ||
| } | ||
|
|
||
| foreach (var serviceEntry in services.Children) | ||
| { | ||
| if (serviceEntry.Value is not YamlMappingNode serviceMapping) | ||
| { | ||
| continue; | ||
| } | ||
|
|
||
| // Extract direct image: references | ||
| foreach (var entry in serviceMapping.Children) | ||
| { | ||
| var key = (entry.Key as YamlScalarNode)?.Value; | ||
| if (string.Equals(key, "image", StringComparison.OrdinalIgnoreCase)) | ||
| { | ||
| var imageRef = (entry.Value as YamlScalarNode)?.Value; | ||
| if (!string.IsNullOrWhiteSpace(imageRef)) | ||
| { | ||
| this.TryRegisterImageReference(imageRef, recorder, fileLocation); | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
| private void TryRegisterImageReference(string imageReference, ISingleFileComponentRecorder recorder, string fileLocation) | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
| { | ||
| if (DockerReferenceUtility.HasUnresolvedVariables(imageReference)) | ||
| { | ||
| return; | ||
| } | ||
|
Comment on lines
+126
to
+130
|
||
|
|
||
| try | ||
| { | ||
| var dockerRef = DockerReferenceUtility.ParseFamiliarName(imageReference); | ||
| if (dockerRef != null) | ||
| { | ||
| recorder.RegisterUsage(new DetectedComponent(dockerRef.ToTypedDockerReferenceComponent())); | ||
| } | ||
| } | ||
| catch (Exception e) | ||
| { | ||
| this.Logger.LogWarning(e, "Failed to parse image reference '{ImageReference}' in {Location}", imageReference, fileLocation); | ||
| } | ||
|
Comment on lines
+125
to
+143
|
||
| } | ||
| } | ||
Uh oh!
There was an error while loading. Please reload this page.