Reorganize repository layout for clarity and maintainability

[michaeltlombardi]

Summary of the new feature / enhancement

As a contributor and maintainer,
I want to clearly understand which folders contain Rust crates, which folders contain resources, and so on,
so that I can quickly orient myself and more readily work on existing or new crates.

Currently, the repository is organized with crates at the top-level of the project folder (like dsc_lib) and in subfolders (like lib/osinfo_lib). It's difficult to get a mental map of what folders belong to what subproject and where to find or place new crates when needed.

Further, there's a mix of locations for resources. Some are defined in the resources folder, like resources/PSScript, while others are at the top-level, like wmi-adapter.

The current layout of the project and subprojects is difficult to understand without diving into every folder or asking a maintainer.

We should consider defining a layout structure and reorganizing the repository to follow it, so contributors can more easily navigate the project and so maintainers can place new subprojects in a defined location.

Proposed technical implementation details (optional)

Summary

I propose reorganizing the project with hierarchical namespacing for resources and extensions, with all crates placed in a single crates folder and symlinked as needed in the other locations. I further propose we rename the crates and folders for consistency and clarity. While this work incurs a significant initial cost, I believe it will simplify maintenance and contribution as the project continues to grow, especially if we plan to continue developing DSC as a monorepo.

That being said, I think there are other options available, and I have laid them out in the remainder of this proposal.

Proposed structures

I have a few different proposals for the structure.

1. Semi-flat structure by programming language.

In this structure:

• All rust crates are moved into a crates or rust folder.

• All subprojects written in PowerShell are placed into a pwsh folder.

• All subprojects written in Python are placed into a python folder.

• All subprojects written in Bash are placed into a bash folder.

• Language-agnostic folders (docs, schema, .config, etc) remain where they are currently located.

  This is true for every proposed structure. The reorganization should affect folders defining projects or modules, not repository-level configuration or information.

Tree view of proposed structure

.
├── .cargo/          # Unchanged
├── .config/         # Unchanged
├── .github/         # Unchanged
├── .pipelines/      # Unchanged
├── .vscode/         # Unchanged
├── archive/         # Unchanged
├── configurations/  # Unchanged
├── docs/            # Unchanged
├── examples/        # Unchanged
├── packaging/       # Unchanged
├── rfc/             # Unchanged
├── schemas/         # Unchanged
├── bash/
│   ├── apt            # moved from resources/
│   └── brew           # moved from resources/
├── pwsh/
│   ├── Microsoft.DSC.Experimental # moved from resources/
│   ├── PSScript                   # moved from resources/
│   ├── reboot_pending             # moved from top-level
│   ├── wmi-adapter                # moved from top-level
│   └── powershell-adapter         # moved from top-level
└── rust/
      ├── dsc/                           # CLI, from top-level
      ├── dsc_lib/                       # library, from top-level
      ├── dscecho/                       # resource, from top-level
      ├── osinfo/                        # resource, from top-level
      ├── osinfo_lib/                    # library, from lib/
      ├── pal/                           # library, from top-level
      ├── process/                       # resource, from top-level
      ├── registry/                      # resource, from top-level
      ├── registry_lib/                  # library, from top-level
      ├── runcommandonset/               # resource, from top-level
      ├── security_context_lib/          # library, from top-level
      ├── sshdconfig/                    # resource, from top-level
      ├── dsctest/                       # resource, test-only, from tools/
      ├── test_group_resource/           # resource, test-only, from tools/
      ├── tree-sitter-dscexpression/     # library, grammar, from top-level
      ├── tree-sitter-ssh-server-config/ # library, grammarm from top-level
      └── y2j/                           # CLI, from top-level

2. Semi-flat structure by purpose.

In this structure:

• All rust libraries are moved into a crates folder. This is also where the dsc CLI crate goes.

• All non-adapter resources are moved into the resources folder, whether implemented in Rust or another language. For rust resources, only the CLI/resource crate goes in this folder. Library crates still go in crates.

  We should consider keeping the CLI/resource crates in crates and creating a symlink in resources - that way the crates all live together, but you can quickly find which crates are for a specific resource, too.

• All extensions are placed in the extensions folder (this seems to be the case for existing extensions already).

• All adapters are placed in the adapters folder.

Tree view of proposed structure

.
├── .cargo/          # Unchanged
├── .config/         # Unchanged
├── .github/         # Unchanged
├── .pipelines/      # Unchanged
├── .vscode/         # Unchanged
├── archive/         # Unchanged
├── configurations/  # Unchanged
├── docs/            # Unchanged
├── examples/        # Unchanged
├── packaging/       # Unchanged
├── rfc/             # Unchanged
├── schemas/         # Unchanged
├── adapters/
│   ├── powershell-adapter # moved from top-level
│   └── wmi-adapter        # moved from top-level
├── crates/
│   ├── dsc/                           # CLI, from top-level
│   ├── dsc_lib/                       # library, from top-level
│   ├── dscecho/                       # resource, from top-level
│   ├── osinfo/                        # resource, from top-level
│   ├── osinfo_lib/                    # library, from lib/
│   ├── pal/                           # library, from top-level
│   ├── process/                       # resource, from top-level
│   ├── registry/                      # resource, from top-level
│   ├── registry_lib/                  # library, from top-level
│   ├── runcommandonset/               # resource, from top-level
│   ├── security_context_lib/          # library, from top-level
│   ├── sshdconfig/                    # resource, from top-level
│   ├── dsctest/                       # resource, test-only, from tools/
│   ├── test_group_resource/           # resource, test-only, from tools/
│   ├── tree-sitter-dscexpression/     # library, grammar, from top-level
│   ├── tree-sitter-ssh-server-config/ # library, grammarm from top-level
│   └── y2j/                           # CLI, from top-level
├── extensions/
│   ├── appx/  # Unchanged
│   ├── bicep/ # Unchanged
│   └── test/  # Unchanged
└── resources/
      ├── apt/                        # Unchanged
      ├── brew/                       # Unchanged
      ├── Microsoft.DSC.Experimental/ # Unchanged
      ├── PSScript/                   # Unchanged
      ├── dscecho                     # symlink to crates/dscecho
      ├── osinfo                      # symlink to crates/osinfo
      ├── process                     # symlink to crates/process
      ├── registry                    # symlink to crates/registry
      ├── runcommandonset             # symlink to crates/runcommandonset
      ├── sshdconfig                  # symlink to crates/sshdconfig
      ├── dsctest                     # symlink to crates/dsctest
      └── test_group_resource         # symlink to crates/test_group_resource

3. Hierarchical structure by namespace for resources/extensions.

This structure is similar to the semi-flat structure by purpose, except that we clearly indicate resource and extension namespacing in the folder hierarchy:

Tree view of proposed structure

.
├── .cargo/          # Unchanged
├── .config/         # Unchanged
├── .github/         # Unchanged
├── .pipelines/      # Unchanged
├── .vscode/         # Unchanged
├── archive/         # Unchanged
├── configurations/  # Unchanged
├── docs/            # Unchanged
├── examples/        # Unchanged
├── packaging/       # Unchanged
├── rfc/             # Unchanged
├── schemas/         # Unchanged
├── adapters/
│   ├── powershell # moved from top-level powershell-adapter
│   └── wmi        # moved from top-level wmi-adapter
├── crates/
│   ├── dsc/                           # CLI, from top-level
│   ├── dsc_lib/                       # library, from top-level
│   ├── dscecho/                       # resource, from top-level
│   ├── osinfo/                        # resource, from top-level
│   ├── osinfo_lib/                    # library, from lib/
│   ├── pal/                           # library, from top-level
│   ├── process/                       # resource, from top-level
│   ├── registry/                      # resource, from top-level
│   ├── registry_lib/                  # library, from top-level
│   ├── runcommandonset/               # resource, from top-level
│   ├── security_context_lib/          # library, from top-level
│   ├── sshdconfig/                    # resource, from top-level
│   ├── dsctest/                       # resource, test-only, from tools/
│   ├── test_group_resource/           # resource, test-only, from tools/
│   ├── tree-sitter-dscexpression/     # library, grammar, from top-level
│   ├── tree-sitter-ssh-server-config/ # library, grammarm from top-level
│   └── y2j/                           # CLI, from top-level
├── extensions/
│   ├── Microsoft.Windows.Appx/
│   │   └── discover/             # moved from extensions/appx
│   ├── Microsoft.DSC.Extension/
│   │   └── bicep/                # moved from extensions/bicep
│   └── test/                     # Unchanged
└── resources/
   ├── DSC.PackageManagement/
   │   ├── apt/                        # moved from resources/apt
   │   └── brew/                       # moved from resources/brew
   ├── Microsoft.DSC.Debug/
   │   └── dscecho                     # symlink to crates/dscecho
   ├── Microsoft.DSC.Experimental/   # Unchanged
   ├── Microsoft.DSC.Transitional/
   │   ├── PSScript/                   # moved from resources/PSScript
   │   └── runcommandonset             # symlink to crates/runcommandonset
   ├── Microsoft/
   │   ├── osinfo                      # symlink to crates/osinfo
   │   └── process                     # symlink to crates/process
   ├── Microsoft.Windows/
   │   └── registry                    # symlink to crates/registry
   ├── Microsoft.OpenSSH.SSHD/
   │   └── sshdconfig                  # symlink to crates/sshdconfig
   └── Test/
      ├── dsctest                     # symlink to crates/dsctest
      └── test_group_resource         # symlink to crates/test_group_resource

Renaming folders

A restructuring would also be a good time to standardize and revise our naming conventions. For the Rust crates, the conventional naming is kebab-case for crates, following the convention <project>-<area>-<subarea>, like dsc-lib. If we don't plan to publish the resource crates separately - if they're only going to be available through DSC itself, or if that's the only supported use - we should consider renaming those crates like dsc-resource-registry and dsc-resource-echo.

Even without moving the folders, renaming the crates would make it much easier to understand at a glance which crate is used for what purpose. Consider the following renames for existing crates:

.
├── dsc/                           # CLI / main package, unchanged.
├── dsc-lib/                       # library, from `dsc_lib` †
├── dsc-lib-osinfo/                # library, from `lib/osinfo_lib`
├── dsc-lib-pal/                   # library, from `pal`
├── dsc-lib-registry/              # library, from `registry_lib`
├── dsc-lib-security_context/      # library, from `security_context_lib`
├── dsc-resource-echo/             # resource, from `dscecho`
├── dsc-resource-osinfo/           # resource, from `osinfo`
├── dsc-resource-process/          # resource, from `process`
├── dsc-resource-registry/         # resource, from `registry`
├── dsc-resource-runcommandonset/  # resource, from `runcommandonset`
├── dsc-resource-sshdconfig/       # resource, from `sshdconfig`
├── dsc-resource-test              # resource, test-only, from `tools/dsctest`
├── dsc-resource-test_group/       # resource, test-only, from `tools/test_group_resource`
├── tree-sitter-dsc-expression/    # library, grammar, from `tree-sitter-dscexpression`
├── tree-sitter-ssh-server-config/ # library, grammar, unchanged.
└── y2j/                           # CLI, unchanged.

Note

† - Even though we change the crate name from dsc_lib to dsc-lib, only the dependency name changes. The items from that crate are still referenced in snake case like dsc_lib::foo. This won't affect the code that depends on this package, unlike the other library renames.

Regardless of whether we move the folders, renaming the crates to a standardized convention makes it much easier to understand the purpose of code without needing to inspect the implementations.

Recommended structure

This is my final recommended structure:

• Moving all crates into the crates folder and renaming the crates.
• Moving the adapter folders into adapters without the -adapter suffix on the subfolders.
• Moving the extensions into namespaced subfolders.
• Moving all non-rust resources into namespaced subfolders under resources and symlinking the resources implemented in Rust.

Tree view of proposed structure

.
├── .cargo/                              # Unchanged
├── .config/                             # Unchanged
├── .github/                             # Unchanged
├── .pipelines/                          # Unchanged
├── .vscode/                             # Unchanged
├── archive/                             # Unchanged
├── configurations/                      # Unchanged
├── adapters/                            # new folder
│   ├── powershell                       # moved from `powershell-adapter`
│   └── wmi                              # moved from `wmi-adapter`
├── crates/                              # new folder
│   ├── dsc/                             # CLI / main package, from `dsc`.
│   ├── dsc-lib/                         # library, from `dsc_lib`
│   ├── dsc-resource-echo/               # resource, from `dscecho`
│   ├── dsc-resource-osinfo/             # resource, from `osinfo`
│   ├── dsc-lib-osinfo/                  # library, from `lib/osinfo_lib`
│   ├── dsc-lib-pal/                     # library, from `pal`
│   ├── dsc-resource-process/            # resource, from `process`
│   ├── dsc-resource-registry/           # resource, from `registry`
│   ├── dsc-resource-registry-lib/       # library, from `registry_lib`‡
│   ├── dsc-resource-run_command_on_set/ # resource, from `runcommandonset`
│   ├── dsc-lib-security_context/        # library, from `security_context_lib`
│   ├── dsc-resource-sshdconfig/         # resource, from `sshdconfig`
│   ├── dsc-resource-test                # resource, test-only, from `tools/dsctest`
│   ├── dsc-resource-test_group/         # resource, test-only, from `tools/test_group_resource`
│   ├── tree-sitter-dsc-expression/      # library, grammar, from `tree-sitter-dscexpression`
│   ├── tree-sitter-ssh-server-config/   # library, grammar, unchanged.
│   └── y2j/                             # CLI, from top-level
├── docs/                                # Unchanged
├── examples/                            # Unchanged
├── extensions/                          # Unchanged
│   ├── Microsoft.Windows.Appx/          # new subfolder
│   │   └── Discover/                    # moved from `extensions/appx`
│   ├── Microsoft.DSC.Extension/         # new subfolder
│   │   └── Bicep/                       # moved from `extensions/bicep`
│   └── test/                            # Unchanged
│       ├── discover/                    # Unchanged
│       └── secret/                      # Unchanged
├── packaging/                           # Unchanged
├── resources/                           # Unchanged
│   ├── DSC.PackageManagement/           # new subfolder
│   │   ├── Apt/                         # moved from `resources/apt`
│   │   └── Brew/                        # moved from `resources/brew`
│   ├── Microsoft.DSC.Debug/             # new subfolder
│   │   └── Echo                         # symlink to `crates/dsc-resource-echo`
│   ├── Microsoft.DSC.Experimental/      # Unchanged
│   ├── Microsoft.DSC.Transitional/      # new subfolder
│   │   ├── PowerShellScript/            # moved from `resources/PSScript`
│   │   └── RunCommandOnSet/             # symlink to `crates/dsc-resource-run_command_on_set`
│   ├── Microsoft/                       # new subfolder
│   │   ├── osinfo                       # symlink to `crates/dsc-resource-osinfo`
│   │   └── process                      # symlink to `crates/dsc-resource-process`
│   ├── Microsoft.Windows/               # new subfolder
│   │   └── registry                     # symlink to `crates/dsc-resource-registry`
│   ├── Microsoft.OpenSSH.SSHD/          # new subfolder
│   │   └── sshdconfig                   # symlink to `crates/dsc-resource-sshdconfig`
│   └── Test/                            # new subfolder
│       ├── dsctest                      # symlink to `crates/dsc-resource-test`
│       └── test_group_resource          # symlink to `crates/dsc-resource-test_group`
├── rfc/                                 # Unchanged
└── schemas/                             # Unchanged