Improve and expose package token replacement as a feature

AGENTS.md

@@ -16,11 +16,12 @@ Compatible with both SDK Pack and NuGetizer. Does not require NuGetizer.
 | Path | Role |
 |------|------|
 | `src/Readme/IncludesResolver.cs` | Pure include-resolution logic (ported from NuGetizer) |
-| `src/Readme/TokenReplacer.cs` | Pure `$token$` replacement (same algorithm as NuGetizer) |
+| `src/Readme/TokenReplacer.cs` | Pure `$token$` replacement + placeholder discovery |
 | `src/Readme/GitHubUrlExpander.cs` | Pure relative-link expansion via Markdig (ported from NuGetizer) |
-| `src/Readme/ProcessReadmeIncludes.cs` | Thin MSBuild task wrapping resolver + tokens + GitHub URLs |
+| `src/Readme/ProcessPackageReadme.cs` | MSBuild task: includes + tokens + GitHub URLs (no unknown-token warn) |
+| `src/Readme/ReplacePackageTokens.cs` | MSBuild task: tokens in arbitrary files (RDM001 on unknown) |
 | `src/Readme/build/Readme.props` | Defaults: `PackReadme`, `ReadmeExpandGitHubUrls` |
-| `src/Readme/build/Readme.targets` | Auto-pack item metadata + pre-pack process + NuGetizer `PackageFile` retarget |
+| `src/Readme/build/Readme.targets` | Auto-pack, `CollectReplacementTokens`, `ProcessPackageReadme`, task UsingTasks |
 | `src/Readme/buildTransitive/*` | Re-exports `build/` for transitive imports |
 | `src/ILRepack.targets` | Merges Markdig into `build/Readme.dll` |
 | `src/Tests/` | Unit tests on fixtures + SDK Pack / NuGetizer pack scenario tests |
@@ -32,7 +33,10 @@ Compatible with both SDK Pack and NuGetizer. Does not require NuGetizer.
 - **Warnings not errors**: missing includes / anchors call `Log.LogWarning` and leave markers in place.
 - **` ```exclude ` fences**: includes inside fenced code blocks with language `exclude` are left literal (for documenting include syntax).
 - **Fragment resolution**: explicit `<!-- #fragment -->` pairs win (placement controls whether a section title is included); otherwise GitHub heading auto-anchors match and **include the heading line** (e.g. `## Usage` for `#usage`).
-- **Token replacement**: after includes, `$token$` placeholders are replaced via `@(ReadmeReplacementToken)` (official NuGet: Id/Version/Author/Title/Description/Copyright/Configuration; plus Authors and Product; consumer-extensible). Case-insensitive names; last value wins for duplicates. Named separately from NuGetizer's `@(PackageReplacementToken)` to avoid item conflicts.
+- **Token replacement**: after includes, `$token$` placeholders are replaced via `@(PackageReplacementToken)` (official NuGet: Id/Version/Author/Title/Description/Copyright/Configuration; plus Authors and Product; consumer-extensible). Case-insensitive names; **last value wins** for duplicates (never fails). Same item name as NuGetizer for coexistence; newer NuGetizers can adopt this model.
+- **`CollectReplacementTokens`**: populates default `@(PackageReplacementToken)` items. Depend on it before using tokens. Consumers add items anytime; `AfterTargets="CollectReplacementTokens"` to remove/replace defaults; `$(CollectReplacementTokensDependsOn)` / `BeforeTargets` for pre-work.
+- **`ProcessPackageReadme`**: pack-time target (`DependsOnTargets=CollectReplacementTokens;$(ProcessPackageReadmeDependsOn)`) runs the process task then retargets pack items to the intermediate file. Does **not** warn on leftover `$token$` (readme may document that syntax).
+- **`ReplacePackageTokens`**: public task for arbitrary InputFile→OutputFile replacement with `Tokens="@(PackageReplacementToken)"`. Emits **RDM001** for remaining unknown placeholders (suppress via `NoWarn`). Suitable for incremental targets (`Inputs`/`Outputs`).
 - **GitHub relative URLs**: after tokens, relative Markdown links/images are rewritten to `https://github.com/{owner}/{repo}/{commit}/…` when `ReadmeExpandGitHubUrls` is true (default), `RepositoryUrl` is a github.com absolute URL, and a commit is available (`RepositoryCommit` → `RepositorySha` → `SourceRevisionId`). Auto-skips otherwise. Opt out with `ReadmeExpandGitHubUrls=false`. Uses Markdig `LinkInline` + `NormalizeRenderer` (same as NuGetizer).
 - **ILRepack Markdig**: Markdig and its polyfill deps are internalized into `build/Readme.dll` so the package ships a single task assembly (no satellite DLLs under `build/`).
 - **Package layout**: development dependency (`DevelopmentDependency=true`); task DLL + props/targets under `build/` (and `buildTransitive/`). Primary output is not under `lib/` (`IncludeBuildOutput=false`).

readme.md

@@ -74,7 +74,7 @@ Absolute `http(s)` includes from a **local** file are always allowed (subject to
 | `ReadmeIncludeScheme` | `https` | Semicolon-separated URI schemes allowed for remote includes (add `http` only if needed) |
 | `ReadmeExpandGitHubUrls` | `true` | Expand relative links/images to raw.githubusercontent.com when `RepositoryUrl` is github.com and a commit is available |
 | `@(ReadmeIncludeDomain)` | _(empty)_ | Hosts allowed for absolute remote includes nested inside remote content |
-| `@(ReadmeReplacementToken)` | Id, Version, Author, Authors, Title, Description, Copyright, Configuration, Product | `$token$` replacements applied after includes |
+| `@(PackageReplacementToken)` | Id, Version, Author, Authors, Title, Description, Copyright, Configuration, Product | `$token$` replacements (defaults from `CollectReplacementTokens`) |
 
 ```xml
 <PropertyGroup>
@@ -90,15 +90,45 @@ Absolute `http(s)` includes from a **local** file are always allowed (subject to
 
 Supports the [official NuGet replacement tokens](https://learn.microsoft.com/en-us/nuget/reference/nuspec#replacement-tokens) (`$id$`, `$version$`, `$author$`, `$title$`, `$description$`, `$copyright$`, `$configuration$`), plus `$authors$` (MSBuild `$(Authors)`) and `$product$` (as in NuGetizer). Replacements run **after** include expansion so tokens inside included files are replaced too.
 
-Extensible via MSBuild items `@(ReadmeReplacementToken)` (separate from NuGetizer's `@(PackageReplacementToken)` to avoid conflicts):
+Defaults are populated by the **`CollectReplacementTokens`** target (same item name as NuGetizer: `@(PackageReplacementToken)`). Duplicate names keep the **last** value — never an error — so Readme and NuGetizer can both contribute tokens.
+
+#### Extend tokens (readme and other files)
+
+Add items anytime; use them case-insensitively as `$company$` in the readme:
 
 ```xml
 <ItemGroup>
-  <ReadmeReplacementToken Include="Company" Value="$(Company)" />
+  <PackageReplacementToken Include="Company" Value="$(Company)" />
 </ItemGroup>
 ```
 
-Use the new token case-insensitively in the readme as `$company$`.
+Override or remove defaults **after** they are collected:
+
+```xml
+<Target Name="CustomizePackageTokens" AfterTargets="CollectReplacementTokens">
+  <ItemGroup>
+    <PackageReplacementToken Remove="Product" />
+    <PackageReplacementToken Include="BuildDate" Value="$([System.DateTime]::UtcNow.ToString('yyyy-MM-dd'))" />
+  </ItemGroup>
+</Target>
+```
+
+Pre-work before defaults (e.g. compute properties) via `$(CollectReplacementTokensDependsOn)` or `BeforeTargets="CollectReplacementTokens"`.
+
+#### Replace tokens in arbitrary files
+
+`ReplacePackageTokens` applies the same `@(PackageReplacementToken)` map to any file. Pair it with `Inputs` / `Outputs` for incremental builds. Unknown remaining `$token$` placeholders emit warning **RDM001** (suppress with `<NoWarn>$(NoWarn);RDM001</NoWarn>`). The package-readme path does **not** warn on unknown tokens (docs may contain `$…$` examples).
+
+```xml
+<Target Name="ProcessPackageEula"
+        DependsOnTargets="CollectReplacementTokens"
+        Inputs="osmfeula.txt"
+        Outputs="$(IntermediateOutputPath)OSMFEULA.txt">
+  <ReplacePackageTokens InputFile="osmfeula.txt"
+                        OutputFile="$(IntermediateOutputPath)OSMFEULA.txt"
+                        Tokens="@(PackageReplacementToken)" />
+</Target>
+```
 
 ### GitHub relative URLs
 

src/Readme/ProcessReadmeIncludes.cs → src/Readme/ProcessPackageReadme.cs

@@ -11,8 +11,9 @@ namespace Readme;
 /// MSBuild task that resolves <c>&lt;!-- include ... --&gt;</c> directives in a package readme,
 /// applies <c>$token$</c> replacements, optionally expands GitHub relative URLs, and writes the
 /// processed content to an output path (typically under BaseIntermediateOutputPath).
+/// Does not warn on unknown <c>$token$</c> placeholders (readme content may document that syntax).
 /// </summary>
-public class ProcessReadmeIncludes : Task
+public class ProcessPackageReadme : Task
 {
     /// <summary>Source readme file path (project readme before include expansion).</summary>
     [Required]
@@ -36,7 +37,8 @@ public class ProcessReadmeIncludes : Task
 
     /// <summary>
     /// Token name/value pairs for <c>$token$</c> replacement after include expansion.
-    /// Maps from <c>@(ReadmeReplacementToken)</c> (metadata <c>Value</c>).
+    /// Maps from <c>@(PackageReplacementToken)</c> (metadata <c>Value</c>).
+    /// Duplicate names keep the last value (coexists with NuGetizer contributions).
     /// </summary>
     public ITaskItem[]? ReplacementTokens { get; set; }
 
@@ -114,7 +116,7 @@ public override bool Execute()
             Directory.CreateDirectory(directory);
 
         File.WriteAllText(OutputFile, content);
-        Log.LogMessage(MessageImportance.Low, "Processed package readme includes: {0} -> {1}", SourceFile, OutputFile);
+        Log.LogMessage(MessageImportance.Low, "Processed package readme: {0} -> {1}", SourceFile, OutputFile);
         return !Log.HasLoggedErrors;
     }
 

src/Readme/Readme.csproj

@@ -8,7 +8,7 @@
     $(TargetPath) or build/Readme.dll blocks Compile/Copy (MSB3027). So we:
       1. Copy the built DLL to a fixed shadow under obj/Readme.Task/ (Build never writes there).
       2. Register UsingTask on that shadow *before* importing build/Readme.targets (first wins).
-      3. Hook Prepare via ProcessPackageReadmeIncludesDependsOn (extension point in shipped targets).
+      3. Hook Prepare via ProcessPackageReadmeDependsOn (extension point in shipped targets).
       4. ContinueOnError on shadow Copy so re-entry after the node already locked the file is OK.
   -->
   <PropertyGroup>
@@ -70,7 +70,7 @@
     Pack always runs after Build (dotnet pack / GeneratePackageOnBuild), so outputs exist.
   -->
   <Target Name="AddReadmePackageAssets"
-          DependsOnTargets="CopyTaskAssemblyToBuild;ProcessPackageReadmeIncludes">
+          DependsOnTargets="CopyTaskAssemblyToBuild;ProcessPackageReadme">
     <ItemGroup>
       <TfmSpecificPackageFile Include="$(MSBuildProjectDirectory)\build\Readme.props" PackagePath="build/Readme.props" />
       <TfmSpecificPackageFile Include="$(MSBuildProjectDirectory)\build\Readme.targets" PackagePath="build/Readme.targets" />
@@ -83,7 +83,7 @@
       <TfmSpecificPackageFile Include="$(MSBuildThisFileDirectory)..\icon.png" PackagePath="icon.png"
                               Condition="Exists('$(MSBuildThisFileDirectory)..\icon.png')" />
       <!--
-        Do not pack the intermediate readme here — ProcessPackageReadmeIncludes already
+        Do not pack the intermediate readme here — ProcessPackageReadme already
         adds it as None with PackagePath=readme.md. A second entry causes NU5118.
       -->
     </ItemGroup>
@@ -108,18 +108,18 @@
       Never point at $(TargetPath) or build/Readme.dll: the node locks the loaded file forever.
     -->
     <_ReadmeTaskAssembly>$(MSBuildProjectDirectory)\obj\Readme.Task\Readme.dll</_ReadmeTaskAssembly>
-    <ProcessPackageReadmeIncludesDependsOn>PrepareSelfHostReadmeTask;$(ProcessPackageReadmeIncludesDependsOn)</ProcessPackageReadmeIncludesDependsOn>
+    <ProcessPackageReadmeDependsOn>PrepareSelfHostReadmeTask;$(ProcessPackageReadmeDependsOn)</ProcessPackageReadmeDependsOn>
   </PropertyGroup>
 
   <!-- First UsingTask wins over build/Readme.targets. -->
-  <UsingTask TaskName="Readme.ProcessReadmeIncludes" AssemblyFile="$(_ReadmeTaskAssembly)" />
+  <UsingTask TaskName="Readme.ProcessPackageReadme" AssemblyFile="$(_ReadmeTaskAssembly)" />
 
   <!--
-    Seed the shadow once. After ProcessReadmeIncludes runs, MSBuild holds a lock on the
+    Seed the shadow once. After ProcessPackageReadme runs, MSBuild holds a lock on the
     shadow for the node lifetime — further copies fail. ContinueOnError allows re-entry
     when this target is a DependsOn of multiple pack targets in the same node.
   -->
-  <!-- No DependsOn Build: ProcessPackageReadmeIncludes runs during pack; Build→Pack→…→Build cycles. -->
+  <!-- No DependsOn Build: ProcessPackageReadme runs during pack; Build→Pack→…→Build cycles. -->
   <Target Name="PrepareSelfHostReadmeTask">
     <MakeDir Directories="$(MSBuildProjectDirectory)\obj\Readme.Task" />
     <Copy SourceFiles="$(TargetPath)"

src/Readme/ReplacePackageTokens.cs

@@ -0,0 +1,76 @@
+using System;
+using System.IO;
+using System.Linq;
+using Microsoft.Build.Framework;
+using Task = Microsoft.Build.Utilities.Task;
+
+namespace Readme;
+
+/// <summary>
+/// MSBuild task that replaces NuGet-style <c>$token$</c> placeholders in an arbitrary file.
+/// Pass <c>Tokens="@(PackageReplacementToken)"</c> after <c>CollectReplacementTokens</c>.
+/// Emits warning <c>RDM001</c> for any remaining unknown placeholders (suppressible via NoWarn).
+/// </summary>
+public class ReplacePackageTokens : Task
+{
+    /// <summary>Warning code for an unknown <c>$token$</c> placeholder after replacement.</summary>
+    public const string UnknownTokenWarningCode = "RDM001";
+
+    /// <summary>Source file path.</summary>
+    [Required]
+    public string InputFile { get; set; } = "";
+
+    /// <summary>Destination path (parent directories are created). May equal <see cref="InputFile"/>.</summary>
+    [Required]
+    public string OutputFile { get; set; } = "";
+
+    /// <summary>
+    /// Token name/value pairs (<c>Include</c> = name, metadata <c>Value</c>).
+    /// Maps from <c>@(PackageReplacementToken)</c>. Duplicate names keep the last value.
+    /// </summary>
+    public ITaskItem[]? Tokens { get; set; }
+
+    public override bool Execute()
+    {
+        if (string.IsNullOrWhiteSpace(InputFile) || !File.Exists(InputFile))
+        {
+            Log.LogError("Input file not found: {0}", InputFile);
+            return false;
+        }
+
+        if (string.IsNullOrWhiteSpace(OutputFile))
+        {
+            Log.LogError("OutputFile is required.");
+            return false;
+        }
+
+        var content = File.ReadAllText(InputFile);
+        var tokens = Tokens?
+            .Select(i => (i.ItemSpec, i.GetMetadata("Value")))
+            ?? Enumerable.Empty<(string, string)>();
+        content = TokenReplacer.Replace(content, tokens);
+
+        foreach (var name in TokenReplacer.FindPlaceholders(content))
+        {
+            Log.LogWarning(
+                subcategory: null,
+                warningCode: UnknownTokenWarningCode,
+                helpKeyword: null,
+                file: InputFile,
+                lineNumber: 0,
+                columnNumber: 0,
+                endLineNumber: 0,
+                endColumnNumber: 0,
+                message: "Unknown package replacement token '${0}$' in '{1}'. Add a PackageReplacementToken item or remove the placeholder.",
+                messageArgs: new object[] { name, InputFile });
+        }
+
+        var directory = Path.GetDirectoryName(OutputFile);
+        if (!string.IsNullOrEmpty(directory))
+            Directory.CreateDirectory(directory);
+
+        File.WriteAllText(OutputFile, content);
+        Log.LogMessage(MessageImportance.Low, "Replaced package tokens: {0} -> {1}", InputFile, OutputFile);
+        return !Log.HasLoggedErrors;
+    }
+}

src/Readme/TokenReplacer.cs

@@ -8,9 +8,17 @@ namespace Readme;
 /// <summary>
 /// Replaces NuGet-style <c>$token$</c> placeholders (case-insensitive token names).
 /// Same algorithm as NuGetizer's readme/license token replacement.
+/// Duplicate names keep the last value (never fails) so Readme and NuGetizer can coexist.
 /// </summary>
 public static class TokenReplacer
 {
+    /// <summary>
+    /// Matches <c>$name$</c> placeholders where <c>name</c> has no whitespace or <c>$</c>.
+    /// </summary>
+    static readonly Regex PlaceholderRegex = new(
+        @"\$([^$\s]+)\$",
+        RegexOptions.CultureInvariant | RegexOptions.Compiled);
+
     /// <summary>
     /// Replace <c>$name$</c> tokens in <paramref name="text"/> using the given name/value pairs.
     /// Duplicate names keep the last value. Token names match case-insensitively.
@@ -35,4 +43,25 @@ public static string Replace(string text, IEnumerable<(string Name, string Value
 
         return expr.Replace(text, match => map[match.Groups[1].Value.ToLowerInvariant()]);
     }
+
+    /// <summary>
+    /// Returns distinct placeholder names found in <paramref name="text"/> (original casing of first occurrence).
+    /// Used after replacement to surface unknown tokens (e.g. task warning RDM001).
+    /// </summary>
+    public static IReadOnlyList<string> FindPlaceholders(string text)
+    {
+        if (string.IsNullOrEmpty(text))
+            return Array.Empty<string>();
+
+        var seen = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
+        var names = new List<string>();
+        foreach (Match match in PlaceholderRegex.Matches(text))
+        {
+            var name = match.Groups[1].Value;
+            if (seen.Add(name))
+                names.Add(name);
+        }
+
+        return names;
+    }
 }