Skip to content

README.md

Latest commit

 

History

History
63 lines (45 loc) · 1.89 KB

README.md

File metadata and controls

63 lines (45 loc) · 1.89 KB

Dataset Processing Function

.NET 10 Azure Isolated Worker Function that processes vendor CSV datasets from ADLS Gen2, validates and transforms records into the platform canonical schema, writes Parquet to the Fabric Lakehouse Bronze layer, and publishes dataset.bronze.available downstream.

Architecture

Clean architecture with four layers:

Project Role
DatasetProcessingFunction Azure Functions host, DI wiring, OTel bootstrap
DatasetProcessingFunction.Application CQRS commands/notifications, port interfaces
DatasetProcessingFunction.Domain Value objects, aggregates, domain services
DatasetProcessingFunction.Infrastructure ADLS, OneLake, Service Bus, Schema Registry

Quick Start (Local Dev)

See specs/001-dataset-ingestion-pipeline/quickstart.md for full local dev setup including Azurite and Service Bus Emulator.

Prerequisites

  • .NET 10 SDK
  • Azure Functions Core Tools v4
  • Docker Desktop (for Azurite + Service Bus emulator)

Run locally

cp src/DatasetProcessingFunction/local.settings.json.template \
   src/DatasetProcessingFunction/local.settings.json
# Edit local.settings.json with your emulator values
dotnet build DatasetProcessingFunction.slnx
cd src/DatasetProcessingFunction && func start

Run tests

# Unit tests
dotnet test tests/DatasetProcessingFunction.UnitTests/

# Integration tests (mocked infrastructure, no Docker required)
dotnet test tests/DatasetProcessingFunction.IntegrationTests/

# All tests with coverage
dotnet test DatasetProcessingFunction.slnx \
  --collect:"XPlat Code Coverage" \
  --results-directory ./TestResults

Deployment

Uses Azure Developer CLI (AZD) with Bicep IaC:

azd auth login
azd up --environment dev

GitHub Actions CI/CD runs on every push to main with OIDC Workload Identity Federation.