feat: centralized indexer config with zod validations #698

0xBison · 2025-05-07T12:45:41Z

Resolves the following issues:

Synopsis:

Consolidates most of the config related and error related code into a single config folder. We now have a singleton that is created and validated at startup. Some initial type validations are performed at startup and there are another set of validations that run after which themself need the config values. We can add validations as we wish to the validations file. On startup failed validations will show like this:

Potential improvements:

Currently logs aren't very prominent due to them being console.log statements. we dont have access to a ponder logger as they dont expose it. i've asked them about this and will follow up. for now it may be ok or we can add some color to validations with the chalk package to make it clear.

…ion and centralised config

changeset-bot · 2025-05-07T12:45:45Z

🦋 Changeset detected

Latest commit: f4c48bb

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 10 packages

Name	Type
ensindexer	Minor
ensadmin	Minor
ensrainbow	Minor
@ensnode/ens-deployments	Minor
@ensnode/utils	Minor
@ensnode/ensrainbow-sdk	Minor
@ensnode/ponder-metadata	Minor
@ensnode/ensnode-schema	Minor
@ensnode/ponder-subgraph	Minor
@ensnode/shared-configs	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

vercel · 2025-05-07T12:45:47Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
admin.ensnode.io	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 22, 2025 5:20pm
ensnode.io	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 22, 2025 5:20pm
ensrainbow.io	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 22, 2025 5:20pm

shrugs

could we use zod transforms to make the schema a bit tighter? in particular where preprocess is used or where a custom error function that includes validation is used (which seems like an antipattern)? https://v4.zod.dev/api#transforms

this feels heavily llm-generated, which is fine in general but i'm worried i had to spend a lot of time reviewing these changes, in particular zod-related behavior where it's not using zod4 changes/features

apps/ensindexer/src/config/app-config.ts

apps/ensindexer/src/config/config.schema.ts

apps/ensindexer/src/config/app-config.ts

apps/ensindexer/ponder.config.ts

apps/ensindexer/src/handlers/Registry.ts

apps/ensindexer/src/config/app-config.ts

apps/ensindexer/src/lib/globals.ts

apps/ensindexer/src/plugins/index.ts

lightwalker-eth

@0xBison Hey excited to see this PR, thank you! I've only partially been able to complete a review of this PR so far, but sharing initial feedback 🫡

apps/ensindexer/.env.local.example

docs/ensnode.io/src/content/docs/ensindexer/usage/configuration.mdx

apps/ensindexer/test/utils/mockConfig.ts

apps/ensindexer/src/config/types.ts

apps/ensindexer/src/lib/plugin-helpers.ts

tk-o · 2025-05-20T13:44:21Z

apps/ensindexer/src/config/config.schema.ts

+  // Add selectedEnsDeployment to the config after as it's a derived value
+  .transform((config) => ({
+    ...config,
+    selectedEnsDeployment: ENSDeployments[config.ensDeploymentChain] as ENSDeploymentGlobalType,


I can see this spot as a nice place to include active plugin projection onto ENSIndexerConfig object.
In this very spot, we already have access to the parsed config value. This enables us to read config.plugins and availableDatasourceNames (which value is returned from Object.keys(config.selectedEnsDeployment) as DatasourceName[]; call):

const activePlugins = getActivePlugins(AVAILABLE_PLUGINS, config.plugins, availableDatasourceNames);

This approach would let us to move the declaration of const AVAILABLE_PLUGINS from ponder.config.ts into config.schema.ts.

Alternatively, if using this transform callback on the ENSIndexerConfigSchema turns out not to be the best place (however, I think it's a place for the purpose), we can move the transformation logic into buildConfigFromEnvironment function call of app-config.ts file.

shrugs

this looks GREAT to me, well done. config.schema.ts feels lovely as well and reads so cleanly. excited to have this in the codebase

shrugs · 2025-05-20T16:38:25Z

apps/ensindexer/src/config/types.ts

+   *   indexedChains object which is ensured by the validateChainConfigs
+   *   function in `validations.ts` and not part of the schema validation here.
+   */
+  indexedChains: Record<number, ChainConfig>;


nit: chainConfigs?

i like this, @lightwalker-eth what are your thoughts, as this was previously chains and you suggested we change it to indexedChains.

Good question -- my understanding is that each ChainConfig in this field is actively being indexed based on the plugins / ensDeploymentChain / ensDeploymentChain.

In other words, let's imagine environment variables were defined with RPC endpoints for mainnet and also for base, however based on the configuration of plugins / ensDeploymentChain / ensDeploymentChain, only the subgraph plugin is activated and therefore the only indexedChain would be mainnet and that base would be excluded from this field.

This distinction is why I was thinking it would be best to name this indexedChains as each of these ChainConfigs should be guaranteed to be actively indexed by the ENSIndexer. This is important for cases like the indexing status page we're building in ENSAdmin, etc.

Appreciate your advice and also if we should refine the docs / validations for this field at all based on this idea 👍

currently this property in the config defines "the set of chain configurations (rpcEndpoint & maxRequests) provided by the environment" and isn't filtered by those that are explicitly indexed.

to derive "the set of all actively indexed chains" in ensadmin, for example, the networks property of the ponder config should be used, as that's the actual set of chains ponder is indexing at runtime. i.e. the plugins determine which networks they require by specifying as such in the networks property of their ponder config.

i don't think it's important for this property in the ENSIndexerConfig (current: indexedChains, proposed: chainConfigs) to only represent explicitly indexed chains, and i think the fact that it represents how the user is configuring ENSIndexer is ideal, actually. the decision making of which chains are actively indexed is done by each of the active plugins, not the ENSIndexerConfig, which is just providing the configuration options for each of the possible networks ensindexer can index from.

so i'm still in favor of chainConfigs or rpcConfigs (and possibly even changing the object to { url: string, maxRequestsPerSecond: number } for minimalism) and don't think any additional re-architecture of this parameter is necessary. and the documentation should be updated to read more like "Configuration options for each indexable chain's rpc"...

apps/ensindexer/src/config/app-config.ts

lightwalker-eth

@0xBison Really super work here! 🚀 This is a partial review but about to hop on some long calls and didn't want to further delay sharing some feedback 🫡

lightwalker-eth · 2025-05-21T14:15:54Z

apps/ensindexer/src/config/types.ts

+   *   indexedChains object which is ensured by the validateChainConfigs
+   *   function in `validations.ts` and not part of the schema validation here.
+   */
+  indexedChains: Record<number, ChainConfig>;


Good question -- my understanding is that each ChainConfig in this field is actively being indexed based on the plugins / ensDeploymentChain / ensDeploymentChain.

In other words, let's imagine environment variables were defined with RPC endpoints for mainnet and also for base, however based on the configuration of plugins / ensDeploymentChain / ensDeploymentChain, only the subgraph plugin is activated and therefore the only indexedChain would be mainnet and that base would be excluded from this field.

This distinction is why I was thinking it would be best to name this indexedChains as each of these ChainConfigs should be guaranteed to be actively indexed by the ENSIndexer. This is important for cases like the indexing status page we're building in ENSAdmin, etc.

Appreciate your advice and also if we should refine the docs / validations for this field at all based on this idea 👍

lightwalker-eth · 2025-05-21T14:18:23Z

apps/ensindexer/src/config/types.ts

+  rpcMaxRequestsPerSecond: number;
+}
+
+export interface GlobalBlockrange {


What about an invariant if both startBlock and endBlock are defined? Should endBlock be enforced to be >= startBlock, or more strictly > startBlock?

i believe that's handled on apps/ensindexer/src/config/config.schema.ts line 56

.refine( (val) => val.startBlock === undefined || val.endBlock === undefined || val.startBlock <= val.endBlock, { error: "END_BLOCK must be greater than or equal to START_BLOCK." }, );

BUT i just tested this with ponder and if start and end blocks are the same, nothing is indexed, even though it feels like just that exact block should be indexed. will log bug with ponder but in the meantime we should switch this comparison to be just < and update the error message.

Ok cool. Will be good to also refine the docs we put on this type to fully define all the expected invariants.

lightwalker-eth · 2025-05-21T14:19:06Z

apps/ensindexer/src/config/types.ts

+  rpcMaxRequestsPerSecond: number;
+}
+
+export interface GlobalBlockrange {


Maybe we can just call this Blockrange since it looks to be a generic idea modeled by this type. What do you think?

To avoid a possible misunderstanding, I am not suggesting to rename the globalBlockrange field on ENSIndexerConfig, only asking about the GlobalBlockrange interface.

sounds good to me, actually we should use re-use the Blockrange type from apps/ensindexer/src/lib/types.ts here

lightwalker-eth · 2025-05-21T14:21:00Z

apps/ensindexer/src/config/types.ts

+/**
+ * Configuration for a single blockchain network (chain) used by ENSIndexer.
+ *
+ * Invariants:


Suggest remove the documentation for this key / chainId out of the ChainConfig interface. I see it is already documented on the indexedChains field 👍

lightwalker-eth · 2025-05-21T14:23:31Z

apps/ensindexer/src/config/types.ts

+
+  /**
+   * The maximum number of RPC requests per second allowed for this chain.
+   * This is used to avoid rate limiting by the RPC provider. This value


Suggest moving the docs for the following idea out of the ChainConfig interface and instead to wherever we are building a ChainConfig instead.

This value is optional and defaults to DEFAULT_RPC_RATE_LIMIT if not specified.

It can cause confusion to document this idea here because the field on this interface is not optional.

lightwalker-eth · 2025-05-21T16:28:04Z

apps/ensindexer/test/utils/mockConfig.ts

+  return clonedObj;
+}
+
+// Default, non-exported mock configuration template


Suggest building this from the environment interface you created for a config.

In other words, all instances of an ENSIndexerConfig, including this mock instance, should be guaranteed to always go through the validation logic we're building.

This will be helpful as over time we'll continue evolving the invariants and schema. It's good that our unit tests fail here if we forgot something.

lightwalker-eth · 2025-05-21T16:39:16Z

apps/ensindexer/src/lib/plugin-helpers.ts

@@ -93,7 +74,15 @@ export function getActivePlugins<PLUGIN extends ENSIndexerPlugin>(

    if (!hasRequiredDatasources) {


In my understanding, this check should happen as part of building and validating an ENSIndexerConfig and not here.

It should be impossible to have an ENSIndexerConfig that fails this test anywhere in our code.

Please see several other related comments I've shared on this PR.

lightwalker-eth · 2025-05-21T16:40:34Z

apps/ensindexer/src/lib/plugin-helpers.ts

  availablePlugins: readonly PLUGIN[],
-  requestedPluginNames: string[],
+  requestedPluginNames: Set<PluginName>,
  availableDatasourceNames: DatasourceName[],


Should we include these DatasourceName values as a field in the config object too? I feel like the config we build should include a data model that's aligned with the plugin -> data source relationship.

Appreciate if this is too complex for this PR. If it is, but you think this is a nice idea, can we please create a follow up GitHub Issue to remember the idea for the future? Thanks

lightwalker-eth · 2025-05-21T16:56:12Z

apps/ensindexer/src/lib/plugin-helpers.ts

+      // all instead of throwing an error here which will only flag a single missing RPC URL.
+      // The code which does that validation is the `validateChainConfigs` function in
+      // `src/config/validations.ts`.
+      transport: http(config.indexedChains[chainId]?.rpcEndpointUrl),


This function should first validate that config.indexedChains[chainId] is defined. If it isn't, it should throw an exception.

After validating that, there should be no need for the ? here.

lightwalker-eth · 2025-05-21T16:56:24Z

apps/ensindexer/src/lib/plugin-helpers.ts

+      // throw an error so we don't need to handle undefined here with a default.
+      // That is already handled in our schema validation so when the RPC url is added
+      // this will not be undefined.
+      maxRequestsPerSecond: config.indexedChains[chainId]?.rpcMaxRequestsPerSecond,


Similar feedback here as above.

lightwalker-eth

@0xBison Once again nice work here 👍 Finished my review with additional feedback. Appreciate your advice! Thanks 🫡

apps/ensindexer/src/config/config.schema.ts

lightwalker-eth · 2025-05-21T19:56:21Z

apps/ensindexer/src/config/config.schema.ts

+import { parse as parseConnectionString } from "pg-connection-string";
+import { z } from "zod/v4";
+
+export const DEFAULT_RPC_RATE_LIMIT = 50;


Suggest we update this to something like 500. As I understand, 50 will cause quite a slow indexing result.

lightwalker-eth · 2025-05-21T20:01:32Z

apps/ensindexer/src/config/app-config.ts

+  return `\n` + issues.map((issue) => issue.message).join("\n");
+}
+
+// loads the relevant environment variables in the shape of the zod schema


Suggested change

// loads the relevant environment variables in the shape of the zod schema

// loads the relevant environment variables in the shape of a ENSIndexerEnvironment

Is that fair?

lightwalker-eth · 2025-05-21T20:04:55Z

apps/ensindexer/src/config/app-config.ts

+ *
+ * This function returns a raw chain config which is not yet validated against the zod schema.
+ */
+export function getChainsFromEnv(): Record<number, RawChainConfig> {


Maybe no need to export?

lightwalker-eth · 2025-05-21T20:06:30Z

apps/ensindexer/src/config/app-config.ts

+ * This function then validates the config against the zod schema ensuring that the config
+ * meets all type checks and invariants.
+ *
+ * **Note:** This function does **not** perform deep validation of the configuration values


Appreciate your advice for what we would need to do so that this function would perform this deep validation?

lightwalker-eth · 2025-05-21T20:14:23Z

apps/ensindexer/src/config/app-config.ts

+/**
+ * Builds the ENSIndexer configuration object from an ENSIndexerEnvironment object
+ *
+ * This function then validates the config against the zod schema ensuring that the config


As I understand zod is an implementation detail of this function and therefore it would be best for us not to mention it in the comments here.

In other words, in theory we should be able to swap out zod for a different parsing / validation framework without it impacting "external" code. As I understand it, zod is an internal implementation detail for how an ENSIndexerInvironment is converted into an ENSIndexerConfig.

apps/ensindexer/test/config.test.ts

apps/ensindexer/src/lib/plugin-helpers.ts

apps/ensindexer/src/config/config.schema.ts

lightwalker-eth

@0xBison @shrugs Looks great! Appreciate all of the special efforts here! 🫡 🚀 🚀 Let's ship it!

0xBison added 4 commits May 7, 2025 00:40

refactor: initial changes for migrating to a zod based schema validat…

e47edf1

…ion and centralised config

chore: fix unit tests, change some defaults, update docs

13b09bc

chore: update ensrainbow url in example env file

d0cfaf9

chore: update local env rainbow comment

0a9c19a

0xBison requested a review from a team as a code owner May 7, 2025 12:45

0xBison added 2 commits May 7, 2025 21:05

Merge branch 'main' into feat/523-indexer-config

fa519bc

merge main

6fb6096

vercel bot deployed to Preview – ensrainbow.io May 7, 2025 13:11 View deployment

vercel bot deployed to Preview – admin.ensnode.io May 7, 2025 13:12 View deployment

vercel bot deployed to Preview – ensnode.io May 7, 2025 13:13 View deployment

chore: run biome

745ef95

vercel bot deployed to Preview – ensnode.io May 7, 2025 13:19 View deployment

vercel bot deployed to Preview – admin.ensnode.io May 7, 2025 13:20 View deployment

vercel bot deployed to Preview – ensrainbow.io May 7, 2025 13:20 View deployment

0xBison mentioned this pull request May 7, 2025

chore: onboarding changes #690

Merged

docs(changeset): centralized indexer config with zod validations

17b65d5

vercel bot deployed to Preview – admin.ensnode.io May 7, 2025 16:06 View deployment

vercel bot deployed to Preview – ensrainbow.io May 7, 2025 16:07 View deployment

vercel bot deployed to Preview – ensnode.io May 7, 2025 16:07 View deployment

shrugs requested changes May 8, 2025

View reviewed changes

lightwalker-eth reviewed May 8, 2025

View reviewed changes

chore: merge

2ec09f7

vercel bot deployed to Preview – ensnode.io May 12, 2025 07:28 View deployment

vercel bot deployed to Preview – ensrainbow.io May 12, 2025 07:28 View deployment

vercel bot deployed to Preview – admin.ensnode.io May 12, 2025 07:30 View deployment

chore: fix minor pr issues

0f3c839

vercel bot deployed to Preview – admin.ensnode.io May 12, 2025 08:03 View deployment

vercel bot deployed to Preview – ensnode.io May 12, 2025 08:04 View deployment

vercel bot deployed to Preview – ensnode.io May 20, 2025 09:26 View deployment

vercel bot deployed to Preview – admin.ensnode.io May 20, 2025 09:27 View deployment

0xBison mentioned this pull request May 20, 2025

Refactor plugin logic #735

Closed

0xBison commented May 20, 2025

View reviewed changes

apps/ensindexer/src/lib/plugin-helpers.ts Show resolved Hide resolved

tk-o reviewed May 20, 2025

View reviewed changes

shrugs approved these changes May 20, 2025

View reviewed changes

chore: small clean of parseEnvironment

8317e8d

vercel bot deployed to Preview – ensrainbow.io May 20, 2025 17:15 View deployment

vercel bot deployed to Preview – ensnode.io May 20, 2025 17:16 View deployment

vercel bot deployed to Preview – admin.ensnode.io May 20, 2025 17:17 View deployment

chore: lint

df19ce5

vercel bot deployed to Preview – ensnode.io May 21, 2025 03:45 View deployment

vercel bot deployed to Preview – admin.ensnode.io May 21, 2025 03:46 View deployment

vercel bot deployed to Preview – ensrainbow.io May 21, 2025 03:46 View deployment

lightwalker-eth requested changes May 21, 2025

View reviewed changes

Feat/validations pr updates (#742)

3caba79

vercel bot deployed to Preview – ensrainbow.io May 22, 2025 14:29 View deployment

vercel bot deployed to Preview – admin.ensnode.io May 22, 2025 14:30 View deployment

vercel bot deployed to Preview – ensnode.io May 22, 2025 14:31 View deployment

lightwalker-eth approved these changes May 22, 2025

View reviewed changes

Merge branch 'main' into feat/523-indexer-config

f4c48bb

shrugs merged commit 86f9f0a into main May 22, 2025
4 of 7 checks passed

shrugs deleted the feat/523-indexer-config branch May 22, 2025 17:16

vercel bot deployed to Preview – ensnode.io May 22, 2025 17:16 View deployment

github-actions bot mentioned this pull request May 22, 2025

Release New Version #687

Merged

vercel bot deployed to Preview – admin.ensnode.io May 22, 2025 17:20 View deployment

vercel bot deployed to Preview – ensrainbow.io May 22, 2025 17:20 View deployment

tk-o mentioned this pull request May 30, 2025

Include active plugins in ENSIndexerConfig #761

Closed

lightwalker-eth mentioned this pull request Jul 6, 2025

Build ENSIndexerConfig upon startup #523

Closed

		@@ -93,7 +74,15 @@ export function getActivePlugins<PLUGIN extends ENSIndexerPlugin>(

		if (!hasRequiredDatasources) {

	// loads the relevant environment variables in the shape of the zod schema
	// loads the relevant environment variables in the shape of a ENSIndexerEnvironment

feat: centralized indexer config with zod validations #698

feat: centralized indexer config with zod validations #698

Uh oh!

Conversation

0xBison commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

changeset-bot bot commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

vercel bot commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shrugs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lightwalker-eth left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shrugs left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shrugs May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lightwalker-eth left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

0xBison commented May 7, 2025 •

edited

Loading

changeset-bot bot commented May 7, 2025 •

edited

Loading

vercel bot commented May 7, 2025 •

edited

Loading

shrugs May 21, 2025 •

edited

Loading