[MISC] Refactor plugin loading for BE and prompt-service to use centralized get_plugin() mechanism#1626
Merged
chandrasekharan-zipstack merged 1 commit intoNov 11, 2025
Conversation
Replaced manual plugin loading implementations across backend, prompt-service, and core modules with the new centralized get_plugin() function. This aligns all plugin loading with the new infrastructure, reduces code duplication, and simplifies maintenance. Removed obsolete loader files (subscription_loader, modifier_loader, processor_loader) that are now replaced by get_plugin(). Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
|
Caution Review failedFailed to post review comments Summary by CodeRabbit
WalkthroughThis diff implements a unified plugin management system across the backend and prompt-service. It removes legacy, ad-hoc plugin loaders (subscription_loader, modifier_loader, processor_loader) and replaces them with a new framework-agnostic PluginManager in unstract.core, plus Django and Flask wrappers. Various backend views and utilities are updated to use dynamic plugin retrieval via Changes
Sequence DiagramsequenceDiagram
participant OldFlow as Old: Static Load
participant NewFlow as New: Dynamic Lookup
participant PluginMgr as PluginManager
participant Plugin as Plugin Module
rect rgb(220, 240, 255)
note over OldFlow: Previous: Load at Startup
OldFlow->>OldFlow: Import loader module
OldFlow->>OldFlow: Scan plugins directory (Django app)
OldFlow->>Plugin: Import plugin modules
OldFlow->>OldFlow: Cache all plugins globally
OldFlow->>OldFlow: Use cached plugins in views
end
rect rgb(240, 255, 240)
note over NewFlow: New: Load on Demand
NewFlow->>PluginMgr: get_plugin("notification")
PluginMgr->>PluginMgr: Check loaded plugins cache
alt Plugin loaded
PluginMgr->>NewFlow: Return plugin metadata + service_class
NewFlow->>Plugin: service_class() instantiate
NewFlow->>Plugin: Call method (e.g., send_sharing_notification)
else Plugin not loaded
PluginMgr->>NewFlow: Return None or empty dict
NewFlow->>NewFlow: Skip notification, continue
end
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes
Areas requiring extra attention:
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
Test ResultsSummary
Runner Tests - Full Report
SDK1 Tests - Full Report
|
|
Deepak-Kesavan
approved these changes
Nov 7, 2025
94463f2
into
feat/UN-2930-plugin-infrastructure
8 checks passed
chandrasekharan-zipstack
added a commit
that referenced
this pull request
Nov 11, 2025
* UN-2930 [FEAT] Add core plugin infrastructure Introduce a generic, framework-agnostic plugin system that supports Django, Flask, and Workers. This foundation enables dynamic loading of plugins with metadata validation, singleton/non-singleton patterns, and support for compiled extensions. Key components: - Generic PluginManager with plugin discovery and validation - DjangoPluginManager wrapper for Django apps - FlaskPluginManager wrapper for Flask apps - Redis client enhancement (REDIS_USERNAME support) - Plugin directory exclusions in .gitignore - Development dependency: debugpy for debugging This is the first PR in a series for UN-2930, establishing the infrastructure needed for subscription usage tracking and other plugin-based features. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Refactored plugin gitignore to avoid explicitly stating plugin names * Added new plugin loading mechanism to BE NOTE: Existing mechanism is left untouched to avoid breaking anything * Addressed code rabbit's review comment * Minor code rabbit comment addressed, updated docs on adding and using plugins * misc: Updated plugin discovery to supported nested plugins upto 2 levels * UN-2930 [FEAT] Enhance plugin_loader with required parameter validation - Update plugin_loader docstring to clarify plugins_dir and plugins_pkg are required - Add parameter validation with helpful ValueError message - Include practical example showing correct usage pattern - Improves developer experience with clear setup requirements Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * UN-2930 [FIX] Add thread-safety to FlaskPluginManager singleton - Add threading.Lock at class level to protect singleton instance - Guard instance creation and state mutations with lock - Prevent race conditions in multi-threaded Flask environments - Ensure thread-safe initialization of app, plugins_dir, and plugins_pkg 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * UN-2930 [REFACTOR] Merge nested conditions and simplify exception handling - Merge nested if statements in FlaskPluginManager.getInstance() into single compound condition - Remove redundant PermissionError from plugin_manager (OSError parent class covers it) 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> * [MISC] Refactor plugin loading for BE and prompt-service to use centralized get_plugin() mechanism (#1626) Refactor plugin loading to use centralized get_plugin() mechanism Replaced manual plugin loading implementations across backend, prompt-service, and core modules with the new centralized get_plugin() function. This aligns all plugin loading with the new infrastructure, reduces code duplication, and simplifies maintenance. Removed obsolete loader files (subscription_loader, modifier_loader, processor_loader) that are now replaced by get_plugin(). Generated with Claude Code Co-authored-by: Claude <noreply@anthropic.com> * UN-2930 [FIX] Track subscription usage only for successful runs (#1620) UN-2930 [FEAT] Implement subscription tracking for successful executions only This PR implements the complete UN-2930 fix to track subscription usage only for successful API calls and workflow executions. Key changes: - Add subscription usage tracking decorator for Django views - Apply decorator to Prompt Studio index and answer-prompt operations - Add workflow completion/failure handlers for defer/commit/discard pattern - Implement batch tracking in workers for completed workflows - Add platform-service plugin integration for subscription tracking - Remove hardcoded SQL from platform-service, use plugin-based approach Components: 1. Decorator (backend/utils/subscription_usage_decorator.py): - Auto-commits usage on success - Auto-discards usage on failure - Non-blocking error handling 2. Prompt Studio Integration: - Track usage for index_document() - Track usage for prompt_responder() 3. Workflow Integration: - Completion handler commits deferred usage for COMPLETED workflows - Failure handler discards all deferred usage 4. Workers Integration: - Batch commit for successful file executions - Calls internal API for efficient batch processing 5. Platform Service: - Plugin-based subscription tracking - Cleaner separation of concerns This ensures usage is only recorded when operations actually succeed, fixing the issue where failed operations were incorrectly tracked. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update backend/configuration/config_registry.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Signed-off-by: Chandrasekharan M <117059509+chandrasekharan-zipstack@users.noreply.github.com> --------- Signed-off-by: Chandrasekharan M <117059509+chandrasekharan-zipstack@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Why
How
Can this PR break any existing features. If yes, please list possible items. If no, please explain why.
No - backward compatibility maintained. The new get_plugin() mechanism is a drop-in replacement for manual loading with identical behavior. All existing plugin interfaces and configuration loading remain unchanged.
Database Migrations
Env Config
Relevant Docs
Related Issues or PRs
Dependencies Versions
Notes on Testing