This repository contains code accompanying the blog [Enable Multi-Agents + Classic ML using Databricks and DSPy!] https://medium.com/@austinchoi/enable-multi-agents-classic-ml-using-databricks-and-dspy-918f78c16e3a
This is primarily an educational demonstration of how you can design Agents to work together to answer a wide variety of questions or figure out and complete tasks in the face of adversity.
Through this demo, you should learn the following:
- You should be using LLMs + Traditional ML (The Vision Agent will demonstrate this)
- Tool calling is almost mandatory to ensure up to date and accurate information (The Pokemon Agent will demonstrate this)
- We can make RAG better with Agents (The Databricks Agent will demonstrate)
A chatbot is not the only use case for this code. In fact, it is likely other Use Case like ETL pipelines would benefit more from this solution. However, to demonstrate the interaction between agents, it is show through a Gradio Interface
Below is a diagram of how the Agents work together to answer a user's question:
Multi-Agent Recreation Much of the set up occurs in the config file to set up your Databricks Environment
config This folder contains the notebooks and assets that will help recreate the demo one for one
| Library | License |
|---|---|
| dspy | MIT |
| langchain | MIT |
| lxml | BSD-3 |
| transformers | Apache 2.0 |
| unstructured | Apache 2.0 |
| llama-index | MIT |
| tesseract | Apache 2.0 |
| poppler-utils | MIT |
| textstat | MIT |
| tiktoken | MIT |
| evaluate | Apache 2.0 |
| torch | BSS-3 |
| pytesseract | Apache 2.0 |
| pillow | MIT-CMU |
| pypdf | BSD-3 |
| llama-index | MIT |