Skip to content

Commit 566cf93

Browse files
xuechendipre-commit-ci[bot]XuhuiRen
authored
Add new DocIndexRetriever example (#405)
* Add DocIndexRetriever example Signed-off-by: Chendi.Xue <[email protected]> --------- Signed-off-by: Chendi.Xue <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: XuhuiRen <[email protected]>
1 parent 7719755 commit 566cf93

File tree

7 files changed

+594
-0
lines changed

7 files changed

+594
-0
lines changed

‎DocIndexRetriever/README.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# DocRetriever Application
2+
3+
DocRetriever are the most widely adopted use case for leveraging the different methodologies to match user query against a set of free-text records. DocRetriever is essential to RAG system, which bridges the knowledge gap by dynamically fetching relevant information from external sources, ensuring that responses generated remain factual and current. The core of this architecture are vector databases, which are instrumental in enabling efficient and semantic retrieval of information. These databases store data as vectors, allowing RAG to swiftly access the most pertinent documents or data points based on semantic similarity.
4+
5+
## We provided DocRetriever with different deployment infra
6+
7+
- [docker xeon version](docker/xeon/) => minimum endpoints, easy to setup
8+
- [docker gaudi version](docker/gaudi/) => with extra tei_gaudi endpoint, faster

‎DocIndexRetriever/docker/Dockerfile‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# Copyright (C) 2024 Intel Corporation
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
FROM python:3.11-slim
5+
6+
COPY GenAIComps /home/user/GenAIComps
7+
8+
RUN apt-get update -y && apt-get install -y --no-install-recommends --fix-missing \
9+
libgl1-mesa-glx \
10+
libjemalloc-dev \
11+
vim \
12+
git
13+
14+
RUN useradd -m -s /bin/bash user && \
15+
mkdir -p /home/user && \
16+
chown -R user /home/user/
17+
18+
WORKDIR /home/user/GenAIComps
19+
RUN pip install --no-cache-dir --upgrade pip && \
20+
pip install --no-cache-dir -r /home/user/GenAIComps/requirements.txt
21+
22+
COPY GenAIExamples/DocIndexRetriever/docker/retrieval_tool.py /home/user/retrieval_tool.py
23+
24+
ENV PYTHONPATH=$PYTHONPATH:/home/user/GenAIComps
25+
26+
USER user
27+
28+
WORKDIR /home/user
29+
30+
ENTRYPOINT ["python", "retrieval_tool.py"]

‎DocIndexRetriever/docker/README.md‎

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
# DocRetriever Application
2+
3+
DocRetriever are the most widely adopted use case for leveraging the different methodologies to match user query against a set of free-text records. DocRetriever is essential to RAG system, which bridges the knowledge gap by dynamically fetching relevant information from external sources, ensuring that responses generated remain factual and current. The core of this architecture are vector databases, which are instrumental in enabling efficient and semantic retrieval of information. These databases store data as vectors, allowing RAG to swiftly access the most pertinent documents or data points based on semantic similarity.
4+
5+
### 1. Build Images for necessary microservices. (This step will not needed after docker image released)
6+
7+
- Embedding TEI Image
8+
9+
```bash
10+
git clone https://github.com/opea-project/GenAIComps.git
11+
cd GenAIComps
12+
docker build -t opea/embedding-tei:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/embeddings/langchain/docker/Dockerfile .
13+
```
14+
15+
- Retriever Vector store Image
16+
17+
```bash
18+
docker build -t opea/retriever-redis:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/retrievers/langchain/redis/docker/Dockerfile .
19+
```
20+
21+
- Rerank TEI Image
22+
23+
```bash
24+
docker build -t opea/reranking-tei:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/reranks/tei/docker/Dockerfile .
25+
```
26+
27+
- Dataprep Image
28+
29+
```bash
30+
docker build -t opea/dataprep-on-ray-redis:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/dataprep/redis/langchain_ray/docker/Dockerfile .
31+
```
32+
33+
### 2. Build Images for MegaService
34+
35+
```bash
36+
cd ..
37+
git clone https://github.com/opea-project/GenAIExamples.git
38+
docker build --no-cache -t opea/doc-index-retriever:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f GenAIExamples/DocIndexRetriever/docker/Dockerfile .
39+
```
40+
41+
### 3. Start all the services Docker Containers
42+
43+
```bash
44+
export host_ip="YOUR IP ADDR"
45+
export HUGGINGFACEHUB_API_TOKEN=${your_hf_api_token}
46+
export EMBEDDING_MODEL_ID="BAAI/bge-base-en-v1.5"
47+
export RERANK_MODEL_ID="BAAI/bge-reranker-base"
48+
export TEI_EMBEDDING_ENDPOINT="http://${host_ip}:8090"
49+
export TEI_RERANKING_ENDPOINT="http://${host_ip}:8808"
50+
export TGI_LLM_ENDPOINT="http://${host_ip}:8008"
51+
export REDIS_URL="redis://${host_ip}:6379"
52+
export INDEX_NAME="rag-redis"
53+
export MEGA_SERVICE_HOST_IP=${host_ip}
54+
export EMBEDDING_SERVICE_HOST_IP=${host_ip}
55+
export RETRIEVER_SERVICE_HOST_IP=${host_ip}
56+
export RERANK_SERVICE_HOST_IP=${host_ip}
57+
export LLM_SERVICE_HOST_IP=${host_ip}
58+
export BACKEND_SERVICE_ENDPOINT="http://${host_ip}:8000/v1/retrievaltool"
59+
export DATAPREP_SERVICE_ENDPOINT="http://${host_ip}:6007/v1/dataprep"
60+
export llm_hardware='xeon' #xeon, xpu, gaudi
61+
cd GenAIExamples/DocIndexRetriever/docker/${llm_hardware}/
62+
docker compose -f docker-compose.yaml up -d
63+
```
64+
65+
### 3. Validation
66+
67+
Add Knowledge Base via HTTP Links:
68+
69+
```bash
70+
curl -X POST "http://${host_ip}:6007/v1/dataprep" \
71+
-H "Content-Type: multipart/form-data" \
72+
-F 'link_list=["https://opea.dev"]'
73+
74+
# expected output
75+
{"status":200,"message":"Data preparation succeeded"}
76+
```
77+
78+
Retrieval from KnowledgeBase
79+
80+
```bash
81+
curl http://${host_ip}:8889/v1/retrievaltool -X POST -H "Content-Type: application/json" -d '{
82+
"text": "Explain the OPEA project?"
83+
}'
84+
85+
# expected output
86+
{"id":"354e62c703caac8c547b3061433ec5e8","reranked_docs":[{"id":"06d5a5cefc06cf9a9e0b5fa74a9f233c","text":"Close SearchsearchMenu WikiNewsCommunity Daysx-twitter linkedin github searchStreamlining implementation of enterprise-grade Generative AIEfficiently integrate secure, performant, and cost-effective Generative AI workflows into business value.TODAYOPEA..."}],"initial_query":"Explain the OPEA project?"}
87+
```
88+
89+
### 4. Trouble shooting
90+
91+
1. check all containers are alive
92+
93+
```bash
94+
# redis vector store
95+
docker container logs redis-vector-db
96+
# dataprep to redis microservice, input document files
97+
docker container logs dataprep-redis-server
98+
99+
# embedding microservice
100+
curl http://${host_ip}:6000/v1/embeddings \
101+
-X POST \
102+
-d '{"text":"Explain the OPEA project"}' \
103+
-H 'Content-Type: application/json' > query
104+
docker container logs embedding-tei-server
105+
106+
# if you used tei-gaudi
107+
docker container logs tei-embedding-gaudi-server
108+
109+
# retriever microservice, input embedding output docs
110+
curl http://${host_ip}:7000/v1/retrieval \
111+
-X POST \
112+
-d @query \
113+
-H 'Content-Type: application/json' > rerank_query
114+
docker container logs retriever-redis-server
115+
116+
117+
# reranking microservice
118+
curl http://${host_ip}:8000/v1/reranking \
119+
-X POST \
120+
-d @rerank_query \
121+
-H 'Content-Type: application/json' > output
122+
docker container logs reranking-tei-server
123+
124+
# megaservice gateway
125+
docker container logs doc-index-retriever-server
126+
```

‎DocIndexRetriever/docker/gaudi/docker_compose.yaml‎

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
2+
# Copyright (C) 2024 Intel Corporation
3+
# SPDX-License-Identifier: Apache-2.0
4+
5+
version: "3.8"
6+
7+
services:
8+
redis-vector-db:
9+
image: redis/redis-stack:7.2.0-v9
10+
container_name: redis-vector-db
11+
ports:
12+
- "16379:6379"
13+
- "8001:8001"
14+
dataprep-redis-service:
15+
image: opea/dataprep-on-ray-redis:latest
16+
container_name: dataprep-redis-server
17+
depends_on:
18+
- redis-vector-db
19+
ports:
20+
- "6007:6007"
21+
environment:
22+
no_proxy: ${no_proxy}
23+
http_proxy: ${http_proxy}
24+
https_proxy: ${https_proxy}
25+
REDIS_URL: ${REDIS_URL}
26+
INDEX_NAME: ${INDEX_NAME}
27+
tei-embedding-service:
28+
image: ghcr.io/huggingface/tei-gaudi:latest
29+
container_name: tei-embedding-gaudi-server
30+
ports:
31+
- "8090:80"
32+
volumes:
33+
- "./data:/data"
34+
runtime: habana
35+
cap_add:
36+
- SYS_NICE
37+
ipc: host
38+
environment:
39+
no_proxy: ${no_proxy}
40+
http_proxy: ${http_proxy}
41+
https_proxy: ${https_proxy}
42+
HABANA_VISIBLE_DEVICES: all
43+
OMPI_MCA_btl_vader_single_copy_mechanism: none
44+
MAX_WARMUP_SEQUENCE_LENGTH: 512
45+
command: --model-id ${EMBEDDING_MODEL_ID}
46+
embedding:
47+
image: opea/embedding-tei:latest
48+
container_name: embedding-tei-server
49+
ports:
50+
- "6000:6000"
51+
ipc: host
52+
depends_on:
53+
- tei-embedding-service
54+
environment:
55+
no_proxy: ${no_proxy}
56+
http_proxy: ${http_proxy}
57+
https_proxy: ${https_proxy}
58+
TEI_EMBEDDING_ENDPOINT: ${TEI_EMBEDDING_ENDPOINT}
59+
LANGCHAIN_API_KEY: ${LANGCHAIN_API_KEY}
60+
LANGCHAIN_TRACING_V2: ${LANGCHAIN_TRACING_V2}
61+
LANGCHAIN_PROJECT: "opea-embedding-service"
62+
restart: unless-stopped
63+
retriever:
64+
image: opea/retriever-redis:latest
65+
container_name: retriever-redis-server
66+
depends_on:
67+
- redis-vector-db
68+
ports:
69+
- "7000:7000"
70+
ipc: host
71+
environment:
72+
no_proxy: ${no_proxy}
73+
http_proxy: ${http_proxy}
74+
https_proxy: ${https_proxy}
75+
REDIS_URL: ${REDIS_URL}
76+
INDEX_NAME: ${INDEX_NAME}
77+
LANGCHAIN_API_KEY: ${LANGCHAIN_API_KEY}
78+
LANGCHAIN_TRACING_V2: ${LANGCHAIN_TRACING_V2}
79+
LANGCHAIN_PROJECT: "opea-retriever-service"
80+
restart: unless-stopped
81+
reranking:
82+
image: opea/reranking-tei:latest
83+
container_name: reranking-tei-server
84+
ports:
85+
- "18000:8000"
86+
ipc: host
87+
entrypoint: python local_reranking.py
88+
environment:
89+
no_proxy: ${no_proxy}
90+
http_proxy: ${http_proxy}
91+
https_proxy: ${https_proxy}
92+
TEI_RERANKING_ENDPOINT: ${TEI_RERANKING_ENDPOINT}
93+
HUGGINGFACEHUB_API_TOKEN: ${HUGGINGFACEHUB_API_TOKEN}
94+
HF_HUB_DISABLE_PROGRESS_BARS: 1
95+
HF_HUB_ENABLE_HF_TRANSFER: 0
96+
LANGCHAIN_API_KEY: ${LANGCHAIN_API_KEY}
97+
LANGCHAIN_TRACING_V2: ${LANGCHAIN_TRACING_V2}
98+
LANGCHAIN_PROJECT: "opea-reranking-service"
99+
restart: unless-stopped
100+
doc-index-retriever-server:
101+
image: opea/doc-index-retriever:latest
102+
container_name: doc-index-retriever-server
103+
depends_on:
104+
- redis-vector-db
105+
- tei-embedding-service
106+
- embedding
107+
- retriever
108+
- reranking
109+
ports:
110+
- "8889:8889"
111+
environment:
112+
- no_proxy=${no_proxy}
113+
- https_proxy=${https_proxy}
114+
- http_proxy=${http_proxy}
115+
- MEGA_SERVICE_HOST_IP=${MEGA_SERVICE_HOST_IP}
116+
- EMBEDDING_SERVICE_HOST_IP=${EMBEDDING_SERVICE_HOST_IP}
117+
- RETRIEVER_SERVICE_HOST_IP=${RETRIEVER_SERVICE_HOST_IP}
118+
- RERANK_SERVICE_HOST_IP=${RERANK_SERVICE_HOST_IP}
119+
- LLM_SERVICE_HOST_IP=${LLM_SERVICE_HOST_IP}
120+
ipc: host
121+
restart: always
122+
123+
networks:
124+
default:
125+
driver: bridge

‎DocIndexRetriever/docker/retrieval_tool.py‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
# Copyright (C) 2024 Intel Corporation
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
import asyncio
5+
import os
6+
7+
from comps import MicroService, RetrievalToolGateway, ServiceOrchestrator, ServiceType
8+
9+
MEGA_SERVICE_HOST_IP = os.getenv("MEGA_SERVICE_HOST_IP", "0.0.0.0")
10+
MEGA_SERVICE_PORT = os.getenv("MEGA_SERVICE_PORT", 8889)
11+
EMBEDDING_SERVICE_HOST_IP = os.getenv("EMBEDDING_SERVICE_HOST_IP", "0.0.0.0")
12+
EMBEDDING_SERVICE_PORT = os.getenv("EMBEDDING_SERVICE_PORT", 6000)
13+
RETRIEVER_SERVICE_HOST_IP = os.getenv("RETRIEVER_SERVICE_HOST_IP", "0.0.0.0")
14+
RETRIEVER_SERVICE_PORT = os.getenv("RETRIEVER_SERVICE_PORT", 7000)
15+
RERANK_SERVICE_HOST_IP = os.getenv("RERANK_SERVICE_HOST_IP", "0.0.0.0")
16+
RERANK_SERVICE_PORT = os.getenv("RERANK_SERVICE_PORT", 8000)
17+
18+
19+
class RetrievalToolService:
20+
def __init__(self, host="0.0.0.0", port=8000):
21+
self.host = host
22+
self.port = port
23+
self.megaservice = ServiceOrchestrator()
24+
25+
def add_remote_service(self):
26+
embedding = MicroService(
27+
name="embedding",
28+
host=EMBEDDING_SERVICE_HOST_IP,
29+
port=EMBEDDING_SERVICE_PORT,
30+
endpoint="/v1/embeddings",
31+
use_remote_service=True,
32+
service_type=ServiceType.EMBEDDING,
33+
)
34+
retriever = MicroService(
35+
name="retriever",
36+
host=RETRIEVER_SERVICE_HOST_IP,
37+
port=RETRIEVER_SERVICE_PORT,
38+
endpoint="/v1/retrieval",
39+
use_remote_service=True,
40+
service_type=ServiceType.RETRIEVER,
41+
)
42+
rerank = MicroService(
43+
name="rerank",
44+
host=RERANK_SERVICE_HOST_IP,
45+
port=RERANK_SERVICE_PORT,
46+
endpoint="/v1/reranking",
47+
use_remote_service=True,
48+
service_type=ServiceType.RERANK,
49+
)
50+
51+
self.megaservice.add(embedding).add(retriever).add(rerank)
52+
self.megaservice.flow_to(embedding, retriever)
53+
self.megaservice.flow_to(retriever, rerank)
54+
self.gateway = RetrievalToolGateway(megaservice=self.megaservice, host="0.0.0.0", port=self.port)
55+
56+
57+
if __name__ == "__main__":
58+
chatqna = RetrievalToolService(host=MEGA_SERVICE_HOST_IP, port=MEGA_SERVICE_PORT)
59+
chatqna.add_remote_service()

0 commit comments

Comments
 (0)