Multimodale Retrieval-Augmented Generation (RAG) mit der Gemini API in der Agent Platform Rezensionen

Debayan B. · Vor 21 Tage überprüft

Harish R. · Vor 21 Tage überprüft

Mohammed Sami U. · Vor 21 Tage überprüft

Sakthi V. · Vor 21 Tage überprüft

Saranishree G. · Vor 21 Tage überprüft

Suba.S G. · Vor 21 Tage überprüft

Jay J. · Vor 21 Tage überprüft

ClientError: 429 RESOURCE_EXHAUSTED. {'error': {'code': 429, 'message': 'Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. Try for 4nd time and not passed Task 4 error : ClientError: 429 RESOURCE_EXHAUSTED. {'error': {'code': 429, 'message': 'Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai.', 'status': 'RESOURCE_EXHAUSTED'}} # # Parameters for Gemini API call. # # reference for parameters: https://google.github.io/generative-ai-python/ # generation_config = types.GenerateContentConfig(temperature=0.2, max_output_tokens=2048) # # Set the safety settings if Gemini is blocking your content or you are facing "ValueError("Content has no parts")" error or "Exception occurred" in your data. # # ref for settings and thresholds: https://google.github.io/generative-ai-python/ # safety_settings = [ # types.SafetySetting( # category=types.HarmCategory.HARM_CATEGORY_HARASSMENT, # threshold=types.HarmBlockThreshold.BLOCK_NONE, # ), # types.SafetySetting( # category=types.HarmCategory.HARM_CATEGORY_HATE_SPEECH, # threshold=types.HarmBlockThreshold.BLOCK_NONE, # ), # types.SafetySetting( # category=types.HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT, # threshold=types.HarmBlockThreshold.BLOCK_NONE, # ), # types.SafetySetting( # category=types.HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT, # threshold=types.HarmBlockThreshold.BLOCK_NONE, # ), # ] # # You can also pass parameters and safety_setting to "get_gemini_response" function Inspect the processed text metadata The following cell will produce a metadata table which describes the different parts of text metadata, including: text: the original text from the page text_embedding_page: the embedding of the original text from the page chunk_text: the original text divided into smaller chunks chunk_number: the index of each text chunk text_embedding_chunk: the embedding of each text chunk text_metadata_df.head() --------------------------------------------------------------------------- NameError Traceback (most recent call last) Cell In[7], line 1 ----> 1 text_metadata_df.head() NameError: name 'text_metadata_df' is not defined Inspect the processed image metadata The following cell will produce a metadata table which describes the different parts of image metadata, including: img_desc: Gemini-generated textual description of the image. mm_embedding_from_text_desc_and_img: Combined embedding of image and its description, capturing both visual and textual information. mm_embedding_from_img_only: Image embedding without description, for comparison with description-based analysis. text_embedding_from_image_description: Separate text embedding of the generated description, enabling textual analysis and comparison. image_metadata_df.head() --------------------------------------------------------------------------- NameError Traceback (most recent call last) Cell In[8], line 1 ----> 1 image_metadata_df.head() NameError: name 'image_metadata_df' is not defined Import the helper functions to implement RAG You will be importing the following functions which will be used in the remainder of this notebook to implement RAG: get_similar_text_from_query(): Given a text query, finds text from the document which are relevant, using cosine similarity algorithm. It uses text embeddings from the metadata to compute and the results can be filtered by top score, page/chunk number, or embedding size. print_text_to_text_citation(): Prints the source (citation) and details of the retrieved text from the get_similar_text_from_query() function. get_similar_image_from_query(): Given an image path or an image, finds images from the document which are relevant. It uses image embeddings from the metadata. print_text_to_image_citation(): Prints the source (citation) and the details of retrieved images from the get_similar_image_from_query() function. get_gemini_response(): Interacts with a Gemini model to answer questions based on a combination of text and image inputs. display_images(): Displays a series of images provided as paths or PIL Image objects. from utils.intro_multimodal_rag_utils import ( display_images, get_gemini_response, get_similar_image_from_query, get_similar_text_from_query, print_text_to_image_citation, print_text_to_text_citation, ) Before implementing a multimodal RAG, let's take a step back and explore what you can achieve with just text or image embeddings alone. It will help to set the foundation for implementing a multimodal RAG, which you will be doing in the later part of the notebook. You can also use these essential elements together to build applications for multimodal use cases for extracting meaningful information from the document. Text Search Let's start the search with a simple question and see if the simple text search using text embeddings can answer it. The expected answer is to show the value of basic and diluted net income per share of Google for different share types. query = "I need details for basic and diluted net income per share of Class A, Class B, and Class C share for google?" Search similar text with text query # Matching user text query with "chunk_embedding" to find relevant chunks. matching_results_text = get_similar_text_from_query( query, text_metadata_df, column_name="text_embedding_chunk", top_n=3, chunk_text=True, ) # Print the matched text citations print_text_to_text_citation(matching_results_text, print_top=False, chunk_text=True) You can see that the first high score match does have what we are looking for, but upon closer inspection, it mentions that the information is available in the "following" table. The table data is available as an image rather than as text, and hence, the chances are you will miss the information unless you can find a way to process images and their data. However, Let's feed the relevant text chunk across the data into the Gemini model and see if it can get your desired answer by considering all the chunks across the document. This is like basic text-based RAG implementation. print("\n **** Result: ***** \n") # All relevant text chunk found across documents based on user query context = "\n".join( [value["chunk_text"] for key, value in matching_results_text.items()] ) instruction = f"""Answer the question with the given context. If the information is not available in the context, just return "not available in the context". Question: {query} Context: {context} Answer: """ # Prepare the model input model_input = instruction # Generate Gemini response with streaming output get_gemini_response( model, # we are passing Gemini model_input=model_input, stream=True, generation_config=types.GenerateContentConfig(temperature=0.2), ) You can see that it returned: "not available in the context" This is expected as discussed previously. No other text chunk (total 3) had the information you sought. This is because the information is only available in the images rather than in the text part of the document. Next, let's see if you can solve this problem by leveraging Gemini and Multimodal Embeddings. Note: We handcrafted examples in our document to simulate real-world cases where information is often embedded in charts, table, graphs, and other image-based elements and unavailable as plain text. Search similar images with text query Since plain text search didn't provide the desired answer and the information may be visually represented in a table or another image format, you will use multimodal capability of Gemini model for the similar task. The goal here also is to find an image similar to the text query. You may also print the citations to verify. query = "I need details for basic and diluted net income per share of Class A, Class B, and Class C share for google?" matching_results_image = get_similar_image_from_query( text_metadata_df, image_metadata_df, query=query, column_name="text_embedding_from_image_description", # Use image description text embedding image_emb=False, # Use text embedding instead of image embedding top_n=3, embedding_size=1408, ) # Markdown(print_text_to_image_citation(matching_results_image, print_top=True)) print("\n **** Result: ***** \n") # Display the top matching image display(matching_results_image[0]["image_object"]) Bingo! It found exactly what you were looking for. You wanted the details on Google's Class A, B, and C shares' basic and diluted net income, and guess what? This image fits the bill perfectly thanks to its descriptive metadata using Gemini. You can also send the image and its description to Gemini and get the answer as JSON: print("\n **** Result: ***** \n") # All relevant text chunk found across documents based on user query context = f"""Image: {matching_results_image[0]['image_object']} Description: {matching_results_image[0]['image_description']} """ instruction = f"""Answer the question in JSON format with the given context of Image and its Description. Only include value. Question: {query} Context: {context} Answer: """ # Prepare the model input model_input = instruction # Generate Gemini response with streaming output Markdown( get_gemini_response( model, # we are passing "gemini-3.5-flash-lite" model model_input=model_input, stream=True, generation_config=types.GenerateContentConfig(temperature=1), ) ) ## you can check the citations to probe further. ## check the "image description:" which is a description extracted through Gemini which helped search our query. Markdown(print_text_to_image_citation(matching_results_image, print_top=True)) Image Search Search similar image with image query Imagine searching for images, but instead of typing words, you use an actual image as the clue. You have a table with numbers about the cost of revenue for two years, and you want to find other images that look like it, from the same document or across multiple documents. Think of it like searching with a mini-map instead of a written address. It's a different way to ask, "Show me more stuff like this". So, instead of typing "cost of revenue 2020 2021 table", you show a picture of that table and say, "Find me more like this" For demonstration purposes, we will only be finding similar images that show the cost of revenue or similar values in a single document below. However, you can scale this design pattern to match (find relevant images) across multiple documents. # You can find a similar image as per the images you have in the metadata. # In this case, you have a table (picked from the same document source) and you would like to find similar tables in the document. image_query_path = "tac_table_revenue.png" # Print a message indicating the input image print("***Input image from user:***") # Display the input image PIL_Image.open(image_query_path) You expect to find tables (as images) that are similar in terms of "Other/Total cost of revenues." # Search for Similar Images Based on Input Image and Image Embedding matching_results_image = get_similar_image_from_query( text_metadata_df, image_metadata_df, query=query, # Use query text for additional filtering (optional) column_name="mm_embedding_from_img_only", # Use image embedding for similarity calculation image_emb=True, image_query_path=image_query_path, # Use input image for similarity calculation top_n=3, # Retrieve top 3 matching images embedding_size=1408, # Use embedding size of 1408 ) print("\n **** Result: ***** \n") # Display the Top Matching Image display( matching_results_image[0]["image_object"] ) # Display the top matching image object (Pillow Image) It did find a similar-looking image (table), which gives more detail about different revenue, expenses, income, and a few more details based on the given image. More importantly, both tables show numbers related to the "cost of revenue." You can also print the citation to see what it has matched. # Display citation details for the top matching image print_text_to_image_citation( matching_results_image, print_top=True ) # Print citation details for the top matching image # Check Other Matched Images (Optional) # You can access the other two matched images using: print("---------------Matched Images------------------\n") display_images( [ matching_results_image[0]["img_path"], matching_results_image[1]["img_path"], ], resize_ratio=0.5, ) The ability to identify similar text and images based on user input, using Gemini and embeddings, forms a crucial foundation for development of multimodal RAG systems, which you explore in the next section. Comparative reasoning Next, let's apply what you have done so far to doing comparative reasoning. For this example: Step 1: You will search all the images for a specific query Step 2: Send those images to Gemini to ask multiple questions, where it has to compare and provide you with answers. matching_results_image_query_1 = get_similar_image_from_query( text_metadata_df, image_metadata_df, query="Show me all the graphs that shows Google Class A cumulative 5-year total return", column_name="text_embedding_from_image_description", # Use image description text embedding # mm_embedding_from_img_only text_embedding_from_image_description image_emb=False, # Use text embedding instead of image embedding top_n=3, embedding_size=1408, ) # Check Matched Images # You can access the other two matched images using: print("---------------Matched Images------------------\n") display_images( [ matching_results_image_query_1[0]["img_path"], matching_results_image_query_1[1]["img_path"], ], resize_ratio=0.5, ) prompt = f""" Instructions: Compare the images and the Gemini extracted text provided as Context: to answer Question: Make sure to think thoroughly before answering the question and put the necessary steps to arrive at the answer in bullet points for easy explainability. Context: Image_1: {matching_results_image_query_1[0]["image_object"]} gemini_extracted_text_1: {matching_results_image_query_1[0]['image_description']} Image_2: {matching_results_image_query_1[1]["image_object"]} gemini_extracted_text_2: {matching_results_image_query_1[1]['image_description']} Question: - Key findings of Class A share? - What are the critical differences between the graphs for Class A Share? - What are the key findings of Class A shares concerning the S&P 500? - Which index best matches Class A share performance closely where Google is not already a part? Explain the reasoning. - Identify key chart patterns in both graphs. - Which index best matches Class A share performance closely where Google is not already a part? Explain the reasoning. """ # Generate Gemini response with streaming output rich_Markdown( get_gemini_response( model, # we are passing "gemini-3.5-flash-lite" model model_input=[prompt], stream=True, generation_config=types.GenerateContentConfig(temperature=1), ) ) ⚠️ Disclaimer: This is not a real investment advise and should not be taken seriously!! ⚠️ Multimodal retrieval augmented generation (RAG) Let's bring everything together to implement multimodal RAG. You will use all the elements that you've explored in previous sections to implement the multimodal RAG. These are the steps: Step 1: The user gives a query in text format where the expected information is available in the document and is embedded in images and text. Step 2: Find all text chunks from the pages in the documents using a method similar to the one you explored in Text Search. Step 3: Find all similar images from the pages based on the user query matched with image_description using a method identical to the one you explored in Image Search. Step 4: Combine all similar text and images found in steps 2 and 3 as context_text and context_images. Step 5: With the help of Gemini, we can pass the user query with text and image context found in steps 2 & 3. You can also add a specific instruction the model should remember while answering the user query. Step 6: Gemini produces the answer, and you can print the citations to check all relevant text and images used to address the query. Step 1: User query # this time we are not passing any images, but just a simple text query. query = """Questions: - What are the critical difference between various graphs for Class A Share? - Which index best matches Class A share performance closely where Google is not already a part? Explain the reasoning. - Identify key chart patterns for Google Class A shares. - What is cost of revenues, operating expenses and net income for 2020. Do mention the percentage change - What was the effect of Covid in the 2020 financial year? - What are the total revenues for APAC and USA for 2021? - What is deferred income taxes? - How do you compute net income per share? - What drove percentage change in the consolidated revenue and cost of revenue for the year 2021 and was there any effect of Covid? - What is the cause of 41% increase in revenue from 2020 to 2021 and how much is dollar change? """ Step 2: Get all relevant text chunks # Retrieve relevant chunks of text based on the query matching_results_chunks_data = get_similar_text_from_query( query, text_metadata_df, column_name="text_embedding_chunk", top_n=10, chunk_text=True, ) Step 3: Get all relevant images # Get all relevant images based on user query matching_results_image_fromdescription_data = get_similar_image_from_query( text_metadata_df, image_metadata_df, query=query, column_name="text_embedding_from_image_description", image_emb=False, top_n=10, embedding_size=1408, ) Step 4: Create context_text and context_images # combine all the selected relevant text chunks context_text = [] for key, value in matching_results_chunks_data.items(): context_text.append(value["chunk_text"]) final_context_text = "\n".join(context_text) # combine all the relevant images and their description generated by Gemini context_images = [] for key, value in matching_results_image_fromdescription_data.items(): context_images.extend( ["Image: ", value["image_object"], "Caption: ", value["image_description"]] ) Step 5: Pass context to Gemini prompt = f""" Instructions: Compare the images and the text provided as Context: to answer multiple Question: Make sure to think thoroughly before answering the question and put the necessary steps to arrive at the answer in bullet points for easy explainability. If unsure, respond, "Not enough context to answer". Context: - Text Context: {final_context_text} - Image Context: {context_images} {query} Answer: """ # Generate Gemini response with streaming output rich_Markdown( get_gemini_response( model, model_input=[prompt], stream=True, generation_config=types.GenerateContentConfig(temperature=1), ) ) Step 6: Print citations and references print("---------------Matched Images------------------\n") display_images( [ matching_results_image_fromdescription_data[0]["img_path"], matching_results_image_fromdescription_data[1]["img_path"], matching_results_image_fromdescription_data[2]["img_path"], matching_results_image_fromdescription_data[3]["img_path"], ], resize_ratio=0.5, ) # Image citations. You can check how Gemini generated metadata helped in grounding the answer. print_text_to_image_citation( matching_results_image_fromdescription_data, print_top=False ) # Text citations print_text_to_text_citation( matching_results_chunks_data, print_top=False, chunk_text=True, ) Conclusions Congratulations on making it through this multimodal RAG notebook! While multimodal RAG can be quite powerful, note that it can face some limitations: Data dependency: Needs high-quality paired text and visuals. Computationally demanding: Processing multimodal data is resource-intensive. Domain specific: Models trained on general data may not shine in specialized fields like medicine. Black box: Understanding how these models work can be tricky, hindering trust and adoption. Despite these challenges, multimodal RAG represents a significant step towards search and retrieval systems that can handle diverse, multimodal data.

Ramdanita -. · Vor 21 Tage überprüft

ClientError: 429 RESOURCE_EXHAUSTED. {'error': {'code': 429, 'message': 'Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. Try for 2nd time and not passed

Ramdanita -. · Vor 21 Tage überprüft

ClientError: 429 RESOURCE_EXHAUSTED. {'error': {'code': 429, 'message': 'Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. Try for 2nd time and not passed

Ramdanita -. · Vor 22 Tage überprüft

Armando A. · Vor 22 Tage überprüft

Nivedith P. · Vor 22 Tage überprüft

Kavya A. · Vor 22 Tage überprüft

Henrique C. · Vor 22 Tage überprüft

navya M. · Vor 22 Tage überprüft

Debayan B. · Vor 22 Tage überprüft

Nithiarasu G. · Vor 22 Tage überprüft

Akash M. · Vor 22 Tage überprüft

SELVI G. · Vor 22 Tage überprüft

Ramya K. · Vor 22 Tage überprüft

Ruth Happlin V. · Vor 22 Tage überprüft

SEONGMIN K. · Vor 22 Tage überprüft

Vishnupriya K. · Vor 22 Tage überprüft

Hemalatha H. · Vor 22 Tage überprüft

Suba.S G. · Vor 22 Tage überprüft