Generative AI Google Brain

gemini-rag-engine-rag-engine-eval-service-sdk

3 min read

Evaluating Vertex RAG Engine Generation with Vertex AI Python SDK for Gen AI Evaluation Service

Source notebook

Repo path: gemini/rag-engine/rag_engine_eval_service_sdk.ipynb · Open on GitHub · intermediate

Evaluates Vertex AI RAG Engine responses with a custom Gen AI Evaluation Service metric.

Summary

The notebook builds a Vertex AI RAG corpus from Cloud Storage files, retrieves context, and generates grounded answers with Gemini. It creates an evaluation dataset containing prompts, retrieved context, and generated responses. It then defines a custom pointwise metric for accuracy, completeness, and groundedness, runs an EvalTask, visualizes results, and cleans up the experiment run and corpus.

Key code patterns

Initialize GenAI and Vertex AI

PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT")
LOCATION = os.environ.get("GOOGLE_CLOUD_REGION", "us-central1")
client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)
vertexai.init(project=PROJECT_ID, location=LOCATION)

Sets project, region, GenAI client, and Vertex AI SDK state before creating RAG and evaluation resources.

Create RAG corpus

rag_corpus = rag.create_corpus(
    display_name="rag-eval-corpus",
    backend_config=rag.RagVectorDbConfig(
        rag_embedding_model_config=rag.RagEmbeddingModelConfig(
            vertex_prediction_endpoint=rag.VertexPredictionEndpoint(
                publisher_model=EMBEDDING_MODEL))))

Creates a Vertex AI RAG corpus backed by a first-party embedding model.

Import files with chunking

transformation_config = rag.TransformationConfig(
    chunking_config=rag.ChunkingConfig(chunk_size=512, chunk_overlap=100))
rag.import_files(
    corpus_name=rag_corpus.name,
    paths=[CURRENT_BUCKET_PATH],
    transformation_config=transformation_config)

Loads Cloud Storage documents into the corpus with explicit chunk size and overlap.

Generate grounded response

rag_retrieval_tool = Tool(retrieval=Retrieval(
    vertex_rag_store=VertexRagStore(
        rag_corpora=[rag_corpus.name], similarity_top_k=10,
        vector_distance_threshold=0.5)))
response = client.models.generate_content(
    model=MODEL_ID,
    contents=prompt,
    config=GenerateContentConfig(tools=[rag_retrieval_tool]))

Uses a Vertex RAG store as a retrieval tool for Gemini generation.

Run custom evaluation

metric = PointwiseMetric(
    metric="custom_question_answering_correctness",
    metric_prompt_template=PointwiseMetricPromptTemplate(
        criteria={"accuracy": "...", "completeness": "...", "groundedness": "..."},
        input_variables=["prompt", "retrieved_context"]))
eval_result = EvalTask(dataset=eval_dataset, metrics=[metric], experiment="test").evaluate()

Defines and runs a custom pointwise metric over the RAG evaluation dataset.

Models & APIs used

  • Models: gemini-2.5-flash, publishers/google/models/text-embedding-005
  • APIs / services: Vertex AI, Vertex AI RAG Engine, Gen AI Evaluation Service, Cloud Storage
  • SDKs / libraries: google-genai, vertexai, google-cloud-aiplatform[evaluation], pandas, tqdm

When to use this

Use this pattern when you need to evaluate grounded RAG responses from Vertex AI RAG Engine with a custom rubric.

Gotchas & caveats

  • Vertex AI API must be enabled for the selected Google Cloud project.
  • Colab requires user authentication before running Google Cloud calls.
  • The notebook requires a runtime restart after installing packages.
  • A Cloud Storage bucket is required before copying public data with gsutil rsync.
  • The Vertex RAG Data Service Agent needs Viewer access to the Cloud Storage bucket.
  • LOCATION defaults to us-central1 when GOOGLE_CLOUD_REGION is unset.
  • The RAG corpus example currently supports Google first-party embedding models.
  • The cleanup cell references aiplatform.ExperimentRun, but aiplatform is not imported in the shown notebook text.

Best practices

  • Use environment variables for GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_REGION when project values are not provided.
  • Configure chunk_size and chunk_overlap when importing files into the RAG corpus.
  • Separate retrieved context collection from grounded response generation before evaluation.
  • Evaluate prompts, retrieved_context, and response together in a pandas DataFrame.
  • Use a custom metric rubric that checks accuracy, completeness, and groundedness.
  • Delete the evaluation ExperimentRun and RAG corpus during cleanup.

Graph View

Backlinks