Generative AI Google Brain

gemini-agent-engine-tutorial-vertex-ai-search-rag-agent

4 min read

Building a Conversational Search Agent with Agent Engine and RAG on Vertex AI Search

Source notebook

Repo path: gemini/agent-engine/tutorial_vertex_ai_search_rag_agent.ipynb · Open on GitHub · intermediate

Builds and deploys a Gemini movie-search RAG agent with LangChain, Agent Engine, and Vertex AI Search.

Summary

This notebook teaches how to build a conversational search agent that uses Gemini with Python tools and LangChain reasoning. It indexes a Kaggle Movies data store in Vertex AI Search, wraps search as a tool, tests the agent locally, deploys it to Agent Engine, grants Discovery Engine access, and queries the remote agent with follow-up questions.

Key code patterns

Initialize Vertex AI

PROJECT_ID = "[your-project-id]"
LOCATION = "us-central1"
STAGING_BUCKET = f"gs://{PROJECT_ID}-agent-engine-staging"
 
import vertexai
vertexai.init(project=PROJECT_ID, location=LOCATION)

Sets the project and region used by the local agent and Agent Engine deployment.

Vertex AI Search tool

def search_kaggle_movies(query: str) -> str:
    from langchain_google_community import VertexAISearchRetriever
    retriever = VertexAISearchRetriever(
        project_id=PROJECT_ID,
        data_store_id=DATA_STORE_ID,
        location_id=LOCATION_ID,
        engine_data_type=1,
        max_documents=10,
    )
    return str(retriever.invoke(query))

Exposes Vertex AI Search retrieval as a Python tool the agent can call.

Session-aware prompt

prompt = {
    "history": lambda x: x["history"],
    "input": lambda x: x["input"],
    "agent_scratchpad": lambda x: format_to_tool_messages(x["intermediate_steps"]),
} | prompts.ChatPromptTemplate.from_messages([
    prompts.MessagesPlaceholder(variable_name="history"),
    ("user", "{input}"),
    prompts.MessagesPlaceholder(variable_name="agent_scratchpad"),
])

Combines chat history, user input, and tool scratchpad messages for multi-turn reasoning.

LangChain Agent Engine agent

agent = LangchainAgent(
    prompt=prompt,
    model="gemini-2.5-flash",
    chat_history=get_session_history,
    model_kwargs={"temperature": 0},
    tools=[search_kaggle_movies],
    agent_executor_kwargs={"return_intermediate_steps": True},
)

Binds Gemini, chat history, tool use, and LangChain execution into an Agent Engine-compatible agent.

Deploy remote agent

client = vertexai.Client(project=PROJECT_ID, location=LOCATION)
remote_agent = client.agent_engines.create(
    agent=agent,
    config={
        "staging_bucket": STAGING_BUCKET,
        "requirements": [
            "google-cloud-aiplatform[agent_engines,langchain]",
            "langchain-google-community",
            "google-cloud-discoveryengine",
        ],
    },
)

Packages the tested local agent with runtime dependencies and deploys it to Agent Engine.

Grant search access

service = discovery.build("cloudresourcemanager", "v1")
project_number = service.projects().get(projectId=PROJECT_ID).execute()["projectNumber"]
 
!gcloud projects add-iam-policy-binding {PROJECT_ID} \
  --member=serviceAccount:service-{project_number}@gcp-sa-aiplatform-re.iam.gserviceaccount.com \
  --role=roles/discoveryengine.editor

Allows the deployed Agent Engine service account to query the Vertex AI Search data store.

Models & APIs used

  • Models: gemini-2.5-flash
  • APIs / services: Vertex AI, Agent Engine, Vertex AI Search, Cloud Resource Manager API, Cloud Storage
  • SDKs / libraries: vertexai, google-cloud-aiplatform[agent_engines,langchain], langchain, langchain-google-community, google-cloud-discoveryengine, google-api-python-client

When to use this

Use this pattern when you need a deployed conversational RAG agent over structured records indexed in Vertex AI Search.

Gotchas & caveats

  • Requires Vertex AI API, Vertex AI Search API, and Resource Manager API to be enabled.
  • Colab requires auth.authenticate_user(); Vertex AI Workbench does not require that Colab-only step.
  • The Vertex AI Search data store and a search app must both exist for the LangChain retriever to work.
  • Enterprise edition features and Generative Responses must be enabled for the Vertex AI Search app.
  • Data import and indexing can take about 5 to 10 minutes; searches may return empty results before indexing completes.
  • The remote Agent Engine service account needs roles/discoveryengine.editor to retrieve documents.
  • Deployment uses a Cloud Storage staging bucket and billable Google Cloud components.

Best practices

  • Test the Vertex AI Search function directly before wiring it into the agent.
  • Test the agent locally before deploying it to Agent Engine.
  • Use ChatMessageHistory and session_id to preserve conversational context across follow-up questions.
  • Set deployment requirements explicitly so Agent Engine has the needed LangChain and Discovery Engine packages.
  • Use temperature 0 for the demonstrated search agent behavior.
  • Delete the deployed agent and optionally the staging bucket after experimentation to avoid unexpected charges.

Graph View

Backlinks