Generative AI Google Brain

gemini-sample-apps-photo-discovery-ag-web-ag-setup-re

3 min read

Building a photo recognition agent: Agent Engine setup

Source notebook

Repo path: gemini/sample-apps/photo-discovery/ag-web/ag_setup_re.ipynb · Open on GitHub · intermediate

Sets up and deploys a Gemini LangChain agent on Agent Engine with Wikipedia and Vertex AI Search tools.

Summary

This notebook teaches how to configure Vertex AI Agent Engine for a photo recognition agent demo by installing dependencies, initializing a project, region, and staging bucket, and defining Python tools. It builds a LangchainAgent with gemini-2.0-flash, tests it locally, then deploys it with agent_engines.create. The tools query Wikipedia for object knowledge and call a Cloud Run /ask_gms endpoint for Google Merch Shop product details from Vertex AI Search.

Key code patterns

Initialize Vertex AI

PROJECT_ID = "<YOUR GOOGLE CLOUD PROJECT ID>"
LOCATION = "us-central1"
STAGING_BUCKET = "gs://<YOUR GCS BUCKET>"
 
import vertexai
vertexai.init(project=PROJECT_ID, location=LOCATION, staging_bucket=STAGING_BUCKET)

Agent Engine setup depends on a configured project, region, and staging bucket.

Wikipedia tool

def query_with_wikipedia(query: str = "Fallingwater"):
    wiki_title = search_wiki_title(query)
    wiki_full_text = get_wiki_full_text(wiki_title)
    return {"answer": wiki_full_text}

Wraps Wikipedia API calls as a Python function that can be passed as an agent tool.

Cloud Run search tool

def find_product_from_googleshop(product_name: str, product_description: str):
    params = {"query": product_name + " " + product_description}
    response = requests.get(GOOGLE_SHOP_VERTEXAI_SEARCH_URL, params)
    item = response.json()
    return {"productDetails": f"{item['gms_name']} is a product sold at Google Merch Shop. The price is {item['price']}. {item['gms_desc']}. You can buy the product at their web site: {item['link']}"}

Connects the agent to a deployed /ask_gms Cloud Run endpoint for Vertex AI Search product lookup.

Create and deploy agent

model_name = "gemini-2.0-flash"
agent = reasoning_engines.LangchainAgent(
    model=model_name,
    tools=[query_with_wikipedia, find_product_from_googleshop],
    agent_executor_kwargs={"return_intermediate_steps": True},
)
remote_agent = agent_engines.create(
    agent,
    requirements=["google-cloud-aiplatform[langchain,agent_engines]", "requests"],
)

Shows the local LangChain agent configuration and the Agent Engine deployment call.

Models & APIs used

  • Models: gemini-2.0-flash
  • APIs / services: Vertex AI, Agent Engine, Vertex AI Search, Cloud Run, Cloud Storage, Wikipedia API
  • SDKs / libraries: google-cloud-aiplatform, google-cloud-discoveryengine, vertexai, langchain, requests, IPython

When to use this

Use this pattern when you need a deployed Agent Engine agent that routes text queries to Python tools and a Cloud Run Vertex AI Search endpoint.

Gotchas & caveats

  • Requires an existing Google Cloud project and the Vertex AI API enabled.
  • The notebook installs pinned packages and then restarts the Jupyter kernel.
  • STAGING_BUCKET must be a valid gs:// bucket for Vertex AI initialization.
  • GOOGLE_SHOP_VERTEXAI_SEARCH_URL is marked please change and requires the /ag-web/app/deploy.sh Cloud Run deployment first.
  • Wikipedia helper functions assume search results and page extracts exist; no error handling is shown.

Best practices

  • Pin setup dependencies for the notebook environment.
  • Restart the runtime after installing packages.
  • Initialize Vertex AI with project, location, and staging bucket before creating Agent Engine resources.
  • Define tool functions with typed arguments, docstrings, and dictionary responses.
  • Test the agent locally with agent.query before deploying it remotely.
  • Pass runtime requirements to agent_engines.create for the deployed agent.

Graph View

Backlinks