Generative AI Google Brain

agents-agent-engine-memory-bank-tutorial-get-started-with-multimodal-agents-with-memory-bank

3 min read

Building a Multimodal Trip Planner with ADK on Vertex AI Agent Engine Memory Bank

Source notebook

Repo path: agents/agent_engine/memory_bank/tutorial_get_started_with_multimodal_agents_with_memory_bank.ipynb · Open on GitHub · advanced

Builds and deploys a multimodal ADK trip planner using Vertex AI Agent Engine Memory Bank.

Summary

The notebook teaches how to configure Vertex AI Agent Engine Memory Bank with managed and custom travel topics, then generate memories from audio, image, and video inputs. It builds an ADK LlmAgent with a memory preload tool and a custom trip budget FunctionTool, tests memory recall across sessions, and deploys the agent to Vertex AI Agent Engine. It also demonstrates explicit session consolidation into Memory Bank for both local and deployed agent flows.

Key code patterns

Memory Bank configuration

memory_bank_config = {
    "customization_configs": [{"memory_topics": travel_topics}],
    "similarity_search_config": {"embedding_model": f"projects/{PROJECT_ID}/locations/{LOCATION}/publishers/google/models/gemini-embedding-001"},
    "generation_config": {"model": f"projects/{PROJECT_ID}/locations/{LOCATION}/publishers/google/models/gemini-2.5-flash"},
}

Defines custom memory topics plus embedding and generation models for Memory Bank.

Generate multimodal memories

response = self.client.agent_engines.memories.generate(
    name=self.agent_engine_name,
    direct_contents_source={"events": events},
    scope={"user_id": user_id},
    config={"wait_for_completion": True},
)

Sends text plus file_data events to Memory Bank and scopes extracted memories to a user.

ADK agent with memory and tool

trip_planner_agent = LlmAgent(
    model="gemini-2.5-flash",
    name="TripPlanner",
    instruction=instruction,
    tools=[tools.preload_memory_tool.PreloadMemoryTool(), budget_tool],
)

Combines automatic memory retrieval with a custom FunctionTool for trip budget estimates.

Persist session to Memory Bank

final_session_state = await session_service.get_session(
    app_name=APP_NAME, user_id=USER_ID, session_id=session.id
)
await memory_bank_service.add_session_to_memory(final_session_state)

Consolidates a multimodal conversation into long-term Memory Bank storage.

Deploy ADK agent

remote_mm_adk_agent = client.agent_engines.create(
    agent=trip_planner_agent,
    config={
        "context_spec": {"memory_bank_config": memory_bank_config},
        "requirements": requirements,
        "staging_bucket": BUCKET_URI,
    },
)

Packages the local ADK agent and deploys it as a Vertex AI Agent Engine endpoint.

Models & APIs used

  • Models: gemini-embedding-001, gemini-2.5-flash
  • APIs / services: Vertex AI, Vertex AI Agent Engine, Vertex AI Memory Bank, Cloud Storage
  • SDKs / libraries: google-cloud-aiplatform, google-adk, google-genai, vertexai

When to use this

Use this pattern to build a deployed ADK agent that remembers user-specific multimodal context across sessions.

Gotchas & caveats

  • Vertex AI Memory Bank is in Preview.
  • A Google Cloud project with the Vertex AI API enabled is required.
  • Colab requires explicit user authentication.
  • The notebook creates a Cloud Storage staging bucket with gsutil.
  • PROJECT_ID, LOCATION, BUCKET_NAME, GOOGLE_CLOUD_PROJECT, GOOGLE_CLOUD_LOCATION, and GOOGLE_GENAI_USE_VERTEXAI must be configured.
  • Image, video, and audio helpers reject unsupported file extensions.
  • Memory persistence requires explicit add_session_to_memory after local or remote conversations.
  • Agent Engine deployment uses a staging bucket and can take a few minutes while building infrastructure.

Best practices

  • Use managed topics for common memory categories and custom topics for domain-specific memory extraction.
  • Provide text context alongside file_data so multimodal memories are grounded in user intent.
  • Scope generated memories with user_id.
  • Use PreloadMemoryTool so relevant memories are fetched at the start of a conversation turn.
  • Instruct the agent not to make up facts when memories are unavailable.
  • Clean up Agent Engine resources after the tutorial to avoid costs.

Graph View

Backlinks