Generative AI Google Brain

gemini-computer-use-intro-computer-use

3 min read

Intro to Computer Use with Gemini

Source notebook

Repo path: gemini/computer-use/intro_computer_use.ipynb · Open on GitHub · intermediate

Builds a Gemini Computer Use browser agent loop with Playwright and screenshots.

Summary

This notebook introduces the Gemini Computer Use tool for automating browser tasks by sending screenshots to Gemini and executing returned UI function calls. It demonstrates setup with google-genai and Playwright, configures the Computer Use tool, executes actions such as navigate, click, and type, then returns screenshots as function responses. It finishes by combining these steps into a multi-turn agent loop with conversation history and browser cleanup.

Key code patterns

Create Vertex GenAI client

client = genai.Client(
    enterprise=True,
    project=PROJECT_ID,
    location="global",
)

Connects the notebook to the Gen AI service on Vertex AI.

Configure Computer Use tool

config = GenerateContentConfig(
    tools=[Tool(computer_use=ComputerUse(
        environment=Environment.ENVIRONMENT_BROWSER,
        excluded_predefined_functions=["drag_and_drop"],
    ))]
)

Enables browser computer-use actions and excludes an unsupported action.

Send prompt with screenshot

contents = [Content(role="user", parts=[
    Part(text=initial_prompt),
    Part.from_bytes(data=screenshot, mime_type="image/png"),
])]
response = client.models.generate_content(
    model=MODEL_ID, contents=contents, config=config
)

Gives the model both the goal and current visual browser state.

Execute function calls

if function_call.name == "navigate":
    await page.goto(function_call.args["url"])
elif function_call.name == "click_at":
    await page.mouse.click(actual_x, actual_y)
elif function_call.name == "type_text_at":
    await page.keyboard.type(function_call.args["text"])

Maps model-requested UI actions to Playwright browser operations.

Return updated state

Part(function_response=FunctionResponse(
    name=name,
    response={"url": current_url},
    parts=[Part(inline_data=FunctionResponseBlob(
        mime_type="image/png", data=screenshot))],
))

Feeds the post-action URL and screenshot back into the conversation.

Models & APIs used

When to use this

Use this pattern to prototype Gemini-driven browser automation that observes screenshots and performs UI actions through Playwright.

Gotchas & caveats

  • Requires google-genai and playwright installation plus Playwright browser setup.
  • Linux environments require playwright install-deps.
  • Colab requires auth.authenticate_user().
  • PROJECT_ID must be set directly or via GOOGLE_CLOUD_PROJECT.
  • The notebook uses location global.
  • Headless browser mode is required for this environment.
  • Production handlers need all supported actions unless actions are explicitly excluded.
  • Safety warnings require user confirmation before proceeding.

Best practices

  • Run Computer Use agents in a secure controlled environment such as a sandboxed VM, container, or dedicated browser profile.
  • Implement client-side action handling and screenshot capture.
  • Append both model responses and function responses to conversation history.
  • Return a FunctionResponse for each executed action, including parallel actions.
  • Handle missing candidates because safety filters may return no candidates.
  • Close the browser and stop Playwright after the session.

Graph View

Backlinks