Generative AI Google Brain

gemini-code-execution-intro-agentic-vision

3 min read

Intro to Gemini Agentic Vision

Source notebook

Repo path: gemini/code-execution/intro_agentic_vision.ipynb · Open on GitHub · intermediate

Uses Gemini code execution for agentic vision tasks: zooming, plotting, and image annotation.

Summary

This notebook teaches how Gemini code execution lets a model generate and run Python code while analyzing images. It configures the Google Gen AI SDK with Vertex AI, sends image prompts to gemini-3.5-flash, enables the code_execution tool, and parses reasoning, generated code, execution results, and returned images. The workflow demonstrates zoom-and-inspect, visual math with chart generation, and pixel-level image annotation.

Key code patterns

Create Vertex AI GenAI client

PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT")
LOCATION = "global"
client = genai.Client(
    enterprise=True,
    project=PROJECT_ID,
    location=LOCATION,
)

Uses a Google Cloud project and Vertex AI location for Gemini calls.

Enable code execution tool

response = client.models.generate_content(
    model=MODEL_ID,
    contents=[image, prompt],
    config=types.GenerateContentConfig(
        tools=[types.Tool(code_execution=types.ToolCodeExecution)]
    ),
)

Allows Gemini to generate and execute Python while solving visual tasks.

Load image content

image_bytes = requests.get(image_path).content
image = types.Part.from_bytes(
    data=image_bytes,
    mime_type="image/jpeg",
)

Shows how to pass downloaded image bytes into a multimodal Gemini request.

Use Cloud Storage image URI

image = types.Part.from_uri(
    file_uri="https://storage.googleapis.com/cloud-samples-data/generative-ai/image/benchmark.jpeg",
    mime_type="image/jpeg",
)

Uses hosted sample images directly as Gemini input parts.

Parse multipart response

for part in response.candidates[0].content.parts:
    if part.text is not None:
        print(part.text)
    if part.executable_code is not None:
        print(part.executable_code.code)
    if part.code_execution_result is not None:
        print(part.code_execution_result.output)
    if part.as_image() is not None:
        display(Image.open(io.BytesIO(part.as_image().image_bytes)))

Separates model reasoning, generated code, execution output, and generated images.

Models & APIs used

  • Models: gemini-3.5-flash
  • APIs / services: Vertex AI, Cloud Storage
  • SDKs / libraries: google-genai, Pillow, requests, IPython

When to use this

Use this pattern when visual reasoning needs code-backed inspection, calculation, plotting, cropping, or annotation.

Gotchas & caveats

  • Requires google-genai installation.
  • Colab users must authenticate with google.colab.auth.authenticate_user().
  • Requires a Google Cloud project or API key; the notebook uses a Google Cloud project.
  • Vertex AI API must be enabled.
  • PROJECT_ID must be set explicitly or available as GOOGLE_CLOUD_PROJECT.
  • The notebook uses LOCATION = “global”.
  • Code execution is enabled only when tools includes types.Tool(code_execution=types.ToolCodeExecution).

Best practices

  • Enable code execution when the model needs to crop, inspect, calculate, plot, or draw instead of guessing from a static image.
  • Parse response parts explicitly to inspect reasoning text, generated code, execution output, and resulting images.
  • Use PIL and IPython display to render images returned from code execution.
  • Use structured image parts from bytes or URI with the correct MIME type.

Graph View

Backlinks