Generative AI Google Brain

gemini-prompts-prompt-optimizer-get-started-with-vertex-ai-prompt-optimizer-multimodality

3 min read

Get Started with Vertex AI Prompt Optimizer - Multimodality

Source notebook

Repo path: gemini/prompts/prompt_optimizer/get_started_with_vertex_ai_prompt_optimizer_multimodality.ipynb · Open on GitHub · intermediate

Optimizes a multimodal Gemini prompt with Vertex AI Prompt Optimizer on MathVista image QA.

Summary

Introduces Vertex AI Prompt Optimizer’s zero-shot and data-driven approaches, then demonstrates the data-driven optimizer for a multimodal question-answering task. The workflow configures a Google Cloud project and bucket, grants service-account roles, defines an image QA prompt template with @@@image/jpeg, validates settings with Pydantic, uploads config to Cloud Storage, runs client.prompt_optimizer.optimize(method=‘vapo’), and reads the best instruction from GCS outputs.

Key code patterns

Initialize project and client

PROJECT_ID = str(os.environ.get('GOOGLE_CLOUD_PROJECT'))
LOCATION = os.environ.get('GOOGLE_CLOUD_REGION', 'us-central1')
BUCKET_URI = f'gs://{BUCKET_NAME}'
! gcloud storage buckets create {BUCKET_URI} --project={PROJECT_ID} --location={LOCATION}
client = vertexai.Client(project=PROJECT_ID, location=LOCATION)

Creates the Cloud Storage workspace and Vertex AI client used by the optimizer job.

Grant backend job roles

SERVICE_ACCOUNT = f'{PROJECT_NUMBER}-compute@developer.gserviceaccount.com'
for role in ['aiplatform.user', 'storage.objectAdmin', 'artifactregistry.reader']:
    ! gcloud projects add-iam-policy-binding {PROJECT_ID} \
      --member=serviceAccount:{SERVICE_ACCOUNT} \
      --role=roles/{role} --condition=None

Prompt Optimizer runs as a backend job and needs model, storage, and component-read permissions.

Multimodal prompt template

system_instruction = '''
Solve the problem given the image.
'''
prompt_template = '''
Problem: {{query}}
Image: {{image}} @@@image/jpeg
Answer: {{target}}
'''

Uses the @@@MIME_TYPE marker so image URIs in the dataset are treated as multimodal inputs.

Validated VAPO config

vapo_data_settings = {
    'system_instruction': system_instruction,
    'prompt_template': prompt_template,
    'has_multimodal_inputs': True,
    'target_model': 'gemini-2.5-flash',
    'optimization_mode': 'instruction',
    'eval_metrics_types': ['question_answering_correctness'],
    'eval_metrics_weights': [1.0],
    'input_data_path': input_data_path,
    'output_path': output_path,
    'project': PROJECT_ID,
}
vapo_data_config = OptimizationConfig(**vapo_data_settings)

Defines instruction-only optimization against a labeled multimodal QA dataset.

Upload and run optimizer

config_path = f'{BUCKET_URI}/config.json'
with epath.Path(config_path).open('w') as config_file:
    json.dump(vapo_data_config.model_dump(), config_file)
result = client.prompt_optimizer.optimize(
    method='vapo',
    config={'config_path': config_path, 'wait_for_completion': True, 'service_account': SERVICE_ACCOUNT},
)

Stores the job configuration in GCS and starts the Vertex AI Prompt Optimizer backend job.

Retrieve best instruction

best_instruction, _ = get_best_vapo_results(output_path)
print('The optimized instruction is:\n', best_instruction)

Parses optimizer output files in GCS so an application can reuse the top-performing prompt.

Models & APIs used

  • Models: gemini-2.5-flash
  • APIs / services: Vertex AI, Cloud Storage, Cloud IAM, Artifact Registry
  • SDKs / libraries: google-cloud-aiplatform, vertexai, google-cloud-storage, pydantic, pandas, etils, protobuf

When to use this

Use this pattern when optimizing a multimodal Gemini prompt against labeled image QA examples stored in Cloud Storage.

Gotchas & caveats

  • Colab requires auth.authenticate_user() before accessing Google Cloud resources.
  • The Vertex AI API must be enabled for the selected project.
  • The notebook creates and later can recursively delete the configured Cloud Storage bucket.
  • The backend job uses the default Compute Engine service account and requires roles/aiplatform.user, roles/storage.objectAdmin, and roles/artifactregistry.reader.
  • Multimodal templates must include a supported MIME marker such as @@@image/jpeg after the media field.
  • The target, optimizer, and source model locations default to us-central1 in the configuration.
  • QPS fields are constrained to integer values and should be set based on available quota.
  • The notebook recommends 50-100 distinct samples for reliable prompt optimization results.

Best practices

  • Use labeled rows containing query, image GCS URI, and target for question_answering_correctness evaluation.
  • Use examples where the current system instruction performs poorly when building an optimization dataset.
  • Validate optimizer settings with the Pydantic OptimizationConfig before submitting the job.
  • Store both optimizer configuration and results in Cloud Storage.
  • Set has_multimodal_inputs to True when optimizing prompts with image inputs.
  • Use wait_for_completion=True when the notebook should block until the optimizer job finishes.
  • Programmatically retrieve the best instruction from GCS outputs for application use.

Graph View

Backlinks