Generative AI Google Brain

gemini-batch-prediction-intro-batch-prediction

3 min read

Intro to Batch Inference with the Gemini API

Source notebook

Repo path: gemini/batch-prediction/intro_batch_prediction.ipynb · Open on GitHub · intro

Runs Gemini batch inference jobs using Cloud Storage and BigQuery inputs and outputs.

Summary

This notebook teaches batch inference with the Gemini API in Gemini Enterprise Agent Platform. It demonstrates preparing JSONL or BigQuery request inputs, creating asynchronous batch jobs with google-genai, polling job state, and reading results from Cloud Storage JSONL or BigQuery tables into pandas DataFrames.

Key code patterns

Create enterprise GenAI client

client = genai.Client(
    enterprise=True,
    project=PROJECT_ID,
    location=LOCATION,
)

Configures the Gemini Enterprise Agent Platform client for batch jobs.

Submit Cloud Storage batch job

gcs_batch_job = client.batches.create(
    model=MODEL_ID,
    src=INPUT_DATA,
    config=CreateBatchJobConfig(dest=BUCKET_URI),
)

Uses a JSONL Cloud Storage input and a Cloud Storage output prefix.

Poll batch job state

while gcs_batch_job.state in (
    "JOB_STATE_RUNNING",
    "JOB_STATE_PENDING",
    "JOB_STATE_QUEUED",
):
    time.sleep(5)
    gcs_batch_job = client.batches.get(name=gcs_batch_job.name)

Batch inference is asynchronous, so results are read only after completion.

Read GCS predictions

fs = fsspec.filesystem("gcs")
file_paths = fs.glob(f"{gcs_batch_job.dest.gcs_uri}/*/predictions.jsonl")
df = pd.read_json(f"gs://{file_paths[0]}", lines=True)
df = df.join(pd.json_normalize(df["response"], "candidates"))

Loads generated JSONL output and flattens candidate responses for analysis.

Submit BigQuery batch job

bq_batch_job = client.batches.create(
    model=MODEL_ID,
    src=INPUT_DATA,
    config=CreateBatchJobConfig(dest=BQ_OUTPUT_URI),
)

Uses a BigQuery table as input and writes predictions to BigQuery.

Models & APIs used

  • Models: gemini-3.5-flash
  • APIs / services: Gemini API, Agent Platform API, Cloud Storage, BigQuery
  • SDKs / libraries: google-genai, pandas, google-cloud-storage, google-cloud-bigquery, fsspec

When to use this

Use this pattern for large Gemini multimodal workloads that are not latency sensitive and should run asynchronously at lower cost.

Gotchas & caveats

  • Enable the Agent Platform API before running batch jobs.
  • Cloud Storage JSONL inputs must be in us-central1 and readable by the service account.
  • BigQuery datasets must be in a supported region such as us-central1; multi-region locations such as us are not allowed.
  • BigQuery input must include a request column of type JSON or STRING containing a valid GenerateContentRequest.
  • BigQuery additional columns cannot use array, struct, range, datetime, or geography types.
  • Only public YouTube or Cloud Storage URIs are supported in fileData or file_data for BigQuery requests.
  • Batch jobs can take time; polling is required before reading outputs.
  • Notebook notes to ignore pip dependency errors.

Best practices

  • Use batch inference for large input sets that are not latency sensitive.
  • Specify explicit Cloud Storage or BigQuery output locations for completed predictions.
  • Check batch job status with client.batches.get before retrieving results.
  • List jobs with client.batches.list when inspecting project batch activity.
  • Delete created batch prediction jobs during cleanup.

Graph View

Backlinks