Generative AI Google Brain

gemini-evaluation-evaltask-approach-evaluate-langchain-chains

3 min read

Evaluate LangChain

Source notebook

Repo path: gemini/evaluation/evaltask_approach/evaluate_langchain_chains.ipynb · Open on GitHub · intermediate

Evaluates a LangChain recipe chatbot with Vertex AI Rapid Evaluation and custom Gemini-based metrics.

Summary

This notebook teaches how to prepare multi-turn chat data, score a LangChain conversational chain, and evaluate its outputs with Vertex AI Rapid Evaluation. It builds a recipe chatbot using ChatVertexAI, defines a custom faithfulness metric backed by Gemini, combines it with built-in metrics, logs results to a Vertex AI experiment, and compares prompt iterations.

Key code patterns

Initialize Vertex AI

PROJECT_ID = "[your-project-id]"
LOCATION = "us-central1"
 
import vertexai
vertexai.init(project=PROJECT_ID, location=LOCATION)

Sets the Google Cloud project and region before using Vertex AI models and evaluation.

Build LangChain Gemini chain

llm = ChatVertexAI(model_name="gemini-2.5-flash", temperature=0)
template = ChatPromptTemplate.from_messages([
    ("system", "You are a conversational bot that produce nice recipes for users based on a question."),
    MessagesPlaceholder(variable_name="messages"),
])
chain = template | llm

Combines a chat prompt template with Gemini through LangChain for conversational response generation.

Batch score conversations

with ThreadPoolExecutor(max_workers) as pool:
    partial_func = partial(batch_generate_message, callable=callable)
    for message in tqdm(pool.map(partial_func, messages.iterrows()), total=len(messages)):
        predicted_messages.append(message)
return pd.DataFrame(predicted_messages)

Runs chain.invoke across decomposed chat turns in parallel and stores model responses for evaluation.

Define custom metric

custom_faithfulness_metric = CustomMetric(
    name="custom_faithfulness",
    metric_function=custom_faithfulness,
)

Registers a client-side custom metric that asks Gemini to return JSON with an explanation and score.

Run EvalTask

metrics = ["fluency", "coherence", "safety", custom_faithfulness_metric]
eval_task = EvalTask(
    dataset=scored_data,
    metrics=metrics,
    experiment=experiment_name,
    metric_column_mapping={"prompt": "user"},
)
eval_result = eval_task.evaluate()

Evaluates scored responses with built-in and custom metrics while logging results to an experiment.

Models & APIs used

  • Models: gemini-2.5-flash
  • APIs / services: Vertex AI, Vertex AI Rapid Evaluation, Vertex AI Experiments
  • SDKs / libraries: vertexai, google-cloud-aiplatform, langchain, langchain-core, langchain-google-vertexai, pandas, yaml

When to use this

Use this pattern to evaluate and compare LangChain conversational chains on multi-turn datasets with Vertex AI metrics and experiment tracking.

Gotchas & caveats

  • Vertex AI API must be enabled for the selected Google Cloud project.
  • The notebook uses billable Vertex AI components.
  • Newly installed packages require a runtime restart in the notebook environment.
  • Colab users may need to authenticate with google.colab.auth.authenticate_user().
  • The default location is us-central1.
  • CustomMetric functions are computed client-side without online evaluation service APIs.
  • The custom metric expects the evaluator model to return valid JSON.

Best practices

  • Decompose multi-turn chats into per-turn examples with conversation_history before batch prediction.
  • Use temperature=0 for repeatable chatbot and evaluator outputs.
  • Map dataset columns explicitly with metric_column_mapping={“prompt”: “user”}.
  • Combine built-in metrics such as fluency, coherence, and safety with a task-specific custom metric.
  • Log evaluation results under an experiment name for run comparison.
  • Compare prompt iterations using Vertex AI experiment data.
  • Clean up created experiments when finished.

Graph View

Backlinks