Skip to content

Vertex AI

This guide shows how to call Google VertexAI models with Adastra LLMGW. It allows you to access Google's Gemini/select self-deployed models hosted in Google Model Garden.

Currently only calling via OpenAI-compatible Chat Completions shape is supported. See VertexAI Compatibility With OpenAI for more details (skip the Authentication part).

Setup

In order to setup the OpenAI client see Azure OpenAI. Then VertexAI models can be called using OpenAI's Chat Completions API shape as follows:

response = client.chat.completions.create(
    model="google/gemini-2.5-pro",
    messages=[{"role": "user", "content": "Tell me a story"}],
    stream=True
)

for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")