Cohere Models over Azure

Azure serves Cohere models over AI Foundry too. LLMGW supports Cohere reranker models over /azure-cohere endpoint. Only reranker models are supported over this endpoint.

Setup

Install the cohere package, which provides a Python client for the Cohere API:

pip install cohere

Configure your endpoint and API key for the Cohere client:

LLMGW_API_ENDPOINT = "https://<llmgw-deployment-url>/azure-cohere/"
LLMGW_API_KEY = "<YOUR_LLMGW_API_KEY>"

Create a client for the Cohere service:

import os
import httpx
import ssl
from openai import AzureOpenAI
import yaml
import cohere

model_name = "cohere-rerank-v4.0-fast"

llmgw_user = "cohere-user"
api_version = "2025-04-01-preview"

# Custom HTTP client is used for custom `llmgw` headers
http_client=httpx.Client(
    headers={
        "llmgw-user": llmgw_user,
    }
)

co = cohere.ClientV2(
    api_key=LLMGW_API_KEY,
    base_url=LLMGW_API_ENDPOINT,
    httpx_client=http_client,
)

documents = [
    {
        "Title": "Incorrect Password",
        "Content": "Hello, I have been trying to access my account for the past hour and it keeps saying my password is incorrect. Can you please help me?",
    },
    {
        "Title": "Cannot Login",
        "Content": "I can't login today?",
    },
]

yaml_docs = [yaml.dump(doc, sort_keys=False) for doc in documents]
query = "What emails have been about password issue?"

# Final call
result = co.rerank(
    model=model_name,
    documents=yaml_docs,
    query=query,
    top_n=3,
)

print(result)

Such call returns similar response:

id='d22c600a-ed03-43af-9fca-34d103def889' results=[V2RerankResponseResultsItem(index=0, relevance_score=0.17418541)] meta=ApiMeta(api_version=ApiMetaApiVersion(version='2', is_deprecated=None, is_experimental=None), billed_units=ApiMetaBilledUnits(images=None, input_tokens=None, image_tokens=None, output_tokens=None, search_units=1.0, classifications=None), tokens=None, cached_tokens=None, warnings=None)

In this example:

• To find where the model parameters are listed, see the configuration page and look for deployment_name. It refers to the model group configured in LLMGW.