Azure <> Helicone NodeJS

mmattlanham

I am having a hard time trying to get Helicone to work with the Azure OpenAI service....

I have tried a few variations and can't seem to get things working. I also don't see anything in the instructions about how to specify a deployment name (I have a deployment for GPT-3.5 and GPT-4) so not sure if that is causing the problem?

Here is my code snippet:

Plain Text

import { Configuration, OpenAIApi } from "openai";

const configuration = new Configuration({
    apiKey: process.env.AZURE_API_KEY,
    basePath: "https://oai.hconeai.com/v1",
    baseOptions: {
      headers: {
        "Helicone-Auth": `Bearer ${process.env.HELICONE_API_KEY}`,
        "Helicone-OpenAI-Api-Base": `https://XXXX.openai.azure.com`,
      }
    }
  });
const openai = new OpenAIApi(configuration);

In the Helicone Dashboard, I am seeing the request being logged as a 404 and saying "Resource not found" do you have anyway to see if the request contains the API key correctly, or understand what is causing it?

I know it is deployed and available, because the following code snippet is when I use the OpenAI Azure NodeJS library directly, which works fine:

Plain Text

import { Configuration, OpenAIApi } from "azure-openai";
const openai = new OpenAIApi(
    new Configuration({
       apiKey: process.env.AZURE_API_KEY,
       azure: {
          apiKey: process.env.AZURE_API_KEY,
          endpoint: process.env.AZURE_URL,
          deploymentName: 'XXXXXXX',
       }
    }),
 );

48 comments

mmattlanham

I've made this into a thread to make it easier to figure out

mmattlanham

I've also tried naming the deployments the same as the model, although Azure doesn't allow . for the 3.5 models, but I have tried making requests with 3.5 and 4 and both fail

Attachment

mmattlanham

Attachment

EEmaGPT

Same here

EEmaGPT

anyone found a solution?

aayoKho

We’re taking a look into this now. Will let you know when we have updates!

EEmaGPT

Thank you!
FYI, I want adapt that curl

1curl "https://newinstance.openai.azure.com/openai/deployments/gpt-4/chat/completions?api-version=2023-03-15-preview" \
2 -H "Content-Type: application/json" \
3 -H "api-key: YOUR_API_KEY" \
4 -d "{
5 "messages": [],
6 "max_tokens": 800,
7 "temperature": 0.7,
8 "frequency_penalty": 0,
9 "presence_penalty": 0,
10 "top_p": 0.95,
11 "stop": null
12}"

But it doesn’t work with helicone and api base header 😵
I get always resource not found

Attachment

CCole

I sent you both a DM. It would be great to hop on a call to debug this together. Additionally, if you're ok with it, please add me to your org so I can take a look: cole@helicone.ai

mmattlanham

Thanks, I've replied

CCole

Looked into the issue and attempted it locally. I created a PR to addrress it. Waiting for review then can merge:

https://github.com/Helicone/helicone/pull/708

Hopefully it fixes the issue. TLDR:

https://learn.microsoft.com/en-us/azure/ai-services/openai/reference

The endpoints look like: https://YOUR_RESOURCE_NAME.openai.azure.com/openai/deployments/YOUR_DEPLOYMENT_NAME/completions?api-version=2023-05-15

But our current logic strips the entire ending and forces it to be:
https://YOUR_RESOURCE_NAME.openai.azure.com/v1/chat/completions

This causes a 404. The azure endpoints require the url structure. The model is not even in the body. Luckily with Scotts recently change, the model name will be retrieved from the response first.

mmattlanham

Ah nice, glad you found it!

mmattlanham

Any idea how long it'll take for the PR to be approved?

CCole

Hopefully today or latest tomorrow.

mmattlanham

Just checking in to see if there has been any progress, I can see some comments on the PR

JJustin

We are looking into this soon, sorry for the delay

CCole

@here try this and let me know if that works!

Plain Text

const configuration = new Configuration({
  apiKey: "AZURE_API_KEY",
  basePath: "https://oai.hconeai.com/openai/deployments/YOUR_DEPLOYMENT_NAME",
  baseOptions: {
    headers: {
      "Helicone-Auth": `Bearer HELICONE_API_KEY`,
      "api-key": "AZURE_API_KEY",
      "Helicone-OpenAI-Api-Base":
        "https://your_resource_name.openai.azure.com",
    },
    params: {
      "api-version": "API_VERSION",
    },
  },
});

CCole

The JSON pretty version currently shows Invalid Prompt, but the pure JSON tab works. This is likely due to a FE mapping that needs to be updated. CC:

CCole

A PR with the Invalid Prompt fix has now been merged. Everything should work now! https://github.com/Helicone/helicone/pull/707

mmattlanham

Unfortunately I'm still getting a 404 and Resource not found everything looks to be set correctly....

Attachment

mmattlanham

Added the api-version part also and still no luck 😦

Attachment

mmattlanham

Let me know if you spot anything I'm doing wrong when you dig into the requests, or if it's helpful to do some live debugging just let me know!

CCole

Is your deployment in this structure:

https://[AZURE_DOMAIN].openai.azure.com/openai/deployments/[DEPLOYMENT]?api-version=2023-03-15-preview

?

CCole

That is what this should build on our backend. It will do this:

Grab the OpenAI basePath path, so /openai/deployments/[DEPLOYMENT]
Then append that to the Helicone-OpenAI-Api-Base which should look like https://[AZURE_DOMAIN].openai.azure.com. It will then add the params to the end as query params. With the final url shaped like:

{Helicone-OpenAI-Api-Base}{OpenAI BasePath Path}{Params}

https://[AZURE_DOMAIN].openai.azure.com/openai/deployments/[DEPLOYMENT]?api-version=2023-03-15-preview

mmattlanham

Yup that lines up with what i have locally

mmattlanham

as per the working curl request i sent before - just doesnt seem to work theough helicone

JJustin

I am actually able to reproduce your issue :/

I am going to take a deeper dive in a bit.... but this is failing for me right now

Plain Text

  const configuration = new Configuration({
    basePath:
      "https://oai.hconeai.com/openai/deployments/text-davinci-003",
    baseOptions: {
      headers: {
        "Helicone-Auth": `Bearer <>`,
        "Helicone-OpenAI-API-Base":
          "https://<>.openai.azure.com",
        "api-key": "<>",
      },
    },
  });
  const openai = new OpenAIApi(configuration);
  const response = await openai.createCompletion({
    prompt: "This is a test",
    model: "text-davinci-003",
  });
  console.log("configuration", response);

mmattlanham

Are you able to see the raw request on your side to check that it doing what you expect and also passing the api key

mmattlanham

well im glad its broken for you also haha

JJustin

yeah the raw request on our side looks good for some reason 🤔

JJustin

I am super busy tonight, so I might not be able to get to this till tomorrow

JJustin

I sent some notes to Cole

mmattlanham

thanks - hopefully we can get it sorted tomorrow then! Appreciate you contining to dig into it!!

CCole

I think in your version you are missing the apiVersion, once you add that it works

CCole

Are you able to call now?

mmattlanham

i cant get mine to work with or wirhoir the version

mmattlanham

without**

CCole

Can you hop on a call?

mmattlanham

yea gimme 5 mins

mmattlanham

I'm available now

JJustin

This is still not working for me either... :/ ?

Plain Text

  const configuration = new Configuration({
    basePath:
      "https://oai.hconeai.com/openai/deployments/text-davinci-003",
    baseOptions: {
      headers: {
        "Helicone-Auth": `Bearer <>`,
        "Helicone-OpenAI-API-Base":
          "https://<>.openai.azure.com",
        "api-key": "<>",
      },
      URLSearchParams: {
        "api-version": "<>",
      },
    },
  });

JJustin

The search params are not getting added 🤔

CCole

We were able to solve it on call. The params belonged inside the baseOptions after the headers:

Plain Text

const configuration = new Configuration({
  apiKey: "AZURE_API_KEY",
  basePath: "https://oai.hconeai.com/openai/deployments/YOUR_DEPLOYMENT_NAME",
  baseOptions: {
    headers: {
      "Helicone-Auth": `Bearer HELICONE_API_KEY`,
      "api-key": "AZURE_API_KEY",
      "Helicone-OpenAI-Api-Base":
        "https://your_resource_name.openai.azure.com",
    },
    params: {
      "api-version": "API_VERSION",
    },
  },
});

CCole

URLSearchParams -> params

CCole

added a github issue: https://github.com/Helicone/helicone/issues/715

mmattlanham

Awesome, thanks for helping solve this! Seeing the latency difference is incredible, same question asked to OpenAI = 17.6s to respond, Azure = 5s - being able to track this is awesome, thanks!

CCole

Wow! That is amazing

JJustin

I cant repro. Would you mind updating the example in the docs here to reflect this change? I dont feel comfortable doing it since I cant get mine to work :/

https://docs.helicone.ai/getting-started/integration-method/azure

CCole

The PR that updates those docs was merged and has now just been deployed to our Mintlify documentation!

Add a reply

Helicone Community Page

Azure <> Helicone NodeJS