Helicone Community Page

Updated last month

Storing Qualitative User Feedback and Evaluations

At a glance

The community member is asking if there is a way to store user feedback or create an evaluation on a request that is neither binary nor a score, but rather qualitative, such as the "expected" model output. The comments suggest that the community member has a specific use case where they have someone manually grading model outputs, and they want to store the correct model output (e.g., "4" for the prompt "2+2" when the model output is "5"). The community members discuss the possibility of storing this information in Helicone, either as "request scoring/grading" or as alternative metadata, and one community member suggests that the community member might be looking for the feature described in the Helicone documentation for "custom properties".

Useful resources
is there any way to store user feedback or create an eval on a request that is neither binary nor a score, but rather qualitative i.e. the "expected" model output?
J
d
7 comments
Hi Deen! I'd love to learn more about what you are trying to measure, can we hop on a call?
sure scheduled
happy to chat regarding my specific use case over the call regardless, but i take it it's not possible today then?
Sorry - can we push back 3 hours?
lets say i have someone manually grading model outputs. Lets say we prompted "2+2" and the model output "5". is there a way in helicone to store what would be the correct model output e.g. "4"? either as "request scoring/grading" or as some alternative metadata (although it feels like this should be a first class citizen)?

the thought is that later this info could ne potentially used in things such as evaluations/finetuning
Hi Deen, sorry for the delay here
lets say i have someone manually grading model outputs. Lets say we prompted "2+2" and the model output "5". is there a way in helicone to store what would be the correct model output e.g. "4"? either as "request scoring/grading" or as some alternative metadata (although it feels like this should be a first class citizen)?

yes you might be looking for this
https://docs.helicone.ai/features/advanced-usage/custom-properties#option-1-pre-define-the-request-id
Add a reply
Sign up and join the conversation on Discord