Why feedback is important
Building AI systems is a difficult task, especially if you want to teach an AI the mechanics of your domain. In order to understand where an AI system is shining and where it is failing, feedback provided by domain experts is crucial.
Feedback can be given in form of thumbs up, thumbs to the side or thumbs down as well as a short free text, and is always linked to an answer of the AI system to your question. For instance, if you ask "How am I insured", and the AI responds with an answer, you can rate and comment the answer.
Feedbacking is relatively simple. It's usually designed to help the engineers of the system understand current problems; it is usually not directly used to train the model "live". In other words, your feedback will not directly impact the AI the moment you provide feedback, but it is used by the people tailoring the AI to your domain to better understand the situation.
Types of feedback
You can provide three types of feedback:
- positive feedback: the answer is what you were hoping for. Great!
- neutral feedback: the answer is okay, but you might not like the way the answer is formulated, or the length of the answer.
- negative feedback: the answer of the model is wrong.
Positive and neutral feedback
When you want provide positive or neutral feedback, you can simply click on the thumbs up or thumbs to the side buttons in the generated answer. The free text comment you see is optional.
Positive and neutral feedback is important, as it helps to understand how often the AI did the job correct. Please feedback - especially in the beginning of an implementation - every answer, even if it is mostly a thumbs up.
Negative feedback
Negative feedback is highly interesting to analyze how the AI system can be improved. Since wrong answers can have different root causes, it is important to provide clear feedback for negative cases; remember, these points are typically read by another human to understand what you would have expected.
In the feedback form, you can also specify the type of issue that happened. Currently, we support:
- Hallucination: This is the most important feedback to collect. Was the answer plain wrong? If so, why? The more precise the feedback is here, the faster the hallucination can be resolved.
- Style
- Missing source
- Old data
- Answer stopped
Answers with non-optimal styles (e.g. you would like to have a bullet point list but receive a fully written response) are not problematic. A hallucination on the other hand is critical; in this case, the model comes up with wrong facts and misleads the human user.
The more in-depth feedback is collected early on in a use case, the better the system can be adapted to tackle the challanges.