🧪 This platform is in early beta. Features may change and you might encounter bugs. We appreciate your patience!
questionnaire_wording_bias
Questionnaire wording bias encompasses the systematic distortion of survey responses caused by how questions are phrased, ordered, or formatted. Even logically equivalent questions yield substantially different response distributions when worded differently — for example, 'allow' versus 'forbid' framing, leading versus neutral questions, or positive versus negative scale anchors.
Asking 'Do you favor allowing abortion in some circumstances?' yields around 77% agreement, while 'Do you favor forbidding abortion in all circumstances?' yields only about 20% agreement — yet both questions logically address the same policy position.
A customer satisfaction survey asks: 'How satisfied were you with our outstanding support team?' The embedded positive descriptor ('outstanding') inflates satisfaction scores compared to a neutral phrasing like 'How satisfied were you with our support team?' — making the service appear more highly rated than it actually is.
An employee engagement survey asks first about specific frustrations with management, then immediately asks 'Overall, how satisfied are you with your job?' The negative priming from the earlier questions causes overall satisfaction scores to drop significantly compared to a version where the overall satisfaction question is asked first, even though nothing about the employees' actual experience has changed.
Binary (yes/no) questions an LLM must answer to identify this aspect:
Does the question framing use loaded or emotionally valenced language?
Type: binaryWould reversing the question wording or scale anchors change the reported results?
Type: binaryIs the question ordering likely to prime certain responses through context effects?
Type: binaryWere the questionnaire items pre-tested for equivalence of phrasing?
Type: binaryQuestionnaire wording bias encompasses the systematic distortion of survey responses caused by how questions are phrased, ordered, or formatted. Even logically equivalent questions yield substantially different response distributions when worded differently — for example, 'allow' versus 'forbid' framing, leading versus neutral questions, or positive versus negative scale anchors.
Words carry connotations beyond their logical content. Question framing activates different mental representations, and positively vs. negatively framed questions tap into different reference points.
Use validated, pre-tested questionnaire items. Split-sample experiments to test equivalent phrasings. Report exact question wording in any citation of survey data. Consider acquiescence response bias in Likert scales.
Public opinion polls on hot-button issues routinely show 15-25 percentage point swings based solely on question wording, making it easy to commission polls that produce desired results.
Use these tools to detect, analyze, or train this aspect.