Authors:
Masaru Shirasuna, Yuto Yoshida
(白砂大、吉田勇翔)

Title:
Attempts to investigate what words trivia experts focus on in quiz questions: Human-AI comparison through “LLM-as-a-judge” approach

Journal(書誌情報）:
PLOS ONE

doi:
10.1371/journal.pone.0358332

論文URL:
https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0358332

Abstract:
Computational capacity and knowledge of humans are more limited than those of large language models (LLMs). However, in buzzer quizzes, human trivia experts can often identify the correct answer even from insufficient information such as only a few words. Investigating how they can make fast and accurate judgments is expected to highlight new characteristics of human intelligence, but little is known about that. In this exploratory and case-based analysis, we predicted that trivia experts and LLMs would differ in which words/phrases in a question are important for identifying the answer, and compared experts’ performance with LLMs’ performance in Japanese trivia questions through an LLM-as-a-judge approach: We regarded LLMs as evaluators and then used their outputs as comparative tools for experts’ evaluations. First, we constructed a quiz question processing system that tokenized question texts based on morphological analysis and then numerically evaluated the importance of each token using GPT-4o/GPT-4.1. Then, we conducted a behavioral experiment wherein actual trivia experts were asked to numerically evaluate the importance of each token, just as LLMs had performed. As a result, trivia experts treated a variety of words/phrases as important, even if each word/phrase did not appear to be strongly associated with the answer. Their evaluation scores tended to accumulate faster than those of the LLMs. This may indicate that trivia experts can make inductive inferences faster (e.g., finding a common concept even from few items). More advanced question-answering systems may be designed by applying trivia experts’ cognitive processes to LLMs’ information processing, and our findings may provide a scaffolding toward such goals.

著者Contact先の email:
m.shirasuna1392[at]gmail.com（[at]を@に変更してください。） (静岡大学・白砂大)

日本語によるコメント（オプション，200-300字で）
早押しクイズの熟達者は、素早く正確に解を導くうえで問題文中のどのような単語を重要視しているのか。この問いを、LLMとの比較を通して実験的に検証しました。「クイズ問題文中の各単語(トークン)がどのくらい重要だと思うか」をLLMと実際のクイズ熟達者とに評価させた結果、クイズ熟達者は、一見すると正答と強く関連しない単語であっても、少数の単語から共通の概念を迅速に見出す(帰納的推論)ことに長けている可能性が示唆されました。人の認知プロセスをLLMの情報処理に応用することで、より高度な応答システムの開発に繋がることが期待されます。

投稿タグ