CHI 2006, Does Think Aloud Work?

CHI2006, April 4th, was a panel session on Wednesday afternoon to discuss ThinkAloud.

The title of the panel discussion is Does Think Aloud Work? How Do We Know? it was

The ThinkAloud protocol, created by Ericsson and Simon, is the most important and most used in usability testing. In this panel discussion, we discussed whether ThinkAloud is really useful and useful.

Among the panelists, Ted Boren, who is on the screen in the picture below, seemed to have a lot to say about usability testing and ThinkAloud. He also seemed to have a lot of experience, and he seemed to have a lot of strength. It was a person I didn't know, but when I searched, I wrote a thesis on ThinkAloud and made a presentation at UPA. I think you need to find it and read the article.


(Photo source of the panel discussion)


Here comes Joe Dumas, author of A Practical Guide to Usability Testing. This book is the oldest usability test book along with the Handbook of Usability Testing, but it is still the best book in my opinion.

Joe is currently at Bentley Colleage in the US, and he seems to be reviewing usability test papers.

Joe showed the video, saying it's a best practice for ThinkAudio. It was assembling scotch tape as described in his book.



Joe Dumas


When asked what to do if a participant does not speak during the usability test, the panelists answered as follows:

“ummm”, “OK” ,”and your are… .”
“keep talking.. :, :remember thinking aloud”

When asked what he would do if he didn't contribute to ThinkAud, Joe answered:

“Just leave it alone”

When asked whether 20 minutes of ThinkAudit training before the main test was bad, Joe answered:

“It takes 20 minutes of training to get good quality data. In 20 years of experience, training has been useful. ”

Who is a good participant?

“Very verbal person”

The moderator always interprets the participant's think-aloud. What?

“Think-aloud itself is not the problem, but moderators need to improve their skills to interpret Think-aloud”

Suddenly, I asked the audience to raise their hand, asking if they had ever learned to think about thinking. There were hardly any.

Then, there was a discussion about RITE. I asked the audience if anyone had done RITE. Quite a lot. Then, when I asked one participant to raise their hand, they heard the same number as the number of people who said RITE.

And the most important Concurrent Thinking Aloud and Retrospective Thinking Aloud were discussed. It has been argued that thinking around while performing a task can affect performance, and that what is said changes participants' behavior. I've actually seen a thesis like this before, but I can't remember. I'll have to look for it.

Because CTA can have these problems, the Retrospective Thinking Aloud technique is sometimes used. In other words, when performing a task, do not speak and remember what you did at that time after completing the task. There is no theoretically verified efficacy and reliability of CTA or RTA yet. Among the presenters, a professor from the University of Washington presented as a thesis the next day, but there was an issue. It seems that CTA or RTA should be verified more theoretically.

On Thursday, during the Expreince Report session, Samsung Electronics gave a presentation entitled “Understanding Users in Consumer Electronics Experience Design”. It was not a thesis, but a session dealing with practical issues. I attended the usability test because they had a new approach called stimulated recall, which was an RTA method, nothing new. Samsung Electronics was doing field research like a school in the UK, recording the use of a setup box at home, and then went to the next day to watch the recorded video and interview what they did at that time. This was an RTA method that was done by recalling the past events. Of course, it could be an RTA or just an interview depending on how you proceeded.

I don't remember whether it was A Practical UT or the Handbook of UT, but I was writing a book that showed the video to the participants after the task was finished using the RTA method and asked what they were thinking at that time. The book I have is the second edition from 1994, but it's an old story.

I have been doing eye tracking in the usability test since 30 to detect data that was not reported due to the difference in speaking and seeing speed with participants in kindergartens, infants, and 40-2004 years of age who do not speak well. . However, I was going to write a thesis on when to do the eye tracking study, but I was rejected at the 2006 Korean HCI Society this year ( 🥺

During the panel, a professor from the University of Washington spoke about this, and in Thursday's Usability Methods session, “The Validity of the Retrospective Think-Aloud Method ad Measured by Eye Tracking” published a paper in Iran. It would be good to refer to. Perhaps it has been a while since a paper like this has been published.

I basically do CTA and ET. And during the debriefing process, RTA is done. Of course, the difference in how you proceed can be an RTA like a Think Around or an opinion like an in-depth interview.

And although it was mentioned in the panel discussion, ThinkAloud in the usability test does not work properly with the ThinkAloud protocol created by Erexson and Simon. Depending on whether you formed a rapper with the participant and practiced well, the ThinkAuded may or may not appear, and it may proceed strangely even with me. Some really funny things happen.

Joe also mentioned it as an important point, but training and how to proceed with the rapper and think-around with the participants is the key. Ted also mentioned, pointing out that the problem is that there are still few standard guidelines for these procedures.

This is know-how, but it doesn't seem to be very well known in the field of experimental psychology or usability testing.



0 If you like the article, please click the heart~ It will be a strength to bloggers (SNS/login/advertising is not related)

Articles you might like

Push-Up Star: Push-Up Counter

Automatic push-up measurement and voice count, various types of push-up measurement and push-up speed analysis, police officers, military fitness test, military academy, student health fitness assessment (POPs), etc. automatically calculated

Add a Comment

Email addresses are not disclosed. Required items *is indicated by

This posting is part of Coupang Partners' activities, and a certain amount of commission is provided accordingly.

If you click the Add Channel button, you can view it in the KakaoTalk view.