Research has explored using Automatic Text Simplification for reading assistance, with prior work identifying benefits and interestsfrom Deaf and Hard-of-Hearing (DHH) adults. While the evaluation of these technologies remains a crucial aspect of research inthe area, researchers lack guidance in terms of how to evaluate text complexity with DHH readers. Thus, in this work we conductmethodological research to evaluate metrics identified from prior work (including reading speed, comprehension questions, andsubjective judgements of understandability and readability) in terms of their effectiveness for evaluating texts modified to be atvarious complexity levels with DHH adults at different literacy levels. Subjective metrics and low-linguistic-complexity comprehensionquestions distinguished certain text complexity levels with participants with lower literacy. Among participants with higher literacy,only subjective judgements of text readability distinguished certain text complexity levels. For all metrics, participants with higherliteracy scored higher or provided more positive subjective judgements overall.
https://doi.org/10.1145/3411764.3445038
The ACM CHI Conference on Human Factors in Computing Systems (https://chi2021.acm.org/)