Songyan Teng
The University of Melbourne, Melbourne, Victoria, Australia
Context sensing on smartphones is often used to understand user behaviour. Amongst the many available sensors, the collection of text is crucial due to its richness. However, previous work has been limited to collecting text only from keyboard input, or intermittently collecting screen text indirectly by taking screenshots and applying optical character recognition. Here, we present a novel software sensor that unobtrusively and continuously captures all screen text on smartphones. We conducted a validation study with 21 participants over a two-week period, where they used our software on their personal smartphones. Our findings demonstrate how data from our sensor can be used to understand user behaviour and categorise mobile apps. We also show how smartphone sensing can be enhanced by using our sensor in conjunction with other sensors. We discuss the strengths and limitations of our sensor, highlighting potential areas for improvement and providing recommendations for its use.
doi.org/10.1145/3613904.3642347
The ACM CHI Conference on Human Factors in Computing Systems (https://chi2024.acm.org/)