Touch screens are notoriously difficult to type. Since we cannot feel the keys, we rely on the sense of sight to move our fingers to the right places and check for errors, a combination of efforts that we cannot achieve at the same time. To truly understand how people type on touchscreens, researchers at Aalto University and the Finnish Center for Artificial Intelligence (FCAI) created the first artificial intelligence model that predicts how people move their eyes and eyes. fingers when typing.
The AI model can simulate how a human user would type a phrase on any keyboard design. It makes mistakes, detects them – but not always immediately – and corrects them, just like humans would. The simulation also predicts how people adjust to alternate circumstances, such as how their writing style changes when they start using a new autocorrect system or a new keyboard design.
“Previously, touchscreen input was mostly understood from the point of view of the movement of our fingers. AI-based methods have helped shed new light on these movements: what we discovered is the importance of deciding when and where to look. Now we can make much better predictions of how people type on their phones or tablets, ”says Dr Jussi Jokinen, who led the work.
The study, which will be presented at ACM CHI on May 12, lays the groundwork for developing, for example, better and even personalized text entry solutions.
“ Now that we have a realistic simulation of how humans type on touchscreens, it should be a lot easier to optimize keyboard designs for better typing – which means fewer errors, more typing. fast and, above all for me, less frustration, ” Jokinen explains. .
In addition to predicting how a generic person would type, the model is also able to take into account different types of users, such as those with motor disabilities, and could be used to develop typing aids or interfaces designed with these. groups in mind. For those who don’t face any particular challenges, it can infer personal writing styles – by noting, for example, errors that repeatedly occur in texts and emails – what kind of keyboard, or automatic correction system, would best serve a user. .
The new approach builds on the group’s previous empirical research, which provided the basis for a cognitive model of how humans type. The researchers then produced the generative model capable of typing independently. The work was carried out as part of a larger project on interactive AI at the Finnish Center for Artificial Intelligence.
The findings are supported by a classic machine learning method, reinforcement learning, which the researchers extended to simulate people. Reinforcement learning is normally used to teach robots to solve tasks by trial and error; the team found a new way to use this method to generate behavior that closely matches that of humans – mistakes, fixes and all.
“We gave the model the same capabilities and limitations that we as humans have. When we asked him to type effectively, he figured out how to best use those abilities. The end result is very similar to the way humans type, without having to teach the model with human data, ”says Jokinen.
Comparison with human typing data confirmed that the model predictions were correct. In the future, the team hopes to simulate slow and fast typing techniques to, for example, design useful learning modules for people who want to improve their typing.
The document, Touchscreen Typing As Optimal Supervisory Control, will be presented on May 12, 2021 at the ACM CHI conference.
Source of the story:
Material provided by Aalto University. Note: Content can be changed for style and length.