Lessons learned from evaluation

A total of 353 end-users have participated in evaluation activities in Cloud4all/GPII. This was organized in three iterative phases (Chalkia, Montalva, de los Rios, & Carmona, 2014), (Chalkia E. B., 2015 ):

  • The first evaluation phase focused on the evaluation of the usability of the Cloud4all/GPII solutions and procedures, in order to gather user feedback early in the Project. Basic and advanced versions of the Project Management Tool (PMT) and initial results of the auto-configuration mechanism of Cloud4all/GPII in basic environments (only Linux and Windows OS) were analysed with end users and stakeholders.
  • The second evaluation phase went one step beyond, moving testing from the usability of the prototypes to the user experience with the scenarios, and how it could be enhanced by using Cloud4all/GPII both on familiar and unfamiliar devices. The focus though was on the assessment of an improved version of PMT and the results of the auto-configuration mechanism of Cloud4all/GPII in a more complex environment (Android OS, Java Phones and some ATs were included in the scenarios together with Linux and Windows OS already tested in the first evaluation).
  • At the third evaluation phase, more naturalistic scenarios were evaluated. The users had to handle a set of devices and applications using a token based on their N&P. The users were asked to navigate through these different devices and applications as if they were in their own environment and validate the auto-configuration procedure and results. The applications used were close to the ones the users could find in their everyday life: Windows and Linux OS for laptop or desktop PCs, Java or Android phones, online banking app, and different AT solutions (such as NVDA, ORCA, Mobile accessibility, Read&Write Gold, and Cloud4Chrome).

The results of the evaluation provided a large amount of user feedback on the usability, usefulness and acceptance of the solutions that has been used to improve the development of the prototypes and tools towards their final versions (Cloud4all, D403.1, 2015).

What is more important for the Project though, is the knowledge gathered on users’ perceptions and behaviour when configuring devices and applications, how users face computer tasks with (non-personalised) familiar and unfamiliar devices, and how Cloud4all/GPII could improve their user experience.

The following items summarize the most relevant lessons learned in this process:

  • Not all users are equally able to specify their needs and preferences and configure their settings. There are individuals who really know what they require in order to use IT and, on the other hand, individuals that have no idea what specific capabilities or features they need. Moreover, adjusting system settings to meet a user’s N&P is often a difficult and frustrating task for them. Indeed, one third of the participants claimed not to make configurations during the tests because of use barriers (e.g. settings they require are not available or the user doesn’t know how to adjust them). If users could receive selective recommendations of convenient accessibility options when they are defining their preferences, that would help them to find solutions to their difficulties, simplify computer tasks, and even lower barriers with ICT in general (Loitsch C. C., 2014).
  • Cloud4all/GPII improves user experience when configuring a solution. When using the PMT to set users’ needs and preferences, its interface was rated as highly usable, the use barriers of configuration were reduced to affect only to 14% of the participants, and the users claimed to adjust settings easier and with better autonomy and competence. Some findings also points to future improvements: additional settings for specific user groups (e.g. people with motor impairments), simpler interfaces and procedures for people with cognitive impairments or snapshotting capabilities for capturing current settings.
  • Cloud4all/GPII improves users’ experience when performing common tasks. In the tests, the participants used a familiar solution (Windows system) as well as non-familiar solutions (Linux, Java phones or Android device) to perform some ordinary tasks, such as reading documents, filling in a form or creating a contact in the phone agenda. These tasks were done both after configuring some settings manually and after personalizing the system with Cloud4all/GPII. It was found that after using Cloud4all/GPII the users improved their performance with these tasks, perceived them as easier and felt more autonomous and competent. In the case of non-familiar systems some users were not even able to access the solution before applying auto-configuration.
  • The user experience of the Cloud4all concept has been scored by end-users as ‘Excellent’. The scores on ease of use, usefulness and intention to use it in the future are above 80 points over 100 in the UX scales used as reference. It should be noticed that most users consider Cloud4all/GPII as more relevant for public than for private devices.