Evaluation

We have asked anonymous users to use our website, both from the student’s and the supervisor’s perspective. Below are the notable responses we have received.

We have asked 4 people who have not been involved with our project to test the FYP Portal for us. We have developed 3 criteria against which to evaluate the success of our project: we are giving the users a scenario where they are a student or a supervisor and asking them how they would approach using the portal, we are asking them to evaluate the user interface for each page they visit, and finally, we are sharing our MoSCoW requirements and asking if they believe each has been fulfilled.

At the start of our project, we interviewed 4 students, 2 academics, and the administrator who is overseeing the FYP supervisor allocations this year.

We decided to go with a semi-structured interview as the method of gathering information from would be users. We probed topics such as what they would like to be able to do with a new system and thoughts they had about the current system. We deemed this more appropriate than a questionnaire as people are more reluctant to be thorough and descriptive in questionnaires. We felt that it was also appropriate to ask open interview questions to really understand what the users want.

Below are notable points from answers to the questions we asked.

Tasks
  1. You want to log in to see what supervisors are available, and you’re interested in finding a supervisor who would supervise Mathematics of Computing.

  2. Assume you didn’t find a supervisor, you want to apply to be automatically allocated a supervisor.

  3. You want to edit your application.

  4. You want to view who your allocated supervisor is.

Expected Solutions
  1. Log in page > Azure AD button > Select Student role > Search bar, “Mathematics of computing”. Results: Kirsty Blackman, Liz Twist, Luke Evans.

  2. Apply > Student Application > Fill application.

  3. Apply > View Application > Edit > Submit.

  4. Apply > Your Allocated Supervisor.

User 1 comments, Student:
  1. A little bit of hesitance when picking web role, curious to see what happens if they picked “Supervisor” instead of “Student”. Otherwise, followed the same steps, but browsed the list of supervisors before searching for “Math”. Found Kirsty Blackman, Liz Twist, Luke Evans.

  2. Their first step was to click “Final Year Projects” in the navigation bar. After that, they correctly clicked Apply > Student Application. Noted the webpage took a bit long to load. Found the form relatively easy to fill out, aside from displaying slight confusion over what “second-choice research group” meant. Completed everything else with no trouble.

  3. Correctly and smoothly progressed through the expected steps.

  4. Correctly progressed through the expected steps but noted there was nothing to see on that page yet.

User 2 comments, Student:
  1. Did not have issues logging in or picking the student role. Browsed the list for a short time before searching for “Mathematics of computing”. Found Kirsty Blackman, Liz Twist, Luke Evans.

  2. Navigated to Apply > Student Application without trouble. Expressed dislike that the form was long. Wished he was redirected to another page once he submitted it.

  3. Navigated to the correct page easily and quickly, remarked there are not too many tabs on the site to get lost in. Noticed there were extra questions, but they read the comment at the top of the page so there was no confusion.

  4. Easily navigated to the page. The only further comments were that the user was satisfied with the navigability of the website.

Tasks:
  1. You are required to log in and register as a supervisor.

  2. You want to know what students you are allocated to supervise this year.

  3. You want to report to the administration, per their instructions, that you agreed to supervise Teodora Lovin.

  4. You want to edit your supervisor application.

Expected Solutions:
  1. Log in page > Azure AD button > Select Supervisor role > Your Supervisor Application > Register as a Supervisor > Fill application.

  2. Your Supervisees > View Supervisees.

  3. Your Supervisees > Register Supervisee > Select Teodora Lovin from lookup.

  4. Your Supervisor Application > View Application > Edit > Submit.

User 3 comments, Supervisor:
  1. Logged in without issue. Claimed it was easy to sign in. Slight surprise when they were asked to pick a role but had no issues executing the task correctly. They took a second to familiarize themselves with the interface, then quickly found the Register as a Supervisor button. Filled in the application successfully, showing a bit of annoyance at having to pick ACM keywords one at a time. Slightly confused at where to find the teams meeting link required in the form.

  2. The user followed the expected steps, commented it was convenient to see everything and everyone they are supervising at a glance.

  3. Didn’t switch pages as per the expected solution, they just used the button on the View Supervisees page. Claimed it was easy to fill out but showed some concern over any privacy issues of being given a list of everyone’s names and emails.

  4. The user has no issues with finding or editing this form. Claimed it was nice to be able to edit the form responses.

User 4 comments, Supervisor:
  1. No comments on signing in. Selected the correct role. Navigated correctly to the supervisor application but wasn’t sure what exactly their supervision capacity was because they had to discuss this with the department.

  2. They navigated correctly to the page to view their supervisees, which was also the home page they started on; however, it was empty.

  3. They used the button on the page they were already on, View Supervisees page to add a supervisee rather than using the navigation bar to access the Register a Supervisee page. They had no issues finding the correct student contact and submitting the form.

  4. They searched through the navigation bar for a second, clicked on their name, and then clicked on Your Application > View Application. They changed the content of some fields and clicked submit. They instinctively refreshed the page to see the updated application.

Summary analysis: The users used the system almost as intended, with few hesitations about where they might find the page they are looking for. In some instances, they took another approach than expected, however these alternative solutions worked perfectly fine and had the same outcome. These tests confirm that the user interface is largely intuitive, with some exceptions to the role selection. The users were overall satisfied with the interface, concluding a successful user acceptance test.

For each page we asked the user to rate from 1-5 how user-friendly they believe our interface to be, where 1 is the lowest score and 5 is the highest. This test is used alongside the User Acceptance test to determine user satisfaction with the website.

Student:

Page

User 1

User 2

User 3

User 4

Log in

5

4

5

5

Home

5

5

5

5

Student Application

4

3

5

4

View Student Application

4

4

5

3

View Allocated Supervisor

4

4

5

4

Notable comments: Student Application was long with many questions. View Student Application showed some fields which weren’t on the Student Application which was confusing, especially when they did not read the comment at the top of the page. Many users commented on how well the website integrates with UCL’s other systems.

Supervisor:

Page

User 1

User 2

User 3

User 4

Log in

5

4

5

5

View Supervisees

4

4

4

4

Register as a Supervisor

5

4

4

4

View Supervisor Application

5

4

5

4

Register Supervisee

5

5

5

5

Notable comments: It was cumbersome having to select ACM keywords one at a time on the Supervisor Application. Every user noted the footer on the View Supervisees Page was not anchored to the bottom of the page, making the page look less professional.

Administrator:

Page

User 1

User 2

User 3

User 4

Log in

5

4

5

5

Matching

3

3

4

3

Notable comments: Too much text on the screen, users did not want to read all of it. User 4 thought the button was a bit of an awkward solution, enquired how we could ensure that the wrong person doesn’t press it.

Summary analysis: Overall, the review has been good. There have been no ratings below 3, and the average rating was 4. The users were overall satisfied with the readability and usability of the website. They highly appreciated it was possible to sign in with UCL credentials and the style was consistent with UCL’s theme. They were not so satisfied with the loading time for some of the webpages.

Once we executed the first two user experience tests, we walked the users through our MoSCoW requirements and showed them some elements of the system implementation. We asked them to tell us if, in their opinion, the requirements have been satisfied.

Must have requirement

User 1

User 2

User 3

User 4

1

Yes

Yes

Yes

Yes

2

Yes

Yes

Yes

Yes

3

Yes

Yes

Yes

Yes

4

Yes

Yes

Yes

Yes

5

Yes

No

Yes

Yes

Comments: User 2 found it confusing that some form fields did not appear in the table underlining it.

Should have requirement

User 1

User 2

User 3

User 4

1

Yes

Yes

Yes

Yes

2

Yes

Yes

Yes

Yes

Comments: All users agreed that we have implemented each of these requirements.

Could have requirement

User 1

User 2

User 3

User 4

1

Yes

Yes

Yes

Yes

2

Yes

No

No

Yes

3

No

No

No

No

Comments: Users 1 and 4 believed that if the table did not get cleared between the year groups, the student’s interests could be monitored over time. All Users thought requirement 3 has not been implemented.

Won’t have requirement

User 1

User 2

User 3

User 4

1

No

No

No

No

2

No

No

No

No

Comments: All Users agreed neither of the Won’t have requirements have been implemented.

Summary analysis: Whilst the users don’t have a complete understanding of our system, we were able to demonstrate to them the functioning feature and their implementations. This enabled them to sign off on the fact they saw it was implemented and functioning.

Task Teodora James Rory
Client Liaison 55% 20% 25%
HCI 33% 33% 33%
Requirement Analysis 33% 33% 33%
Research 33% 33% 33%
Creating Presentations 60% 20% 20%
Filming and Editing Presentation 0% 0% 100%
Algorithm Development 0% 100% 0%
Database Development 100% 0% 0%
Web Role Development 0% 100% 0%
Portal Interface Development and Design 33% 33% 33%
Report Writing 40% 20% 40%
Project Website Development 10% 45% 45%
Blog Writing 80% 20% 0%
Overall 37% 35% 28%

Students and supervisors should only be able to submit a single Student Application or Supervisor Application. Despite this, the form does not become unavailable to them after they fill it out once. Ideally, it would disappear to stop them from accidentally submitting another application, which would cause the “View Application” page to malfunction.

On the Supervisor Application, the ACM Keyword selection and Research Group selections are implemented using subgrids rather than table fields. Therefore, they can not be made required fields even though we believe they should be.