We worked with the City of Boston to evaluate their current "how to" pages with a broad range of users. During our evaluative usability testing phase, we found that scanning was the most prominent behavior when completing a task on boston.gov. This scanning behavior was causing our users to miss important details related to their task. How might we improve the pages to ensure that users are getting all the relevant information they need?
People need a way to digest important details quickly when completing everyday tasks on boston.gov.
They miss some things entirely, or don't feel confident in others, and fall back on inefficient and often incorrect analog methods.
Turning tabs into accordions will improve the scannability of how-to pages through progressive disclosure. This will mean users can feel more confident they haven’t missed any crucial details.
What were our research goals?
- Review and gain insights from previous research that was done by other design agencies
- Determine the current site's usability through user interviews and evaluative usability testing
- Identify any potential opportunities for improvement
Who were our users?
When working on projects in the past, we had a user in mind when designing. However this particular project created a challenge for us because with working with the City of Boston, virtually every person is our user. We tackled this problem by identifying the users' behavior.
What did we find?
- We found the same improvement opportunities as other design agencies. Most users were using the site to pay parking tickets and pay excise tax but were having trouble with navigating the site which led them to calling/going to City Hall to complete their task.
- Other areas of improvement include making the search icon more prominent on the home page, using consistent taxonomy to avoid confusion, and continuing to improve SEO as most users found their way to the site via Google.
- We heard stories from residents at City Hall that indicated that either information was missing on the "how to" pages or the information was there and the user just completely missed it. We started to wonder why this was happening.
Narrowing our scope
With only three weeks to complete this project, we decided to focus on the "how to" pages. The Digital Team at City of Boston wanted us to find out if these pages were effective in helping users find information. The other research that we had done regarding navigation issues was currently being addressed by the Digital Team at City of Boston so we felt that our time could be better spent in researching the "how to" pages and determine if anything needed to be improved.
Evaluative Usability Testing of Current Site
- We tested 15 people on the original site, with people in our immediate network, at City Hall, and with local organizations like Tech Goes Home and Little Brothers Friends of the Elderly. We wanted to ensure that we were testing with a broad age range of users.
- We tested our users by giving them a set of tasks to complete. We found that users were failing on task 1 and so that should be an area to focus and figure out why that was occurring.
- During our evaluative usability testing of the current site we had 8 users complete a System Usability Scale (SUS) survey. The results indicated that these users felt that the site was easy to use and they would recommend using this site. All of the scores beat the national average of 68.
What were users saying in response to the "How to" pages?
Why were users having difficulty in finding information?
During our usability testing, we could see that users were frequently scanning the page for the information. We tested on both mobile and desktop versions and people were missing the tabs which meant they were missing 3/4 of the information on the page. When using the site boston.gov, users are very task focused (pay a parking ticket, pay taxes, etc.) and so they want to quickly get through this process. This also made sense given that previous research done by the Nielsen Norman Group that states that "79 percent of users always scanned any new page they came across. Only 16 percent read word-by-word." When they couldn't find the information, they would:
- Give up completely
- Do the process incorrectly
- Call or go to City Hall to complete their task
How do we get users to read the content?
- Focus them on information through progressive disclosure
- Give them information at the point of need
- Make it scannable
We first experimented with sketching out different ideas for "how to" pages. Some ideas that we liked but we didn't take forward was the idea of a decision tree model to guide users through their task, annotated graphical representation of parking tickets we thought could be useful for non-English speakers, and a printable checklist.
These ideas were not implemented because we wanted to make a universal change to all the "how to" pages. While we thought these could be useful, these ideas seemed too narrow in order for us to apply changes universally.
Make tabs into accordions
We knew that users were missing the information that was hidden in other tabs. We experimented with 2 alternatives. The first being changing the tabs to accordions. We stole this idea from mass.gov. The second option being to take the tabs out of the header and convert them into buttons. We tested these 2 alternatives with 15 users on a paper prototype.
We overwhelmingly found from this quick paper prototype test that users preferred accordions. We decided to proceed with making a mid-fi prototype with accordions to test further with users at City Hall and Boston Public Garden. You can read more about the paper prototype testing results here.
Make content relevant at the point of need
We could see from our initial usability testing of the current site that users were having trouble with finding key information related to their task. We had to come up with a way to make this information "pop". Since we really didn't know what pieces of information users would want to see first and where it should be I suggested that we do a participatory design exercise with users by dissecting parts of the "how to" pages and see where users wanted to put the information. In doing this exercise we learned that users wanted payment method-related information given to them at the point of need. Giving them all the information upfront was overwhelming. We then started to experiment where to put this payment method-related information.
- Should information go inside the accordion?
- Should it go outside the accordion?
- Should it be in both places?
Make content scannable
Information that was relevant needed to have something eye-catching to draw the users' attention. We had seen that boston.gov had used exclamation point icons to alert the user so we wanted to experiment and see if that would help in finding the information more quickly.
Usability Testing of Our Redesign
Throughout this design we had tested 56 users in total, 29 with our redesign. We had two questions that we wanted to answer for our redesign.
- Can people use accordions?
- Are people able to find relevant information for a given task?
We found that all users were able to use accordions. Users knew how to open and close the accordions without hesitation. Users were able to navigate the different methods of payment when we asked them to switch from paying online to pay by mail. The only improvement that we heard from users was that they wanted to see all their options for payment at the top without having to scroll down. We iterated and decided to make the accordions smaller so that they could fit before the fold of the screen.
We found that the accordions seemed to help with users being able to find information that was located on the right side of the page or inside the accordion (about 80% of users were successful) but information that was in the "Keep in Mind" section at the bottom was still being missed (about 40% of the time). This suggests that information in this "Keep in Mind" section is not being read. We think that we were successful in getting 80% of users to complete their task because we had put the information both inside the accordion at point of need and also on the right hand side. We think this purposeful redundancy is useful especially when dealing with such a wide user base. Some people find it in the accordions, others find it on the side.
Both in the current site and our redesign, we asked users to find where they would mail their parking payment. Users would tell us that they would mail their parking ticket payment to the address on the right hand side. However they actually needed to send their payment to the address located within the step as seen below. The same problem occurred when paying by phone as well. In order to solve this problem we moved the contact information to the bottom of the page to avoid confusion.
For a more detailed explanation of our design choices, please view our annotated wireframes created by my teammate cass Hebert.
We felt really confident in changing tabs to accordions. All users were able to use accordions and we think it helped to make the page more scannable getting users to find relevant information more quickly.
We need more testing to validate where to put information on the page. We think that the addition of exclamation point icons were helpful in making certain information "pop". Our recommendation to the City of Boston is to work with individual departments on their "how to" pages and learn what information should be prioritized. For example, "What do people call your office the most for?" If that information isn't on the page, add it. If it is on the page, put it at the point of need during the process or add some visual cue like the icon. We would have liked to continue our research in this area but we wanted to make sure that we had a winner with the accordions.
I am so grateful to have had the opportunity to work with the City of Boston. It was definitely a unique experience in that we were designing for everyone. Not to mention that we came into this after IDEO had done their redesign so we weren't even sure if there was anything to improve upon. But as it turns out, there's always something to be improved. You can make a whole project about changing one aspect of the site and it can have a huge impact. I learned that while our design choices are meant to benefit our users, it also has to make sense for the client.
I was heavily involved with the research aspect of this project including interviews and multiple rounds of usability testing. If I could go back in time I would have told myself that you really need to come up with a better note-taking system. Handwritten notes, both my own and teammates, were hard to discern and took time to transcribe electronically. I wish I had made an interview notes template in the beginning so everyone on my team was following the same format and also so I could see all the users' responses on one page. It would have made my life easier when trying to pull out the common themes. But you live, you learn, you iterate.
What I learned from field research
Guerilla-style interviews and testing are hard work. But I'm so glad that I did it, and I did it a lot. With my teammate Michael Greig by my side, we took on the task of guerilla-style interviews and usability testing at City Hall and at Boston Public Garden. I learned many tricks along the way.
- Don't try to approach people in pairs. It scares them off.
- Have a laptop out. It makes you seem more credible.
- Location, Location, Location- Setup shop where people are coming and going. Also if people are waiting in a long line they are more likely to chat.
- Tell them upfront how much time it's going to take them to complete.
- Make them feel involved in the design process. "The City of Boston wants your feedback so together we can make their site even better."
- And most importantly, DON'T GIVE UP. There were many times that we felt dejected but we kept going for that chance that someone would stop and chat. And they did. I'll say that everyday we went out to do interviews or testing, we always gained more information than what we had so it was always worth the effort to get the feedback from real users.
Special thanks to my team: