smule | design

Usability Test on

Sing! Karaoke by Smule

Evaluation report for a karaoke-based mobile application created by Smule

Overview

End Date: 12/2014
Duration: 4 weeks
Group project (5)

My Role

UX Researcher
UI Researcher

Tools

Interview report
Usability testing
Heuristics evaluation
Comparative analysis
Survey report

Background

Sing! Karaoke is a karaoke-based product created by Smule, a company that designs mobile music products that focus on social media interaction. Sing! Karaoke is available for free download via the App Store (Apple’s marketplace for IOS apps) and Google Play (Google’s marketplace for Droid apps). The product offers “in-app” purchases to enhance the user experience, however, the user can benefit from limited functionality by using the items that are offered for

free. Users benefit from the product by having the ability to sing and record their voice to backing tracks offered within the app. The application uses the microphone function of the mobile device and allows the user to hear their own voice as well as the backing track though the headphone output.

Goals

The goal of our study is to provide Smule recommendations on how to enhance the Sing! Karaoke product to attract more customers, create a better user experience, and propel Sing! Karaoke to be among the top three karaoke-based applications offered. Our team of researchers conducted three usability and two heuristic tests to gather information of user interaction for the Sing! Karaoke application. This report will include qualitative data gathered via a set of questions and a scenario that users were asked to participate in. Our team sought out individuals who were familiar with the Sing! Karaoke app as well as users who were new to the product.

High-Level Goal

The high level goal for our team is to attract more customers for the Sing! Karaoke app and to enhance the user experience of the application. While the app itself is extremely popular, we feel confident that the data we are collecting from our usability and heuristic tests will help form recommendations that will allow the app to become more attractive to potential customers. Data collection in this report includes not only the successes garnered from test subjects who used the app, but also frustrations encountered during the tests as the users attempted to navigate through Sing! Karaoke’s interface and participate in options the app offered. Customer satisfaction was one of our primary focuses in collecting our data. Users were asked questions regarding the difficulty level of using the application, their satisfaction with in-app options, and their overall feeling regarding their experiences.

Mid-Level Goal

Our mid-level goal is to give Smule recommendations that will help Sing! Karaoke app to become - and maintain when achieved - one of the top three karaoke applications on the market. Our team believes the recommendations for improvements we will offer will meet help Smule meet this goal.

Plan and Method

Plan

The evaluators of this project utilized various sources of information to capture a precise portrayal of Smule Sing! Karaoke App. This evaluation was planned utilizing a four-tier planning approach as presented in Figure 1 below. The qualitative data were analyzed to triangulate the perceptions about the Smule Sing! Karaoke App, as well as to develop effective recommendations. The following steps were used for the evaluation:

1. Initiation

In the initiation phase, the evaluators began the process by holding an initial team meeting on November 14, 2014. During the initial meeting, the team aimed to gain a common understanding of the projects by deciding the application. The team also gathered background information about the application and developed high-level and mid-level goals.

2. Data collection

Phase two consisted of data collection. Preliminary data was collected from 3 (three) usability tests with the users and 2 (two) heuristics evaluation with the experts in the field of music (see Appendix C). The usability tests consisted of a set of questions and scenarios, while the heuristic evaluations comprised 10 heuristic principles suggested by Nielsen and 4-5 additional questions the team generated. The evaluators also noted some essential information during the interview.

3. Data Analysis

The third phase of evaluation consisted of data analysis. Planned analysis methods include thematic analysis as well as content analysis of the interview results, scenarios and heuristic evaluation principles. The content analysis method was used to find out relevant and important data from the interview results related to the goals of the evaluation. The thematic analysis method was used in order to generate some themes or topics so that the team was able to group the findings.

4. Final Production

Based on the results of data analysis, the team identified the findings as well as constructed some recommendations for future implementation. Following this phase, the team will present initial findings on December 10, 2014. After collaborative revisions, the final report will then be submitted.

Methodology

For the testing methodology, it includes participants’ information, scenarios for usability and heuristics for the heuristic evaluation. The method our team adopted is two heuristic evaluations with 2 experts and three usability testing with 3 test subjects.

The length of each usability test was around 40 minutes; including:

Pre-test preparation: 10 minutes/ Task Scenarios: 5 minutes/ Post-test interview: 25 minutes

Two heuristic evaluations took more than one hour; including:

Pre-test preparation: 10 minutes/ Observation: 30 minutes/ Similar app comparison: 20 minutes/ Post-test conversation: 10 minutes

Nielson’s 10 Heuristics:

1). Visibility of system status

2). Match between system and the real world

3). User control and freedom

4). Consistency and standards

5). Error prevention

6). Recognition rather than recall

7). Flexibility and efficiency of use

8). Aesthetic and minimalist design

9. Help users recognize, diagnose, and recover from errors

10). Help and documentation

Conclusions

Tasks

The tasks that were performed by the participants are indispensable to the validity of the test.

1. Usability Tests:

We provided the participants a scenario and interview questions below:

“You’ve just downloaded and installed this interesting karaoke app named Smule. You’re not into pop music and want to sing “Summertime” by George Gershwin, attempt to find this song in Smule.

What was your impression of the initial layout of the app?

How readily available were the tools at your disposal to find songs? What method did you use? Do you think there could/should be a better way?

Overall how easily were you able to find the song?

Assuming you’ve found the song, tap the “sing” icon to sing it. Don’t do anything past this.

What is your impression of the screen you now see?

How does it make you feel?

You don’t want to sing in a group, but you’re low on cash. You’ll notice that you have 10 credits, but the song is 65 credits. Go to “Get More Credits” at the top. Attempt to acquire the needed 65 credits in whatever way you chose/feel most comfortable with. [Start Timer]

How easy was the overall process?

At any point in time were you frustrated with the system?

Did you feel adequately compensated for what you were asked to do?

Keeping in mind that this is one of the only ways these songs can be free (and thus some aspects of this process are a necessary evil), what suggestions do you have for improving or streamlining the process of acquiring credits?

Were you able to easily find opportunities for credits that you were comfortable with doing?

What if any opportunities did you turn down because you were not comfortable with them? Why?”

2. Heuristic Evaluation

Jakob Nielson’s 10 Heuristics have been adapted for our heuristic evaluation and other 10 project-specific heuristics have been added as well (5 for each evaluation).

Project-specific Heuristics:

For Evaluation of Participant 1

1). Singing Tips on Mobile Devices

2). Privacy Protection

3). Hot Recordings Relevance

4). Free Music Trial

5). High quality of backing track music

For Evaluation of Participant 2

1). Social Media integration

2). Privacy Setting

3). Advertisements

4). Interactivity (Interaction with the partners)

5). Sound System or Mixer (Voice Production)

The results of the usability tests and heuristic evaluations revealed several prominent design issues with the Smule: Sing! Karaoke app. These issues encompassed problems with both the visual design and the procedural design, and were highly consistent between participants. All identified design problems were organized into 5 separate categories shown below.

1. Privacy Setting

A great many off the “offers” which users could use to acquire more credits (see: Credit system), elicited strong objections from users. Two of the three usability test subjects were very much concerned with giving their email information to entities unknown, which were clearly not associated with Smule, but which Smule directed users to for free credits. The number of “offers” which required personal information from users was overall too high and harmed the users’ experiences. They made users uncomfortable and required them to expend extra effort finding offers they were comfortable with.

A solution to the issue with personal data and “entities-unknown” as a privacy issue is detailed below. Without addressing that issue directly, however, Smule can still improve the user’s experience by separating “offers” into categories. Since the majority of offers which caused discomfort were survey related (due to their ubiquitous requirement for the user’s email), placing them in a single category would mean that users who are concerned with privacy could quickly and easily access offers which they are comfortable with accepting.

Separate from just concerns of privacy however, is actually the concern of the legitimacy of some of Smule’s offers. Indeed it came to the attention of the investigators that some of Smule’s advertising partners may in fact be disreputable (even if technically legal) business partners. We recommend that Smule re-evaluate some of their advertising partnerships. One user: Serena C noted that a survey offer which was presented asked the user about their preference for CheetosTM or Flamin’ Hot CheetosTM, however in the fine print was the following:

This is an advertisement powered by Gift-Rewards.net, an independent rewards program that is not associated nor endorsed / sponsored by any of the above listed merchandise or brands …

This would rule out Gift-Rewards.net as a market research company in service of Frito-LayTM, and implies that the entity is not interested in the user’s survey response, but rather in acquiring their email. The acquisition of such information ought not be necessary if this were a legitimate survey.

2. Credit System

Smule Sing! Karaoke uses a credit system in order to make songs available to users. Songs which are considered free typically require the user to accumulate a set number of credits to access them. The credit values are not consistent across different titles and the exact method by which the product arrives at these values is unclear. The song “Summertime” from Porgy and Bess by George & Ira Gershwin currently costs 65 credits. Users can acquire credits via 3 main methods (divided further into several sub-methods):

Facebook: Users can like or connect to Smule: Sing! Karaoke on facebook for a one time reward of 5 and 10 credits respectively.

Advertisements: Users can watch a 10-30 second advertisements for 2 credits. It appears that this can be repeated indefinitely, but that is unclear. A user would have to view 33 such advertisements per 65 point song, and thus Smule: Sing! Karaoke would need a tremendous number of advertisements (without repeats) available to make more than a handful of songs available for free using this method.

Offers: Offers can be largely divided into 3 primary categories:
- Downloading and starting another app.
- Taking a survey
- Signing up for a paid subscription to some other service.

This credit system is the only means by which users can sing a song (individually) for free in Smule Sing! Karaoke.

Users took issue with both methods b, and, c, and by extension the credit price of individual songs. It was noted by both Serena C, and Rebekah H, that a 2 credit reward for a 10-30 second advertisement was frustrating and discouraged both users. “Summertime”’s credit price was standard for other songs on the site, and at 65 credits would mean that a user would have to sit through 33 advertisements starting from 0 credits, if they wanted to play the song for free solely through watching ads.

One might note that giving such a low point value to advertisements may be a strategy on the part of Smule to encourage users to primarily use offers rather than watch ads to sing songs for free. The implication here would be that users should use ads only “round out” their credits if offers provided them with a number of credits just below the threshold required for the song. While this is a logical approach the negative user reaction would seem to indicate that there is substantial room for improvement.

We recommend that advertisements not be listed as an option unless the user’s current credit surplus is very close to the price of the current song. This way the implied use of watching advertisements which falls out from Smule’s current model will align with the presentation of the option to the user.

The other option for reducing user frustration with the lack of credits given from watching advertisements would be to simply give more points for advertisements (even 5 points/ad would mean a user would have to watch 13 ads back to back to afford the average song), or lower overall credit prices. This is the most obvious option. Consider that the average internet user familar with Youtube.com generally equates one youtube video with ~10mins of content with ~15-90 seconds of advertising. Thus it is a user’s expectation that a relatively limited number of ads should account for a significant amount of content. This is also further evidence of why ads should be subordinated in the UI to offers, if the company chooses to keep its current model of credit pay-outs. The UI and the users’ own experiences bias them to expect more credits from watching ads than what Smule pays out.

Given these two possible means of addressing the problem we recommend that Smule conduct an economic analysis of the cost/benefit of both methods. It is not unreasonable to at least investigate the possibility that giving more credits for advertisements will lead to an increase in advertisement viewing since users may very well likely sing more total songs per session as a result. This is especially important because of the user’s distaste for the available offers (see: privacy concerns). Offers adequately compensated users in their opinion, but again, see privacy concerns.

3. Tutorial/ Help

The Tutorial guide only appears once upon the initial use of the app by a given user. However, many users ignore tutorials in general in favor of a quick “head-first” approach. These users prefer to use guides and manuals only when confronted with problems as a result of their haste (or an unclear UI). While there is a help button, it provides only FAQs, and a great many of them at that. The preponderance of FAQs in the list led to expert user Zhuoyi Q suggesting that there be a search function. We recommend not only that Zhouyi’s suggested help-search be implemented, but also that the tutorial be accessible as a part of this help menu. Furthermore, non FAQ documentation, as in actual user guide pages would be useful to have as a part of the help system, and is another recommendation of this committee.

The colors used are aesthetically pleasing, but otherwise the home screen’s layout is cluttered. The surface of icon of Smule is complex (confusing to some users). Some icons do not appear to have clear functional details associated with them, forcing users to, guess as to their functionality. The most important aspect of designing an individual icon is that it has to be instantly recognizable, and its function easily decerned. Along these same lines, icons must not be deceptive as to their function (maliciously/intentionally or otherwise), or imply some meaning or function that is not the case. This was a problem noted by Serena C in particular, who used the term “bait and switch” for the “sing” button. While some “sing”

4. Design

buttons have a “vip only” icon which implies they must be paid for others do not. By extension the implication is that any song without “vip only” must be free. However, since they cost credits, this is not the case. Serena C mentioned that she felt the app was being disingenuous and that the screen which comes up after pressing “sing” which asks for money or credits to sing the song alone was a let-down. They could be easily solved by indicating which songs (if any) are truly free and indicating which songs cost credits. Curiously this is a feature already of one of Smule’s other apps “Magic PainoTM”.

The apps layout came under criticism from 1 expert and 2 usability test subjects. In particular, the lack of prominence of the search box led to both users taking longer than needed to find the song in question, and was noted by the expert. The ordering of song categories was also confusing. Expert user Andrew R recommended that the categories be organized alphabetically, which as the authors of this evaluation, we endorse.

5. Backing Track

Enable users to sing with the original and high-quality backing track.

This is probably the most disappointing part of this app. Due to copyright issues, the app cannot provide original and high-quality backing track music, while the Chinese Karaoke app -- Changba can provide them very well. Instead, Sing! only offers a version of very cheap and coarse MIDI backing track, which was a pretty frustrated experience for expert users like Zhuoyi.

Our team would recommend that the app need to negotiate with recording companies for being authorized original backing tracks, or provide a high-quality alternative of backing track music.

If you are reading this line... Congratulation!

Because you might be the only person other than me in this world who reads through the report！

Though I turned it to my professor of HCI class and received A+, well, I do believe he did not finish the whole thing...