Process

TIMELINE: 3-weeks

PROJECT TEAM: 1 Researcher & 1 Marketing Copywriter

MY ROLE: Research & Synthesizing Actionable Insights

MY METHODS: Prototyping, Unmoderated Concept Testing

TOOLS: Usertesting.com, Figma

Study Setup

Defining our users

We defined Credit Builders as a group of customers who are:

• taking actions to bump up their score

• frequently monitoring their credit score

• have already seen improvements to their credit score

• interested in refinancing their auto loan(s) because their credit score has improved

We knew very little about this group of users and had a few assumptions we wanted to test.

Research Objective

Does the emerging persona, Credit Builders, find value in being recognized for their efforts to improve their credit score?

Additionally, how does trust play a role here? Do the users trust the messaging and offer we are providing? Does the brand build or degrade trust?

Methodology

Working with a copywriter, I created 3 mock emails to test these assumptions and a potential new product (Rate Insurance). I then conducted an unmoderated concept test on usertesting.com to see how these three messages resonated with participants.

I created 3 variations of the test, each with 4 participants, for a total of 12 participants. Each participant would first see one version of the email and answer questions centered around their comprehension and trust of the content. For the second half of the test, participants saw all 3 versions of the email.

Screener Questions

To narrow down our participants and find our ideal participants, I created the following screener:

Test Versions

The Congratulatory email focused on celebrating the recent increase in the user’s credit score and letting them know they have “unlocked a lower car payment and this new 9.59% APR”. The imagery is very celebratory, with confetti and people jumping.

Assumption tested: Credit Builders enjoy being recognized for their efforts to improve their scores.

The Educational email focused on giving participants a tip on how to save money on interested by refinancing now.

Assumption tested: Credit Builders believe it will save money to refinance later when their credit score reaches a certain point, and they can get a better APR.

The Rate Insurance email tests a potential new marketing product that allows customers to apply for refinance now with the advertised APR. If their score improves later on, they can get an even lower APR applied to their loan.

Assumption tested: Credit Builders believe it will save money to refinance later when their credit score reaches a certain point, and they can get a better APR.

Test Questions

My focus for this test was to understand the Credit Builders a bit more and the assumptions we had about them. I also wanted to see how much they trusted the messaging and offers.

Examples of questions asked:

• Take as much time as you need to review the content presented. What, if anything, draws your attention or stands out to you?

• Why do you think you received this message?

• How does getting this email make you feel?

• How many points do you think your credit score would have to improve to receive a message like this?

• How trustworthy do you think this message is?

  • Participants were able to choose answers on a Likert scale

    • Very trustworthy —> Very untrustworthy

• In one sentence, describe the difference between the three messages. Describe it in your own words, not using language from the page you see.

Insights

Key Takeaways

• Trust in the message reflects the trust in the brand, not necessarily the message's content.

  • Participants with an existing relationship with this company had a higher level of trust and viewed the offers as personalized and special.

  • Participants with no existing relationship with the brand felt the personalization was intrusive.

• Participants valued being acknowledged for improved scores

  • Participants enjoyed being recognized for their efforts to improve their scores. It made them feel like their hard work had paid off.

  • Acknowledging the credit score change also provided context for why they received the offer.

Balance of trust, value & intrusion

There is a strong overlap between the user’s existing trust in the company and the user’s perception of the message.

• A high level of trust makes users feel like we are “on their side,” providing them with a special, personalized offer.

• A low level of trust makes the user feel like the message is a little intrusive

• Higher Comprehension leads to a higher perceived value of the message.

  • If the customer does not understand why they are receiving the message, it doesn’t seem as valuable.

Detailed Findings

Marketing Quiz: Testing the Comprehension of Content

At the end of the test, participants were asked multiple-choice questions to assess the lasting impression and memorableness of the content. They were asked to answer without referring back to the content. All questions could be answered in the headline or top paragraph of the email.

• [On the Congratulatory Email], you were told your credit score went up. What happened because of this?

• [On the Educational Email], you were told you could be overpaying on your car loan. What does this email recommend you consider as the best way you may save?

• [On the Rate Insurance Email], what will happen the next time your credit score goes up?

Marketing Quiz Results

• Most participants were able to answer the question correctly for the first versions they saw.

• Rate Insurance saw the most confusion on the details of the offer.

• Although the participants did not necessarily prefer the educational email, they seemed to understand the main purpose of the content.

Congratulatory Email Findings

What resonated:

• Most participants mentioned feeling “happy” when seeing this email.

• The personalization & imagery really stood out to participants.

• Many participants mentioned “no harm to your credit”

• All participants understood that they received this message because of their recently improved credit score.

What didn’t:

• When compared to the Rate Insurance version for the test, some participants felt like this offer was more “ambiguous” and “less direct”

  • “…guaranteed this rate IF you’re approved felt like the guarantee aspect felt contradictory”

• A few participants who had negative feelings felt like they were being watched or that this message felt more like a “sales pitch” than others.

  • These participants noted they did not have an existing relationship with the company.

Educational Email Findings

What resonated:

• Most participants did understand they could save money by refinancing.

• Most participants felt this email was trustworthy and they were likely to explore refinancing

What didn’t:

• Most of these participants did say they found value in getting an email recognizing their credit score improving. However, they did not think credit improvement was the trigger for this message.

  • Some participants were unsure of the relevancy for this message.

  • This message could be made more clear by telling themy they are receiving this for a specific reason (e.g. improving their score)

• Participants found this message to be harder to digest than the others.

  • Unclear if there was an offer and what the offer was.

  • Harder to skim because “there is a lot of text based detail”

Rate Insurance Email Findings

What resonated:

• All participants found this email very trustworthy or somewhat trustworthy

• When compared to other versions, participants said this offer seemed more direct and liked the concept of the offer.

• Most participants understood they received this email because of their recently improved credit score.

What didn’t:

• While users stated this offer seemed the clearest, there was confusion around the details of the offer?

  • What would happen if their credit score goes down? Does their APR increase?

  • Would this change happen automatically or would you have to apply to refinance again?

Understanding Value & Trust

Comprehension of why they received the email

• It was clearer to participants that the Congratulatory & Rate Insurance emails were sent to them because of their recent increase in credit score.

• Most participants that were shown the Educational email believed it was just a general message because they have a current auto loan.

  • Participants were unsure of the relevancy of this message.

General Trustworthiness of the Message

• Most participants found the messaging somewhat or very trustworthy

  • When asked why they trusted the message, most participants mentioned it was because of the company’s brand.

  • Some participants stated general reasons, like “the offer seems legitimate,” or they did not notice any red flags they looked for in scams.

• The few participants who said they felt neutral or untrusting of the message mentioned that they were unfamiliar or did not have a current relationship with the company.

Trust in Advertised APR

• One of our assumptions we tested was the distrust of advertised APRs

• For the versions of the test with stated APRs, participants were asked what they thought the rates would be if they refinanced.

• Most participants seemed to believe that they would receive the advertised APR if they refinanced. These participants stated the reason was because the offer was specific to their good credit score.

Value in Recognition for Credit Improvement

• Most participants felt there was value in being acknowledged for their credit score improvement.

  • Several participants mentioned that they found value because it makes them “feel like my hard work paid off,” and it can serve as “motivation”

  • A few participants also mentioned that the recognition made it “clear why the offer was provided”

• The few participants who did not find value in the recognition mentioned that it was because they had no relationship with the company

  • “You are not my organization; you are not my loan company. How were you able to know?”

Stated Preferences

When asked directly about their preferences, participants seemed to slightly prefer the Congratulatory email. Several participants stated they enjoyed the personalization and the imagery. A few participants mentioned it was also “easier to comprehend on the first pass”

Participants who favored the Rate Insurance email stated that they felt this offer was more direct and more informative about the offer. They also believed it was easier to read and understand

Participants preferred the other options over the Educational email because they preferred the personalization of the others and it was more difficult to skim and understand quickly

Summary & Next Steps

Recommendations

• Focus on Credit Builders who have existing products or services with the company.

  • This group will find the most value in getting personalized messaging and acknowledgment for their credit score improvement.

• Run a bandit test to help determine which version resonates more (Congratulatory vs Rate Insurance)

What happened next?

The marketing team got the confirmation they needed on the value of pursuing this emerging persona before they invested the tech effort to clean up the data on this group of customers.

I used this study to create a template our marketing could reproduce to test other campaigns independently with little oversight. I led the marketing team through a second concept test using this template and offered guidance during the synthesis. For the third test, our marketing partners were able to get insights on another potential campaign without input from my team.