Building HP’s First Participant Panel
Summary:
I spearheaded HP’s first ever user panel for a comprehensive study, establishing the framework’s 4 key pillars to enable other teams to replicate this process. This involved navigating various challenges and nuances, including identifying compliant programs and legal forms that adhere to HP’s policies, while also considering onboarding and approval timelines.
The case study is summarized by this visual, but to learn more about each pillar, the impact of our work and its future, and additional challenges we faced, keep reading to find out!
The original model (found on the case study icon) was built by CJ, and I adapted it for my case study
Introduction:
HP Instant Ink is an as needed, subscription-based ink replenishment service designed for specific HP printers, providing customers with a convenient, unique, and cost-effective way to maintain a steady supply of ink. As the lead UX Researcher on the Instant Ink Design Team, I spearheaded building HP’s first participant panel, and in doing so, established a replicable process for other research teams to recruit participants from their own customer base.
The Need for a Participant Panel
Instant Ink (II) wanted to conduct user interviews on a specific customer segment: II customers who were also enrolled in the HP Paper Add-on Service and utilized their paper in a particular manner. However, because this as-needed paper subscription was a recent add-on to Instant Ink, coupled with the specific user behavior we were seeking, this subgroup was too small to be found on our user testing platform’s panel.
A common issue we faced on our user testing platform was that participants could be fraudulent
Recruiting through an agency would have been more expensive and involved a longer timeline
So, we built our own panel.
Pillar 1: Reaching out to HP Customers
Teams involved: Researcher/moderator, ReOps team, Manager, Legal team, CRM team
Tools: Qualtrics, Sharepoint List, PowerAutomate (for automation)
Artifacts: Screener survey, Recruitment email
In order to reach out to Instant Ink customers, I had to collaborate with Instant Ink’s respective CRM (Customer Relationship Management) and Legal team.
I crafted a recruitment email to Instant Ink customers, informing them that HP Instant Ink was recruiting participants for a study. If participants were interested, I asked them to fill out a Qualtrics survey, which acted as our screener survey.
The recruitment email informed users on the following:
Interview dates
Payment incentive
Requirements (ex: minimum age, mandatory legal forms)
The Qualtrics email captured:
Standard demographic info (ex: age range, income range)
Customers’ Instant Ink plan
Specific user behaviors
A huge benefit from this recruitment method was that we did not have to screen for fraudulent Instant Ink and Paper Add-on customers.
Before deployment, both the recruitment email and survey underwent rigorous legal review. The CRM team manages the timing of HP’s email dispatch to prevent customer overload. Customers meeting the study criteria were transitioned to the next phase via automated processes (which will be explored in the next section).
Pillar 2: Coordinating with Participants
Programs: Adobe Sign, UserTesting.com
Tools: Proxy Mailbox (to protect my identity)
Artifacts: Templatize all coordination emails
Participants that met my study’s criteria were sent legal forms to sign via Adobe Sign. Only after signing, would participants receive an invitation link to schedule an interview time with me on UserTesting.com.
Our most significant challenge was gathering suitable legal forms participants needed to sign before I could interview them. Although our org had an existing NDA from prior in-house studies, we did not have a specific Informed Consent form for video recording. I collaborated closely with the legal team to craft an Informed Consent form that aligned with HP’s policies and could be used in future studies as well.
How automation fits in:
Originally, I coordinated with participants manually. However my ReOps team and I noticed a significant drop-off rate as participants progressed through the recruitment funnel. My coworker, Roberto hypothesized that users were losing interest due to the varying wait time between signing up on the Qualtrics/recruitment survey and scheduling a time with me.
To counter this, Roberto automated all of the coordination steps by utilizing Power Automate and various coding languages.
This automation significantly reduced scheduling time and funnel abandonment; a sample size that previously took us 2 weeks to schedule was completed in just 2 hours.
Pillar 3: Conducting the Interview
User interviews were conducted on UserTesting.com, and using the platform was no different than interviewing UserTesting’s panel.
At this point, you may have noticed that I haven’t mentioned any major cons of this method yet, aside from the manual labor involved. A huge con for this method was a horrible no-show rate of 60%. We quickly knew this was going to be an issue after our first three participants (in a row!) failed to show.
To combat this, we implemented automated email reminders to participants and noticed that those who attended their interview would confirm their attendance ahead of time. While this strategy did not improve the no-show rate, it did help us anticipate which participants were likely to miss their interview.
We had to explicitly state that if a participant did not show up within the first 10 minutes, they would be considered a no-show and would no longer be eligible for compensation (did I have someone show up 40 minutes late and ask why they could not enter the meeting room? Yes, yes I did)
In addition to automating the reminder emails, we also templatized an, “Are you experiencing any technical difficulties?” email that would be sent to participants if they were more than 3 minutes late
And the reason for this was unlike a third party’s panel, participants did not face any consequences for missing their interview times.
Pillar 4: Post Interview
After the interview concluded, participants received an automated thank-you email. Compensation was distributed via Tango, where participants could choose among 234 digital gift cards, including a Mastercard gift card.
From an operational standpoint, the signed legal forms were uploaded to the appropriate HP platform.
Impact and Future Work
Impact
This project was a huge win for HP! Previously, we relied on third-party panels for recruiting HP customers, which introduced biases into our studies. However, with this project, we achieved three major milestones.
Successfully recruited a niche target group, which our user testing panel was unable to do
HP now has a direct path for engaging with customers for feedback and the infrastructure to replicate this framework
A large number of HP Instant Ink and Paper Add-on customers have expressed their interest in participating in future user studies
Future Work
Moving forward, the next phase of this work involves reaching out to the interested II and Paper Add-on customers to invite them to join HP Instant Ink’s official user panel. Once this initial group is onboarded, the plan is to expand the user panel by including general II customers and then extending to non-II but HP customers. This expansion effort entails identifying new CRM contacts, securing legal compliance for the user panel, and maintaining up-to-date information for the panel members.
Conclusion & Additional Notes
The most difficult part of this project was identifying the necessary components, such as programs, legal forms, and collaboration with teams, that align with HP's policies while considering onboarding and approval timelines.
For instance…
There was a 90 day delay to fund our Tango account due to HP’s policy. To get around this, I asked to use another org’s Tango account temporarily. Although this was a viable solution, it meant interdepartmental cooperation and borrowing funds.
However I then discovered that the other org’s Tango account lacked the MasterCard payment option due to an outdated HP policy. To overcome this, I had to coordinate between Legal and the relevant team to allow for MasterCard payments.
Despite these challenges, establishing these foundational pillars enabled HP to directly engage with its customers for user feedback. This work also lays the groundwork for future scalability and sharpens my ability in spearheading Research Ops initiatives within the company.