Supriya Pandit is a digital product management leader with experience in retail, ecommerce, and software development. She began her product management journey at Ticketmaster and later joined Macy’s, where she helped develop their first mobile app and onsite search experience. Supriya served in product leadership at GSPANN Technologies, Art.com, BOLD, and, most recently, Backcountry. She has a graduate degree in film and television and cites the commonality that holds both together as the “ability to tell a story with empathy.”
In our conversation, Supriya shares her experience integrating machine learning (ML) technology into a large retailer’s web presence and the associated challenges, including scale, data quality, attributing products, and connecting in-store data to dotcom data. She also discusses how her team at BOLD flagged a substantial user dropoff in a specific user segment — high school students — and leveraged rapid prototyping to fix it.
BOLD is the parent company of brands like MyPerfectResume and MyPerfectCoverLetter, and I was the director of product management. You can upload your resume or provide basic information for your desired role and company, and the software generates the best content to use to fill your resume. When I joined, we used a one-size-fits-all model. It felt overwhelming to the customer and we couldn’t decipher which segment was struggling the most. That was my deep foray into understanding segmentation. We started looking at funnels, behaviors, and drop-off rates.
We found that a chunk of the people who came to our website were high school students. They were very early into their professional lives but had to submit resumes for part-time jobs or college applications. What they lacked was the ability to synthesize babysitting, dog walking, or raking someone’s yard into real-life work experiences because they undervalued the importance of those jobs.
When we saw a massive drop-off in students, we were left wondering what the root cause was. We realized that the reason was fear and an inability to articulate their skills, so we created a questionnaire that made it easy for them to take an experience that they had as a student and translate that into a relevant skill on their resume. We saw a huge positive impact on our finances, as well as an uptick in how students were converting on the site.
I’ve been using ChatGPT and testing it out for other reasons — mainly integrating that into building my own company. ChatGPT can make every resume sound similar. It doesn’t give you a unique perspective of your life. If I use it on the same day, 10 different times, and ask questions in different ways, I still see a repetitiveness to the ChatGPT output. People are smart — it’s very easy to discern when something was made with ChatGPT.
While it’s a great tool, I would caution people to use it carefully. Make the effort to create something that personally speaks to who they are. You don’t want to create a situation where the hiring manager is not going to trust you. Many people, particularly HR folks who interview fresh graduates, are pretty good at picking up that scent.
Rigorous, extensive research. We’d hired a research team and got a ton of qualitative and quantitative data. We also created a cohort of our users that we could incentivize to come talk to us. We were talking to them and looking at the behavioral data. Our analytics team was holding our hand all the way along on the journey. We were dissecting data from the student cohort from every angle possible.
At least once a month, we also look at the session data. We had a flag in the data that could create that segment for us, and we looked at their behavior to create the right questions to ask during research. As the open-ended research started giving us enough cues, we started identifying where the problems were. We then started rapid prototyping, which was amazing. This was actually one of the most exciting moments for me. I would sit with my team on these calls, and we would have one researcher lead. It’s very skillful work.
We’d all join a Zoom call and have our senior researcher walk the user through the prototype. The questions had to be very precise and not open-ended so we could extract accurate information. Often, multiple customer reviews were done on the same day. After the call, we would review the outcome as a team, and often, the designers would make tweaks on the same day. Then, on the next call, we would show a new customer the new version of the prototype.
Doing so helped us make sure that we were getting to the bottom of the issue we had identified. It’s good to run your prototypes by at least seven to 10 people. We needed to refine the prototype with enough people so that, in the end, we felt good about this solution and could send it to the engineers to build. This was also very cost-effective in ensuring minimal wastage of engineering resources.
The big problem that I faced with large retailers was data — data quality, data integrity, and data attribution. Eventually, dotcom wanted to marry their data with the store. For example, we wanted to make sure that what was available online could be picked up from a store. Or what was available in the store reflected the right inventory numbers on the website.
That is very difficult to do at a company. The naming conventions for a product are not consistent. For example, I may call a pant “pants” while you might call them “trousers.” That completely ruins how a product is attributed. One of the favorite parts of my job, aside from creating user experiences, was building search. And search is completely based on how well your data is attributed.
Another great example is a common search term — “gravy boat.” If a merchant has not attributed a product “gravy boat,” you might see something with a boat or might see something with gravy, and neither is what the customer is looking for. This is when attributes become extremely powerful. We incorporated machine learning into attributing data at scale.
We worked with our teams internally, as well as with some external companies that were building ML models specifically for retail. For example, people loved to shop for handbags, so we made sure that we worked with the merchants and curated enough examples of handbags so that the machine learning team had a robust chunk of data to work with. The machine learning algorithms then started to recognize those images based on visual search and could attribute hundreds and thousands of products more efficiently. Merchants no longer had to individually attribute data across dotcom and the store.
As product managers and executives, we are unique and occasionally lose sight of other teams. We are very excited about what we want to do. We want to run with it. We see all the business aspects: revenue, impact on conversion, customer frustration, etc. We have OKRs and the revenue numbers we need to bring in.
Not everybody has the visibility or is looking at a problem in the same way. The skill that we had to learn as a product team was to be able to speak the language of other teams impacted by our product. Not all teams consume information the same way. The first challenge is that everybody has a day job. During the attribution project, the buyers and merchants had to spend a lot of time looking at the output of a machine-learning algorithm and say whether it looked right or not.
We learned that the sooner we got a financial model in front of teams, the more inclined they were to help, and the more inclined they were to commit. We also learned to be accommodative in our asks. If they did not have the time, we needed to respect that. Thanksgiving and Black Friday sales are critical and we could not distract the crew for our longer-term attribution project since those results were not immediate. We often reduced the scope of our work. From that, we learned humility, consistent communication, and presenting data in ways people could relate to.
One of the CEOs I worked with would ask, “Do you have a plan?” It’s a straightforward but extremely powerful question. It forced me to think about how to plan projects in an all-inclusive way — especially if they were cross-functional. As a retailer, executing a project in November is a non-starter since the entire company is focused on holiday sales.
But, what I could do is use November and December to plan. I could create cross-functional alignment and get my numbers together. The storytelling process could start so that in January, when they come up for air, they could get excited and see exactly how they were going to impact next year’s metrics. Q4 is a time to build relationships, plan, and be helpful to the business at large. That way, you can build internal trust. You don’t want to be a product executive whose only focus is OKR-based. It was crucial to be on the journey alongside the cross-functional team.
I’ll use the same example. There was tremendous nervousness about using machines to attribute because machines can misattribute. If you misattribute things online and rely on algorithms to create recommendations, you are at great risk of creating terrible product recommendations. Retailers were used to a high-touch process where they hand-curated product recommendations. There was a lot of nervousness and anxiety around giving up that control.
Building trust, showing them examples, and bringing these teams along on the journey from the get-go was critical. We helped them be part of the solution instead of handing them a solution. My learning was to make no assumption on how well I understood another person’s job.
At one of the companies I worked at, our CEOs were focused on identifying the most underserved cohort and building solutions for them. One of the most underserved cohorts is people who are retired. They have many shareable skills, but we live in an ageist culture that is unable to tap this wealth of knowledge.
There is so much wisdom that goes untapped. Every generation believes that they are spectacularly unique in what they’re going through — I think we repeat the same mistakes but the flavors are always changing
I am hoping to use ML in a way that maps the needs of folks who are young but don’t have the means to train themselves with the skills of people who don’t need the money and are willing to provide that help. This creates a purpose for older folks — giving them another chance at making an impact. Isn’t that what we all want in the end?
LogRocket identifies friction points in the user experience so you can make informed decisions about product and design changes that must happen to hit your goals.
With LogRocket, you can understand the scope of the issues affecting your product and prioritize the changes that need to be made. LogRocket simplifies workflows by allowing Engineering, Product, UX, and Design teams to work from the same data as you, eliminating any confusion about what needs to be done.
Get your teams on the same page — try LogRocket today.
Want to get sent new PM Leadership Spotlights when they come out?
Hypergrowth happens when a company experiences an exceptionally rapid rate of expansion, typically more than 40 percent annual growth.
Detractors have long-term effects like negative brand perception, reduced customer loyalty, and a decrease in sales.
To proactively address liability concerns, you can create an internal product recall team for dealing with risks and ensuring quality.
Our Galileo AI Fall ’24 release makes Highlights available to all Pro and Enterprise customers. We’re also debuting Ask Galileo, which enables you to directly ask questions about what you see in session replays.
Learn more →
