Ayan Basu is AVP, Head of Product of TOEFL at ETS, a nonprofit educational testing and assessment organization. He started his career as an engineer and worked for nearly a decade at Samsung Austin Semiconductor before transitioning to product management. Ayan has worked in various product and leadership roles in industries such as electronics, data, and cybersecurity. He currently oversees ETS’ largest revenue-generating product: the English proficiency test.
In our conversation, Ayan talks about what it’s like to work on a product that spans more than 160 countries and differs slightly in various markets. He shares the challenges that come with navigating so many types of stakeholders, including students, teachers, test-takers, governments, and universities, and the importance of accepting that not everybody can be on the same page. Ayan also discusses ETS’ efforts of using AI to enhance the learning process and develop more accurate ways to improve assessment takers’ skills.
With the different user segments you mentioned, research and assessments are core to what we do. There’s a lot of psychometric analysis and science-backed research that goes into developing these learning products and solutions. We have the TOEFL, which is what I manage. There’s also the GRE, TOEIC, Praxis, and other assessments. There are different kinds of products for test takers, students, teachers, licensing certification, etc., but also other solutions that we offer to institutions where they can place their students into different levels of classes.
We have a B2B model as well. With governments, we do a lot of research on different policies around education and help them in designing new ones going forward. Another aspect of working with governments is the immigration policies that they have with TOEFL. For instance, going to the UK and Australia, it’s more of a requirement to pass an English test if you’re not a native language speaker. Immigration authorities use our English proficiency test for things like student visas or work visas. ETS has been around for decades and has had a lot of impact worldwide.
It’s super challenging. This is the most complex set of products I’ve worked on because we have all different kinds of users and feedback coming from everywhere — governments, B2B, organizations, institutions, students, and test takers. And test takers can be teenagers, young adults, or working professionals in corporations. We’re catering to just about everybody you can think of.
If you think about taking the TOEFL test from a student perspective, we have to make sure they have a good experience and that the university that accepts the TOEFL and associated scores. The students need the test to prove the proficiency of their English skills so that when they go to universities, they can fit right in. We have to test for those things. Not only do we have to make the students happy, we also have to make sure the university is getting the right kind of student who can be successful.
The added complexity of what I’m doing right now is that not only do we have a global product strategy, but also we need to have some local market specializations. This is around asset creation, positioning, messaging, and some market-specific services that we can offer. For example, we have the TOEFL test, and we have a whole ecosystem for that. Test takers have to prepare for it, so we have test preparation solutions. And how test prep works in China versus India versus EMEA is so different. Even though the test is the same, we have to think about how to create these test prep products with different strategies by market.
There are multiple channels. There are feedback buttons within the products that we offer, and we also do regular surveys with actual test takers with institutions. We regularly visit them. We have an advisory board with certain institutions and a core set of folks that give us feedback. Governments and authorities provide us with regular feedback too.
We have a lot of data coming in from all over the place. We have a market research team that’s regularly serving the market and looking at trends, as well as a competitive intelligence team. We look at what competitors are doing and keep a finger on the pulse of what’s happening in the industry. All of these things we have to collate to feed into what the product roadmap actually looks like and what the future product strategy is.
Specifically, a big area of feedback for us is in China and India. They have giant networks of study abroad agents who help students figure out what test they should take, where they should apply to university, what the application process is like, the financing, visa requirements, etc. There are a billion different things that they have to think about. We have to monitor what’s happening in the competitive landscape so that boots-on-the-ground feedback relays what we need to do next.
As humans, it’s natural to gravitate toward hearing what we want to hear. We tune things out that we don’t resonate with. I think it’s always hard to face the truth and hear uncomfortable things about what’s wrong with your product. As a product manager, you become very passionate about what you’re putting out there. The proof is in the pudding when it’s in the hands of users and they’re giving you feedback. You have to take a deliberate step back and spend more time with the unhappy users to truly understand the root cause of the unhappiness, and figure out if it’s the thing that you should work on based on that prioritization framework.
At the moment, we’re spending a lot of time with the universities because ultimately, they decide what they accept in terms of test results. Asking questions around “If something isn’t working for you, why is it not working?” is helpful. Kevin Kelly has a really good quote on this in his book Excellent Advice for Living: Wisdom I Wish I’d Known Earlier. He said something like, “If you ask somebody a question the first time, they’re not going to respond with what you want to hear.” Keep digging. By the third or fourth question, you’re going to get the true answer.
With these universities, I don’t want to ask them one question and say, “Hey, how’s the test?” I want to keep poking into the details and if I get a little thread, keep pulling on the thread until the actual truth is revealed.
The feedback that we’ve gotten is that our registration and signup flow is not easy. We have survey data so we can look at the results and see what is going on. A key source of feedback is also the data through your products and looking at user behavior and metrics.
One of the low-hanging fruit quick fixes is to simplify that initial signup flow that we identified in our survey data. Be able to quickly create an account and go through the process. Those are small things around the test-taking experience that we are actively addressing.
Everybody has an opinion. At the end of the day, no matter how much feedback or data you have, you have to rely on some level of gut and instinct. I’ve been in product for a long time. To a certain extent, I can lean on my past experiences so I know how to synthesize that feedback and conflicting perspectives.
The philosophy that I like to take is that of Jeff Bezos and Amazon: disagree and commit. Not everybody will be on the same page, but it’s not my job to convince everybody that this is the right thing. Hopefully, most people are aligned, and that’s the goal, but you have to move fast. If you try to get everybody to be on the same page, it’ll never happen.
Ultimately, as head of product, I have to make the final decision and need to use my best judgment by using my instincts and experiences. And the more experience you have in different industries in different organizations, the better, but the core PM principles remain the same no matter where you work.
There’s always going to be this push and pull. It’s easy to get sucked into the loop of servicing your largest enterprise customers and forgetting about long-term strategic things. That’s where, as a product leader, you need to have an open dialogue with the people above you. As long as you have that, you can get to a consensus of long-term versus short-term and servicing your immediate customers. It’ll never be perfect, but if you don’t talk about it constantly and if you don’t raise the flag when needed, you’ll never be able to accomplish the big things you want to do strategically.
Ideally, the way you would set up is you have 15–20 percent of your team dedicated to long-term, strategic things that are like multi-year projects. Then, the “run the business” team works on immediate customer impact, especially the big ones that need something tomorrow. Having those things balanced is very critical. There may be quarters where you have to drop everything to take care of a hundred fires that are burning all at the same time. But the only way to get through it is to make sure everybody always knows what’s going on. The worst thing you can do is operate in isolation.
Internally, we’re developing tools and mechanisms using AI to more easily automate content generation and assessment creation. Also, people are using AI in their daily lives today, and they’re learning a lot through that. Our goal is to figure out how we can enhance that learning process and develop more accurate ways to measure and improve assessment takers’ skills. We want to better prepare them for life and what they want to accomplish academically and professionally, and AI is the key part of that. We’re viewing AI as a way to speed up and enhance how quickly people can learn these skills and become proficient.
We are using AI today by building machine learning models to optimize how we do things and are looking at it as an integral part of developing assessment products and scoring them. For example, AI can potentially provide feedback for users who are learning how to do better on our English test. There are many more enhancements we’re looking at using AI to improve the user experience and provide more value.
As I mentioned, we’re at an inflection point. Everything’s changing with the way universities operate, with AI, and with education. It’s a new space for me. That is exciting because I am learning a lot and trying to put a strategic vision in place for the transformation of what we do. From an English proficiency test perspective, I’m thinking about what the next version of this looks like. What space do we play in?
A lot of companies talk about changing lives and changing the world. We actually do. Through education, we’re making an impact on people’s lives. That part to me is very exciting.
AI is now at the core of everything, so I’m also thinking about how that is changing Gen Z and Gen Alpha. How will they work and how will they go through their lives? Those are all thoughts and things we have to solve for. The team I’m building will be forward-looking — what does TOEFL look like two years from now? What does ETS look like? It’s a very exciting time to be here because we’re going through this transformation.
The last thing I’ll say is it is a global initiative that we’re doing. The team that I’m hiring is worldwide — India, China, Europe, and the US. The impact we have is in nearly every country. We’re in 160 countries or so. We have a huge impact on a global basis. This is extremely exciting for any product manager to operate at this scale.
LogRocket identifies friction points in the user experience so you can make informed decisions about product and design changes that must happen to hit your goals.
With LogRocket, you can understand the scope of the issues affecting your product and prioritize the changes that need to be made. LogRocket simplifies workflows by allowing Engineering, Product, UX, and Design teams to work from the same data as you, eliminating any confusion about what needs to be done.
Get your teams on the same page — try LogRocket today.
Want to get sent new PM Leadership Spotlights when they come out?
Hypergrowth happens when a company experiences an exceptionally rapid rate of expansion, typically more than 40 percent annual growth.
Detractors have long-term effects like negative brand perception, reduced customer loyalty, and a decrease in sales.
To proactively address liability concerns, you can create an internal product recall team for dealing with risks and ensuring quality.
Our Galileo AI Fall ’24 release makes Highlights available to all Pro and Enterprise customers. We’re also debuting Ask Galileo, which enables you to directly ask questions about what you see in session replays.
Learn more →
