Ultimately, it’s very, very hard to assess someone’s product ability from a regular interview. Asking them to walk you through a scenario is a useful way to see their ability to do the job, and also how they communicate and persuade. It’s a useful time to put unexpected things to them to see how they react.
I recommend leaving the scenario until last as it’s quite a big ask for both you and them. I normally like to provide the candidate with the scenario ahead of time and make the hiring manager available to answer clarifying questions over email.
A scenario might be a small question “what would you change about our landing page?” or it might be big “what should we change about our product?”.
There are a few different ‘types’ of scenario that you can try.
Hypothetical for your company: A made up challenge that you could realistically face. This would let you understand what they know about your industry and the specific challenges you might face.
Real for your company: Talk through a challenge that you’re actively working on. This lets you see what they’d be like to brainstorm with. Matt Parish points out that if you don’t come from a product background yourself, these tasks are easiest to evaluate.
Hypothetical for another company: Give them a company or ask them to think of one. This can be useful as it is neutral (you know the same amount as them about it).
Totally hypothetical: You can test them on something completely abstract, to see how they handle challenges. I believe that Monzo used to ask designers to design a toaster from scratch to see how they approached problems.