AMA: GitLab VP, Product, Mike Flouton on AI Product Management

I think this is hard to say, but if I had to guess I think this does evolve into a specialized function. At Barracuda we had a platform PM function and I was lucky enough to work with a phenomenal PM who owned threat detection efficacy. He interfaced with our data scientists and ML engineers on a daily basis, and had to be very comfortable with concepts like precision and recall, model retraining, model ops, etc. That said, the LLM cloud providers have made it so easy to consume third party models via API there's a level of abstraction that makes specialized important, but less so than if you have an in house ML engineering team. So I'd imagine there's plenty opportunity for specialized PMs and generalists to play in this field.

First and foremost, there are certain traits I value in any PM and I'd say these are the basic non-negotiables. * Intelligence. You can't coach height in basketball and you can't coach raw smarts in PM * Curiosity. Great PMs ask a lot of questions. * Strong customer focus. You love talking to customers and learning. * No jerks. I want humble, low ego people who enjoy finding out they're wrong because it means they learned something new. For an AI hire, I touched on this in a few other questions but I want someone who has done the work and understands AI. People who know the difference between supervised and unsupervised learning, what a random forest is, how to measure the quality of a model, what LLMs are good for (and what they aren't), etc. Information is so democratized today it's easier than ever to learn this stuff before expecting to show up and add value as an AI PM learning on the job.

Let me preface this by defining a product team as PM, UX and Engineering. I'd suggest there are at least two sets of metrics you should be looking at. First and foremost, don't forget you're here to solve a customer problem. Judge success according in how the capability is driving that specific outcome just like you would any other product. That could be the time it takes a customer to do a task, number of phishing attacks detected, sales volume of your sellers on a marketplace or rides taken by customers. To supplement those outcome metrics, you want to look at some supporting metrics as well. These may be specific to AI. Classic examples are * Precision and recall * F-score * False positives and false negatives * True positives and true negatives There's a slew of others that I'm not going to list or explain, but if you're an AI PM you need to know them to be able to talk to your ML engineers. I'd recommend taking a course or working your way through a book. Don't be afraid to get your hands dirty and write some python in a Jupyter notebook!

I've often said that PM is misunderstood as a relatively junior and technical job. It's actually best when it's treated as a strategic function, and being technical is a bonus but not necessarily the be all end all. I do think AI PM might be a bit of an exception to the last piece. You need to understand the enabling technology a bit better as an AI PM than as a PM in other domains. So I'd say you need to have the desire an ability to get into the weeds a bit. To be clear, the intent here is in no way to trample on the authority of you engineering counterparts. Always let them own the "how." But being fluent in AI will let you have much more meaningful technical tradeoff conversations. As for where to learn, I learn best by doing. I worked my way through a book on ML models in python several years ago and had a blast and got a better sense of the art of the possible. If you're the type that learns better in structured learning there are great courses online or at local universities.

For sure. There are certain things AI does really well, and others that humans or traditional deterministic algorithms do better. That sounds obvious, but there's some nuance here that users often miss. At Barracuda, we were the first major email security vendor to market with an AI based approach to stopping phishing and impersonation attacks. We invested a tremendous amount of time training the models to spot attacks that traditional solutions missed. We didn't train those models to catch run of the mill spam (the "easy" stuff) because there are mature solutions that catch those attacks, including others in our portfolio. What we didn't anticipate is that missing the "easy" stuff that we didn't train the models to detect undermined confidence in our ability to catch the stuff we were actually trying to detect. Users assumed that if we trained the models to catch one type of attack, it should automatically catch spam. What customers didn't appreciate is that AI doesn't work that way. It just underscores the need to be vigilant and engage with customers as often as possible - their perception is reality and you'll need to adjust to it, not the other way around.

This will be anti-climactic, but my advice is "don't forget you're a PM." Stay laser focused on the customer, their pain and problems and solve the problem. AI is an incredibly powerful tool to solve customer problems, but that's all it is. A tool to solve problems. And it's not always the best tool. We have a beautiful new hammer, don't go around looking for unnatural nails. Work from problem to solution to tech (AI when appropriate), don't start with the tech and work back to a problem.