How do you know if your product launch for a transformative product bet is failing or if the bet was wrong?

Behavioral metrics.

If you've changed the behavior of your customer base to to something else, you're succeeding. If not, and you haven't been able to retain or continue a habit-forming change, go back to the drawing board.

It's important to ensure you have these metrics lined up even into beta so you're measuring change, not just revenue our core output and you can explain WHY.  

Retention is the easiest thing to measure across this. But I like to pair metric + and activity. So 30-day sustained activity of X that is growing by Y.  

Yes it's more complicated to track, but then you can really diagnose what caused the changes better, and can drive better strategy for the future.