Product execution mock interview: measuring success of a bank app (w/ FinTech PM)

hello everyone kentucky vessel from the rocketblocks team here in this mock interview video we're sitting down with ava morgenstein who is a product manager at jpmorgan focused on payments products and in this particular mock interview video we're going to stay close to her area of expertise and we're going to be walking through a product analytics and metrics oriented question asking her to think through the success metrics that she would use to measure whether a particular consumer banking app is succeeding or not so let's go ahead and jump in okay so imagine you are a product manager at a bank it could be a existing bank that's been established for a while or one of those sort of new hot fintech companies but basically this is a consumer-facing bank and they have a mobile app that they've put out and i want you to help us sort of think through like if you were the pm owning that mobile app what sort of success metrics would you set out and use to determine whether this mobile app is really you know successful in achieving its goals sure do you mind if i ask a clarifying question here yeah sure are we curious about the success of the application itself on your iphone or are we curious about the success of the the bank or the fintech overall that's a really good question i think right now the way to think about it is let's constrain the problem to like the success of the mobile app itself and not the success of the overall bank in meeting its objectives gotcha that makes sense um so i think i would divide this this question into two different components i think on one hand given that this is a for-profit company or startup what we would want to be looking at is revenue and understanding how much revenue is being driven from the app itself okay so the first component and i'll go more into that in a second and the second component is really user adoption and growth that's really critical for understanding do people use it is it important is it our usage growing etc so i'll go into both of those in a little bit more detail okay so i imagine that just given the the age that we live in much of the banking that this bank does is done via its mobile app um so because of that the the revenue aspect is pretty critical for the application here um and i'm going to use chime as an example because that's the one i'm familiar with but chime is essentially a relatively new bank founded in 2013 and they allow for checking and savings accounts free of charge for all of their customers from a revenue standpoint the way that they actually generate revenue is by charging part of the merchant fee on payments that their clients make on chime debit cards and then they also have a yield on some of their high yield savings accounts which they take a portion of so given those revenue objectives and kind of revisiting our revenue component of what our metrics are what we would want to understand is on this mobile app how many transact excuse me how many debit payments are made and i'm assuming that chime can make payments through through apple pay or through zell or through venmo etc and what notional or excuse me what percentage is actually charged for the merchant and there can be a variety of merchants here depending on where a customer is actually making a purchase so we would want to know what rate all those merchants are charging and what percentage of those rates chime actually gets to take home and that would be a really key metric for understanding revenue growth from the perspective of the debit card side of revenue so that's that's one or two metrics for that topic got it and just to make sure i understand that because there was some level of detail you went into you're saying okay so you have a chime mobile app that you use and and along with that mobile app it sounds like you're presuming you have a debit card that you get with it and then you're saying well there's you know payments that the end user might make and maybe they use zell or uh apple pay and so the way that connects into revenues you're saying the that end user is not is it that they are initiating those apps or sorry those payments from within the chime app or that they are connecting their chime debit card to those other payment services the latter was essentially them connecting their chime debit card to all the payment services that they would be using through that mobile app and of course if you wanted to look at overall company revenue you would go into all debit card usage but for your clarifying point this would be specifically for the mobile app and specifically for other mobile applications on the user's device got it okay and that would then be turn into the metrics just to rehash them of notional transacted and then the merchant rates and the percentage of the merchant rates that chime can earn got it okay sounds good so moving on to that that really covers revenue for us for the debit card side only there is also revenue from a savings account perspective but because this is intractable from the overall business of chime i'll skip over this one in terms of the mobile app okay sounds good so moving on to the user metrics i think in order to understand the success of the mobile app what we would need would be a few different things we need to get an understanding of accounts and we need to get an understanding of actual app usage and how that influences how many accounts are opened and specifically how much money is saved in the high-yield savings accounts um which actually i'm going to unwind to that that's not super mutually exclusive um so for our category of user growth and user adoption let's just keep it with the number of accounts and then the number of transactions and clients that we have okay got it so accounts and number of transactions yeah so just to start with kind of the simple one i think we want to know let's start at the client level so we have a total number of users or clients that bank with chime overall we'll want to know what that number is so we can kind of compare it as a notional metric for how many actually on the mobile app and then we'll know how many customers are on the mobile app um and that's just an aggregate customer number so we have a good sense of of where we are overall as a bank and then what percentage of adoption our customers have for our mobile platform yeah the next aspect would certainly be the number of accounts that those users have open and that all of these are important together is because you could have many customers and many more accounts because people open accounts and then leave the bank and so for these accounts essentially what we would want to understand is how many total accounts are there and how many total accounts are actually opened on the mobile app so how feasible is it to open an account on the mobile app can kind of be understood from the perspective of the proportion of mobile app opened accounts to total accounts got it and that covers our account perspective i'll take a pause there again and so tell me a little bit about you know thinking about chime as a business and let's let's say mobile is important for them they believe that most of banking in the future will be done via like an interface like a mobile app that's what consumers want et cetera what type of numbers would you like to see for things like percentage of our users using the mobile app and percent of uh you know accounts created via mobile app versus um you know the desktop website or something like that i think you would hope to see between 90 and 100 for a successful banking application i think no-brainer there it's very clear that the banking industry is moving towards a lot of mobile activity as opposed to desktop activity and in-person activity and if you're under that 90 metric then that's an indicator that your your interface on your app isn't accessible enough or isn't widely known about enough to actually be able to have most of your customers leveraging it got it okay and then how about the was that number for both the number of people using the mobile app and the number of accounts created via mobileverse desktop was 90 plus for both yes absolutely okay yeah okay and i think that that does depend on the account opening software that's being used um because of course that process can be quite complex and i'm saying that assuming that it's possible to open an account on the mobile app which i realize it's not for every single fintech or financial institution got it okay okay cool how about on the transaction side sure so i think a really clear indicator of how often uh excuse me of how successful a banking application is is how many transactions are actually executed on it and i know we talked about the notional value of these transactions on the revenue side but for the actual user side and the adoption side what we're most curious about is the number of transactions per account the number of transactions per day and the number of transactions per customer per unit of time and this will help us gain an understanding of how people actively use this app and we can compare it hypothetically to some of our competitors maybe some established banks and maybe some non-banking institutions like venmo that also allows people to transact got it okay and tell me like take me a step deeper on some of those metrics so you talked about transactions per day transactions per i think was transactions per account per customer and then you know transactions um maybe maybe it was transaction transactions over some other unit of time um i may have missed the third one or maybe maybe it was just day versus some other unit of time but like break down a little bit for me what are the different types of transactions you would expect and is there any reason to um look or measure any of those specific transactions more granular granularly or does it just matter to like look at say transactions gotcha so it sounds like kind of the first level of granularity that we could approach this question with would be payments versus receipts like how many transactions are being sent out from the chime account versus being received to the time account and the the complexity that that adds and the reason that it would be useful to track that discrepancy is because if people are only paying from their chime accounts and never receiving funds to them that really limits our functionality as a bank and would also indicate that there is some aspect of paying to a chime account that becomes difficult or there's some aspect of the user question that's being answered by the chime account that we don't yet understand why are time users only paying etc so i think having an understanding of the balance of that and making sure it was in keeping with expected amounts given the application and what we expect from the rest of the industry would be pretty important there got it and help me just because i'm less familiar personally with the particular like fintech lingo but when you say receipts this is money coming into an account versus money flowing out of a chime account is that the way to think about it exactly or you wouldn't even necessarily want it to be a china account because it could be china users sending to each other but from the perspective of the account owner a payment is when they are sending money out of their account and a receipt is when they're receiving money into their account got it and how should one think about like if i'm using the chime app and say uh doing mobile check deposit is that something that's considered like where does that get does that get captured in the transactions the way you're thinking of them currently i would think that should be captured in the transactions because that's part of the notional that's flowing into the bank and that's certainly something that we want to be tracking got it okay so that's a receipt into a chime customers account that's right got it okay and then how do you think about like you know i imagine you know typical mobile banking apps there's like a few things you know typically there's minimal amount of things let's call the table stakes number of things you can do like mobile check deposit um usually sending sending payments transferring payments um what other things like might be possible or do there any other things that we would want to measure when we're thinking about user adoption of the mobile app or is it just actual sort of transactions where money is moving yeah i think what you're bringing up is really important and i think what it actually reflects is beyond the functionality of the application and the the functionality i'm just going to assume is primarily to send transactions based on my understanding from of chimes business from an outsider's perspective but i think what you're bringing up is that there's a huge amount of user experience that's intrinsic and understanding why clients will adopt our product or not adopt our product and that's actually a total deal breaker for us and i think this is like the last category of the user adoption and user growth so i'm glad you brought it up but essentially we want to be understanding what the impact to our clients is of having that front-end user interface and how much they enjoy using it or how much they want to use it again and i think a few metrics that we could use to track these could be broken up into the perspective of time and how we're measuring time and then also actual user feedback um i think it's pretty difficult to tell how people find a user interface sometimes without at least one if not both of the metrics in those two categories so for the time category what we would be interested in would be kind of the soas or the time that it takes after clicking on something to go to the next page um that's very important and then also the time that a user spends on the application um and i think you'd have to do some research to understand what a high amount of time indicated versus a low amount of time but i would imagine that there's a sweet spot amount of time which indicates that a client can log on can do their business and get off in a short period of time that indicates they were able to accomplish what they wanted to accomplish but not that it was so difficult it took them too long so that's kind of what i mean by the time metrics and those are things that you can actually build your application to track if you so choose yeah the other metrics are actually like do people like using the product and to be able to attain this i think what you would do would be talk to your users and understand exactly what they want to use the application for and exactly how they plan to interact with it and after the product is live which of course it is you would want to understand what are their pain points what's annoying to them what do they have to go around that they don't want to go around and to be able to to get to these answers i think you would really need to have a significant level of design strategy involved in conversing with your customer population and understanding what improvements you need to make the application better got it and when you say design strategy what can you unpack that a little bit what do you mean when you say design strategy when talking to customers sure we're essentially saying how can we ask the customer what they need from us in a way that is asking them a question that they can answer even if they're not consciously aware of it yet got it okay okay got it so it's like sort of methodology of teasing the information out of customers or potential customers it's that and i would also say it's really saying our product is designed for the end user this is the target audience that we need to meet everything for and so we want to make sure that no matter what the functionality that we as the product developers believe should exist in it that really what it does is it serves the end user need and in doing so allows the products to be adopted and used and earn the company revenue got it okay and tell me a little bit about i thought your point on time was interesting and the idea there being this sort of sweet spot of time time spent in app for example and so it sounds like you're saying look like if if it's just like 20 seconds on average maybe that's not good because maybe people are getting confused and leaving and if it's on the other hand if it's 20 minutes you know maybe that's not good either because like why is it taking so long how you know if you could sort of wave your wand and magically get some sort of metric around this like how would you can you think of any sort of ways where you you figure out what or how to sort of quantify what that sweet spot looks like yeah absolutely i think you'd want to start with getting an understanding of what people log on to the app to do okay and if you can get an understanding of what that is and also of kind of how often they do it then you'll have a good understanding of how long you might expect them to be on the application for and then yeah so i think that's kind of like just a starting point saying i'm logging on to check my debit account balance and then i'm leaving is a lot different than saying i'm logging on to check my debit account balance then i'm sending 10 of my friends money and then i'm doing my credit card bill um that's a much different time perspective so i think to create one metric would be pretty tricky especially for using the application for different things depending on your user population but i think what's most important is understanding what people are going on for and in an ideal world how long that activity should take a normal user to be able to accomplish and i think there are several different ways that you could go about doing that which i'm happy to go into further but i really think that it is specific to the activity that's being accomplished and the intention that the user is opening the application with got it okay um i think that makes sense i completely understand like different use cases will have sort of different expected um sort of time commitments to to take care of even if even if everything is running smoothly um cool sounds good so we we talked about a lot of different metrics you sort of classified things into revenue uh and user adoption metrics which i think sort of makes sense one sort of last tactical question on like user adoption one thing i don't think we spent much time talking about is like you know should we track like the number of times people log in a day like the number of just pure logins matter or um we did talk about time spent on out but like tell me about like or logins is that type of sort of general metric that's often used in like the consumer world like you know tracking daily active users for someone like snapchat or facebook or whoever does does that metric make sense here should we care about logins should we track that i i think it would be an interesting piece of information but i do also think that what you would learn from counting logins is also attained by some of the other information that we discussed so by looking at logins we're seeing how actively are people actually using the application but this is a banking application and frequency of use is not necessarily an indicator that the application is performing well right like you don't want to log on to your jpmorgan or chase account like 20 times in a day that sounds terrible um no one's interested in that and so what i think that we're tracking with logins is what we're trying to track is activity right and we've tracked activity pretty well through transactions and we can further categorize transactions by transactions that occur kind of in the app or in the app through other apps and then transactions that occur with a material debit card and that will also provide you with some more understanding of the frequency that being said frequency is important to track above a certain time range if someone's not logging in once a week that's a pretty good indicator that they're not really actively using our banking app anymore and that's so i would say that it really depends on the amount or the time unit that you're interested in and that as long as someone is logging into that app once or twice a week you're probably good on that front and then the other granularity you would get from the logins is really better covered by some of the other metrics we discussed ah okay that that's an interesting distinction so essentially you're saying you know transactions um and measuring the metrics we've set up to measure transactions are gonna capture whether we have like a healthy rate of activity happening on the app including the fact that you need to log in to do those transactions but maybe logins is useful for predicting or thinking about which percentage of the the population is maybe likely to turn or something like that because of what you're saying like if they're not you know like something like percent of users who haven't logged in in two weeks or something that may actually be an interesting metric and an actionable metric because maybe there's some sort of product changes or you know enhancements you could do targeted towards that group exactly got it okay sounds good um last question is so again talked about these you know a lot of different metrics revenue and user adoption were the two big buckets that you used and sort of used to create an organization around these um if you had to prioritize a few like think about you know the metrics where when you walk in in the morning you're going to glance at and make sure nothing seems like it's gone crazy wrong of all the type of stuff we talked about would you prior which ones would you prioritize and why yeah that's a great question um i think from and we're still talking from the applications perspective right yep gotcha i think that i would definitely want to know the number of transactions that were executed in the app versus executed physically and and that's just like the most important off the bat for me because the trend will tell me what the actual adoption is of using the application and also the adoption of it relative to actual physical methods of payment or receipt so that would be my number one far and away um and then the big like if something's gone that's like up 20 or down 20 it'd be like five alarm fire well maybe not if it's up 20 although that would probably be interesting as well right and i think kind of in keeping with that i'd be interested in the notional that's actually being transacted on the app once again because that is related to our revenue and also because it's a similarly indicative metric of how much the application is being used for its primary purpose which is payments okay sounds good any last uh any last bits you want to add yeah absolutely i think kind of overall from a product perspective while you do want to track the revenue that this application is producing for your company the user experience is what you shouldn't ever take your eye off of because that will be the path to all of your future customer satisfaction and actually delivering value um so while as a product manager you might be prioritizing the revenue so you can understand it that should never be um kind of instead of the user experience because they go hand in hand and you're there primarily to deliver a great product got it sounds good thank you

Channel: rocketblocks

Views: 15,873

Keywords:

Id: KW3YvcJw7Bo

Length: 23min 27sec (1407 seconds)

Published: Tue May 25 2021