Transcript of the App Performance Café episode with Prasanna Vijayanathan , from Linkedin, originally published on June 4, 2020. More details in the podcast episode page.
Rui Costa:
Hi, everyone! Welcome to the second episode of the App Performance Cafe. Today I have the pleasure to bring in Prasanna from LinkedIn. He will start by sharing with us what he understands by mobile app performance, and then he will take us to this mind-blowing concept of personalized performance and network quality as a service - which is something that for me was a new concept: to make sure that every user gets the best experience possible, regardless of his conditions. Thank you, and I hope you enjoy it.
Episode Start
Rui Costa:
Hi everyone! Welcome to the App Performance Cafe. I'm Rui and I'm very excited to have here with me today Prasanna Vijayanathan from LinkedIn. Thank you Prasanna for accepting our invitation to be here today. Prasanna can you tell us a little bit more about yourself? What do you do at LinkedIn?
Prasanna Vijayanathan:
Sure, thanks Rui. So currently I work at LinkedIn as part of the Performance Engineering Team. I manage a team of engineers to focus on how the LinkedIn applications, both mobile and web apps perform to our members, our end users. And we also look at how our edge infrastructure performs and how members can connect to LinkedIn elsewhere. So that is the purview of our team. And that's what I do at LinkedIn.
Rui Costa:
Cool, exciting. I've been reading and actually viewing a bunch of talks from you and your team - so, very excited. So, I always like to start with this eye level question, a very simple one so that we can all be on the same page during the conversation: So from your perspective, why should we care about mobile app performance?
Prasanna Vijayanathan:
That's actually a very good question. We have thought a lot about this too. I think the first step is to understand what is mobile app performance, app performance in general. To me, anything that affects how a member or a user of your product, interacts with your product, anything that affects the way they interact - it falls under the realm of performance for your application. So the better experience your users are getting, the better is the performance of your product or reputation. And so if you want to make your users in LinkedIn - we use the term members - we want to make our members happy, then the application should perform better.
Rui Costa:
Yeah, cool. I read in one of your articles a mantra that you guys have, which is something along the lines of "Make LinkedIn feel instantaneous", something like that, right?
Prasanna Vijayanathan:
That is our core mantra for the Performance tTeam at LinkedIn.
Rui Costa:
Very interesting, so what are the key challenges that you see when following your path towards reaching this ... I would say utopia, but hopefully not?
Prasanna Vijayanathan:
Yes, it always feels like utopia when you keep making things better work, the goalpost keeps moving the other way. I think, let's think about what I said for a moment. If we want to make every member's experience on all of our applications as fast as possible and make them feel instantaneous: how do you figure out what is that experience like? For one, what performance or the experience of an application means to each user would be different. Some people might be happy looking at the spinning wheel for five seconds and wait for something to load. And some people may not even want to spend a second on it, if they don't see something they like, they would close the application and go - we don't know. So there are so many contributing factors to performance, and this has become even more challenging to understand and define, and then go on to measure and optimize. Mostly because of the diversity in these factors, think about the number of different types of mobile phones out there all over the world. Think about the thousands of internet service providers offering various network plans and speeds, and then the fidelity of network connections themselves. Like think about the global diversity and especially for products that are reaching out all the world these days pretty easily, given the place or availabilities. The global scale of the diversity is the main challenge for performance for applications to it.
Rui Costa:
Interesting. So you mentioned user expectations are a factor of diversity. But you also mentioned the network, either it's the provider or even the phone location, whatever: What kind of challenges do you see there? So for example, I've read in your in your article, like the rise of customized content delivery, things that you guys were testing on the edge, where you do optimizations in one point, and the outcome of the optimization varies significantly across users, across networks, across locations, I would say. So can you share a little bit more about that?
Prasanna Vijayanathan:
Yes. So the diversity that I was talking about, we've run various experiments in the past, in the traditional sense of performance as well. Identify a bottleneck, look at the data, optimize for the bottom network DC and roll it out to everyone, see if it helps a majority of your users and that's a performance win. But then because of said diversity in network noises, client- side characteristics, global scale - all of that - what we continue seeing is every time you make a general performance optimization, that would help some members, but there will be some members who actually see a degraded performance because of that. Well, don't see much improvement at all.
This is what I refer to as a selective penalty, in my earlier blog that you mentioned about, that by improving my rolling out optimizations, we might improve some members, but then we might degrade others. And this became a sort of a trend and a lot of optimizations a few years ago that we did.
And that's when I started thinking, how could we maybe not do it? Not do an optimization if it doesn't help that member. And that gave rise to the idea of customizing content delivery or what we call personalizing performance. Just like today on all the different applications out there the type of content that, I assume for any user on the internet, is tailor made to their preferences.Similarly, can we can tailor make content to fit within a certain performance threshold for each user every time they interact with any application? So that is the idea of personalizing performance.
Rui Costa:
Interesting. So what is that all about? Different users getting a different app, or a different app experience in the sense of, for example progressive content delivery, is it about providing a lighter experience for different users? Or are the, what really strikes me is number one: How do you know which user is the one that is fetching that content and to which experience you should adapt? And then how do you do that at adaptation? Is it about reach content versus lighter versions? Is it about optimizations at the edge that differ from say, region-to-region, device-to-device, where do you see that going?
Prasanna Vijayanathan:
I see it going into all of those directions. So, let me explain: Let's say, if you want to try and understand what is it about an application's performance that is important to said member, looking at time of interaction? There could be so many factors that can have a role in it, but at a high level, you can break it down, keeping your application servers constant. Your infrastructure that connects to your members, and then the application itself on the platform that they're running it on.
These two factors add up the major portion of the diversity that we see that affects performance. Of course, there could be other factors within a server, but it is more in the control of the application provider. So one of the things that we have thought about and we've built now is something called network quality as a service. And we've talked about this in some of our talks at some conferences. It's basically the idea that if we can identify how the network is performing for that member, at that time, and expose that information.
So any, any other feature on the application can use it, how they want. Similarly how the device is performing - maybe you're running low on battery, or your CPU's red-hot, or maybe you're running on a really old phone compared to the latest iPhones out there- so those diversities, if we expose that information, there is all the opportunities that you've talked about on the table now. And that is how we are approaching this challenge. If we can expose all that information in real time for each interaction, he can let on the feature engineers and application owners figure out how they can use it.
Rui Costa:
So it's like having a performance monitor on the ends of the user, right? It's having something that can assess all the conditions that may affect his experience. Interesting. How do you adapt from that? So first, before that - How do you actually measure that?
I say that for example, I'm a network engineer, so I always think about by default on networks and on wireless networks in particular. So I'm thinking they are so unstable from the point of view of the time it takes to build something: how can we be able to get the good assessment live from the application perspective? So, that then the rest of the application adjusts in real time, because I guess that's what you're aiming, adjusting real time to this fluctuation of experience.
Prasanna Vijayanathan:
Yup. That's actually a great question and we are still learning. So it totally depends on what each application is built for. So when you build an application, what does that you want your product or application to provide value to your users or members? And that would be your true north metric. And all other performance metrics that you define and monitor and measure, should add up to improving that true north metric, whatever you define for your product. And that is the way we are also looking at it, for our true North metric for giving better experience to our members. How do all these optimizations that we are doing in real time, adding up to improving that experience for our members. And that, is how we are looking at this.
Rui Costa:
Wow. It also has to take into account what the difference between different goals in the given page of the application. So for example, do you see pages where for the user it's more critical to get lighter content, say text for example, in comparison with other pages where loading richer content is actually critical from the end user perspective, even if it takes longer to load. So I'm trying to think about the trade off between rich content and time - and user frustration. So how does the user react to that?
Prasanna Vijayanathan:
In fact, that is one of the first things that we tried. And we've talked about this in our other conferences, too. Using this information, let's say the first thing we tried was using this network quality as a service. Can we adapt the resolution of images that show up on your mobile web application? So if the network quality currently is not that great - let's say you're connected on 2G or 3G networks - then maybe show a lower resolution image, free up some bandwidth, so the application actually loads and shows you something meaningful. And then let's say you transition to a faster wifi connection, you're just walking along and now the network suddenly looks better. So get that information out, so the next set of images that you see when you connect to wifi is actually going to be a better resolution. And so it gives you a better experience and you're interacting with the application - such optimizations. And we've seen that it actually helps improve the experience of our members on the application. This can be true for images, videos, or any other content that's on your page. Even thinking forward, looking forward, what kind of application even renders on what part of the application shows up for you and so on - depending on what that member is looking for at that time and what would fit well for that given interaction.
Rui Costa:
And in the end, let me take you into a network challenge. That's something that I'm, as you know, like I'm really into. So we have this, say performance monitoring at the end user, so that you can adapt the experience of the end user. But still, what can we do?
Well, there are many perspectives to this, but I'm thinking specifically from a networking or handling the networking challenges perspective, what can we do to optimize and ensure that irrespective of what are the network conditions ,the user ends up getting the best experience possible. How can you ensure that I'm thinking from an application perspective, from an infrastructure perspective - what kind of optimizations can we bring to the table or strategies to adapt?
Because as you mentioned, it fluctuates so much that you cannot rely on a very specific optimization for a very specific use case because then comes, I don't know, noontime and everyone is on the, on the same link and there it goes the benefit. What kind of optimizations do you have in mind and strategy for this?
Prasanna Vijayanathan:
So some ideas that I have, and this is pretty general to any application developer, a lot of companies have adopted this strategy of having the lighter version of their app that shows up. So that is one option of just build lightweight application that can load very quickly and you have the minimum product for members to start interacting with- that is one option.
Then options of that, if there is enough space on the device, but the network is almost like offline: Can you give like an offline experience? That is another idea that can be explored. Especially, depending on the rapid changes in network qualities, like I mentioned earlier, like toggling between the 2G networks, 4G to wifi and so on. You can start customizing the content like the images I spoke about, the ton of videos that are out there on applications today. You can set initial bit rates, that is a challenge today - even though there are adaptive video players that adapt to network quality, but what would the initial start bitrate be? And things like that. How do you buffer queue content, like you download images and keep them ready, but is there enough space on your cache to do that - all that, all that comes into picture. So it's not, so network is definitely a key factor. But then there are also other factors, especially when you're talking about mobile performance. Just in terms of networks we've had in the past, we experimented with what type of PCP condition control can we provide from the server. Can that be different to different end points and so on. Those are all optimizations that are possible. But going beyond just the transport and going to the application, I think all of these are potential opportunities.
Rui Costa:
Yeah, we could go on around transport and protocols for ages, but the goal, or otherwise people would just fall asleep or get too excited.
Prasanna Vijayanathan:
I mean the opportunities there are limited to how much are we transferring over the wire, right? Can we almost go down to zero in the wire and just generate everything on the client? Is that a possibility? I would think so, yes. But again, trade off between home much can you put on the client and the device? How much can you put on the server and how much can go on the wire and the network? I think this is a balance that keeps happening through decades of technology.
Rui Costa:
Yeah, and that's why when we had the previous call, when you were talking about this network quality as a service, I got like. It clicked because it's exactly what we should do. From optimizing the experience because even if you have perfect infrastructure, optimize development, everything is light, efficient: then you have network, all devices, battery - you name it - which you don't control. So you have to have something and ideally something that as you say, customized and personalized to the end user. And I guess that can only be done if you're sitting virtually by the side of every user and trying to assess the experience and adapt. And so this idea of "end user monitoring", I don't know how to call that. Trying to sum it up and trying to bring a different angle: I've read in your article that you are seeing interesting things when you were optimizing the speed of the application, performance of the application. You were seeing that, number one, there is an impact in terms of user engagement, but you had some interesting comments on the level of impact it takes when you compare users in good networks to users in bad networks, well bad, slower networks or devices you also have, so that's one. The other thing that struck me is that you found an anomaly in the data, which then you say, and I really believe you're right. That, so the anomaly was that there was a bunch of users for which the lighter version of the application was not improving engagement. And your comment is that this is probably due to the reason that this set of users actually prefers to wait to get the richer content, something like that. I don't know the exact words, but it was something along these lines. Can you comment a little bit on that?
Prasanna Vijayanathan:
Sure. So, the first part of your reference was more on the lines of this selective penalty that I spoke about. Where some users can benefit and others don't when you roll out a general optimization. And then the second portion is where we found that there is a line between giving more and more feature rich content to users, versus giving it fast - give less features, less rich content - but deliver it fast. And I think the answers that you are talking about in that talk, I was referring to, we actually hit that line. Where we found that there are users who really enjoy the fast experience, even though the content was light, but there were members who didn't like the fast experience they preferred the more feature-rich content. That is pretty intuitive, and it was actually amazing that in our analysis we ran into that line between performance and features.
Rui Costa:
Yeah. So I will put your LinkedIn profile in the description of the episode. Probably the link for the article that we're talking about as well, so that people can take a deeper look, the data is so interesting. It's so - it's simple and at the same time mind-blowing - at least from my perspective, it was like a little bit mind blowing.
Alright, cool. So I asked the same thing to Nolan last time. So to end up, what would you say should be the key takeaways from your experience working on performance that every performance engineer should consider?
Prasanna Vijayanathan:
Interesting. Let me take a moment to think about this. The world of performance, there are aspects that you can control and there are aspects and a lot of things that are outside your control. For the pieces that you do control, that's where we go in and identify bottlenecks, optimize, and make it better for the pieces that you don't control.
My recommendation is we try to understand what they are, and why they are the way they are and adapt to it. So both of these approaches to performance are critical, especially the latter as we have a growing in terms of diversity. So make sure you look at more times of performance, the piece that you can control and the pieces that you can't.
Rui Costa
Wow. Fantastic. That's a very catchy phrase and it really sums it up, like a control what you can and what you can control- please adapt. That's perfect. Prassana, thank you so much. We were able to keep it short, hopefully the audience is happy that. Thank you guys for listening, hope you have enjoyed it, and see you next week.
Prasanna Vijayanathan:
Alright, thanks a lot for having me.
Rui Costa
I hope you have enjoyed the conversation. I will leave Prassana's LinkedIn address in the description of the episode, as well as the link for the post that we were discussing throughout our conversation. Please follow us on the usual podcast platforms like Spotify and Apple podcast, and visit performancecafe.codavel.com
See you next week.