Accelerate
Troubleshoot
Observe
Android builds can be complex and challenging to optimize, yet Android teams usually have the fewest resources to work on the build. So how did Salesforce get their Android build from 20-minutes down to less than 4-minutes? In this webcast, we discuss top Android build issues and lessons learned with Jason Schroeder of the Salesforce Android team along Rooz Mohazzabi from Gradle, including:
- Build modularization to make the most of your processing power
- Using the build cache to accelerate developer and CI builds
- Using build scans to optimize and debug collaboratively
Top Android Build Issues
Thanks to everyone that attended the webinar and thanks for your questions!
Rooz: Thank you all for joining, I’m Rooz. I work here at Gradle on the customer success team, and have been involved with Android for many years, and [I am] pleased to have our special guest here, Jason, from Salesforce. Jason why don’t you introduce yourself and tell us a bit about your role and what you’ve been working on and I guess what brought you to us and how we met and all that-
Jason: Cool, thanks for having me today Rooz, my name is Jason and I work at Salesforce. I’ve been at Salesforce for 14 years now. When I started at Salesforce I was an entry-level developer working on Blackberry and later iPhone. I switched into management for several years and led up to 45 people and about 6 months ago I switched back to an individual contributor role where, now, I am an architect working on the mobile build engineering team for Salesforce. So, I and my team specialize in the building the vertical for IOS and Android for all of our products. What brought me to Gradle was, I think for way too long our builds have languished and have not had proper maintenance and investment. And I think we’ve changed that six months ago when the builds were getting out of hand. We brought in a bunch of licenses for Develocity which helped us to analyze the hotspots and make measurable improvements in our build times, which led to greater productivity for developers because they are not taking as much time to build their software.
Rooz: Awesome, and tell me about how you measure productivity and why you care about it in general; or, is it someone from above that says “hey, we really need to focus on development.” Where does that all happen, because that’s why we’re here today, right-?
Jason: I think it’s both. Last fall, I wrote a pitch for our president of technology and said we should form a team that centralizes the knowledge for excellence in build engineering. Specifically, for mobile; for many years it has not been a priority, we’ve duck taped and bubble gummed it together to get a build out to store and I think our breaking point were builds over an hour.
“And it was not ok anymore, and now we’re really putting in the calories to bring those build times down and today we’re gonna talk about specifically how we got our biggest Android app, the Salesforce app for Android down from 20 minutes down to more like 4 or 5 minutes.”
Rooz: Other than build times, what else means developer productivity to you? Some folks look at developer happiness, some folks look at Linkedin, some folks look at how they ship features faster and how they commit to publishing; so, what is success for you-?
Jason: So, we were getting critical feedback in employee surveys saying our build infrastructure was lacking, so that was kind of the first metric we were trying to fix. So, it is employee survey results – are they happy with our toolchain? How long does it take to get a feature to customers’ hands? And that’s where we’re going next. One of my measures on this year’s goals is time to get a build to the app store GooglePlay or iTunes. From a commit to release candidate, there may be some additional testing that’s not part of my goals, but how do we get the CI/CD pipeline as tight as possible? We looked at measuring success versus failures on CI for our pull request evaluation and we thought that would be a good goal. We want the pull requests to be very successful, but we really pivoted on that because the point of CI and the point of the computer evaluation of your proposed change is to find feedback. And if its catching things and finding test failures, or lint things, or other complaints then the CI is doing its job and we want it to find things. We don’t want it to be more red than green, but we want it to catch things if it’s never catching anything maybe it’s not testing the right pieces
Rooz: Alright, so topics: were going to talk about some of the top Android build issues, we’re going to talk about modularization, we’re going talk about build-cache of course accelerating local developer cache, and CI builds, we’re going to talk about build scans and tools that Jason and team have used, and tools that we have built to help you optimize builds and debug, accelerate debugging and find build related liability issues and then well do Q and A at the end
Jason: Let’s talk about a typical Android project and I know everyone builds things differently, but this might look familiar to you. You’ve probably got an app at the center of all these things and you’ve probably got some java, maybe some Kotlin?. Maybe you probably depend on some Android libraries, maybe some that you build or maybe some that come from Google or others. And hopefully, you have tests, whether those be Espresso tests or Unit tests you may be working with fast lane to help you manage your app and interacting with Google play store. And you may have code generators or other plugin type things to help your app build. So, you’ve kinda got a lot going on here. There might be many projects in your workspace and how do you get all those things to build efficiently?. That’s what we’re going talk about today so tell us a little bit about the build cache why it’s important
Rooz: In 2017 we released the build cache, the local build cache, and the remote build cache. The shared build caches. And for us in terms of build time, before Develocity we were doing was a lot of consulting and training and support and “make my builds faster” was one of the top things. And we get it it’s still one of the lowest hanging fruits on developer productivity.
What are you going to do with a build that is under three minutes, you really can’t do much, right, you can’t get coffee, ya know, it’s this short amount of time. But those are some of the most painful builds and actually, Google’s engineering productivity team did talk about this in a presentation that I saw at Scale conference that said builds under five minutes are really the most destructive to the workflow. And they could see that developers would actually stop working. So the first year was all about getting the build cache going. In year 2 and year 3 what we started seeing from our users was, ok well the cache is now a commodity-how can I make this cache as effective as possible. Hans our founder often talks about your maximum build performance and how to reach that. And to reach that you need to monitor cache hit percentages. You need tools to debug cache misses so for us it’s been a lot about how we build better tools to optimize the cache. And that’s a lot of what we see today versus two years ago. How do I get to 80 to 90 percent cache hits? or how do I get to the most cache hits that are possible?
Jason: Yeah, so the build cache really helped Salesforce. That was the biggest thing we did to bring down our build times because we’ve got a lot of projects, like 30-40 projects in our workspace. And some of those get updated a lot and some of them don’t. And there might be one project that hasn’t been updated in a year. Why should CI spend time building that project that never changes? Is there something we can do to reuse something that’s already been built?. And we got that pretty much out of the box with the build cache. So, the build cache excels at computation-intensive tasks. For example, if you need a bunch of prime numbers in your app. For example, calculating those prime numbers is a computation-intensive task. You would not want to do that 100 times a day on your CI system. But if the cache has got it in the cache we can quickly pull out the precompiled bit and run with it so you don’t have to spend CP cycles doing that. Something you might not wanna cache is something that maybe changes over time. So maybe if you’re doing something with a stock price or the weather you don’t want to cache that because maybe if you’re using yesterday’s weather that might not be good for your application. And like you were saying if you want to help Gradle, put the right things in the cache. So, things that you think are going to get reused should be in the cache. Maybe throw-away branches should not be in the cache because they might evict something that might be more important and that will bring your cache hit rate down. So, you can use Develocity to study how well your cache is working. So let’s look at some cacheable Android tasks. These are a bunch of tasks that I saw in a build scan that were cacheable and some of those looked quite resource intensive. You know merge, debug resources compile java these are things that take inputs make outputs and if you don’t have to spend CPU time doing that it makes your build faster. So here is a quick slide on how to turn it on there’s a much more detailed guide at the bottom of the page. So basically, you want the build cache for your developers. You don’t want a local build cache for your CI machine because maybe that’s going to get evicted. You don’t want to spend time caching stuff to the CI machine’s disk because it may be doing something else in a few minutes and you may not get to reuse. But you do want to use the remote build cache for both your engineers and your CI. That’s kind of like your hub for your cache that persistent cache. If you have an ephemeral build machine that disappears you lose all that work its done. You can save that for its successor to the remote build cache.
Rooz: I’m gonna throw you an off-script question here
Jason: Lay it on me
Rooz: You know they were coming when you started looking at and using Develocity to see how well it is being used, were you seeing folks not using the build cache or folks that didn’t know the build cache was on, were you getting much of that?
Jason: So when we started, even the local build cache was turned off. The project was using old versions of Gradle and old versions of plugins. I think we were using a version of Gradle that was so old that parallelization was a beta feature, and you had to opt-in so that was turned off. So, turning on a lot of these flags gave us kind of out of the box improvement. Build-cache helped a lot so we didn’t really tell developers they had to do it. We checked it into CI and they kind of got it with everybody else and with the build scans we could start to measure and see -look it getting better for people, not just the Jenkins machines but also for the developers
Rooz: Because you have a lot of Android developers right?
Jason: We do
Rooz: How do you know who is doing what and how is everyone using the builds in the most optimal way?
Jason: They’re all over the place and we’ve got developers all over the continental US. You want your build-cache kind of near your people. If you have to go halfway around the world for the build cache it may not be as effective as something that’s down the street from you or something you’ve got a high bandwidth link to. We run the build cache inside Salesforce’s network so we didn’t need to turn on authentication because only inside our VPN can access it so there were no additional passwords to set up or anything
Rooz: and the local build cache is on once you’re on?
Jason: it’s on your disk
Rooz: the latest version of Gradle it automatically turns on
Jason: So it worked great, so how do you know if it’s on or not, so this is an example of a build scan from Develocity, and if you go to the performance on the left you should see a build cache tab if you don’t see this tab at all build cache is probably off but here’s a little bit how to read this first off at the top of the page you’ve got tasks whose outputs were requested from cache
so, we have 469 tasks 95% of them were cache hit from the remote build cache which is a pretty good hit rate
Rooz: yeah I would say so
Jason: I would say don’t spend time getting it higher than that the local cache was disabled because this was done on a Jenkins job the push was enabled we have Jenkins push builds because those are the cleanest environments we have you may not know what is on miscellaneous developer workstations but what is on our build machines is highly sanitized and trusted so we only do pushes from there. And then how to read some of these numbers down here so we hit the cache 447 times we downloaded 84 megabytes of cached products at 2.3 megabytes per seconds and we spent 37 serialized seconds doing, this this is a parallelized thing but if you stretched them all out we would spend 37 seconds on this we uploaded 22 products to the cache at 6.2 megabytes per second and down here it shows our pack and unpack times basically compressing and deflating things up this is like the mechanics the cost for using the cache shall we say but the benefits of sending and receiving far outweigh the costs of using it
Rooz: and for folks, sorry to interrupt you, for folks who don’t know what a build scan is it its essentially a scan of your build
Jason: thanks for that
Rooz: So, every time you run a build you can run your build, depending on which version of Gradle you’re on, with the dash scan command and it will basically send your build (metadata?) into this very nice, rich UI where you can search it and what not and you can get that from scans. Gradle. Com or just search for build scans
Jason: At Salesforce, we use Develocity but if I don’t have that can I get build scans?
Rooz: You can but the data does go to our EC2 servers, so probably folks at certain organizations you know, big banks and whatnot might not allow that, there might be some sensitive or custom information on your libraries and which libraries you’re using right so if you’re working on a secret project …..
Jason: so maybe good for one-offs or…
Rooz: Yeah it’s good to test it out and check – look at your build. So what Jason is showing is a view of Develocity from all your scans. Specifically, you can see in the query box showing CI builds. So, show me all the successful CI builds from the last 7 days so what are we looking at here, Jason?
Jason: so, let’s drill into it and I want to talk about how to read this because this is how you measure where you are today, so you know how to focus your resources and where to improve
Rooz: is this the first place as an administrator and someone who owns the builds and owns developer productivity where do you usually start is this one of the first windows do you look at
Jason: This is where I start
Rooz: how often do you look at this
Jason: daily
Rooz: daily, ok
Jason: so were training developers to stop sending me Jenkins console logs and send me the build scans because I get far more data from that and if they send me the Jenkins link the first thing I go get is the build scan link. So, let’s talk about how to read this for a second so in that box it shows this is one scan out of that whole one-week interval. I think we had 600 builds in a week on Jenkins. How do we read this, its starts today at 2 pm the build time took 2 mins and 58 seconds, this is wall time so if you started a stopwatch and ended your stopwatch this is how much it costs. Many of the other numbers you see here are serial times so if they’re running in parallel those wall times will be shorter so if we stretched our all our serial tasks into one long string it would’ve taken five and a half minutes so just by turning on parallelization we’ve got a 2x speed increase ,and I didn’t do anything. I added a one-liner to my Gradle-dot-properties, boom, so we also save 9 mins by avoiding things and we’ll talk about that in the next slide. And our build-cache costs us one minute, but well also measure what it saved us. So here’s a serial task execution if we stretched them all out it would’ve been twice as long. And here’s how much time you save by avoiding tasks so they can be up to date may be because you already did it and nothing has changed. So I don’t need to do it again or I get it from a cache either from local or from the remote build cache, see this has saved me 9 minutes for this build so if I did not have the build cached or if it was misconfigured my job would’ve succeeded but it might’ve taken 9 more minutes and this is what it cost me to use the build cache. On the left I’ve got build time and now I can see the fifth percentile, 95th percentile and then the mean, the average is I’m spending about 3 mins doing this and over the last week we did 638 builds on Jenkins and if I summed up all that build time we spent 3 days building out of the last seven days so Jenkins was pretty busy. The next box shows serial task execution so if I stretched all these out on average we’re like 8 and a half minutes so a little bit more maybe not as good as that one we were looking at, but this is the average for the week. Avoidance savings – so these are the things the build cache is the helping with or the up to date checks are helping with, so we saved, and these are net numbers after the cost of the build cache. We’re saving four days of build time so we’re saving more time than we actually spent building. We’re saving more than half of that on average over this week which is incredible on an average build we’re saving 9 mins of build time
Rooz: and when you look at this dashboard on a daily basis what are you looking for
Jason: I’m looking for the peaks and I want to see why those are so tall. Typically, these are days or scans that had a lot of tests for this project we used fire-based test lab and Espresso tests on simulators so those may take 20 mins or so, those are not quite parallelizable through Gradle. You can parallelize them a little bit through fire-based test lab, so that kind of a something to Gradle so some of these tasks are going to look like that you might want to see the ones that have mysteriously low cache hit ratios and why. Maybe something got pushed to the cache server and a bunch of important or desirable things got evicted. Maybe your cache is too small? Things like that. And then drilling into the avoidance savings so we saved about 9 minutes, half of those were because tasks were up to date and half of those from the build cache. So that’s 50/50. That’s pretty good and the up to date means that none of the inputs have changed so I don’t need to do this task again. So, I think of it like if you run java compiler twice but you didn’t change anything. There’s nothing to compile because you already have the outputs, so it’s important for plugin developers to declare your inputs and your outputs so Gradle knows if somethings changed or not. If you don’t declare those or if your outputs overlap with other outputs Gradle will get confused and not be able to do these up to date checks which makes your build slower. So, the build cache costs us on average 25 seconds but remember were saving 9 minutes. The net avoidance number is the net number and then you can break this out and see the uploading time downloading time zipping and unzipping for the artifacts downloading takes more time than uploading which you would expect so these ratios look good I wouldn’t worry about these numbers for us
Rooz: and since you have developers all over the world I would say how do you deal with the cache node issue well I guess my question is how many cache nodes you do have
Jason: so we have one Develocity server in the same datacenter as most of our build infrastructure and we have a standalone build-cache right next to it we’ve put them on different machine so they don’t compete for disk space or IO, but I’d flip that and turn it to Gradle because I think you have more international developers do you have more than one build cache
Rooz: yeah we do we have one in Germany and one in the US west coast and one in Australia because that’s kinda where-
Jason: and they all synchronize resources?
Rooz: yeah
Jason: so, your developer’s kind of pick the one closest to them?
Rooz: yeah though before we had that feature our developers in Australia had a great internet connection right they weren’t in the local cafe trying to download from the remote cache but it was still doing pretty well but you know the smart cache replication feature certainly helped and it’s easier to set up the cache
Jason: so, you saw a measurable improvement
Rooz: yeah absolutely and we look at the same basically view that you’re looking at now
Jason: by the power of movie magic, I put together the avoidance savings with the cost of the cache so net we’re saving four days of time over a week of builds and it cost us four hours so that’s a really good trade-off in my book
Rooz: wow yeah absolutely and out of these four days of time
Jason: seven days
Rooz: sorry yeah how often are developers actually waiting for CI builds to finish, I mean they’re not waiting for all CI builds to finish some of our data shows 20 percent for some other organizations it’s more like 50 percent or 80 percent, we are going into a world where you see these ephemeral builds and folks are triggering a lot of CI builds so what are you seeing kind of from your side as far as the impact of-
Jason: -of this
Rooz: yeah feedback cycles right, on something – how often are they actually like hey I need to wait here for this CI build to finish to see what’s going on how often does that happen
Jason: Yeah so our times are taking our builds are taking the Gradle chunk as you can see is like seven minutes according to this screenshot. There are other pieces that go before that you know we have to get the Gradle wrapper we have to copy the code from Git? (21:58) to the build machine. We have to send the results back to the server when it’s done so developers may issue a pull request you know and then Jenkins will build it and give feedback. And that it gives feedback throughout the whole life cycle so we have different stages. First, we typically do a linter or a code format check so if you did all your tabs and spaces wrong you know that’s going to tell you immediately and then it might tell you about your code style like “Rooz you’re not checking for null pointers? (22:26) left and right this is not a good, not a good pull request go back and rework it” and then it will run tests and then it will do perf and all those other things. So, we’ve set it up to give you multiple git hub checks as it works through the process so your kind of getting feedback as its building. But I would say it’s long enough, it’s too long and the developers might contact switch? (22:47) or get a coffee um it’s definitely not down to five minutes but it’s enough that they’re not quite blocked but they are going to switch to another task. I think the most important impact is that it helping them with their development life cycle so as they might be switching branches or switching projects you know that cache that local cache is going to help them build faster. So like I said we’ve got maybe 30-40 Gradle projects in the workspace if they switch branches a bunch of those projects might be invalid dated but a bunch of them might still be cacheable and we don’t have to build you don’t have to do a Gradle clean and Gradle build don’t do a Gradle clean let the build cache and let the cache work, you can rely on it
Rooz: and speaking of which did you have any issues there with folks not trusting the cache and saying you know what I’m just going to run clean, I mean it happens less with Android as certain other JVM build tools
Jason: we see it and I’ve seen it and the build scans help me find that cause I can see what tasks people are using and I can say hey you don’t have to do clean or hey clean cost you this much time try it without a clean and see if you get a better result or we can catch Jenkins jobs that are calling clean that we just get rid of because its wasted time and that and that’s’ brought down build times too
Rooz: cool cool thanks for sharing that
Jason: of course, so let’s get some audience interaction
Rooz: alright yeah yeah let’s do a poll here and see who is using the build cache today I would say it’s about 10 % percent maybe 20 percent yeah, so what would you say to those folks I mean we talked about it of course
Jason we talked about it I would definitely turn it on for local builds because you’ll get some impact for the average developer and if you can try and set it up for remote builds Gradle makes it available as part of Develocity I think you can get it as a jar? (24:49) or as a doctor image (24:49) so it’s very easy to prototype
Rooz: yeah as Jason said the build cache is your best friend
Jason: yes so let’s talk a little bit more about Android stuff now that we know how to read the build scans and what to measure that’s how you can measure the changes your making don’t just go willy-nilly change stuff because your gut might make you think it’s going to be faster, go and measure it. So, Android x and the Jetifier. So Android support library came out many years ago it is a set of add-on’s from Google to make Android better, and at Google.io 2018, Google announces Android x and Android x is replacing the support library um it’s going to be semantically version and a little more version compatible. So if your dependencies have different versions of the support library they will all play a little better if you’re using Android x. Android x is very similar to the support library but they changed the package so google helped by producing a tool called the Jetifier and what this will do is it will go and rewrite your dependencies to use Android x for you because you can’t mix and match the support library and Android x which is good for you because you can start using Android x and you use the Jetifier to transform all your dependencies, its bad because it costs time to do that rewrite and we were seeing 30 seconds per flavor so if you’ve got debug internally and release flavors that’s a minute and a half thereof rewriting your dependencies. You can measure that with the build scan, I think the task is called transform task, if you can update your libraries to be Android x so you don’t need the jetifier that’s the best way to go forward and if you’re writing a library please go switch your library to Android x so that you’re customers and your users can switch to your latest version and we can all move past the support library
Parallel, this is one thing that some of our projects did not have on because the Gradle scripts had not been touched in a long time there’s a lot of cobwebs all over them but turning this on was a really quick fix. Tasks that Gradle figured out could be parallelized and didn’t depend on each other started running in parallel immediately if you need to make a debug build and a release build at the same time those a really easy example of two things that the computer will do at the same time they don’t really depend on each other and Gradle will figure out all the task dependencies between them and Gradle will know how many cores> (27:17) your machine has and will decide the best number of threads to use automatically
Rooz: and we’ve seen this I mean I have personally seen this because I work with Tony on a lot of Develocity trials and proof of concepts we look at the timeline view of the build scan and we see that its quite parallelized and Tony sometimes asks “ hey maybe you should throw some more cores at this and bring that down” from your side any tips and tricks on getting more parallelization right is that something that you look at like ya know it’s pretty great how we have it now but it with working at Salesforce I’m sure you have these massive machines, unlimited hardware power at your fingertips and I’m sure you could throw more cores at it right? How often do you look at getting the most out of parallelization and what tips and tricks do you have?
Jason: sure so when you look at your build scan, a healthy one, when you look at the timeline view it stacks and shows all your tasks, a healthy one might look kind of like a brick wall with a bunch of little rectangles very long and very tall because you want all those threads being used. A very unhealthy one will be a long skinny bar because they’re all done in one thread that would be bad what it’s probably looking like is very tall at the beginning is it kinds gets narrower as we get to the end of the build process and maybe all those inputs are getting merged into an output file so look at those later things see if they can be broken up at all. Check if you have depends on that don’t need to be there, like if you’re producing a report, like if you use fine books if you’re producing an HTML report you have to run fine books. First, you can’t do the report before you do the scan so some of these do depend on each other. There’s another option where you can say should run before or after, which you can tell it if your doing serial, do it this order but it doesn’t necessarily have to be and if it has the cores it will do it. So if you can find some things that maybe don’t need to depend on each other or removing those depends on or changing them to should’s then you can get it a little more parallel and you know more than 8 or more than 12 cores might be overkill but it depends on what you’re building. If you have 30 projects like our workspace then maybe 30 cores will be good to get you going maybe not but in the build scans. It will tell you how many cores are available to it and use the data to tune it for your environment
Rooz: alright thanks for the insight on that
Jason: as we saw on the earlier build scans we turn on parallel and for this build scan we saw a 2x improvement so the other thing with this is gives Gradle a whole bunch of things to do at once. Don’t like one-off these Gradle commands as part of your build script. Don’t do assemble and then a lint and then a release, because every time we do that like in the second example Gradle has to start up and we didn’t talk about it today but there’s also a Gradle daemon. 30:20 you can turn on that will keep some of that cache information warm but there’s still startup costs that running the Gradle wrapper. If you do the first one Gradle has an opportunity to parallelize those two tasks if you’re looking at the second example there’s no way to parallelize those because they are completely separate processes. If you can try to give Gradle a bunch of things to do at once depends on your pipeline, repositories and dependencies so here we’re getting into some more code on the screen. So, I want you to remember that the order of our repositories where you get your dependencies it does matter. So, if we have 30 or 40 projects and those have I don’t know let’s say we have 400 dependencies to get and that includes Google tools and libraries ya know. Gradle is going to go check what do we got one, two, three, four, five for different repositories its gonna go look in all those repositories for all these things and its gonna go in order. So if you can hint to it like this uh Google one is your dependencies are more likely to be in Google than J center put Google at the top um and if you’ve, IF you’ve got dependencies in more than one repository the higher one is going to win. So, if you have an internal Maven you can also use these repository filters and I gave a long link at the bottom, but this helps hint to Gradle what kinds of dependencies are in this Maven. So, for example, if you have got a maven server at your company that has company artifacts you can tell Gradle to go look there for it. You know with this example it will never go look in Maven. My company for a com. google dependency and that will cut down on your network io and that will help Gradle find your dependencies faster while making less network requests. You can also do some tricks here to make sure that you don’t accidentally get a com.my company resources from J center. Say for example you maybe have com.mycompany. Internally you never want to look in J center for those it can be a security feature too to make sure you’re not getting dependencies from unexpected places
Let’s take a poll so this example was with a groovy build script that Gradle supports column? Build scripts too so what are you all using for your build scripts
Rooz: and can we ask what you are using for your build scripts before we kick off or as I am kicking off this poll here
Jason: yeah so the majority of our Android projects are using groovy scripts out of legacy but for new projects were using Kotlin and for simple ones, we do move them over to Kotlin. it gives us because the engineers are using Kotlin now too. Now they don’t have to know java groovy and Kotlin they’ll know java and Kotlin and they’ll be a little more comfortable with the build scripts
Rooz: alright and we have and I uh think that’s good enough for us to close the poll and show the results
Jason: everyone is using groovy
Rooz: everyone is using alright ok let’s get back into it
Jason: so, if you’ve got internal infrastructure and it really depends on your company and what you’re using if you’ve got internal infrastructure use it, your Jenkins might be internal. If you can set up a cache for the Gradle wrapper or other things you can do that. So, here’s an example of a Jenkins pipeline that checks if your project is using the Gradle wrapper and it will go rewrite that properties file to say get the Gradle wrapper from mirror.mycompany.com the Gradle wrapper is the binary distributions a good 80 – 90 megabytes. If you need to fetch that for every single job that bandwidth adds up you know you may not have unlimited bandwidth from your data center to services.Gradle.org. If you uh can get it internally it may be more reliable and faster to get it that way and if you need to give HTTP basic off you can do that here with these Gradle wrapper users and Gradle wrapper password don’t put those in your source control but you can expose those as environment variables and through your Jenkins file so this means that for build machines we kinda shortcut it and get the Gradle wrapper from an internal company mirror? That might be very close to the build machines with more bandwidth but the average joe developer might just get it from services.Gradle.org, maybe they are not installing the Gradle wrapper very often Gradle wrappers a good way to have different versions of Gradle for different projects or different branches without having to install more than one Gradle on your build machine
so were a little poll happy right now but what CI system are you using
Rooz: and here are the results Jenkins at 59% team city at 4% we see a lot of circle ci Travis and other as well
Jason: so, the other important Gradle plugin that you must be used as an Android developer is the Android Gradle plugin from Google. It’s important to upgrade this Google is constantly making performance improvements and bug fixes, for example, Android Gradle plugin 3.3 came out in January it fixed a major memory leak in lint and is also used more configuration avoidance API’s which means it doesn’t create a whole bunch of tasks unless they’re actually about to get run. So you may not create release tasks if your only making a debug build which makes the configuration step for Gradle run faster and 3.3.1 has a parallelization fix for lint you can see more about this issue with the link at the bottom of the screen, uh 3.4 came out in April which used more configuration avoidance API’s which means that the warm-up for Gradle is faster. More parallelization and conversions of transforms to tasks which means you’ll get more out of the something build cache and as we saw with Salesforces example the build cache made a big improvement on performance so these you may not think to upgrade ago but you get measurable improvements out of this not just new features but fixes and performance improvements. AGP 3.5 is around the corner and I think the biggest feature will be caching of Android unit tests so if you’re doing Robo electric or JVM unit tests they’ll cache which means if your java sources don’t change your tests won’t have to run but you’ll reuse that the outputs from last time and it’s important to note they are only cached when the tests pass so if you have tests failures it’s not gonna write that to your build-cache uh more transforms will be converted to cache-ful tasks which make them more cacheable there’s a class loader leak and data binding will get better with incremental annotation processing which means that you don’t have to start your builds from scratch you can reuse outputs
Rooz: and we had a question earlier from the uh questions toolbar about our transforms cached yeah so that’s coming in 3.5 um. You can try out the beta now and I should add a little bit of context here the google Android SDK they own the Android Gradle plugin and we work closely with them on Gradle build tools side
Jason: yeah
Rooz: and you wanna make a thing as I said build times and caching is your best friend
Jason: yes
Rooz: right the lowest hanging fruits to developer productivity is build times and the best we’ve found how to do that is through caching. So, with each version of the Gradle build tool, there are more things cacheable. There is more data about cacheability and ways for you to optimize your build so there’s more things going to build scans and more things going to Develocity and we work closely with the Android tools team to take advantage to make Gradle build tool have features that make every version of the Android Gradle plugin faster and more cacheable. That is one of their top objectives of that team is improving build times so everybody understands the priority of this and wants to make this better and faster
Jason: Yeah and I think the same could be said about updating Gradle
Rooz: yeah you wanna stay on the latest right we added some amazing dependency management features around dependency locking and resolution rules and what not to with Gradle 5.0 and of course with every version of Gradle there’s more things that are gonna be cacheable and faster, faster builds is part of our reason for existence build happiness
Jason: perfect if these issues from Android Gradle plugin are important to you take a moment to go click on them and star them and let the google Android team know that you want to see them worked on I think we’ve got one more poll
Rooz: all right yeah so let’s see what version of Android Gradle plugin you all here are using and Jason while we let the polls come in how I mean you work at a large organization multiple Android apps it’s not easy being on the latest all the time you have millions of users around the world right, customers and whatnot how uh how do you deal with staying up to date and how up to date are you and team and what are your kinda tips and tricks for keeping folks up to date because you have all these different Android apps at Salesforce right
Jason: That’s a good question. My build engineering team does have a best practices example app uh that we use internally so that is kinda our first step at testing any of these things so those are on ADP 341 Gradle 541 which are both the latest versions available today build scans 2.3 that’s kinda the first place we go to test releases and then well start sending pull requests to some of our most complicated apps like the Salesforce app. If that’s passing then you know we can kind of trickle it down to all the other apps because if our most complicated use case app is happy then pretty much the other ones will be smooth sailing and we use the Develocity to study what versions something? People are using so we can go poke and see oh this is on our really old version let’s spend some times improving that
Rooz: so, let’s share the results I should take a screenshot of this and send it to our friends at the Android tools team let me just do that here boom alright, alright let’s keep it going
Jason: ok I think we’re ready for questions Rooz
Rooz: alright were in Qu and A already well yeah it is kinda time
So, isn’t using maven local considered a code sme? Hmm, I don’t know what that…
Jason: depends on your project if you are publishing things to your local maven may be because you have not set all your projects in one workspace you can do that we use some recognative? Modules that like to put things into maven local, so we put it there first its gonna work for your developers but not for your CI machines
Rooz: Gradle wrapper assemble debug takes a decent amount of time after enabling parallelism and build cache but Gradle We build takes about 20-30 mins or so how can we reduce that
Jason: I would start with a build scan if you don’t have a Develocity you can send one up to Gradle’s public site and you’ll get a link emailed to you and you can choose who’s gonna see that scan or not and that from there you can really look at your timeline view you can sort your tasks by longest execution and see if that makes sense to you if there’s any red flags from
there you can also see which tasks were run and if it says it was uh not up to date but you expected it to be the scan can tell you why maybe some files were out of date maybe something had changed that you didn’t change if more than one task are outputting files to the same directory Gradle can get confused and rerun tasks
Rooz: are there any hints and tips when JNI is involved seems to be having a big impact on build times and maybe you should talk about what JNI is for the other folks
Jason: we do have one project that used JNI. JNI is where you write C++ code and you compile it with the Android NDK and you may have some java native interoperability (JNI) so you’ll want the NDK some of those tasks may not be parallelized yet I’m just starting to look into some of that now for one of our projects again I would maybe ask on the forums and try to see if those outputs are cacheable they may not be because they maybe a little more platform dependent
Rooz: alright, if my CI is publishing cache to build cache node? With different version name and version code, how will that impact in cache hit percentage? I guess from other team members
Jason: When you say version name and version code it makes me think that you’re talking about APKs and that’s a little more high level than what the cache is gonna store the cache will look at individual tasks and store those outputs so, for example, you may have a java compilation task or a Kotlin compilation task those intermediate files are what’s going to be cached and not cached. Gradle is very good at not mixing those up it will look at maybe uh check some for the input files to really make sure that the same inputs produce the same outputs as when we talked about the weather example. If there are outside influences to those files that might cause cache misses or some confusion but that’s kinda down at the plugin level to make sure that those tasks are cacheable and a build scan will tell you which tasks were cacheable and which were not and if not it will typically tell you why it was not cacheable maybe the task didn’t have caching enabled or maybe there was a miss
Rooz: how many modules are too many is their performance cost to too many modules and this is from James
Jason: I’m sure there is a limit. I can say that our biggest project has got about 40 modules in it were trying to pare it down as I said some of those modules haven’t been touched in a year we would like to take those out and convert them to aars? Instead, if you’re working in a mono repo type thing build-cache would be even more important to you especially if not all those are under active development more parallelization might be something to look into for that one too if you’ve got a lot of modules there might be more opportunity to parallelize things
Rooz: alright here’s a capped? question how beneficial is cap.incremental.apt? Will that make a difference in build time
Jason: I think the easiest answer to that is to measure it yourself um turn it on turn it off do some build scans and measure it you can use buildscan.tag to tag your builds with that on and tag your builds with that off just like in my screenshots. I was looking at the CI tag uh and based on that you can maybe do a handful of runs or whatever is statistically significant for you to measure it I think it depends, I don’t have an answer off my cuff but I think the most data-centric answer is to test it
Rooz: to measure yeah
Jason: both maybe on your workstation and on CI
Rooz: Yeah so you should just run a build scan on both of those and look at the performance tab and check it out alright last call for questions I think you’re in the clear unless you want me to throw some more questions at you, no I’ll save that
Jason: Thank you
Rooz: Thanks Jason for your time uh just checking in if I see anything else, ok um yeah uh thank you, Jason, for coming here and sharing some of your best practices with us and yeah and thanks to you all for joining with the webinar and well see you on the next one
Jason: Thanks for having me Rooz