Making the Most out of the Build Cache

Build cache infrastructure can be critical to accelerate your builds. But don’t expect to achieve your maximum potential build acceleration right out of the box.  This webcast will show you how you:

• Assess the effectiveness of the build cache for your builds
• Identify and prioritize tasks to optimize for build caching
• Debug build cache misses and determine the root cause
• Monitor build cache effectiveness over time
• Create a distributed build cache cluster for faster builds

Making the Most out of the Build Cache

Thanks to everyone that attended the webinar and thanks for your questions!

Rooz: Good morning, everyone. My name is Rooz, I’ve been working in customer support with Gradle for 5 and 1/2 years now, mostly working with Tony. We’ve been helping users and customers adopt our tools and become proficient with them, especially with caching. That’s the big topic, caching and that’s why we’re here today.

Tony: My name is Tony Robalik, I’m a software engineer at Gradle, and I’ve been focused on customer support primarily working on the Gradle enterprise team, helping make the Gradle enterprise tool more effective for our customers. Before that, I was on the Android team lead for Chess.com, where we were actually rewriting our app from scratch with modern architecture and in Kotlin.

Rooz: So let’s run through the agenda for today.

Tony: So the agenda is pretty straight forward. I think to get started it makes sense to just talk about build-cache basics really quickly just so we’re all on the same page as far as how the build cache works. Once that’s complete we’ll talk about the benefits of the build cache, that is why would we even bother using it. Then we will talk about build-cache monitoring, debugging, acceleration, and how that can accelerate your build. And then we’ll talk about build-cache replication, that is if you have a globally distributed team, how do you make the build cache work in that scenario. And then finally, of course, we will have time for a Q&A.

Rooz: All right, so we want to talk about the impact of caching on developer productivity. In the last few years since we launched the cache for Gradle, the remote and local cache, we’ve seen a lot of folks have been interested in how do I improve build times. And the best way is with caching and the impact is fairly huge. So the Gradle team ourselves were one of the first to do this really well. We save around 60 days every 24 hours. Tableau one of our customers and users are saying they save around 15 hours every two weeks. But it’s not that easy as we will learn more today.

Tony: Let’s start with our first poll.

Rooz: So who’s using the build cache today, that’s what we want to know. So let’s kick this poll off here. All right, so we got 35% using the local cache only and a good amount of folks not using the build cache, and only 6% using the remote cache. 

Tony: Yeah, that’s pretty interesting if using the remote cache only, we’ll get more into that later. So let’s keep going. All right, so build-cache basics. What is the build cache, and how does it work? So at a high level, the way the Gradle build cache works, it’s almost like an extension of incremental build, which is all those up to date checks you see on your build. Whenever you ask Gradle to invoke a task, like clean, check, test or build, it looks at all the inputs to that task. It can be a hash of those inputs and create what is called a cache key from those hashes. And then it will look in the build cache, either the local build cache or the remote build cache. And if it finds that cache, gets the cache hit, it will take those outputs on the build cache and then substitute them into your build.  If there’s a cache miss, then it has to execute the task, which obviously will take longer than pulling it from the build cache. 

The reason I have the final bullet point on the screen, that the build cache is always on, is that there is never a scenario in which it makes sense to turn the build cache off. That’s the point I want to hammer home repeatedly throughout this Presentation.  We all are familiar with the idea of having a dry code, you don’t repeat yourself, don’t do anything twice. It’s the same with the build. You want to have dry builds, don’t build something twice if you only have to build it once. That’s what the cache is all about.

Rooz: And that’s what you mean by dry?

Tony: Yeah, just dry builds, do it once and only once.

Rooz: And what does this diagram do?

Tony: So this diagram is sort of a visual representation of what I just said, which is when you invoke a Gradle build. The first thing it’s going to do is check if that task is already up to date, and it’s local memory, and if it’s up to date it won’t have to run the task again. If, however, the task is not up to date, Gradle will then look in the local cache to see if a task has been run anywhere in the past. And if that’s the case, it’ll pull that task from the cache and substitute the outputs saving you the time from executing the task. If it’s not in the local cache then it goes to the remote cache. And so that’s where the real cache flow comes into play as you can share those outputs across your entire team and CI infrastructure. So again, it’s not just having dry builds locally, but dry builds across your entire team. If it’s a cache hit with the remote cache, it will then copy that output into the local cache so that next time, it’ll be faster.

Rooz: So every consecutive build after this point, and from the remote cache, is going to get faster because it’s going to auto-populate.

Tony: Exactly

Rooz: And you don’t have to do anything to make that happen?

Tony: No, it’s just totally transparent, Gradle handles all of that for you.

Rooz: Great. So for the folks on the web, and so everybody gets how the cache works, there is a little raise your hand tool on go to webinar that you can use. So raise your hand if that didn’t make sense, or if it’s still not clear on how the cache works.

So we’ve got a couple of folks, that are still not clear on how the cache works, and you can message us the question of exactly what you’re not sure about and we will get to it at the end.

Tony: Yeah, we’ll do all the Q&A at the end.

Rooz: All right, so one thing, Tony, I get this question all the time, “Hey, I’m already doing caching with Artifactory or Maven Central, why do I need cache if I’m already doing that?”.

Tony: So this is a common question we get from people who are new to the build cache. It is a different thing than a binary repository, and when I say binary repository, I mean something like, Artifactory, that hosts your libraries, all of your dependencies, everything you see in the dependencies block of your Gradle build script. Those are your binary dependencies and those come from a binary repository like Artifactory, and the build cache is not that. If the binary repository is everything that’s external to your build, the build cache is everything that’s internal to your build. It’s every task itself that can be cached, as opposed to the output of the entire build.

Rooz: So you’re essentially caching every step in the build process.

Tony: Exactly, every step gets cached.

Rooz: And that’s what makes it super fast. And that’s what you mean by dry build, so I’ve already done this, it’s up to date, don’t do it again.

Tony: Exactly right. To make that a bit more clear we have a concrete example of that, which is caching test execution. This is something you can’t do with say, Artifactory or Maven Central, but with the build cache, you can. Let’s say you have some expensive integration tests as part of your test suite and so then you run all your tests, it takes a few minutes, or maybe even a few hours in some really bad cases. And then you make a small incremental change to your code, say, change one line of code and one of your modules and then obviously, you want to rerun the tests for that module. But there’s no reason to rerun the tests from all the other modules that didn’t change. And so Gradle build cache has your back, it keeps your builds dry and all those other tests all will just come from the build cache, and the reports are already there, and they’re done, and then it’s delivered to you.

Rooz: So this is for unit tests and integration tests? 

Tony: Yeah, anything that’s a test task.

Rooz: And that’s why we’ve seen that the remote cache is just really doing really well with improving CI build.

Tony: Absolutely right.

Rooz: It’s just caching those and that’s why we have these long Gradle core builds that are now super fast.

Tony: Yeah, and if you’re continuously integrating, then your CI builds will be just a little bit different than the build before it, which means it gets a lot of benefit from that build cache.

Rooz: So one question with regards to that that I got from the folks, “Can you give a specific example of cache versus binary? So i.e. Gradle 3.5 is local. So caching downloads 4.5.1 and caches it locally.”.

Tony: So that’s what I was just trying to explain here, is that you don’t host your test executions in a binary repository. This is not like a library that your software product consumes, it’s something, only the Gradle build cache can provide.

Rooz: So it’s basically only caching dependencies, rather than it caching every step of the building process including the test.

Tony: Yeah, it’s like caching the build process versus caching the published artifacts of some site. J Unit is an example of something that would be important to your build in dependencies as an external dependency.

Let’s do another poll.

Rooz: All right. So, “What are you building today?”, that’s the poll. So, Tony, this cache works for everyone, whether you’re building Java, Android, C, or C++. If you’re using the Gradle plugins, this will work for everything.

Tony: That’s correct. Yes

Rooz: So let’s see what folks are building today. All right, so Java is the winner right, lots of Android and I’m excited that native folks are in here as well. That’s pretty cool, I’m curious about the other, send us your others, we’d love to hear from you. So we’ve got some Golang, building Golang with Gradle.

Tony: Nice. That’s the build cache basics out of the way. Let’s talk about in more detail why you would want to use the build cache.

Rooz: So Tony, one thing I think would be interesting to share. I’ve been pretty impressed, I mean, I haven’t read code in a very long time. But I’ve been pretty impressed with how well we are using the cache internally with how CI populates the cache. We have a Gradle today, we have multiple different teams all over Europe, Asia, Australia, and all over the US. So it’s not an easy thing and I think it’d be really cool to share how we’re pulling this off.

Tony: So there are three primary scenarios in which the build cache really helps out with local builds that are built on your local development machine. And the first one is the first build of the day. So let’s imagine that you have your morning coffee,  you turn on your computer at 7 or 8 or 9 AM depending on when you like to get up. And you pull master and then you open up your ID and the first thing you’re going to see is it’s going to be thinking. But then it’s going to try to build the project and if you’re not using the build cache that could take a while if you have a big project. But with the build cache, you’re going to find that most of those sort of interbuild components, the process we talked about, have already been built on CI. So Gradle will pull those components, those build outputs from the CI build, down to your local development machine and it could save you a tremendous amount of time. I think we see like 40% or more build speed improvement with just that first build of the day. So it just makes your morning a little bit sweeter.

Rooz: And the first build of the day totally sucks for everybody and the bigger your project gets the worse it is.

Tony: And you’ve got a cold Gradle demon. You know, you’ve just opened up slack and you have 30 Chrome tabs and it’s all-consuming memory and CPU time. So you want to make sure that that first build is fast and that’s where the cache has your back.

Rooz: So the remote cache, the big benefit of the cache, is usually the first build of day and then the local cache kind of takes over.

Tony: And then the other two examples are like switching branches and updating your branch. So in branch-based development it’s really common that your colleague, you know she has a PR, it’s been built on CI she wants you to review her code, and so you want to see it in action. So you check out her branch and you build it. But because this argument is built on CI and built somewhere, and it’s populated the remote build cache, you can build her branch much faster than if you had to build it all from scratch on your own machine. So that’s also a really big use case for the build cache both local and remote.

And then the final example is when you’re in the middle of the day, you are getting a branch with upstream changes. So you’re pulling master, you’re rebasing your branch or something, you’re merging code in. And once again, that code has already been built somewhere else in the world. So keep your builds dry. Build only once. And that’s where the build cache really kicks butt.

Rooz: All right, let’s move on.  So CI builds, I talked about this earlier. One thing I’m seeing from our users when we are doing Gradle enterprise customer support, checking in, and looking at the dashboards from our users. And the remote cache is just crushing it with CI builds. I’m seeing around 40% on it. So how does that really work? Because you’re supposed to run clean for CI.  So I get that question a lot. But with caching CI, I mean we brought up the test, but with the CI how does that work because I get that question a lot.

Tony: Right. So it’s very common to run all CI builds are clean builds. And the reason this is the case is that we want to ensure that there is no sort of incremental task outputs present in the build. We want to have a clean sort of working space for the build. So that’s what we clean, we clean up a lot. And we clean up all the incremental outputs because it’s hard to hit incremental tasks right sometimes. The Gradle core tasks are all very well tested and we’re confident that they work right. But if there’s a custom incremental task that could have a problem that’s hard to debug and so we clean the workspace and we run the build. And so while it does improve safety. It does impose very heavy performance costs since those builds take a lot longer than if they were incremental builds. And so this is a case where the build cache really comes into play because it’s caching all of those build outputs from each of those clean builds. And so like I said before if you’re continuously integrating which we all should be doing with our code, then every one of the CI builds will only be a little bit different than the one before, for the most part, you know in most cases. So the second build versus the first build has a small change. You know, most of the output from the first build is still quite valid. And so the remote build cache in the local cache can support pull of outputs into the second build and save a tremendous amount of time. And we are seeing like really huge savings cutting build time in half, basically in many cases on CI

Rooz: And one question I have here with regards to, I think it was from the last slide but somewhat is applicable here. Build cache works for the same branch or between different branches?

Tony: Yeah between branches. The way the build cache works is it just it’s completely agnostic as to where the build was run, whether it CI or local, whether it’s to run on branch A or branch B. All it cares about is the inputs and outputs to a task. So the way the build cache works is it computes the hash of every input to the task creates a cache key from the aggregate of all those inputs and then that key is valid doesn’t matter where it was created, whether it was on your computer, your colleagues computer, your CI server doesn’t matter. Yeah, so it just doesn’t matter what branch it’s on.

Rooz: All right.

Tony: There are a couple of other points, I think is useful to make here, which is that it’s become increasingly common for teams to use what we call ephemeral build agents. And by that, I mean like Docker containers or what have you. So if you’re spinning up a Docker container running a build then tearing it down. What that means is that every build is on a fresh workspace, completely fresh. And so once again, the build cache has got your back. It doesn’t matter that it’s in a Docker container. It can still pull those artifacts, and save all that time in those ephemeral builds. 

The final point I want to make here is that we have seen a number of teams have really, really interesting, but quite complex sort of hand-grown tooling for this problem. So they already have a very long CI build and it might take an hour or more and they want to make that build faster which makes sense. We all want to make our builds faster. And so they don’t use to build cache yet. And what they do is they have this specialized tooling that they’ve written themselves that will, for example, parse get commits looking for you know which files have changed so that they can tell those CI agents you know which parts of the build to run. And that works in some cases, but it’s error-prone,  it’s complex, and it’s hard to maintain. And what’s really great about the build cache is a sort of it can erase all that complexity because you just tell Gradle build everything just build everything. And then it will already handle the cases for you where something has been built before and hasn’t changed and pulled from the build cache. So it just helps Gradle build everything. And then the remote cache ensures that your build is dry and you don’t have to worry about all this complex crazy tooling that you’ve written yourself.

Rooz: And that makes sense because I’ve heard and been in discussions with multiple folks saying, hey, we are not using the cache we don’t trust it. We’re not sure how reliable it is. And we’ve built some tooling around that kind of makes your life easier, which you’ll show in a few.

Tony: Yeah yeah. So the best thing I can say there is you should try it out. We have many large companies that use the build cache every day producing hundreds of thousands of builds daily and build cache is absolutely essential to their workflow. And what you will see in the next part of the presentation is all the tooling we have around the build cache to help you monitor it and ensure it’s working the way you want it to work. So you can have that confidence going forward.

So I’ve talked a bit about CI. I want to describe briefly how Gradle’s CI works. Obviously, this is a very simplified diagram, our build is much more complex than just this, but sort of a high level. Well, we have this concept of a seed build, which just compiles everything. What it does is it compiles stuff for test execution for CI itself and also compiles things that only developers might need, even if the CI doesn’t need it. And the reason it does this is because they can populate the build cache with all those outputs. So the developers once again, keep their builds dry and this sort of fan-out pattern we think is really useful as something that’s enabled by the build cache and we would actually recommend it if you’re a build cache user. I’ll show you how this works in more detail in a little bit.

The final thing I wanted to say in the section is not every task should be cacheable. Some tasks, for example, are just faster to run than to pull from the cache. And an example of that is the jar task or a copy or a sync task. Those tasks are already just copying files, which is just what a build cache does. So there’s no point in caching that because it’s like just kind of redundant,  it just wastes resources. And then another example of when it doesn’t make sense to cache a task is in an Android build. There’s a task called out package application that builds your APK. OK, It’s probably the case that every one of your builds is going to result in a new APK file. Right, you’re going to change a resource, changing a lot of code, you need a new APK to run on your app and in that case, if you’re caching that task you know it’s just a waste of time because every build is going to be different anyway. So it’s just wasting resources. But having said that most tasks do benefit from the build cache and once again, I want to reiterate that the build cache should always be on. There’s never a use case that I can think of in which it makes sense to have a build cache off.

Rooz: And I have a question with regards to that. Cache is enabled by default correct? And it works as long as it runs on the same machine?

Tony: That is not actually quite correct. The build cache is not enabled by default. You have to enable it. And there’s a couple of ways to do it, which there are various links at the bottom of these slides that sort of point to more resources. And of course, at the very end of the presentation,  I have a slide with a bunch of links for further reading. You can learn how to do that. But just to really simply the build cache is not enabled by default. You have to opt into it. And the build cache once again, it does not matter which machine A build that’s run on the build cache will pull those artifacts from any of those machines as long as you have remote caching enabled and it’ll just work.

Rooz: All right. So one thing that you know we get a lot from our users is that “ I have turned on the cache, I set up the remote cache, but it’s not enough.” Right, this one and done just by turning it off. You know, “I’m getting low cache hits, please help me”. And I think that’s one of the big areas why we’re here. And so why do we need to monitor caching and maybe it’s good to point out how we do it internally with Develocity.

Tony: Absolutely, right. So as Rooz said, it’s not enough to just turn the cache on, you will certainly see a benefit if you do that. But it will not be the maximum benefit you could have. And so we’re going to show you how you can use Develocity to monitor the build cache and one of the great things about build scans and Develocity. it really helps to make the build transparent. For a lot of users, the build process is sort of like black magic and we want to move away from that idea. We think that the build is a manageable and maintainable thing and it can be treated as such and Develocity really helps to prove that point because it gives you all this insight into what happened during your build. So it is incredibly useful. 

So the first thing your point is about, how to monitor the build cache effectiveness. The first thing that I do as someone trying to help out other teams making their build faster via the build cache is I will look at what we call the performance dashboard. So let’s do that right now. Right, so this is the Develocity performance dashboard for the Gradle organization. Before I go too deeply into what you’re looking at I want to just do some quick filters. This is just the past day, by the way, this is in the past 24 hours, the Gradle organization has produced almost 10,000 builds. So we’re cranking pretty hard. Let’s just look at Gradle core builds. Let’s look at only successful builds. And I want to look at the compile all builds OK. So now this is just in the past 24 hours. This is our seed build. This is the thing that seeds our build cache. We have it tagged with compile all. I’ll describe this. This is in detail before going into the build cache specific stuff that’s very interesting and useful. And this view has a ton of information in it is just really condensed and this is quite amazing. So starting from the bottom up. What you see is this bar chart, every bar is a build and you click on the bar, you can navigate to the build scan for that build which I’ll do in a minute. You see all these summary statistics for the build. So at a glance, I know that in the past day this compile all build has taken almost three minutes on average. And I see that we’ve avoided 18 and a half minutes from various Gradle performance-enhancing features.

Rooz: So is that like real human time or how does that work. If I really wanted to know because some of that is incremental builds, right.

Tony: Yeah, you know, it’s a good question. So to answer that you take a quick step back and describe more how the bars are visualized. So the white circle in each bar is the actual build time. So this build, for example, only took just shy of two minutes to run. But you see the bar is much higher in that because it shows the various performance aspects of that build. The blue color is the serial task executions. That would be if every task ran in series or end to end. This is how long the build would take, almost 14 minutes. But in point of fact, the build takes just under three minutes. So there is a parallelization factor happening here. This build is very highly parallelized. It’s in fact 5.3 times faster than it would be if it all ran in series. So that’s the blue bar. And then the gray you see here is going to be avoidant savings. And there are various aspects to that. So we have up to date checks. These are tests that are already up to date. They’re already sort of in-memory in the Gradle daemon. They don’t need to be re-executed or pull from the build cache they’re just already there. And then we have the local and the remote build cache and we see here that these two aspects of the cache together are you know like 15 minutes. But again, it’s not wall clock savings. It’s not human savings. It’s CPU time savings. To get the avoidance savings from a human perspective. The heuristic we use is you divide this number 18 minutes by the parallelization factor of 5.3. And then that’s the human time savings. Now I’m not going to embarrass myself trying to do the math live but yeah.

Rooz: Three threes. So around three minutes and some change on average, it’s been saving. And these are all local builds.

Tony: You know these are these actually all CI builds in this case. The CI would run the compile all tasks.

Rooz: So it’s saving more or so it’s decreased the build time by half. Yeah by more than half.

Tony: Yeah by a significant amount.

Rooz: So it would have been six minutes on average if we hadn’t used this remote cache on the CI build.

Tony: That’s exactly right. And so what you see here is why I can click on any one of these and get more details about it. And the one I’m interested in and for this purpose is the avoidance savings. And I can see that for each one of these builds you know the blue is local build cache and then the gray is the remote build cache and how they impact the speed of all of these builds. So this again is a reminder of what we’re doing here if we want to monitor the effectiveness of the build cache. And I can already see it’s quite effective. The mean build time for these builds for the past day is about three minutes, but it’s cutting that in half because it would have been twice as long without the build cache. So you know that this build is pretty highly optimized. And then I can see that in more detail by navigating to one of these build scans and let’s see what that looks like.

Rooz: And if you wanted to recreate that dashboard yourself you’d need some sort of I guess hooks in the build system to send build times back to a spreadsheet or something and graph it.

Tony: Yes.

Rooz: Yes, I mean, I’ve seen other folks do this before with a spreadsheet. I mean, that’s why we built this product because we wanted to monitor that for ourselves.

Tony: Yeah, we had a pretty funny case where someone said, well, I could just do that with Auk.

Rooz: Yes, right. I remember that.

Tony: I wouldn’t definitely recommend that. So what you see here is a build scan for that build that I just clicked on. There is a lot of information here. I could give a whole webinar just on this one view, but I won’t do that. But I want to show you is how you use the build scan to debug cache effectiveness. So the first thing I would do as someone who didn’t know anything about this build. So just make sure the cache is turned on. So I go down to switches. And I see that the cache is on. These are all the Gradle switches. So the cache is on, continue on failure, daemon, and parallel are all turned on. So that’s all good. And then I would go to the Performance tab. 

So from here, I can see various performance features of this build and in the build cache tab shows me already that we have a really high cache hit rate 98% of the casual tasks came from the cache that’s built. As you can see, it’s really highly optimized. If we want to see details on it,  there are some misses. Let’s see what that’s about. On a test execution tab, I can actually click down here on these. These are casual tasks that were executed. So there’s only five of them. I click on that. I’m now in the timeline view and the timeline view is really cool. It shows all the tasks in your build, and how they were parallelized by these sort of stacked charts here. And what I’ve done by clicking on that link is I’ve sort of auto filtered this view to only show the tasks that I care about. So I’m looking at casual tasks that succeeded or failed. They’re highlighted in the timeline view and they also have this sort of list view at the bottom. So these are the five tasks that are casual but didn’t come from the cache. And I want to know why that is. If my goal is to pick the lowest hanging fruit what I might do is I go over to order and the longest and now I see at the very top here, this is the longest-running task that was casual and this is where I would start as a someone who’s maintaining the build to make my builds faster. 

Now, in this case, I could guess that this task didn’t come from the cache for a good reason. Probably some code really did change it had to be rebuilt. So that just makes sense. But in the general case, I can’t necessarily guarantee that. And this was where I would start for as far as the lowest hanging fruit.

Rooz: So we use this with the dashboard to find what are the longest-running tasks in your organization that developers run the most. And that’s where you would kind of start as far as the lowest hanging fruit.

Tony: Yeah And just something I may not be clear to people is that build scans are super extensible. So all these things at the top here are tags, these tags are custom tags that we add to our build scans and you can do that too if you’re a build scan user it’s very easy. Similarly, for these links, these are all custom linked and there are even custom values, which are just key-value pairs. So I know you know the branch name, and the get status, for example. This has all really useful context when debugging build issues, including with the cache.

Rooz: One question we have with regards to that, “Will parallelization of the Gradle build affect the build cache, especially for parallel test cases?”

Tony: Yeah, it doesn’t matter if it’s parallel in fact. One really interesting thing you might see, which I saw once a few months ago. So the way parallelization works is you have various modules of your build all running in parallel, all being built in parallel in the right order, of course. It’s actually possible to have one of those pipelines so broken, as modules will build something and then another module in a different parallel stream of work could actually pull that same task output out of the cache to use it itself. So you even see caching within build sometimes and that will happen only if the inputs to both of those tasks are identical, but that can happen. And it’s really cool to see that in practice. It means the build cache works really well.

Rooz: Yeah, so the build cache is no magic pill basically. You need monitoring tools to stay on top of this thing and get the most out of it.

Tony: Yeah. We’ll get some nice juicy examples. Coming up next, we’ve got a poll.

Rooz: OK. So as our final poll, who has used a build scan before? Or who is using Develocity? All right, so this poll results are in. A good amount of folks have used build scans. So that’s great news. Well, 31% is “What are build scans”. So we will spend some time at the end maybe to go deeper into build scans. We highly recommend you check it out and we use those tools. As Tony showed to debug cache misses and get the most out of your cache.

Tony: I’m going to be honest here. This used to be two slides. But it turns out it’s hard to talk about detecting cache misses without also talking about debugging cache misses. So I’m going to do that and do some live demos right now and show you what this stuff looks like in practice with real builds. And at the bottom of the screen, you see a variety of links. Again, those are all collected at the end, in one big slide. But in particular, the ones on the left are links to Gradle build tool source code. So you can see what we actually use ourselves for extending build scans. And it’s pretty cool stuff.

It wouldn’t be a Gradle webinar without some command line stuff. So let’s do that. I’m going to run some builds locally and show you what it looks like when there are issues. So let’s run a build. By the way, This is the last time I’m going type Gradle W because it’s too long. And I want to have a better keyboard performance.

Rooz: You have a question here. “You don’t need to be a Gradle Enterprise user to use the build scan?”

Tony: That’s correct.

Rooz: Yeah, Develocity is a collection of your build scans.

Tony: That’s correct. Once again, if you want to try it out right now and just run any Gradle build from the command line at –scan, It will look like that. And you get a build scan.

Rooz: And it does go right to our cloud So it is somewhat public. So if you’re working for a Wall Street bank or something that might not be too happy with you sharing that information…

Tony: But you can delete them. So let’s keep that in mind.

Rooz: Yep. And as far as sensitive information being sent it’s from what I understand, is that the libraries are in the build scan.

Tony: And it just builds better data. Nothing no source code is included. Just data about the build itself. Yeah, we’ll show its library.

Rooz: It’ll show if you’re using some secret library, that you know, you’re working on flying cars and nobody should know right. That might be in there. So be it.

Tony: But as you can see, the URL has this sort of crazy hash. So it’s security through obscurity. So I did the first build and I’ve made no code changes. So the second build should have a pretty high cache hit rate. Let’s do it. And I see in fact, actually eight. So I’m doing one more thing to make a bit more. There we go. So I’m going to put on console=plain just so we can see this in a bit more detail because that build was so fast that it’s hard to see. So what we see is that several tasks had to be executed. This one compileJava, for example, a test was executed and given that I made no code changes, I would have expected these to come from the cache. But they didn’t. So I wanted to why that is. And to dig into that, I can look at the build scan. 

Rooz: And we do this sometimes with our users. We call that a cacheability test. Maybe take a second to talk about that.

Tony: Basically how it works is it’s a no-op build, and a no-op build should do nothing, since if it did something, then something is wrong. So then you use build scans and some other Develocity features to dig into why that might be. So that’s what I’ll do right now. 

You see already there are some interesting things about this build scan. I have a cache miss tag. I’ll show you how that works later. It’s just a custom tag but it’s pretty interesting how it gets computed. And then there’s this custom link called Git Commit scans and what this will do. It’s actually pretty neat. One thing it’s going to show off is that every aspect of the Develocity UI the view is basically specified by the URL. So that means it’s really easy to do custom links that do all kinds of crazy things. And in this case, what I’ve done is that custom link does Git Commit scans. This will show me all of the builds that are on the same Git Commit which can be really useful for debugging cache issues. 

Furthermore, this particularly builds a sort of build comparison. So this is the build I just ran. I can compare it to this one. So a build comparative I also call a build diff. It shows you everything that is different between the two builds. So I go to compare. I now see this Build Comparison view and in particular, we’re focused on the task inputs tab. So we see that there are four tasks that had different inputs between these two builds. And once again, that was unexpected. There it was a no-op build, so why has anything changed whatsoever. But the build comparison tells me exactly what changed. That’s just super useful. 

So for example, we have this create source task. It had a timestamp property and it’s very convenient when build authors use names like that and I can know exactly what’s going on as the timestamp is going to change with every build. But sometimes they’re not quite so nice to us and might be a different name. That makes it harder to understand what’s going on. 

This is a process resources task. I can actually drill down into this. And I can see a lot of details about this task input. So this is a file property with what we call relative path normalization. And to understand this, you need to understand that when Gradle is running a task it’ll normalize all the inputs to every task. Sometimes the normalization is basically nothing,  where it does no work at all and other times it’ll do various things like it will treat the path to that file input as if it were a relative path. Or maybe, as if it were an absolute path or in some cases, the path doesn’t matter at all. And only cares about the name of the file or in other cases, the normalization is just called none, which means it doesn’t even care about the name of the file, the file name can change, only the contents matter.

So in this case, we have relative path normalization to a file named build-info.properties. And so that’s weird. I want to know what happened here that made this task run. So what’s really cool about this is, first of all, you should know that this view only has sort of the hash of the content, not the content itself. So that information is a bit harder to get at. But I can easily just cat this file and see what’s going on. 

Let’s do that. What I see here is that this file is a timestamp and a username. So I wanted to include the username here just to show that it’s not always a timestamp. That’s the problem. So what we can see here is that this file is a problem for two reasons. One is that it’ll change on every single build because the timestamp will always be different. And it also changes on every single machine, not everyone’s name Tony, unfortunately, or maybe, fortunately. So this means at this task, the inputs aren’t relocatable all across machines. So I’ll get more into that later.

So we see that issue, we can look at compile Java task itself. It also has an issue, which is that it has a generatedClass, which has a name only normalization because, in Java, we don’t care about the path the files, we don’t care about the name of the files. And this class has a problem, let’s see what that looks like. So I’m going to copy this to save myself some typing and cat that as well. And here we see the problem once again, is a timestamp. So I know this might seem like a trivial example, and who would do this. But I see this in practice all the time in real-world builds. A great example of this is an Android build, the normalization to the file it was fine. It was as it should have been. But the file contents contain a unique idea or something. And that was just a mistake that someone made years ago. And no one noticed until they had this Gradle tool, the build comparison to dig into it. So this is the thing that you shouldn’t do because it’ll make the build cache work much less effectively.

And then finally, we have one more thing, which is this test task, which again I didn’t want this to run twice. I want it to come from the cache the second time. And I see that it has two things going on. It’s a runtime class pass normalization. There are two kinds of classpath normalization. There are compiled classpath, which is things required to compile the project. And then there is a runtime classpath, which is what’s needed at runtime. This runtime differs from the compile classpath in that it also cares about resources. So again, it’s this build info thing. And I want to make this point a little bit more strongly you might think, why is this file even here. But I’m sure a lot of enterprise Java users understand it. You might need some kind of audit trail, you want to make sure you can audit every build. And so you need some kind of data about that build. That’s what the file is for. But it’s unfortunate that it impacts the cacheability of the task. But we’ll show you how to solve that later. So that is why that exists.

Rooz: OK. It took me a while to kind of get that one right. And you know, you want to build something. And it should be up to date. It should come from the cache. It’s not. So that means some things are changing in these task inputs and we’re using that build diff to find out. Hey, what has changed, where does it live, which file is it basically.

Tony: And then you can correct those really easily.

Rooz: Because it’s like finding a needle in a Haystack once your project gets fairly large.

Tony: Yeah. So what I just showed you was an example of actually running a build and then observing an issue in real-time. I saw that this task was not up to date, didn’t come from the cache. And that confused me as a user. And so I wanted to dig into that. That’s what I just did.

Now the next example I want to show is a bit more proactive. Where we turn the cache on for the first time. And we want to see how cacheable our build is and one great way to do that is what I call the relocatability test. And what that means is for the remote build cache to work well, task inputs must be quote-unquote relocatable. So they have to be able to locate from one machine to another. And that should still be valid. It should be agnostic as to where it ran.

Rooz: And so this is the test that when we’re doing Develocity trials and proof of concept, this is the test that we run as one of the early parts of the process to see kind of what’s going on.

Tony: Exactly And so the way that works is I will simply want to make a copy. This is like a sort of like a cheap way of doing it. So I’m going to copy that whole project to a new directory and go back into the original directory and run the tasks. So Gradle clean check. OK, so that would populate the cache, that’s how the test works basically. So you run the task, you wanted to check. And then you changed to the directory of the copy and then I’ll run the same task again. And what I see is that a number of tasks had to be executed more than I expected. And so to dig into that. 

Well, actually, I think I ran the wrong test. So I want to clear the cache locally. And that’s just an alias by the way for basically just moving some cache artifacts locally. So I ran the wrong task. So it’s Gradle clean. I have this task name, again it’s really helpful when people give tasks such useful names. OK, there we go. Let’s run that again. OK So here we see that way more tasks were executed here than I expected. I thought this would have been all coming from the cache because what I’m trying to get at here is that you have the project in two different directories. So there are different absolute paths to the project. So what this test will do, it’ll flag any issues with absolute paths to your task inputs.

Let me do another one of these things where I was going to look at a build comparison between these two things. And then what I’ll see in this task inputs comparison, I see that there is a file called config.properties with absolute path normalization. And so this is really common, absolute path normalization actually is the default for historical reasons. And so if you’re creating a custom task and you don’t specify the path normalization for that task. It’s going to by default pick absolute path, which means that the absolute path of the property matters. And so obviously, if you have a project in two different directories on my local machine, they’ll have different absolute paths right. So that means the inputs of that task are no longer relocatable and if it breaks locally like that it’ll also break remotely. So it’s going to have an issue with the remote cache as well.

So that’s a really easy thing to fix. Let’s go into. This is my IntelliJ. I’m going to navigate to that task. I see there’s a task called unrelocated task and someone commented that off for some silly reason, so let’s uncomment that. Now we know it’s a config file. So there’s no need to have any note, not an absolute path, not a relative path. It’s just a no sensitivity whatsoever to that file. And now we’re going to do is I’m going to just remove that copy. I am going to copy it again, and go back into the original project directory. And I’m going to clear my cache one more time just to make sure it’s clean for this task. And then I am going to run the task. So it’s really clean on top of making it a little bit faster. So it is on task, there we go. Let’s go into the copy directory run it one more time. And now we see that it came from the cache.

So let’s see what it looks like in practice once again and do a build comparison. Here we go. The inputs are identical so that fixed the problem. So that was a really straightforward thing to do. And once again, that was a trivial example for demonstration purposes, but it’s really common to see that in practice 

There was one other example I wanted to show here, which is what we can do to sort of again proactively flag issues for later debugging. So you can instrument these builds in a way. So let’s see what that looks like. See what I mean by that. I’m going to go to once again, the performance dashboard. And I’m going to refresh it. So these are all my builds and many of them are very fast because they’re just help tasks. I’ve got a custom tag called cache miss. Here they are. I’ll show you the source code for this. So again, this is my proposal for my build.Gradle file. I have a lot of interesting stuff in here, by the way, this is a great example of ways to torture Gradle. I have many bad practices in this file just for demo purposes. We’ll share this go with you guys later. So you can see them how you can make Gradle unhappy.

Rooz: But we have some Gradle gurus on this webinar. They might just call you out on this Tony.

Tony: Yeah, it’s fun making Gradle suffer. So the way to do this kind of custom tagging is basically the sort of Gradle taskGraph thing. I’m not going to go into detail on this because we are kind of running low on time, but this demonstrates how easy it is to add custom tags based on criteria you define to your real build scans. 

So what I’ve done here is I basically said build cache is on. There’s a cache miss that I didn’t expect. And I haven’t tagged it yet. So it’s tagged with cache miss. So that’s what we see with. These are all tasks that I expected to come from the cache. But didn’t. And I can even demonstrate this as well. It’s a really neat thing you can do or you can sort of negate certain criteria. So I want to look at only not:dirty builds. So when I say this, what I’m saying is if you look at some of these scans they have a tag called dirty which indicates that the git status is dirty, and I don’t want to look at those. I want only those tasks that have no dirty status, which just kind of a cool thing that you can do. And then what I would do is I would actually just do a build comparison. So I can just compare these two builds. And again, it’s the same view you saw earlier. There’s a lot of reasons why this build wasn’t cacheable because of all the various pathologies introduced for demo purposes, but it’s neat that you can actually have a custom tag like that to help you. Like every week or so you can go back and look at these issues and resolve problems in a more proactive fashion.

Rooz: So cache misses. So what do we need to do here to get more and debug these?

Tony: Yeah, sure. So I showed you some of the issues. I already showed you one way to resolve an issue, which is changing the path normalization to a certain task inputs and there’s one other thing I haven’t shown yet, which is this idea of one-time classpath normalization which is a pretty cool tool. So let’s see this in practice.

I’m going to run some builds.  Again, “g” is just an alias for Gradle W. So I’m going to run this twice, do a comparison again. And I mentioned this earlier. But you see the test task has this runtime classpath normalization and it points to this building photo properties. Now I want this file to exist. I want it to be there for auditing purposes. But I don’t want it to impact the success or failure of a test because it doesn’t have any impact on that. So I don’t want to make the test not cacheable. So what I’m going to do is, use my super-secret flag I introduced and you know if only this were something as part of the Gradle core,  we could tell our tasks to be stable. But it’s something I’ve added just for demo purposes. I do this again, do it twice. So the second time it should come from the cache to do that comparison OK. And here we see that this no longer shows the issue with the test task not coming from the cache. Because you don’t see it on the screen, it means, the test task has the same inputs for both builds. And so what we’ve done if we have a look at it here. You can even see it. It’s over here. Testing came from the cache this time.

So the way I achieved that was by using runtime classpath normalization. And so what you would do is you’d have this block of code here. So we basically just told Gradle that build-info.properties is something that can be ignored for this kind of test purposes. Obviously, I have it wrapped because I want to have fun with Gradle.

So there’s only one more thing to show you guys, I brought this up earlier in the presentation. If you have distributed teams this can be very interesting for you if you want to be able to have multiple nodes and you’ve got team members in Europe, and in Asia, and in America. And obviously, if you have just a single remote cache node in the US because we’re limited by the speed of light, that cache hit and will not perform as well for the European as it will for the Americans. And so to resolve that issue Develocity supports this idea of having a sort of cache node network. And we can replicate data between these multiple cache nodes.

So I want to show how easy it is to set one of those things up. It will only take a moment, there we go. So it’s super easy to set up just a docker command. That’s all it takes to set up a remote cache node. Now I can verify that it works by going to the localhost. OK. So this is the cache node UI. You would connect to the Develocity by editing these fields. And you can see how it’s obviously empty, this is a new cache. You can figure it in various ways. So it’s super easy to set up a remote cache node just simple to docker command and then, of course, if you take it down, you can just docker remove or whatever.

Rooz: One question I have for you, Tony. So what if you’ve enabled this remote cache and Bo from our team is at Starbucks and he’s running this massive build and it’s slow. How would we know his build is slow because his network sucks from Starbucks in Asia?

Tony: Sure. Well, an easy way to see what that looks like is just to go on the performance dashboard screen. So here I would see that, for example, because there’s no free lunch right. Everything has some cost the build cache for the Gradle build is actually pretty low. You know we’re spending very little time uploading and downloading and zipping and unzipping our cache artifacts. So this is how you would actually keep an eye on the overhead of the cache and the sort of the networks.

Rooz: And in an actual build scan you could see that as well, right?

Tony: Yeah. So I could pick one that has a high value for example. Navigate to It, go the Performance tab, then build cache. Now we see all the aspects of that. Like how long things took. How many objects were packed and unpacked for example? The network speed as well. Yeah, right here 5.

Rooz: Right. So that’s where you would determine, hey, maybe I shouldn’t use this remote cache from Starbucks in my local neighborhood or something.

Tony: Yeah, you could keep an eye on those issues like that.

Rooz: All right. Should we just jump into the questions? 

Tony: Ok let’s jump into it.

Rooz: All right, let’s do it. “Is Python supported for build cache? Some explanation.”

Tony: Well, there are several Python plugins for running Gradle builds

Rooz: There’s a Python Gradle plugin that LinkedIn folks open-sourced.

Tony: Sure so I guess as long as the plugin is well-written and the tasks are well written then there should be no problem with caching those tasks.

Rooz: All right. So the person who asked that feel free to email us. Yeah, or us send us a build scan and we’ll take a look at it.

Tony: Yeah, in fact, that’s a good point Rooz. If there were any problems that you have, the first thing you’ll hear Gradler say to you is, send me a build scan because it is our number one tool for diagnosing these kinds of issues.

Rooz: All right. OK, let’s jump into this one. “So the reliability of the cache depends on how well we define input-output tests? Ill-defined tasks may create a debugging hell with the cache producing fake false positives.” What do you think?

Tony: So that can happen if you have an ill-defined task. It’s not something I see too often in practice. I think the question you’re asking is, is it possible to sort of poison the cache? And if you want to be really deliberate about it, of course, you can do anything you like in the world. But that is something I don’t see too often in practice. Like all of the Gradle core task core plugins are very well-defined. If you’re actually a plug-in author and you want to ensure that that isn’t happening to you, you want to make sure that you apply in your plug-in, the Java Gradle plugin plugin. And what that will do is adds a task to the check task which verifies all your tasks are well defined. So if you’re plugin author definitely make sure you’re using that plugin for your plugin development work and that’ll ensure that your tasks aren’t poorly defined 

Rooz: Got it. All right, and if that wasn’t clear you know send us a follow-up. Alright, next one. “How can we enable Gradle cache on docker containers?” Did we already asked that?

Tony: Yeah, It just works. Doesn’t matter where it’s run.

Rooz: Yeah. And there is a tutorial I’m sure somewhere.

Tony: Yeah In fact, I can show it to you guys. This is all it takes to enable the cache. That’s it. Gradle properties, just turn that on and you’re good to go.

Rooz: So here’s one with our Develocity question. “In Develocity, how can I see the build cache activities from build-cache admin?” This person specifically wants to see details of the latest cache events, have IP but it has the same IP address. I guess it’s from F5 for all requests or something.

Tony: So what we have here is, Develocity has two aspects to it. There’s the build scans which I’ve already shown you a lot of information about and there’s also build-cache administration. So from here, we can see how the build cache is being used. And so in the past 24 hours, we see that the cache hit rate is very high for the built-in cache. We even see all the other cache nodes that our team has available to us. California, Frankfurt  Germany, Sydney Australia. And so this is one useful thing to look at to see how well the cache is performing over time. 24 hours, seven days, 30 days.

We can look at the various nodes attached to our cache and to our Develocity instance and including details about these nodes. So I can see what the cache size is, maximum artifact size. All these interesting things I can even see. Latest cache events. So I know what’s being evicted what was a miss, what things have been copied, that sort of thing. So it’s a huge amount of detail you can get.

If you really want to get down into the weeds on this. There’s even like a health monitoring section. Yeah, and if you’re interested in a particular artifact you can actually take the copy this key. Plug it into entries and see what its status is.

Rooz: Alright.

Tony: What do you got next?

Rooz: Alright. “I am also building JavaScript NPM when I remove Pax just from the build. Will they also be removed by the cache component, or do they stay in the cache pollution until I run Gradle clean in my CI build?” Interesting question.

Tony: I think it gets back to our earlier slide about sort of the difference between the binary repository and the build cache. However, I think that with NPM we actually have an example on the Gradle docs about how to cache NPM tasks. And it won’t just sort of hang around forever. 

One thing that happens with the build cache is if a cache artifact that you saw here under the nodes use, a sort of all the activity on that cache. And what you’ll see over time is things get evicted right. There’s sort of the least recently used aspect to this. So when a thing hasn’t been used for a while it will get evicted from the cache. It won’t it won’t stay around polluting you forever. It will go away in your cache size. And you can specify the maximum size you want for your build cache. So right here and they’ll keep making sure your cache Isn’t growing out of bounds.

Rooz: Here’s a question about tagging build scans. “How can a tag for each build scan with a unique ID?”

Tony: Oh, sure you could do that. So this is my build.Gradle for these examples I’ve been showing. There are various tags I have, so for example, I’m tagging the branch name. I’m tagging that the status is dirty. I have already showed you the tag for the build cache miss. If you want to tag someone with a unique ID, you can do that as well. You could easily tag with the commit ID, so I could just add that down here, for example. Now what we’ll see is if I run this build, is that this build has that git commit part of one of its tags. So you can easily tag someone with a unique ID like a git commit. Very simple.

Rooz: OK. This is a question about what command used in one of your command line demos here.  “Wondering what’s the command line argument you typed for seeing verbose output console dash dash please.”

Tony: So if I want to run any given thing, normally you see this sort of rich view where it just has this. It’s very attractive, but if you want to see all the tasks in a bit more detail you just dash dash console equals plain and there are a variety of arguments you can supply, not just plain. There is rich, there is auto. So that’s how that works. And if you just Google, “Gradle command line parameters” or something you’ll find all this documentation is on our website.

Rooz: “Do you have a tutorial to enable Gradle cache on an Android project?” Is it different from Android and Java as far as enabling?

Tony: No, as far as enabling there’s no difference whatsoever. There’s a bit of difference in how it’s used because the Android plugin is different in Java but there’s no difference in enabling it’s the exact same thing as I showed you earlier. It’s just as simple as this.

Rooz: OK. Easy. “Do you use a femoral build agent or is there a more efficient way to take advantage of the Gradle Daemon? (Up to date checks, local build-cache, et cetera, and CI specifically)”

Tony: So at Gradle, we do not use the ephemeral agents. We actually have persistent CI infrastructure. We have a number of agents, dozens of them.

Rooz: And dumb question. I haven’t read code. What are ephemeral agents?

Tony: Oh, they’re just things that sort of they spin up for a build and then they spin back down when the builds over. So they’re like mayflies, you know they exist for a moment in time that is gone and has benefits for the sort of immutable infrastructure idea. But it does have a performance penalty as we all know. So we don’t use that ourselves at Gradle.

Rooz: “We use the OWASP dependency check plug-in which checks dependencies against natural vulnerability database to fail builds that contain security vulnerabilities. By using the Gradle cache can skip this task because there’s no actual code change. However, a vulnerability in an existing dependency may have been discovered and added to the remote MVD database. So we, therefore, don’t fail the build when we should. Is there a good way to handle this?”

Tony: Absolutely. So if you want a task to always run what you would do is you get a reference to a task. You would say task up to date when false, done. Now the test will never be up to date. It will always run.

Rooz: All right. Easy.

Tony: But you don’t want to abuse that because you don’t want to just do that Willy nilly at all, and make your builds all slower.

Rooz: “Is Develocity required to use a remote build cache?”

Tony: For projects that use Gradle it is not required to use Develocity to use it for the remote build cache. You can, in fact, set up your own remote build cache node and just use it. The problem, of course, is that you have no monitoring then, it’s hard to see how the cache is performing and then to verify issues with it. So you can do it.

Rooz: “Which part of the system decides, does it make sense to cache query cache or not? Speaking of those tasks that should not be cached.”

Tony: So you want to know which tasks should be cached?  Is that the question?

Rooz: So does it make sense to cache/query cache or not? Which part of this system decides that?

Tony: Oh, so you as the build author or the plugin author decide which tasks to make cacheable. So to make this really concrete for you. I’ve got some custom tasks in my build script and I have to tell Gradle that they’re cacheable. If I don’t have this annotation if that’s just not there, then the task will never be ever used in the cache. But in addition to having this annotation, you have to specify the outputs and inputs and then and then it just works. But yeah, you have to tell Gradle. It’s an opt-in, not an opt-out.

Rooz: But some tasks are cacheable right out of the box though.

Tony: Exactly yeah. If for some reason you didn’t want a task to be cached similar to this thing, here you’d say cacheIf{false}. But I again, don’t overuse that. It’s going to cost you.

Rooz: “So does it make sense to have CI builds running build all the time to have cache always warm?”

Tony: Yes, we do that ourselves. I also showed you earlier and I’ll show it again. If you go to the performance dashboard for Develocity for the Gradle organization. If I just look at the past week. There we go. We see that just in the past week, there’s about 10,000 builds here. So we are running Gradle builds all the time to make sure our cache is warm. Every single build has that sort of seed task in it.

Rooz: “What are reasonable allocations of build-cache overhead?”

Tony: Oh so this value right here. That’s a good question. We don’t have any sort of hard number for what makes sense. I’ve seen this be much higher in the case where there is a team that had just a single cache node in the US. But they had developers in Europe. The download in time was several seconds and that was unfortunate, but they know what the solution to that is, which is just to add a second cache node near their European developers.

One thing you should keep in mind, which is not obvious to many people who first see the performance dashboard is that this overhead number, you do not need to do manual math to figure out the true benefit of the cache. This remote build cache value here, 56 seconds, already accounts for these four seconds of overhead so you don’t need to do the math yourself. So this is still a benefit. Without having to account for the four seconds of overhead.

Rooz: “What is your idea font?” Alright, we have somebody who likes your idea font here.

Tony: This is fever code. I like it for the font ligatures.

Rooz: “How would we use the build scan to debug while the clean assemble works while an assembled doesn’t?”

Tony: So that would be a situation where I would again, I would use the build comparison. 

Rooz: Which is a Develocity feature. So you could use the trial of it. There’s a 30-day free trial that you can do if you needed to.

Tony: Yeah. So what you do is this build comparison feature is for more than just for cache issues. If you have a failed build and successful build. This is a fantastic way to see what makes the two builds different from each other. And then you can from that point reason about what may have caused the one build to fail. It’s a really common scenario that we see all the time.

Rooz: And I think one of the main reasons we made that is because we got so many questions saying, hey this thing passed on CI and it’s failing on my machine.

Tony: Absolutely 

Rooz: A colleague ran this build and it passed, and it’s failing when I run it. What’s going on?

Tony: And one issue there is if you don’t have Develocity. And you need to debug a CI failure. That’s going to be a real big pain in the butt right. You’ve to run a clean build. It could take half an hour, depending on how complex the build is or an hour or who knows.

Rooz: Just to reproduce it?

Tony: Yeah, just to reproduce it and then good luck if it’s a race condition, but with build scans, we have a persistent record of every build in your organization, which makes it dead simple to see what’s going on.

Rooz: So you can find that crazy, weird anomaly failure that happened three months ago that Frederick ran.

Tony: Yeah, and you can even filter this view to show only failed builds for example. So we can do that now.

Rooz: And we can even do that per user let’s not embarrass anybody right now. Let’s see here. What else? “How would you put your Maven repo and your build cache to their best uses?” Interesting question.

Tony: Well, I think this whole webinar was about that, right, about how to make the best of the build cache. If you have a more specific question, feel free to ask.

Rooz: So the way you showed us, to recap, you’re looking at the cache misses. Right, you’re looking at cache miss percentage. You’re looking at the tasks., and the task execution of the build scan and see what’s what. So those are the three ways. You know you kind of approach.

So my question to you. I’m going to throw one in here for you, Tony. So what is your favorite? What is your go-to? Right, you have a build scan and the cache hit rate is low. What’s your go-to? What’s the first thing you do?

Tony: Yeah. So that’s a great question.

Rooz: I mean, I maybe show us one can we find one of your builds in here and see how long your builds take.

Tony: We’re not going to do that.

Rooz: All right. OK, well fair enough there.

Tony: But you first thing I do, absolutely. So if we go to the cache examples project again to simplify the view I do really love this custom one that I have added to my build scans which lets me see all the other build on the same commit. And then it populates a comparison for me already.

Rooz: So hold on, why are you doing that? What’s the main motivation?

Tony: So the idea is that if, let’s say that a first build of the day, I get to work in the morning. And I know that the exact commit that  I just build on my computer was already built on CI and like the cache rate does not look like, what it should be. So this is precisely what I’m going to do. And I go to the build scan, create a comparison against the CI build. And then see what the differences were in the inputs, and then from there, I can mix all kinds of judgments about how to improve that.

It’s something that I do a lot with some of the Android plugin in and not to call them out or anything, but I have found a number of bug reports against the Android plugin and over the past few months to help improve the cacheability of that project. And it’s going to be for any Android developers out there. I think you should expect big things from the build cache for Android builds over this summer.

Rooz: And the caching of transform tasks, that’s in there?

Tony: Yeah it’ll be, I think it’s coming in 3.5. Yeah, our plug-in 3.5.

Rooz: Yeah that’s a big one.

Tony: Yeah, it will be caching those transforms, saving a lot of time.

Rooz: “What’s the difference between Android and Java plug-in from Develocity perspective?”

Tony: Again, Develocity and build scans don’t care what kind of plugins you have applied. It’ll show you all the plugins that have been applied. Like if we look here, we can see all the plugins applied to a build and a build scan. Like here we can see that it has build scan plug-in applied, it has the Java plugin applied. For an Android build, it would have a very similar look it would also show you the Android plugins that are applied. Like the com.android.library, com.andoid.application and the feature plugins as well. So we would see them here. That’s the only difference really.

Rooz: All right, another question. “Android Gradle plug-in had a build cache functionality separate from Gradle’s build-cache feature. Is this still relevant and how does this conflict with Gradle’s build-cache at all?”

Tony: Right. So there’s no conflict whatsoever. The Android build-cache stuff is complementary to the Gradle build cache. The way that works is its sort of like a workaround for the Android team for issues in their transform tasks. It’s sort of caches things within the task as opposed to Gradle caches tasks themselves. That Android task is for within a specific kind of task to help avoid some work. And that will actually be removed eventually in favor of the Gradle build cache like as a sole solution probably sometime this year.

Rooz: All right.

Tony: But yeah they’re complementary. They don’t impact, they don’t conflict at all. Just to be clear.

Rooz: I almost forgot this one. So one of our Gradle gurus, old friends of Gradle design here and he disagrees with you on some areas when you should not, well, in cases of disabling the cache. So he’s saying, “Hey rebuilding when there are environmental or system dependencies that are not properly captured with caching.”

Tony: Right. So every one of you knows your build better than I do. So if you know for a fact that your build has certain things that aren’t captured well by the build as it currently stands then you’re right. I’m not right about that. But what that means is you can probably make some improvements to your build to capture that information as properly as proper inputs to your tasks and then maybe eventually, once you’ve gotten to that point, you can turn on caching. But yeah, I’m not going to tell anyone that I know their build better than they do.

Rooz: And so he continues, “Proving that fresh baseline builds can work from base dependencies and source code schedule build that validates that the build environment is stable and works.”

Tony: What’s the question exactly?

Rooz: So it’s another time where you should not use the cache. Right, this is a response to cache that should always be on the. Right, in bold that you had put on that slide right. Yeah, it is a bold assertion. So proving that fresh baseline builds can work from the base dependency and source code.

Tony: OK sure if that’s important to you. An easy way to do that on an ad hoc way is just saying Gradle check, you know no build cache. And there you go. So if you want to have the cache on for most circumstances. But for every now and then run a build without the build cache, you can just do that. And that’ll solve that problem.

Rooz: What about third-party tests? One of the things you mentioned is those can be problematic because you don’t know who’s written those. The inputs and outputs and whatnot, everything would be all over the place.

Tony: Yeah. So I showed this earlier, which is that caching is an opt-in not an opt-out. You have to annotate your task with a cacheable task for it to be a cacheable task. If a third party plugin author has done this. It’s sort of on them and also it’s on you. If you are the build maintainer and you are just adding someone’s plug-in without checking out what it is. I mean, you can file bugs against their repo saying, hey, you should fix this there’s an issue with your tasks. But you know, you shouldn’t be adding libraries to your project, not knowing what they are. We’ve seen issues with that in the past where people just adding any old thing. So you know you should vet your third party dependencies.

Rooz: “What about for using Gradle in deployment scenarios where just caching doesn’t make sense at all?” Is that something that folks should worry about.

Tony: So we use the cache as you might expect religiously even for deployments. And we do have a very large customers who use it toward deployments, but for some people, it’s a step too far to deploy a build that’s been cached. And so that’s fine. If that makes you uncomfortable. There’s nothing to say that you have to use the cache in that case, you know run most of your CI builds with the cache enabled. All of your developer builds with the cache enabled that’s already saving you a ton of time. But for that critical time when you’re going to deploy your final binary just to run it without the cache. If that makes you feel more comfortable, you totally should do that.

Rooz: Yeah, I got another one for you, myself personally. A hard one. I don’t know if you’re ready for this. How do we know if someone is not using the cache? Let’s say someone is you know, for whatever reason, reliability, I don’t trust that, I don’t want to use, I just love running clean, and I just like typing those words. How would we know if someone is not using the cache?

Tony: OK 

Rooz: Without checking individual build scans.

Tony: There are a couple of things you can do. If you look in the source code, which I will make available to people after the talk. So one thing you can do is this is how you know the cache is enabled for a build. So what I might do is say is cache enable and then tag if not cache enabled, build scan tag no-cache. So it’s as simple as something like that, if I want to track whether builds have the cache turned off I’ll just check this and I’ll show you this in action. Let’s run a build with the cache off. Of course, it’s a failure if I type something wrong. I’ll get rid of this for now.

Rooz: I didn’t say it was going to be easy, but so far you’re making pretty, you’re making live demos. It’s the best.

Tony: All right. So here we see no-cache. So now what I could do is I could go to my performance dashboard or my scan list and just search for no cache. Well, ah that Tony guy, he’s just running builds without cache.

Rooz: But you needed to have this kind of setup before. So if you wanted to kind of cache this.

Tony: So that’s for caching if you want also check on clean issues like we shouldn’t be running clean too often right, guys. So we can already search for people. And we already have that in the system. These are all the builds that have clean turned on. And I could easily see who’s running these clean builds. It’s that TCAgent guy on CI.

Rooz: Yeah.  So you can query clean for Java projects, right? So for Android projects, we query assemble debug. This is the problematic task that everyone wants to improve the build times. It’s a great representation of local builds. But what about for the developer builds in Java. What are some cool things to search for and filter in our dashboard? Right. Hey, I want to improve local developer build times for a particular project. What are some interesting things to filter for? To kind of really narrow this thing, because you have incremental builds right. So the average build time is all over the place. Right, so what do you do Tony? And it’s a hard one I know. We’re still trying to figure it out. But I’m wondering what you do here.

Tony: So here I have filtered the performance dashboard just so local builds. And again, this is a custom tag easy to set up with these custom tags. I’ve shown several times now. So I’m looking at just local builds in the past day. If I’m concerned about developer productivity you know I can look at only failed builds, like look at only the slowest of builds and then I would dig into that. See this looks like a sanity check build. So you can see a lot of interesting data from the build scan to really help you figure out what’s going on in any given building. So I know a lot of Mac would talk cores cause all these flags are turned on.

Rooz: And there’s a way we could see who has the slowest builds in our organization. Let’s not do that online. But like we could do that?

Tony: Yes, we could do that. Yeah, well, I mean, the simple way of doing that is just to look at the reference dashboard. I take away this filter and look at the past week.

Rooz: That’s what we used to catch build time regressions as well. Right, hey, something’s going on. How do we help this person? It’s not a spying mechanism. But it’s more of, hey, let’s one or a few folks make everybody’s builds fast kind of a thing.

Tony: Yes. So we already see this couple of spikes here and if wanted to I could dig into that and see what’s going on.

Rooz: All right. Here’s another one from our one of our friends, “Gradle glory here. I’ve run into a few issues with custom init scripts and setting runtime parameters like caching. What are your best practices there?”

Tony: Yeah. So we do recommend using init scripts for setting up things like that across all builds on your local machine. As far as best practices I’d say that’s a good thing to do. I’d like to know what the specific issues really are so I could really answer the question in more detail.

Rooz: We can message them about that. “Would you recommend caching results of shadow jar tasks?”

Tony: In general, we don’t cache jar tasks for the reasons specified. It’s just moving stuff around, which is just as fast, if not faster, than pulling from the cache, which is, again, just moving stuff around.

Rooz: “What about NPM install node modules folder?”

Tony: We actually have an example, in our documentation of how to cache the NPM tasks. So it’s something you could totally do to save some time.

Rooz: “By default, the local cache is maintained in user/username.gradle. If the location is moved to network drive instead, would it be safe for multiple developers to use it concurrently?”

Tony: So for sure the way that the local cache works, in general, is that its sort of like first win. So I’ll show you what it looks like. This is my Gradle cache. If I just list them, you see they’re just these hashes right. So assuming that if you believe that our hashing strategy is good. Which I do. What that means is that for there to really be a collision here, there has to be identical inputs. And in that case, it doesn’t matter which one wins, if the first one wins that’s fine. And it would be completely OK to share this build cache directory across the whole organization as we did with a remote cache. This is essentially what it is already.

Rooz: “What are the hardware VM specs to host Gradle cache?”

Tony: So this is our manuals that show the specs for Develocity itself. But for the build cache, you would just go to this page and it shows all the information you need for that. I’m trying to find the right link.

Rooz: It’s probably in the “before you install” maybe.

Tony: Yeah but it’s basically here somewhere. We have all those specs publicized in our documentation.

Rooz: So documentation in general, this is where you would send folks?

Tony: Yeah, you’d send folks here. 

Rooz: So Gradle.com docs.

Tony: Yeah, there is a plug-in user manual.

Rooz: “Is there a difference between Artifactory build cache and Develocity build cache?” And this person is referring to not the repository in a binary repository cache. Do we have any data on that?

Tony: We don’t have any good data right now and how they differ. We believe based on some ad hoc testing that ours is faster. But I don’t have any sort of information to really prove that point right now, unfortunately.

Rooz: I think that’s all we have. We don’t want you to miss your flight so we should cut it off soon. Thanks, everyone for attending. Email us if you have any other questions. And anything else would you like to say, Tony?

Tony: Thanks for attending everyone. It’s been a good time.

Rooz: All right. Great Thanks. Have a good one.