We run our CI on Jenkins. The below image is a decent explanation of the whole setup (non-relevant steps are removed for the sake of simplicity).
To summarize, our overall process:
- We have a master and a beefier agent machine (both VMs running on the cloud) running all our CI.
- Currently, we have 26 git repos that are managed by this system.
- All our CI jobs are multi-branch Jenkins project.
- All repos are Gradle projects.
Jenkins details: 1. All CI jobs run on Jenkins agent. 2. The Jenkins agent is configured with 4 executors.
The problem:
Multiple Gradle builds running in parallel step on each other. All Gradle builds use the same grade global cache ( /var/lib/jenkins/.gradle/caches ) and while one executor is trying to compile, the other executor runs clean which does :
rm -rf /var/lib/jenkins/.gradle/caches
This results in problems like:
- Compilation failed
- Class not found
- Could not resolve all artifacts for configuration ‘:classpath’.
etc.
To bypass this, I have disabled build caching. This has definitely stabilized the builds, but now it is taking around 40% longer to complete a CI job. So, I have to enable back caching as soon as possible.
Question:
What is the optimal way out of it? I’m considering configuring separate global cache per project (in the root of the project) but have not tested it yet. Do you have any other suggestions?
Any pointer is highly appreciated. Thanks in advance.