Full compileTestJava run when only one test class changed

Hej,

I am using Gradle 6.5.1 on a single java project.

A clean compilation of the test folder for this project takes about 8 minutes.
If I change a single test class that is not used by other test classes Gradle starts compiling again the whole test folder and it takes again 8 minutes.

My expectation is that Gradle would only need to compile this file and not the whole folder.

I tried compiling with and without build-cache and I don’t see any difference in the behaviour.
All the previous tasks are marked as up-to-date by Gradle.

Here is the info output of Gradle.

....
Task :classes UP-TO-DATE
Skipping task ':classes' as it has no actions.
:classes (Thread[Execution worker for ':',5,main]) completed. Took 0.0 secs.
:compileTestJava (Thread[Execution worker for ':',5,main]) started.

Task :compileTestJava
Cache entries evicted. In-memory cache of ....gradle/caches/journal-1/file-access.bin: Size{400} MaxSize{400}, CacheStats{hitCount=0, missCount=0, loadSuccessCount=0, loadExceptionCount=0, totalLoadTime=0, evictionCount=521}
Performance may suffer from in-memory cache misses. Increase max heap size of Gradle build process to reduce cache misses.
Excluding []
Excluding []
Custom actions are attached to task ':compileTestJava'.
Build cache key for task ':compileTestJava' is c5d9263873a9750b37aef521e4d0b615
Task ':compileTestJava' is not up-to-date because:
  Input property 'stableSources' file <the class name of the changed file>.java has changed.
Created classpath snapshot for incremental compilation in 0.015 secs. 640 duplicate classes found in classpath (see all with --debug).
Class dependency analysis for incremental compilation took 0.03 secs.
Compiling with JDK Java compiler API.

There is this line Performance may suffer from in-memory cache... and I tried to compile with GRADLE_OPTS=-Xmx1G and GRADLE_OPTS=-Xmx512m. I also checked that the daemons have applied the memory settings via ps ax | grep gradle. But it makes no difference and the same message was coming up with these settings. It always says MaxSize{400}.

• Is my expectation right that the change to this one file only would not need a full recompile of the test folder?
• Is this message about max heap changing relevant and how can I avoid his message?
• How can I speed up the compile time for one changed test class?
  Because in a scenario where I would like to run certain tests on a change with -t it takes now 8 minutes after every change.

Thanks for any help in advance.