Make PlatformScalaCompile Cacheable. Issue:#3648

[devishree90]

Context

Issue: #3648
This change will make the tasks like compilePlayBinaryScala as Cacheable. (I'm working on play project which uses gradle as build system. we are trying to levarage Build caching to improve build time. Currently, most of the scala tasks depend on 'compilePlayBinaryScala' and 'compilePlayBinaryScala' not being cacheable triggers cascading non-cacheability on other tasks as well)

Please suggest recommended tests for this change.

Contributor Checklist

• Review Contribution Guidelines
• Make sure that all commmits are signed off to indicate that you agree to the terms of Developer Certificate of Origin.
• Check "Allow edit from maintainers" option in pull request so that additional changes can be pushed by Gradle team
• Provide integration tests (under <subproject>/src/integTest) to verify changes from a user perspective
• Provide unit tests (under <subproject>/src/test) to verify logic
• Update User Guide, DSL Reference, and Javadoc for public-facing changes
• Ensure that tests pass locally: ./gradlew <changed-subproject>:check

Gradle Core Team Checklist

• Verify design and implementation
• Verify test coverage and CI build status
• Verify documentation
• Recognize contributor in release notes

[wolfs]

Hi @devishree90, thank you for your PR!

Regarding test coverage: I would expect this change to have the same coverage as we do for the ScalaCompile task which is created by the scala plugin. The following tests come to mind:

• ScalaCompileRelocationIntegrationTest
• CachedScalaCompileIntegrationTest

Since you are especially interested in making compilePlayBinaryScala cacheable, I would suggest to add tests for this scenario. There should be tests to cover the scenarios described in the build cache guide.

[devishree90]

@wolfs - I have added tests as suggested. I have a failing test which i need help with. Scenario which verifies Caching could be used from two different directories is failing in this commit. I suspect this could be because of the directory structure which i'm creating/using. (I compared tmp/CompilePlayBinaryScala.analysis in both folders and only difference i see is directory path.
). Could you help me investigate this failure?

[eriwen]

@wolfs or @lptr can I assign this to you?

[devishree90]

@wolfs @lptr @eriwen - Is anyone working on this PR?

[lptr]

There's been some holidays, but the team is back now. We'll take a look at this PR this week.

[devishree90]

Thank you @lptr

[wolfs]

Thanks for the update on the PR!
Awesome, now we have some tests!
The tests already show a problem: source in PlatformScalaCompile requires the correct relocatability annotations. Moreover, the Java version should also be tracked when compiling Scala sources.

[wolfs]

Do we need the ivy and maven lightbend repositories here? I think the maven ones should be enough, right?

[wolfs]

This probably needs to be indented 4 spaces more. I would leave out the .stripIndent(), since it only makes the test a little bit harder to read.
I prefer indenting multiline strings like this:

    def multiLineString = """
        This is the text
        which goes over multiple lines
    """

[wolfs]

❌ As seen by the failing compilation is cached if the build executed from a different directory, it is not enough to only add @CacheableTask to PlatformScalaCompile. It is also necessary to add the override of getSource with the correct path sensitivity:

gradle/subprojects/scala/src/main/java/org/gradle/api/tasks/scala/ScalaCompile.java

Lines 70 to 77 in e6b2c90

	/**
	* {@inheritDoc}
	*/
	@Override
	@PathSensitive(PathSensitivity.NAME_ONLY)
	public FileTree getSource() {
	return super.getSource();
	}

Moreover, the Java version should be tracked for Scala compilation, too. For ScalaCompile, this is done here:

gradle/subprojects/scala/src/main/java/org/gradle/api/tasks/scala/ScalaCompile.java

Lines 95 to 106 in e6b2c90

	/**
	* The Java major version of the JVM the Scala compiler is running on.
	*
	* @since 4.1
	*/
	@Incubating
	@Input
	// We track this as an input since the Scala compiler output may depend on it.
	// TODO: This should be replaced by a property in the Scala toolchain as soon as we model these.
	protected String getJvmVersion() {
	return JavaVersion.current().getMajorVersion();
	}

We should add a test for PlatformScalaCompile, too, similar to this:

gradle/subprojects/scala/src/integTest/groovy/org/gradle/scala/compile/UpToDateScalaCompileIntegrationTest.groovy

Lines 66 to 89 in c8f6881

	def "compile is out of date when changing the java version"() {
	def jdk7 = AvailableJavaHomes.getJdk(VERSION_1_7)
	def jdk8 = AvailableJavaHomes.getJdk(VERSION_1_8)
	buildScript(scalaProjectBuildScript('0.3.13', '2.11.8'))
	when:
	executer.withJavaHome(jdk7.javaHome)
	run 'compileScala'
	then:
	executedAndNotSkipped(':compileScala')
	when:
	executer.withJavaHome(jdk7.javaHome)
	run 'compileScala'
	then:
	skipped ':compileScala'
	when:
	executer.withJavaHome(jdk8.javaHome)
	run 'compileScala', '--info'
	then:
	executedAndNotSkipped(':compileScala')
	}

I suggest adding both to AbstractScalaCompile.

[devishree90]

Updated as suggested

[wolfs]

❌ This test fixture doesn't seem to be very idiomatic for play. IOW, I think this should be a somewhat minimal play application with the corresponding classes.
It would probably also be nice to see the caching working under the presence of routes and twirl templates.

[devishree90]

I have added classes for minimal play application. Please let me know if more changes are required

[wolfs]

The CI build currently fails: https://builds.gradle.org/viewLog.html?buildId=10556883&tab=buildResultsDiv&buildTypeId=Gradle_Check_SanityCheck
I guess this line needs to be added to language-scala.gradle:

useTestFixtures(project: ":plugins") // includes core test fixtures

[devishree90]

@wolfs - I have added more test & updated language-scala.gradle as suggested. How do i trigger CI builds to verify my fix?

[devishree90]

@wolfs - I see tests are still failing. Looking at following link and i dont understand exception which i see there. https://builds.gradle.org/viewLog.html?buildId=10583070&buildTypeId=Gradle_Check_SanityCheck. Am i looking at right build?

[wolfs]

No worries, I am fixing the build and will finish this PR. I already have the your work + build fixing on a branch if you want to have a look: https://github.com/gradle/gradle/tree/wolfs/build-cache/platform-scala-compile

[devishree90]

Thank you for the update @wolfs

[wolfs]

@devishree90 The PR has been merged and will be released as part of Gradle 4.6.
Thank you for your work and your patience.

Could you try out a nightly if caching works for your project (e.g. 4.6-20180131151911+0000)?

[wolfs]

@devishree90 Did you have a chance to try out caching for a real world play project? I would really be interested in your results.

[devishree90]

Sorry @wolfs . I totally missed your message. I tested it out. It works perfectly well. I could see 'compilePlayBinaryScala' tasks are loaded from cache. Thanks for following-up.

Let me know if you are interested in specific information.

[wolfs]

@devishree90 Some timings would be nice. Something like how much your build takes without the build cache, and how long a fully cached build takes.

This is something we describe in the build cache guide.