MLE-31165 Bumping to Spark 4.2.0

[rjrudin]

This matches the Spark connector. Makes necessary bumps to Hadoop and the AWS SDK as well.

[Copilot]

Pull request overview

Updates Flux’s dependency alignment to match the Spark connector by bumping Spark to 4.2.0, updating related Hadoop/AWS SDK versions, and centralizing the Jackson version to keep the enforced Jackson range consistent with Spark/LangChain4j constraints.

Changes:

• Bump Spark to 4.2.0 and Hadoop to 3.5.0, with AWS SDK aligned to the Hadoop-selected version.
• Add a jacksonVersion property and wire root dependency resolution to use it (instead of a hard-coded value).
• Add a forced lz4-java patch version to address a CVE via Gradle resolutionStrategy.

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated no comments.

File	Description
gradle.properties	Bumps Spark/Hadoop/AWS SDK versions and introduces jacksonVersion for centralized version control.
flux-cli/build.gradle	Adds a forced lz4-java version and adjusts Spark dependency handling accordingly.
build.gradle	Uses jacksonVersion when forcing Jackson modules to 2.21.x.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

flux-cli/src/test/java/com/marklogic/flux/api/GenericFilesImporterTest.java:100

• The test currently contains commented-out assertions and a temporary explanation tied to an external connector PR. This makes the test harder to maintain and will require another edit when the connector behavior changes again. Prefer asserting on the common contract (exception message) while allowing either FluxException or ConnectorException until the underlying behavior is stabilized.

        // This is currently catching a ConnectorException instead of a FluxException due to a change in the Spark
        // connector via PR 683 for the Spark connector. That change should be undone with the real fix being that
        // FailedRequest should become serializable in the Java Client.
//        FluxException ex = assertThrows(FluxException.class, command::execute);
        ConnectorException ex = assertThrows(ConnectorException.class, command::execute);

flux-cli/src/test/java/com/marklogic/flux/impl/importdata/ImportOrcFilesTest.java:174

• abortOnReadFailure is intended to test the ORC import command, but this method invokes "import-parquet-files" (a few lines above). That means the test is exercising the wrong CLI command and may miss regressions in the ORC path.

            "--permissions", DEFAULT_PERMISSIONS
        );

        assertTrue(stderr.contains("Command failed") && stderr.contains("footer is too large"),