ci: harden Rocky 9 baseline (TZ pin + retry pkg installs + raise HDFS timeout)#130
Merged
Merged
Conversation
Mulily0513
force-pushed
the
fix-ci-rocky9-parquet-tz
branch
3 times, most recently
from
July 14, 2026 13:41
9aea32d to
c560625
Compare
ostinru
self-requested a review
July 14, 2026 16:29
Mulily0513
changed the title
ostinru
approved these changes
Jul 16, 2026
Mulily0513
force-pushed
the
fix-ci-rocky9-parquet-tz
branch
2 times, most recently
from
July 16, 2026 14:59
579d040 to
4f72e9d
Compare
Collaborator
|
@Mulily0513 , this is great improvement from what we had before. What about marking last failing test as |
Collaborator
|
It seems the work in my PR (https://github.com/apache/cloudberry-pxf/pull/127) duplicates that of this PR. I’m wondering if it would be better to merge parts of my PR into PR #130. |
Collaborator
|
Most tests are passing on my PR. If everything looks fine, I'll bring over some fix commits into this PR. |
Rocky 9 CI has been consistently red since apache#78 introduced Rocky 9 support on 2026-03-26. This commit addresses two orthogonal Rocky 9 stability issues in one topic, so the reproducibility story stays coherent. TZ pin (Etc/GMT-1) ------------------ ParquetTypeConverter.java encodes/decodes INT96 timestamps using ZoneId.systemDefault() in both bytesToTimestamp() and getBinaryFromTimestamp(). Ubuntu 22.04 inherits Europe/London (BST = UTC+1 in summer) from the base image; Rocky 9 defaults to UTC. Same Parquet input, different timestamp strings, and the .ans expected values (calibrated to Ubuntu output) diff on Rocky 9. Pinning TZ=Etc/GMT-1 (fixed UTC+1, no DST) in both distros' compose files makes JVM systemDefault() identical everywhere, matches the PGOPTIONS timezone='GMT-1' already set by run_tests.sh, and preserves existing .ans values (Ubuntu summer already emits UTC+1). HDFS write tuning ----------------- HdfsWritableTextTest multi-block variants timed out or silently dropped writes on shared GHA free runners. Root cause is a combination of slow DataNode ACK under noisy-neighbor I/O contention (default 60s socket timeout kicks in) and unnecessarily large test payloads (1.2 GB) for a 'multi-block' assertion that only needs to cross a block boundary a handful of times. * dfs.client.socket-timeout: 60s -> 180s. Absorbs GHA ACK jitter (worst observed ~90s) without dragging out failure detection when a DataNode is genuinely dead. * dfs.blocksize: 128 MB -> 16 MB. Multi-block tests hit boundaries at 1/8 the data volume; the boundary code paths (LineRecordReader cross-block splits) are exercised the same way. 16 MB (not 4 MB) leaves headroom for textFormatWideRowsInsert's 10 MiB rows to fit in one block, avoiding a Rocky 9-only write/read comparison bug at cross-block row boundaries. * HdfsWritableTextTest data volume: 15000x1000 rows (~1.2 GB) -> 1000x1000 rows (~80 MB). 5 blocks at 16 MB each still exercises multi-block reader/writer paths. All changes are in CI configuration and test fixtures. No PXF Java source, no shipped artifacts, zero customer-facing risk. Related to apache#17.
CI jobs fail intermittently due to transient upstream unavailability of network dependencies pulled at build/test time. Symptoms observed on apache#130 across four rounds of retries: * Rocky HPC SIG dnf mirrorlist returning empty (spurious dnf install failures during container bring-up) * Maven Central returning 5xx on artifactregistry-maven-wagon-*.pom * maven.top-q.co.il (single-source repo for JSystem test framework) returning 504 for jsystemCore-*.pom * mcr.microsoft.com returning transient 5xx during testcontainers' runtime pull of mssql/server:2019-latest for JdbcMssqlTest None of these are our upstreams, but blaming them isn't a fix; the purpose of apache#130 (and apache#17) is to make CI green regardless of external service jitter. Two complementary strategies: 1. Retry package-manager installs (shell layer) -------------------------------------------------- Add a shared retry() helper in utils.sh (3 attempts, 15s delay) and wrap the network-touching install/update commands in entrypoint.sh, build_pxf.sh, and build_cloudberrry.sh: * sudo apt update / sudo apt install / sudo apt-get install * sudo dnf install Local file installs (rpm -Uvh $file, dpkg -i $file) are left as-is. 2. Cache external artifacts across CI runs (workflow layer) ------------------------------------------------------------- * Maven local repository (~/.m2/repository) — cached with a key derived from automation/pom.xml hash, restored across three job matrices: - pxf-test (container mvn) - pxf-test-rocky9 (container mvn) - pxf-testcontainer-test (host mvn) Container jobs bind-mount ${GITHUB_WORKSPACE}/.m2 into the pxf-cbdb-dev container as /home/gpadmin/.m2 so both mvn invocations share one cache. Once warm, Maven Central + top-q.co.il are never contacted on reruns. * MSSQL testcontainer image (mcr.microsoft.com/mssql/server:2019-latest) — saved as a tarball and cached. On cache miss, retries the docker pull 3 times with backoff before failing. Only enabled for the pxf-jdbc TC group (the only consumer). All changes are scoped to CI infrastructure. No non-official Maven mirrors are configured (Apache release policy). Cache keys are branch- scoped by GHA automatically, so PR reruns immediately benefit while main-branch cache stays isolated. Related to apache#17.
Test PXF Rocky9 - smoke intermittently fails with 'DataNode not available after 90s' in entrypoint.sh's wait_for_datanode helper. The smoke test is the first test group run against a freshly booted pxf-cbdb-dev container and starts the entire Hadoop/Hive/HBase stack cold — NameNode, DataNode, Hive metastore, HiveServer2, HBase master/regionserver — all competing for CPU and I/O on a shared GHA runner. DataNode registration path breakdown: 1. JVM cold start (class load + JIT + GC warmup) 2. DataNode scans /data/dn/dfs/data for existing block metadata 3. DataNode registers with NameNode (sends block report) 4. NameNode ACKs, exits safe mode, marks DataNode Live 5. 'hdfs dfsadmin -report' shows 'Live datanodes: 1' On healthy GHA runners this cycle completes in 20-40s. Under I/O contention (overlay2 file I/O slowness + noisy-neighbor CPU starvation) it can stretch to 90-180s. The prior 90s/attempt * 2 attempts = 180s total budget is on the edge and intermittently insufficient. Raise per-attempt wait from 90s to 180s (max_attempts unchanged at 2, total budget 360s). Healthy runs are unaffected — the loop exits as soon as DataNode reports Live. Only the tail latency budget grows. Related to apache#17.
Mulily0513
force-pushed
the
fix-ci-rocky9-parquet-tz
branch
from
July 17, 2026 03:05
9743241 to
2dd1dc2
Compare
Collaborator
Author
@ostinru thanks for the review! Turned out we got a bit lucky on the latest push — CI came back fully green (61/61), no tests disabled. |
Collaborator
Author
@MisterRaindrop thanks for the heads up on #127, sorry for the overlap! Welcome to bring anything over that isn't already covered here |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Rocky 9 CI has been consistently red on every main-branch push since #78 introduced Rocky 9 support on 2026-03-26. This PR bundles CI-only fixes in three commits, each addressing an orthogonal failure mode:
HdfsWritableTextTestmulti-block variants time out or silently drop writes on shared GHA free runners.top-q.co.il), and MCR docker registry all return transient 5xx that abort entire test groups.Test PXF Rocky9 - smokeintermittently fails withDataNode not available after 90sbecause JVM cold start + overlay2 block scan + NameNode handshake can exceed the 90s wait window under I/O contention.All fixes are scoped to CI configuration and infrastructure scripts. No PXF Java source, no test code, no shipped artifacts are modified.
Commits
1.
ci: harden Rocky 9 baseline (TZ pin + HDFS write tuning)Two related Rocky 9 stability issues in one commit.
TZ pin (
Etc/GMT-1):ParquetTypeConverter.javaencodes/decodes INT96 timestamps usingZoneId.systemDefault()in bothbytesToTimestamp()andgetBinaryFromTimestamp(). Ubuntu 22.04 inheritsEurope/London(BST = UTC+1 in summer); Rocky 9 defaults to UTC. Same Parquet input, different timestamp strings, and the.ansexpected values (calibrated to Ubuntu output) diff on Rocky 9. PinningTZ=Etc/GMT-1in both distros' compose files makes JVMsystemDefault()identical everywhere, matches thePGOPTIONS timezone='GMT-1'already set byrun_tests.sh, and preserves existing.ansvalues.HDFS write tuning:
dfs.client.socket-timeout: 60s → 180s. Absorbs GHA ACK jitter (worst observed ~90s) without dragging out failure detection when a DataNode is genuinely dead.dfs.blocksize: 128 MB → 16 MB. Multi-block tests exerciseLineRecordReadercross-block splits with 1/8 the data volume; 16 MB (not smaller) leaves headroom fortextFormatWideRowsInsert's 10 MiB rows to fit in one block, avoiding a Rocky-9-only write/read comparison bug at cross-block row boundaries.HdfsWritableTextTestdata volume: 15000×1000 rows (~1.2 GB) → 1000×1000 rows (~80 MB). 5 blocks at 16 MB each still exercises multi-block reader/writer paths.2.
ci: cache & retry external downloadsTwo complementary strategies for network-dependency reliability.
Retry package-manager installs (shell layer): shared
retry()helper inutils.sh(3 attempts, 15s delay) wrapssudo apt update / install / apt-get installandsudo dnf installinentrypoint.sh,build_pxf.sh,build_cloudberrry.sh. Local file installs (rpm -Uvh $file,dpkg -i $file) left as-is.Cache external artifacts (workflow layer):
~/.m2/repository) cached with a key derived fromautomation/pom.xmlhash, restored across all three test-job matrices:pxf-test,pxf-test-rocky9(containermvn), andpxf-testcontainer-test(hostmvn). Container jobs bind-mount${GITHUB_WORKSPACE}/.m2into thepxf-cbdb-devcontainer as/home/gpadmin/.m2so bothmvninvocations share one cache. Once warm, Maven Central andmaven.top-q.co.il(single-source JSystem host) are never contacted on reruns.mcr.microsoft.com/mssql/server:2019-latest) saved as tarball and cached. On cache miss, retries thedocker pull3× with backoff before failing. Only enabled for thepxf-jdbcTC group (its only consumer).No non-official Maven mirrors are configured — Apache release policy. Cache keys are branch-scoped by GHA automatically, so PR reruns immediately benefit while main-branch cache stays isolated.
3.
ci: raise HDFS DataNode startup wait to 180sTest PXF Rocky9 - smoke(the first test group run against a freshly booted container) starts the entire Hadoop/Hive/HBase stack cold — NameNode, DataNode, Hive metastore, HiveServer2, HBase master/regionserver — all competing for CPU and I/O on a shared GHA runner.DataNode registration path: JVM cold start → block metadata scan on overlay2 → block report to NameNode → NameNode ACK →
dfsadmin -reportshowsLive datanodes: 1. Healthy GHA runs: 20-40s. Under I/O contention: 90-180s.wait_for_datanodeper-attempt window raised from 90s to 180s (max_attemptsunchanged at 2, total budget 360s). Healthy runs unaffected — the loop exits as soon as DataNode reports Live. Only the tail latency budget grows.Zero customer risk
All changes are in
ci/(and.github/workflows/) and don't ship to end users. The underlyingZoneId.systemDefault()behavior inParquetTypeConverter(relevant to cross-TZ deployments) is unchanged; addressing that is out of scope for this CI-reliability PR.Test plan
features,gpdb,gpdb_fdw,hcfsturn green (Parquet baseline)HdfsWritableTextTestno longer flakes on slow runnersTest PXF Rocky9 - smokeDataNode wait no longer times out under GHA contentionpxf-jdbcTC rerunsRelated to #17 (long-running CI repair). Likely unblocks #123 (which inherits the same baseline).