[Bug]: Linux HopRun ignores HOP_EMPTY_STRING_DIFFERS_FROM_NULL=Y while Windows HopRun honors it

[tw102609]

Apache Hop version?

2.18.1

Java version?

21

Operating system

Linux

What happened?

Environment
Apache Hop 2.18.1
Official Docker image (apache/hop:2.18.1)
Linux (Docker)
Windows 11
Java 21
SQL Server (Microsoft JDBC 13.4.0)
Summary

When running the same project and workflow:

Windows GUI preserves empty strings.
Windows hop-run preserves empty strings.
Linux hop-run (official Docker image) converts empty strings to NULL.

This occurs even though HOP_EMPTY_STRING_DIFFERS_FROM_NULL=Y is supplied through every supported mechanism we could identify.

Expected behavior

When

HOP_EMPTY_STRING_DIFFERS_FROM_NULL=Y

is configured,

empty strings ("") should remain empty strings during pipeline execution and be written to SQL Server as '', not NULL.

This is the behavior observed on Windows.

Actual behavior

On Linux using the official Docker image:

Empty strings become NULL.
Inside the pipeline:
Get Variables

reports

HOP_EMPTY_STRING_DIFFERS_FROM_NULL = N

despite every configuration source specifying Y.

Reproduction

Environment JSON contains

{
"variables": [
{
"name": "HOP_EMPTY_STRING_DIFFERS_FROM_NULL",
"value": "Y"
}
]
}

Docker also receives

-e HOP_EMPTY_STRING_DIFFERS_FROM_NULL=Y

and

-e HOP_CONFIG_OPTIONS="--config-file-set-variables=HOP_EMPTY_STRING_DIFFERS_FROM_NULL=Y"

The startup script confirms before Hop starts:

Shell variable:
HOP_EMPTY_STRING_DIFFERS_FROM_NULL=Y

Environment JSON:
"name": "HOP_EMPTY_STRING_DIFFERS_FROM_NULL"
"value": "Y"

Process environment:
HOP_EMPTY_STRING_DIFFERS_FROM_NULL=Y

However, inside the pipeline:

Get Variables

returns

HOP_EMPTY_STRING_DIFFERS_FROM_NULL = N
Behaviour comparison
Environment Get Variables Result
Windows GUI Y Empty strings preserved
Windows HopRun Y Empty strings preserved
Linux Docker HopRun N Empty strings become NULL
Minimal pipeline

A minimal pipeline consisting of

Generate Rows
(String field,
Set Empty String = Yes)
|
Table Output

writes

NULL

to SQL Server on Linux Docker.

The same pipeline writes

''

(empty string)

on Windows.

Additional investigation performed

Verified:

Same Hop version (2.18.1)
Same workflow
Same project
Same environment JSON
Same SQL Server JDBC driver
Same Java major version
Same SQL Server destination

Verified before pipeline startup:

HOP_EMPTY_STRING_DIFFERS_FROM_NULL=Y

Verified inside pipeline:

HOP_EMPTY_STRING_DIFFERS_FROM_NULL=N

Tried:

Docker ENV
Environment JSON
HOP_CONFIG_OPTIONS
Windows HopRun
Windows GUI
Linux HopRun

No configuration change caused Linux HopRun to honor the value.

Question

Is HOP_EMPTY_STRING_DIFFERS_FROM_NULL expected to work under Linux HopRun?

If so, is there another supported mechanism for configuring it in the official Docker image?

If not, this appears to be a Linux-specific inconsistency between Windows HopRun and Linux HopRun.

Issue Priority

Priority: 2

Issue Component

Component: Infrastructure, Component: Hop Run

[hansva]

This is an issue; our Docker container has no clean way to override variables in some cases.
The only solution is to update hop-config directly in the container

[tw102609]

Thanks for confirming. Since the container is intended to be ephemeral, would you recommend updating hop-config.json as part of the container startup (for example in our custom checkout-and-run-job-hop.sh before HopRun executes), or is there another supported initialization hook you would recommend?

[hansva]

I thought we had a ticket somewhere in our backlog. I would like to make it possible to override all those settings with environment variables; that way, you should be able to use Docker environment variables.

At this point, I would indeed use your custom checkout-and-run and do a sed on the config file to replace the value

[tw102609]

Thanks!

[tw102609]

I tried updating hop-config.json from my checkout-and-run-job-hop.sh, but the official Docker entrypoint performs hop-conf.sh --project-create and hop-conf.sh --environment-create after my extension script exits. That appears to recreate or overwrite the configuration before HopRun starts. Is there a supported hook that runs after hop-conf.sh but before HopRun, or would you recommend replacing the default entrypoint entirely? Thanks.

[hansva]

I'll try to take a look at this, but project-create/environment-create should not overwrite or reset the variables section in that file.

[tw102609]

Sure, thanks. Let me know if I can provide anything else. For what its worth I added some more diagnostics to the custom entrypoint. Attaching

logging-with-diagnostics.log