`julia_connect()` fails due to dylibs

[edwardlavender]

On MacOS julia_connect() may fail with various dylib errors:

Error: Error happens in Julia.
InitError: could not load library "/Users/lavended/.julia/artifacts/71f842209dc88269523ab21f38b8696ed4039b5b/lib/libproj.25.dylib"
dlopen(/Users/lavended/.julia/artifacts/71f842209dc88269523ab21f38b8696ed4039b5b/lib/libproj.25.dylib, 0x0001): Library not loaded: @rpath/libsqlite3.0.dylib
  Referenced from: <4C4C4435-5555-3144-A196-DCBF483577EE> /Users/lavended/.julia/artifacts/71f842209dc88269523ab21f38b8696ed4039b5b/lib/libproj.25.9.7.1.dylib
  Reason: tried: '/Users/lavended/.julia/artifacts/71f842209dc88269523ab21f38b8696ed4039b5b/lib/./libsqlite3.0.dylib' (no such file), '/Users/lavended/.julia/artifacts/71f842209dc88269523ab21f38b8696ed4039b5b/lib/./libsqlite3.0.dylib' (no such file), '/Users/lavended/.julia/juliaup/julia-1.12.5+0.aarch64.apple.darwin14/Julia-1.12.app/Contents/Resources/julia/lib/julia/libsqlite3.0.dylib' (no such file), '/Users/lavended/.julia/juliaup/julia-1.12.5+0.aarch64.apple.darwin14/Julia-1.12.app/Contents/Resources/julia/lib/

[edwardlavender]

These errors are related to https://github.com/katrinabrock/juli-slipRs

[edwardlavender]

This is a workflow that worked on MacBook Pro 2023 (Apple M2 Pro), MacOS Sequoia v 15.7.4.

For testing, create a new project and run the following code:

library(JuliaCall)
julia_setup()
julia_command("using Pkg")
julia_command('Pkg.activate(".")')
julia_command('Pkg.add("ArchGDAL")')
julia_command("using ArchGDAL")

Error: Error happens in Julia.
InitError: could not load library "/Users/lavended/.julia/artifacts/71f842209dc88269523ab21f38b8696ed4039b5b/lib/libproj.25.dylib"
dlopen(/Users/lavended/.julia/artifacts/71f842209dc88269523ab21f38b8696ed4039b5b/lib/libproj.25.dylib, 0x0001): Library not loaded: @rpath/libsqlite3.0.dylib
  Referenced from: <4C4C4435-5555-3144-A196-DCBF483577EE> /Users/lavended/.julia/artifacts/71f842209dc88269523ab21f38b8696ed4039b5b/lib/libproj.25.9.7.1.dylib
  Reason: tried: '/Users/lavended/.julia/artifacts/71f842209dc88269523ab21f38b8696ed4039b5b/lib/./libsqlite3.0.dylib' (no such file), '/Users/lavended/.julia/artifacts/71f842209dc88269523ab21f38b8696ed4039b5b/lib/./libsqlite3.0.dylib' (no such file), '/Users/lavended/.julia/juliaup/julia-1.12.5+0.aarch64.apple.darwin14/Julia-1.12.app/Contents/Resources/julia/lib/julia/libsqlite3.0.dylib' (no such file), '/Users/lavended/.julia/juliaup/julia-1.12.5+0.aarch64.apple.darwin14/Julia-1.12.app/Contents/Resources/julia/lib/

This error occurs even without R geospatial libraries (e.g., terra) being loaded!

Instructions

Create Overrides.toml

The first step is to create .julia/artifacts/Overrides.toml. For me, this lives in /Users/lavended/.julia/artifacts/Overrides.toml. We use this file to direct Julia to use the same libraries that R is using, which on MacOS live in /usr/lib/ or /opt/homebrew/Cellar/. For each problematic library, we need to find the (i) local copy and (ii) (j) the _jll version on JuliaBinaryWrappers.

Resolve SQLite

• First, we'll try to resolve the libsqlite3.0.dylib error.
• Go to Google and search for lib sqlite _jll Julia or similar. We are looking for a link to https://github.com/JuliaBinaryWrappers. https://github.com/JuliaBinaryWrappers/SQLite_jll.jl is the closest option.
• View Project.toml and extract the name and uuid:

name = "SQLite_jll"
uuid = "76ed43ae-9a5d-5a62-8c75-30186b810ce8"

• Find the local version of SQLite. Manually search /usr/lib and /opt/homebrew/Cellar/ or try:

pkg-config --variable=libdir sqlite3 

• Update Overrides.toml with the uuid, the package name (minus the _jll) and the directory (minus the lib/), following the structure below. Note that the library must be within a lib/ folder so if this does not exist, you need to create a symlink (see advice above).

[76ed43ae-9a5d-5a62-8c75-30186b810ce8]
SQLite = "/usr/"

• Save the file, restart R and try again:

Error: Error happens in Julia.
InitError: could not load library "/Users/lavended/.julia/artifacts/71f842209dc88269523ab21f38b8696ed4039b5b/lib/libproj.25.dylib"
dlopen(/Users/lavended/.julia/artifacts/71f842209dc88269523ab21f38b8696ed4039b5b/lib/libproj.25.dylib, 0x0001): Library not loaded: @rpath/libsqlite3.0.dylib
  Referenced from: <4C4C4435-5555-3144-A196-DCBF483577EE> /Users/lavended/.julia/artifacts/71f842209dc88269523ab21f38b8696ed4039b5b/lib/libproj.25.9.7.1.dylib
  Reason: tried: '/Users/lavended/.julia/artifacts/71f842209dc88269523ab21f38b8696ed4039b5b/lib/./libsqlite3.0.dylib' (no such file), '/Users/lavended/.julia/artifacts/71f842209dc88269523ab21f38b8696ed4039b5b/lib/./libsqlite3.0.dylib' (no such file), '/Users/lavended/.julia/juliaup/julia-1.12.5+0.aarch64.apple.darwin14/Julia-1.12.app/Contents/Resources/julia/lib/julia/libsqlite3.0.dylib' (no such file), '/Users/lavended/.julia/juliaup/julia-1.12.5+0.aarch64.apple.darwin14/Julia-1.12.app/Contents/Resources/julia/lib/

Resolve PROJ

• Now we try to resolve the libproj.25.dylib error
• Go to https://github.com/JuliaBinaryWrappers/PROJ_jll.jl
• From Project.toml, extract the name and uuid:

name = "PROJ_jll"
uuid = "58948b4f-47e0-5654-a9ad-f609743f8632"

• Update Overrides.toml:

[58948b4f-47e0-5654-a9ad-f609743f8632]
PROJ = "/opt/homebrew/Cellar/proj/9.7.1/"

• Save, restart R, retry:

Error: Error happens in Julia.
InitError: could not load library "/Users/lavended/.julia/artifacts/99a3cc4fca09bb791c8b76c40daa9abee5fff3b3/lib/libgeotiff.5.2.4.dylib"
dlopen(/Users/lavended/.julia/artifacts/99a3cc4fca09bb791c8b76c40daa9abee5fff3b3/lib/libgeotiff.5.2.4.dylib, 0x0001): Library not loaded: @rpath/libproj.25.dylib
  Referenced from: <4C4C4488-5555-3144-A11A-36E0B24B28B0> /Users/lavended/.julia/artifacts/99a3cc4fca09bb791c8b76c40daa9abee5fff3b3/lib/libgeotiff.5.2.4.dylib
  Reason: tried: '/Users/lavended/.julia/artifacts/99a3cc4fca09bb791c8b76c40daa9abee5fff3b3/lib/./libproj.25.dylib' (no such file), '/Users/lavended/.julia/artifacts/99a3cc4fca09bb791c8b76c40daa9abee5fff3b3/lib/./libproj.25.dylib' (no such file), '/Users/lavended/.julia/juliaup/julia-1.12.5+0.aarch64.apple.darwin14/Julia-1.12.app/Contents/Resources/julia/lib/julia/libproj.25.dylib' (no such file), '/Users/lavended/.julia/juliaup/julia-1.12.5+0.aarch64.apple.darwin14/Julia-1.12.app/Contents/Resources/julia/

Resolve libgeotiff

• Go to https://github.com/JuliaBinaryWrappers/libgeotiff_jll.jl
• From Project.toml, extract the name and uuid:

uuid = "06c338fa-64ff-565b-ac2f-249532af990e"
version = "100.702.400+0"

• Update Overrides.toml:

[06c338fa-64ff-565b-ac2f-249532af990e]
libgeotiff = "/opt/homebrew/Cellar/libgeotiff/1.7.4/"

• Save, restart R, retry:

Error: Error happens in Julia.
InitError: could not load library "/opt/homebrew/Cellar/libgeotiff/1.7.4/lib/libgeotiff.5.2.4.dylib"
dlopen(/opt/homebrew/Cellar/libgeotiff/1.7.4/lib/libgeotiff.5.2.4.dylib, 0x0001): tried: '/opt/homebrew/Cellar/libgeotiff/1.7.4/lib/libgeotiff.5.2.4.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/Cellar/libgeotiff/1.7.4/lib/libgeotiff.5.2.4.dylib' (no such file), '/opt/homebrew/Cellar/libgeotiff/1.7.4/lib/libgeotiff.5.2.4.dylib' (no such file)

• Now, it looks like Julia is looking for a libgeotiff.5.2.4.dylib file which does not exist.

  • First, let's check we have an up-to-date version of libgeotiff installed by running the code below. libgeotiff seems to be up to date!

    brew update
    brew upgrade
    

  • Second, let's check the contents of the /opt/homebrew/Cellar/libgeotiff/1.7.4/lib/ directory:

    libgeotiff.dylib
    libgeotiff.5.dylib
    libgeotiff.a
    pkgconfig/
    

  • As a hack, we create a symlink in R from libgeotiff.5.dylib to libgeotiff.5.2.4.dylib and hope for the best:

    Sys.symlink(
      "/opt/homebrew/Cellar/libgeotiff/1.7.4/lib/libgeotiff.5.dylib",
      "/opt/homebrew/Cellar/libgeotiff/1.7.4/lib/libgeotiff.5.2.4.dylib")
    

• Restart R, retry:

Error: Error happens in Julia.
InitError: could not load library "/Users/lavended/.julia/artifacts/6cb83a902c715db12c050e6b239b60e4feae2ab3/lib/libgdal.38.dylib"
dlopen(/Users/lavended/.julia/artifacts/6cb83a902c715db12c050e6b239b60e4feae2ab3/lib/libgdal.38.dylib, 0x0001): Library not loaded: @rpath/libgeotiff.5.dylib
  Referenced from: <4C4C44DF-5555-3144-A15B-0C57CBDE7175> /Users/lavended/.julia/artifacts/6cb83a902c715db12c050e6b239b60e4feae2ab3/lib/libgdal.38.3.12.2.dylib
  Reason: tried: '/Users/lavended/.julia/artifacts/6cb83a902c715db12c050e6b239b60e4feae2ab3/lib/./libgeotiff.5.dylib' (no such file), '/Users/lavended/.julia/artifacts/6cb83a902c715db12c050e6b239b60e4feae2ab3/lib/./libgeotiff.5.dylib' (no such file), '/Users/lavended/.julia/juliaup/julia-1.12.5+0.aarch64.apple.darwin14/Julia-1.12.app/Contents/Resources/julia/lib/julia/libgeotiff.5.dylib' (no such file), '/Users/lavended/.julia/juliaup/julia-1.12.5+0.aarch64.apple.darwin14/Julia-1.12.app/Contents/Resources/julia/lib

Resolve GDAL

• Go to https://github.com/JuliaBinaryWrappers/libgeotiff_jll.jl
• From Project.toml, extract the name and uuid:

name = "GDAL_jll"
uuid = "a7073274-a066-55f0-b90d-d619367d196c"

• Update Overrides.toml:

[a7073274-a066-55f0-b90d-d619367d196c]
GDAL = "/opt/homebrew/Cellar/gdal/3.12.2/"

• Save, restart R, retry and it works!

TLDR

• libsqlite3.0.dylib is fixed by overriding SQLite_jll.jl with sqlite3 in /usr/lib/
• libproj.25.dylib is fixed by overriding PROJ_jll.jl with libproj in /opt/homebrew/Cellar/proj/9.7.1/lib/
• libgeotiff.5.2.4.dylib is fixed by overriding libgeotiff_jll with libgeotiff in /opt/homebrew/Cellar/proj/9.7.1/lib/
• libgdal.38.dylib is fixed by overriding GDAL_jll with libgdal in /opt/homebrew/Cellar/gdal/3.12.2/lib

The final Overrides.toml is:

[76ed43ae-9a5d-5a62-8c75-30186b810ce8]
SQLite = "/usr/"

[58948b4f-47e0-5654-a9ad-f609743f8632]
PROJ = "/opt/homebrew/Cellar/proj/9.7.1/"

[06c338fa-64ff-565b-ac2f-249532af990e]
libgeotiff = "/opt/homebrew/Cellar/libgeotiff/1.7.4/"

[a7073274-a066-55f0-b90d-d619367d196c]
GDAL = "/opt/homebrew/Cellar/gdal/3.12.2/"