GH-44515: [D] Add Initial Support

[kassane]

Rationale for this change

Add Dlang support. Initial implementation based on c_glib, using gir-to-d generator.

Tools used

• ldc2 (based on LLVM backend + dmd-frontend) toolchain - works in macOS ARM64, too.
• dub (pkg-manager, like cargo [rust])
• gtkd-developers/gir-to-d

What changes are included in this PR?

Add D implementation only. No modify any other language support.

Are these changes tested?

Unit tests will also be added during development.

Are there any user-facing changes?

It adds a new programming language alternative for attracting more users (the small D community).

References

• GIR in Apache Arrow - Error: OutputStream matches conflicting symbols gtkd-developers/gir-to-d#45
  C++
• https://dlang.org/spec/cpp_interface.html
• https://dlang.org/articles/template-comparison.html
• https://dlang.org/articles/cpptod.html
  C
• https://dlang.org/spec/interfaceToC.html
• https://dlang.org/articles/ctod.html

cc: @kou

• GitHub Issue: [D] D programming language Implementation of Arrow #44515

[github-actions]

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - Contributing Overview

[github-actions]

⚠️ GitHub issue #44515 has been automatically assigned in GitHub to PR creator.

[kou]

Could you remove auto generated files?
Can we add a shell script that generates files and fix them (by sed or something)?

[kassane]

Could you remove auto generated files?

Ok.

Can we add a shell script that generates files and fix them (by sed or something)?

It's possible, but is it viable?
Because there are so many lines and the fixes vary ( looking at previous commits).

Note that it is possible to use meson to configure the build of D projects (officially supported). However, it may be interesting to make build.d with D (std/phobos2) features to solve.

[kassane]

Now, base build.d done. (try run dub build to test)

• cpp:cmake (config, build): 🆗 - the flags can be revised later!
• c_glib:meson (setup, build): 🆗 - the flags can be revised later!

Right now, I will review the issue of gir-to-d. If needed, it requires an improved fork to fix the D bindings generation.
Note: Still need to fix D's CI later.

[kou]

I'm not familiar with D but why do we need to create a D program just build the D bindings?

Can we simplify this?

• Can we detect already installed Apache Arrow GLib by pkgconf --cflags --libs arrow-glib instead of building Apache Arrow C++ and GLib?
• Can we create a shell script that just runs dub run girtod -- --gir-directory $(pkgconf --variable=girdir arrow-glib) --input $(pkgconf --variable=girdir arrow-glib/Arrow-1.0.gir --output source --user-runtime-linker?

[kassane]

I'm not familiar with D but why do we need to create a D program just build the D bindings?

Like cargo (Rust), dub only get D files (all *.d in rootpath) by default.

build.d will only be applied for building non-D project.
For this use case, it is assumed that the user does not have Arrow installed.

Can we simplify this?

• Can we detect already installed Apache Arrow GLib by pkgconf --cflags --libs arrow-glib instead of building Apache Arrow C++ and GLib?

dub.json: libs = [...] use pkg-config by default on posix.

Also is possible add specialized libraries like: libs-posix|libs-windows

• Can we create a shell script that just runs dub run girtod -- --gir-directory $(pkgconf --variable=girdir arrow-glib) --input $(pkgconf --variable=girdir arrow-glib/Arrow-1.0.gir --output source --user-runtime-linker?

Yeah! dub.json can add multi commands in preBuildCommands. Like dub.json from glibD project.

[kou]

it is assumed that the user does not have Arrow installed.

Can we require users to install Apache Arrow GLib before they install the D bindings?

FYI: We have the APT/Yum repositories for Apache Arrow GLib: https://arrow.apache.org/install/

• Can we create a shell script that just runs dub run girtod -- --gir-directory $(pkgconf --variable=girdir arrow-glib) --input $(pkgconf --variable=girdir arrow-glib/Arrow-1.0.gir --output source --user-runtime-linker?

Yeah! dub.json can add multi commands in preBuildCommands. Like dub.json from glibD project.

Could you try this approach?

[kassane]

Could you try this approach?

"preBuildCommands": [
		"dub run girtod -- -i Arrow-1.0.gir -o source --use-runtime-linker",
		"dub run girtod -- -i ArrowDataset-1.0.gir -o source --use-runtime-linker",
		"dub run girtod -- -i ArrowFlight-1.0.gir -o source --use-runtime-linker"
	],
"dependencies": {
		"glibd": {
			"repository": "git+https://github.com/gtkd-developers/GlibD.git", // <= get glibd + gir-to-d, generating gio bindings 
			"version": "1546823185334c4727d378baf890fa13d9fa4cbd" // latest commit
		}
	},

Works: bb07030

dub build-ouput

dub test -f --root=d
             Generating test runner configuration 'arrow-d-test-unittest' for 'unittest' (library).
     Pre-gen Running commands for glibd
    Existing package girtod found locally
0 packages fetched, 1 already present, 0 failed
             Building package girtod in /home/kassane/.dub/packages/girtod/0.23.2/girtod/
     Pre-gen Running commands for girtod
    Starting Performing "debug" build using /home/kassane/zig/ldc2-master/bin/ldc2 for x86_64.
    Building girtod 0.23.2: building configuration [application]
     Linking girtod
     Running ../../../girtod/0.23.2/girtod/girtod -i src -o generated --use-runtime-linker
copying file [src/gtkd] to [generated/gtkd]
    Starting Performing "unittest" build using /home/kassane/zig/ldc2-master/bin/ldc2 for x86_64.
    Building glibd 2.4.3+commit.2.g1546823: building configuration [library]
    Building arrow-d ~master: building configuration [arrow-d-test-unittest]
   Pre-build Running commands
             Building package girtod in /home/kassane/.dub/packages/girtod/0.23.2/girtod/
     Pre-gen Running commands for girtod
    Starting Performing "debug" build using /home/kassane/zig/ldc2-master/bin/ldc2 for x86_64.
    Building girtod 0.23.2: building configuration [application]
     Linking girtod
     Running ../../.dub/packages/girtod/0.23.2/girtod/girtod -i Arrow-1.0.gir -o source --use-runtime-linker
             Building package girtod in /home/kassane/.dub/packages/girtod/0.23.2/girtod/
     Pre-gen Running commands for girtod
    Starting Performing "debug" build using /home/kassane/zig/ldc2-master/bin/ldc2 for x86_64.
    Building girtod 0.23.2: building configuration [application]
     Linking girtod
     Running ../../.dub/packages/girtod/0.23.2/girtod/girtod -i ArrowDataset-1.0.gir -o source --use-runtime-linker
             Building package girtod in /home/kassane/.dub/packages/girtod/0.23.2/girtod/
     Pre-gen Running commands for girtod
    Starting Performing "debug" build using /home/kassane/zig/ldc2-master/bin/ldc2 for x86_64.
    Building girtod 0.23.2: building configuration [application]
     Linking girtod
     Running ../../.dub/packages/girtod/0.23.2/girtod/girtod -i ArrowFlight-1.0.gir -o source --use-runtime-linker

[kassane]

For manual fix need add APILookup.txt and use custom configs, like
https://github.com/gtkd-developers/AtkD/blob/master/src/APILookup.txt
doc: https://github.com/gtkd-developers/gir-to-d/blob/master/Readme_APILookup

[kou]

All .dockerfiles are matched...

[kassane]

@kou,
I have not listed the d/ files in .dockerignore.
One question: is docker_ruby already included in c_glib?

Previously the CI/CD section, I just attempted adapting swift-lang recipe.