How to debug an error with "Errors encountered parsing headers, unable to proceed with conversion"

[sloede]

While trying to generate bindings for a C library in deps/build.jl, I cannot get past the call to convert_headers, since it always throws an error as

│ ERROR: LoadError: Errors encountered parsing headers, unable to proceed with conversion
│ Stacktrace:
│  [1] error(::String) at ./error.jl:33
│  [2] (::CBindingGen.var"#18#21"{var"#1#2",Array{String,1}})(::String) at /home/mschlott/.julia/packages/CBindingGen/BiQcs/src/CBindingGen.jl:193
│  [3] mktempdir(::CBindingGen.var"#18#21"{var"#1#2",Array{String,1}}, ::String; prefix::String) at ./file.jl:709
│  [4] mktempdir(::Function, ::String) at ./file.jl:707 (repeats 2 times)
│  [5] convert_headers(::Function, ::Array{String,1}; args::Array{String,1}) at /home/mschlott/.julia/packages/CBindingGen/BiQcs/src/CBindingGen.jl:169
│  [6] top-level scope at /mnt/ssd/home/mschlott/code/SEAL.jl/deps/build.jl:56
│  [7] include(::String) at ./client.jl:457
│  [8] top-level scope at none:5
│ in expression starting at /mnt/ssd/home/mschlott/code/SEAL.jl/deps/build.jl:56

I have tried looking at CBindingGen.jl:193, but I cannot figure out how to make LibClang tell me what is actually the problem (e.g., is it a bad include directory, or is it something in the headers themselves etc.). Can you give me advice on how to proceed here, i.e., how to debug the error that caused this message?

[sloede]

Alright, so with the help of LibClang.clang_formatDiagnostic, which I very dirtily hacked into the CBindingGen package code, I managed to get a somewhat meaningful error message:

/home/mschlott/.julia/artifacts/098fe6e2292cbf4947cdf5c2f169f0caf3f691d8/include/SEAL-3.6/seal/c/defines.h:7:10: fatal error: 'cstddef' file not found

But now I am stumped again: cstddef is - as far as I know - a standard library header, so why is libclang not able to find it? Do I need to provide some additional include directories - and if yes, how to find them 🤔 ?

[sloede]

Aha! cstddef seems to the C++ counterpart to stddef.h, which is a C header. So maybe the problem stems from the fact that defines.h in the SEAL library really is a C++ header file containing C++ code to produce the actual C interface. While the symbols I'm actually interested in are correctly declared as extern "C", the existence of C++ code in the header seems to preclude the use of LibClang and thus CBindingGen. Or am I missing something here, @krrutkow ?

Just to be sure, I changed the #include <cstddef> in defines.h to #include <stddef.h> and it actually continues to parse the file until it hits another error in a line starting with a static_assert... So that seems to confirm that there is an issue with C++ code...

[krrutkow]

It is true that CBinding* is focused on C-only API's, but it looks like SEAL tries to provide a C API at least... I didn't dig through the SEAL "C" headers too much, but I think you can work around their problems without modifying them at all.

First, create a fake cstddef file echo '#include <stddef.h>' > cstddef and in your call to convert_headers be sure to add "-I." to the args array argument so the fake header be found.

If the static_assert is the only non-C code, then it should be possible to tell the static_assert to go away with a preprocessor definition for it, so add "-Dstatic_assert(a...)=" to args.

Let me know how far that gets you.

[sloede]

First, create a fake cstddef file echo '#include <stddef.h>' > cstddef and in your call to convert_headers be sure to add "-I." to the args array argument so the fake header be found.

If the static_assert is the only non-C code, then it should be possible to tell the static_assert to go away with a preprocessor definition for it, so add "-Dstatic_assert(a...)=" to args.

Thanks for these suggestions! I went ahead and removed the problematic lines (or disabled them as suggested), but that just kicks the can down the road. Now it complains with

 /home/mschlott/.julia/artifacts/098fe6e2292cbf4947cdf5c2f169f0caf3f691d8/include/SEAL-3.6/seal/c/batchencoder.h:16:1: error: expected identifier or '('

where the offending line is
https://github.com/microsoft/SEAL/blob/5c586e2480dc0a43fb76a23a3dcc3daecb3764e8/native/src/seal/c/batchencoder.h#L16

The SEAL_C_FUNC macro should evaluate to extern "C" and is defined here:
https://github.com/microsoft/SEAL/blob/5c586e2480dc0a43fb76a23a3dcc3daecb3764e8/native/src/seal/c/defines.h#L62

Any idea why LibClang is having so much trouble parsing these C++ files? Or is this a general limitation that I am just not aware of?

[krrutkow]

The usage of LibClang in CBindingGen is as a C parser, it is possible (and I will get around to it some day) to use it for parsing C++ though. Adding "-DSEAL_C_DECOR=", "-Wno-macro-redefined" to args would get it farther along. Depending on _MSC_VER, you might need "-DSEAL_C_DECOR=__declspec(dllimport)" instead.

BTW, I am working on a "v2.0" of the CBinding* framework which should be closer to supporting parsing C++, but it improves usability and bindings compile/load times considerably. So keep the feature ideas coming! 😃

[sloede]

Adding "-DSEAL_C_DECOR=", "-Wno-macro-redefined" to args would get it farther along.

Unfortunately, this did not really help. I even added #define SEAL_C_FUNC extern "C as the last line of defines.h, but I still get the following errors:

/home/mschlott/.julia/artifacts/098fe6e2292cbf4947cdf5c2f169f0caf3f691d8/include/SEAL-3.6/seal/c/batchencoder.h:16:1: error: expected identifier or '('
/home/mschlott/.julia/artifacts/098fe6e2292cbf4947cdf5c2f169f0caf3f691d8/include/SEAL-3.6/seal/c/batchencoder.h:18:1: error: expected identifier or '('
/home/mschlott/.julia/artifacts/098fe6e2292cbf4947cdf5c2f169f0caf3f691d8/include/SEAL-3.6/seal/c/batchencoder.h:20:1: error: expected identifier or '('
/home/mschlott/.julia/artifacts/098fe6e2292cbf4947cdf5c2f169f0caf3f691d8/include/SEAL-3.6/seal/c/batchencoder.h:22:1: error: expected identifier or '('

and many more like them. I think the problem is that any non-C code just does not get parsed properly, maybe with the exception to some macros and C++-style single-line comments.

Even if I manually change line 16 in batchencoder.h above from

SEAL_C_FUNC BatchEncoder_Create(void *context, void **batch_encoder);

to

extern "C" long BatchEncoder_Create(void *context, void **batch_encoder);

the error remains unchanged. It is only fixed if I go for pure C with

long BatchEncoder_Create(void *context, void **batch_encoder);

Thus, to me it seems like as if LibClang as used by CBindingGen is not able to handle most (any?) kind of pure C++ constructs. Or is there anything I can do to make it treat the code like C++ instead of C?

[sloede]

BTW, I am working on a "v2.0" of the CBinding* framework which should be closer to supporting parsing C++, but it improves usability and bindings compile/load times considerably. So keep the feature ideas coming!

That's great to hear - and thanks again for your fast and helpful support. I do like CBinding and CBindingGen and especially enjoy the latter's simple interface (compared to Clang.jl for binding generation). We have already realized two other libraries that make use of it (P4est.jl and KROME.jl).

[krrutkow]

The extern "C" is what makes it C++ code, and it will not parse then. So if you just remove that from the defines.h file you should able to proceed. There is also the use of bool so adding "-Dbool=_Bool" to args would take care of that, but somewhere SDKDDKVer.h was being included which I did not have available. So that is as far as I could take it.

[sloede]

Thanks, this got me a little further. I have now an additional file helper.h that is included first, which contains the following code:

// Include stdbool.h to introduce `bool`
#include "stdbool.h"

// Include defines.h and then redefine the SEAL_C_FUNC macro to remove the `extern "C"`
#include "seal/c/defines.h"
#define SEAL_C_FUNC HRESULT

Added to that, I have a file cstddef that just #includes stddef.h (as per your suggestion). However, the next error then came from another file that was missing C++ includes like algorithm or vector, and when I added those as well, I get now these

/home/mschlott/.julia/artifacts/098fe6e2292cbf4947cdf5c2f169f0caf3f691d8/include/SEAL-3.6/seal/util/clang.h:9:34: error: token is not a valid binary operator in a preprocessor subexpression
/home/mschlott/.julia/artifacts/098fe6e2292cbf4947cdf5c2f169f0caf3f691d8/include/SEAL-3.6/seal/util/defines.h:118:1: error: unknown type name 'namespace'
/home/mschlott/.julia/artifacts/098fe6e2292cbf4947cdf5c2f169f0caf3f691d8/include/SEAL-3.6/seal/util/defines.h:118:15: error: expected ';' after top level declarator

Thus it seems like it just goes on and on and on, trying to trick the C++ files into looking like valid C files, and I am not sure I will be able to fix everything from outside (plus, it is very ugly).

I also tried a different tack and passed ("-x", "c++", "-std=c++17") as arguments to LibClang, which got rid of all errors without any tricks. However, this generated a ton of warnings from CBindingGen about macros it was unable to convert and in the end, the file with the generated bindings only contained some macro definitions from defines.h but nothing else. Do you know if invoking LibClang as a C++ parser would work at all, i.e., is it worth to further investigate this? Or is this already part of your effort for v2.0?

P.S.: If it helps and if you'd be willing to have a look at it, I can also push my current work to a branch of SEAL.jl such that you can reproduce my results.

UPDATE Here is the related code JuliaCrypto/SEAL.jl#20

[krrutkow]

Ok, thanks for the access! I should get the new version out sometime in the next week, but it is C-only still. It is totally possible to use LibClang to parse C++ too, I just haven't done that yet. Generating C++ bindings for the SEAL library might be a nice first use case as its mostly a C API, just not proper C.

[krrutkow]

CBinding.jl v1 now reports compilation errors, and a bare minimum support for C++ to support the SEAL library will be available in an upcoming release of it.