[Bug]: Issue when generating ballerina client code with grpc containing oneof and google.profotbuf.Empty #7230
Comments
|
@maxwellb2 Thanks for reporting the issue. It seems like an issue with the protoc-tool when handling the type In the meantime, if you are stuck, you can proceed by making a slight change in your proto definition as shown below. syntax = "proto3";
package test;
service TestService {
rpc TestRPC(RequestMessage) returns (ResponseMessage);
}
enum TestEnum {
ENTRY_ONE = 0;
ENTRY_TWO = 1;
}
message ComplexType {
uint32 foo = 1;
TestEnum bar = 2;
}
message OptionalComplexType {
oneof value {
ComplexType complex_value = 1;
// A field to represent `None`. It's usually left as an empty field.
Empty none = 2;
}
}
message OptionalString {
oneof value {
string name = 1;
// A field to represent `None`. It's usually left as an empty field.
Empty none = 2;
}
}
message RequestMessage {
repeated OptionalComplexType req = 1;
}
message ResponseMessage {
repeated OptionalString resp = 1;
}
message Empty {}Instead of using the |
daneshk
added
the
Reason/EngineeringMistake
|
Thanks for the workaround @daneshk! We will use our own Empty message. |
|
We are using version 2201.10.1 (the current latest version) |
|
The issue will remain open till we do a distribution patch release. |
|
Hi @daneshk I just tried the example proto spec here (with the custom
Unfortunately, I'm still not able to build the client and am seeing the following error: I also tried changing our internal I'm testing this with Ballerina 2201.10.1 on Linux. |
|
@rukmal Based on the stack trace, it seems like there is an error when trying to compile the proto file with the protoc executor( Please execute the following command, providing the directory where the proto file resides and the proto file path. /tmp/protoc-3.21.7-linux-x86_64.exe --proto_path="<absolute directory path where proto files resides>" "<absolute filepath of the root proto file>" --descriptor_set_out="schema-descriptor.desc"For example,
Execute the below command, |
|
Hi @daneshk, I can't directly execute the protoc --proto_path="/home/rukmal/tmp/protoc_test" "/home/rukmal/tmp/protoc_test/service.proto" --descriptor_set_out="schema-descriptor.desc"This is building the descriptor set correctly, and I see the output: Again, I don't think the issue is with the protobuf schema, or my installation of Ballerina. This schema is working as expected with both For reference, I'm using the following shell.nix file on my system to create the Ballerina dev environment: { pkgs ? import <nixpkgs> {} }:
let
version = "2201.10.1"; # Define the version here
codeName = "swan-lake"; # Define the codeName here
ballerina = pkgs.stdenv.mkDerivation {
pname = "ballerina";
inherit version;
src = pkgs.fetchzip {
url = "https://dist.ballerina.io/downloads/${version}/ballerina-${version}-${codeName}.zip";
hash = "sha256-gKxJnoNWYE3ozQ0JvMgHgrg/DCkvFnJqZAecgqvJGq8=";
};
nativeBuildInputs = [ pkgs.makeWrapper ];
installPhase = ''
runHook preInstall
cp -rv distributions/ballerina-${version} $out
runHook postInstall
'';
preFixup = ''
wrapProgram $out/bin/bal --set JAVA_HOME ${pkgs.openjdk}
'';
meta = with pkgs.lib; {
description = "Open-source programming language for the cloud";
license = licenses.asl20;
platforms = pkgs.openjdk.meta.platforms;
};
};
in
pkgs.mkShell {
buildInputs = [
ballerina
pkgs.openjdk
];
shellHook = ''
echo "Ballerina ${version} is ready to use!"
export JAVA_HOME=${pkgs.openjdk}
'';
}We tested this on a MacOS machine and are seeing the same error too. This is the $ protoc --version
libprotoc 24.4 |
|
@rukmal, is it possible for you to share your modified proto schema or something similar that I can use to reproduce the issue, please? It would be really helpful to find out the problem with the Ballerina tool. |
|
I'm using the test schema we discussed above: syntax = "proto3";
package test;
service TestService {
rpc TestRPC(RequestMessage) returns (ResponseMessage);
}
enum TestEnum {
ENTRY_ONE = 0;
ENTRY_TWO = 1;
}
message ComplexType {
uint32 foo = 1;
TestEnum bar = 2;
}
message OptionalComplexType {
oneof value {
ComplexType complex_value = 1;
// A field to represent `None`. It's usually left as an empty field.
Empty none = 2;
}
}
message OptionalString {
oneof value {
string name = 1;
// A field to represent `None`. It's usually left as an empty field.
Empty none = 2;
}
}
message RequestMessage {
repeated OptionalComplexType req = 1;
}
message ResponseMessage {
repeated OptionalString resp = 1;
}
message Empty {}The |
|
Hi @daneshk please disregard the previous comment about not sharing our spec. I've created and have attached a subset of our specification here for testing purposes by the team. Please note that some of these messages are distributed across different files in our setup, and are imported with The imports are not an issue, as we have been using the syntax = "proto3";
package usm;
import "google/protobuf/duration.proto";
import "google/protobuf/empty.proto";
service USMDataService {
// Get unicode interface nodes given a unicode string
rpc GetUnicodeInterfaceNodeIdentifiers(UnicodeInterfaceNodeIdentifiersRequest) returns (UnicodeInterfaceNodeIdentifiersResponse);
// Get a synthesized route from the graph
rpc GetSynthesizedRoute(SynthesisRequest) returns (SynthesisResponse);
}
// Synthesis interface node request
message UnicodeInterfaceNodeIdentifiersRequest {
// Unicode string to fetch interface nodes for
string unicode_string = 1;
}
// Synthesis interface node response
message UnicodeInterfaceNodeIdentifiersResponse {
// Names used from the string to fetch the interface nodes
repeated string names = 1;
// Interface nodes for the unicode string (corresponds to the names above)
repeated OptionalInterfaceNodeIdentifier node_ids = 2;
}
// Synthesis request message
message SynthesisRequest {
// Interface nodes for the synthesis operation
repeated NodeIdentifier interface_nodes = 1;
// (Optional) VoyagerIdentifier of the graph to use for synthesis
optional VoyagerIdentifier voyager_id = 2;
// (Optional) Bidirectional step limit for the synthesis operation
optional uint32 bidirectional_step_limit = 3;
}
// Synthesis response message
message SynthesisResponse {
// Response routes from the synthesis process
repeated RouteSynthesisResponse responses = 1;
// Number of bidirectional steps taken for the synthesis operation
uint32 bidirectional_steps_taken = 2;
// Number of forward nodes scanned
uint32 forward_scanned_count = 3;
// Number of reverse nodes scanned
uint32 reverse_scanned_count = 4;
// Duration of the synthesis operation
google.protobuf.Duration duration = 5;
}
// Optional interface node identifier
message OptionalInterfaceNodeIdentifier {
oneof value {
// ID of the Interface Node
NodeIdentifier node_id = 1;
// A field to represent `None`. It's usually left as an empty field.
google.protobuf.Empty none = 2;
}
}
// Optional interface node name
message OptionalInterfaceNodeName {
oneof value {
// Name of the interface node
string name = 1;
// A field to represent `None`. It's usually left as an empty field.
google.protobuf.Empty none = 2;
}
}
// Route Synthesis response
message RouteSynthesisResponse {
// Route interface nodes
repeated OptionalInterfaceNodeIdentifier interface_nodes = 1;
// Route interface node names
repeated OptionalInterfaceNodeName interface_node_names = 2;
// Number of non-null interface nodes in the route
uint32 interface_node_count = 3;
// Hidden nodes in the route
repeated LongNodeIdentifier hidden_node_long_ids = 4;
// Boolean indicating whether the route is novel, or was present in the loaded data
bool is_novel = 5;
}
// Node type
enum NodeType {
// Error (nodes always have a node type)
UNSPECIFIED = 0;
// Interface node
INTERFACE = 1;
// Hidden node
HIDDEN = 2;
}
// Interface device
enum InterfaceDevice {
// Null device (never this for interface nodes)
NULL = 0;
// Reference device
REFERENCE = 1;
// Unicode device
UNICODE = 2;
}
// Node implementation
enum NodeImplementation {
// Error implementation (LongeNodeIdentifier always has one)
ERROR = 0;
// Base Node
BASE_NODE = 1;
// Collided Node
COLLIDED_NODE = 2;
// Projected Node
PROJECTED_NODE = 3;
}
// Voyager Identifier
message VoyagerIdentifier {
// Run ID of the Voyager Identifier
uint32 run_id = 1;
// Iteration ID of the Voyager Identifier
uint32 iteration_id = 2;
}
// Node Identifier
message NodeIdentifier {
// Type of the node
NodeType node_type = 1;
// Numeric index of the node
uint32 node_idx = 2;
// (Optional) interface device; necessary if node_type is INTERFACE
optional InterfaceDevice device = 3;
}
// Long Node Identifier
message LongNodeIdentifier {
// Node IDs of the Node
repeated NodeIdentifier node_ids = 1;
// Node implementation
NodeImplementation node_impl = 2;
// Voyager ID of the Node (Base graph if this is None)
optional VoyagerIdentifier voyager_id = 3;
}Please note that I already tried replacing I tried creating the schema descriptor file with protoc for this subset of our specification too, and it works as expected. This is the output of running the ballerina GRPC tool against this spec (I deleted the $ bal grpc --mode client --input ./service.proto --output ./
Downloading the protoc executor file - protoc-3.21.7-linux-x86_64.exe
Download successfully completed. Executor file path - /tmp/protoc-3.21.7-linux-x86_64.exe
Successfully extracted the library files.
Oct 07, 2024 2:37:21 PM io.ballerina.protoc.protobuf.cmd.GrpcCmd generateBalFile
SEVERE: An error occurred when generating the proto descriptor.
io.ballerina.protoc.protobuf.exception.CodeGeneratorException: Invalid command syntax. Stream closed
at io.ballerina.protoc.protobuf.utils.BalFileGenerationUtils.handleProcessExecutionErrors(BalFileGenerationUtils.java:110)
at io.ballerina.protoc.protobuf.utils.BalFileGenerationUtils.generateDescriptor(BalFileGenerationUtils.java:83)
at io.ballerina.protoc.protobuf.cmd.DescriptorsGenerator.generateRootDescriptor(DescriptorsGenerator.java:148)
at io.ballerina.protoc.protobuf.cmd.GrpcCmd.generateBalFile(GrpcCmd.java:251)
at io.ballerina.protoc.protobuf.cmd.GrpcCmd.execute(GrpcCmd.java:165)
at java.base/java.util.Optional.ifPresent(Optional.java:178)
at io.ballerina.cli.launcher.Main.main(Main.java:58)
Caused by: java.io.IOException: Stream closed
at java.base/java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:168)
at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:270)
at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:313)
at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:188)
at java.base/java.io.InputStreamReader.read(InputStreamReader.java:177)
at java.base/java.io.BufferedReader.fill(BufferedReader.java:162)
at java.base/java.io.BufferedReader.readLine(BufferedReader.java:329)
at java.base/java.io.BufferedReader.readLine(BufferedReader.java:396)
at io.ballerina.protoc.protobuf.utils.BalFileGenerationUtils.handleProcessExecutionErrors(BalFileGenerationUtils.java:105)
... 6 more
An error occurred when generating the proto descriptor. Invalid command syntax. Stream closed |
|
Thanks alot @rukmal. I just replaced syntax = "proto3";
package usm;
import "google/protobuf/duration.proto";
service USMDataService {
// Get unicode interface nodes given a unicode string
rpc GetUnicodeInterfaceNodeIdentifiers(UnicodeInterfaceNodeIdentifiersRequest) returns (UnicodeInterfaceNodeIdentifiersResponse);
// Get a synthesized route from the graph
rpc GetSynthesizedRoute(SynthesisRequest) returns (SynthesisResponse);
}
// Synthesis interface node request
message UnicodeInterfaceNodeIdentifiersRequest {
// Unicode string to fetch interface nodes for
string unicode_string = 1;
}
// Synthesis interface node response
message UnicodeInterfaceNodeIdentifiersResponse {
// Names used from the string to fetch the interface nodes
repeated string names = 1;
// Interface nodes for the unicode string (corresponds to the names above)
repeated OptionalInterfaceNodeIdentifier node_ids = 2;
}
// Synthesis request message
message SynthesisRequest {
// Interface nodes for the synthesis operation
repeated NodeIdentifier interface_nodes = 1;
// (Optional) VoyagerIdentifier of the graph to use for synthesis
optional VoyagerIdentifier voyager_id = 2;
// (Optional) Bidirectional step limit for the synthesis operation
optional uint32 bidirectional_step_limit = 3;
}
// Synthesis response message
message SynthesisResponse {
// Response routes from the synthesis process
repeated RouteSynthesisResponse responses = 1;
// Number of bidirectional steps taken for the synthesis operation
uint32 bidirectional_steps_taken = 2;
// Number of forward nodes scanned
uint32 forward_scanned_count = 3;
// Number of reverse nodes scanned
uint32 reverse_scanned_count = 4;
// Duration of the synthesis operation
google.protobuf.Duration duration = 5;
}
// Optional interface node identifier
message OptionalInterfaceNodeIdentifier {
oneof value {
// ID of the Interface Node
NodeIdentifier node_id = 1;
// A field to represent `None`. It's usually left as an empty field.
Empty none = 2;
}
}
// Optional interface node name
message OptionalInterfaceNodeName {
oneof value {
// Name of the interface node
string name = 1;
// A field to represent `None`. It's usually left as an empty field.
Empty none = 2;
}
}
// Route Synthesis response
message RouteSynthesisResponse {
// Route interface nodes
repeated OptionalInterfaceNodeIdentifier interface_nodes = 1;
// Route interface node names
repeated OptionalInterfaceNodeName interface_node_names = 2;
// Number of non-null interface nodes in the route
uint32 interface_node_count = 3;
// Hidden nodes in the route
repeated LongNodeIdentifier hidden_node_long_ids = 4;
// Boolean indicating whether the route is novel, or was present in the loaded data
bool is_novel = 5;
}
// Node type
enum NodeType {
// Error (nodes always have a node type)
UNSPECIFIED = 0;
// Interface node
INTERFACE = 1;
// Hidden node
HIDDEN = 2;
}
// Interface device
enum InterfaceDevice {
// Null device (never this for interface nodes)
NULL = 0;
// Reference device
REFERENCE = 1;
// Unicode device
UNICODE = 2;
}
// Node implementation
enum NodeImplementation {
// Error implementation (LongeNodeIdentifier always has one)
ERROR = 0;
// Base Node
BASE_NODE = 1;
// Collided Node
COLLIDED_NODE = 2;
// Projected Node
PROJECTED_NODE = 3;
}
// Voyager Identifier
message VoyagerIdentifier {
// Run ID of the Voyager Identifier
uint32 run_id = 1;
// Iteration ID of the Voyager Identifier
uint32 iteration_id = 2;
}
// Node Identifier
message NodeIdentifier {
// Type of the node
NodeType node_type = 1;
// Numeric index of the node
uint32 node_idx = 2;
// (Optional) interface device; necessary if node_type is INTERFACE
optional InterfaceDevice device = 3;
}
// Long Node Identifier
message LongNodeIdentifier {
// Node IDs of the Node
repeated NodeIdentifier node_ids = 1;
// Node implementation
NodeImplementation node_impl = 2;
// Voyager ID of the Node (Base graph if this is None)
optional VoyagerIdentifier voyager_id = 3;
}
message Empty {}
I will try to reproduce the issue in a different machine(linux machine) and get back to you soon. |
|
I just tried the above spec on my machine, and it's not working unfortunately... I attached the nix configuration I'm using to build on linux here, so please let me know if there are any other questions I can help with. The Ballerina work (handled by @maxwellb2) is luckily enough happening on a Mac, so we will try to build the updated spec (replacing This is the system I'm using: $ uname -a
Linux mercury 6.11.0 ballerina-platform/ballerina-release#1-NixOS SMP PREEMPT_DYNAMIC Sun Sep 15 14:57:56 UTC 2024 x86_64 GNU/Linux |
|
@daneshk I was able to generate the ballerina client code without error on MacOS using the custom |
|
This fix will be released with 2201.10.4 patch |
|
Thanks @daneshk, I'll work on getting a reproduction example in a dockerfile for you |
Description
I am seeing a null pointer error when attempting to generate off of my GRPC spec. I wrote this demo GRPC spec similar to my internal set up to replicate the issue.
Steps to Reproduce
Create the
service.protofile as follows:When I test with:
$ bal grpc --mode client --input ../api/service.proto --output .I get the following error:
Affected Version(s)
No response
OS, DB, other environment details and versions
No response
Related area
-> gRPC Tool
Related issue(s) (optional)
No response
Suggested label(s) (optional)
No response
Suggested assignee(s) (optional)
No response
The text was updated successfully, but these errors were encountered: