Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Orchestration Trigger input binding fails with "Orchestration history state was either missing from the input or not a string value" #2804

Open
EmilDamsbo opened this issue Apr 26, 2024 · 12 comments · May be fixed by #2950
Labels

Comments

@EmilDamsbo
Copy link

Description

Running into the uncaught exception "Orchestration history state was either missing from the input or not a string value".

Setup

  1. Have two functions F1 and F2, the first is an HTTP trigger and the other is an orchestrator function
  2. HTTP trigger F1 schedules a new orchestration instance for F2, and adds an additional input object
  3. Before my debugger enters F2, the input binding fails:
[2024-04-26T21:46:38.116Z] An error occurred while executing the orchestrator function 'DispatchJob'.
[2024-04-26T21:46:38.117Z] Result: An error occurred while executing the orchestrator function 'DispatchJob'.
Exception: System.InvalidOperationException: Orchestration history state was either missing from the input or not a string value.
[2024-04-26T21:46:38.118Z]    at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.DurableTaskFunctionsMiddleware.RunOrchestrationAsync(FunctionContext context, BindingMetadata triggerBinding, FunctionExecutionDelegate next) in /_/src/Worker.Extensions.DurableTask/DurableTaskFunctionsMiddleware.cs:line 55
[2024-04-26T21:46:38.119Z]    at Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore.FunctionsHttpProxyingMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\extensions\Worker.Extensions.Http.AspNetCore\src\FunctionsMiddleware\FunctionsHttpProxyingMiddleware.cs:line 34
[2024-04-26T21:46:38.120Z]    at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.FunctionsOrchestrator.EnsureSynchronousExecution(FunctionContext functionContext, FunctionExecutionDelegate next, FunctionsOrchestrationContext orchestrationContext) in /_/src/Worker.Extensions.DurableTask/FunctionsOrchestrator.cs:line 81
[2024-04-26T21:46:38.121Z]    at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.FunctionsOrchestrator.RunAsync(TaskOrchestrationContext context, Object input) in /_/src/Worker.Extensions.DurableTask/FunctionsOrchestrator.cs:line 51Stack:    at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.DurableTaskFunctionsMiddleware.RunOrchestrationAsync(FunctionContext context, BindingMetadata triggerBinding, FunctionExecutionDelegate next) in /_/src/Worker.Extensions.DurableTask/DurableTaskFunctionsMiddleware.cs:line 55
[2024-04-26T21:46:38.122Z]    at Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore.FunctionsHttpProxyingMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\extensions\Worker.Extensions.Http.AspNetCore\src\FunctionsMiddleware\FunctionsHttpProxyingMiddleware.cs:line 34
[2024-04-26T21:46:38.123Z]    at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.FunctionsOrchestrator.EnsureSynchronousExecution(FunctionContext functionContext, FunctionExecutionDelegate next, FunctionsOrchestrationContext orchestrationContext) in /_/src/Worker.Extensions.DurableTask/FunctionsOrchestrator.cs:line 81
[2024-04-26T21:46:38.124Z]    at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.FunctionsOrchestrator.RunAsync(TaskOrchestrationContext context, Object input) in /_/src/Worker.Extensions.DurableTask/FunctionsOrchestrator.cs:line 51.
[2024-04-26T21:46:38.676Z] Executed 'Functions.DispatchJob' (Failed, Id=56e78500-0d6f-4448-880e-47e1b1d9571a, Duration=7706ms)
[2024-04-26T21:46:38.678Z] System.Private.CoreLib: Exception while executing function: Functions.DispatchJob. Microsoft.Azure.WebJobs.Extensions.DurableTask: Orchestration history state was either missing from the input or not a string value.
[2024-04-26T21:46:38.681Z] c663be9b1bde4b788c83378174a3bb1c: Function 'DispatchJob (Orchestrator)' failed with an error. Reason: Orchestration history state was either missing from the input or not a string value.. IsReplay: False. State: Failed. RuntimeStatus: Failed. HubName: TestHubName. AppName: . SlotName: . ExtensionVersion: 2.13.2. SequenceNumber: 5. TaskEventId: -1
  1. I debugged, and it seems the orchestration trigger input binding is somehow binding a full object rather than a string in here:
    a.. https://github.com/Azure/azure-functions-durable-extension/blob/dev/src/Worker.Extensions.DurableTask/DurableTaskFunctionsMiddleware.cs#L49
    b. triggerInputData is expected to have a .value that is either null or string, but in my case it's a {Microsoft.Azure.Functions.Worker.Extensions.DurableTask.FunctionsOrchestrationContext}. The whole triggerInputData object after binding:
{
    "BindingMetadata": {
        "Name": "context",
        "Type": "orchestrationTrigger",
        "Direction": 0
    },
    "Value": {
        "IsAccessed": false,
        "Name": {
            "Name": "DispatchJob",
            "Version": ""
        },
        "InstanceId": "262ca4d6610f479da9e344d0681be433",
        "CurrentUtcDateTime": "2024-04-26T21:47:53.5989272Z",
        "IsReplaying": false,
        "Parent": null,
        "Entities": {}
    }
}

Expected behavior

I expect the orchestration trigger to bind successfully, and start executing my function code.

Actual behavior

Function 'DispatchJob (Orchestrator)' failed with an error. Reason: Orchestration history state was either missing from the input or not a string value.

Relevant source code snippets

// insert code snippet here

Known workarounds

I haven't encountered any workarounds yet.

App Details

  • Durable Functions extension version (e.g. v1.8.3): Microsoft.Azure.Functions.Worker.Extensions.DurableTask v 1.1.2
  • Azure Functions runtime version (1.0 or 2.0): V4,0
  • Programming language used: C# ,NET 8 (isolated worker)

Screenshots

image

@cgillum
Copy link
Collaborator

cgillum commented Apr 26, 2024

This is incredibly strange. Does your app contain any customizations, like middleware, injected services, etc., - i.e. things that would normally be done in the Program.cs file?

I can't think of any reason why the trigger binding value would be an object, and why it would be a fully populated FunctionsOrchestrationContext of all things as shown in your screenshot.

It would be great if you could provide a minimal repro app for us to look at.

@EmilDamsbo
Copy link
Author

We have quite a large app built around this, and there is potentially some middlewares at play. I'll work to produce a minimum repro of this within the week.
Some additional context would also be that the caller to the orchestrator is an abstract generic class which the entrypoint function inherits.

So something like

public abstract class FunctionBase<TRequest>
{
    protected async Task<IActionResult> DispatchJobFromRequest(TRequest request, DurableTaskClient context)
    {
        [...]
        // using the generated extension method for class-based orchestrators, but this reproed without it too
        // https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-dotnet-isolated-overview#source-generator-and-class-based-activities-and-orchestrations
        await context.ScheduleNewDispatchJobInstanceAsync(request);
        [...]
        return new ObjectResult(...])
    }
}

public class RequestHandler : FunctionBase<MyRequest>
{
    [Function]
    public async Task<IActionResult> HandleRequest([HttpTrigger([...])] HttpRequest request, [DurableClient] DurableTaskClient context)
    {
        MyRequest convertedRequest = [convert HttpRequest to MyRequest somehow]

        return this.DispatchJobFromRequest(convertedRequest, context);
    }
}

And the orchestrator function DispatchJob is defined elsewhere not inheriting from the FunctionBase.
This is the source-generated extension for the DurableTaskClient (censored the namespace and part of the object name), and I don't see any reason why it would pass the fully-formed FunctionsOrchestrationContext here.
image

@EmilDamsbo
Copy link
Author

It seems to be related to inheritance when passing objects as an input parameter to the ScheduleNewOrchestrationInstanceAsync-method, because I now have a not-entirely-minimal repro. I will try to upload it tomorrow once I've removed any traces of company IP.

@EmilDamsbo
Copy link
Author

I found the offending line. I had previously followed some of the advice in this thread #2527

So the functions' HostBuilder looked like this:

var host = new HostBuilder()
    .ConfigureFunctionsWebApplication(
        builder => new DurableTaskExtensionStartup().Configure(builder)
    )
    .ConfigureServices(services =>
    {
        services.AddHttpClient();
    })
    .Build();

host.Run();

Now this manual call to DurableTaskExtensionStartup.Configure had fixed some of my source generation issues, however it is also causing some unintended middleware issues unbeknownst to me. So when I removed line 3 there with the explicit DurableTaskExtensionStartup.Configure()-call, I stopped encountering the issue.

Repository here demonstrates the issue. it's about as minimal as it gets, it's based on one of the public templates:
https://github.com/EmilDamsbo/OrchestrationInputBindingRepro
I recommend adding some kind of mechanism in DurableTaskExtensionStartup to ensure there isn't duplicated middlewares.

@cgillum
Copy link
Collaborator

cgillum commented Apr 30, 2024

@jviau can you take a look? This seems related to another issue that you've helped users with before.

@cgillum cgillum added P2 Priority 2 and removed Needs: Triage 🔍 labels Apr 30, 2024
@jviau
Copy link
Contributor

jviau commented May 1, 2024

@EmilDamsbo I imagine what happened is your source generation was eventually fixed, so that line: new DurableTaskExtensionStartup().Configure(builder) was now being implicitly called for you. Having that manual call wound up double registering our types, leading to some undefined behavior.

@EmilDamsbo
Copy link
Author

I agree that this is most likely what happened after all my experimentation. Would it be possible to catch this DI double-registration on the durable extensions side or should I open an issue elswhere?

@jviau
Copy link
Contributor

jviau commented May 2, 2024

We could do some work on the durable side to make this registration re-enterable.

@lilyjma
Copy link
Contributor

lilyjma commented Aug 20, 2024

@EmilDamsbo - if you're still seeing the issue, can you provide a latest minimal repro please? We'd like to know what exactly is the trigger of the exception. Thanks!

@cgillum
Copy link
Collaborator

cgillum commented Oct 26, 2024

Closing since the underlying issue has been fixed.

@cgillum cgillum closed this as completed Oct 26, 2024
@MarcinJuraszek
Copy link

MarcinJuraszek commented Oct 29, 2024

This issue still exists when using the new IHostApplicationBuilder logic being introduced right now (currently in -preview2 versions of Azure Functions). The documentation says to setup the host like this:

var builder = FunctionsApplication.CreateBuilder(args);

// (...) removed because doesn't seem to matter?

var host = builder.Build();

And later calls for builder.ConfigureFunctionsWebApplication() to be used to enable ASP.NET Core integration:

If you want to use ASP.NET Core integration, call builder.ConfigureFunctionsWebApplication().

However, the moment you add that ConfigureFunctionsWebApplication() call, Durable Framework starts failing with:

Exception: System.InvalidOperationException: Orchestration history state was either missing from the input or not a string value.

To reproduce, simply create a new Azure Functions app with a Durable Function in it, update the SDKs to preview versions and change the host setup from

var host = new HostBuilder()
    .ConfigureFunctionsWebApplication()
    .ConfigureServices(services =>
    {
        services.AddApplicationInsightsTelemetryWorkerService();
        services.ConfigureFunctionsApplicationInsights();
    })
    .Build();

host.Run();

to

var hostBuilder = FunctionsApplication.CreateBuilder(args);

hostBuilder.ConfigureFunctionsWebApplication();

var host = hostBuilder.Build();

host.Run();

The versions I've used to reproduce it:

<PackageReference Include="Microsoft.ApplicationInsights.WorkerService" Version="2.22.0" />
<PackageReference Include="Microsoft.Azure.Functions.Worker" Version="2.0.0-preview4" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.ApplicationInsights" Version="1.4.0" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.DurableTask" Version="1.1.7" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http" Version="3.2.0" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" Version="2.0.0-preview4" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="2.0.0-preview2" />

@cgillum
Copy link
Collaborator

cgillum commented Oct 30, 2024

Reactivating per @MarcinJuraszek as it seems like a new regression might have been introduced in some of the current preview nuget packages.

@jviau - any ideas what might be going on here?

@cgillum cgillum reopened this Oct 30, 2024
@cgillum cgillum added P1 Priority 1 dotnet-isolated and removed P2 Priority 2 labels Oct 30, 2024
@jviau jviau linked a pull request Oct 30, 2024 that will close this issue
15 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants