Task Failures In Detail

Task processing can encounter failures at various stages within the Service Executor. Here are the key stages where failures may occur and how to handle them:

Task Preparation: During the setup phase before the actual task execution begins.
Task Execution: While the task is running, which can be due to:
- An exception being thrown within the task code.
- The task exceeding its defined timeout limit.
Task Finalization: When serializing the output and acknowledging the message to the broker.

Each of these failure types requires different handling strategies, which are detailed in the following sections.

Exception During Task Preparation

The task preparation phase includes:

Instantiating the Service class
Deserializing the method arguments
Processing the (optional) getTimeoutSeconds function, if an execution timeout is set

If any exception occurs during this phase, the task is immediately marked as failed, and this information is sent to the workflow that dispatched the task. Retrying is not attempted because these operations should be deterministic. Additionally, the message is acknowledged to prevent reprocessing.

These errors typically stem from issues in the implementation code of the Services during the development phase. If it happens in production, you should:

Deploy a fix for the underlying code issue.
Retry the failed tasks once the fix is in place.

Exception During Task Execution

An exception during the execution occurs:

If the Service implementation throws this exception.
When the execution runs for longer than the defined execution timeout.

When an exception is thrown by the Service implementation during the execution of a task:

The exception is caught and logged (as a warning).
The task is automatically scheduled for retry, based on its retry policy.
The underlying message is acknowledged.

If the maximum number of retries is reached, the task is immediately marked as failed, logged as an error, and this information is sent to the workflow that dispatched the task.

In this situation, you should:

Deploy a fix for the underlying code issue, if any.
Retry the failed tasks once the fix is in place.

The workflow that dispatched a task remains unaware of any retry attempts occurring for that task. From the workflow's perspective, it only receives the final outcome: either the task has succeeded after potentially multiple retry attempts, or it has ultimately failed once all retry attempts have been exhausted. This abstraction allows the workflow to focus on the overall task completion status rather than the intricacies of the retry mechanism.

Exception During Task Finalization

The task finalization phase includes:

Serializing the output of the task.
Sending the result to the workflow that requested the task.

If any exception occurs during the serialization of the output, the task is immediately marked as failed, and this information is sent to the workflow that dispatched the task. Retrying is not attempted because the operation is deterministic. Additionally, the message is acknowledged to prevent reprocessing.

In this case, you should:

Deploy a fix for the serialization issue.
Retry the failed tasks once the fix is in place.