Tuning webhook performance parameters in Jira Data Center
Platform Notice: Data Center - This article applies to Atlassian products on the Data Center platform.
Note that this knowledge base article was created for the Data Center version of the product. Data Center knowledge base articles for non-Data Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
In Jira 9 and earlier, webhooks are processed synchronously, with the most computationally intensive work performed during the HTTP request. Administrators can opt-in to asynchronous webhooks by adding a feature flag and performing a rolling restart.
Starting in Jira 10.0.0, asynchronous webhooks have become the default and only option. This change significantly enhances the end-user experience, as processing is now handled by a dedicated thread pool instead of being tied to the user request.
However, in both Jira 9 and 10, environments with exceptional webhook usage may exceed the guardrails set by the default parameters. The following parameters allow you to customize these limits to meet your organization’s needs.
Environment
Jira Data Center 10.3.5 LTS / 10.6.0 and later.
These parameters are unavailable in Jira 10.0.0-10.3.4 LTS / 10.4.0-10.5.0. Please upgrade to the current LTS bug fix release or the latest feature release. For more details, please consult the Jira Software release notes and Upgrading Jira applications.
For all versions of Jira Data Center 9, please review the workarounds on JRASERVER-74492 - Getting issue details... STATUS .
Diagnosis
Problem 1: processing queue
You observe the following error in your atlassian-jira.log file:
2025-03-25 14:05:00,000+0000 http-nio-8080-exec-1 ERROR charlie 1x1x1 abcdef 0.0.0.0 /secure/AjaxIssueAction.jspa [c.a.event.internal.AsynchronousAbleEventDispatcher] There was an exception thrown trying to dispatch event [com.atlassian.jira.event.issue.IssueEvent@cafebabe[issue=TIS-1,comment=<null>,worklog=<null>,changelog=[GenericEntity:ChangeGroup][issue,10000][author,charlie][created,2025-03-25 14:04:59.00][id,10000],eventTypeId=2,sendMail=true,params={eventsource=action, baseurl=https://jira.example.com},subtasksUpdated=true,spanningOperation=Optional.empty]] from the invoker [com.atlassian.event.internal.ComparableListenerInvoker@cafebabe]
java.lang.RuntimeException: Task com.atlassian.webhooks.internal.DefaultWebhookService$$Lambda$1234/0x0000000cafebebe@cafebabe rejected from java.util.concurrent.ThreadPoolExecutor@cafebabe[Running, pool size = 5, active threads = 5, queued tasks = 250, completed tasks = 80000]. Listener: com.atlassian.jira.plugins.webhooks.spi.JiraWebhookEventPublisher event: com.atlassian.jira.event.issue.IssueEventSimplified:
Task rejected from ThreadPoolExecutor[Running, pool size = 5, active threads = 5, queued tasks = 250, completed tasks = 80000]Problem 2: HTTP dispatch queue
You observe the following error in your atlassian-jira.log file:
2025-03-25 14:05:00,000+0000 webhook-dispatcher:thread-1 WARN anonymous [c.a.w.internal.publish.DefaultWebhookDispatcher] Could not dispatch jira:issue_updated webhook to <WEBHOOK_URL>; a maximum of 500 dispatches are already in flight
2025-03-25 14:05:00,000+0000 webhook-dispatcher:thread-1 DEBUG anonymous [c.a.w.internal.publish.DefaultWebhookDispatcher] Skipping webhook invocation [f63f8397-7328-48d8-9c22-2a4eb2e30954] to <WEBHOOK_URL> (Too many webhook dispatches already in flight)Simplified:
Could not dispatch <EVENT_NAME> webhook ... a maximum of <MAX_IN_FLIGHT_DISPATCHES_COUNT> dispatches are already in flight
Too many webhook dispatches already in flightProblem 3: failure exponential backoff
You observe the following error in your atlassian-jira.log file:
2025-03-25 14:05:00,000+0000 webhook-dispatcher:thread-1 WARN anonymous [c.a.w.internal.history.DefaultInvocationHistoryService] Request to <WEBHOOK_URL> resulted in an error: Webhook failed too many times. Skipping this webhook for the next 11933msSimplified:
Webhook failed too many times. Skipping this webhook for the next <DELAY>msSolution
Tuning webhook event/JQL configuration
Have you considered adjusting your webhook configuration within Jira? Reducing webhook scope will decrease workload and avoid limits. Please consult Best practices on working with webhooks in Jira Data Center for recommendations.
These parameters differ from Jira 9, which used JVM arguments instead of application properties. Any webhook JVM arguments carried over from Jira 9 will not affect Jira 10.
The following parameters can be added to the jira-config.properties file (Advanced Jira application configuration).
Parameters for Problem 1
| Title | Property | Description | Default Value |
|---|---|---|---|
Queue size | webhooks.executor.queue.size | The size of the webhook processing queue. Any additional pending requests will be dropped and unrecoverable. | 250 |
Executor thread pool size | webhooks.executor.thread.count | The number of webhook-dispatcher worker threads. Additional threads may increase CPU usage but will improve queue throughput. | 5 |
Enqueue timeout |
| How long (milliseconds) a webhook event-producing thread will wait to add to a full queue before timing out. Setting this value too high could force users to wait indefinitely for their HTTP request to return. | 100 |
Parameters for Problem 2
| Title | Property | Description | Default Value |
|---|---|---|---|
Max in-flight webhooks | webhooks.max.in.flight.dispatches | The maximum number of webhooks that can be dispatched concurrently. This includes webhook requests that are currently in transit over the network, as well as those waiting for HTTP connections. Any additional requests will be dropped and unrecoverable. Before increasing this limit, consider tuning the I/O thread pool size, max number of HTTP connections per host, and max HTTP connections. | 500 |
Dispatch timeout | webhooks.dispatch.timeout.millis | How long (milliseconds) a If unsuccessful, the webhook will be dropped. | 250 |
HTTP connections per host |
| The maximum number of concurrent HTTP connections allowed per host for dispatching webhooks. | 5 |
Total HTTP connections |
| The maximum number of concurrent HTTP connections for dispatching webhooks. A higher number of HTTP connections with insufficient I/O threads will increase the number of in-flight webhooks. | 200 |
I/O thread pool size |
| The number of HTTP I/O threads. Additional threads may increase CPU usage but will improve queue throughput. | 3 |
Parameters for Problem 3
| Title | Property | Description | Default Value |
|---|---|---|---|
Backoff trigger count | webhooks.backoff.trigger.count | The maximum number of consecutive failures before a webhook is deemed unhealthy and subsequently skipped. A single successful dispatch will mark the webhook as healthy. | 5 |
Backoff initial delay | webhooks.backoff.initial.delay.seconds | How long (seconds) an unhealthy webhook is skipped when first deemed unhealthy. | 10 |
Backoff exponent |
| An unhealthy webhook is skipped with an initial delay. If the webhook continues to fail, this duration will keep on increasing based on this delay exponent up to the max delay or until the webhook is healthy. The backoff delay is computed as:
up to the configured max delay. | 1.2 |
Backoff max delay |
| The maximum duration for which an unhealthy webhook is skipped. | 2 |