[Azure] Add AMQP-over-WebSockets support to Event Hub integrations

packages/azure_ai_foundry/_dev/build/docs/README.md

@@ -36,6 +36,23 @@ Refer to the [Azure Logs](https://docs.elastic.co/integrations/azure) page for m
 
 **Authentication (Event Hub):** The Event Hub input supports two authentication methods: **connection string** (default) and **client secret** (Microsoft Entra ID). For setup steps, required RBAC roles (Azure Event Hubs Data Receiver, Storage Blob Data Contributor), and configuration options, see the [Azure Logs integration](https://docs.elastic.co/integrations/azure) or [Filebeat azure-eventhub input](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-azure-eventhub.html) documentation.
 
+#### AMQP-over-WebSockets
+
+By default, the integration uses AMQP ports `5671` and `5672` to communicate with Event Hubs. If these ports are blocked, set **Event Hubs transport protocol** to **AMQP-over-WebSockets** in the advanced options. This tunnels AMQP over HTTPS on port `443`.
+
+This option requires the Event Hub processor v2 (**Processor version** set to `v2`, which is the default) and Elastic Agent 9.2.4 or later.
+
+#### Proxy support
+
+Proxy support is optional and requires **Event Hubs transport protocol** set to **AMQP-over-WebSockets**.
+
+To enable it:
+
+1. In the advanced options, set **Event Hubs transport protocol** to **AMQP-over-WebSockets**.
+2. Define the `HTTPS_PROXY` environment variable for the Elastic Agent process, for example `HTTPS_PROXY=http://proxy.example.com:8080`. Elastic Agent routes both Event Hubs and Storage Account traffic through the proxy.
+
+This requires processor v2 and Elastic Agent 9.2.4 or later.
+
 #### Native logging
 
 The Microsoft Foundry provides native logging and monitoring to track the telemetry of the service. The `Audit` and `RequestResponse` log categories come under the native logging. However, the default logging doesn't log the inputs and outputs of the service. This is useful to ensure that the services operates as expected.
@@ -115,4 +132,4 @@ For more details on ECS fields, check the [ECS Field Reference](https://www.elas
 {{fields "metrics"}}
 
 ## Alerting Rule Template
-{{alertRuleTemplates}}
+{{alertRuleTemplates}}

packages/azure_ai_foundry/changelog.yml

@@ -1,3 +1,14 @@
+- version: "0.13.0"
+  changes:
+    - description: Add support for AMQP-over-WebSockets transport protocol and proxy configuration.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/20559
+    - description: Add the Event Hub processor version option and switch the default processor version to v2.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/20559
+    - description: Add the remaining Event Hub processor v2 options (processor update interval, processor start position, partition receive timeout, partition receive count, and checkpoint migration).
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/20559
 - version: "0.12.0"
   changes:
     - description: Publish Agentless to Elastic Managed integrations name change.

packages/azure_ai_foundry/data_stream/logs/agent/stream/azure-eventhub.yml.hbs

@@ -43,6 +43,25 @@ storage_account_key: {{storage_account_key}}
 {{#if resource_manager_endpoint}}
 resource_manager_endpoint: {{resource_manager_endpoint}}
 {{/if}}
+{{#if processor_version}}
+processor_version: {{processor_version}}
+{{/if}}
+migrate_checkpoint: {{migrate_checkpoint}}
+{{#if processor_update_interval}}
+processor_update_interval: {{processor_update_interval}}
+{{/if}}
+{{#if processor_start_position}}
+processor_start_position: {{processor_start_position}}
+{{/if}}
+{{#if partition_receive_timeout}}
+partition_receive_timeout: {{partition_receive_timeout}}
+{{/if}}
+{{#if partition_receive_count}}
+partition_receive_count: {{partition_receive_count}}
+{{/if}}
+{{#if transport}}
+transport: {{transport}}
+{{/if}}
 tags:
 {{#if preserve_original_event}}
   - preserve_original_event

packages/azure_ai_foundry/data_stream/logs/manifest.yml

@@ -113,6 +113,108 @@
         description: >-
           (Optional when **Authentication Type** is **Client Secret**) Microsoft Entra ID authority endpoint. Defaults to https://login.microsoftonline.com (Azure Public Cloud).
           Change for other Azure environments: Azure Government (https://login.microsoftonline.us), Azure China (https://login.chinacloudapi.cn), or Azure Germany (https://login.microsoftonline.de).
+      - name: processor_version
+        type: select
+        title: Processor version
+        multi: false
+        required: false
+        show_user: false
+        default: v2
+        options:
+          - text: v1 (legacy)
+            value: v1
+          - text: v2
+            value: v2
+        description: >
+          The processor version that the integration should use. Possible values are v1 (legacy) and v2 (recommended).
+          The v2 event hub processor is recommended for typical use cases. Defaults to v2.
+      - name: processor_update_interval
+        type: text
+        title: Processor update interval
+        multi: false
+        required: false
+        show_user: false
+        default: 10s
+        description: >-
+          (Processor v2 only) How often the processor should attempt to claim partitions.
+
+          Default is `10` seconds.
+      - name: processor_start_position
+        type: select
+        title: Processor start position
+        multi: false
+        required: false
+        show_user: false
+        default: earliest
+        options:
+          - text: earliest
+            value: earliest
+          - text: latest
+            value: latest
+        description: >-
+          (Processor v2 only) Controls from what position in the event hub the processor should start processing messages for all partitions.
+
+          Possible values are `earliest` and `latest`.
+
+          `earliest` starts processing messages from the last checkpoint, or the beginning of the event hub if no checkpoint is available.
+
+          `latest` starts processing messages from the latest event in the event hub and continues to process new events as they arrive.
+
+          Default is `earliest`.
+      - name: partition_receive_timeout
+        type: text
+        title: Partition receive timeout
+        multi: false
+        required: false
+        show_user: false
+        default: 5s
+        description: >-
+          (Processor v2 only) Maximum time to wait before processing the messages received from the event hub.
+
+          The partition consumer waits up to a "receive count" or a "receive timeout", whichever comes first.
+
+          Default is `5` seconds.
+      - name: partition_receive_count
+        type: text
+        title: Partition receive count
+        multi: false
+        required: false
+        show_user: false
+        default: 100
+        description: >-
+          (Processor v2 only) Maximum number of messages from the event hub to wait for before processing them.
+
+          The partition consumer waits up to a "receive count" or a "receive timeout", whichever comes first.
+
+          Default is `100` messages.
+      - name: transport
+        type: select
+        title: Event Hubs transport protocol
+        multi: false
+        required: true
+        show_user: false
+        default: amqp
+        options:
+          - text: AMQP
+            value: amqp
+          - text: AMQP-over-WebSockets
+            value: websocket
+        description: >
+          (Processor v2 only) The transport protocol used to connect to Event Hubs.
+          Supported values are AMQP and AMQP-over-WebSockets. Defaults to AMQP.
+      - name: migrate_checkpoint
+        type: bool
+        title: Migrate checkpoint information
+        multi: false
+        required: false
+        show_user: false
+        default: true
+        description: >-
+          (Processor v2 only) Flag to control if the processor should perform the checkpoint information migration from processor v1 to v2 at startup.
+
+          The checkpoint migration converts the checkpoint information from the v1 format to the v2 format, so the processor resumes from where v1 left off instead of reprocessing the event hub retention window.
+
+          Default is `true`.
       - name: preserve_original_event
         required: true
         show_user: true

packages/azure_ai_foundry/docs/README.md

@@ -36,6 +36,23 @@ Refer to the [Azure Logs](https://docs.elastic.co/integrations/azure) page for m
 
 **Authentication (Event Hub):** The Event Hub input supports two authentication methods: **connection string** (default) and **client secret** (Microsoft Entra ID). For setup steps, required RBAC roles (Azure Event Hubs Data Receiver, Storage Blob Data Contributor), and configuration options, see the [Azure Logs integration](https://docs.elastic.co/integrations/azure) or [Filebeat azure-eventhub input](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-azure-eventhub.html) documentation.
 
+#### AMQP-over-WebSockets
+
+By default, the integration uses AMQP ports `5671` and `5672` to communicate with Event Hubs. If these ports are blocked, set **Event Hubs transport protocol** to **AMQP-over-WebSockets** in the advanced options. This tunnels AMQP over HTTPS on port `443`.
+
+This option requires the Event Hub processor v2 (**Processor version** set to `v2`, which is the default) and Elastic Agent 9.2.4 or later.
+
+#### Proxy support
+
+Proxy support is optional and requires **Event Hubs transport protocol** set to **AMQP-over-WebSockets**.
+
+To enable it:
+
+1. In the advanced options, set **Event Hubs transport protocol** to **AMQP-over-WebSockets**.
+2. Define the `HTTPS_PROXY` environment variable for the Elastic Agent process, for example `HTTPS_PROXY=http://proxy.example.com:8080`. Elastic Agent routes both Event Hubs and Storage Account traffic through the proxy.
+
+This requires processor v2 and Elastic Agent 9.2.4 or later.
+
 #### Native logging
 
 The Microsoft Foundry provides native logging and monitoring to track the telemetry of the service. The `Audit` and `RequestResponse` log categories come under the native logging. However, the default logging doesn't log the inputs and outputs of the service. This is useful to ensure that the services operates as expected.
@@ -504,3 +521,4 @@ The following alert rule templates are available:
 
 
 
+

packages/azure_ai_foundry/manifest.yml

@@ -1,7 +1,7 @@
 format_version: 3.4.0
 name: azure_ai_foundry
 title: "Microsoft Foundry"
-version: "0.12.0"
+version: "0.13.0"
 source:
   license: "Elastic-2.0"
 description: "Collects Microsoft Foundry logs and metrics"

packages/azure_app_service/_dev/build/docs/README.md

@@ -23,6 +23,23 @@ Refer to the [Azure Logs](https://docs.elastic.co/integrations/azure) page for m
 
 **Authentication (Event Hub):** The Event Hub input supports two authentication methods: **connection string** (default) and **client secret** (Microsoft Entra ID). For setup steps, required RBAC roles (Azure Event Hubs Data Receiver, Storage Blob Data Contributor), and configuration options, see the [Azure Logs integration](https://docs.elastic.co/integrations/azure) or [Filebeat azure-eventhub input](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-azure-eventhub.html) documentation.
 
+### AMQP-over-WebSockets
+
+By default, the integration uses AMQP ports `5671` and `5672` to communicate with Event Hubs. If these ports are blocked, set **Event Hubs transport protocol** to **AMQP-over-WebSockets** in the advanced options. This tunnels AMQP over HTTPS on port `443`.
+
+This option requires the Event Hub processor v2 (**Processor version** set to `v2`, which is the default) and Elastic Agent 8.19.10, 9.1.10, 9.2.4, or later.
+
+### Proxy support
+
+Proxy support is optional and requires **Event Hubs transport protocol** set to **AMQP-over-WebSockets**.
+
+To enable it:
+
+1. In the advanced options, set **Event Hubs transport protocol** to **AMQP-over-WebSockets**.
+2. Define the `HTTPS_PROXY` environment variable for the Elastic Agent process, for example `HTTPS_PROXY=http://proxy.example.com:8080`. Elastic Agent routes both Event Hubs and Storage Account traffic through the proxy.
+
+This requires processor v2 and Elastic Agent 8.19.10, 9.1.10, 9.2.4, or later.
+
 ## App Service Logs
 
 Collects different types of logs from Azure App Service via Event Hub.

packages/azure_app_service/changelog.yml

@@ -1,4 +1,15 @@
 # newer versions go on top
+- version: "1.2.0"
+  changes:
+    - description: Add support for AMQP-over-WebSockets transport protocol and proxy configuration.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/20559
+    - description: Add the Event Hub processor version option and switch the default processor version to v2.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/20559
+    - description: Add the remaining Event Hub processor v2 options (processor update interval, processor start position, partition receive timeout, partition receive count, and checkpoint migration).
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/20559
 - version: "1.1.0"
   changes:
     - description: Add client secret authentication support for Azure Event Hub with RBAC.

packages/azure_app_service/data_stream/app_service_logs/agent/stream/azure-eventhub.yml.hbs

@@ -43,6 +43,25 @@ storage_account_key: {{storage_account_key}}
 {{#if resource_manager_endpoint}}
 resource_manager_endpoint: {{resource_manager_endpoint}}
 {{/if}}
+{{#if processor_version}}
+processor_version: {{processor_version}}
+{{/if}}
+migrate_checkpoint: {{migrate_checkpoint}}
+{{#if processor_update_interval}}
+processor_update_interval: {{processor_update_interval}}
+{{/if}}
+{{#if processor_start_position}}
+processor_start_position: {{processor_start_position}}
+{{/if}}
+{{#if partition_receive_timeout}}
+partition_receive_timeout: {{partition_receive_timeout}}
+{{/if}}
+{{#if partition_receive_count}}
+partition_receive_count: {{partition_receive_count}}
+{{/if}}
+{{#if transport}}
+transport: {{transport}}
+{{/if}}
 tags:
 {{#if preserve_original_event}}
   - preserve_original_event

packages/azure_app_service/docs/README.md

@@ -23,6 +23,23 @@ Refer to the [Azure Logs](https://docs.elastic.co/integrations/azure) page for m
 
 **Authentication (Event Hub):** The Event Hub input supports two authentication methods: **connection string** (default) and **client secret** (Microsoft Entra ID). For setup steps, required RBAC roles (Azure Event Hubs Data Receiver, Storage Blob Data Contributor), and configuration options, see the [Azure Logs integration](https://docs.elastic.co/integrations/azure) or [Filebeat azure-eventhub input](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-azure-eventhub.html) documentation.
 
+### AMQP-over-WebSockets
+
+By default, the integration uses AMQP ports `5671` and `5672` to communicate with Event Hubs. If these ports are blocked, set **Event Hubs transport protocol** to **AMQP-over-WebSockets** in the advanced options. This tunnels AMQP over HTTPS on port `443`.
+
+This option requires the Event Hub processor v2 (**Processor version** set to `v2`, which is the default) and Elastic Agent 8.19.10, 9.1.10, 9.2.4, or later.
+
+### Proxy support
+
+Proxy support is optional and requires **Event Hubs transport protocol** set to **AMQP-over-WebSockets**.
+
+To enable it:
+
+1. In the advanced options, set **Event Hubs transport protocol** to **AMQP-over-WebSockets**.
+2. Define the `HTTPS_PROXY` environment variable for the Elastic Agent process, for example `HTTPS_PROXY=http://proxy.example.com:8080`. Elastic Agent routes both Event Hubs and Storage Account traffic through the proxy.
+
+This requires processor v2 and Elastic Agent 8.19.10, 9.1.10, 9.2.4, or later.
+
 ## App Service Logs
 
 Collects different types of logs from Azure App Service via Event Hub.