7.58.0

Agent

Prelude

Release on: 2024-10-21

• Please refer to the 7.58.0 tag on integrations-core for the list of changes on the Core Checks

Upgrade Notes

• Changes behavior of the timeout for Network Path. Previously, the timeout signified the total time to wait for a full traceroute to complete. Now, the timeout signifies the time to wait for each hop in the traceroute. Additionally, the default timeout has been changed to 1000ms.

New Features

• Added capability to tag any Kubernetes resource based on labels and annotations. This feature can be configured with kubernetes_resources_annotations_as_tags and kubernetes_resources_labels_as_tags. These feature configurations are associate group resources with annotations-to-tags (or labels-to-tags) map For example, pods can be associated with an annotations-to-tags map to configure annotations as tags for pods. Example: {`pods`: {`annotationKey1`: tag1, `annotationKey2`: tag2}}
• The Kubernetes State Metrics (KSM) check can now be configured to collect pods from the Kubelet in node agents instead of collecting them from the API Server in the Cluster Agent or the Cluster check runners. This is useful in clusters with a large number of pods where emitting pod metrics from a single check instance can cause performance issues due to the large number of metrics emitted.
• NPM - adds UDP "Packets Sent" and "Packets Received" to the network telemetry in Linux.
• [oracle] Add the active_session_history configuration parameter to optionally ingest Oracle active session history samples instead of query sampling.
• Added config option logs_config.tag_truncated_logs. When enabled, file logs will come with a tag truncated:true if they were truncated by the Agent.

Enhancement Notes

• [DBM] Bump go-sqllexer to 0.0.14 to skip collecting CTE tables as SQL metadata.
• Agents are now built with Go 1.22.7.
• Add the ability to tag cisco-sdwan device and interface metrics with user-defined tags.
• Add support for setting a custom log source from resource attribute or log attribute datadog.log.source.
• The default UDP port for traceroute (port 33434) is now used for Network Traffic based paths, instead of the port detected by NPM.
• [oracle] Add oracle_client_lib_dir config parameter.
• [oracle] Increase tablespace check interval from 1 to 10 minutes.
• [oracle] Don't try to fetch execution plans where plan_hash_value is 0
• The OTLP ingest endpoint now maps the new OTel semantic convention deployment.environment.name to env
• Prevents the use of the process_config.run_in_core_agent.enabled configuration option in unsupported environments.
• APM: Trace payloads are now compressed with zstd by default.

Security Notes

• Bump embedded Python version to 3.12.6 to address CVE-2024-4030 and CVE-2024-4741.
• Update cURL to 8.9.1.
• Update OpenSSL to 3.3.2 (on Linux & macOS) in order to mitigate CVE-2024-6119.

Bug Fixes

• Adds missing support for the logs config key to work with AD annotations V2.
• Fix agent jmx [command] subcommands for container environments with annotations-based configs.
• Fixed issue with openSUSE 15 RC 6 where the eBPF tracer wouldn't start due to a failed validation of the tcp_sendpage probe.
• Fixed a rare issue where short-lived containers could cause logs to be sent with the wrong container ID.
• Fix Windows Process Agent argument stripping to account for spaces in the executable path.
• Fixes issue with the kubelet corecheck where kubernetes.kubelet.volume.* metrics were not properly being reported if any matching namespace exclusion filter was present.
• OOM Kill Check now reports the cgroup name of the victim process rather than the triggering process.
• The process agent will no longer exit prematurely when language detection is enabled or when there is a misconfiguration stemming from process_config.run_in_core_agent.enabled's default enablement in Kubernetes.
• Change the datadog-security-agent Windows service display name from Datadog Security Service to Datadog Security Agent for consistency with other Agent services.
• Fix a bug preventing SNMP V3 reconnection.

Other Notes

• Add metric origins for the Kubeflow integration.
• Add functional tests to Oracle using a Docker service to host the database instance.
• Adds Agent telemetry for Oracle collector.

Datadog Cluster Agent

Prelude

Released on: 2024-10-21 Pinned to datadog-agent v7.58.0: CHANGELOG.

New Features

• Added capability to tag any Kubernetes resource based on labels and annotations. This feature can be configured with kubernetes_resources_annotations_as_tags and kubernetes_resources_labels_as_tags. These feature configurations are associate group resources with annotations-to-tags (or labels-to-tags) map For example, deployments.apps can be associated with an annotations-to-tags map to configure annotations as tags for deployments. Example: {`deployments.apps`: {`annotationKey1`: tag1, `annotationKey2`: tag2}}
• The Kubernetes State Metrics (KSM) check can now be configured to collect pods from the Kubelet in node agents instead of collecting them from the API Server in the Cluster Agent or the Cluster check runners. This is useful in clusters with a large number of pods where emitting pod metrics from a single check instance can cause performance issues due to the large number of metrics emitted.

Enhancement Notes

• Added a new option for the Cluster Agent ("admission_controller.inject_config.type_socket_volumes") to specify that injected volumes should be of type "Socket". This option is disabled by default. When set to true, injected pods will not start until the Agent creates the DogstatsD and trace-agent sockets. This ensures no traces or DogstatsD metrics are lost, but it can cause the pod to wait if the Agent has issues creating the sockets.

Bug Fixes

• Fixed an issue that prevented the Kubernetes autoscaler from evicting pods injected by the Admission Controller.

7.57.2

Prelude

Release on: 2024-09-24

Enhancement Notes

• Agents are now built with Go 1.22.7.

Bug Fixes

• Fix OOM error with cluster agent auto instrumentation by increasing default memory request from 20Mi to 100Mi.
• Fixes a panic caused by running the Agent on readonly filesystems. The Agent returns integration launchers and handles memory gracefully.

7.57.1

Agent

7.57.1

Prelude

Release on: 2024-09-17

• Please refer to the 7.57.1 tag on integrations-core for the list of changes on the Core Checks

Bug Fixes

• APM: When the UDS listener cannot be created on the trace-agent, the process will log the error, instead of crashing.
• Fixes memory leak caused by container check.

Datadog Cluster Agent

7.57.1

Prelude

Released on: 2024-09-17 Pinned to datadog-agent v7.57.1: CHANGELOG.

7.57.0

Agent

7.57.0

Known bugs

• ECS Fargate deployments may cause increases in RAM and CPU usage. For more information see #27523.

Prelude

Release on: 2024-09-09

• Please refer to the 7.57.0 tag on integrations-core for the list of changes on the Core Checks

Upgrade Notes

• Update cURL to 8.9.1.
• Update OpenSSL from 3.0.14 to 3.3.1 (on Linux and macOS).

New Features

• The agent diagnose command now includes a --json option to output the results in JSON format.
• Add integration value for device metadata.
• APM: In order to allow for automatic instrumentation to work in Kubernetes clusters that enforce a Restricted Pod Security Standard, which require all containers to explicitly set a securityContext, an option to configure a securityContext to be used for all initContainers created by the auto instrumentation has been added. | This can be done through the DD_ADMISSION_CONTROLLER_AUTO_INSTRUMENTATION_INIT_SECURITY_CONTEXT environment value, or admission_controller.auto_instrumentation.init_security_context configuration -in both cases a json string should be supplied.
• Adds a kube_runtime_class tag to metrics associated with Kubernetes pods and their containers.
• Expose the Agent's get host tags function to python checks using the new datadog_agent.get_host_tags method.
• Implement static allowlist of Kubernetes events to send by default. This feature is only enabled when filtering_enabled is set to true in the kubernetes_apiserver integration configuration.
• Adds a new launcher to handle incoming logs from integtrations.
• Add optional reverse DNS enrichment of private IP addresses to NDM NetFlow.
• On Windows, the default value for the service inference feature is now enabled.

Enhancement Notes

• Turn on Orchestrator Explorer by default in the core agent
• Added new source_host tag to TCP/UDP logs to help users understand where their logs came from.
• Added support to handling UDP/TCP Logs when running the containerized agent.
• APM: Allow custom HTTP client to be provided when instantiating the trace-agent configuration. This feature is primarily intended for the OpenTelemetry exporter.
• APM: Add default UDS listeners for traces (trace-agent) and dogstatsd (core-agent) on /var/run/datadog/apm.socket and /var/run/datadog/dsd.socket, respectively. These are used in the Single Step APM Instrumentation, improving the onboarding experience and minimizing the agent configuration.
• For the [Inferred Service Dependencies beta](https://docs.datadoghq.com/tracing/guide/inferred-service-opt-in/?tab=java), add two new peer.hostname precursor attributes, out.host and dns.hostname. This will improve coverage of inferred services because some tracer integrations only place the peer hostname in one of those attributes.
• APM stats for internal service overrides are now aggregated by the _dd.base_service tag only, enhancing visibility into specific base services.
• Include spans with span.kind=consumer for aggregation of stats on peer tags.
• IP address quantization on all peer tags is done the backend during ingestion. This change updates the Agent to apply the same IP address quantization. This reduces unnecessary aggregation that is currently done on raw IP addresses. And therefore, improves the aggregation performance of stats on peer tags.
• APM: Add new setting to disable the HTTP receiver in the trace-agent. This setting should almost never be disabled and is only a convenience parameter for OpenTelemetry extensions. Disabling the receiver is semantically equivalent to setting the receiver_port to 0 and receiver_socket to "".
• Agents are now built with Go 1.22.6.
• [NDM] Adds the option to collect BGP neighbors metrics from Cisco SD-WAN.
• [NDM] Add option to collect cloud application metrics from Cisco SD-WAN.
• [Cisco SD-WAN] Allow enabling/disabling metrics collection.
• Report the hostname of Kubernetes events based on the associated pod that the event relates to.
• Introduces a parser to extract tags from integration logs and attach them to outgoing logs.
• Implement External Data environment variable injection in the Admission Controller. Format for this new environment variable is it-INIT_CONTAINER,cn-CONTAINER_NAME,pu-POD_UID. This new variable is needed for the New Origin Detection spec. It is used for Origin Detection in case Local Data are unavailable, for example with Kata Containers and CGroups v2.
• Upgraded JMXFetch to 0.49.3 which adds support for jsr77 j2ee statistics and custom ConnectionFactory. See 0.49.3 for more details.
• Windows Agent Installer gives a better error message when a gMSA account is provided for ddagentuser that Windows does not recognize.
• Uninstalling the Windows Agent MSI Installer removes specific subdirectories of the install path to help prevent data loss when PROJECTLOCATION is misconfigured to an existing directory.
• Adds a default upper limit of 10000 to the number of network traffic paths that are captured at a single time. The user can increase or decrease this limit as needed.
• Language detection can run on the core Agent without needing a gRPC server.
• Add Hostname and ExtraTags to CollectorECSTask.
• Collect SystemInfo for Pods and ECS Tasks.
• Implement API that allows Python checks to send logs for eventual submission.
• Users can use DD_ORCHESTRATOR_EXPLORER_CUSTOM_SENSITIVE_ANNOTATIONS_LABELS to remove sensitive annotations and labels. For example: DD_ORCHESTRATOR_EXPLORER_CUSTOM_SENSITIVE_ANNOTATIONS_LABELS="sensitive-key-1 sensitive-key-2". Keys should be separated by spaces. The agent removes any annotations and labels matching these keys.
• Add the ability to tag interface metrics with user-defined tags.

Security Notes

• Fix CVE-2024-41110.

Bug Fixes

• Results of agent config did not reflect the actual runtime config for the other services. This will have other Datadog Agent services (e.g. trace-agent) running as a systemd service read the same environment variables from a text file /etc/datadog-agent/environment as the core Agent process.
• [DBM] Bump go-sqllexer to 0.0.13 to fix a bug where the table name is incorrectly collected on PostgreSQL SELECT ONLY statement.
• [Cisco SD-WAN] Do not collect unspecified IP addresses.
• Fix container.net.* metrics accuracy on Linux. Currently container.net.* metrics are always emitted with high cardinality tags while the values may not represent actual container-level values but POD-level values (multiple containers in a pod) or host-level values (containers running in host network). With this bug fix, the container.net.* metrics aren't emitted for containers running in host network and a single timeseries is emitted by pods when running multiple containers. Finally, in non-Kubernetes environments, if multiple containers share the same network namespace, container.net.* metrics won't be emitted.
• Fix duplicate logging in Process Agent component's Enabled() method.
• Fixed bug in kubelet check when running in core agent that was causing kubernetes.kubelet.container.log_filesystem.used_bytes to be reported by the check for excluded/non-existing containers. The metric was being reported in this case without tags. This bug does not exist in the python integration version of the kubelet check.
• Fixes a bug on Windows in the driver installation custom actions that could prevent rollback from working properly if an installation failed or was canceled.
• Update pro-bing library to include fix for a Windows specific issue with large ICMP packets
• [oracle] Fix wrong durations for cloud databases.
• Stop chunking outputs in manual checks for container, process, and process_discovery checks to allow JSON unmarshaler to parse output.
• Remove the original pod annotation on consul
• Fix pod status for pods using native sidecars.
• Fix a regression where the Agent would fail to start on systems with SysVinit.
• APM: Fixes issue where the number of HTTP decoders was incorrectly set if setting GOMAXPROCS to milli-cpu values.

Other Notes

• Add metrics origins for vLLM integration.
• Add deprecation warnings when running process checks on the Process Agent in Linux. This change prepares for the deprecation of processes and container collection in the Process Agent, occurring in a future release.
• Add metric origin for the AWS Neuron integration

Datadog Cluster Agent

7.57.0

Prelude

Released on: 2024-09-09 Pinned to datadog-agent v7.57.0: CHANGELOG.

New Features

• The Cluster Agent now supports activating Continuous Profiling using Admission Controller.
• LimitRange and StorageClass resources are now collected by the orchestrator check.

Enhancement Notes

• The auto-ins...

7.56.2

Agent

7.56.2

Prelude

Release on: 2024-09-02

• Please refer to the 7.56.2 tag on integrations-core for the list of changes on the Core Checks

Bug Fixes

• Fix issue causing GUI to fail when opening with Internet Explorer on Windows.

Datadog Cluster Agent

7.56.2

Prelude

Released on: 2024-09-02 Pinned to datadog-agent v7.56.2: CHANGELOG.

7.56.1

Agent

7.56.1

Prelude

Release on: 2024-08-29

• Please refer to the 7.56.1 tag on integrations-core for the list of changes on the Core Checks

Bug Fixes

• Fixed a nil pointer dereference issue in the Tailer.DidRotate function that was causing the Agent to panic.

Datadog Cluster Agent

7.56.1

Prelude

Released on: 2024-08-29 Pinned to datadog-agent v7.56.1: CHANGELOG.

7.56.0

Agent

Prelude

Release on: 2024-08-16

• Please refer to the 7.56.0 tag on integrations-core for the list of changes on the Core Checks

Upgrade Notes

• Change default bind address in OTLP ingest from 0.0.0.0 to localhost. Please refer to the following blog post: https://opentelemetry.io/blog/2024/hardening-the-collector-one for additional information around this change.
• Update cURL to 8.7.1.

New Features

• The core Agent now supports multiple configuration files in addition to the main datadog.yaml file. The -E flag can be used to specify additional configuration files to be loaded sequentially after the main datadog.yaml.
• When DD_SERVERLESS_STREAM_LOGS is enabled, DD_EXTENSION prints collected logs like agent stream-logs.
• Add full support of CIS Amazon Linux 2 Benchmark in CSPM.
• Add full support of CIS Amazon Linux 2023 Benchmark in CSPM.
• NPM - Adds the capability to track and report failed TCP connections to the Datadog backend. This feature is disabled by default.

Enhancement Notes

• Add the use_apiserver_cache option to the kubernetes_state_metrics check to reduce the pressure on the underlying storage engine etcd. Requires Kubernetes 1.19+.

• APM: Add obfuscation support for OpenSearch statements within span metadata. This feature works in the same way as the existing Elasticsearch one, and is enabled by default. It is configured by binding apm_config.obfuscation.opensearch.* parameters to new obfuscation environment variables. In particular, bind: apm_config.obfuscation.opensearch.enabled to DD_APM_OBFUSCATION_OPENSEARCH_ENABLED: It accepts a boolean value with default value true.

  apm_config.obfuscation.opensearch.keep_values to DD_APM_OBFUSCATION_OPENSEARCH_KEEP_VALUES It accepts a list of strings of the form ["id1", "id2"].

  apm_config.obfuscation.opensearch.obfuscate_sql_values to DD_APM_OBFUSCATION_OPENSEARCH_OBFUSCATE_SQL_VALUES It accepts a list of strings of the form ["key1", "key2"].

• Agents are now built with Go 1.22.4.

• Agents are now built with Go 1.22.5.

• Bump dependency msodbcsql18 to version 18.3.3.1.

• Adds config kubernetes_events_source_detection.enabled which is false by default. When set to true, this sets the source of kubernetes events to specific integrations based on the name of the controller that emitted it. All kubernetes events will have the tag orchestrator:kubernetes. For controller names that do not match any of the known integrations, the source will still be set to kubernetes by default.

• Introduces a bundle_unspecifed_events config to the docker integration. When bundle_unspecifed_events and unbundle_events are true, Docker events are unbundled according to collected_event_types and the remaining events are bundled after excluding the filtered_event_types and collected_event_types.

• The Agent will now ignore empty configuration files in conf.d. Users can use this functionality to avoid creating broken integrations when deploying agents with provisioning systems that do not allow skipping files entirely.

• Introduces an bundle_unspecified_events config to the kubernetes_apiserver integration. When bundle_unspecified_events and unbundle_events are true, Kubernetes events are unbundled according to collected_event_types and the remaining events are bundled.

• Improve utility functions that start or stop a service to better manage the SERVICE_START_PENDING and SERVICE_STOP_PENDING states by waiting for the service transition to complete. This will improve handling of concurrent execution of agent.exe start and agent.exe restart commands for the Windows Agent.

• Make the oom_kill check capture the OOM score and the OOM score adjustment of the process being killed.

• Oracle integration will now auto-adjust the size of the SQL substring requested by the activity sampler to better support users with multi-byte character sets, for example, Korean, Arabic, etc. This should alleviate crashes caused by long queries using these characters.

• The OTLP ingestion endpoint now supports the same settings and protocol as the OpenTelemetry Collector OTLP receiver v0.103.0.

• APM: Probabilistic Sampler now only looks at the lower 64 bits of a trace ID by default to improve compatibility in distributed systems where some apps may truncate the trace ID. To maintain the previous behavior use the feature flag probabilistic_sampler_full_trace_id.

• Adds the source of the payload for Processes-owned messages.

• Add tags to CollectorManifest

• Add image and imageID fields to pods ContainerStatuses.

• The orchestrator check can now scrub sensitive data from probes in pods specifications.

• The Agent now populates the git.repository_url and git.commit.sha tags from the values of the DD_GIT_REPOSITORY_URL and DD_GIT_COMMIT_SHA container environment variables.

• Implement the kubernetes_persistent_volume_claims_as_tags configuration that allows users to disable PersistentVolumeClaim for Kubernetes resources.

• Upgrade the NTP check client library 'beevik/ntp' from v0.3.0 to v1.3.4

• Use cloud-provided hostname as default when running the Agent in AKS.

• APM: Enabled zstd compression by default on trace payloads.

Deprecation Notes

• APM: DD_APM_MAX_TPS config setting is deprecated in favor of the more accurate DD_APM_TARGET_TPS. Accordingly, when configured through YAML, max_traces_per_second is deprecated in favor of target_traces_per_second. The setting behavior remains the same, only the name is changed to more accurately reflect the existing logic.

Security Notes

• Updating OpenSSL to 3.0.14 to address CVE-2024-4741 (on Linux and macOS).

Bug Fixes

• Upgrades the pro-bing library to fix a Windows-only bug with too-long ICMP packets being received
• Fix ExtraTags mapping for CollectorManifest.
• Fix a bug in the Agent where it could potentially fetch logs of short-lived Kubernetes jobs twice if the CRI is Docker.
• Re-enable printing of checks metadata in the datadog-agent status collector section.
• Fix OTLP status output not being displayed in the GUI.
• Fix issue where init config for ping took priority over instance config.
• Fix diagnose command for logs endpoints and related warnings about unknown config keys.
• Fixes oracle.tablespace.offline metric not emitting 1 when tablespace is offline.
• APM: Show probabilistic sampling configuration in Agent status when enabled.
• Add a field to differentiate between empty and undefined podSelector or namespaceSelector for network policies.
• Fixed a bug where the file tailing position is always set to the beginning, this fix allows users to explicitly set a starting position.
• All datadog public endpoints have the maximum requirements to close idle connections after 60s being idle. If a given client keeps it for longer, the server will close it, and the client will likely see the issue during the next write, leading to a connection reset error. The idle timeout should be therefore set under a minute. This PR is reducing the timeout from 90 to 30s.
• Windows: Added driver rollback properties to ensure that all services and drivers are uninstalled or rolled back after an installation or upgrade failure.

Other Notes

• Add metric origins for community Python integrations.

Datadog Cluster Agent

Prelude

Released on: 2024-08-16 Pinned to datadog-agent v7.56.0: CHANGELOG.

Upgrade Notes

• Disables default injection of the .NET profiler dependency for Kubernetes auto_instrumentation.

Enhancement Notes

• Mark the NetworkPolicy collector as stable in the Cluster Agent
• Enabled language detection automatically in the injected agent sidecar on EKS Fargate when APM SSI is enabled. This is only available for users using the admission controller to automatically inject the agent sidecar.
• The orchestrator check can now scrub sensitive data from probes in pods specifications.

Bug Fixes

• Fixes issue where the external metrics server would sometimes return metrics which had not been updated for longer than the configured external_metrics_provider.max_age as valid. In connection with this fix, a new config (external_metrics_provider.query_validity_period) has been added to account for the delay between when metrics are resolved and when they are queried by the various autoscaling controllers. It is set to 30 seconds by default.

7.55.3

Agent

7.55.3

Prelude

Release on: 2024-08-01

• Please refer to the 7.55.3 tag on integrations-core for the list of changes on the Core Checks

Enhancement Notes

• Agents are now built with Go 1.21.12.

Security Notes

• Fix CVE-2024-41110.

Datadog Cluster Agent

7.55.3

Prelude

Released on: 2024-08-01 Pinned to datadog-agent v7.55.3: CHANGELOG.

7.55.2

Agent

7.55.2

Prelude

Release on: 2024-07-25

• Please refer to the 7.55.2 tag on integrations-core for the list of changes on the Core Checks

Security Notes

• Fix CVE-2024-6257.

Bug Fixes

• Fixes an issue introduced in 7.55.0 with container metrics. In some rare cases, container metrics (cpu, memory, limits, etc.) could be incorrect and not reflect actual resources usage.

Datadog Cluster Agent

7.55.2

Prelude

Released on: 2024-07-25 Pinned to datadog-agent v7.55.2: CHANGELOG.

7.55.1

Agent

Prelude

Release on: 2024-07-12

• Please refer to the 7.55.1 tag on integrations-core for the list of changes on the Core Checks

Bug Fixes

• Fix a regression where the agent would fail to start on systems with SysVinit

Datadog Cluster Agent

Prelude

Released on: 2024-07-12 Pinned to datadog-agent v7.55.1: CHANGELOG.