.. _version_history_1.40.0: 1.40.0 (Pending) ================= Incompatible behavior changes ----------------------------- *Changes that are expected to cause an incompatibility if applicable; deployment changes are likely required* * **build**: Bumped the hermetic LLVM/Clang toolchain from 18 to 22. This upgrades the default compiler used by ``--config=clang`` and may surface new warnings or diagnostics in downstream builds that pin to the Envoy toolchain. * **ssl**: Removed AWS-LC as a selectable SSL library, along with the ``--config=aws-lc-fips`` build configuration. AWS-LC was previously the only way to build for the ppc64le architecture; ppc64le builds should now use ``--config=openssl`` instead. Minor behavior changes ---------------------- *Changes that may cause incompatibilities for some users, but should not for most* * **ext_proc**: The external processing filter now logs the target URI as the destination when using the ``google_grpc`` service. Previously, only the cluster name of the ``envoy_grpc`` service was logged. * **happy_eyeballs**: The happy eyeballs sorting of a multi-address host's address list now happens once when the address list is created or refreshed, instead of on every upstream connection attempt. The order in which connection attempts are made is unchanged. * **redis_proxy**: The Redis proxy codec is stricter about malformed RESP wire input that was previously accepted silently: negative aggregate or bulk length headers other than the spec's ``*-1`` / ``$-1`` null forms, integer lines carrying no digits, integers outside the signed 64-bit range, and messages exceeding new nesting-depth, cumulative-element, inline-command-element and scalar-token limits are now treated as protocol errors that close the connection. A single bulk string, blob error or verbatim string payload is additionally capped at 512 MiB, matching Redis's default ``proto-max-bulk-len``; deployments whose backends raise ``proto-max-bulk-len`` beyond that default are affected by this cap. Locally generated error replies also have ASCII control bytes replaced with spaces so attacker-influenced text cannot inject RESP framing. The remaining changes are only visible to peers sending non-conforming or abusive wire data. * **server**: Fixed container-aware CPU limit detection (#45410) not being enabled by default. The minimum of the cgroup CPU limit, CPU affinity, and hardware thread count, added in #40997 and documented as the default in the v1.37.0 release notes, was only applied when ``--cpuset-threads`` was set. It is now applied whenever ``--concurrency`` is not set, so worker threads are sized to the cgroup CPU limit in containerized deployments without requiring ``--cpuset-threads``. Detection can still be disabled by setting ``ENVOY_CGROUP_CPU_DETECTION`` to ``false``. * **stats**: Added the runtime guard ``envoy.reloadable_features.enable_stats_explicit_tags`` (default ``false``). When set to ``true`` and the stats configuration carries no custom tags (empty :ref:`stats_tags ` and :ref:`use_all_default_tags ` left at its default of ``true``), the stats store uses the tags supplied by the calling code (the explicit-tags logic) and propagates scope-level tags onto every stat, instead of re-parsing the flat stat name. The guard is evaluated once at startup. There is no visible change to users while the guard remains ``false``. * **watchdog**: Configuring the :ref:`envoy.watchdog.backtrace_action ` now causes Envoy to install a process-wide ``SIGUSR2`` signal handler and to send ``SIGUSR2`` to stuck threads in order to capture their backtraces. Deployments that rely on ``SIGUSR2`` for other purposes should avoid enabling this action. Bug fixes --------- *Changes expected to improve the state of the world and are unlikely to have negative effects* * **access_log**: Fixed a bug where :ref:`omit_empty_values ` had no effect for ``json_format``. Because the JSON formatter pre-serializes the template when loading the configuration, keys whose command operators evaluated to null were still emitted (for example ``{"key":null}`` instead of ``{}``). When ``omit_empty_values`` is set, the JSON formatter now omits keys with null values, removes nested objects that become empty, and preserves empty arrays, matching the documented behavior. This behavioral change can be reverted by setting the runtime guard ``envoy.reloadable_features.json_formatter_omit_empty_values`` to ``false``. * **credential_injector**: Fixed a bug where a credential loaded from a file-based generic secret was injected into the request header verbatim, including any trailing newline commonly present in secret files. Since HTTP header values cannot contain CR/LF, this produced an invalid header and the request failed. Trailing CR/LF characters are now stripped from the credential before injection, and a credential consisting only of CR/LF characters is treated as missing. * **grpc_http1_reverse_bridge**: Fixed a crash (SEGFAULT) in the ``grpc_http1_reverse_bridge`` filter when ``withhold_grpc_frames`` is enabled without ``response_size_header`` and the upstream response body exceeds the downstream HTTP/2 stream flow control window. The filter now uses the upstream ``Content-Length`` header to stream the response incrementally instead of buffering and releasing it all at once. * **http**: Fixed a bug where response metadata added by HTTP encoder filters could be dropped when a later encoder filter sent a direct local reply before final response headers were encoded to the codec. Saved response metadata is now flushed before the local reply ends the stream. This behavior can be temporarily reverted by setting the runtime guard ``envoy.reloadable_features.direct_local_reply_flush_saved_response_metadata`` to ``false``. * **listener**: Fixed a crash at startup when a UDP or QUIC listener was configured with ``bind_to_port: false``. This combination was never functional and is now rejected at configuration load with a validation error. * **mcp**: Fixed a memory usage issue in the MCP JSON-RPC parser by optimizing node allocation for unneeded fields. * **mcp_json_rest_bridge**: Fixed a path-traversal issue in the ``mcp_json_rest_bridge`` HTTP filter where a path-template variable's value (taken from attacker-controlled tool-call arguments) was installed verbatim into the upstream request ``:path``, so a value such as ``../../admin/secrets`` produced raw path traversal. Traversal segments (``.`` / ``..``) are now rejected for every template variable, and a "simple" variable (for example ``{id}``) additionally has ``/`` percent-encoded to confine it to a single path segment. Variables with an explicit pattern such as ``{name=projects/*}`` may still legitimately span multiple segments. * **mcp_json_rest_bridge**: mcp_json_rest_bridge: Fixed a bug where headers-only upstream responses (e.g., HTTP 204 No Content) were passed through to MCP clients without a JSON-RPC response body, causing MCP SDK timeouts or exceptions. The filter now synthesizes a valid JSON-RPC response: an empty ``ToolResult`` for ``tools/call`` requests and a server error for ``tools/list`` requests. * **open_telemetry**: Fixed the OpenTelemetry access loggers (both the gRPC and HTTP variants) ignoring configured :ref:`formatters ` when building :ref:`custom_tags `. Previously a custom tag whose value used a formatter extension command failed with ``Not supported field in StreamInfo``, even though the same command worked in ``body`` and ``attributes``. The configured command parsers are now passed through to custom-tag creation. * **redis_proxy**: Fixed a use-after-free in the Redis cluster ``CLUSTER SLOTS`` discovery. A cluster refresh (periodic resolve timer or DNS update) that arrived after ``CLUSTER SLOTS`` completed but while the zone-discovery ``INFO`` requests it triggered were still in flight could start a second discovery, overwrite the in-flight callbacks and free memory still referenced by the outstanding requests. * **reverse_tunnel**: Fixed a bug in the reverse tunnel downstream socket interface (``envoy.bootstrap.reverse_tunnel.downstream_socket_interface``) where handshake ``additional_headers`` values that use a ``ThreadLocal``-backed substitution formatter (such as ``%FILE_CONTENT%``, or secret/SDS-backed formatters) resolved to an empty string on the worker thread that assembles the handshake request. The handshake formatters were built in the bootstrap extension constructor, which runs before the worker threads register with the ``ThreadLocal`` system, so the formatter providers' thread-local slots were never populated on the workers. The formatters are now built in ``onServerInitialized()``, after the workers are registered, so their values propagate to every worker thread. * **reverse_tunnel**: Fixed a bug in the reverse tunnel downstream socket interface (``envoy.bootstrap.reverse_tunnel.downstream_socket_interface``) where handshake ``additional_headers`` values that use a substitution formatter (such as ``%FILE_CONTENT%``, or secret/SDS-backed formatters) were sent as the raw, unsubstituted template on every reverse connection. The reverse connection listen socket snapshots the handshake formatters when it is created, which can happen before ``onServerInitialized()`` builds them; that null snapshot is then reused for every re-dial, so the handshake fell back to emitting the literal ``additional_headers`` value. The handshake headers are now resolved from the live bootstrap extension when the request is assembled, so post-initialization dials substitute the value correctly. * **thrift_proxy**: Fixed a 32-bit integer overflow in the ``thrift_proxy`` lax (non-strict) binary protocol decoder. A message name length of 0xFFFFFFF7 or greater wrapped the insufficient-data check in ``readMessageBegin`` and raised a spurious decode error that closed the downstream connection. The check is now performed in 64-bit arithmetic and the decoder waits for more data instead, matching the strict binary protocol. * **tls**: Fixed a memory leak in the OpenSSL compatibility layer where ``SSL_get0_peer_certificates()`` called ``SSL_get_peer_certificate()`` without freeing the returned reference. Each call leaked one ``X509`` refcount, preventing the certificate and its sub-allocations from being freed when the connection closed, causing unbounded memory growth in certain deployments. * **tls**: Fixed upstream TLS client session caching so sessions are scoped by the effective SNI used for the connection. This prevents a session learned for one upstream SNI from being offered on a connection using a different SNI. The existing ``max_session_keys`` setting continues to limit the total number of cached sessions. This behavior can be temporarily reverted by setting runtime guard ``envoy.reloadable_features.scope_upstream_tls_session_cache_by_sni`` to ``false``. * **upstream**: Fixed a bug where ``upstream_bind_config`` with port ``0`` could cause ephemeral port exhaustion by reserving an ephemeral port during ``bind()``. Envoy now automatically enables ``IP_BIND_ADDRESS_NO_PORT`` to defer port allocation until ``connect()``. This change can be temporarily reverted by setting runtime guard ``envoy.reloadable_features.upstream_bind_config_fix_port_exhaustion`` to ``false``. Removed config or runtime ------------------------- *Normally occurs at the end of the* :ref:`deprecation period ` * **http2**: Removed the runtime guard ``envoy.reloadable_features.safe_http2_options`` and the legacy code path it guarded. HTTP/2 connections now always fall back to the safe defaults (max concurrent streams of 1024, 16 MiB initial stream window and 24 MiB initial connection window) when the corresponding options are unset, and the unused legacy default constants are removed. * **oauth2**: Removed the runtime guard ``envoy.reloadable_features.oauth2_cleanup_cookies`` and the legacy code path it guarded. The OAuth2 filter now always removes the OAuth flow cookies (``OauthHMAC``, ``OauthExpires``, ``RefreshToken``, ``OauthNonce`` and ``CodeVerifier``, including their suffixed names) from a request before it is forwarded upstream, so these cookies are no longer exposed to the backend service. * **on_demand**: Removed the runtime guard ``envoy.reloadable_features.on_demand_track_end_stream`` and the legacy code path it guarded. The on-demand filter now always tracks the downstream ``end_stream`` state to decide whether a stream with a fully read body can be recreated, instead of rejecting all requests that carry a body. * **tracing**: Removed the runtime guard ``envoy.reloadable_features.trace_refresh_after_route_refresh`` and the legacy code path it guarded. The HTTP connection manager now always refreshes the trace decision and decorator when the route is refreshed, and charges the tracing statistics from ``chargeStats`` rather than from the old un-refreshed code path. * **wasm**: Removed the runtime guard ``envoy.reloadable_features.wasm_use_effective_ctx_for_foreign_functions`` and the legacy code path it guarded. The ``set_envoy_filter_state`` and ``clear_route_cache`` Wasm foreign functions now always resolve the effective context (``contextOrEffectiveContext``) instead of the current context. New features ------------ * **access_log**: Added the ``%LISTENER_NAME%`` access log command operator, which logs the name of the listener that accepted the downstream connection. * **access_log**: Added the ``DS_HS_BEG`` (downstream TLS handshake begin, i.e. when the ClientHello was received) and ``DS_HS_END`` (downstream TLS handshake end) time points to the :ref:`%COMMON_DURATION% ` access log formatter. These are populated for both TLS and QUIC downstream connections. Also added the ``%DOWNSTREAM_CX_RTT%`` access log formatter returning the last measured round trip time of the downstream connection in milliseconds. * **admin**: Added ``invert_filter`` query parameter to the ``/stats`` and ``/stats/prometheus`` admin endpoints. When set, the ``filter`` regex is inverted so matching stats are excluded from the output (e.g. ``/stats?filter=server&invert_filter``). * **aws_eventstream_parser**: Added the :ref:`aws_eventstream_parser ` filter. This filter extracts values from AWS EventStream HTTP response bodies (used by AWS Bedrock streaming APIs) and writes them to dynamic metadata for observability, logging, and cost tracking use cases. * **dynamic_modules**: Added stats sink snapshot getters that expose each metric's tag-extracted name and its tags (name/value pairs) for counters, gauges, and text readouts, so a dynamic module can reconstruct the dimensional metric names Envoy's built-in formatters produce. Available through the Rust SDK ``MetricSnapshot`` tag accessors. * **dynamic_modules**: Added the ``envoy_dynamic_module_callback_listener_filter_set_filter_state_typed`` and ``envoy_dynamic_module_callback_listener_filter_get_filter_state_typed`` ABI callbacks so a dynamic-module listener filter can write and read typed filter state, mirroring the existing bytes setter/getter. Unlike the bytes variant which stores a raw ``Router::StringAccessor``, the typed setter uses the key's registered ``ObjectFactory`` to build a properly typed filter state object, so a built-in Envoy filter that reads the key as a typed object can consume it. The Rust SDK exposes these as ``EnvoyListenerFilter::set_filter_state_typed`` and ``EnvoyListenerFilter::get_filter_state_typed``. * **http**: Added support for forwarding the issuer of the client certificate in the ``x-forwarded-client-cert`` (XFCC) header via the new :ref:`issuer ` field of ``SetCurrentClientCertDetails``. When enabled, the ``Issuer`` key is added in text format and the ``issuer`` field is added in JSON format. Defaults to disabled. * **mcp**: Added a ``NOOP`` traffic mode to ``McpFilter``. * **quic**: Added support for memory optimization in QUIC by resetting the internal SSL object after the handshake finishes. This can be enabled by setting the runtime guard ``envoy.reloadable_features.quic_enable_reset_ssl_after_handshake`` to ``true``. * **quic**: Upstream QUIC connections now present the client certificate configured in the cluster's :ref:`upstream TLS context ` when the upstream server requests one. Previously configured client certificates were silently not sent over HTTP/3. Client certificates using a private key provider are not supported over QUIC and are now rejected at configuration load time. This behavior change can be reverted by setting the runtime guard ``envoy.reloadable_features.quic_upstream_client_certificates`` to ``false``; the guard is evaluated when a cluster's transport socket is created, so flipping it takes effect on clusters created or updated afterwards. * **redis_proxy**: Added RESP3 protocol support to the Redis proxy via the new :ref:`protocol_version ` listener setting (default ``RESP2`` keeps the existing behavior). When set to ``RESP3``, downstream clients negotiate with an explicit ``HELLO 3`` handshake — data commands sent beforehand are rejected with ``-NOPROTO`` and counted by the new ``downstream_rq_noproto`` counter — and every new upstream connection performs a ``HELLO 3`` handshake (combined with ``AUTH`` or AWS IAM credentials when configured, followed by ``READONLY`` where applicable) before serving traffic; requests issued during the handshake are held and replayed in order, and negotiation failures are tracked by the new per-cluster ``upstream_resp3_hello_failure`` counter. Independently of the setting, the proxy now answers ``HELLO``, ``CLIENT SETNAME`` and ``CLIENT SETINFO`` locally so that modern Redis clients can complete their connection setup, and the codec understands all RESP3 frame types, down-converting them for RESP2 connections. * **reverse_tunnel**: Added experimental inline JWT authentication for the reverse tunnel handshake via the new :ref:`jwt_validation ` field on the ``envoy.filters.network.reverse_tunnel`` filter. When configured, the bearer token carried in the handshake request is verified (signature, issuer, audiences, and ``exp``) against an inline ``local_jwks`` before the connection is accepted and its socket registered, so a forged or expired token cannot establish a usable reverse tunnel. A ``jwt_validation`` block requires an ``issuer``, and tokens without an ``exp`` claim are rejected. Verified claims are published as dynamic metadata so the existing ``validation`` block can bind a claimed identifier to a verified claim via ``%DYNAMIC_METADATA(namespace:claim)%``. Only inline JWKS (synchronous verification) is supported; remote JWKS fetching is not yet implemented. * **reverse_tunnel**: The downstream reverse-tunnel initiator (``envoy.bootstrap.reverse_tunnel.downstream_socket_interface``) now includes two additional identifiers in the HTTP handshake it sends to the acceptor: ``x-envoy-reverse-tunnel-worker-id`` (the initiator worker dispatcher name, e.g. ``worker_2``) and ``x-envoy-reverse-tunnel-connection-id`` (the initiator's per-connection id). Both are surfaced in the initiator access log via the new ``worker_id`` and ``connection_id`` fields of the ``envoy.reverse_tunnel.initiator`` dynamic metadata namespace. The upstream acceptor (``envoy.bootstrap.reverse_tunnel.upstream_socket_interface``) now parses these headers and exposes them on every reverse-tunnel lifecycle event as the ``initiator_worker_id`` and ``initiator_connection_id`` fields of the ``envoy.reverse_tunnel.lifecycle`` dynamic metadata namespace and as the ``envoy.reverse_tunnel.initiator_worker_id`` / ``envoy.reverse_tunnel.initiator_connection_id`` connection filter-state keys. Together these let tunnels originating from different workers/connections of the same initiator be told apart and correlated across both ends. * **stats**: Added per-cluster and per-listener ``stats_matcher`` configuration that overrides the bootstrap :ref:`stats_config ` matcher for the specific cluster or listener. When this field is configured, legacy ``envoy.stats_matcher`` metadata is ignored. * **tracing**: tracing: added :ref:`set_instrumentation_scope ` option to the OpenTelemetry tracer to allow controlling the emission of the instrumentation scope name and version in traces. * **watchdog**: Added :ref:`envoy.watchdog.backtrace_action`, a new watchdog action that logs a stack backtrace of stuck threads when the watchdog fires. A configurable cooldown prevents duplicate backtraces for the same thread.