[Serve] Add RAY_SERVE_HAPROXY_TCP_NODELAY env var

[eicherseiji]

Summary

Adds RAY_SERVE_HAPROXY_TCP_NODELAY environment variable (default 0) that, when set to 1, adds option http-no-delay to the HAProxy config defaults section. This sets TCP_NODELAY on both client and server connections, disabling Nagle's algorithm.

Replaces #61455 which was accidentally corrupted by a bad force push.

Why this matters

HAProxy's default behavior leaves Nagle's algorithm enabled, which buffers small writes before flushing to the wire. For LLM serving workloads where responses are streamed token-by-token as small SSE frames, this introduces artificial latency — the kernel holds back each token waiting for more data to coalesce, directly inflating TTFT and tail ITL.

Benchmark results

Setup: Qwen2.5-0.5B-Instruct, 256 requests, max concurrency 32, prefix-heavy prompt (640 input tokens/req)

Metric	Nagle ON (default)	Nagle OFF (TCP_NODELAY)	Delta
Mean TTFT	356.8 ms	120.9 ms	-66%
Median TTFT	274.1 ms	69.0 ms	-75%
P99 TTFT	939.0 ms	715.6 ms	-24%
P99 ITL	255.9 ms	53.4 ms	-79%
Output throughput	2,481 tok/s	2,521 tok/s	~0%
Request throughput	19.4 req/s	19.8 req/s	~0%

Nagle causes ~3x mean TTFT regression and ~5x P99 ITL regression with no meaningful throughput benefit. The effect will be more pronounced at lower concurrency / interactive use cases.

Code change

The implementation is a one-liner in the Jinja template, gated behind an env var:

# python/ray/serve/_private/haproxy_templates.py
    {%- if config.tcp_nodelay %}
    # Set TCP_NODELAY on all connections
    option http-no-delay
    {%- endif %}

# python/ray/serve/_private/constants.py
RAY_SERVE_HAPROXY_TCP_NODELAY = (
    os.environ.get("RAY_SERVE_HAPROXY_TCP_NODELAY", "0") == "1"
)

Test plan

• Verify generated haproxy.cfg contains option http-no-delay when env var is set
• Existing HAProxy unit tests pass (test_controller_haproxy.py)
• New test: test_start_with_tcp_nodelay — verifies HAProxy starts successfully with tcp_nodelay=True and config contains the directive
• Benchmarked with/without on Qwen2.5-0.5B-Instruct (results above)

[gemini-code-assist]

Code Review

This pull request introduces the RAY_SERVE_HAPROXY_DISABLE_NAGLE environment variable to disable Nagle's algorithm in HAProxy, which can improve latency for certain workloads. The implementation correctly adds the necessary HAProxy directives. My feedback includes a couple of suggestions to enhance code consistency and simplify the HAProxy configuration.

[gemini-code-assist]

For consistency with other boolean environment variables in this file (e.g., RAY_SERVE_LOG_TO_STDERR), it would be better to use the get_env_bool utility function. This also makes the check more robust by handling values like "true" in addition to "1".

RAY_SERVE_HAPROXY_DISABLE_NAGLE = get_env_bool("RAY_SERVE_HAPROXY_DISABLE_NAGLE", "0")