USING AMBASSADOR

Understanding Envoy Proxy HTTP Access Logs

Ambassador uses Envoy’s default access log format

Ambassador uses Envoy Proxy as its core L7 routing engine. Envoy Proxy provides a configurable access logging mechanism. Ambassador uses the default format string for Envoy’s access logs. These access logs provide an extensive amount of information that can be used to troubleshoot issues.

Reading Ambassador Access Logs

You can read the log file using kubectl logs:

 (⎈ | gke:default)$ kubectl logs ambassador-796cb689d9-jsptz ambassador
ACCESS [2019-08-22T09:14:59.499Z] "GET /.ambassador-internal/openapi-docs HTTP/1.1" 200 - 0 989 5 1 "10.52.2.21" "Go-http-client/1.1" "bc391742-0ad4-4f0d-9e00-6e81266a1480" "ambassador" "10.55.253.138:5000"
ACCESS [2019-08-22T09:14:59.506Z] "GET /callback/.ambassador-internal/openapi-docs HTTP/1.1" 503 UH 0 19 3 - "10.52.2.21" "Go-http-client/1.1" "c8d23a4b-c203-468a-abfb-ef47aca58e23" "ambassador" "-"
...

The Ambassador access log format

Let’s dissect each entry.

Start time

The start time of the request.

Method

The HTTP method used for the request.

X-Envoy-Original-Path

The original HTTP path requested by the client.

Protocol

Either HTTP/1.1 or HTTP/2. If the protocol is TCP, the value will be -.

Response Code

The HTTP response code. If the request is a TCP request, the value will be — .

Response Flags

These provide additional details about the response or connection if any above and beyond the standard response code. Possible values for HTTP and TCP requests include UH (no healthy upstream hosts); UF(upstream connection failure); UO (upstream overflow); NR (no route configured); URX (rejected because of upstream retry limit or maximum connection attempts reached). For HTTP requests, an additional set of values are possible, including:

• DC downstream connection termination
• LH Local service failed health check request
• UT Upstream request timeout
• LRConnection local reset
• URUpstream remote reset
• UCUpstream connection termination
• DIThe request processing was delayed for a period specified via fault injection. Note that Ambassador does not currently support fault injection.
• FI The request was aborted with a response code specified via fault injection. Note that Ambassador does not currently support fault injection.
• RL The request was ratelimited locally by the rate limiting filter.
• UAEX The request was denied by the external authorization service.
• RLSEThe request was rejected because there was an error in rate limit service.
• IH The request was rejected because it set an invalid value for a strictly-checked header in addition to 400 response code.
• SI Stream idle timeout in addition to 408 response code.
• DPE The downstream request had an HTTP protocol error.
• UPE The upstream response had an HTTP protocol error.
• UMSDR The upstream request reached max stream duration.

Bytes Received / Bytes Sent

The body bytes received or sent. For WebSocket connections, the Bytes Sent will include response header bytes.

Response Duration

The total duration, in milliseconds, of the request from the start time to the first byte read from the upstream host.

Upstream Service Time

The time, in milliseconds, spent by the upstream host processing the request. This is useful if you want to compare the service time compared to network latency.

X-Forwarded-For

The XFF HTTP header field identifies the originating IP address of the client. Ambassador enables XFF by default.

User-Agent

The user agent string, which allows the server to identify the specific type of software request agent.

Request ID

The x-request-id header is used by Envoy to uniquely identify each request. This is especially important for distributed tracing and stable access logging across multiple microservices.

Host (or Authority)

The value of the Host (HTTP/1.1) or Authority (HTTP/2) header.

Upstream Host

The upstream host URL, i.e., the target destination for the request.

Further reading

For more details about the access log configuration, see the Envoy Proxy access log documentation. Thanks to Megan O’Keefe for her original tweet about Envoy access logs in Istio:

You’ll see some strong similarities between Istio and Ambassador access logs (after all, both are based on Envoy Proxy). That said, there are some subtle differences as Ambassador is solely an edge gateway, while Istio is a broader mesh (what’s the difference?).

Get Involved

Interested in Ambassador? Join our Slack and get started with Ambassador Edge Stack.

And, if Ambassador is working well for you, we’d love to hear about it. Drop us a line in the comments below, or @ambassadorlabs on Twitter.

This post was last updated on March 22, 2021.