USING AMBASSADOR
Understanding Envoy Proxy HTTP Access Logs
Ambassador uses Envoy’s default access log format
Ambassador uses Envoy Proxy as its core L7 routing engine. Envoy Proxy provides a configurable access logging mechanism. Ambassador uses the default format string for Envoy’s access logs. These access logs provide an extensive amount of information that can be used to troubleshoot issues.
Reading Ambassador Access Logs
You can read the log file using kubectl logs:
(⎈ | gke:default)$ kubectl logs ambassador-796cb689d9-jsptz ambassador
ACCESS [2019-08-22T09:14:59.499Z] "GET /.ambassador-internal/openapi-docs HTTP/1.1" 200 - 0 989 5 1 "10.52.2.21" "Go-http-client/1.1" "bc391742-0ad4-4f0d-9e00-6e81266a1480" "ambassador" "10.55.253.138:5000"
ACCESS [2019-08-22T09:14:59.506Z] "GET /callback/.ambassador-internal/openapi-docs HTTP/1.1" 503 UH 0 19 3 - "10.52.2.21" "Go-http-client/1.1" "c8d23a4b-c203-468a-abfb-ef47aca58e23" "ambassador" "-"
...The Ambassador access log format
Let’s dissect each entry.
Start time
The start time of the request.
Method
The HTTP method used for the request.
X-Envoy-Original-Path
The original HTTP path requested by the client.
Protocol
Either HTTP/1.1 or HTTP/2. If the protocol is TCP, the value will be -.
Response Code
The HTTP response code. If the request is a TCP request, the value will be — .
Response Flags
These provide additional details about the response or connection if any above and beyond the standard response code. Possible values for HTTP and TCP requests include UH (no healthy upstream hosts); UF(upstream connection failure); UO (upstream overflow); NR (no route configured); URX (rejected because of upstream retry limit or maximum connection attempts reached). For HTTP requests, an additional set of values are possible, including:
DCdownstream connection terminationLHLocal service failed health check requestUTUpstream request timeoutLRConnection local resetURUpstream remote resetUCUpstream connection terminationDIThe request processing was delayed for a period specified via fault injection. Note that Ambassador does not currently support fault injection.FIThe request was aborted with a response code specified via fault injection. Note that Ambassador does not currently support fault injection.RLThe request was ratelimited locally by the rate limiting filter.UAEXThe request was denied by the external authorization service.RLSEThe request was rejected because there was an error in rate limit service.IHThe request was rejected because it set an invalid value for a strictly-checked header in addition to 400 response code.SIStream idle timeout in addition to 408 response code.DPEThe downstream request had an HTTP protocol error.UPEThe upstream response had an HTTP protocol error.UMSDRThe upstream request reached max stream duration.
Bytes Received / Bytes Sent
The body bytes received or sent. For WebSocket connections, the Bytes Sent will include response header bytes.
Response Duration
The total duration, in milliseconds, of the request from the start time to the first byte read from the upstream host.
Upstream Service Time
The time, in milliseconds, spent by the upstream host processing the request. This is useful if you want to compare the service time compared to network latency.
X-Forwarded-For
The XFF HTTP header field identifies the originating IP address of the client. Ambassador enables XFF by default.
User-Agent
The user agent string, which allows the server to identify the specific type of software request agent.
Request ID
The x-request-id header is used by Envoy to uniquely identify each request. This is especially important for distributed tracing and stable access logging across multiple microservices.
Host (or Authority)
The value of the Host (HTTP/1.1) or Authority (HTTP/2) header.
Upstream Host
The upstream host URL, i.e., the target destination for the request.
Further reading
For more details about the access log configuration, see the Envoy Proxy access log documentation. Thanks to Megan O’Keefe for her original tweet about Envoy access logs in Istio:
You’ll see some strong similarities between Istio and Ambassador access logs (after all, both are based on Envoy Proxy). That said, there are some subtle differences as Ambassador is solely an edge gateway, while Istio is a broader mesh (what’s the difference?).
Get Involved
Interested in Ambassador? Join our Slack and get started with Ambassador Edge Stack.
And, if Ambassador is working well for you, we’d love to hear about it. Drop us a line in the comments below, or @ambassadorlabs on Twitter.
This post was last updated on March 22, 2021.
