Arnish Cache Please Try Again Later

Introduction

Varnish is a widely used contrary proxy and HTTP accelerator. Information technology sits in front of an HTTP service, and caches the responses to improve the response times observed by the clients, and possibly to reduce the load on the upstream backend service.

Besides bringing performance improvements, Varnish tin can be likewise useful for shielding the clients from any outage of the backend service, since if the upstream is temporarily out of service, Varnish can keep serving the cached content, so the clients don't have to notice anything (if the data they demand is already present in the cache).

In guild to use the basic caching capability, and to likewise gracefully autumn back when the backend is down, we have to control 2 properties of the response object.

• beresp.ttl: This is the corporeality of time until Varnish keeps an object in cache, and serves it up from there to the clients until it tries to fetch it once more from the upstream.
• beresp.grace: This is the corporeality of fourth dimension an object is kept in the cache even after the ttl is expired. This is interpreted on pinnacle of the ttl, so for case if both ttl and grace is set to 1m, then the objects can stay in the cache for a total of 2 minutes.

If a request comes in for which the response is already in the enshroud, and the ttl has not passed still, then Varnish returns the cached value to the customer without trying to fetch a fresh i from the backend.

On the starting time request that comes subsequently ttl expired, if grace is set and is non expired yet, then Varnish will immediately return the cached value to the customer, and at the aforementioned time it will trigger a background fetch for a fresh response. If the fetch is successful, it replaces the stale object in the enshroud with the fresh ane.

This beliefs means that if the backend is not available, that won't bear upon the end clients until the end of the grace period, since Varnish will ever render the cached object to the client, and but do a background fetch asynchronously. Which will fail if the backend is down, merely that won't hurt the object already stored in the cache.
And if the grace period is over, Varnish will purge the cached object, and won't return it to clients any more than. So if the backend is down, the clients will receive a 503 Service Unavailable response from Varnish.

In do, grace can be much longer than ttl. For instance we tin fix ttl to a couple of minutes to make sure the clients regularly become fresh data, but we can gear up grace to multiple hours, to give us time to fix the backend in instance of an outage.

5xx responses

By default Varnish only falls back to the buried values during the grace period if the backend tin can't exist connected to, or if the request times out. If the backend returns a 5xx error, Varnish considers that normal response, and returns information technology to the client (it might even store it in the enshroud, depending on our configuration).
This is an understandable default, it might exist confusing if our clients did non receive some server errors which could contain valuable data well-nigh the trouble with the backend.

On the other manus, nosotros might also consider 5xx responses outages, and want to gracefully fall back to the cached values instead. I wanted to attain the following objectives.

• Never cache 5xx responses. This is important, because this'll make Varnish try to fetch the response again upon every asking. (It would be wasteful to enshroud a 500 response for ten minutes, when the outage might only final 1-ii minutes.)
• If we are in the grace period, and the backend returns a 5xx, fall back to the cached response and ship that to the client.
• If we don't have the response in the enshroud at all, then return to the client the actual 5xx error that was given past the backend.

This can be accomplished with Varnish pretty easily, just information technology needs some configuration.

Configuration

In lodge to customize the behavior of Varnish, we take to edit its configuration file (which ordinarily resides at /etc/varnish/default.vcl).

The config file is using the VCL syntax, which is a C-similar domain specific linguistic communication specifically for the customization of Varnish. It took me some getting used to earlier I could wrap my head around it, an actual production configuration tin can get complicated, which can be a scrap intimidating at first. In this mail I won't talk most the basics, but will just rather show the bits necessary for this particular setup. If you'd like to learn the basics of VCL, I recommend reading a comprehensive guide, for example The Varnish Book.

In the VCL configuration we can customize various subroutines, with which we can claw into varios steps of the asking lifecycle. The but subrouting we're going to look at is sub vcl_backend_response, which is triggered correct after a response was received from the backend.

A very elementary version of this would be the following, which enables caching for 1 minute, and enables svelte fallback for 10 minutes on top of that.

            sub vcl_backend_response {     set beresp.ttl = 1m;     set beresp.grace = 10m; }                      

In order to attain the objective of gracefull fallback on 5xx errors, we'll need to use the post-obit additional tools.

• beresp.condition: With this nosotros can bank check the condition code returned past the backend. This will be used to handle 5xx responses specially.
• bereq.is_bgfetch: This field shows whether this was a backend request sent asynchronously when the customer received a response already from the enshroud, or if it's a synchronous asking that was sent when we didn't find the response in the enshroud.
• beresp.uncacheable: Nosotros can set this to true to prevent a response from beingness cached.
• abandon: This is a return keyword that we can use to make Varnish completely abandon the request, and render a synthesized 503 to the customer (or just do nothing, in case of an asynchronous background fetch).

With all these tools in our belt we can implement the requirements with the following logic.

            sub vcl_backend_response {     prepare beresp.ttl = 1m;     set up beresp.grace = 10m;      # This cake will make sure that if the upstream returns a 5xx, only we have the response in the cache (even if it'due south expired),     # nosotros fall back to the cached value (until the grace period is over).     if (beresp.condition == 500 || beresp.status == 502 || beresp.status == 503 || beresp.status == 504)     {         # This bank check is important. If is_bgfetch is true, it means that we've found and returned the cached object to the client,         # and triggered an asynchoronus background update. In that example, if information technology was a 5xx, we have to abandon, otherwise the previously buried object         # would be erased from the enshroud (even if we set uncacheable to true).         if (bereq.is_bgfetch)         {             render (carelessness);         }          # We should never cache a 5xx response.         set beresp.uncacheable = true;     } }                      

In the in a higher place code I'm simply handling the standard 500, 502, 503 and 504 codes, merely of form you can extend it if your backend is returning something else.

The key of the solution is using abandon in combination with if (bereq.is_bgfetch). If we returned abandon on every 5xx without checking bereq.is_bgfetch, then Varnish would always render its built-in 503 response to the client instead of the 5xx sent by the backend, so our client could never encounter the actual backend errors.

Of import: The field bereq.is_bgfetch is merely available starting from Varnish five.two.0. And depending on our Os version and installation method, we might exist on an before version, and so this is important to check with varnishd -5.

The configuration syntax of Varnish takes a chip of getting used to, but if set up properly, information technology provides an invaluable tool for improving the performance of our services, and meanwhile protecting our clients from outages. And I hope this post will be helpful when handling server side errors.

lunaalivink.blogspot.com

Source: https://blog.markvincze.com/how-to-gracefully-fall-back-to-cache-on-5xx-responses-with-varnish/

Share this post