Magento 2 and Varnish 6
- Install Varnish 6 on Debian 10
- Configure NGINX
- Modify the Varnish system configuration
- Modify default.vcl
- Configure Magento to use Varnish
- Export a Varnish Configuration File
- The Varnishing of the TopMenu
Install Varnish 6 on Debian 10
In order to get Varnish up and running type sudo apt-get install varnish. Enter the following command to display the version of Varnish you are running:
varnishd -V
varnishd (varnish-6.1.1 revision efc2f6c1536cf2272e471f5cff5f145239b19460)
Copyright (c) 2006 Verdens Gang AS
Copyright (c) 2006-2015 Varnish Software AS
Configure NGINX
Configure your web server to listen on a port other than the default port 80 because Varnish responds directly to incoming HTTP requests, not the web server. In the sections that follow, we use port 8080 as an example:
nano /etc/nginx/sites-available/magento.conf
Change the server to listen to port 8080:
upstream fastcgi_backend {
server unix:/run/php/php7.2-fpm.sock;
}
server {
listen 8080 default_server;
server_name your-server.de;
set $MAGE_ROOT /var/www/html/magento;
include /var/www/html/magento/nginx.conf.sample;
}
Test and reload NGINX:
nginx -t
service nginx reload
Modify the Varnish system configuration
As a user with root privileges, open your Vanish configuration file in a text editor:
nano /etc/default/varnish
Set the Varnish listen port to 80:
VARNISH_LISTEN_PORT = 80
Make sure that DAEMON_OPTS contains the correct listening port for the -a parameter:
DAEMON_OPTS =
'-a :80 \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-S /etc/varnish/secret \
-s malloc,256m'
Save your changes to the Varnish configuration file and exit the text editor.
Modify default.vcl
This section discusses how to provide minimal configuration so Varnish returns HTTP response headers. This enables you to verify Varnish works before you configure Magento to use Varnish.
- Back up default.vcl:
cp /etc/varnish/default.vcl /etc/varnish/default.vcl.bak
Open /etc/varnish/default.vcl in a text editor and locate the backend default and replace the value of .host with the fully qualified hostname or IP address and listen port of the Varnish backend or origin server; that is, the server providing the content Varnish will accelerate. Replace the value of .port with the web server’s listen port:
backend default {
.host = "my.domain.com";
.port = "8080";
}
Save your changes to default.vcl and exit the text editor and restart Varnish:
service varnish restart
Unfortunatly this did not change the Port Varnish was running on:
netstat -tulpn | grep varnish
tcp 0 0 0.0.0.0:6081 0.0.0.0:* LISTEN 1634/varnishd
tcp 0 0 127.0.0.1:6082 0.0.0.0:* LISTEN 1634/varnishd
tcp6 0 0 :::6081 :::* LISTEN 1634/varnishd
Following this guide, applying changes to the default service is best done by creating a new file /etc/systemd/system/varnish.service.d/customexec.conf:
# create the drop in directory
mkdir /etc/systemd/system/varnish.service.d
# create the drop in file. The name is irrelevant, as long as it ends in .conf
nano /etc/systemd/system/varnish.service.d/customexec.conf
Here you only need to add the settings you want do change, everything else will be loaded from the default definition file.
[Service]
ExecStart=/usr/sbin/varnishd -j unix,user=vcache -F -a :80 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m
Afterwards, tell systemctl to reload it's config files and to restart the service
systemctl daemon-reload
service varnish restart
Now I received an error message:
service varnish restart
Failed to restart varnish.service: Unit varnish.service has a bad unit file setting.
See system logs and 'systemctl status varnish.service' for details.
systemctl status varnish.service
Warning: The unit file, source configuration file or drop-ins of varnish.service changed on disk● varnish.service - Varnish HTTP accelerator
Loaded: bad-setting (Reason: Unit varnish.service has a bad unit file setting.)
Drop-In: /etc/systemd/system/varnish.service.d
└─customexec.conf
Active: active (running) since Fri 2020-02-07 11:10:15 CET; 33min ago
Docs: https://www.varnish-cache.org/docs/6.1/
man:varnishd
Main PID: 1634 (varnishd)
Tasks: 217 (limit: 4915)
Memory: 71.4M
CGroup: /system.slice/varnish.service
├─1634 /usr/sbin/varnishd -j unix,user=vcache -F -a :6081 -T localhost:6082 -f /etc/v └─1646 /usr/sbin/varnishd -j unix,user=vcache -F -a :6081 -T localhost:6082 -f /etc/v
Feb 07 11:10:15 Magento2 systemd[1]: Started Varnish HTTP accelerator.
Feb 07 11:10:16 Magento2 varnishd[1634]: Debug: Version: varnish-6.1.1 revision efc2f6c1536cf227Feb 07 11:10:16 Magento2 varnishd[1634]: Debug: Platform: Linux,4.19.0-6-amd64,x86_64,-junix,-smFeb 07 11:10:16 Magento2 varnishd[1634]: Version: varnish-6.1.1 revision efc2f6c1536cf2272e471f5Feb 07 11:10:16 Magento2 varnishd[1634]: Platform: Linux,4.19.0-6-amd64,x86_64,-junix,-smalloc,-Feb 07 11:10:16 Magento2 varnishd[1634]: Debug: Child (1646) Started
Feb 07 11:10:16 Magento2 varnishd[1634]: Child (1646) Started
Feb 07 11:10:16 Magento2 varnishd[1634]: Info: Child (1646) said Child starts
Feb 07 11:10:16 Magento2 varnishd[1634]: Child (1646) said Child starts
Feb 07 11:43:32 Magento2 systemd[1]: varnish.service: Service has more than one ExecStart= setti
The duplicated ExecStart line is in /lib/systemd/system/varnish.service:
cat /lib/systemd/system/varnish.service
[Unit]
Description=Varnish HTTP accelerator
Documentation=https://www.varnish-cache.org/docs/6.1/ man:varnishd
[Service]
Type=simple
LimitNOFILE=131072
LimitMEMLOCK=82000
ExecStart=/usr/sbin/varnishd -j unix,user=vcache -F -a :6081 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m
ExecReload=/usr/share/varnish/varnishreload
ProtectSystem=full
ProtectHome=true
PrivateTmp=true
PrivateDevices=true
[Install]
WantedBy=multi-user.target
Note that the drop-in should have an empty ExecStart= to prevent this problem:
[Service]
ExecStart=
ExecStart=/usr/sbin/varnishd -j unix,user=vcache -F -a :80 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m
Varnish is now running on port 80:
netstat -tulpn | grep varnish
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 2972/varnishd
tcp 0 0 127.0.0.1:6082 0.0.0.0:* LISTEN 2972/varnishd
tcp6 0 0 :::80 :::* LISTEN 2972/varnishd
And I am able to access the Magento store again on port 80. You can check that your shop is served by Varnish by querying the HTTP header of your page:
curl -I http://my.domain.com/
HTTP/1.1 200 OK
Server: nginx/1.14.2
Date: Fri, 07 Feb 2020 12:46:23 GMT
Content-Type: text/html; charset=UTF-8
Set-Cookie: X-Magento-Vary=6b1086e51dc44ada73095007287c835c2e8a8cb2; expires=Fri, 07-Feb-2020 13:46:23 GMT; Max-Age=3600; path=/; HttpOnly
X-Magento-Cache-Control: max-age=86400, public, s-maxage=86400
X-Magento-Cache-Debug: MISS
X-Magento-Tags: store,cms_b,FPC
Pragma: no-cache
Cache-Control: max-age=0, must-revalidate, no-cache, no-store
Expires: Thu, 07 Feb 2019 12:46:23 GMT
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Content-Encoding: gzip
Vary: Accept-Encoding
X-Varnish: 98400
Age: 0
Via: 1.1 varnish (Varnish/6.1)
Connection: keep-alive
Before you can look at headers, you must set Magento for developer mode. You can also use the magento deploy:mode:set command.
Make sure Varnish is running then enter the following command on the Varnish server and click on a link on your website:
varnishlog
Configure Magento to use Varnish
- Log in to the Magento Admin as an administrator.
- Click Stores > Configuration > Advanced > System > Full Page Cache.
- From the Caching Application list, click Varnish Caching.
- Enter a value in the TTL for public content field.
- Expand Varnish Configuration and enter the following information:
| Field | Description |
|---|---|
| Access list | Enter the fully qualified hostname, IP address, or Classless Inter-Domain Routing (CIDR) notation IP address range for which to invalidate content. More information |
| Backend host | Enter the fully qualified hostname or IP address and listen port of the Varnish backend or origin server; that is, the server providing the content Varnish will accelerate. Typically, this is your web server. More information |
| Backend port | Origin server's listen port. |
| Grace period | The grace period determines how long Varnish serves stale content if the backend is not responsive. The default value is 300 seconds. |
- Click Save Config.
Export a Varnish Configuration File
- Click one of the export button to create a
varnish.vclyou can use with Varnish.
-
Upload the file to
/etc/varnish/varnish.vcl -
It is recommended to change the value of
acl purgeto the IP address of the Varnish host.
acl purge {
"localhost";
}
- Edit
nano /etc/default/varnishto add the new configuration file:
DAEMON_OPTS =
'-a :80 \
-T localhost:6082 \
-f /etc/varnish/varnish.vcl \
-S /etc/varnish/secret \
-s malloc,256m'
- Static files should not be cached by default, but if you want to cache them, you can edit the section
Static files cachingin the VCL to have the following content:
# Static files should not be cached by default
return (pass);
# But if you use a few locales and don't use CDN you can enable caching static files by commenting previous line (#return (pass);) and uncommenting next 3 lines
#unset req.http.Https;
#unset req.http./* */;
#unset req.http.Cookie;
- And reload Varnish and NGINX:
systemctl daemon-reload
service varnish restart
service nginx restart
- Now that you’re using the
default.vclgenerated for you by Magento, you can perform some final verifications to make sure Varnish is working.
curl -I -v --location-trusted 'http://my.domain.com'
The Varnishing of the TopMenu
After switching to Varnish I noticed that my topmenu module was no longer loading. Checking the page source showed me that the Edge Side Include was failing:
The problem in my case was a ttl="3600" parameter given in the menu component that has to be deleted:
<?xml version="1.0"?>
<page xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:View/Layout/etc/page_configuration.xsd">
<body>
<referenceContainer name="div.sidebar.main">
<block class="TemplateMonster\Megamenu\Block\Html\Topmenu"
name="catalog.sidebarnav"
template="TemplateMonster_Megamenu::html/topmenu.phtml"
ifconfig="megamenu/config/megamenu_general_show_left"
before="-"
ttl="3600" />
</referenceContainer>
<referenceBlock name="catalog.topnav" template="TemplateMonster_Megamenu::html/topmenu.phtml" />
</body>
I found this block in themeXXX\app\code\TemplateMonster\Megamenu\view\frontend\layout\default.xml in the template source. And on the server inside the magento installation dir /vendor/magento/module-theme/view/frontend/layout/default.xml.
There might also be issues with HTTPS on the backend side:
nano /etc/default/varnish
In the DAEMON_OPTS variable, add -p feature=+esi_ignore_https, -p feature=+esi_ignore_other_elements, -p feature=+esi_disable_xml_check. This would look like:
DAEMON_OPTS =
'-a :80 \
-p feature=+esi_ignore_other_elements \
-p feature=+esi_disable_xml_check \
-p feature=+esi_ignore_https \
-T localhost:6082 \
-f /etc/varnish/varnish.vcl \
-S /etc/varnish/secret \
-s malloc,256m'
When you change this, you need to run service varnish restart for the changes to take effect.
Going to Production (HTTPS)
I am now going to use NGINX to add SSL encryption to my cached content - since Varnish does not natively supports encryption.
/etc/varnish/default.vcl
Magento will still be served on port 8080 by NGINX - so we have to set the Varnish default backend accordingly (this is the default - nothing to change here):
backend default {
.host = "localhost";
.port = "8080";
.first_byte_timeout = 6000s;
}
/etc/default/varnish
For testing I set the Varnish listening port to port 80 - I am now reversing it back to port 6081 - the reasoning is that I will tell NGINX to automatically upgrade all incoming traffic on port 80 (HTTP) to port 443 (HTTPS). Port 6081 will only be used on localhost and can be blocked to outside traffic:
DAEMON_OPTS="-a :6081 \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-S /etc/varnish/secret \
-s malloc,256m"
/etc/systemd/system/varnish.service
The SystemD service file also specifies the ports being used by Varnish. This seems redundant (they are also defined in the varnish configuration). So either remove them (i did not try) or revert them back to use the default ports 6081 and 6082:
[Unit]
Description=Varnish HTTP accelerator
Documentation=https://www.varnish-cache.org/docs/6.1/ man:varnishd
[Service]
Type=simple
LimitNOFILE=131072
LimitMEMLOCK=82000
ExecStart=
ExecStart=/usr/sbin/varnishd -j unix,user=vcache -F -a :6081 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m
ExecReload=/usr/share/varnish/varnishreload
ProtectSystem=full
ProtectHome=true
PrivateTmp=true
PrivateDevices=true
[Install]
WantedBy=multi-user.target
/etc/nginx/conf.d/default.conf
The NGINX server needs 3 blocks:
- server block port 80: forced upgrade to HTTPS
443 - server block port 8080: execute the Magento backend and make your shop available to Varnish
- server block port 443:
proxy_passthe Varnish cash from port6081to the web and add encryption
upstream fastcgi_backend {
server unix:/run/php/php7.3-fpm.sock;
}
server {
listen 80;
listen [::]:80;
server_name my.server-domain.com;
return 301 https://$server_name$request_uri;
}
server {
listen 8080;
server_name my.server-domain.com;
set $MAGE_ROOT /var/www/magento2;
set $MAGE_MODE developer; # or production
include /var/www/magento2/nginx.conf.sample;
fastcgi_read_timeout 600;
proxy_read_timeout 600;
set $MAGE_DEBUG_SHOW_ARGS 1;
set $MAGE_ROOT /var/www/magento2;
location ~ (index|get|static|report|404|503)\.php$ {
fastcgi_pass unix:/run/php/php7.0-fpm.sock;
fastcgi_buffers 1024 4k;
fastcgi_read_timeout 600s;
fastcgi_connect_timeout 600s;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
location / {
try_files $uri $uri/ /index.php$is_args$args;
proxy_read_timeout 600s;
proxy_connect_timeout 75s;
proxy_headers_hash_max_size 5120000;
proxy_headers_hash_bucket_size 640000;
}
#proxy the php scripts to php-fpm
location ~ \.php$ {
include /etc/nginx/fastcgi.conf;
fastcgi_intercept_errors on;
fastcgi_pass unix:/var/run/php7.3-fpm.sock;
}
}
server {
listen 443 ssl http2 default_server;
listen [::]:443 ssl;
server_name my.server-domain.com; # managed by Certbot
ssl on;
# ssl_certificate /etc/nginx/pems/public_cert.pem;
# ssl_certificate_key /etc/nginx/pems/privat_key.pem;
ssl_certificate /etc/letsencrypt/live/my.server-domain.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/my.server-domain.com/privkey.pem; # managed by Certbot
include /etc/nginx/ssl/ssl-params.conf; # add additional SSL parameter
include /etc/nginx/conf.d/header.conf; ## add additional HEADER
location / {
proxy_pass http://127.0.0.1:6081;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Port 443;
proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;
fastcgi_buffer_size 32k;
fastcgi_buffers 4 32k;
}
}
Check if Varnish is Running
curl -I https://my.domain.com
HTTP/2 200
server: nginx
date: Thu, 29 Apr 2021 09:19:57 GMT
content-type: text/html; charset=UTF-8
vary: Accept-Encoding
vary: Accept-Encoding
strict-transport-security: max-age=31536000
content-security-policy: upgrade-insecure-requests;
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
x-magento-cache-control: max-age=86400, public, s-maxage=86400
age: 35212
x-magento-cache-debug: HIT
grace: none
pragma: no-cache
expires: -1
cache-control: no-store, no-cache, must-revalidate, max-age=0
accept-ranges: bytes
x-frame-options: DENY
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
cache-control: public, must-revalidate, proxy-revalidate, max-age=0
You should see either a x-magento-cache-debug: HIT or x-magento-cache-debug: MISS. If you are not seeing it:
-
Make sure that you activated the DEVELOPER mode in the Magento Server Block in NGINX:
set $MAGE_MODE developer; -
Activate Full Page Cache under System/Cache Management:
bin/magento cache:status
Current status:
config: 1
layout: 1
block_html: 1
collections: 1
reflection: 1
db_ddl: 1
compiled_config: 1
eav: 1
customer_notification: 1
config_integration: 1
config_integration_api: 1
google_product: 1
full_page: 0
config_webservice: 1
translate: 1
vertex: 1
Excluding Backend URLs
default.vcl
Add exclude statements to /etc/varnish/default.vcl:
sub vcl_recv {
if (req.url == "/example.html") {
return(pass);
}
# Static files caching
if (req.url ~ "/downloads/") {
# Static files should not be cached by default
return (pass);
# But if you use a few locales and don't use CDN you can enable caching static files by commenting previous line (#return (pass);) and uncommenting next 3 lines
#unset req.http.Https;
#unset req.http.SSL-OFFLOADED;
#unset req.http.Cookie;
}
if (req.url ~ "^/(pub/)?(media|static)/") {
# Static files should not be cached by default
return (pass);
}
return (hash);
}
sub vcl_backend_response {
if (bereq.url == "/example.html") {
set beresp.uncacheable = true;
set beresp.ttl = 120s;
return(deliver);
}
if (bereq.url ~ "/downloads/") {
set beresp.uncacheable = true;
set beresp.ttl = 120s;
return(deliver);
}
if (bereq.url ~ "^/(pub/)?(media|static)/") {
set beresp.uncacheable = true;
set beresp.ttl = 120s;
return(deliver);
}
return (deliver);
}
Testing
Now check your pages for the x-magento-cache-debug: HIT result:
curl -I https://www.instar.com/
HTTP/2 200
server: nginx
date: Tue, 27 Jul 2021 07:30:13 GMT
content-type: text/html; charset=UTF-8
vary: Accept-Encoding
vary: Accept-Encoding
strict-transport-security: max-age=31536000
content-security-policy: upgrade-insecure-requests;
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
x-magento-cache-control: max-age=86400, public, s-maxage=86400
age: 83
x-magento-cache-debug: HIT
grace: none
pragma: no-cache
expires: -1
cache-control: no-store, no-cache, must-revalidate, max-age=0
accept-ranges: bytes
x-frame-options: DENY
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
cache-control: public, must-revalidate, proxy-revalidate, max-age=0
access-control-allow-origin: *
curl -I https://www.instar.com/static/test.txt
HTTP/2 200
server: nginx
date: Tue, 27 Jul 2021 07:31:10 GMT
content-type: text/plain; charset=utf-8
content-length: 10
last-modified: Tue, 06 Jul 2021 06:55:08 GMT
etag: "60e3fe4c-a"
accept-ranges: bytes
x-frame-options: DENY
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
cache-control: public, must-revalidate, proxy-revalidate, max-age=0
access-control-allow-origin: *
curl -I https://www.instar.com/instar/downloads/IN-9008-PoE_FW_4.1.2.48_WebUI_3.0.\(328\)_update.zip
HTTP/2 200
server: nginx
date: Tue, 27 Jul 2021 07:32:19 GMT
content-type: application/zip
content-length: 7962962
last-modified: Mon, 14 Dec 2020 09:45:54 GMT
etag: "5fd73452-798152"
accept-ranges: bytes
pragma: no-cache
expires: -1
cache-control: no-store, no-cache, must-revalidate, max-age=0
x-frame-options: DENY
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
cache-control: public, must-revalidate, proxy-revalidate, max-age=0
access-control-allow-origin: *