Virtualised Backup Performance

Whilst recently experiencing performance issues running a virtualised backup on a customers site, I thought I would blog some useful steps to help diagnose issues with virtualised backup performance with products such as vRanger and Veeam a lot of these also apply to physical backups.

This is a list of things to consider / try when trying to diagnose the issues you are having, I will add to it over time but please feel free to comment further suggestions or changes.

• Network
• ESX
• Storage
• Anti-Virus / Third Party Tools
• Veeam  Specific Settings

Network

1. Are you using a Gb connection or 100Mb lan connection?
2. Is the ESX host, VM and Backup server all seeing the expected network connection speed?
3. Do a ping test between all the effected elements, are you loosing any packets? are you seeing the expected response times?
4. Is the backup server on the same subnet as the ESX Host or VM being backed up? If not could the routing device be causing a bottle neck? Is it a 100Mb routing device or Gb? Do a tracert to ensure you are taking the expected route. Are you able to temporarly put this backup server on the same network and test backup performance?
5. Are you using the latest drivers for your NIC?
6. What is your backup server / VM / ESX host currently doing? Could this be causing a networking bottleneck?
7. Try a standard file transfer between your VM and the backup server, how does this perform? Use Veeam FastSCP to do a file transfer between the ESX host and your backup server, how does this perform?

ESX

1. Are you using VCB or Network backup? If you are using VCB try network backup, how does this effect the performance? Have you followed the relevent VCB articles on VMware’s website? Are you using a FC or iSCSI SAN? If you are using iSCSI are you expecting a performance increase by using VCB? or are you just looking to remove backup from LAN or take load away from the ESX boxes?
2. How much memory have you reserved for the service console? Try and set it to 800Mb, does this improve the performance?
3. Have you reserved CPU for the service console? Try increasing this from the default settings.

Storage

1. Where are you storing your backups? What performance do you get when copying a standard file to this store? How are you connecting to this storage and what are the bandwidth limitation of this connection?
2. Are you connecting to a NAS or SAN via iSCSI? If you are using a software iSCSI initiator how is this effecting the CPU usage of your backup server?
3. Are the drivers and firmware for you FC card up to date?
4. Is anything else being activly written to this device? How many backups are you trying to run at once to this store?

Anti Virus / Third Party Tools

1. Is your anti virus causing a bottle neck for reads / writes? Is it possible to setup exclusions or temporaliy disable the AV to test? (Please ensure you don’t leave your AV permently disabled!?)
2. Is there any other software on the server that could be causing a bottleneck for your backup?

Veeam Specific Settings

1. If using Network mode have you enabled the SSH client on the ESX host as instructed when you added your hosts?
2. If using full fat ESX rather than ESXi try changing the “Data Transfer Engine” under properties of your host in Veeam to “Force Service Console Agent Mode” note, the default “Automatic” mode will actually always attempt to use SSH on “fat” ESX host if it configured (previous bullet).
3. How many backups are you running at once? Note when using network backup mode, multiple backups may slow down indvidual job rates, but the overall speed maybe quicker, test different mixes until you find a sweet spot. For VCB SAN mode though, backup storage disk speed is often a bottleneck, so unless parallel jobs write to different disks, there’s little sense to run many jobs in parallel.
4. What commpression level is your backup set to? If currently set to best try Optimal.