On a modern operating system, it is likely that some form of threat management system is installed. While in the past only anti-virus and port/application-based firewall solutions were common on hosts, intrusion prevention systems (IPS) are increasingly becoming the norm. These can perform traffic and behaviour analysis, and block suspicious activity.
To a threat management system, MOUNT10’s behaviour can appear suspicious. If not excluded from threat management system monitoring, they can prevent MOUNT10 from backing up successfully.
It is important to understand how MOUNT10 can be disrupted by threat management systems, and what to look for when troubleshooting. This guide demonstrates a number of scenarios in general terms.
Scenarios
- Anti-virus interferes with Backup Client installation
- Firewall / IPS prevents transmission to Storage Platform
- Firewall / IPS prevents communication between MOUNT10 processes
- General troubleshooting strategy for anti-virus/IPS problems
- Exclusions
I. Anti-virus interferes with Backup Client installation
1. Installation interference
Symptom: Backups will not run and sometimes a connection to the Storage Platform cannot be established.
Example error message: Various – Backup Client may halt indefinitely, e.g. “Building Selection List”.
Cause: This scenario typically occurs when anti-virus exclusions have not been applied to exclude the MOUNT10 installation folder (default location: C:\Program Files\MOUNT10\Backup Client or Backup Client SE). The anti-virus scans the installation folder and incorrectly identifies the working files and folders, including those of the Java installation, as potential virus threats. It often deletes or blocks access to these.
Solution: Exclude the installation folder from anti-virus scans. A Backup Client reinstall, and reconnect to the Backup Account may be required to get the Backup Client back into a known good state.
2. Backup Client database lock and corruption
Symptom: The Backup Client reports a database error when trying to run a backup, or reports that the database is locked.
Example Error Message:
- In the MOUNT10 GUI or backup log file: “file is encrypted or is not a database” or “Could not calculate differences between new and previous backup sets:database is full”.
- In the MOUNT10 GUI or backup log file: “Unable to recover from exception in backup process”.
- In the MOUNT10 GUI or backupservice.log file: “com.attix5.sqlite.SQLiteException: database is locked”.
Cause: Anti-virus has either scanned and corrupted the Backup Client database (backuplist.db) or has a file handle on the database, preventing the Backup Client from getting an exclusive lock.
Solution: Exclude the database folder from anti-virus scans. It may be necessary to delete the backuplist.db file and reconnect to the Backup Account in order to get the last known good version of the Backup Client database.
3. Cache corruption
Symptom: Files are flagged as “trouble” by either the Backup Client or Storage Platform and larger than expected data transfers are occurring. Errors can also be seen during the adding/patching phase of a Staged Backup.
Example error message:
- In the Backup Client log when patching “Unable to patch file” and “Trouble file encountered and added as full file”.
- In the Backup Client log when updating the cache “Patching cache file failed”, “Could not apply patch to file…”, “Could not move backup file to cache… reason: Trying to move a file to the cache that does not exist” and “Flagged file for full backup”.
- During a Staged Backup in the adding/patching file stage, in the backupservice.log file: “WARN com.attix5.service.spcomms.StoragePlatformConnection – File not found in toBackup”.
Cause: Files in the cache or ToBackup folders have been altered by the anti-virus application which incorrectly handled them as a threat. The cache file checksums either do not exist or no longer match the Backup Client database, cannot be patched, and so must be resent in full. This can result in larger than expected data transfers. Deletion of files in the ToBackup folder can cause the backup to fail.
Solution: Exclude the cache and ToBackup folders from anti-virus scans. This will prevent the files being altered and prevent them being resent in full.
4. Anti-virus interferes with file read and write operations
Symptom: File reads fail or are unusually slow. Some files within the VHDTemp and WindowsImageBackup are shown as failing to rename in the backupservice.log files, which causes the System State to fail.
Example error message: Backup Client log may show “Unable to read file”. Slow backups will not show this error, but the time taken to process each file will be noticeably slower than normal.
The following may be seen in the backupservice.log when System State rename failures are occurring:
- “Could not rename RenamedFolder”
- “WBAdminPlugin – Could not delete E:\WindowsImageBackup\SERVERNAME\SystemStateBackup\RenamedFolder\VHD-Hex-ID” (e.g. 5106e16c-60d3-11de-ae0d-806e6f6e6963)
- “WBAdminPlugin – Could not rename E:\WindowsImageBackup\SERVERNAME\SystemStateBackup\RenamedFolder to E:\WindowsImageBackup\SERVERNAME\SystemStateBackup\Backup YYYY-MM-DD HHMMSS” (e.g. 2013-28-02 093759)
Cause: File reads and writes are being monitored by anti-virus on-access scanning, which can cause read failure or slow backup speeds. The failure to rename is caused by anti-virus holding the file open during this process.
Solution: Exclude the installation folder and MOUNT10 Backup Client service (a5backup.exe or a5backup64.exe on 64-bit machines) from anti-virus scanning. Also ensure WindowsImageBackup and VHDTemp are excluded from scanning.
5. Communication error during restore
Symptom: During the Receiving files process of the restore the client received a communications error
Example error message: Error: 14:17:11 Communications error: bad record MAC
Cause: Certain anti-virus/security applications intercept and sometimes modify the ssl packets leading to a “bad mac record” error (eg Webroot secureAnywhere supplied by some online Banking sites).
Solution: Uninstall/Disable the anti-virus application. Alternatively, resuming the restore works in most cases.
II. Firewall / IPS prevents transmission to Storage Platform
1. Backup Client cannot contact AccountServer
Symptom: The Backup Client will not authenticate against the AccountServer. The AccountServer can be contacted by the Storage Platform Communication Test tool or telnet. The Backup Client reports a read I/O failure, and does not proceed to send data to the StorageServer. Multiple retries may be seen.
Example error message: In the Backup Client log when initiating a connection “Cannot connect to Storage Platform: Connection refused: connect” or “IOException connecting to Storage Platform: Connection timed out: connect”.
Cause: Traffic from the Backup Client to the AccountServer is intercepted and blocked or dropped by a firewall or IPS, as it believes it to be suspicious.
Solution: Ensure that traffic from the MOUNT10 Backup Client service to the AccountServer is permitted in the firewall or IPS rules.
2. Backup Client cannot contact StorageServer
Symptom: The Backup Client does not send data to the StorageServer, or stops sending after a period of time. The Backup Client reports a read I/O failure. Multiple retries may be seen. Backups do not complete. In some circumstances, only small file selections will back up successfully.
Example error message: In the Backup Client log when initiating a connection “Could not create backup: Failed to initiate streaming backup” or “IOException connecting to Storage Platform: Connection timed out: connect”.
If a connection drops unexpectedly “Backup transfer failed”, “sendBackup could not send backup file: Exception in writer thread: Connection reset by peer: socket write error” or “sendBackup could not send backup index file: Connection reset by peer: socket write error”.
Cause: Traffic from the Backup Client to the StorageServer is intercepted and blocked or dropped by a firewall or intrusion prevention system, as it believes it to be suspicious. The drop may not happen immediately, but after a period of time or volume of data.
Solution: Ensure that traffic from the MOUNT10 Pro Backup Client Service to the AccountServer is permitted in the firewall or IPS rules.
Note: There are other issues that produce the same or very similar symptoms to these. Speed and duplex settings mismatch or errors can cause similar network traffic drops. Also, ISP-based traffic shaping or throttling can prevent effective communication. These causes should not be ruled out when investigating AccountServer and StorageServer communication issues.
III. Firewall / IPS prevents communication between MOUNT10 processes
1. System tray cannot open Backup Client GUI
Symptom: When right clicking the system tray icon and clicking Open, it is reported that “The Backup Service is not running.” On inspecting the Windows Services menu, the Backup Service is running. Restarting the system tray application does not resolve the problem.
Example error message: In a dialog box “The Backup Service is not running”.
Cause: A host firewall / IPS is blocking communication between the system tray application and the Backup Service. Believing the service is stopped, the system tray does not start the Backup Client GUI.
Solution: Ensure that the system tray application (A5Loader.exe on Desktop and Laptop Edition, SERunner.exe on Server Edition) and Backup Service (a5backup.exe, a5backup64 on 64-bit machines) are permitted in the firewall or IPS rules.
2. Backup Client GUI and Service cannot communicate
Symptom: The Backup Client GUI does not load the backup selection, does not save settings when trying to close, or is unresponsive. The Backup Service is running.
Note: This behaviour is similar to the Backup Service having stopped.
Example error message: Might not display an error message. The GUI may fail to load the backup selection tree from the service. For example:
Cause: A host firewall / IPS is blocking communication between the Backup Client GUI and the Backup Client Service.
Solution: Ensure that the Backup Client GUI (javaw.exe) and Backup Client Service are permitted in the firewall or IPS rules.
3. Backup Service and Exchange agent service cannot communicate
Symptom: When running an Exchange SIR Plus backup or during configuration, a communication error is shown.
Example error message: In the backup log file “Could not communicate with the Exchange Agent: Error in call with Exchange Agent, Connection refused: connect.”
Cause: A host firewall / IPS is blocking communication between the Backup Client Service and the Exchange Agent Service (A5EA.exe) used by the SIR Plus plug-in.
Solution: Ensure that the Exchange Agent Service and Backup Client Service are permitted in the firewall or IPS rules.
4. Storage Platform Console and Backup Client Remote Management cannot communicate
Symptom: The Storage Platform Console cannot connect to the Remote Management of a Backup Client.
Example error message: In a Storage Platform Console dialog box “The operation has timed out”.
Cause: A host firewall / IPS is blocking communication between the Storage Platform Console and the Backup Client Service.
Solution: Ensure that access to the Remote Management port (default 9091) of the Backup Client Service is permitted in the firewall or IPS rules.
Note: When connecting over a network, bear in mind that network-based firewalls and IPS systems can also prevent Remote Management connections being established. Additionally, any Backup Clients sitting behind Network Address Translation will require port forwarding to be configured.
IV. General troubleshooting strategy for anti-virus/IPS problems
If the MOUNT10 software is not behaving as expected, an anti-virus (AV) program or IPS is usually the cause. Possible interactions also exist with third-party software such as disk optimisation and monitoring software.
We recommend the following steps to troubleshoot the issue:
1. Apply AV exclusions
- Apply exclusions to the AV for MOUNT10 folders and executables.
- Test the results after each exclusion and repeat the process until all reasonable configurations have been tried.
Note:
- Often “On Access” and “Scheduled” exclusions are separate, and may need to be configured separately. Similarly process and file access may require separate configuration.
- Due to the large variety of AV/IPS software, exclusion requirements may vary greatly. However, refer to the sections E, F and G below for a exclusion recommendations.
2. Eliminate known third-party software
- Possible interactions exist with third-party software such as disk optimisation and monitoring software. If exclusions can be applied, exclude the MOUNT10 folders and processes (as per E-F below) or disable the software.
Note: Other third-party backup applications running concurrently with MOUNT10 can also cause problems and could be mistaken for AV behaviour. - Test the result after each action.
3. Disable the AV
- Failing Step 1 and 2, identify and disable any other active AVs.
- Disable the main AV altogether, testing the outcome between actions.
4. Uninstall all AVs
- Temporarily remove the AV software to rule out other AV processes running in the background that are not subject to configurable exclusions.
Note: MOUNT10 have found that some AVs disregard exclusions and continue to run anyway.
5. Eliminate unknown third-party software interactions
An unknown piece of third-party software is interacting with MOUNT10. We recommend a process of elimination in a lab environment where problems can be recreated and software uninstalled.
V. Exclusions
1. Recommended exclusions based on defaults for SE
Installation
- C:\Program Files\MOUNT10\Backup Client ESE\ and all sub folders.
- System State Folders for Windows 2008 (assuming C: is used by System State).
- C:\VHDTemp\.
- C:\WindowsImageBackup\.
- \\.\GLOBALROOT\Device\ (especially important when doing System State backups)
If any working folders or plug-in “dump” locations are moved, ensure that these are also excluded.
Processes
- C:\Program Files\MOUNT10\Backup Client SE\a5backup.exe (or a5backup64.exe on 64 bit systems)
- C:\Program Files\MOUNT10\Backup Client SE\SERunner.exe
- C:\Program Files\MOUNT10\Backup Client SE\SplitVHD.exe and SplitVHDX.exe
- Javaw.exe (Launched by SERunner.exe process)
TCP ports
- 9091 for Remote Management
- 8011 for Exchange Agent Service for SIR Plus
- The other ports used by the Backup Client are randomly selected, but can be manually specified in the a5backup.properties file:
- service.rpc-server.localhost.port=port number 1-65535
- service.rmi.port=port number 1-65535
- service.port=port number 1-65535
The ports must be unique, and not clash with any existing services. Ensure the GUI is closed and that the service is stopped when editing the file.
2. Recommended exclusions based on defaults for DL
Installation
C:\Program Files\MOUNT10\Backup Client\.
If any working folder locations are moved, ensure that these are also excluded.
Processes
- C:\Program Files\MOUNT10\Backup Client\a5backup.exe (or a5backup64.exe on 64 bit systems)
- C:\Program Files\MOUNT10\Backup Client\A5Loader.exe
- Javaw.exe (Launched by A5Loader.exe process)
TCP ports
- 9091 for Remote Management
- The other ports used by the Backup Client are randomly selected, but can be manually specified in the a5backup.properties file:
- service.rpc-server.localhost.port=port number 1-65535
- service.rmi.port=port number 1-65535
- service.port=port number 1-65535
The ports must be unique, and not clash with any existing services. Ensure the GUI is closed and service is stopped when editing the file.
3. Recommended exclusions based on defaults for ESE
Installation
- C:\Program Files\MOUNT10\Backup Client ESE\ and all sub folders.
- C:\ProgramData\MOUNT10\ and sub-folders
Processes
- C:\Program Files\MOUNT10\Backup Client ESE\a5backup64.exe
- C:\Program Files\MOUNT10\Backup Client ESE\SERunner.exe
- C:\Program Files\MOUNT10\Backup Client ESE\a5Loader.exe
- Javaw.exe (Launched by SERunner.exe process)
TCP ports
- The other ports used by the Backup Client are randomly selected, but can be manually specified in the a5backup.properties file:
- service.rpc-server.localhost.port=port number 1-65535
- service.rmi.port=port number 1-65535
- service.port=port number 1-65535
The ports must be unique, and not clash with any existing services. Ensure the GUI is closed and that the service is stopped when editing the file.