Hi all,
In this blog I wanted to share a lessons learned when using VMware NSX with Zerto for disaster recovery purposes. Just a bit of background first, I am using for this blogpost a 3-site architecture:
- Management site: here vSphere, NSX manager and the Zerto Virtual Manager are hosted. This site manages both Site A and Site B.
- Site A: Our production site with a web and an application server.
- Site B: Our DR site for fail-over purposes.
Background
For Disaster Recovery (DR) we use Zerto to protect both our
WEB and APP servers. Zerto copies the data from both VM’s from Site A to Site B,
so in case site A becomes unavailable we can recover the servers.
To protect the servers from a network and security
perspective we use VMware NSX and leverage the Distributed Firewall (DFW) and NSXs’
micro-segmentation capabilities. For our application servers we create an APP
security group and for our WEB servers we create a WEB security group.
SG-APP
SG-WEB
Notice that we use the VM object for the membership in the
security groups. This is key to the problem we are facing.
Once the security groups are configured we can configure the
DFW. Here we create the following rules.
For the web server anyone should be able to access this on port 443. Meanwhile the web server needs to talk to the application
server on port 50123 to retrieve its configuration. All other traffic is being
blocked.
The problem
The application works fine on site A and only
the allowed traffic is passing through. Now we want to test the DR strategy.
Using Zerto we have created a Virtual Protection Group (VPG) protecting the
application (APP and WEB) from Site A to Site B.
Now we test the failover process and bring the application
online at Site B. Both the VM’s failover without any issues and boot up fine. However,
when we try and access the application using our favourite browser we noticed
that the application is no longer working…. We also notice that when we console
onto the web server, it is unable to read its configuration from the app
server.
Unfortunately this is expected behavior, so let’s unpack
this a bit further.
Under the bonnet
In VMware a virtual machine is a unique object
with an ID attribute. You can find this when looking in the .VMX file of the virtual
server under VC.UUID. The below snippet is from our APP server before we fail
the server across with Zerto.
Once we fail over the app server and look at the .VMX file
we see that the UUID is different.
Similar to our WEB server: before Zerto failover
And after Zerto failover it is
As you can see the UUID’s are different after each failover.
Now I am not 100% sure, but I believe that this unique
object is used to link the VM object with the NSX security group inside the vCentre’s
database.
When Zerto protects the virtual server it only copies across
the virtual disks as the disks contains the data. Then whenever a failover is invoked
Zerto creates a new VM object within vCenter and attaches the replicated
virtual disks to this new VM object. (Just a quick side note here, I believe
other DR solutions integrating with VMware use this same method.)
This causes a problem when using NSX in certain scenarios. Especially
when using the “Virtual Machine” objects to populate the security groups.
The figure below shows the security groups memberships after
the fail-over. As you can see our servers are no longer in their security
groups…. resulting to the firewall rules no longer being applied.
Solution
During a real-life Disaster Recovery situation every minute
counts, so having to manually re-add all the new VM objects to the security
groups will cost precious time. Automation here is key!
Below is a quick and dirty Powershell script that adds the new VM
objects into the existing security groups ensuring the firewall rules are again
applied correctly. The intent for this script is only to showcase a potential
way on how to use automation to re-apply the firewall rules after a failover. Please modify this script to suit your needs and test
adequately. Disclaimer: use this script at your own risk!!!
There are 2 main functions:
- backup-nsxsecuritygroupconfig à this function creates an extract of all security groups and their members and saves it into an XML file
- restore-securitygroupmember à this function reads in the XML file and compares it with the current security group configuration. Any missing members are added back into the security group
The script uses a common denominator between the original VM
object and the new one. Fortunately we can use the MAC address IF (AND ONLY IF)
you have configured the Zerto VPG to retain the MAC address of the servers
after fail-over. THIS IS KEY to using the script.
Prerequisites
VMware PowerCli:(version i used is 6.5)
Component Versions
---------------
VMware Cis Core PowerCLI
Component 6.5 build 4624453
VMware VimAutomation
Core PowerCLI Component 6.5 build 4624450
VMWare ImageBuilder
PowerCLI Component 6.5 build 4561891
VMWare AutoDeploy
PowerCLI Component 6.5 build 4561891
VMware Vds PowerCLI
Component 6.5 build 4624695
VMware Cloud
PowerCLI Component 6.5 build 4624821
VMware HA PowerCLI
Component 6.0 build 4525225
VMware HorizonView
PowerCLI Component 7.0.2 build 4596620
VMware Licensing
PowerCLI Component 6.5 build 4624822
VMware PCloud
PowerCLI Component 6.5 build 4624825
VMware Storage
PowerCLI Component 6.5 build 4624820
VMware vROps
PowerCLI Component 6.5 build 4624824
VMware vSphere
Update Manager PowerCLI 6.5 build 4540462
PowerNSX
PowerCLI C:\> Get-PowerNsxVersion
Version
Path
Author
CompanyName
------- ----
------
-----------
0.0
C:\powernsx\NSXpowershell\PowerNSX.psm1
NOTE: PowerNSX currently is not a supported module by
VMware.
#import the powershell NSX module and update the path correctly.
Import-Module C:\temp\NSXpowershell\PowerNSX.psm1
#function to go through the security groups and save it to the file specified.
function backup-nsxsecuritygroupconfig($savefile){
#variables
$nsxsecgroups =@()
$output ="security group name `t member"
$strsecgroup = ""
$strsecgroupmember = ""
$strsecgroupexclusion = ""
$strsecgroupsg=""
#create the xml document and its specifications.
[xml]$newXML = New-Object system.Xml.XmlDocument
$dec =$newXML.CreateXmlDeclaration("1.0","utf-8",$null)
$newXML.AppendChild($dec)
#create the root node
$root = $newxml.Createnode('element',"SecurityGroups",$null) # arguments are type node meaning has to be element, then the name of it.
#get the list of security groups and a list of the VM's in vcenter
$nsxsecgroups = Get-NsxSecurityGroup
$vm = get-vm
foreach($g in $nsxsecgroups){
#create the xml node
$secgrpnode = $newxml.CreateNode('element',"secgroup",$null)
$secgrpnode.SetAttribute("name",$g.name)
#check for static and dynamic security groups.
if($g.member){
#loop through each member object in the security group
foreach($m in $g.member){
#clear the variables.
$member = $null
$memberid = $null
$membermac = $null
$mac ="" # empty the mac address variable to ensure no false info is taken accross.
$mac = ($vm | Where-Object{$_.id -eq "VirtualMachine-"+$g.member.objectid} | Get-NetworkAdapter).MacAddress #if there are interfaces this variable is an array
$member = $newxml.CreateNode('element',"member",$null) #create the member element
$member.SetAttribute("name",$m.name)
$memberid = $newxml.CreateNode('element',"member-id",$null) #create the member-id element and add it to the member element
$memberid.innertext = $m.objectid
$member.AppendChild($memberid)
if($m.objectid -like "vm*"){#only vm elements have mac addresses
$membermac = $newxml.CreateNode('element',"mac-address",$null)
$membermac.innertext = $mac
$member.AppendChild($membermac)
}
$secgrpnode.AppendChild($member) # add the member element to the security group.
}
#verify if there is no exclusions.
if($g.excludeMember)
{
foreach($m in $g.excludemember){
$member = $null
$memberid = $null
$membermac = $null
$member = $newxml.CreateNode('element',"member",$null)
$member.SetAttribute("name",$m.name)
$memberid = $newxml.CreateNode('element',"member-id",$null)
$memberid.innertext = $m.objectid
$member.AppendChild($memberid)
if($m.objectid -like "vm*"){#only vm elements have mac addresse
$membermac = $newxml.CreateNode('element',"mac-address",$null)
$membermac.innertext = $mac
$member.AppendChild($membermac)
}
$secgrpnode.AppendChild($member)
}
}
}
$root.AppendChild($secgrpnode)
}
$newXML.AppendChild($root)
$newXML.Save($savefile)
}
function get-NewIDfromMAC($macaddr){
<#
.SYNOPSIS
function to retrieve the VM ID based on the MAC address. It returns the VM ID
.DESCRIPTION
The Get-NewIDfromMAC retrieves the VM id from vCenter based on the MAC address provided.
.PARAMETER macaddr
the MAC address from the VM where the ID needs to be retrieved from.
#>
$vmid = $vm| Get-NetworkAdapter | Where-Object{$_.MacAddress -eq $macaddr} ## we assume there is only 1 vm with the mac address
if($vmid.Count){
$tmpstr = $vmid[0].Parentid
}else{
$tmpstr = $vmid.Parentid
}
return $tmpstr
}
function restore-securitygroupmember($backupfile){
<#
.SYNOPSIS
function to restore the VM member object into the security group
.DESCRIPTION
This function reads the backup file and compares the current security groups with its members against the security groups and their members of the backup file. If the member is missing it will perform a search based upon the mac address and adds it to the security group.
.PARAMETER
backupfile. the path and the location of the backup file in the xml format.
#>
#read the xml file
[xml]$xmlbackup = Get-Content $backupfile
#populate array with current security groups and memberships
$nowSecGroup = Get-NsxSecurityGroup
#create an all VM array to use to add the newly created vm's into the security group.
$vm = Get-VM
#declare the variables.
$missingmembers = @()
$existbool = $false
foreach($grp in $xmlbackup.SecurityGroups.secgroup)
{
#check if there are any VM's associations in the members.
if($grp.member.'member-id' -like "vm-*"){
#go through the current security groups to cross check with the xml file security groups
foreach($nsg in $nowSecGroup){
#clear the temporary name variable as a string
$tmpname = ""
#ensure its the matching security group.
if($grp.name -eq $nsg.name){
#verify if the member is an array, if it is only one than compare directly.
#check if the member count is equal ifso move on.
if($grp.member.'member-id'.count -ne $nsg.member.objectid.count){
#find what members are missing
foreach($m in $grp.member){
$existbool = $false
#verify if there are any members in the current security group
if($nsg.member.objectid.count -eq 0){
#add the member to the security group.
$id = get-NewIDfromMAC($m.'mac-address')
$newmember = $vm | Where-Object{$_.id -eq $id}
Add-NsxSecurityGroupMember -SecurityGroup $nsg -Member $newmember
}else{##when the security is missing a member but already has other members to it.
foreach($n in $nsg.member){##check if the member exist.
if($m.InnerText -eq $n.name){
#value already exists
$existbool = $true
}
}
if( -not $existbool){#if the member does not exist then add it to the security group.
$id = get-NewIDfromMAC($m.'mac-address')
$newmember = $vm | Where-Object{$_.id -eq $id}
Add-NsxSecurityGroupMember -SecurityGroup $nsg -Member $newmember
}
}
}
}
}
}
}
}
}
#create connection to nsx and vcloud
$nsxServer = "enter the nsx server address"
$usernamensx = "enter the username to access the nsx server"
$passwordnsx = "enter the password to access the nsx server" #if your password contains special characters use the single quotes
$usernamevc = "enter the username to access the vcenter server"
$passwordvc = "enter the password to access the vcenter server"
Connect-NsxServer -Server $nsxServer -Username $usernamensx -Password $passwordnsx -VIUserName $usernamevc -VIPassword $passwordvc
backup-nsxsecuritygroupconfig("c:\temp\backupnsxsgconfig.xml")
restore-securitygroupmember -backupfile "C:\temp\backupnsxsgconfig.xml"
Comments
Post a Comment