vSphere 6 : vSphere Client is ALIVE!!

UPDATE: I have updated the below to fix the mis-information

There has been a long standing rumor that VMware is killing off the VIClient.  When the vSphere Beta orginally came out, i was bummed to see it missing.  It had been replaced with a very clunky and slow client, that looked a lot like the full Web Client.  Honestly, it was terrible and i was very disappointed.

I was very pleasantly surprised when i fired up the latest build of vSphere and clicked on the “Download vSphere Client” and the normal looking vSphere Client installed.  I was a bit excited, the icons looked the same as the old client i liked so much.  Sure enough, it looks the same and works the same as the old 5.5 client. That also means that in order to use new features you will need to use the Web Client.

vMotion between AMD & Intel

UPDATE: It seems Virtual Hardware 10 VMs will NOT vMotion even with the below done, might want to hang on to v8 for now if this is something you want to do.

Its always been said that you can’t vMotion VMs between very different processor types.  In fact w/o enabling EVC you can’t vmotion between even different generations of processors, let alone between AMD and Intel processors.  This isn’t really true, well mostly anyway.  There are certain parameters you can set in vCenter and sometimes the VMs that will allow you to actually do this vMotion.

I must state that this is in NO way supported by VMware at all, in fact if you ask their support teams to help you do this they will tell you its impossible since the CPUs aren’t really virtualized and are passed through to the VMs.  Please do not do this to your production systems or at least to anything really important, there are weird bugs.

In my home basement lab i have an AMD Opteron 2350 server and my newly aquired Intel L5520 based boxes.  These are running vSphere 5.5 U2.  I wanted to use the AMD box since it still has a decent amount of RAM and cores, enough for some management servers anyway.  I wanted to be able to vMotion VMs w/o powering them off to move them, that’s just annoying.

In order to allow me to do this i enabled two advanced options in vCenter.  The first was config.migrate.test.CpuCompatibleError  this was set to false.  In addition i set config.migrate.test.CpuCompatibleMonitorSupport also set to false.

Now this allowed me to actually vMotion most of my VMs with nothing more then a warning in the “Migrate” wizard dialog.  There was not a bit of hiccup on these VMs, sweet!!  However, not everything was roses, some of my VMs required hiding the NX bit or even doing some custom CPU masking.  Here is the VMware link to do the custom masking. VMWare CPU Masking

One thing i did notice is that some machines actually played nicer when being moved from Intel to AMD.  The VM would use various AMD bits but not the Intel.  This was the most frustrating on my Horizon View Connection Server.  Now, I could go and mask each feature on these VMs, however i just made it easy and disabled DRS on that VM, and used the Intel as the primary host and it would only move to the AMD if there was an HA event or i needed to do maintenance.

VMware is betting big on EVO

If you didn’t watch the VMworld Keynote today, or read anything on twitter, or were just too busy, it is Monday, then you missed VMware’s big announcement.  They are moving into what they call the “hyper-converged” infrastructure.  They are helping develop pre-built compute nodes with vSphere already bundled in.  All of this is a single SKU as well. They are calling this new product line EVO.  There are two products currently announced in the EVO line.  The first actually exists already and that is EVO:RAIL.

EVO:RAIL is designed for your small to medium shops, that are looking to run up to 400 server VMs or 1,000 VDI desktops in this hyper-converged platform.  This is not a product that we will see running your Fortune 100’s datacenters.  The base compute nodes are NOT built by VMware, as some rumored!!  These appliances are built by various EVO partners, Dell, Fujitsu, Inspur, Net One Systems Co. and Supermicro.  These partners will all GA their appliances in the 2nd half of this year….so shortly.  EMC is also involved but will not be shipping until 2015.   These appliances are a 2U box with 4 separate compute nodes within the 2U footprint.  Each appliance is designed to run 100 server VMs or 250 VDI VMs. Right now you can scale up to 4 appliances per cluster

What makes these appliances pretty neat is that VMware vSphere will be automagicly setup on these nodes.  The entire RAIL product line and experience  is designed for the non-vcp crowd, the very green admin who knows very little about VMware administration.  Everything is done through a simplified GUI that walks you through everything.

When an appliance is first racked/stacked and powered on, an actual Webserver VM is powered up as well.  This web server, running Java FYI, will then be connected to in order to setup the environment.  The setup wizard asks very simplified information, hostname prefix for all the nodes, desired passwords, and basic networking information such as IP and VLAN numbers.  Now most of these fields are filled in automatically with a default set of information.  All the values can be changed, and the wizard is smart enough to look for critical errors, such as overlapping IP ranges for example.  The wizard will then run against all the seen nodes and begin to setup vCenter, HA, DRS, & VSAN for storage.  If there is an error the wizard will show the error and prompt for information in order to fix it.  It will not just die and leave you in an unfinished and unknown state. (I’m looking at you EMC, as i’ve had terrible luck with their wizards doing just that)

Once the environment is up, the same webpage you used to build the environment is used to manage it.  It presents a very simple dashboard that gives you lots of information about the health and status of the EVO:RAIL cluster.  There is a section that allows you to create new VMS, including uploading ISOs, specifying the VM “size” (right now its a pretty static, small, medium & large) as well as the “Security Options”.  This can be none, basic (what is created today), a secure setting and a total locked down option.  (My believe is that there must be some NSX bits involved, however i was told no… not sure i believe that)  The VM can then be administered from another part of this GUI.   You can power it on, view the console, and make minor configuration changes.  This is making VMware be able to be used by the user that has next to no knowledge about VMware.  I typically use the analogy of the janitor being able to do it.

The EVO:RAIL system will be updated and managed by a separate set of firmware and patches from the general repos and VUM.  I’ll repeat you do NOT use VUM to update these appliances.  This is excellent news since VUM and i are not friends.  These appliances will be treated as just that a single block, not a separate compute node, storage equipment and vsphere running on top.  This makes is stupid simple to work with.

Here is the other thing i really like.  You are not forced to use this simplified GUI.  Lets say i am a VMware admin, and i have a normal environment with blades, running vCAC, vCOPS and doing some really cool things.  My company decides EVO:RAIL is a great fit for a remote office that needs servers onsite, but doesn’t want a full blown setup.  I can still use the same tools i use today, vSphere Web Client, vCAC, vCOPS, etc to manage, deploy and monitor my new EVO:RAIL clusters.

VMware did also make mention of EVO:RACK.  It is still in tech preview, so think alpha stage.  This will be the fully blown out, mega datacenter version of EVO.  This will involve top of rack switches, full storage platforms, Rack-Mount servers, more vSphere components such as NSX, vCloud, etc.  This could be very very interesting and i’d love to see more about it, however they aren’t talking much about it in any sessions or over at the booths.

I will keep some of my opinions on the product line until i can get to touch it and play with it more.  Right now it seems like something i wouldn’t get much exposure to, as World Wide tends to go after the larger companies, who typically want or need to do a more traditional “build your own” solution.  It is still an interesting leap, and something that could become very powerful, especially once Rack is available.

Micro-Segmentation on NSX!!

I recently attended a really cool presentation by Scott Lowe about the ability to do Micro-segmentation with NSX.  This is, in my opinion, the biggest use case for NSX and something that impresses me a lot.

Before we deep dive on what is NSX & Micro-Segmentation, first where do most of us stand today with our networks in regards to security.  Most environments use a Perimeter type security model.  Unfortunately, it tends to not be very resilient and can have a lot of issues.  First off if your able to breach this “shell” and get access to the interior servers you typically have a pretty open environment.  There is typically little security protecting East/West traffic between servers, coupled with the ever increasing network traffic between these servers its fairly easy to hide any rouge traffic while attacking other servers.  Once you have access and control you can then launch your attack when its best.  A perfect example of this is the recent big box stores, Target for example.  The attackers got in and waited some time before actually stealing data.

How can we best combat this today? The best option is to utilize a Least Privilege or 0-Trust security model.  This puts firewalls between everything, all servers both North/South and East/West.  I have actually seen a customer do this, not only did they spend a TON of money on both physical and virtual firewalls but it was an administrative nightmare.  They had a lot of touch points and one of the biggest problems was actually identifying rules and what they did, and why they were there.  There were rules still in place for servers and applications that had been decommissioned earlier.  It also was a nightmare to try and open up ports between applications and equipment.  They had to touch multiple firewalls all while trying to monitor the traffic to catch those ports that vendors don’t always list in their docs about “needed firewall ports”.

NSX and its Micro-Segmentation is an awesome answer to this problem!  NSX at its heart is Network Virtualization. NSX allows us to decouple the Network from the hardware and allow centralized management of this decoupled network.  With NSX we separate out the various network “planes”.  First, there is the Management plane, this is typically vCenter with the Network & Security Plugin installed.  This is where we define all the various rules and policies.  Next, the Control Plane. This is NSX Manager & NSX Controllers. This is the decision maker and controls the rules states as well as their definitions and keeps track of everything.  Finally we have the Distributed Data planes, these are all the modules loaded into all of the ESXi hypervisiors, and enables the Distributed Routers, Distributed Firewalls and switches.  This is where all the actual packet switching happens.  All of the configurations can be manually done or can use various REST APIs to do some automation, typically with vCAC.

Ok, now that we know what the basic NSX is, why is micro-segmentation so awesome?? NSXs Micro-segmentation allows us to implement a true 0-Trust Model security policy, easily and without the complication and huge costs and administration overhead that it typically takes.  We can now put an intelligent firewall and routing between every single VM!!  The part that is really cool is that we are doing this at the hypervisor level.  This allows the traffic to not have to “hairpin” to go through a firewall or router.  The traffic is typically analyzed at the originating vnic level.  (It can be done on the receive side, but isn’t recommended)  The rules are also dynamic and follow the VM wherever it goes in the environment.  The rules are removed if the VMs are off or deleted.  They are re-created when the VM is powered on, and CAN be automatically created when a VM is provisioned.

NSX Micro-Segmentation allows us to have the datacenter security sit in a “sweet spot”.  It is close enough to the VMs and workloads that it can provide intelligence of workloads and granular control.  However, unlike many agent based tools that can provide this level of intelligence and granular control, the security features are in the hypervisor and not in the VM.  This allows a compromised VM to still have all its security features intact.  In fact as we’ll see later on, NSX can potentially recognize the compromise and perform specific actions against the VM.

NSX allows for a very flexible network design.  At a very high level it is comprised of three main types of “tenants” or types of networks. We can Isolate networks from each other.   We can also Segment networks, which means we can allow certain types of traffic between various points.  Finally NSX allows us to bring in more “Advanced Services” via 3rd Party applications that plugin to NSX.  Lets dive into each more.

Isolation networks allows us to create just that, completely isolated networks.  These networks have absolutely no knowledge of each other.  They also have no knowledge of the underlying physical connections.  They could both be running over a big flat network and still not see each other.  They can have the exact same IP space as well.  This is excellent for keeping Dev, Test and Production networks separate, yet all running within the same vSphere Environment!! This is also used for multi-tenant environments, where the workloads shouldn’t ever see each other.

Segmented networks are most common types of networks, these are the traditional 3-teir App scenario. Without NSX, typically you’d have a Perimeter firewall, a DMZ then the inside firewall or firewalls that lead to various networks.  The rules are always in place on the firewalls no matter what the workload states are.  The rules also have to be manually inputted into each firewall, and moved in some cases if workloads move, depending on network design. With NSX we have some flexibility and choices.  We can create logical networks per group, app, BU, etc and then apply the rules to each VM.  We can also be a bit silly and just dump everything onto one giant logical switch and apply the rules to the VMs that way.  Either way is the same result, the rules are applied at the VMs, the logical networks can be designed however makes the most sense for the environment.

The “Advanced Services” functions of NSX brings a lot of really neat functionality and intelligence to traffic flows that weren’t really possible before at the scale and easy of administration that NSX brings. With NSX can setup the firewalls to do intelligent and dynamic routing.  We can send traffic through a Malware scanner and then based on the output of the scans, we can do different things with the traffic.  You can also send traffic to a deep packet scanner.  You can also just pass the traffic straight to its destination.  With these advanced services we can build If/Then type rules for the traffic.  For example, if traffic is sent to TrendMicro and a virus is found, NSX can quarantine it and not allow the VM to pass any traffic until its resolved.  Also, if during a scan, a big vulnerability due to old software is found, NSX can then monitor all that VMs traffic via IPS until fixed.  If a different scan is run and it finds sensitive data, we can encrypt the traffic and restrict it while its investigated.  I think this is a very very neat feature, and something that should make both network and security guys very happy! There are a bunch of vendors that are providing various plugins for this, such as, Rapid 7, McAfee, PaloAlto, Symantec & TrendMicro.

One thing that Scott mentioned that makes a ton of sense and is really cool is around Security Groups.  Security Groups allow the administrator to group VMs together logically using a variety of static and dynamic variables, such as Datacenter, Virtual Machine, vNIC, OS Type, User ID, Security Tag (applied by IPS, Malware scanner), etc.  Then various policies you create, Firewall rules, send to IPS, etc, are then applied to given security groups and tags within the group.  In reality this is the only way to really do NSX at any real scale.  It would be very time consuming to create rules for each and every VM.  By using the security groups VMs can become part of the groups and get certain policies applied to it automatically.

NSX and the Micro-Segmentation feature are all very simply managed within the vSphere Web Client. If your still using the old .NET client, you need to stop or you’ll be very very sad in the near future.  These are all under the Network & Security plugin.  This is where you can create your Security Groups along with their elements or attributes.  You also create all your various policies here.  You can also view all the events, and logs here.  There is also an area where you can View all of the traffic flows from VM to VM, and even create rules against the seen flows.  It does make it a bit easier to create the rulesets from real flows rather then trying to set them up ahead of time, if you’d like.

One thing that Scott also mentioned were a few use cases for this micro segmentation.  There was one use case that i actually will be exploring more in depth in the near future because it interested me a lot.  He mentioned that an admin can use NSX and Micro-Segmentation with VMware View.  In a very typical VMware View design there tends to be various pools for different user types, such as internal workers and external or offshore users who shouldn’t be able to access certain things, or setup of different pools for dev teams who only should be accessing their development machines.  This usually means different VLANs, with firewall and routing rules to accomplish this segmenting.  However, NSX can apply firewall & routing rules based upon a logged in User ID.  This means you could actually have a single pool for all our users on a single flat network and because NSX can apply the locked down firewall and routing rules to the VM when a “restricted user” logs in, we accomplish the same goal as the more complicated setup.  Now thats awesome!

Now Scott had a lot of really interesting slides and visualizations in his presentation that unfortunately i can’t use.  I would look for more information at VMworld.  In addition there is a good White-Paper here.



Looking forward to VMworld 2014??

So i’m pretty stoked about being able to go to VMworld 2014 in San Francisco this year. In my opinion not only should you be going, but you should be excited as well. I don’t care if your a PS Delivery geek like me, a Sales guy, or an Admin who deals with the day to day fires, you should be going and excited because there is something for everybody there.

I look at it this way, there are three main reasons one should go this conference. Education, Socialization and Exposure.

The biggest and most important is Education. Without breaking any NDAs and getting myself into trouble, i will say there will be some pretty cool stuff announced and show this year from VMware. I’m quite excited to hear what they have to say about it and what they feel the future of some of it is. That being said there are also some awesome breakout sessions that i have been to in the past at previous VMworld conferences. Some of my favorites are the really technical deep dives where you can actually see peoples eyes glaze over, but learning whats really going on under the covers is the best, or at least i think so. Lets see, the “lessons learned” ones are usually pretty informative as well, esp if its in an area that you can see yourself or your company moving towards, like VDI or IaaS.

More in terms of education is the opportunities to really ask experts the tough questions, and i’m not talking about just playing “stump the chump”, but real “how does this work” or “why can’t i do this?” type questions. I have seen and heard some amazing things from these almost random conversations that occur. In fact some of the things that i’ve wow’d customers with have come from those exact sessions.

Also i can’t stress this enough, take advantage of the discounted testing for certifications. I know it would be a lot better if they gave away one free one instead HINT HINT to VMware, but it makes it a really easy sell to your management to take a shot at a certification. In fact last VMworld i decided since it was cheaper why not just take my VCAP-DCA, i hadn’t studied, but as it turns out i was able to pass it. I was happy because i got a new cert, my management was happy because it was 1/2 off.

Onto Socialization! This has honestly become my favorite part about going to conferences, any of them actually. I’m not talking about the crazy parties where you can’t remember how you got back to your hotel, that never happens anyway… I’m referring to just meeting up and hanging out with, not only some of your co-workers, but people you don’t get to see until these type of events. If there is one thing any seasoned IT guy knows is this is a people industry, it’s not what you know, but who you know. You run into the same people over and over at different jobs. I have had a blast hanging at the events like the Hall Crawls, or the big party on Wednesday night. I also like to just meetup at a local eatery and talk and hang out. Now typically us being nerds does mean the conversations usually end up technical, but the best part is they end up in lots of different disciplines all over the place. You learn things that you had NO idea about. I remember hanging out with a good buddy of mine who taught me about OpenDaylight and what it can do, we just happened to meetup at a local place and chatted for hours about all kinds of cool technical stuff and where we saw the market going.

Finally a cool thing is going to the Solutions exchange and seeing what cool technology is out there. Yes it can be a bit of a pain and hectic with 1000’s of people running around and everybody trying to get their badge scanned and what not. But i have learned about some cool tech in previous years and have met contacts that allowed me to be able to get some equipment into our lab to do some testing and to try some of this stuff out. It also doesn’t hurt that sometimes there is some pretty cool swag out there.

Anyway i highly encourage you guys to go to the conference. If you’d like to register, you may do so here. If you’d like to see some more official information about it before you make your decision, here is the link.

Virtual SSDs in VMWare products

A colleague of mine asked me for help with setting up VSAN and asked how to setup VMDKs as SSDs.

While i at first gave them the older SATP information, i found an awesome page at Virtually Ghetto and had to send it along.


Instead of me rehashing it all out, i’d rather send you to Mr. Lam’s excellent page on it.




5.5 U1b now out – Fixes NFS and Heartbleed issues

So for those who didn’t know, ESXi 5.5 u1 had a pretty seveare issue relating to NFS.

So occasionally any connections to NFS storage would end up in an All Paths Down (APD) condition. This is obviously pretty poor as things tend to break when the storage is ripped out from underneath the VMs running on the hosts.

This has been a known bug by VMware. This issue had absolutely nothing to do with Network or Storage hardware, however NetApp had come out with a patch that would help prevent the issue.

In addition ESXi 5.5 was vulnerable to the Heartbleed issues. If you read that and are confused, well you must have lived under a rock.

VMware has released 5.5 U1b that has the patches baked in. If you don’t want to do a full update, the patch is here, http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2077361

A collegue of mine has create a script that you can run on your ESXi hosts to update the server if you don’t have VUM installed. You must enable SSH to the host.

# open firewall for outgoing http requests:
esxcli network firewall ruleset set -e true -r httpClient
# Install the ESXi 5.5 pre-U1 Heartbleed Fix Image Profile from the VMware Online depot
esxcli software profile update -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-5.5.0-20140401020s-standard --allow-downgrades
# Reboot your host

Custom UCS/FlexPod Build Script

 UPDATE: Working with some of our internal guys, its come to my attention that some of the script has broken with the newer UCSM versions.  I will be updating this to be more “adaptable”, however use the script for ideas and feel free to kang any code from it for now.


So i started working on developing a Powershell script that will grab variables from an Excel sheet and create a UCS Build off of that.

I am at a point that the build actually works quite well now. I’m pretty proud of myself since i’m NOT a deep Powershell guy. This came about from looking at other UCS Powershell scripts and a lot of tweaking and testing.

Anyway this script will continue to grow and its functionality expand. My end goal is to be able to do a base FlexPod build by scripting, including UCS, Nexus Switches, Netapp and VMware.

It will take a lot of time, and i may never really use the script but its more of a pet project to not only see if i can do it, but also grow my Powershell skillset.

Here is the github if you’d like to follow/assist or download and play with it a bit.


Creating Custom ESXi Images Video

This is a video i did to show how to create a custom ESXi image iso and zip, incorporating specific VIBs.

In this case it was to change out the Cisco net-enic & scsi-fnic drivers to match the version that is on the HCL for the UCS version we were running.  The Cisco created ESXi version didn’t even have the right version.

I do know that there are a 100 videos out there for this, but i had created it, so why not share it 🙂

So i have this embedded, but since we’re looking at text, i really do recommend going straight to youtube and watch it, since the screen will be larger.