Friday 22 July 2016

SAP HANA TDI on Cisco UCS and VMware vSphere - Part 2

ESXi Host

UCS Service Profile

The ESXi host provides the platform where virtual machines are running on. The service profile contains the configuration of the hardware in a Cisco UCS environment. Service Profile Templates or vSphere Auto Deploy can be used to ease the ESXi deployment process. In this example, a standalone service profile creation is shown.

For each vSwitch, it is recommended to configure two uplink interfaces with MTU 9000 as trunk. The VLAN assignment takes place in the port group configuration of the vSwitch.

SAP HANA TDI on Cisco UCS and VMware vSphere - Part 2

In order to get the best performance for virtualization, certain BIOS features should be enabled. The c-states can be controlled by the hypervisor and do not necessarily have to be disabled. It depends on the performance needs vs. power saving aspects how balanced this configuration should be.

SAP HANA TDI on Cisco UCS and VMware vSphere - Part 2

SAP HANA TDI on Cisco UCS and VMware vSphere - Part 2

SAP HANA TDI on Cisco UCS and VMware vSphere - Part 2
VMware vSphere screenshot
Although the use of VM-FEX is optional, it is recommended to enable all Intel Direct IO features.

SAP HANA TDI on Cisco UCS and VMware vSphere - Part 2

NUMA Locality

It is beneficial for HANA when a VM with 2 vSockets get scheduled on the same physical socket rather than being spread out to remote sockets. While distributing workload of the same VM on multiple sockets to get the full computing power of the core might be beneficial for other workloads, HANA benefits largely from L3 caching effects  and therefore the VM should stick to its sockets as much as possible. Using sched.vcpu.affinity per VM is a quite harsh way to configure NUMA locality and also limits VM operational tasks, so let's look at another more gentle approach with quite the same benefits.

VMware KB 2003582 describes the two parameters numa.vcpu.preferHT and numa.preferHT, one for a single VM, the other for the entire ESXi host. Using one of these parameters solves the problem of vCPUs getting scheduled to distant NUMA nodes and therefore leverages L3 caching effects for the VM, too.


SAP HANA has different types of network communication channels to support the different SAP HANA scenarios and setups.

SAP HANA TDI on Cisco UCS and VMware vSphere - Part 2
Source: SAP SE
On the basis of the listed network requirements, every server must be equipped with two 1 or 10 Gigabit Ethernet (10 Gigabit Ethernet is recommended) interfaces for scale-up systems to establish communication with the application servers and users (client zone). If the storage for SAP HANA is external and accessed through the network, two additional 10 Gigabit Ethernet or 8-Gbps Fibre Channel interfaces are required (storage zone). Scale-out systems need a 10 Gigabit Ethernet link for internode communication (internal zone). When using multiple ESXi hosts in a vSphere Cluster with enabled DRS, at least one additional 10 Gigabit Ethernet link is required for vMotion traffic.

For the internal zone, storage zone and the vMotion network, it is recommended to configure jumbo frames end-to-end.

The network traffic can be consoliated by using the same vSwitch with several load-balanced uplinks.

SAP HANA TDI on Cisco UCS and VMware vSphere - Part 2

Regarding the network adapter placement, the devices might show up in an unexpected order. In order to align vNIC and vmnic enumeration, apply vNIC Placement Policy according to the UCS Manager Server Management Guide and follow VMware KB 2091560 and KB 2019871.


The storage system must be certified as part of an appliance, independent of the appliance vendor, or must be certified as SAP HANA TDI Storage in the Certified SAP HANA Hardware Directory.

It is recommended to physically separate the the origin (VMFS LUN or NFS export) of the datastores providing data and log for performance reasons. Additional performance classes can be distinguished:
CategoryRead PerformanceWrite Performance
OS boot disk medium low
/hana/shared medium low
/hana/data very high high
/hana/log high very high
backup low medium



  1. Good job! Fruitful article. I like this very much. It is very useful for my research. It shows your interest in this topic very well. I hope you will post some more information about the software. Please keep sharing!!
    AWS Training in Chennai
    AWS Course in Chennai
    German Classes in Chennai
    Python Training in Chennai
    AWS Training in OMR
    AWS Training in Velachery
    AWS Training in Adyar