In late 2018, when I first read this whitepaper on SAP HANA Adoption of Non-Volatile Memory it kept me hooked on Persistent Memory technology and its usage with HANA. Ever since then, I regularly followed the announcement of this persistent memory technology and on 2nd April, 2019 Intel announced the general availability of Intel Optane DC Persistent Memory. Later after that, IBM introduced Virtual Persistent Memory (vPMEM) in the virtualization platform of IBM Power Systems. So now we have two technology with which we can adopt HANA to enable the use of Persistent Memory.
Recently, I have worked on configuring IBM’s Virtual Persistent Memory (vPMEM) on POWER9 System for SAP HANA. So I thought of writing this blog to provide more insight on both – Intel and IBM’s persistent memory technology and pen down the steps on how to configure persistent memory on IBM POWER9 Server.
This blog is focused on providing an overview on each persistent memory technology and how HANA as a database enables the use of Persistent Memory.
Intel’s new Persistent Memory called Intel Optane DC Persistent Memory, is a new type of computer storage, combining speeds approaching those of dynamic RAM (DRAM) along with RAM’s byte-by-byte address-ability, plus the permanence of solid-state disks (SSDs). It resides on the memory bus which allows Persistent Memory (PMEM) to have DRAM-like access to data, which means you can achieve almost the same speed with slightly higher latency of DRAM alongside non-volatility of SSDs.
Like conventional RAM, persistent memory is installed directly into motherboard memory slots. As such, it is supplied in the same physical form factor as RAM – as DIMMs. These are known as NVDIMM: non-volatile dual inline memory module or DCPMM: DC Persistent Memory Module. So, your system will have a mixture of both conventional RAM and DCPMM, where operating system and applications will execute in conventional RAM, with the DCPMM providing very fast supplementary storage.
When Intel introduced beta program in 2018 of Intel Optane DC Persistent Memory, it has disclosed its two different operating modes. These modes determine which capabilities of the Intel memory are active and available to software.
Memory Mode
When memory mode is used for DCPMM, the application and operating system perceive a pool of volatile memory, no differently than it does today on DRAM-only systems. In this mode, no specific persistent memory programming is required in the applications, and the data will not be saved in the event of a power loss. So existing application will continue to work the way it works today with Optane memory mode like virtualization deployment.
In Memory mode, the DRAM acts as a cache for the most frequently-accessed data, while the Intel Optane DC Persistent Memory provide large memory capacity. Cache management operations are handled by the Intel Xeon Scalable processor’s memory controller. When data is requested from memory, the memory controller first checks the DRAM cache, and if the data is present, the response latency is identical to DRAM. If the data is not in the DRAM cache, it is read from the Intel Optane DC Persistent Memory with slightly longer latency. Applications with consistent data retrieval patterns the memory controller can predict will have a higher cache hit-rate, and should see performance close to all-DRAM configurations, while workloads with highly-random data access over a wide address range may see some performance difference versus DRAM alone. Also, data is volatile in Memory Mode; it will not be saved in the event of power loss.
Persistence is enabled in the second mode, called App Direct which is also a suitable mode for HANA In-Memory database.
So, the question is why use Optane with memory mode, if it’s not persistent?
If you consider the largest DRAM DIMM that is currently available in market is 128 GB, whereas the smallest DCPMM available is 128 GB and goes up to 512 GB and it is also cheaper than the traditional DRAM. Since Optane with memory mode acts just like a traditional DRAM, legacy applications can work seamlessly like virtualized database deployment and big data analytics. This will suffice for the use cases where existing application needs more memory with low TCO.
App Direct Mode
With App Direct Mode, applications and operating system are explicitly aware that there are two types of direct load/store memory in the platform and can direct which type of data read or write is suitable for DRAM or Intel Optane DC Persistent Memory. So, in order to use mode, your application like Operating System, Virtualization, HANA In-Memory Database should be aware on the usage of DRAM or Intel Optane DC Persistent Memory.
Operations that require the lowest latency and don’t need permanent data storage can be executed on DRAM and the data that needs to be made persistent or structures that are very large can be routed to the Intel Optane DC Persistent Memory
So, in order to use App Direct mode of Intel Optane DC Persistent Memory, application should be enabled with persistent memory-aware file system. Below are some of the applications and virtualization that support App Direct mode
SLES 12 SP 04 / SLES 15
RHEL 7.6, 7.7, and 8.0 for SAP Solutions
Windows Server 2019
VMware ESX v6.7
HANA In-Memory (HDB 2.0 Revision 35)
Above are the applications, that supports app-direct persistent memory approach. Reference of the application is not in context with HANA.
Irrespective of which persistent memory technology you use – Intel’s DCPMM or IBM’s vPMEM, HANA logic of using persistent managed file system remain same. So, in this section I will try to highlight how HANA enable the use of Persistent Memory.
NOTE: As servers are getting more and more powerful with each new release, above comparison might be different when done in future.
As per SAP Note 2786237 – Sizing SAP HANA with Persistent Memory, SAP Support server configuration with persistent memory which follow certain ratio between DCPMMs to DIMMs for optimal performance.
SAP HANA 2.0 SPS 03/04 currently supports various capacity ratios between Intel Optane DCPMMs and DIMMs. Ratio examples include the following:
◉ 1:1 ratio: A single 128-GB Intel Optane DCPMM is matched with a single 128-GB DDR4 DIMM, or a 256-GB Intel Optane DCPMM is matched with a single 256-GB DRAM DIMM.
◉ 2:1 ratio: A 256-GB Intel Optane DCPMM is matched with a 128-GB DRAM DIMM, or a 128-GB Intel Optane DCPMM is matched with a 64-GB DDR4 DIMM.
◉ 4:1 ratio: A 512-GB Intel Optane DCPMM is matched with a 128-GB DDR4 DIMM, or a 256-GB Intel Optane DCPMM is matched with a 64-GB DRAM DIMM.
Different-sized Intel Optane DCPMMs and DIMMs can be used together as long as supported ratios are maintained as below
Recently, I have worked on configuring IBM’s Virtual Persistent Memory (vPMEM) on POWER9 System for SAP HANA. So I thought of writing this blog to provide more insight on both – Intel and IBM’s persistent memory technology and pen down the steps on how to configure persistent memory on IBM POWER9 Server.
This blog is focused on providing an overview on each persistent memory technology and how HANA as a database enables the use of Persistent Memory.
Intel’s Persistent Memory Approach – Overview
Intel’s new Persistent Memory called Intel Optane DC Persistent Memory, is a new type of computer storage, combining speeds approaching those of dynamic RAM (DRAM) along with RAM’s byte-by-byte address-ability, plus the permanence of solid-state disks (SSDs). It resides on the memory bus which allows Persistent Memory (PMEM) to have DRAM-like access to data, which means you can achieve almost the same speed with slightly higher latency of DRAM alongside non-volatility of SSDs.
Like conventional RAM, persistent memory is installed directly into motherboard memory slots. As such, it is supplied in the same physical form factor as RAM – as DIMMs. These are known as NVDIMM: non-volatile dual inline memory module or DCPMM: DC Persistent Memory Module. So, your system will have a mixture of both conventional RAM and DCPMM, where operating system and applications will execute in conventional RAM, with the DCPMM providing very fast supplementary storage.
When Intel introduced beta program in 2018 of Intel Optane DC Persistent Memory, it has disclosed its two different operating modes. These modes determine which capabilities of the Intel memory are active and available to software.
Memory Mode
When memory mode is used for DCPMM, the application and operating system perceive a pool of volatile memory, no differently than it does today on DRAM-only systems. In this mode, no specific persistent memory programming is required in the applications, and the data will not be saved in the event of a power loss. So existing application will continue to work the way it works today with Optane memory mode like virtualization deployment.
In Memory mode, the DRAM acts as a cache for the most frequently-accessed data, while the Intel Optane DC Persistent Memory provide large memory capacity. Cache management operations are handled by the Intel Xeon Scalable processor’s memory controller. When data is requested from memory, the memory controller first checks the DRAM cache, and if the data is present, the response latency is identical to DRAM. If the data is not in the DRAM cache, it is read from the Intel Optane DC Persistent Memory with slightly longer latency. Applications with consistent data retrieval patterns the memory controller can predict will have a higher cache hit-rate, and should see performance close to all-DRAM configurations, while workloads with highly-random data access over a wide address range may see some performance difference versus DRAM alone. Also, data is volatile in Memory Mode; it will not be saved in the event of power loss.
Persistence is enabled in the second mode, called App Direct which is also a suitable mode for HANA In-Memory database.
So, the question is why use Optane with memory mode, if it’s not persistent?
If you consider the largest DRAM DIMM that is currently available in market is 128 GB, whereas the smallest DCPMM available is 128 GB and goes up to 512 GB and it is also cheaper than the traditional DRAM. Since Optane with memory mode acts just like a traditional DRAM, legacy applications can work seamlessly like virtualized database deployment and big data analytics. This will suffice for the use cases where existing application needs more memory with low TCO.
App Direct Mode
With App Direct Mode, applications and operating system are explicitly aware that there are two types of direct load/store memory in the platform and can direct which type of data read or write is suitable for DRAM or Intel Optane DC Persistent Memory. So, in order to use mode, your application like Operating System, Virtualization, HANA In-Memory Database should be aware on the usage of DRAM or Intel Optane DC Persistent Memory.
Operations that require the lowest latency and don’t need permanent data storage can be executed on DRAM and the data that needs to be made persistent or structures that are very large can be routed to the Intel Optane DC Persistent Memory
So, in order to use App Direct mode of Intel Optane DC Persistent Memory, application should be enabled with persistent memory-aware file system. Below are some of the applications and virtualization that support App Direct mode
SLES 12 SP 04 / SLES 15
RHEL 7.6, 7.7, and 8.0 for SAP Solutions
Windows Server 2019
VMware ESX v6.7
HANA In-Memory (HDB 2.0 Revision 35)
Above are the applications, that supports app-direct persistent memory approach. Reference of the application is not in context with HANA.
IBM’s Persistent Memory Approach – Overview
After the release of Intel’s Optane DC Persistent Memory, last year IBM introduced a new capability called Virtual Persistent Memory (vPMEM) in the Virtualization Platform (IBM PowerVM) of IBM Power Systems.
IBM approach for persistent memory is an enhancement in their virtualization platform of Power Systems. So basically, if you have IBM POWER9 System, you can enable the use of Persistent Memory for HANA database.
IBM’s Virtual Persistent Memory is an enhancement in PowerVM that introduces the ability to configure persistent volumes using the existing DRAM technology. Virtual Persistent Memory isn’t a new type of physical memory but is a way to use the PowerVM hypervisor to create persistent memory volumes out of the DRAM that is already installed on the system. This is being made available on existing POWER9 processor-based systems with just a firmware upgrade and that too at no additional costs.
Intel’s DCPMM vs IBM’s vPMEM – Quick Checks
| Intel Optane DC Persistent Memory (DCPMM) | IBM Virtual Persistent Memory (vPMEM) |
| Separate DCPMM module need to be purchased in order to configure Persistent Memory on Intel based Servers | Additional hardware is not required in order to configure Persistent Memory on IBM POWER9 System. |
| Persistent Memory can be configured only on Intel’s Cascade lake architecture which was introduced last year. So old architecture like Haswell, Skylake will not be enable for Persistent Memory | All existing POWER9 Server can be configured to use Persistent Memory, just by performing firmware upgrade. So, you need not to replace your entire hardware |
| Intel’s Persistent based solution is built on separate DIMM module which as slightly higher latency as compared to DRAM. | IBM’s Persistent based solution is built on DRAM, so it preserves the same run-time performance |
| For On-Premise customer, VMware is preferred choice for virtualization on HANA. Currently SAP HANA is not yet supported by SAP on vSphere on persistent memory | The solution is based on virtualization i.e. it is configured at LPAR level. So, customer can run multiple VMs on a Single System. |
| Data is preserved even when power is off. | vPMEM does not preserve memory across server power down. But it will preserve the data if you reboot your HANA, Linux OS (i.e. LPAR) |
| Ratio needs to be followed between DRAM and PMEM when configuring HANA database. (SAP Note 2786237) | As solution is based on DRAM, there are no ratio restriction applies here. Everything resides on DRAM and sizing is performed just like SAP’s HANA Sizing guidelines. |
HANA Adoption of Persistent Memory
Irrespective of which persistent memory technology you use – Intel’s DCPMM or IBM’s vPMEM, HANA logic of using persistent managed file system remain same. So, in this section I will try to highlight how HANA enable the use of Persistent Memory.
Background: HANA In Memory database keeps all its data in main memory or DRAM. Disk storage is still required to allow restarts in case of power failure and for permanent persistency. The idea behind HANA was to integrate transactional and analytics workload within the same database management system. This is been achieved with columnar engine which exploits modern hardware, compression of database content and maximum parallelization. In the transactional workload of the SAP Business Applications, more than 80% of all statements are read access. The remaining data write operations consists mainly of inserts, a few updates, and very rare deletes.
HANA’s relational in-memory columnar store is tuned for such enterprise workloads. The trade-off favors the large majority of read accesses through the choice of a columnar database. The performance impact on the row-oriented smaller transactional workload is acceptable due to the fast in-memory access.
HANA compress its columnar store table with extensive compression technique in order to optimize the utilization of available memory. But compression pays off only when the data does not change often (computational effort spent to compress data is leveraged over a longer duration). So, in order to overcome this costly overhead of compression every time data gets changes, HANA columnar tables are divided into two table fragments: Main and Delta. Main Fragments are reader friendly, it contains most of the data and it changes rarely (it uses sorted dictionaries, N-bit and other compression techniques). On the other hand, delta fragments are writer friendly, it contains remaining smaller part of data.
Currently both Main and Delta table fragments are stored in main memory or DRAM. But now with Intel’s DCPMM and IBM’s vPMEM in place, HANA database has been intelligently designed where it identifies the presence of Persistent Memory and stores its Main table fragment of columnar store in DCPMM/vPMEM and Delta fragment, Row Store in DRAM. Now with this persistent memory storage leads to significantly less time loading data from traditional persistent storage into memory. So now HANA doesn’t have to wait for data to be loaded into memory as most of the data is already in the memory. This enables rapid restart and recovery times immediately with full performance benefits of SAP HANA.
Intel (x86) vs IBM (ppc64le) – Server Comparison
As being from Service Industry, I often been asked about the hardware comparison between the two processor – x86 vs ppc64le. So, I usually prepare a table which helps people to understand the technical aspect of the server in terms of no. of CPUs, maximum memory, support of persistent memory, virtualization support etc.
NOTE: Comparison is entirely based on the technical aspect.
For pcc64le Processor, I have considered E950 POWER9 Server and for x864 Processor, 2nd Generation Intel Xeon Scalable Processors (Cascade Lake) 8280M which is released on Q2’19 and support DCPMM.
| pcc64le Processor | x86 Processor | Remarks | |
| CPUs | |||
| # of Cores / Processor | 12 | 28 | |
| # of Parallel Thread / Core | 8 | 2 | |
| Total # of Parallel Thread / Processor | 8 x 12 = 96 | 2 x 28 = 56 | |
| Memory | |||
| Max. # of Memory Channel / Processor | 8 | 6 | |
| # of DIMM Module / Channel | 4 | 2 | |
| Total # of DIMM Module / Processor | 4 x 8 = 32 | 2 x 6 = 12 | |
| Total # of DIMM Module / Server | 32 x 2 = 64 | 12 x 2 = 24 | 2 Socket Server is considered |
| Available DDR4 (RAM) Size | 128 GB | 128 GB | 256 GB is still not available in market |
| Total Memory (RAM) / Server (2 Socket) | 128 GB x 64 = 8 TB | 128 GB x 24 = 3 TB | |
| Persistent Memory | |||
| Persistent Memory Support | Yes (vPMEM) | Yes (Optane DC) | |
| Persistent Memory Mode* | Volume Mounted on DRAM | App Direct Mode | |
| DRAM:PMEM Ratio Restriction | No Restriction | SAP Note 2786237 | |
| Server Supported for Persistent Memory | All POWER9 | Cascade Lake Servers | |
| Available PMEM DIMMS | —- | 128 / 256 / 512 GB | |
| Total Memory / Server (with PMEM) | 128 GB x 64 DIMMS = 8 TB | 128 GB x 12 DIMMs = 1.5 TB 512 GB x 12 DCPMMs = 6 TB Total = 7.5 TB |
IBM: Persistence based solution is based on DRAM. Intel’s Server sizing calculation is done based on 4:1 ratio As 256 GB DRAM is still not available in market, current hardware can extend upto 7.5 TB on 2 Socket Cascade lake processor. |
| Miscellaneous | |||
| Hypervisor | PowerVM | 3rd Party Vendor (like VMware) | Each hyperscaler use their own hypervisor |
| Support of PMEM with Virtualization | HANA is supported for PowerVM on POWER9 Servers | Persistent Memory is not certified by SAP to be used with VMWare Virtualization** | Support of hyperscaler hypervisor needs to be checked with cloud vendor. |
| Availability | Persistent Memory is only available as TDI*** | Persistent Memory is only available as TDI*** | |
NOTE: As servers are getting more and more powerful with each new release, above comparison might be different when done in future.
Intel’s Supported Ratios of DCPMMs to DIMMs
As per SAP Note 2786237 – Sizing SAP HANA with Persistent Memory, SAP Support server configuration with persistent memory which follow certain ratio between DCPMMs to DIMMs for optimal performance.
SAP HANA 2.0 SPS 03/04 currently supports various capacity ratios between Intel Optane DCPMMs and DIMMs. Ratio examples include the following:
◉ 1:1 ratio: A single 128-GB Intel Optane DCPMM is matched with a single 128-GB DDR4 DIMM, or a 256-GB Intel Optane DCPMM is matched with a single 256-GB DRAM DIMM.
◉ 2:1 ratio: A 256-GB Intel Optane DCPMM is matched with a 128-GB DRAM DIMM, or a 128-GB Intel Optane DCPMM is matched with a 64-GB DDR4 DIMM.
◉ 4:1 ratio: A 512-GB Intel Optane DCPMM is matched with a 128-GB DDR4 DIMM, or a 256-GB Intel Optane DCPMM is matched with a 64-GB DRAM DIMM.
Different-sized Intel Optane DCPMMs and DIMMs can be used together as long as supported ratios are maintained as below
Persistent Storage Sizing – Data, Log and Shared
Intel Servers
The storage size for the file system is based on the amount of memory (DRAM + Intel Optane DCPMM) on the SAP HANA host. For a single-node system with 9 TB of memory (3-TB DRAM + 6-TB Intel Optane DCPMM), the recommended file system sizes are as follows:
/hana/data = 1.2 x memory (DRAM + Intel Optane DCPMM) = 1.2 x 9 TB = 10.8 TB
/hana/log = 512 GB
/hana/shared = 1 TB
IBM Servers
As vPMEM volume is mounted on top of the persistent storage requirement will be followed in the same manner as that of RAM, so the recommended file system size for 8 TB (2 Socket Server) are as follows:
/hana/data = 1.2 x DRAM = 1.2 x 8 TB = 9.6 TB
/hana/log = 512 GB
/hana/shared = 1 TB
The size of volume which needs to be mounted on RAM for vPMEM on POWER9 Server will be discussed in next blog, which will focused on how to configure vPMEM on POWER9 Server.
No comments:
Post a Comment