I talked earlier this week about the recent persistent memory summit (see my post Persistent Memory). If DRAM gets faster or higher capacity, then there isn't anything that software engineers need to change, whether they're writing the operating system, middleware, or applications. If a system has persistent memory then there are a number of things that might be different. If the persistence is provided by NVDIMMs, then if you choose to ignore it, then you won't see any change how your software behaves. If it is implemented using a technology lke 3DXpoint (Optane is Intel's name) then you might find the performance degraded since it is slower (500ns access versus 100ns for DRAM). On the other hand, when 3DXpoint is used as the underlying memory technology for SDDs, then you might see the performance improve because it is faster than NAND flash. However, if you want to take advantage of the persistence, then you will need to make changes to how you program, it doesn't come for free.

Paul Grun

The challenge is that persistent memory (despite the name) isn't really memory, but it isn't really storage either, as Paul Grun of Cray pointed out in his talk. In fact, it's not even monolithic, with the green stratum in the middle of the triangle really splitting into one level which is local (and very memory-like) and one level which is remote (and very storage-like).

Paul pointed out some areas where it seems using persistent memory will be very attractive:

• Database Applications; e.g., a modifiable, an in-memory database that survives power cycles
• Data Analytics; e.g., create a persistent database once, run new queries repeatedly
• Graph Analytics; e.g., operate on larger graphs than can fit in local memory
• Commercial Applications; e.g., enable collaboration on large scale projects
• HPC Applications; e.g., scalability, parallel applications

One thing Paul emphasized is that not all applications value persistence, and for those apps, the convergence of memory and storage is not interesting. Sometimes the extra amount of memory per dollar, or the increased speed versus flash, are interesting, and no changes are necessary to take advantage of those. On the other hand, as Paul said at the end of his talk, this will enable new use cases, some of which haven't been thought of yet.

Another issue Paul talked about was redundancy. As he put it:

If data is not in two places it is nowhere.

So if persistence is really required, then data written to persistent memory also needs to find its way to a second storage location (such as persistent memory on a second server).

Andy Rudoff

Andy Rudoff, who works for Intel in his day job, is the head of the SNIA Non-Volatile Memory Programming Technical Working Group.

He started by reminding everyone of some things that everyone knows about persistent memory, and some things they forget. First, the things everyone knows. Persistent memory is:

• Read and written just like normal memory (DRAM) with load and store instructions.
• It is persistent, and retains its value like storage.
• It is exposed to applications via the SNIA NVM TWG model.

And the things people often forget. Persistent memory is not:

• Something only accessed in blocks (like a disk).
• Too slow to be accessed by load/store. In fact, the working definition is that it is fast enough that it is reasonable to stall the CPU when waiting for an access to complete.
• The programming model includes the storage APIs. If you are using a persistent memory SSD then you just access it through the normal APIs.

That means that the easiest way to use persistent memory is either to access a PM SSD through the normal API or to map part of the persistent memory into the application address space just like another file (DAX, or direct access). Or it can just be treated as very large non-persistent memory. None of these approaches require the code to be modified.

However, many applications can take advantage of persistent memory by being aware of it and treating it differently. There is nothing to stop you just using load/store and rolling your own, but generally, it is better to make use of libraries that hide a lot of the low-level details. There are several libraries to facilitate this:

• libpmem: low-level support for local persistent memory
• librpmem: low-level support for remote access to persistent memory
• libpmemlog: interface to create a persistent memory resident logfile (for journaling database updates, for example)
• lipbpmemobj: Interface for persistent memory allocation, transactions, and general facilities
• libpmemblk: Interface to create arrays of persistent memory resident blocks, of the same size, atomically updated

These libraries can be used to create attractive functionality that takes advantage of the persistent memory but hides it from the user. For example, there is a libpmemkv that implements a persistent key-value pair storage facility. The application needs nothing extra, and after a restart, all the key-value pairs will still be in the store.

Here is a full stack example. Working up from the bottom of the diagram we have:

• The persistent memory, that just looks like part of the processor address space.
• A file system that exposes the persistent memory in the form of memory-mapped files.
• libpmem that abstracts away all the hardware details so that everything looks the same on all systems.
• libpmemobj that provides atomic transactions and a persistent memory allocator, and hides the details of what it takes to give the atomicity guarantees such as serialization.
• LLPL that provides Java transactions and allocators.
• Cassandra, that uses Java containers to create persistent memory aware versions. The caller of Cassandra sees the same API as they do if a completely different implementation (without persistent memory, say) is in use.
• Application: unchanged.

Andy gave the lessons so far:

• There are lots of ways to use persistent memory without app modifications.
• Try first to use existing APIs (for example, an app can be configured for SSD tier.
• Try next to use highest abstraction possible (e.g. key-value store, simple log interface).
• Try next to use a transaction library (in libpmemobj).
• Finally, if you must, program to raw mapped access.

Sign up for Sunday Brunch, the weekly Breakfast Bytes email.