Brian's Brain

This blog post references my feature article 'Solid-State Drives Challenge Hard Disks' in EDN's November 13, 2008 edition. It's one of a series of web addendums to the print writeup.

Sometimes when I do a hands-on benchmarking project, I'm wracked with doubt because (being more of a technology generalist, versus a single-technology-focused content expert) I fear that I've overlooked a key characteristic (or few) in my testing. This time I'm again wracked with doubt, but for a different reason...given the technology 'deep dive' afforded by my employment background, I know that pragmatic resource and time constraints have precluded me from exploring additional interesting evaluation avenues.

Such is life, I guess. In lieu of testing ad infinitum myself, I'll instead attempt in the paragraphs that follow to document both the capabilities and limitations of this project, so that you can expand the scope of my efforts with your own work (and if you do so, please consider posting your results for my and others' benefit!). First off, below please find relevant excerpts from SiSoftware's Sandra documentation for the tests I ran (typos in the source documentation remain uncorrected by me).

File Systems:

Benchmark mounted file systems (i.e. volumes). Shows how your file systems connected to storage adapters and storage hosts compare to other devices in a typical computer.

This is not the raw disk performance that other benchmarks test - but the speed of the volume itself that depends on many more factors like file system, operating system cache, position on disk, etc. Thus this is the performance you get at the file system level.

• Read Test: Buffered, Random, Sequential
• Write Test: Buffered, Random, Sequential
• Seek Test

Drive Index: is a composite figure representing an overall performance rating based on the average of the read, write, and seek tests, and file and cache size. The Drive Index is intended to represent drive performance under typical use in a PC. A larger number means better performance. The weighting of the results is not equal it represents the distribution of different files sizes as used on these devices (obtained through field research).

Physical Disks:

Benchmark hard disks (i.e. the disk itself, not the file system). Shows how your physical disks connected to the storage adapters or hosts compare to other disks in a typical computer.

As the test measures raw performance it is independent on the file system the disk uses and any volumes mounted off the disk.

• Read Test: Sequential across disk
• Write Test: Sequential across disk
• Seek Test: random, full stroke 

Drive Index: is a composite figure representing an overall performance rating based on the highest read or write speed across the whole disk. Thus the higher the better.

Access Time: is the average time to read a random sector on the disk, analogous to latency response time. Thus the lower the better.

Removable Storage:

Benchmark removable and flash devices. Shows how your removable storage and flash memory devices compare to other typical devices.

Such devices are typically small, portable media devices. They are connected into a computer typically via USB (however there are other interface alternatives). These devices are increasing in popularity as users are searching for small (key ring or pen sized devices), which they can use for transferring and backing up the users templates, pictures, documents and more. When Sandra refers to such devices it is referring to the two integrated parts the controller and the memory typically flash internal to the device.

As with many tests there are other external factors that may effect the results and may cause a repeat test to produce a slight difference these include but are not limited in motherboard chipset / architecture, USB hub, interface or controller.

Explanations:

The typical usage model for these devices is file operations, such as writing a file to the device, reading a file from it, and deleting a file, this benchmark exercises the devices in terms of these operations.

To measure the "raw" cluster level performance of the device, it is recommended to also test it by means of the File Systems module.

The following characteristics are measured for each of the four representative files sizes of 512 Bytes (representing a minimal single data cluster file), 32kB, 256kB, 2MB and 64MB. The weighting of the results is not equal it represents the distribution of different files sizes as used on these devices (obtained through field research):

Read Performance: expressed both in terms of Operations per Minute (i.e. the number of files read per minute) and the corresponding net transfer rate in kB/second.

Write Performance: similarly expressed both in terms of Operations per Minute (i.e. the number of files written in a minute) and the corresponding net transfer rate in Kbytes/second.

Delete Performance: measuring the number of files deleted per minute.

For each of the four file sizes a Combined Index is then calculated, stating the combined performance in terms of Combined Operations Per Second, with respect to a mix of write, read and delete operations.

Combined Device Index: is a composite figure representing an overall performance rating based on the average of the Combined Index figures over the four file sizes. (Higher is better, i.e. better performance)

The in the graphical window the Combined Index (y axis) is plotted as a function of the file size (x axis). The y axis can be toggled between linear and logarithmic scale. Please note that the x axis scale is non-linear and somewhat distorted for ease of comprehension. The chart is intended for easy to use “one-glance” comparison, and it should NOT to be used for data readout.

Endurance Factor: is a figure representing the Wear and Life Expectancy of flash devices; this is obtained by dividing the average performance (normal condition, i.e. sequential write) to the lowest performance (high-stress condition, i.e. same block re-write).

It measures the relative improvement of endurance caused by the wear levelling or flash management algorithm; the absolute endurance of a device (i.e. its expected life-time) is directly dependent, in addition to this Endurance Factor, on the nominal manufacturer rating of maximum erase/reprogram cycles, which is typically 100,000+ for SLC and 10,000+ for MLC devices. (Higher is better, i.e. longer life-time for the device)

As the factor is a ratio, it is useful only in comparing devices rather than having a direct, physical interpretation associated to its numerical value.

Continue reading with Part 2 of this post, 'Solid-State Drives Challenge Hard Disks: SiSoftware Speaks, And Testing Qualifiers'...