Few days ago I got a awesome piece of hardware – Fusion-IO Duo 640GB Multi Level Cell – MLC, in fact it’s single PCI express card with 2 Fusion-IO modules, 320GB each). Within following week I will be doing performance testing and compare results with data provided by vendor.
FusionIO Duo MLC 640GB
Technical details:
| ioDrive Duo Capacity | 320GB | 640GB | 640GB | 1.28TB |
|---|---|---|---|---|
| Nand Type | Single Level Cell (SLC) | Multi Level Cell (MLC) | ||
| Read Bandwidth (64KB) | 1.5 GB/s | 1.5 GB/s | 1.5 GB/s | 1.5 GB/s |
| Write Bandwidth (64KB) | 1.5 GB/s | 1.5 GB/s | 1.0 GB/s | 1.1 GB/s |
| Sequential Read IOPS (512 Byte) | 261,000 | 273,000 | 196,000 | 185,000 |
| Sequential Write IOPS (512 Byte) | 262,000 | 252,000 | 285,000 | 278,000 |
| Mixed IOPS (75/25 r/w) | 238,000 | 236,000 | 138,000 | 150,000 |
| Access Latency (512 Byte) | 26 μs | 26 μs | 29 μs | 30 μs |
| Bus Interface | PCI-Express x4/x8 or PCI Express 2.0 x4 | |||
| Warranty | 5 years or maximum endurance used | |||
| Supported Operating Systems | |
|---|---|
| Microsoft Windows | 64-Bit Microsoft XP/Vista/Win7/Server 2003/2008/2008 R2 |
| Linux | RHEL 5/6; SLES 10/11; OEL 5/6; CentOS 5/6; Debian Squeeze; Fedora 15/16;openSUSE 12; Ubuntu 10/11 |
| UNIX | Solaris 10 U8/U9/U10 x64; OpenSolaris 2009.06 x64; OSX 10.6/10.7, HP-UX* 11i |
| Hypervisors | VMware ESX 4.0/4.1/ESXi 4.1/5.0,Windows 2008 R2 with Hyper-V, Hyper-V Server 2008 R2 |
As you can see vSphere5.1 ESXi is not on the list but it is supported. Download a ESXi5.1 drivers from FusionIO support page and install on hypervisor.
Installation procedures
- Insert Fusion-IO card into proper slot on motherboard – refer to motherboard vendor documentation
- Verify is the FusionIO card was detected by ESXi5.1 on PCI bus
- Log in to ESXi5 via SSH or use DCUI
- from a command line type lspci |more
~ # lspci 00:00:00.0 Bridge: Intel Corporation Xeon E3-1200 Processor Family DRAM Controller 00:00:01.0 Bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port [PCIe RP[00:00:01.0]] 00:00:02.0 Display controller: Intel Corporation Xeon E3-1200 Processor Family Integrated Graphics Controller 00:00:16.0 Communication controller: Intel Corporation P8P67 Deluxe Motherboard 00:00:1a.0 Serial bus controller: Intel Corporation P8P67 Deluxe Motherboard 00:00:1b.0 Multimedia controller: Intel Corporation 6 Series/C200 Series Chipset Family High Definition Audio Controller 00:00:1c.0 Bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 [PCIe RP[00:00:1c.0]] 00:00:1c.2 Bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 3 [PCIe RP[00:00:1c.2]] 00:00:1c.3 Bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 4 [PCIe RP[00:00:1c.3]] 00:00:1c.4 Bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 [PCIe RP[00:00:1c.4]] 00:00:1c.5 Bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 6 [PCIe RP[00:00:1c.5]] 00:00:1c.6 Bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 [PCIe RP[00:00:1c.6]] 00:00:1c.7 Bridge: Intel Corporation 82801 PCI Bridge [PCIe RP[00:00:1c.7]] 00:00:1d.0 Serial bus controller: Intel Corporation P8P67 Deluxe Motherboard 00:00:1f.0 Bridge: Intel Corporation Z68 Express Chipset Family LPC Controller 00:00:1f.2 Mass storage controller: Intel Corporation Cougar Point 6 port SATA AHCI Controller [vmhba0] 00:00:1f.3 Serial bus controller: Intel Corporation P8P67 Deluxe Motherboard 00:02:00.0 Bridge: PLX Technology, Inc. PEX 8616 16-lane, 4-Port PCI Express Gen 2 (5.0 GT/s) Switch 00:03:04.0 Bridge: PLX Technology, Inc. PEX 8616 16-lane, 4-Port PCI Express Gen 2 (5.0 GT/s) Switch 00:03:05.0 Bridge: PLX Technology, Inc. PEX 8616 16-lane, 4-Port PCI Express Gen 2 (5.0 GT/s) Switch 00:03:06.0 Bridge: PLX Technology, Inc. PEX 8616 16-lane, 4-Port PCI Express Gen 2 (5.0 GT/s) Switch 00:05:00.0 Mass storage controller: Fusion-io ioDimm3 00:06:00.0 Mass storage controller: Fusion-io ioDimm3 00:0a:00.0 Serial bus controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller 00:0b:00.0 Network controller: Realtek Realtek 8168 Gigabit Ethernet [vmnic0] 00:0c:00.0 Bridge: ASMedia Technology Inc.
Card has been detected – 00:05:00.0 Mass storage controller: Fusion-io ioDimm3
Drivers installation
- Download a ESXi5 drivers offline package iomemory-vsl-2.3.11.183-offline_bundle-909505.zip from FusionIO support page – Drivers – FusionIO account is needed.
- Upload drives on ESXi host – you can use vSphere client or WinSCP
- Install drives on ESXi
- remotely using vCLI, esxcli –server <servername> software vib install -d <path to the offline bundle>
- from console – esxcli software vib install -d <offline-bundle>
~ # esxcli software vib install -d /vmfs/volumes/SATA01/install/iomemory-vsl-2.3.11.183-offline_bundle-909505.zip Installation Result Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective. Reboot Required: true VIBs Installed: Fusion-io_bootbank_block-iomemory-vsl_2.3.11.183-1OEM.500.0.0.472560 VIBs Removed: VIBs Skipped: ~ #
- reboot ESXi host
Check Fiusion-IO card status:
~ # fio-status Unable to get product information for /dev/fct0:0. Unable to get format information for /dev/fct0. Unable to get data volume information for /dev/fct0. Unable to get system monitor information for /dev/fct0. Unable to get product information for /dev/fct1:0. Unable to get format information for /dev/fct1. Unable to get data volume information for /dev/fct1. Unable to get system monitor information for /dev/fct1. Found 2 ioDrives in this system with 1 ioDrive Duo Fusion-io driver version: 2.3.11 build 183 Adapter: ioDrive Duo ioDrive Duo HL SN:59649 External Power: NOT connected Connected ioDimm modules: fct0: HP ioDIMM 320GB SN:73388 fct1: HP ioDIMM 320GB SN:51321 fct0 Status unknown: Driver is in MINIMAL MODE: Firmware is out of date. Update firmware. HP ioDIMM 320GB SN:73362 Located in slot 0 Upper of ioDrive Duo SN:73388 PCI:05:00.0 Firmware v5.0.6, rev 101583 Geometry and capacity information not available. Sufficient power available: Unknown Internal temperature: 57.1 degC, max 57.1 degC fct1 Status unknown: Driver is in MINIMAL MODE: Firmware is out of date. Update firmware. HP ioDIMM 320GB SN:51301 Located in slot 1 Lower of ioDrive Duo SN:51321 PCI:06:00.0 Firmware v5.0.6, rev 101583 Geometry and capacity information not available. Sufficient power available: Unknown Internal temperature: 55.6 degC, max 55.6 degC ~ #
As you can see card is running in minimal mode and firmware on card is outdated, which means Firmware upgrade is necessary
Upgrade FusionIO card firmware on ESXi 5.1
- Download firmware package iodrive_107053.fff from FusionIO support page
[box type=”warning”] Make sure that driver version and firmware version are matches[/box]
- Upload firmware package on ESXi5.1 host
- Execute fio-update-iodrive -d /dev/fctX where X is a device number, in my case I have to upgrade firmware on both cards /dev/fct0 and /dev/fct1
[box type=”warning”] Firmware upgrade takes around 15 minutes !! DO NOT STOP UPGRADE UTILITY OR POWER SERVER OFF[/box]
~ # fio-update-iodrive -d /dev/fct1 /vmfs/volumes/SATA01/install/iodrive_107053.fff Warning: ioDrive at '/dev/fct1' has incomplete internal identification information. Even though the ioDrive will continue to run, some utilities and SDK functions may have problems identifying and enumerating this ioDrive. Device ID 1 (/dev/fct1) Updating device firmware from 5.0.6.101583 to 5.0.7.107053 WARNING: DO NOT TURN OFF POWER OR RUN ANY IODRIVE UTILITIES WHILE THE FIRMWARE UPDATE IS IN PROGRESS Please wait...this could take a while Progress ------------------------- \ 0: 100% Results ------------------------- 0: Firmware updated successfully You MUST now reboot this machine before the new firmware will be activated!
- updated second device and reboot ESXi host
Verify if drives appear in ESXi host inventory
- Start vSphere client go to Configuration –> Storage adapters on a right hand side windows in storage adapters section two new storage adapters for iomemory-vsl
FusionIO card in ESXi inventory
Now, just create VMFS5 filesystem and DONE Fusion-IO DUO installed on vSphere 5.1 ESXi
Great and detailed very good article and very informative. Thank you very much for this
Did you format the FIO? What block size? Did you know you can overprovision for maximum performance? You can go as high as 30some percent to achieve twice the throughput I believe. As soon as my UPS gets here with my darn servers, I’ll be installing my FIO cards to join the big leagues.. Thanks