User ManualTable of ContentsTable of Contents31 Introduction9Who Should Read this Guide9How this Guide is Organized9Overview10Interoperability11Conventions Used in this Guide11Table 1-1. Typographical Conventions (Continued)11Documentation12Contact Information132 Feature Overview15What’s New in this Release15Table 2-1. QLogic Adapter Model Numbers (Continued)15New Features16Supported Distributions and Kernels17Table 2-2. InfiniPath/OpenFabrics Supported Distributions and Kernels (Continued)17Compiler Support18Software Components183 Step-by-Step Installation Checklist21Hardware Installation21Software Installation224 Hardware Installation25Hardware Installation Requirements25Hardware25Table 4-1. Adapter Models and Related Platforms (Continued)25Form Factors26Cabling and Switches27Table 4-2. QLogic InfiniBand Cables27Optical Fibre Option28Configuring the BIOS28Safety with Electricity29Unpacking Information29Verify the Package Contents29List of the Package Contents29Figure 4-1. QLogic QLE7280 with IBA7220 ASIC31Figure 4-2. QLogic QLE7140 Card with Riser, Top View31Figure 4-3. QLogic QHT7040/QHT7140 Full and Low Profile Cards with Riser, Top View32Unpacking the QLogic Adapter32Hardware Installation33Hardware Installation for QLE7240, QLE7280, or QLE7140 with PCI Express Riser33Dual Adapter Installation33Installation Steps33Figure 4-4. PCIe Slot in a Typical Motherboard34Figure 4-5. QLogic PCIe HCA Assembly with Riser Card35Figure 4-6. Assembled PCIe HCA with Riser36Hardware Installation for QHT7140 with HTX Riser36Figure 4-7. HTX Slot37Figure 4-8. QLogic QHT7140 Adapter with Riser Card38Figure 4-9. Assembled QHT7140 with Riser39Hardware Installation for QLE7240, QLE7280, and QLE7140 Without a PCI Express Riser39Hardware Installation for the QHT7140 Without an HTX Riser40Figure 4-10. QHT7140 Without Riser Installed in a 3U Chassis41Switch Configuration and Monitoring41Cabling the Adapter to the InfiniBand Switch41Completing the Installation425 Software Installation43Cluster Setup43Types of Nodes in a Cluster Environment43Supported Linux Distributions44Table 5-1. InfiniPath/OpenFabrics Supported Distributions and Kernels44Setting Up Your Environment44Downloading and Unpacking the InfiniPath and OpenFabrics Software45Choosing the Appropriate Download Files45Table 5-2. Available Packages (Continued)45Unpacking the InfiniPath tar File46Check for Missing Files or RPMs47Installing the InfiniPath and OpenFabrics RPMs48Table 5-3. InfiniPath and OpenFabrics RPMs to Use for Each Node in a Cluster48Choosing the RPMs to Install49Table 5-4. RPMS to Install49Using rpm to Install InfiniPath and OpenFabrics50Installed Layout51Starting the InfiniPath Service52InfiniPath and OpenFabrics Driver Overview53Figure 5-1. Relationship Between InfiniPath and OpenFabrics Software53Configuring the InfiniPath Drivers53Configuring the ib_ipath Driver54Configuring the ipath_ether Network Interface54ipath_ether Configuration on Red Hat54ipath_ether Configuration on SLES56OpenFabrics Drivers and Services Configuration and Startup58Configuring the IPoIB Network Interface59Configuring and Administering the VNIC Interface60OpenSM65SRP66Using OFED SRP66MPI over uDAPL68Other Configuration: Changing the MTU Size68Starting and Stopping the InfiniPath Software68Rebuilding or Reinstalling Drivers After a Kernel Upgrade70Rebuilding or Reinstalling Drivers if a Different Kernel is Installed71Further Information on Configuring and Loading Drivers71LED Link and Data Indicators71Table 5-5. LED Link and Data Indicators (Continued)71Adapter Settings72Customer Acceptance Utility73Table 5-6. ipath_checkout Options74Removing Software Packages75Uninstalling InfiniPath and OpenFabrics RPMs75Uninstalling OFED 1.3 Software75Downgrading RPMs75Additional Installation Instructions75Installing Lustre76Installing QLogic MPI in an Alternate Location76Installing on an Unsupported Distribution77Managing and Installing Software Using Rocks78Installing Rocks and InfiniPath RPMs79Further Information on Rocks and kickstart83A Installation Troubleshooting85Hardware Issues85Node Spontaneously Reboots85Some HTX Motherboards May Need Two or More CPUs in Use85BIOS Settings85MTRR Mapping and Write Combining86Edit BIOS Settings to Fix MTRR Issues86Use the ipath_mtrr Script to Fix MTRR Issues87Issue with Supermicro® H8DCE-HTe and QHT704087Software Installation Issues87Version Number Conflict with opensm-* on RHEL5 Systems88OpenFabrics Dependencies88OpenFabrics Library Dependencies88Missing Kernel RPM Errors89Resolving Conflicts90mpirun Installation Requires 32-bit Support90ifup on ipath_ether on SLES 10 Reports "unknown device"91Lockable Memory Error on Initial Installation of InfiniPath91B Configuration Files93Table B-1. Configuration Files (Continued)93C RPM Descriptions95InfiniPath and OpenFabrics RPMs95Different Nodes May Use Different RPMs95RPM Version Numbers and Identifiers95RPM Organization96Documentation and InfiniPath RPMs96Table C-1. InfiniPath Documentation/RPMs96Table C-2. InfiniPath/RPMs97Table C-3. InfiniPath-Devel/RPMs97Table C-4. InfiniPath-MPI/RPMs98OpenFabrics RPMs98Table C-5. OpenFabrics Documentation/RPMs98Table C-6. OpenFabrics/RPMs (Continued)98Table C-7. OpenFabrics-Devel/RPMs (Continued)101Table C-8. OpenSM/RPMs102Table C-9. Other HCAs/RPMs (Continued)102Table C-10. Other HCAs-Devel/RPMs103Table C-11. OtherMPIs/RPMs (Continued)103Index107Size: 1.7 MBPages: 114Language: EnglishOpen manual
