Technology for Everyday Needs

AMD today announced general availability of the next generation 45nm AMD Opteron Quad core processors. The official press release is available here.

Major improvements in the new Opteron architecture (code named Shanghai) in this release are as follows:

• 45nm Manufacturing Process
• Larger Cache
• Support for DDR2 800 MHz
• Upcoming enhancements for HyperTransport 3.0 (HT3)
• Other micro-architecture enhancements like
• AMD SmartFetch
• AMD CoolCore
• Maintains platform compatibility leading to better return on investment.

What benefits can you expect from the new processor?

Energy Efficiency without sacrificing Performance: The new generation of AMD Opteron processors utilize the latest AMD 45nm manufacturing process. This process allows greater clock speeds leading to higher core frequencies without dissipating too much heat. As per AMD’s announcement, new generation processors deliver 35% more performance while drawing up to 35% less power. The new manufacturing process also allows much higher clock speeds than the previous generation quad core processors. Over all, AMD Opteron processors combined with support for DDR2 memory offers platform level energy efficiency and 100% x86 compatibility.

Improved Application Performance: The latest generation processors feature two major enhancements affecting application performance: DDR2 800 MHz support and larger cache. The latest AMD Opteron (Shanghai) improves on the previous generation of AMD Opteron processors (Barcelona) with the support of 800 MHz DDR2 memory. This memory technology offers improved memory bandwidth over the previous generation of processors and offer much better energy efficiency than Fully Buffered DIMM (FB-DIMM) technology. A 200% increase in L3 cache to 6 MB benefits a number of applications across verticals, like databases, virtualiztion, JAVA applications, scientific applications, media applications and more. A faster memory bus combined with a larger cache with out complicated prefetching and snooping algorithms offers overall application efficiency.

HyperTransport 3.0 (HT3) Support: AMD Opteron processors provide unparalleled scalability and aggregate memory bandwidith by employing AMD DirectConnect architecture with HyperTransport. Previous generations offered a 1GHz HyperTransport link among the processors. Next generation enhancements planned for Q2 2009 include support for coherent HyperTransport 3.0 offering up to 17.6 GB/s of bandwidth for inter-processor communication. cHT3 will enahce platform scalability for systems featuring 4 or more AMD Opteron processors and will enahce application performance for DP platforms.

Micro-architecture Enhancements: The next generation 45nm AMD Opteron processor (Shanghai) also features enhancement to numerous other micro-architectural features. Some are listed below:

AMD SmartFetch: This technology allows cores to enter in to a “halt” state when the processor core becomes idle. In a ‘halt” state, the processor does not consume power and enhances power efficiency. This technology does not affect application performance in any way thus offering better power efficiency with no penalties on performance.

AMD CoolCore: This technology allows powering down selected sections of the processors. When a particular section is not being used, that section will be powered down in order to enhance power efficiency.This technology does not affect application performance in any way thus offering better power efficiency with no penalties on performance.

Enhanced Virtualization Performance: The next generation 45nm AMD Opteron processor (code named Shanghai) offers unsurpassed enhancements in virtualization performance. Combined with the architectural enhancements like 45nm manufacturing, larger cache, higher frequencies, higher memory bandwidth, cHT3 support, the new processor delivers faster “world switch” time enhancing virtual machine efficiency. AMD’s innovating AMD-V featuring Rapid Virtualization Indexing reduces overhead associated with software virtualization. L3 cache index disable proivdes improved data integrity as well.

With 45 nm AMD Opteron quad core processor (Shanghai), AMD continues to build on its platform strengths while addressing certain drawbacks in Barcelona processors. AMD Opteron (Shanghai) can be used on all systems, with a BIOS upgrade, supporting the Barcelona processors. Customers can avail themselves of the new processors with a simple in socket upgrade without the associated costs of a total hardware replacement. Application software will instantly experience the performance enhancements that come with 45nm AMD Opteron Shanghai processors.

HPC Systems, Inc. a platinum partner of AMD now supports 45nm AMD Opteron Shanghai processors across the product line. Systems featuring the latest generation of AMD Opteron processors are immediately available.

Read the press release here.

Share This

In RedHat 5.1, the maximum file system size is increased to 16 TB from 8TB. However, getting mkfs to format a volume larger than 2 TB is not straight forward.

We do  ship large volumes to customers regularly. We recommend that customers use XFS for large volumes for performance and size considerations. However, sometimes customers want only ext3 because of the familiarity with the file system.

Before being able to format a volume,  you must be able to create a volume greater than 2 TB. fdisk cannot do this.

You will need to use GNU Parted (parted) to create partitions larger than 2 TB. Details on how to use parted can be found here and here

A simple example of using parted, we assume are working on /dev/sdb of size 10 TB from a RAID controller.

$> parted /dev/sdb

GNU Parted 1.8.9
Using /dev/sdd
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted)

(parted) mkpart primary ext3 10737418240
(parted) print
(parted) quit


A straight forward mkfs command on any volume larger than 2 TB will yield the following error:

mkfs.ext3: Filesystem too large.  No more than 2**31-1 blocks
(8TB using a blocksize of 4k) are currently supported.

A simple workaround is to force mkfs to format the device in spite of the size:

mkfs.ext3 -F -b 4096 /dev/<block device>

mkfs.ext3 -F -b 4096 /dev/<path to logical volume> if you are using LVM

In order to use the above command you need to have e2fsprogs 1.39 or above. The above command also sets block size to 4kb.

You could also use -m0  to set the reserved blocks to zero.

Note that ext3 is not recommended for large volumes. XFS is better suited for that purpose.

Further reading:

RedHat Knowledgebase  Article

 Knowplace

Unixgods

Share This

AMD (NYSE: AMD) today announced it has achieved the top spot on the VMware^® VMmark™ virtualization benchmark for x86 servers with the Quad-Core AMD Opteron™ processor-based HP ProLiant DL585 G5. AMD now holds the top three spots on the 16-core VMmark benchmark. This latest result is further proof that Quad-Core AMD Opteron processors provide a high-performance virtualization solution that allows data center managers to make large-scale virtualization deployments and do so at an attractive price point.

Read more here

Share This

I had some problems compiling BLACS with OpenMPI and GCC on RHEL 5 / CentOS 5. So, here is how I got it to compile and pass the tests successfully:

OpenMPI: 1.2.5

GCC: 4.1.2

F77 = gfortran

F90 = gfortran

CC = gcc

CXX = g++

Bmake file used: BMAKES/Bmake.MPI-LINUX

Changes made to Bmake:

COMMLIB = MPI

#  ————————————-
#  Name and location of the MPI library.
#  ————————————-
MPIdir = /home/test/openmpi-install/
MPILIBdir =
MPIINCdir = $(MPIdir)/include
MPILIB =

SYSINC =

INTFACE = -DAdd_

TRANSCOMM = -DUseMpi2

WHATMPI =

SYSERRORS =

#=============================================================================
#=========================== SECTION 3: COMPILERS ============================
#=============================================================================
#  The following macros specify compilers, linker/loaders, the archiver,
#  and their options.  Some of the fortran files need to be compiled with no
#  optimization.  This is the F77NO_OPTFLAG.  The usage of the remaining
#  macros should be obvious from the names.
#=============================================================================
F77            = $(MPIdir)/bin/mpif77
F77NO_OPTFLAGS =
F77FLAGS       = $(F77NO_OPTFLAGS) -O3 -mtune=amdfam10 -march=amdfam10
F77LOADER      = $(F77)
F77LOADFLAGS   =
CC             = $(MPIdir)/bin/mpicc
CCFLAGS        = -O3 -mtune=amdfam10 -march=amdfam10
CCLOADER       = $(CC)
CCLOADFLAGS    =
Of special importance are the flags:

INTFACE = -DAdd_

TRANSCOMM = -DUseMpi2

If INTFACE is not set correctly, make tester will fail with following messages:

blacstest.o(.text+0x4c): In function `MAIN__':

: undefined reference to `blacs_pinfo_’

blacstest.o(.text+0×6e): In function `MAIN__’:

: undefined reference to `blacs_get_’

blacstest.o(.text+0×8b): In function `MAIN__’:

: undefined reference to `blacs_gridinit_’

blacstest.o(.text+0×94): In function `MAIN__’:

More such errors follow.

If TRANSCOMM is not set correctly, make tester will complete sucecssfully and you will be able to successfully execute C interface tests also. When executing FORTRAN interface tests, the following messages are seen:

BLACS WARNING ‘No need to set message ID range due to MPI communicator.

‘from {-1,-1}, pnum=1, Contxt=-1, on line 18 of file ‘blacs_set_.c’.
BLACS WARNING ‘No need to set message ID range due to MPI communicator.’
from {-1,-1}, pnum=3, Contxt=-1, on line 18 of file ‘blacs_set_.c’.
BLACS WARNING ‘No need to set message ID range due to MPI communicator.’
from {-1,-1}, pnum=0, Contxt=-1, on line 18 of file ‘blacs_set_.c’.
BLACS WARNING ‘No need to set message ID range due to MPI communicator.’
from {-1,-1}, pnum=2, Contxt=-1, on line 18 of file ‘blacs_set_.c’.
[comp-pvfs-0-7.local:30119] *** An error occurred in MPI_Comm_group
[comp-pvfs-0-7.local:30118] *** An error occurred in MPI_Comm_group
[comp-pvfs-0-7.local:30118] *** on communicator MPI_COMM_WORLD
[comp-pvfs-0-7.local:30118] *** MPI_ERR_COMM: invalid communicator
[comp-pvfs-0-7.local:30119] *** on communicator MPI_COMM_WORLD
[comp-pvfs-0-7.local:30119] *** MPI_ERR_COMM: invalid communicator
[comp-pvfs-0-7.local:30119] *** MPI_ERRORS_ARE_FATAL (goodbye) 

Share This

In the previous post, we tried Hyper-V with only 16 cores as per the release notes. Now we addedd another 8 CPUS (16 cores AMD Opteron) to the same system. This was to test the x64 Windows Server 2008 on 32 cores than Hyper-V. We already did this for the x86 version here.

The system did boot up just fine. Here is a screen shot.

With that taken care of, we quickly browsed through the event logs to see if the Hyper-V service / hypervisor failed to start as per the release notes. There was no such message. The only way to test if the hypervisor has started or not is to fire up the Server Manager and try to boot up the virtual machines.

We were pleased to see that the hypervisor indeed started and there was no problem booting up the virtual machines. And here is a screenshot.

This opens up a wide range of usage cases. With appropriate capacity planning, the entire data center for a small company can be replaced with one 5U server and two for a highly available setup.

Cheers,

Kalyan

Share This

After a successful install and test of x86 Windows Server 2008, it was time to put the x64 version through the same test.

We will talk about the x64 installation and experiences in another post.In this post, we will focus on the Hyper-V installation, configuration and experiences on our 16 core AMD Opteron server, A5808-32.

Hyper-V is the new Microsoft hypervisor technology included with certain SKUs of Windows Server 2008 x64 version. Hyper-V, included in the Windows Server 2008 RC1, is still in beta stages.

Installation:

Hyper-V is a new server role. This role has to be added after Windows Server 2008 installs and boots up. This installs the Hypervisor and reboots the server.

Now a new role shows up under “Roles” in the “Server Manager”. The new role is “Hyper-V” and has the new category “Microsoft Hyper-V Servers”. Your server will show up under this category. In the future, we may see other categories than just “Microsoft Hyper-V Servers”. This could be a place holder for the future management framework talked about here.

Creating a new VM is pretty straight forward. Select New->Virtual machine from the Server Manager and follow the wizard. Here is a screen shot of FedoraCore 6 x86_64 installation. Look, Ma … Linux on Windows!

Next up, Windows XP. Here is a screenshot of Windows XP install and FC6 & Windows XP VMs active on the Hyper-V server.

Over all experience:

It works well. We did not face any major issues with installation of Hyper-V or of VM’s. The management capabilities are polished. High Availability (HA) is implemented as a part of the Windows HA services.

There was no problem with mouse or keyboard inputs. No sticky mouse issues.

Certain Linux distros need noacpi flag to boot as VMs under Hyper-V.

Microsoft integration services CD ROM was not recognized by FC6. It logs “this disc does not contain any tracks I recognize.” Anyway, the integration services are suported only on Windows Server 2003 SP2 and Windows Server 2008.

Installation of both Windows XP and FC 6 was much longer than on VMWare or an actual system.

For a beta release, Hyper-V is suprisingly fully usable and comes fully integrated with Window Server 2008.

As per release notes, Hyper-V does not support more than 16 cores. So we configured our 32-core system with only 16 cores.

Future Work:

Test HA services

Capacity Planning

Quick Migration

Comments:

We used remote desktop connection to connect to the server and manage it. Once the mouse is captured inside the VM, there is no way to release it without going to the console of the Windows server and releasing it from there. Also the we could not use the mouse inside the FC6 from Remote Dektop. We could not find what the key sequence was to send Ctrl+Alt+<– to the remote machine using RemoteDekstop. Hopefully once a standalone application, like VI3, is released for Windows Hypervisor, it would be much easier to manage.

Edit:

If you are looking for detailed installation instructions, this post is useful.

Share This

Windows Server 2008 hit RC1 milestone recently. Windows Server is a popular choice on our 8 socket Opteron server, A5808-32, for a number of customers. RC1 is a good time for small vendors to test the compatibility of the OS with their servers and storage.

Here is a screen shot of Windows Server 2008 RC1 with 32 cores - 8 sockets, quad core AMD Opteron. This is the x86 version with full install.

Notice in the screen shot a report from CPU-Z utility - 8 processors, 4 cores and processor model. Windows version is displayed in the winver dialog and on the extreme left is the list of processors from the windows device manager.

Installation experiences:

• First install hung at 60%. Cold restart
• No video driver. Standard VGA. Well, you wouldn’t want Aero interface on your server anyway. We did not try to install the Windows 2003 version of the driver as the display was very usable.
• All NIC’s successfully detected
• On-board SATA successfully detected
• 32-cores (8 sockets) successfully detected
• System was configured with only 16GB instead of the maximum 256GB. We do not see any issues for use with 256GB
• iSCSI initiator successfully mounted a remote volume
• Windows “System” control panel applet does not display processor or memory details.

Now to try the x64 and Hyper-V versions. Stay tuned.

Cheers,

Kalyan

Share This

Trying VMotion was a natural next step after our success in initial tests with VMWare ESX on A5808-32. iSCSI services on HiPerStor provide for a very affordable shared storage for VMWare ESX. Well, lets jump right in. Read the rest of this entry »

Share This

Well, what a pleasant way to start the day!

After many successful tests yesterday (instal & test, iSCSI & NFS), I populated the system with 8 quad core AMD Opteron processors and left for the day hoping to come back and fire up ESX on 32 cores.

And here it is …. ESX working like a charm on 32 cores.

I hope to get those VMMark numbers out soon!

 Cheers,

Kalyan

Share This

After we successfully installed and tested VMWare ESX server 3.5 with Linux and Windows machines, it was time to test HiPerStor with VMWare ESX server.

If you are not familiar with HiPerStor, it is a storage product from HPC Systems, Inc. featuring support for NFS, SMB/CIFS, iSCSI and Apple shares. read more about it here.

To start with, we would like to use the iSCSI target features of HiPerStor and add a new VMFS datastore to our ESX server. ESX comes with a builtin iSCSI initiator and hence there is no need for additional iSCSI hardware.

Since the HiPerStor has no default iSCSI volumes defined, the first step is to open up the web management GUI and add a new iSCSI volume. With that taken care of, ESX server has to be configured, as follows, to be able to access iSCSI volumes.

• Create VMKernel port group
• A corresponding Service Console port group on the same subnet as the VMKernel port group
• Enable ESX software iSCSI initiator
• Discover new LUNs after configuring appropriate security settings
• Utilize the new storage device as VMFS datastore

None of the above steps posed any serious challenges. Discovering the iSCSI volumes was a breeze. However, we need to manually initate a “rescan” on the software iSCSI initator before the LUNs show up in a reasonable time. ESX was also happy to extend an already “hybrid” VMFS datastore we created earlier (SCSI partition on LSI controller + raw SATA disk on on-board controller). Well, thats nice.

But for now, I chose to create a separate datastore from the iSCSI volume so it will show up in the following screenshot.

You can also notice in the screenshot that there is also an NFS datastore created from an NFS share on HiPerStor. There cannot be a better testament to the versatility of HiPerStor than this screenshot. You should also know that the same HiPerStor system is concurrently serving up a bunch of CIFS shares as well. On the background you can catch a glimpse of the HiPerStor management GUI.

Now, to install a VM on the iSCSI datastore and boot it up. For this time, lets try it with SLES 10. And here is a screenshot of SLES 10 up & running successfully from the iSCSI datastore provided by our own HiPerStor.

You can notice in the background, the datastore information for this VM (SLES 10) and other key parameters. 

Next up: time measurements of booting up a VM from local disk and VM from an iSCSI disk, if I can figure out a way to measure the time accurately.

EDIT: I had to login to the ESX server and execute the commands to enable iSCSI before I could see the LUNs from the storage server.

Cheers,

Kalyan

Share This