[[Image:VIO-Server-General.png]]

[[Image:VIO-Server-Details.png]]

[[Image:HA-VIO-Server-Details.png]]

TECHNOLOGY PPT, Page 51

Virtual I/O Server

Provides an operating environment for virtual I/O administration
- Virtual I/O server administration
- Restricted scriptable command line user interface (CLI)
Minimum hardware requirements
- POWER5 VIO capable machine
- Hardware management console
- Storage adapter
- Physical disk
- Ethernet adapter
- At least 128 MB of memory
Capabilities of the Virtual I/O Server
- Ethernet Adapter Sharing
- Virtual SCSI disk
  - Virtual I/O Server Version 1.1 is addressed for selected configurations, which include specific models of EMC, HDS, and STK disk subsystems, attached using Fiber Channel
- Interacts with AIX and Linux partitions

The Virtual I/O Server provides a restricted scriptable command line user interface (CLI). All aspects of Virtual I/O server administration are accomplished through the CLI, including:

Device management (physical, virtual, LVM)
Network configuration
Software installation and update
Security
User management
Installation of OEM software
Maintenance tasks
The creation and deletion of the virtual client and server adapter is managed by the HMC GUI and POWER5 server firmware. The association between the client and server adapters is defined when the virtual adapters are created.

The optional Advanced POWER Virtualization hardware feature, which enables micro-partitioning on a POWER5 servers, is required to activate the Virtual I/O Server. A small logical partition with the enough resources to share to other partitions is required. The following is a list of minimum hardware requirements to create the Virtual I/O Server partition:

POWER5 server, the VIO capable machine.
Hardware management console to create the partition and assign resources.
Storage adapter: The server partition needs at least one storage adapter.
Physical disk: A disk large enough to make sufficient-sized logical volumes on it.
Ethernet adapter: Allows securely route network traffic from a virtual Ethernet to a real network adapter.
Memory: At least 128 MB of memory.
The Virtual I/O Server provides the Virtual SCSI (VSCSI) Target and Shared Ethernet adapter virtual I/O function to client partitions. This is accomplished by assigning physical devices to the Virtual I/O Server partition, then configuring virtual adapters on the clients to allow communication between the client and the Virtual I/O Server.

TECHNOLOGY PPT, Page 52

Installation CD when Advanced POWER Virtualization feature is ordered
Configuration approaches for high availability
- Virtual I/O Server
  - LVM mirroring
  - Multipath I/O
  - EtherChannel
- Second virtual I/O server instance in another partition

Installation of the Virtual I/O Server partition is performed from a special mksysb CD that will be provided to customers that order the Advanced POWER Virtualization feature.

This is a dedicated software for the virtual I/O server operations, so the virtual I/O server software is only supported in virtual I/O server partitions.

The Virtual I/O Server partition itself is configured using a command line interface. Defining partition resources such as virtual Ethernet or virtual disk connections to client systems requires use of the HMC.

Virtual I/O server supports the following operating systems as virtual I/O client:

AIX 5L Version 5.3
SUSE LINUX Enterprise Server 9 for POWER
Red Hat Enterprise Linux AS for POWER Version 3
Red Hat Enterprise Linux AS for POWER Version 4

When we talk about providing high availability for the virtual I/O server, we are talking about incorporating the I/O resources (physical and virtual) on the virtual I/O server as well as the client partitions into a configuration that is designed to eliminate single points of failure.

The virtual I/O server per se is not highly available. If there is a problem in the virtual I/O server or if it should crash, the client partitions will see I/O errors and not be able to access the adapters and devices which are backed by the virtual I/O server.

However, redundancy can be built into the configuration of the physical and virtual I/O resources at several stages.
Since the virtual I/O server is an AIX based appliance, redundancy for physical devices attached to the virtual I/O server can be provided by using capabilities like LVM mirroring, Multipath I/O, and EtherChannel.

When running two instances of the virtual I/O server, you can use LVM mirroring, Multipath I/O, EtherChannel, or Multipath routing with dead gateway detection in the client partition to provide highly available access to virtual resources hosted in the separate virtual I/O server partitions.

TECHNOLOGY PPT, Page 53

Virtual SCSI

Allows sharing of storage devices
Vital for shared processor partitions
- Overcomes potential limit of adapter slots due to Micro-Partitioning
- Allows the creation of logical partitions without the need for additional physical resources
Allows attachment of previously unsupported storage solutions

The virtualization features of the POWER5 platform support up to 254 partitions, while the biggest planned server only provides up to 160 I/O slots. With each partition requiring at least one I/O slot for disk attachment and another one for network attachment, this puts a constraint on the number of partitions. To overcome these physical limitations, I/O resources have to be virtualized. Virtual SCSI provides the means to do this for storage devices.
On the other hand, virtual I/O has a value proposition to it. It allows the creation of logical partitions without the need for additional physical resources. This facilities on demand computing and server consolidation. Virtual I/O also provides a more economic I/O model by using physical resources more efficiently through sharing.

Furthermore, virtual I/O allows attachment of previously unsupported storage solutions. As long as the virtual I/O server supports the attachment of a storage resource, any client partition can access this storage by using virtual SCSI adapters.

For example, at the time of writing, there is no native support for EMC storage devices on Linux. By running Linux in logical partition of a POWER5 server, this becomes possible.

A Linux client partition can access the EMC storage through a virtual SCSI adapter. Requests from the virtual adapters are mapped to the physical resources in the virtual I/O server partition. Driver support for the physical resources is therefore only needed in the virtual I/O server partition.

TECHNOLOGY PPT, Page 54, FIGURE MISSING

VSCSI server and client architecture overview

Virtual SCSI is based on a client/server relationship.
The virtual I/O resources are assigned using an HMC.
Virtual SCSI enables sharing of adapters as well as disk devices.
Dynamic LPAR operations allowed.
Dynamic mapping between physical and virtual resources on the virtual I/O server.

Virtual SCSI is based on a client/server relationship. The virtual I/O server owns the physical resources and acts as the server. The logical partitions access the virtual I/O resources provided by the virtual I/O server as the clients. The virtual I/O resources are assigned using an HMC.

Often the virtual I/O server partition is also referred to as hosting partition and the client partitions as hosted partitions.

Virtual SCSI enables sharing of adapters as well as disk devices.

To make a physical or a logical volume available to a client partition, it is assigned to a virtual SCSI server adapter in the virtual I/O server partition.

The client partition accesses its assigned disks through a virtual SCSI client adapter. It sees standard SCSI devices and LUNs through this virtual adapter.

Virtual SCSI resources can be assigned and removed dynamically. On the HMC, virtual SCSI target and server adapters can be assigned and removed from a partition using dynamic logical partitioning. The mapping between physical and virtual resources on the virtual I/O server can also be done dynamically.
This chart shows an example where one physical disk is split up into two logical volumes inside the virtual I/O server. Each of the two client partitions is assigned one logical volume, which it accesses through a virtual I/O adapter (vSCSI Client Adapter). Inside the partition, the disk is seen as normal hdisk.

TECHNOLOGY PPT, Page 55, FIGURE MISSING

Virtual devices

Are defined as LVs in the I/O server partition
- Normal LV rules apply
Appear as real devices (hdisks) in the hosted partition
Can be manipulated using Logical Volume Manager just like an ordinary physical disk
Can be used as a boot device and as a NIM target
Can be shared by multiple clients

A disk owned by the virtual I/O server can either be exported and assigned to a client partition as a whole or it can be split into several logical volumes. Each of these logical volumes can then be assigned to a different partition.

A virtual disk device is mapped by the server VSCSI adapter to a logical volume and presented to the hosted partition as a physical direct access device. There can be many virtual disk devices mapped onto a single physical disk. The system administrator will create a virtual disk device by choosing a logical volume and binding it to a VSCSI server adapter.

The virtual I/O adapters are connected to a virtual host bridge, which AIX treats much like a PCI host bridge. It is represented in the ODM as a bus device whose parent is sysplanar0. The virtual I/O adapters are represented as adapter devices with the virtual host bridge as their parent.

On the virtual I/O server, each logical volume or physical volume that is exported to a client partition is represented by a virtual target device, which is a child of a virtual SCSI server adapter.

On the client partition, the exported disks are visible as normal hdisks; however, they are defined in subclass vscsi. They have a virtual SCSI client adapter as parent. Note that virtual disks can be used as boot devices and as NIM targets.

Virtual disks can be shared by multiple clients, allowing for configurations using concurrent LVM, for example.

TECHNOLOGY PPT, Page 56, FIGURE MISSING

SCSI RDMA and Logical Remote Direct Memory Access

SCSI transport protocols define the rules for exchanging information between SCSI initiators and targets.
Virtual SCSI uses the SCSI RDMA Protocol (SRP).
SCSI initiators and targets have the ability to directly transfer information between their respective address spaces.
SCSI requests and responses are sent using the Virtual SCSI adapters.
The actual data transfer, however, is done using the Logical Redirected DMA protocol.

The SCSI family of standards provides many different transport protocols that define the rules for exchanging information between SCSI initiators and targets. Virtual SCSI uses the SCSI RDMA Protocol (SRP), which defines the rules for exchanging SCSI information in an environment where the SCSI initiators and targets have the ability to directly transfer information between their respective address spaces.

SCSI requests and responses are sent using the Virtual SCSI adapters that communicate through the POWER Hypervisor.
The actual data transfer however is done directly between a data buffer in the client partition and the physical adapter in the Virtual I/O Server by using the Logical Remote Direct Memory Access (LRDMA) protocol.
This chart shows how the data transfer using LRDMA works.

TECHNOLOGY PPT, Page 57, FIGURE MISSING

Virtual SCSI security

Only the owning partition has access to its data.
Data-information is copied directly from the PCI adapter to the client's memory.

Using Virtual SCSI means the Virtual I/O Server acts like a storage box to provide the data. Instead of SCSI or Fiber cable, the connection is done by the POWER Hypervisor. The Virtual SCSI device drivers of the I/O Server and the POWER Hypervisor ensures that only the owning partition has access to its data. Neither other partitions nor the I/O server itself are able to make the client data visible. Only the control-information is going through the I/O Server; the data-information, however, is copied directly from the PCI-adapter to the clients memory.

TECHNOLOGY PPT, Page 58, FIGURE MISSING

Performance considerations

Twice as many processor cycles to do VSCSI as a locally attached disk I/O (evenly distributed on the client partition and virtual I/O server)
- The path of each virtual I/O request involves several sources of overhead that are not present in a non-virtual I/O request.
- For a virtual disk backed by the LVM, there is also the performance impact of going through the LVM and disk device drivers twice.
If multiple partitions are competing for resources from a VSCSI server, care must be taken to ensure enough server resources (CPU, memory, and disk) are allocated to do the job.
If not constrained by CPU performance, dedicated partition throughput is comparable to doing local I/O.
Because there is no caching in memory on the server I/O partition, it's memory requirements should be modest.

Enabling VSCSI may not result in a performance benefit. This is because there is an overhead associated with Hypervisor calls, and because of the several steps involved for the I/O requests from the initiator to target partition, VSCSI will use additional CPU cycles when processing I/O requests. This will not give the same performance from VSCSI devices as from dedicated devices. The use of Virtual SCSI will roughly double the amount of CPU time to perform I/O as compared to using directly attached storage. This CPU load is split between the Virtual I/O Server and the Virtual SCSI Client. Performance is expected to degrade when multiple partitions are sharing a physical disk, and actual impact on overall system performance will vary by environment. The base-case configuration is when one physical disk is dedicated to a partition.

The following are general performance considerations when using Virtual SCSI:
Since VSCSI is a client/server model, the CPU utilization will always be higher than doing local I/O. A reasonable expectation is a total of twice as many cycles to do VSCSI as a locally attached disk I/O (more or less evenly distributed on the client and server).

If multiple partitions are competing for resources from a VSCSI server, care must be taken to ensure enough server resources (CPU, memory, and disk) are allocated to do the job.

If not constrained by CPU performance, dedicated partition throughput is comparable to doing local I/O.
There is no data caching in memory on the server partition. Thus, all I/Os that it services are essentially synchronous disk I/Os. Because there is no caching in memory on the server partition, its memory requirements should be modest.

The path of each virtual I/O request involves several sources of overhead that are not present in a non-virtual I/O request. For a virtual disk backed by the LVM, there is also the performance impact of going through the LVM and disk device drivers twice.

(IBM eServer p5 Virtualization - Performance Considerations, SG24-5768)

TECHNOLOGY PPT, Page 59, FIGURE MISSING

Limitations

Hosting partition must be available before hosted partition boot.
Virtual SCSI supports FC, parallel SCSI, and SCSI RAID.
Maximum of 65535 virtual slots in the I/O server partition.
Maximum of 256 virtual slots on a single partition.
Support for all mandatory SCSI commands.
Not all optional SCSI commands are supported.

Supported devices

At the time of writing, virtual SCSI supports FC, parallel SCSI, and SCSI raid devices. Any other devices, such as SSA, tape, or CD-ROM, are not supported.

Number of adapters

Virtual SCSI itself does not have any limitations in terms of the number of supported devices or adapters. However, the virtual I/O server partition supports a maximum of 65535 virtual I/O slots. A maximum of 256 virtual I/O slots can be assigned to a single partition.

Obviously, every I/O slot needs some resources to be instantiated. Therefore, the size of the virtual I/O server puts a limit to the number of virtual adapters that can be configured.

SCSI commands

The SCSI protocol defines mandatory and optional commands. While virtual SCSI supports all the mandatory commands, not all optional commands are supported.

TECHNOLOGY PPT, Page 60, FIGURE MISSING

Implementation guideline

Partitions with high performance and disk I/O requirements are not recommended for implementing VSCSI.
Partitions with very low performance and disk I/O requirements can be configured at minimum expense to use only a portion of a logical volume.
Boot disks for the operating system.
Web servers that will typically cache a lot of data.

Partitions with high performance and disk I/O requirements are not recommended for implementing VSCSI. Partitions with very low performance and disk I/O requirements can be configured at minimum expense to use only a logical volume. Using a logical volume for virtual storage means that the number of partitions is no longer limited by hardware, but the trade-off is that some of the partitions will have less than optimal storage performance. The suitable applications for VSCSI might be the boot disks for the operating system or Web servers that will typically cache a lot of data.

Virtual LAN

[[Image:Virtual-Lan-General.png]]

[[Image:Virtual-Lan.png]]

[[Image:Virtual-Lan-Alternate.png]]

[[Image:HA-Virtual-Lan.png]]

[[Image:Virtual-Lan-Flowchart.png]]

TECHNOLOGY PPT, Page 63, FIGURE MISSING

Virtual network segments on top of physical switch devices.
All nodes in the VLAN can communicate without any L3 routing or inter-VLAN bridging.
VLANs provides:
- Increased LAN security
- Flexible network deployment over traditional network devices
VLAN support in AIX is based on the IEEE 802.1Q VLAN implementation.
- VLAN ID tagging to Ethernet frames
- VLAN ID restricted switch ports

Virtual LAN (VLAN) is a technology used for establishing virtual network segments on top of physical switch devices. If configured appropriately, a VLAN definition can straddle multiple switches. Typically, a VLAN is a broadcast domain that enables all nodes in the VLAN to communicate with each other without any L3 routing or inter-VLAN bridging. In the diagram shown in this chart, two VLANs (VLAN 1 and 2) are defined on three switches (Switch A, B, and C). Although nodes C-1 and C-2 are physically connected to the same switch C, traffic between two nodes can be blocked. To enable communication between VLAN 1 and 2, L3 routing or inter-VLAN bridging should be established between them; this is typically provided by an L3 device.

The use of VLAN provides increased LAN security and flexible network deployment over traditional network devices.
VLAN support in AIX is based on the IEEE 802.1Q VLAN implementation. The IEEE 802.1Q VLAN is achieved by adding a VLAN ID tag to an Ethernet frame, and the Ethernet switches restricting the frames to ports that are authorized to receive frames with that VLAN ID. Switches also restrict broadcasts to the logical network by ensuring that a broadcast packet is delivered to all ports that are configured to receive frames with the VLAN ID that the broadcast frame was tagged with.

A port on a VLAN capable switch has a default PVID that indicates the default VLAN the port belongs to. The switch adds the PVID tag to untagged frames that are received by that port. In addition to a PVID, a port may belong to additional VLANs and have those VLAN IDs assigned to it that indicates the additional VLANs the port belongs to.
A port will only accept untagged packets or packets with a VLAN ID (PVID or additional VIDs) tag of the VLANs the port belongs to. A port configured in the untagged mode is only allowed to have a PVID and will receive untagged packets or packets tagged with the PVID. The untagged port feature helps systems that do not understand VLAN tagging communicate with other systems using standard Ethernet.

Each VLAN ID is associated with a separate Ethernet interface to the upper layers (IP and so on) and creates unique logical Ethernet adapter instances per VLAN (for example, ent1, ent2, and so on). You can configure multiple VLAN logical devices on a single system. Each VLAN logical devices constitutes an additional Ethernet adapter instance. These logical devices can be used to configure the same Ethernet IP interfaces as are used with physical Ethernet adapters.

TECHNOLOGY PPT, Page 64

Virtual LAN overview

Enables inter-partition communication.
- In-memory point to point connections
Physical network adapters are not needed.
Similar to high-bandwidth Ethernet connections.
Supports multiple protocols (IPv4, IPv6, and ICMP).
No Advanced POWER Virtualization feature required.
- POWER5 Systems
- AIX 5L V5.3 or appropriate Linux level
- Hardware management console (HMC)

The Virtual Ethernet enables inter-partition communication without the need for physical network adapters in each partition. The Virtual Ethernet allows the administrator to define in-memory point to point connections between partitions. These connections exhibit similar characteristics, as high bandwidth Ethernet connections supports multiple protocols (IPv4, IPv6, and ICMP). Virtual Ethernet requires a POWER5 system with either AIX 5L V5.3 or the appropriate level of Linux and a Hardware Management Console (HMC) to define the Virtual Ethernet devices. Virtual Ethernet does not require the purchase of any additional features or software, such as the Advanced Virtualization Feature.

Virtual Ethernet is also called Virtual LAN or even VLAN, which can be confusing, because these terms are also used in network topology topics. But the Virtual Ethernet, which uses virtual devices, has nothing to do with the VLAN known from Network-Topology, which divides a LAN in further Sub-LANs.

TECHNOLOGY PPT, Page 65, FIGURE MISSING

Virtual Ethernet connections

VLAN technology implementation
- Partitions can only access data directed to them.
Virtual Ethernet switch provided by the POWER Hypervisor
Virtual LAN adapters appears to the OS as physical adapters
- MAC-Address is generated by the HMC.
1-3 Gb/s transmission speed
- Support for large MTUs (~64K) on AIX.
Up to 256 virtual Ethernet adapters
- Up to 18 VLANs.
Bootable device support for NIM OS installations

The Virtual Ethernet connections supported in POWER5 systems use VLAN technology to insure that the partitions can only access data directed to them. The POWER Hypervisor provides a Virtual Ethernet switch function based on the IEEE 802.1Q VLAN standard, which allows partition communication within the same server.

Partitions wishing to communicate through a Virtual Ethernet channel will need to create an additional in-memory channel. This will require a user to be able to request the creation of an in-memory channel between partitions on the HMC. The kernel would create a virtual adapter for each memory channel indicated by the firmware. A normal AIX configuration routine would create the device special files. A virtual LAN adapter appears to the operating system in the same way as a physical adapter. A unique Media Access Control (MAC) address is also generated when the user creates a Virtual Ethernet adapter. A prefix value can be assigned for the system so that the generated MAC addresses in a system consists of a common system prefix, plus an algorithmically-generated unique part per adapter. The MAC-Address of the virtual adapter is generated by the HMC.

The transmission speed of Virtual Ethernet adapters is in the range of 1-3 Gigabits per second, depending on the transmission (MTU) size. The Virtual Ethernet Adapter supports, as Gigabit (Gb) Ethernet, Standard MTU-Sizes of 1500 Byte and Jumbo frames with 9000 Byte. Additionally for Gb Ethernet, the MTU-Size of 65280 Bytes is also supported in Virtual Ethernet. So, the MTU of 65280 Bytes can be only used inside a Virtual Ethernet.

A partition can support up to 256 Virtual Ethernet adapters with each Virtual Ethernet capable of being associated with up to 18 VLANs.

The Virtual Ethernet can also be used as a bootable device to allow such tasks as operating system installations to be performed using NIM.

TECHNOLOGY PPT, Page 66, FIGURE MISSING

Virtual Ethernet switch

Based on IEEE 802.1Q VLAN standard
- OSI-Layer 2
- Optional Virtual LAN ID (VID)
- 4094 virtual LANs supported
Up to 18 VIDs per virtual LAN port
Switch configuration through HMC

The POWER Hypervisor Switch is consistent with IEEE 802.1 Q. This standard defines the operation of virtual LAN (VLAN) bridges that permit the definition, operation, and administration of virtual LAN topologies within a bridged LAN infrastructure. It works on OSI-Layer 2 and supports up to 4096 networks (4096 VIDs). The Hypervisor works as a virtual Ethernet switch and maintains queues for each VLAN in its own memory. IEEE needs a Virtual LAN ID (VID). The LAN ID is optional in the above implementation. When this option is selected while adding a new Virtual LAN interface at the HMC, a VID can be chosen. Up to 4094 Virtual LANs are supported. Up to 18 VIDs can be configured per Virtual LAN port.

The authority to communicate between LPARs is granted by configuring ports on a virtual Ethernet switch maintained by the Hypervisor. The switch configuration is defined using the HMC.
When frames are sent across the network, a tag header is used to indicate to which VLAN a frame belongs. This ensures that the switch forwards the frame to only those ports that belong to that VLAN. Untagged packets are handled by adding the port VLAN identifier (PVID) to each frame.

TECHNOLOGY PPT, Page 67, FIGURE MISSING

How it works

When a message arrives at a Logical LAN Switch port from a Logical LAN adapter, the POWER Hypervisor caches the message's source MAC address to use as a filter for future messages to the Adapter. If the port is configured for VLAN headers, the VLAN header is checked against the port's allowable VLAN list. If the message specified VLAN is not in the port's configuration, the message is dropped. Once the message passes the VLAN header check, it passes into destination MAC address processing. If the port is NOT configured for VLAN headers, the Hypervisor (conceptually) inserts a two byte VLAN header (based upon the port's configured VLAN number). Next, the destination MAC address is processed by searching the table of cached MAC addresses (built from messages received at Logical LAN Switch ports (see above)). If a match for the MAC address is not found and if there is no Trunk Adapter defined for the specified VLAN number, then the message is dropped; otherwise, if a match for the MAC address is not found and if there is a Trunk Adapter defined for the specified VLAN number, then the message is passed on to the Trunk Adapter. If a MAC address match is found, then the associated switch port's configured, allowable VLAN number table is scanned for a match to VLAN number contained in the message's VLAN header. If a match is not found, the message is dropped. Next, the VLAN header configuration of the destination switch port is checked. If the port is configured for VLAN headers, the message is delivered to the destination Logical LAN adapters, including any inserted VLAN header. If the port is configured for no VLAN headers, the VLAN header is removed before being delivered to the destination Logical LAN adapter.

TECHNOLOGY PPT, Page 68, FIGURE MISSING

The measurements shown were taken using a 4-way POWER5 systems and AIX 5L V5.3 with several partitioning configurations. SMT (Simultaneous Multi Threading) is turned on for POWER5 systems. Virtual LAN adapters and the Gigabit Ethernet adapter default settings where used.
The Virtual Ethernet connections generally take up more processor time than a local adapter to move a packet (DMA versus copy). For shared processor partitions, performance will be gated by the partition definitions (for example, entitled capacity and number of processors). Small partitions communicating with each other will experience more packet latency due to partition context switching. In general, high bandwidth applications should not be deployed in small shared processor partitions. For dedicated partitions, throughput should be comparable to a 1 Gigabit Ethernet for small packets providing much better performance than 1 Gigabit Ethernet for large packets. For large packets, the Virtual Ethernet communication is copy bandwidth limited.
The throughput of the Virtual Ethernet scales nearly linear with the allocated capacity entitlements. The linear scaling of Virtual Ethernet with CPU entitlements shows that there is no measurable overhead when using shared processors versus dedicated processors for the throughput between Virtual LANs. Throughput is increasing, as expected, with growing MTU-Sizes (from MTU-Size 1500 to 9000 with factor ca. >3 and from 1500 to 65394 with factor >7).
The Virtual Ethernet adapter has higher raw throughput at all MTU sizes. On MTU 9000, the difference in throughput is very large, due to the fact that the in-memory copy that Virtual Ethernet uses to transfer data is more efficient at larger MTU.

TECHNOLOGY PPT, Page 69, FIGURE MISSING

Limitations

Virtual Ethernet can be used in both shared and dedicated processor partitions provided with the appropriate OS levels.
A mixture of Virtual Ethernet connections, real network adapters, or both are permitted within a partition.
Virtual Ethernet can only connect partitions within a single system.
A system's processor load is increased when using virtual Ethernet.

The following are limitations that must be considered when implementing an Virtual Ethernet.
Virtual Ethernet uses the system processors for all communication functions instead of offloading that load to processors on network adapter cards. As a result, there is an increase in system processor load generated by the use of Virtual Ethernet.

(Introduction to Advanced POWER Virtualization on IBM eServer p5 Servers, SG24-7940)

TECHNOLOGY PPT, Page 70, FIGURE MISSING

Implementation Guidelines

Know your environment and the network traffic.
Choose a high MTU size, as it makes sense for the network traffic in the Virtual LAN.
Use the MTU size 65394 if you expect a large amount of data to be copied inside your Virtual LAN.
Enable tcp_pmtu_discover and udp_pmtu_discover in conjunction with MTU size 65394.
Do not turn off SMT.
No dedicated CPUs are required for virtual Ethernet performance.

Because there is only a little experience with Virtual LANs until now, this guideline should not be taken as a good performance guarantee; they are only for orientation.

Know your environment and the network traffic.

Choose a high MTU size, as it makes sense for the network traffic in the Virtual LAN.

Use the MTU size 65394 if you expect a large amount of data to be copied inside your Virtual LAN.

Enable tcp_pmtu_discover and udp_pmtu_discover in conjunction with MTU size 65394, if there is a communication to physical adapters.

Do not turn off SMT (Simultaneous Multi-Threading) unless your applications demand it.

The throughput in Virtual LANs scale linear with CPU entitlements, so there is no need for dedicated CPUs for partitions because of Virtual LAN performance.

TECHNOLOGY PPT, Page 72, FIGURE MISSING

Shared Ethernet Adapter

Connects internal and external VLANs using one physical adapter.
SEA is a new service that acts as a layer 2 network switch.
- Securely bridges network traffic from a virtual Ethernet adapter to a real network adapter
SEA service runs in the Virtual I/O Server partition.
- Advanced POWER Virtualization feature required
- At least one physical Ethernet adapter required
No physical I/O slot and network adapter required in the client partition.

Using a Shared Ethernet Adapter (SEA), you can connect internal and external VLANs using one physical adapter.

Shared Ethernet Adapter is a new service that acts as a layer 2 network switch to securely bridge network traffic from a Virtual Ethernet to a real network adapter. The Shared Ethernet Adapter service runs in the Virtual I/O server partition.

TECHNOLOGY PPT, Page 72, FIGURE MISSING

Shared Ethernet Adapter

Virtual Ethernet MAC are visible to outside systems.
Broadcast/multicast is supported.
ARP (Address Resolution Protocol) and NDP (Neighbor Discovery Protocol) can work across a shared Ethernet.
One SEA can be shared by multiple VLANs and multiple subnets can connect using a single adapter on the Virtual I/O Server.
Virtual Ethernet adapter configured in the Shared Ethernet Adapter must have the trunk flag set.
- The trunk Virtual Ethernet adapter enables a layer-2 bridge to a physical adapter
IP fragmentation is performed or an ICMP packet too big message is sent when the shared Ethernet adapter receives IP (or IPv6) packets that are larger than the MTU of the adapter that the packet is forwarded through.

The Shared Ethernet Adapter allows partitions to communicate outside the system without having to dedicate a physical I/O slot and a physical network adapter to a client partition. The Shared Ethernet Adapter has the following characteristics:

Virtual Ethernet MAC are visible to outside systems.
Broadcast/multicast is supported.
ARP and NDP can work across a shared Ethernet.
In order to bridge network traffic between the Virtual Ethernet and external networks, the Virtual I/O Server partition has to be configured with at least one physical Ethernet adapter. One Shared Ethernet Adapter can be shared by multiple VLANs and multiple subnets can connect using a single adapter on the Virtual I/O Server.

A Virtual Ethernet adapter configured in the Shared Ethernet Adapter must have the trunk flag set. Once an Ethernet frame is sent from the Virtual Ethernet adapter on a client partition to the POWER Hypervisor, the POWER Hypervisor searches for the destination MAC address within the VLAN. If no such MAC address exists within the VLAN, it forwards the frame to the trunk Virtual Ethernet adapter that is defined on the same VLAN. The trunk Virtual Ethernet adapter enables a layer-2 bridge to a physical adapter.

The shared Ethernet directs packets based on the VLAN ID tags. It learns this information based on observing the packets originating from the virtual adapters. One of the virtual adapters in the Shared Ethernet Adapter is designated as the default PVID adapter. Ethernet frames without any VLAN ID tags are directed to this adapter and assigned the default PVID.

When the shared Ethernet receives IP (or IPv6) packets that are larger than the MTU of the adapter that the packet is forwarded through, either IP fragmentation is performed and the fragments forwarded or an ICMP packet too big message is returned to the source when the packet cannot be fragmented.

TECHNOLOGY PPT, Page 74, FIGURE MISSING

Virtual Ethernet and Shared Ethernet Adapter security

VLAN (virtual local area network) tagging description taken from the IEEE 802.1Q standard.
The implementation of this VLAN standard ensures that the partitions have no access to foreign data.
Only the network adapters (virtual or physical) that are connected to a port (virtual or physical) that belongs to the same VLAN can receive frames with that specific VLAN ID.

Similar to Virtual SCSI, the POWER Hypervisor also provides the connection between different partitions when using Virtual Ethernet. Inside the server, the POWER Hypervisor acts like an Ethernet switch. The connection to the external network is done by the virtual I/O Servers shared Ethernet function. This part of the I/O Server acts as a Layer 2 bridge to the physical adapters. The Virtual Ethernet implementation fulfills the IEEE 802.1Q standard, which describes VLAN (virtual local area network) tagging. This means that a VLAN ID tag is inserted into every Ethernet frame. The Ethernet switch restricts the frames to the ports that are authorized to receive frames with that VLAN ID. Every port of an Ethernet switch can be configured to be a member of several VLANs. Only the network adapters, both virtual and physical ones, which are connected to a port (virtual or physical) that belongs to the same VLAN can receive these frames. The implementation of this VLAN standard ensures that the partitions have no access to foreign data.

TECHNOLOGY PPT, Page 75, FIGURE MISSING

Performance considerations

The measurements shown were taken using a 4-way POWER5 system and AIX 5L V5.3 with several partitioning configurations. SMT (Simultaneous Multi Threading) is turned on, on POWER5 systems. Virtual LAN adapters and the Gigabit Ethernet adapter default settings were used.

The shared Ethernet adapter allows the adapters to stream data at media speed as long as it has enough CPU entitlements. This chart shows the throughput of the Virtual I/O-Server at MTU sizes of 1500 and 9000 in both modes, simplex and duplex.

CPU utilization per Gigabit of throughput is higher with Shared Ethernet adapter, as it has to receive from one end and send it out the other end, and because of the bridging functionality in the Virtual I/O-Server.

TECHNOLOGY PPT, Page 76, FIGURE MISSING

System processors are used for all communication functions, leading to a significant amount of system processor load.
One of the virtual adapters in the SEA on the Virtual I/O server must be defined as a default adapter with a default PVID.
Up to 16 Virtual Ethernet adapters with 18 VLANs on each can be shared on a single physical network adapter.
Shared Ethernet Adapter requires:
- POWER Hypervisor component of POWER5 systems
- AIX 5L Version 5.3 or appropriate Linux level

You must consider the following limitations when implementing Shared Ethernet Adapters in the Virtual I/O Server:
Because Shared Ethernet Adapter depends on Virtual Ethernet, which uses the system processors for all communication functions, a significant amount of system processor load can be generated by the use of Virtual Ethernet and Shared Ethernet Adapter.

One of the virtual adapters in the Shared Ethernet Adapter on the Virtual I/O Server must be defined as the default adapter with a default PVID. This virtual adapter is designated as the PVID adapter and Ethernet frames without any VLAN ID tags are assigned the default PVID and directed to this adapter.

Up to 16 Virtual Ethernet adapters with 18 VLANs on each can be shared on a single physical network adapter. There is no limit on the number of partitions that can attach to a VLAN. So the theoretical limit is very high. In practice, the amount of network traffic will limit the number of clients that can be served through a single adapter.

Shared Ethernet Adapter requires the POWER Hypervisor component of POWER5 systems and therefore cannot be used on POWER4 systems. It also cannot be used with AIX 5L Version 5.2, because the device drivers for Virtual Ethernet are only available for AIX 5L Version 5.3 and Linux. Thus, there is no way to connect a AIX 5L Version 5.2 system to a Shared Ethernet Adapter.

TECHNOLOGY PPT, Page 77, FIGURE MISSING

Implementation guideline

Know your environment and the network traffic.
Use a dedicated network adapter if you expect heavy network traffic between Virtual Ethernet and local networks.
If possible, use dedicated CPUs for the Virtual I/O Server.
Choose 9000 for MTU size, if this makes sense for your network traffic.
Don't use Shared Ethernet Adapter functionality for latency critical applications.
With MTU size 1500, you need about 1 CPU per gigabit Ethernet adapter streaming at media speed.
With MTU size 9000, 2 Gigabit Ethernet adapters can stream at media speed per CPU.

Because there is only a little experience with Virtual I/O server and Shared Ethernet Adapter until now, these guidelines should not be taken as a good performance guarantee; they are only for orientation.

Know your environment and the network traffic.
Don't use the Shared Ethernet Adapter functionality of the Virtual I/O-Server if you expect heavy network traffic between Virtual LANs and local networks.
Use a dedicated network adapter instead.
If possible, use dedicated CPUs for the Virtual I/O-Server (no shared processors).
Choose 9000 for MTU size, if this makes sense for your network traffic.
Don't use the Shared Ethernet Adapter functionality of the Virtual I/O-Server for latency critical applications.
With MTU size 1500, you need about 1 CPU per gigabit Ethernet adapter streaming at media speed.
With MTU size 9000, 2 Gigabit Ethernet adapters can stream at media speed per CPU.