Reasons for Using Jumbo Frames in RAC Interconnect

We are in the process of re-introducing Oracle RAC back into the organisation. I am having difficulty explaining the reasons for using Jumbo frames over the interconnect.

The books I have consulted seem to assume that there would not be much resistance to using Jumbo frames. Thus, their explanations for choosing Jumbo frames are sparse, and they concentrate on implementation details.

In Oracle 10g RAC: Grid, Services & Clustering by Murali Vallath,

Jumbo frame support is designed to enhance Ethernet networking throughput and reduce significantly the large CPU utilization of large file transfers like large multimedia files or large datafiles by enabling more efficient larger payloads per packet. By sending larger payloads per packet, fewer packets need to be routed, reducing the CPU overhead and potentially improving networking throughput. By using jumbo frames, the transfer frame sizes for Ethernet can be increased to 9,000 bytes.

p. 157

Unfortunately, none of the reasons given by Vallath are relevant to a RAC interconnect. The interconnect network is supposed to be on the same subnet, and must not be routed, at all. The interconnect is not supposed to be used for file transfer—it is used for cache fusion and RAC signalling.

In Expert Oracle RAC 12c by Riyaj Shamsudeen, Syed Jaffar Hussain, Kai Yu, Tariq Farooq,

Fragmentation and reassembly are performed in memory. Excessive fragmentation and reassembly issues can lead to higher CPU usage (accounted to kernel mode CPU usage) in the database nodes.

p. 255, Chapter 9: Network Practices

In Expert Oracle RAC Performance Diagnostics and Tuning by Murali Vallath, the recommendations are:

Best Practice Jumbo frames provide overall performance improvements in a RAC environment and should be used.

Kindle Location 2896

Oracle supports and recommends the use of jumbo frames as a best practice for the private interconnect in a RAC configuration.

Kindle Location 3082

In Recommendation for the Real Application Cluster Interconnect and Jumbo Frames (Doc ID 341788.1), the benefits are said to come from increased performance:

For RAC Interconnect traffic, devices correctly configured for Jumbo Frame improves performance by reducing the TCP, UDP, and Ethernet overhead that occurs when large messages have to be broken up into the smaller frames of standard Ethernet. Because one larger packet can be sent, inter-packet latency between various smaller packets is eliminated. The increase in performance is most noticeable in scenarios requiring high throughput and bandwidth and when systems are CPU bound.

When using Jumbo Frames, fewer buffer transfers are required which is part of the reduction for fragmentation and reassembly in the IP stack, and thus has an impact in reducing the latency of a an Oracle block transfer.

As illustrated in the configuration section, any incorrect setup may prevent instances from starting up or can have a very negative effect on the performance.

Emphasis Mine

Based on this reading, the best summary I can come up with is that Jumbo Frames have the potential to reduce CPU usage under high CPU load and heavy traffic on the interconnect. The reason for this is due to the reduction in fragmentation and reassembly effort.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s