We are in the process of re-introducing Oracle RAC back into the organisation. I am having difficulty explaining the reasons for using Jumbo frames over the interconnect.
The books I have consulted seem to assume that there would not be much resistance to using Jumbo frames. Thus, their explanations for choosing Jumbo frames are sparse, and they concentrate on implementation details.
Jumbo frame support is designed to enhance Ethernet networking throughput and reduce significantly the large CPU utilization of large file transfers like large multimedia files or large datafiles by enabling more efficient larger payloads per packet. By sending larger payloads per packet, fewer packets need to be routed, reducing the CPU overhead and potentially improving networking throughput. By using jumbo frames, the transfer frame sizes for Ethernet can be increased to 9,000 bytes.
Unfortunately, none of the reasons given by Vallath are relevant to a RAC interconnect. The interconnect network is supposed to be on the same subnet, and must not be routed, at all. The interconnect is not supposed to be used for file transfer—it is used for cache fusion and RAC signalling.
In Expert Oracle RAC 12c by Riyaj Shamsudeen, Syed Jaffar Hussain, Kai Yu, Tariq Farooq,
Fragmentation and reassembly are performed in memory. Excessive fragmentation and reassembly issues can lead to higher CPU usage (accounted to kernel mode CPU usage) in the database nodes.
p. 255, Chapter 9: Network Practices
In Expert Oracle RAC Performance Diagnostics and Tuning by Murali Vallath, the recommendations are:
Best Practice Jumbo frames provide overall performance improvements in a RAC environment and should be used.
Kindle Location 2896
Oracle supports and recommends the use of jumbo frames as a best practice for the private interconnect in a RAC configuration.
Kindle Location 3082
In Recommendation for the Real Application Cluster Interconnect and Jumbo Frames (Doc ID 341788.1), the benefits are said to come from increased performance:
For RAC Interconnect traffic, devices correctly configured for Jumbo Frame improves performance by reducing the TCP, UDP, and Ethernet overhead that occurs when large messages have to be broken up into the smaller frames of standard Ethernet. Because one larger packet can be sent, inter-packet latency between various smaller packets is eliminated. The increase in performance is most noticeable in scenarios requiring high throughput and bandwidth and when systems are CPU bound.
When using Jumbo Frames, fewer buffer transfers are required which is part of the reduction for fragmentation and reassembly in the IP stack, and thus has an impact in reducing the latency of a an Oracle block transfer.
As illustrated in the configuration section, any incorrect setup may prevent instances from starting up or can have a very negative effect on the performance.
Based on this reading, the best summary I can come up with is that Jumbo Frames have the potential to reduce CPU usage under high CPU load and heavy traffic on the interconnect. The reason for this is due to the reduction in fragmentation and reassembly effort.