Linux / 服务器 · 2022年4月2日 0

关于Mellanox mlx5 ethtool计数器

现在懒死了,八个月没有更新博客了,今天在处理一个业务网卡丢包的问题,监控查看丢包源是在机器进方向丢包的,通过ethtool查看计数器的值和ifconfig的dropped值哪个是一样的(也可以相近的毕竟也有个别其他情况丢的包,比如crc等。),判断丢包类型,通过rx_out_of_buffer计数器确认丢包类型,表面意思看起来和buffer有关系,但不确定具体的解释,通过网上搜索,确认这个计数器的解释,最终通过修改ring buffer降低了业务的丢包率。修改方法比较简单,如下:

查看当前设置:

ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX: 8192
RX Mini: 0
RX Jumbo: 0
TX: 8192
Current hardware settings:
RX: 8192
RX Mini: 0
RX Jumbo: 0
TX: 8192

修改设置:

ethtool -G eth0 rx 8192

ethtool -G eth0 tx 8192

ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX: 8192
RX Mini: 0
RX Jumbo: 0
TX: 8192
Current hardware settings:
RX: 8192
RX Mini: 0
RX Jumbo: 0
TX: 8192

中午11:30左右修改前后丢包对比

在修改的过程中遇到了一个莫名其妙的bug,导致我尝试在第四次修改的时候服务器自己重启了,在这没有去排查具体重启的原因,重启之后就可以正常顺利的修改了。有重要业务的同学,建议停服再操作,我当时报错的记录如下:

由于所搜的历程比较麻烦,所以做个记录分享下,Mellanox mlx5 ethtool计数器的解释,摘自Mellanox 官网。原链接:https://support.mellanox.com/s/article/understanding-mlx5-ethtool-counters

Understanding mlx5 ethtool Counters

Description:

This post shows the list of ethtool counters applicable for ConnectX-4 and above (mlx5 driver). All counters listed here are available via ethtool starting with MLNX_OFED 4.0.

Note: The post also provides a reference to ConnectX-3/ConnectX-3 Pro counters that co-exist for the mlx4 driver (see notes below).
 

References

Release Notes

This document was updated to match Linux kernel 4.20

Counters Overview

There are several counter groups, depends where the counter is being counted. In addition, each group of counters may have different counter types.

Counter Groups

  • Ring – software ring counters
  • Software Port – An aggregation of software ring counters.
  • vPort counters – traffic counters and drops due to steering or no buffers. May indicate on NIC issues. These counters include Ethernet traffic counters (including Raw Ethernet) and RDMA/RoCE traffic counters.
  • Physical port counters – the physical port connecting NIC to the network. May indicate on NIC issues or link or network issue. This measuring point holds information on standardized counters like IEEE 802.3, RFC2863, RFC 2819, RFC 3635 and additional counters like flow control, FEC and more. Physical port counters are not exposed to virtual machines.
  • Priority Port Counters – A set of the physical port counters, per priory per port.

Counter Types

Counters are divided to three Types

  • Traffic Informative Counters – counters which counts traffic. These counters can be used for load estimation of for general debug.
  • Traffic Acceleration Counters – counters which counts traffic which was accelerated by Mellanox driver or by hardware. The counters are an additional layer to the informative counter set and the same traffic is counted in both informative and acceleration counters. Acceleration counters are marked with [A].
  • Error Counters – Increment of these counters might indicate a problem. Each of these counter has an explanation and correction action.

Statistic can be fetched via the ip link or ethtool commands. ethtool provides more detailed information.

ip –s link show <if-name>

ethtool -S <if-name>

Acceleration Mechanism

The following acceleration mechanisms have dedicated counters:

  • TSO (TCP Segmentation Offload) – increasing outbound throughput and reducing CPU utilization by allowing the kernel to buffer multiple packets in a single large buffer. The NIC split the buffer into packet and transmits it
  • LRO (Large Receive Offload) – increasing inbound throughput and reducing CPU utilization by aggregation of o multiple incoming packet of a single stream to a single buffer
  • CHECKSUM (Checksum) – calculation of TCP checksum (by the NIC). The following CSUM offload are available (refer to skbuff.h for detailed explanation)
    • CHECKSUM_UNNECESSARY
    • CHECKSUM_NONE – no CSUM acceleration was used
    • CHECKSUM_COMPLETE – Device provided CSUM on the entire packet
    • CHECKSUM_PARTIAL – Device provided CSUM
  • CQE Compress – compression of Completion Queue Events (CQE) used for sparing bandwidth on PCIe and hence achieve better performance.

Counters Description

Ring / Software Port Counters

The following counters are available per ring or Software port.

These counters provide information on the amount of traffic that was accelerated by the NIC. The counters are counting the accelerated traffic in addition to the standard counters which counts it (i.e. accelerated traffic is counted twice).

The counter names in the table below refers to both ring and port counters. the notation for ring counters includes the [i] index without the braces. the notation for port counters doesn’t include the [i]. a counter name rx[i]_packets will be printed as rx0_packets for ring 0 and rx_packets for the software port

Ring / Software Port Counter Table

CounterDescriptionType
rx[i]_packetsThe number of packets received on ring i.ConnectX-3 naming : rx[i]_packetsInformative
rx[i]_bytesThe number of bytes received on ring i.ConnectX-3 naming : rx[i]_bytesInformative
tx[i]_packetsThe number of packets transmitted on ring i.ConnectX-3 naming : tx[i]_packetsInformative
tx[i]_bytesThe number of bytes transmitted on ring i.ConnectX-3 naming : tx[i]_bytesInformative
tx[i]_tso_packetsThe number of TSO packets transmitted on ring i [A].Acceleration
tx[i]_tso_bytesThe number of TSO bytes transmitted on ring i [A].Acceleration
tx[i]_tso_inner_packetsThe number of TSO packets which are indicated to be carry internal encapsulation transmitted on ring i [A]Acceleration
tx[i]_tso_inner_bytesThe number of TSO bytes which are indicated to be carry internal encapsulation transmitted on ring i [A].Acceleration
rx[i]_lro_packetsThe number of LRO packets received on ring i [A].Acceleration
rx[i]_lro_bytesThe number of LRO bytes received on ring i [A].Acceleration
rx[i]_csum_unnecessaryPackets received with a CHECKSUM_UNNECESSARY on ring i [A].Acceleration
rx[i]_csum_nonePackets received with CHECKSUM_NONE on ring i [A].Acceleration
rx[i]_csum_completePackets received with a CHECKSUM_COMPLETE on ring i [A].Acceleration
rx[i]_csum_unnecessary_innerPackets received with inner encapsulation with a CHECK_SUM UNNECESSARY on ring i [A].Acceleration
tx[i]_csum_partialPackets transmitted with a CHECKSUM_PARTIAL on ring i [A].Acceleration
tx[i]_csum_partial_innerPackets transmitted with inner encapsulation with a CHECKSUM_PARTIAL on ring i [A].Acceleration
tx[i]_csum_nonePackets transmitted with no hardware checksum acceleration on ring i.Informative
tx[i]_stoppedtx_queue_stopped (1)Events where SQ was full on ring i. If this counter is increased, check the amount of buffers allocated for transmission.Error
tx[i]_waketx_queue_wake (1)Events where SQ was full and has become not full on ring i.Error
tx[i]_droppedtx_queue_dropped (1)Packets transmitted that were dropped due to DMA mapping failure on ring i. If this counter is increased, check the amount of buffers allocated for transmission.Error
rx[i]_wqe_errThe number of wrong opcodes received on ring i.Error
tx[i]_nopThe number of no WQEs (empty WQEs) inserted to the SQ (related to ring i) due to the reach of the end of the cyclic buffer. When reaching near to the end of cyclic buffer the driver may add those empty WQEs to avoid handling a state the a WQE start in the end of the queue and ends in the beginning of the queue. This is a normal condition.Informative
rx[i]_mpwqe_fragThe number of WQEs that failed to allocate compound page and hence fragmented MPWQE’s (Multi Packet WQEs) were used on ring i. If this counter raise, it may suggest that there is no enough memory for large pages, the driver allocated fragmented pages. This is not abnormal condition.Informative
rx[i]_mpwqe_filler_cqesThe number of filler CQEs events that where issued on ring i.berfore kernel 4.19 name was rx[i]_mpwqe_fillerInformative
rx[i]_cqe_compress_blksThe number of receive blocks with CQE compression on ring i [A].Acceleration
rx[i]_cqe_compress_pktsThe number of receive packets with CQE compression on ring i [A].Acceleration
rx[i]_cache_reuseThe number of events of successful reuse of a page from a driver’s internal page cache – supported from Kernel 4.9Acceleration
rx[i]_cache_fullThe number of events of full internal page cache where driver can’t put a page back to the cache for recycling (page will be freed) – supported from Kernel 4.9Acceleration
rx[i]_cache_emptyThe number of events where cache was empty – no page to give. driver shall allocate new page – supported from Kernel 4.9Acceleration
rx[i]_cache_busyThe number of events where cache head was busy and cannot be recycled. driver allocated new page – supported from Kernel 4.9Acceleration
rx[i]_xmit_moreThe number of packets sent with xmit_more indication set on the skbuff (no doorbell) – Supported from kernel 4.8Acceleration
tx[i]_cqesThe number of completions received on the CQ of TX ring. Supported from kernel 4.19Informative
ch[i]_pollThe number of invocations of NAPI poll of channel. Supported from kernel 4.19Informative
ch[i]_armThe number of times the NAPI poll function completed and armed the completion queues on channelSupported from kernel 4.19Informative
ch[i]_aff_changeThe number of times the NAPI poll function explicitly stopped execution on a CPU due to a change in affinity, on channel. Supported from kernel 4.19Informative
rx[i]_congst_umrThe number of times an outstanding UMR request is delayed due to congestion, on ringSupported from kernel 4.19Error
ch[i]_eventsThe number of hard interrupt events on the completion queues of channel. Supported from kernel 4.19Informative
rx[i]_mpwqe_filler_stridesThe number of strides consumed by filler CQEs on ring. Supported from kernel 4.19Informative
rx[i]_xdp_tx_xmitThe number of packets forwarded back to the port due to XDP program XDP_TX action (bouncing). these packets are not counted by other software counters. These packets are counted by physical port and vPort counters – supported from kernel 4.9Before kernel 4.19 name was rx[i]_xdp_txInformative
rx[i]_xdp_tx_fullThe number of packets that should have been forwarded back to the port due to XDP_TX action but were dropped due to full tx queue. these packets are not counted by other software counters. These packets are counted by physical port and vPort countersyou may open more rx queues and spread traffic rx over all queues and/or increase rx ring sizesupported from kernel 4.9Error
rx[i]_xdp_tx_errThe number of times an XDP_TX error such as frame too long and frame too short occurred on XDP_TX ring of RX ring. Supported from kernel 4.19Error
rx[i]_xdp_tx_cqesrx_xdp_tx_cqe (1)The number of completions received on the CQ of the XDP-TX ring. Supported from kernel 4.19Informative
rx[i]_xdp_dropThe number of packets dropped due to XDP program XDP_DROP action. these packets are not counted by other software counters. These packets are counted by physical port and vPort counters – supported from kernel 4.9Informative
rx[i]_xdp_redirectThe number of times an XDP redirect action was triggered on ring. .Supported from kernel 4.19Acceleration
tx[i]_xdp_xmitThe number of packets redirected to the interface(due to XDP redirect). These packets are not counted by other software counters. These packets are counted by physical port and vPort counters – Supported from kernel 4.19Informative
tx[i]_xdp_fullThe number of packets redirected to the interface(due to XDP redirect), but were dropped due to full tx queue. these packets are not counted by other software counters. you may enlarge tx queues. Supported from kernel 4.19Informative
tx[i]_xdp_errThe number of packets redirected to the interface(due to XDP redirect) but were dropped due to error such as frame too long and frame too short . Supported from kernel 4.19Error
tx[i]_xdp_cqesThe number of completions received for packets redirected to the interface(due to XDP redirect) on the CQ . Supported from kernel 4.19Informative
rx[i]_cache_waiveThe number of cache evacuation. This can occur due to page move to another NUMA node or page was pfmemalloc-ed and should be freed as soon as possible. Supported from kernel 4.14Acceleration

Notes:

(1) The corresponding ring and global counters do not share the same name (i.e. do not follow the common naming scheme).

vPort Counters

Counters on the eswitch port that is connected to the VNIC.

 

vPort Counter Table

CounterDescriptionType
rx_vport_unicast_packetsUnicast packets received, steered to a port including Raw Ethernet QP/DPDK traffic, excluding RDMA trafficInformative
rx_vport_unicast_bytesUnicast bytes received, steered to a port including Raw Ethernet QP/DPDK traffic, excluding RDMA trafficInformative
tx_vport_unicast_packetsUnicast packets transmitted, steered from a port including Raw Ethernet QP/DPDK traffic, excluding RDMA trafficInformative
tx_vport_unicast_bytesUnicast bytes transmitted, steered from a port including Raw Ethernet QP/DPDK traffic, excluding RDMA trafficInformative
rx_vport_multicast_packetsMulticast packets received, steered to a port including Raw Ethernet QP/DPDK traffic, excluding RDMA trafficInformative
rx_vport_multicast_bytesMulticast bytes received, steered to a port including Raw Ethernet QP/DPDK traffic, excluding RDMA trafficInformative
tx_vport_multicast_packetsMulticast packets transmitted, steered from a port including Raw Ethernet QP/DPDK traffic, excluding RDMA trafficInformative
tx_vport_multicast_bytesMulticast bytes transmitted, steered from a port including Raw Ethernet QP/DPDK traffic, excluding RDMA trafficInformative
rx_vport_broadcast_packetsBroadcast packets received, steered to a port including Raw Ethernet QP/DPDK traffic, excluding RDMA trafficInformative
rx_vport_broadcast_bytesBroadcast bytes received, steered to a port including Raw Ethernet QP/DPDK traffic, excluding RDMA trafficInformative
tx_vport_broadcast_packetsBroadcast packets transmitted, steered from a port including Raw Ethernet QP/DPDK traffic, excluding RDMA trafficInformative
tx_vport_broadcast_bytesBroadcast packets transmitted, steered from a port including Raw Ethernet QP/DPDK traffic, excluding RDMA trafficInformative
rx_vport_rdma_unicast_packetsRDMA unicast packets received, steered to a port (counters counts RoCE/UD/RC traffic) [A]Acceleration
rx_vport_rdma_unicast_bytesRDMA unicast bytes received, steered to a port (counters counts RoCE/UD/RC traffic) [A]Acceleration
tx_vport_rdma_unicast_packetsRDMA unicast packets transmitted, steered from a port (counters counts RoCE/UD/RC traffic) [A]Acceleration
tx_vport_rdma_unicast_bytesRDMA unicast bytes transmitted, steered from a port (counters counts RoCE/UD/RC traffic) [A]Acceleration
rx_vport_ rdma _multicast_packetsRDMA multicast packets received, steered to a port (counters counts RoCE/UD/RC traffic) [A]Acceleration
rx_vport_ rdma _multicast_bytesRDMA multicast bytes received, steered to a port (counters counts RoCE/UD/RC traffic) [A]Acceleration
tx_vport_ rdma _multicast_packetsRDMA multicast packets transmitted, steered from a port (counters counts RoCE/UD/RC traffic) [A]Acceleration
tx_vport_ rdma _multicast_bytesRDMA multicast bytes transmitted, steered from a port (counters counts RoCE/UD/RC traffic) [A]Acceleration
rx_steer_missed_packetsNumber of packets that was received by the NIC, however was discarded because it did not match any flow in the NIC flow table. supported from kernel 4.16Error
rx_packetsRepresentor only: packets received, that were handled by the hypervisor. supported from kernel 4.18Informative
rx_bytesRepresentor only: bytes received, that were handled by the hypervisor. supported from kernel 4.18Informative
tx_packetsRepresentor  only: packets transmitted, that were handled by the hypervisor. supported from kernel 4.18Informative
tx_bytesRepresentor  only: bytes transmitted, that were handled by the hypervisor. supported from kernel 4.18Informative

Physical Port Counters

The physical port counters are the counters on the external port connecting adapter to the network. This measuring point holds information on standardized counters like IEEE 802.3, RFC2863, RFC 2819, RFC 3635 and additional counters like flow control, FEC and more.

Physical Port Counter Table

CounterDescriptionType
rx_packets_phyThe number of packets received on the physical port. This counter doesn’t include packets that were discarded due to FCS, frame size and similar errors.ConnectX-3 naming : rx_packetsInformative
tx_packets_phyThe number of packets transmitted on the physical port.ConnectX-3 naming : tx_packetsInformative
rx_bytes_phyThe number of bytes received on the physical port, including Ethernet header and FCS.ConnectX-3 naming : rx_bytesInformative
tx_bytes_phyThe number of bytes transmitted on the physical port.ConnectX-3 naming : tx_bytesInformative
rx_multicast_phyThe number of multicast packets received on the physical port.ConnectX-3 naming : rx_multicast_packetsInformative
tx_multicast_phyThe number of multicast packets transmitted on the physical port.ConnectX-3 naming : tx_multicast_packetsInformative
rx_broadcast_phyThe number of broadcast packets received on the physical port.ConnectX-3 naming : rx_broadcast_packetsInformative
tx_broadcast_phyThe number of broadcast packets transmitted on the physical port.ConnectX-3 naming : tx_broadcast_packetsInformative
rx_crc_errors_phyThe number of dropped received packets due to FCS (Frame Check Sequence) error on the physical port. If this counter is increased in high rate, check the link quality using rx_symbol_error_phy and rx_corrected_bits_phy counters below.ConnectX-3 naming : rx_crc_errorsError
rx_in_range_len_errors_phyThe number of received packets dropped due to length/type errors on a physical port.ConnectX-3 naming : rx_in_range_length_errorError
rx_out_of_range_len_phyThe number of received packets dropped due to length greater than allowed on a physical port.If this counter is increasing, it implies that the peer connected to the adapter has a larger MTU configured. Using same MTU configuration shall resolve this issue.ConnectX-3 naming : rx_out_range_length_errorError
rx_oversize_pkts_phyThe number of dropped received packets due to length which exceed MTU size on a physical portIf this counter is increasing, it implies that the peer connected to the adapter has a larger MTU configured. Using same MTU configuration shall resolve this issue.ConnectX-3 naming : rx_frame_errorsError
rx_symbol_err_phyThe number of received packets dropped due to physical coding errors (symbol errors) on a physical port.Error
rx_mac_control_phyThe number of MAC control packets received on the physical port.Informative
tx_mac_control_phyThe number of MAC control packets transmitted on the physical port.Informative
rx_pause_ctrl_phyThe number of link layer pause packets received on a physical port. If this counter is increasing, it implies that the network is congested and cannot absorb the traffic coming from to the adapter.Informative
tx_pause_ctrl_phyThe number of link layer pause packets transmitted on a physical port. If this counter is increasing, it implies that the NIC is congested and cannot absorb the traffic coming from the network.Informative
rx_unsupported_op_phyThe number of MAC control packets received with unsupported opcode on a physical port.Error
rx_discards_phyThe number of received packets dropped due to lack of buffers on a physical port. If this counter is increasing, it implies that the adapter is congested and cannot absorb the traffic coming from the network.ConnectX-3 naming : rx_fifo_errorsError
tx_discards_phy The number of packets which were discarded on transmission, even no errors were detected. the drop might occur due to link in down state, head of line drop, pause from the network, etcError 
tx_errors_phyThe number of transmitted packets dropped due to a length which exceed MTU size on a physical port.Error
rx_undersize_pkts_phyThe number of received packets dropped due to length which is shorter than 64 bytes on a physical port. If this counter is increasing, it implies that the peer connected to the adapter has a non-standard MTU configured or malformed packet had arrived.Error
rx_fragments_phyThe number of received packets dropped due to a length which is shorter than 64 bytes and has FCS error on a physical port. If this counter is increasing, it implies that the peer connected to the adapter has a non-standard MTU configured.Error
rx_jabbers_phyThe number of received packets d due to a length which is longer than 64 bytes and had FCS error on a physical port.Error
rx_64_bytes_phyThe number of packets received on the physical port with size of 64 bytes.Informative
rx_65_to_127_bytes_phyThe number of packets received on the physical port with size of 65 to 127 bytes.Informative
rx_128_to_255_bytes_phyThe number of packets received on the physical port with size of 128 to 255 bytes.Informative
rx_256_to_511_bytes_phyThe number of packets received on the physical port with size of 256 to 512 bytes.Informative
rx_512_to_1023_bytes_phyThe number of packets received on the physical port with size of 512 to 1023 bytes.Informative
rx_1024_to_1518_bytes_phyThe number of packets received on the physical port with size of 1024 to 1518 bytes.Informative
rx_1519_to_2047_bytes_phyThe number of packets received on the physical port with size of 1519 to 2047 bytes.Informative
rx_2048_to_4095_bytes_phyThe number of packets received on the physical port with size of 2048 to 4095 bytes.Informative
rx_4096_to_8191_bytes_phyThe number of packets received on the physical port with size of 4096 to 8191 bytes.Informative
rx_8192_to_10239_bytes_phyThe number of packets received on the physical port with size of 8192 to 10239 bytes.Informative
link_down_events_phyThe number of times where the link operative state changed to down. In case this counter is increasing it may imply on port flapping. You may need to replace the cable/transceiver.Error
rx_out_of_bufferNumber of times receive queue had no software buffers allocated for the adapter’s incoming traffic.Error
module_bus_stuckThe number of times that module’s I2C bus (data or clock) short-wire was detected. You may need to replace the cable/transceiver – supported from kernel 4.10Error
module_high_tempThe number of times that the module temperature was too high. If this issue persist, you may need to check the ambient temperature or replace the cable/transceiver module – supported from kernel 4.10Error
module_bad_shortedThe number of times that the module cables were shorted. You may need to replace the cable/transceiver module – supported from kernel 4.10Error
module_unplugThe number of times that module was ejected – supported from kernel 4.10Informative
rx_buffer_passed_thres_phyThe number of events where the port receive buffer was over 85% full. Supported from kernel 4.14Informative
tx_pause_storm_warning_eventsThe number of times the device was sending pauses for a long period of time – supported from kernel 4.15Informative
tx_pause_storm_error_eventsThe number of times the device was sending pauses for a long period of time, reaching time out and disabling transmission of pause frames. on the period where pause frames were disabled, drop could have been occurred – supported from kernel 4.15Error
rx[i]_buff_alloc_err / rx_buff_alloc_errFailed to allocate a buffer to received packet (or SKB) on port (or per ring)Error
rx_bits_phyThis counter provides information on the total amount of traffic that could have been received and can be used as a guideline to measure the ratio of errored traffic in rx_pcs_symbol_err_phyrx_corrected_bits_phy.Informative
rx_pcs_symbol_err_phyThis counter counts the number of symbol errors that wasn’t corrected by FEC correction algorithm or that FEC algorithm was not active on this interface. If this counter is increasing, it implies that the link between the NIC and the network is suffering from high BER, and that traffic is lost. You may need to replace the cable/transceiver. The error rate is the number of rx_pcs_symbol_err_phy divided by the number of rx_phy_bits on a specific time frame.Error
rx_corrected_bits_phyThe number of corrected bits on this port according to active FEC (RS/FC). If this counter is increasing, it implies that the link between the NIC and the network is suffering from high BER. The corrected bit rate is the number of rx_corrected_bits_phy divided by the number of rx_phy_bits on a specific time frameError
phy_raw_errors_lane[l]This counter counts the number of physical raw errors per lane [l] index. The counter counts errors before FEC corrections. If this counter is increasing, it implies that the link between the NIC and the network is suffering from high BER, and that traffic might be lost. You may need to replace the cable/transceiver. Please check in accordance with rx_corrected_bits_phy.
Supported from kernel 4.20
Error

Priority Port Counters

The following counters are physical port counters that being counted per L2 priority (0-7).

Note: ‘p’ in the counter name represents the priority.

Priority Port Counter Table

CounterDescriptionType
rx_prio[p]_bytesThe number of bytes received with priority p on the physical port.ConnectX-3 naming :rx_prio_[p]_bytes. this counter also counts packets with no vlanInformative
rx_prio[p]_packetsThe number of packets received with priority p on the physical port.ConnectX-3 naming : rx_prio_[p]_packets. this counter also counts packets with no vlanInformative
tx_prio[p]_bytesThe number of bytes transmitted on priority p on the physical port.ConnectX-3 naming :tx_prio_[p]_bytes.Informative
tx_prio[p]_packetsThe number of packets transmitted on priority p on the physical port.ConnectX-3 naming : tx_prio_[p]_packets.Informative
rx_prio[p]_pauseThe number of pause packets received with priority p on a physical port. If this counter is increasing, it implies that the network is congested and cannot absorb the traffic coming from the adapter.Note: This counter is available only if PFC was enabled on priority p. Refer to HowTo Configure PFC on ConnectX-4 .ConnectX-3 naming : rx_pause_prio_pInformative
rx_prio[p]_pause_durationThe duration of pause received (in microSec) on priority p on the physical port. The counter represents the time the port did not send any traffic on this priority. If this counter is increasing, it implies that the network is congested and cannot absorb the traffic coming from the adapter.Note: This counter is available only if PFC was enabled on priority p. Refer to HowTo Configure PFC on ConnectX-4 .ConnectX-3 naming : rx_pause_duration_prio_pInformative
rx_prio[p]_pause_transitionThe number of times a transition from Xoff to Xon on priority p on the physical port has occurred.Note: This counter is available only if PFC was enabled on priority p. Refer to HowTo Configure PFC on ConnectX-4 .ConnectX-3 naming : rx_pause_transition_prio_pInformative
tx_prio[p]_pauseThe number of pause packets transmitted on priority p on a physical port. If this counter is increasing, it implies that the adapter is congested and cannot absorb the traffic coming from the network.Note: This counter is available only if PFC was enabled on priority p. Refer to HowTo Configure PFC on ConnectX-4 .ConnectX-3 naming : tx_pause_prio_pInformative
tx_prio[p]_pause_durationThe duration of pause transmitter (in microSec) on priority p on the physical port.Note: This counter is available only if PFC was enabled on priority p. Refer to HowTo Configure PFC on ConnectX-4 .ConnectX-3 naming : tx_pause_duration_prio_pInformative
rx_prio[p]_buf_discardThe number of packets discarded by device due to lack of per host receive buffers. Supported from kernel 5.3Informative
rx_prio[p]_cong_discardThe number of packets discarded by device due to per host congestion. Supported from kernel 5.3Informative
rx_prio[p]_markedThe number of packets ecn marked by device due to per host congestion. Supported from kernel 5.3Informative
rx_prio[p]_discardThe number of packets discarded by device due to lack of receive buffers. Supported from kernel 5.6Infornative

Device Counters

CounterDescriptionType
rx_pci_signal_integrityCounts physical layer PCIe signal integrity errors, the number of transitions to recovery due to Framing errors and CRC (dlp and tlp).If this counter is raising, try moving the adapter card to a different slot to rule out a bad PCI slot. Validate that you are running with the latest firmware available and latest server BIOS version.Error
tx_pci_signal_integrityCounts physical layer PCIe signal integrity errors, the number of transition to recovery initiated by the other side (moving to recovery due to getting TS/EIEOS).If this counter is raising, try moving the adapter card to a different slot to rule out a bad PCI slot. Validate that you are running with the latest firmware available and latest server BIOS version.Error
outbound_pci_buffer_overflowThe number of packets dropped due to pci buffer overflow. If this counter is raising in high rate, it might indicate that the receive traffic rate for a host is larger than the PCIe bus and therefore a congestion occurs. Supported from kernel 4.14Informative
outbound_pci_stalled_rdThe percentage (in the range 0…100) of time within the last second that the NIC had outbound non-posted reads requests but could not perform the operation due to insufficient posted credits. Supported from kernel 4.14Informative
outbound_pci_stalled_wrThe percentage (in the range 0…100) of time within the last second that the NIC had outbound posted writes requests but could not perform the operation due to insufficient posted credits. Supported from kernel 4.14Informative
outbound_pci_stalled_rd_eventsThe number of seconds where outbound_pci_stalled_rd was above 30%. Supported from kernel 4.14Informative
outbound_pci_stalled_wr_eventsThe number of seconds where outbound_pci_stalled_wr was above 30%. Supported from kernel 4.14Informative
dev_out_of_bufferThe number of times the device owned queue had not enough buffers allocatedError

Full List of Counters

# ethtool -S eth5

 

NIC statistics:

rx_packets: 10

rx_bytes: 3420

tx_packets: 18

tx_bytes: 1296

tx_tso_packets: 0

tx_tso_bytes: 0

tx_tso_inner_packets: 0

tx_tso_inner_bytes: 0

tx_added_vlan_packets: 0

tx_nop: 0

rx_lro_packets: 0

rx_lro_bytes: 0

rx_ecn_mark: 0

rx_removed_vlan_packets: 0

rx_csum_unnecessary: 0

rx_csum_none: 0

rx_csum_complete: 10

rx_csum_unnecessary_inner: 0

rx_xdp_drop: 0

rx_xdp_redirect: 0

rx_xdp_tx_xmit: 0

rx_xdp_tx_full: 0

rx_xdp_tx_err: 0

rx_xdp_tx_cqe: 0

tx_csum_none: 18

tx_csum_partial: 0

tx_csum_partial_inner: 0

tx_queue_stopped: 0

tx_queue_dropped: 0

tx_xmit_more: 0

tx_recover: 0

tx_cqes: 18

tx_queue_wake: 0

tx_udp_seg_rem: 0

tx_cqe_err: 0

tx_xdp_xmit: 0

tx_xdp_full: 0

tx_xdp_err: 0

tx_xdp_cqes: 0

rx_wqe_err: 0

rx_mpwqe_filler_cqes: 0

rx_mpwqe_filler_strides: 0

rx_buff_alloc_err: 0

rx_cqe_compress_blks: 0

rx_cqe_compress_pkts: 0

rx_page_reuse: 0

rx_cache_reuse: 0

rx_cache_full: 0

rx_cache_empty: 2688

rx_cache_busy: 0

rx_cache_waive: 0

rx_congst_umr: 0

rx_arfs_err: 0

ch_events: 75

ch_poll: 75

ch_arm: 75

ch_aff_change: 0

ch_eq_rearm: 0

rx_out_of_buffer: 0

rx_if_down_packets: 15

rx_steer_missed_packets: 0

rx_vport_unicast_packets: 0

rx_vport_unicast_bytes: 0

tx_vport_unicast_packets: 0

tx_vport_unicast_bytes: 0

rx_vport_multicast_packets: 2

rx_vport_multicast_bytes: 172

tx_vport_multicast_packets: 12

tx_vport_multicast_bytes: 936

rx_vport_broadcast_packets: 37

rx_vport_broadcast_bytes: 9270

tx_vport_broadcast_packets: 6

tx_vport_broadcast_bytes: 360

rx_vport_rdma_unicast_packets: 0

rx_vport_rdma_unicast_bytes: 0

tx_vport_rdma_unicast_packets: 0

tx_vport_rdma_unicast_bytes: 0

rx_vport_rdma_multicast_packets: 0

rx_vport_rdma_multicast_bytes: 0

tx_vport_rdma_multicast_packets: 0

tx_vport_rdma_multicast_bytes: 0

tx_packets_phy: 0

rx_packets_phy: 0

rx_crc_errors_phy: 0

tx_bytes_phy: 0

rx_bytes_phy: 0

tx_multicast_phy: 0

tx_broadcast_phy: 0

rx_multicast_phy: 0

rx_broadcast_phy: 0

rx_in_range_len_errors_phy: 0

rx_out_of_range_len_phy: 0

rx_oversize_pkts_phy: 0

rx_symbol_err_phy: 0

tx_mac_control_phy: 0

rx_mac_control_phy: 0

rx_unsupported_op_phy: 0

rx_pause_ctrl_phy: 0

tx_pause_ctrl_phy: 0

rx_discards_phy: 0

tx_discards_phy: 0

tx_errors_phy: 0

rx_undersize_pkts_phy: 0

rx_fragments_phy: 0

rx_jabbers_phy: 0

rx_64_bytes_phy: 0

rx_65_to_127_bytes_phy: 0

rx_128_to_255_bytes_phy: 0

rx_256_to_511_bytes_phy: 0

rx_512_to_1023_bytes_phy: 0

rx_1024_to_1518_bytes_phy: 0

rx_1519_to_2047_bytes_phy: 0

rx_2048_to_4095_bytes_phy: 0

rx_4096_to_8191_bytes_phy: 0

rx_8192_to_10239_bytes_phy: 0

link_down_events_phy: 0

rx_prio0_bytes: 0

rx_prio0_packets: 0

tx_prio0_bytes: 0

tx_prio0_packets: 0

rx_prio1_bytes: 0

rx_prio1_packets: 0

tx_prio1_bytes: 0

tx_prio1_packets: 0

rx_prio2_bytes: 0

rx_prio2_packets: 0

tx_prio2_bytes: 0

tx_prio2_packets: 0

rx_prio3_bytes: 0

rx_prio3_packets: 0

tx_prio3_bytes: 0

tx_prio3_packets: 0

rx_prio4_bytes: 0

rx_prio4_packets: 0

tx_prio4_bytes: 0

tx_prio4_packets: 0

rx_prio5_bytes: 0

rx_prio5_packets: 0

tx_prio5_bytes: 0

tx_prio5_packets: 0

rx_prio6_bytes: 0

rx_prio6_packets: 0

tx_prio6_bytes: 0

tx_prio6_packets: 0

rx_prio7_bytes: 0

rx_prio7_packets: 0

tx_prio7_bytes: 0

tx_prio7_packets: 0

module_unplug: 0

module_bus_stuck: 0

module_high_temp: 0

module_bad_shorted: 0

ch0_events: 9

ch0_poll: 9

ch0_arm: 9

ch0_aff_change: 0

ch0_eq_rearm: 0

ch1_events: 23

ch1_poll: 23

ch1_arm: 23

ch1_aff_change: 0

ch1_eq_rearm: 0

ch2_events: 8

ch2_poll: 8

ch2_arm: 8

ch2_aff_change: 0

ch2_eq_rearm: 0

ch3_events: 19

ch3_poll: 19

ch3_arm: 19

ch3_aff_change: 0

ch3_eq_rearm: 0

ch4_events: 8

ch4_poll: 8

ch4_arm: 8

ch4_aff_change: 0

ch4_eq_rearm: 0

ch5_events: 8

ch5_poll: 8

ch5_arm: 8

ch5_aff_change: 0

ch5_eq_rearm: 0

rx0_packets: 0

rx0_bytes: 0

rx0_csum_complete: 0

rx0_csum_unnecessary: 0

rx0_csum_unnecessary_inner: 0

rx0_csum_none: 0

rx0_xdp_drop: 0

rx0_xdp_redirect: 0

rx0_lro_packets: 0

rx0_lro_bytes: 0

rx0_ecn_mark: 0

rx0_removed_vlan_packets: 0

rx0_wqe_err: 0

rx0_mpwqe_filler_cqes: 0

rx0_mpwqe_filler_strides: 0

rx0_buff_alloc_err: 0

rx0_cqe_compress_blks: 0

rx0_cqe_compress_pkts: 0

rx0_page_reuse: 0

rx0_cache_reuse: 0

rx0_cache_full: 0

rx0_cache_empty: 448

rx0_cache_busy: 0

rx0_cache_waive: 0

rx0_congst_umr: 0

rx0_arfs_err: 0

rx0_xdp_tx_xmit: 0

rx0_xdp_tx_full: 0

rx0_xdp_tx_err: 0

rx0_xdp_tx_cqes: 0

rx1_packets: 10

rx1_bytes: 3420

rx1_csum_complete: 10

rx1_csum_unnecessary: 0

rx1_csum_unnecessary_inner: 0

rx1_csum_none: 0

rx1_xdp_drop: 0

rx1_xdp_redirect: 0

rx1_lro_packets: 0

rx1_lro_bytes: 0

rx1_ecn_mark: 0

rx1_removed_vlan_packets: 0

rx1_wqe_err: 0

rx1_mpwqe_filler_cqes: 0

rx1_mpwqe_filler_strides: 0

rx1_buff_alloc_err: 0

rx1_cqe_compress_blks: 0

rx1_cqe_compress_pkts: 0

rx1_page_reuse: 0

rx1_cache_reuse: 0

rx1_cache_full: 0

rx1_cache_empty: 448

rx1_cache_busy: 0

rx1_cache_waive: 0

rx1_congst_umr: 0

rx1_arfs_err: 0

rx1_xdp_tx_xmit: 0

rx1_xdp_tx_full: 0

rx1_xdp_tx_err: 0

rx1_xdp_tx_cqes: 0

rx2_packets: 0

rx2_bytes: 0

rx2_csum_complete: 0

rx2_csum_unnecessary: 0

rx2_csum_unnecessary_inner: 0

rx2_csum_none: 0

rx2_xdp_drop: 0

rx2_xdp_redirect: 0

rx2_lro_packets: 0

rx2_lro_bytes: 0

rx2_ecn_mark: 0

rx2_removed_vlan_packets: 0

rx2_wqe_err: 0

rx2_mpwqe_filler_cqes: 0

rx2_mpwqe_filler_strides: 0

rx2_buff_alloc_err: 0

rx2_cqe_compress_blks: 0

rx2_cqe_compress_pkts: 0

rx2_page_reuse: 0

rx2_cache_reuse: 0

rx2_cache_full: 0

rx2_cache_empty: 448

rx2_cache_busy: 0

rx2_cache_waive: 0

rx2_congst_umr: 0

rx2_arfs_err: 0

rx2_xdp_tx_xmit: 0

rx2_xdp_tx_full: 0

rx2_xdp_tx_err: 0

rx2_xdp_tx_cqes: 0

tx0_packets: 1

tx0_bytes: 60

tx0_tso_packets: 0

tx0_tso_bytes: 0

tx0_tso_inner_packets: 0

tx0_tso_inner_bytes: 0

tx0_csum_partial: 0

tx0_csum_partial_inner: 0

tx0_added_vlan_packets: 0

tx0_nop: 0

tx0_csum_none: 1

tx0_stopped: 0

tx0_dropped: 0

tx0_xmit_more: 0

tx0_recover: 0

tx0_cqes: 1

tx0_wake: 0

tx0_cqe_err: 0

tx1_packets: 5

tx1_bytes: 300

tx1_tso_packets: 0

tx1_tso_bytes: 0

tx1_tso_inner_packets: 0

tx1_tso_inner_bytes: 0

tx1_csum_partial: 0

tx1_csum_partial_inner: 0

tx1_added_vlan_packets: 0

tx1_nop: 0

tx1_csum_none: 5

tx1_stopped: 0

tx1_dropped: 0

tx1_xmit_more: 0

tx1_recover: 0

tx1_cqes: 5

tx1_wake: 0

tx1_cqe_err: 0

tx2_packets: 0

tx2_bytes: 0

tx2_tso_packets: 0

tx2_tso_bytes: 0

tx2_tso_inner_packets: 0

tx2_tso_inner_bytes: 0

tx2_csum_partial: 0

tx2_csum_partial_inner: 0

tx2_added_vlan_packets: 0

tx2_nop: 0

tx2_csum_none: 0

tx2_stopped: 0

tx2_dropped: 0

tx2_xmit_more: 0

tx2_recover: 0

tx2_cqes: 0

tx2_wake: 0

tx2_cqe_err: 0

 

The number of packets dropped due to XDP program XDP_DROP action. these packets are not counted by other software counters. These packets are counted by physical port and vPort counters – supported from kernel 4.9