IPQ4xx Ethernet Analysis

Table of Contents

Device tree

  1. QCA Ethernet DMA Documentation/devicetree/bindings/net/qca-ess.txt
  2. QCA Ethernet Switch ESS-SWITCH is used to forward the packet among LAN, WAN and Host processor. Documentation/devicetree/bindings/arm/msm/ess-switch.txt
  3. QCA Ethernet PHY mode ess-psgmii
  4. MDIO Interface ipq40xx-mdio

The three *_bmp seem to be which physical ports are associated with each of two VLANs:

0x01  0000 0001 -- CPU
0x1e  0001 1110 -- LAN
0x20  0010 0000 -- WAN
edma@c080000 {
        compatible = "qcom,ess-edma";
        reg = <0xc080000 0x8000>;
        qcom,page-mode = <0>;
        qcom,rx_head_buf_size = <1540>;
        qcom,wan_port_id_mask = <0x10>;
        qcom,mdio_supported;
        qcom,phy_mdio_addr = <4>;
        qcom,poll_required = <1>;
        qcom,forced_speed = <1000>;
        qcom,forced_duplex = <1>;

        interrupts = <0 65 1>,
                        <0 66 1>,
                        <0 67 1>,
                        <0 68 1>,
                        <0 69 1>,
                        <0 70 1>,
                        <0 71 1>,
                        <0 72 1>,
                        <0 73 1>,
                        <0 74 1>,
                        <0 75 1>,
                        <0 76 1>,
                        <0 77 1>,
                        <0 78 1>,
                        <0 79 1>,
                        <0 80 1>,
                        <0 240 1>,
                        <0 241 1>,
                        <0 242 1>,
                        <0 243 1>,
                        <0 244 1>,
                        <0 245 1>,
                        <0 246 1>,
                        <0 247 1>,
                        <0 248 1>,
                        <0 249 1>,
                        <0 250 1>,
                        <0 251 1>,
                        <0 252 1>,
                        <0 253 1>,
                        <0 254 1>,
                        <0 255 1>;
        gmac0 {
                local-mac-address = [000000000000];
        };
        gmac1 {
                local-mac-address = [000000000000];
        };
};

ess-switch@c000000 {
        compatible = "qcom,ess-switch";
        reg = <0xc000000 0x80000>; /* 512KB */
        switch_access_mode = "local bus";
        resets = <&gcc ESS_RESET>, <&gcc ESS_MAC1_CLK_DIS>, \
                <&gcc ESS_MAC2_CLK_DIS>, <&gcc ESS_MAC3_CLK_DIS>, \
                <&gcc ESS_MAC4_CLK_DIS>, <&gcc ESS_MAC5_CLK_DIS>;
        reset-names = "ess_rst","ess_mac1_clk_dis", \
                "ess_mac2_clk_dis","ess_mac3_clk_dis", \
                "ess_mac4_clk_dis", "ess_mac5_clk_dis";
        clocks = <&gcc GCC_ESS_CLK>;
        clock-names = "ess_clk";
        switch_cpu_bmp = <0x1>;  /* cpu port bitmap */
        switch_lan_bmp = <0x1e>; /* lan port bitmap */
        switch_wan_bmp = <0x20>; /* wan port bitmap */
};
ess-psgmii@98000 {
        compatible = "qcom,ess-psgmii";
        reg = <0x98000 0x800>; /* 2k */
        psgmii_access_mode = "local bus";
        resets = <&gcc ESS_PSGMII_ARES>;
        reset-names = "psgmii_rst";
};

mdio@90000 {
        #address-cells = <1>;
        #size-cells = <1>;
        compatible = "qcom,ipq40xx-mdio";
        reg = <0x90000 0x64>;
        phy0: ethernet-phy@0 {
                reg = <0>;
        };
        phy1: ethernet-phy@1 {
                reg = <1>;
        };
        phy2: ethernet-phy@2 {
                reg = <2>;
        };
        phy3: ethernet-phy@3 {
                reg = <3>;
        };
        phy4: ethernet-phy@4 {
                reg = <4>;
        };
};

source files

  • /net/ethernet/eth.c
  • /drivers/net/ethernet/qcom/essedma/
  • /drivers/net/phy/mdio-ipq40xx.c

CONFIG_RFS_ACCEL

It macro controls the Accelerated RFS(Receive Flow Steering). And the kernel documents related it are:

And other complementary techniques in the Linux networking stack to increase parallelism and improve performance for multi-processor systems.

  • RFS: Receive Flow Steering
  • RSS: Receive Side Scaling
  • RPS: Receive Packet Steering

Receive Flow Steering

While RPS steers packets solely based on hash, and thus generally provides good load distribution, it does not take into account application locality. This is accomplished by Receive Flow Steering (RFS). The goal of RFS is to increase datacache hitrate by steering kernel processing of packets to the CPU where the application thread consuming the packet is running. RFS relies on the same RPS mechanisms to enqueue packets onto the backlog of another CPU and to wake up that CPU.

Accelerated RFS

Accelerated RFS is to RFS what RSS is to RPS: a hardware-accelerated load balancing mechanism that uses soft state to steer flows based on where the application thread consuming the packets of each flow is running. Accelerated RFS should perform better than RFS since packets are sent directly to a CPU local to the thread consuming the data. The target CPU will either be the same CPU where the application runs, or at least a CPU which is local to the application thread’s CPU in the cache hierarchy.

init module

ret = platform_driver_register(&edma_axi_driver);
static struct platform_driver edma_axi_driver = {
        .driver = {
                .name    = edma_axi_driver_name,
                .owner   = THIS_MODULE,
                .of_match_table = edma_of_mtable,
        },
        .probe    = edma_axi_probe,
        .remove   = edma_axi_remove,
        .shutdown = edma_axi_shutdown,
};

Eth driver probe

#define EDMA_WAN 0
#define EDMA_LAN 1
/* VLAN tag */
#define EDMA_LAN_DEFAULT_VLAN 1
#define EDMA_WAN_DEFAULT_VLAN 2
  1. create net_device for wan and lan port
  2. 16 tx queues, 4 tx queues per core (4 cores)
  3. 4 rx queues, 1 rx queue per core
  4. alloc tx queues, tx rings
  5. alloc rx queues, rx rings
  6. fill adapter and netdev structure
  7. Register the netdevice
  8. register net sysctl table
  9. Disable all 16 Tx and 8 rx irqs
  10. Reset
  11. populate per_core_info, do a napi_Add, request 16 TX irqs, 8 RX irqs, do a napi enable
  12. Configure misc things
  13. Enable All 16 tx and 8 rx irq mask
  14. init statistics timer (edma_statistics_timer)

net device ops

static const struct net_device_ops edma_axi_netdev_ops = {
        .ndo_open               = edma_open,
        .ndo_stop               = edma_close,
        .ndo_start_xmit         = edma_xmit,
        .ndo_set_mac_address    = edma_set_mac_addr,
        .ndo_select_queue       = edma_select_xps_queue,
#ifdef CONFIG_RFS_ACCEL
        .ndo_rx_flow_steer      = edma_rx_flow_steer,
        .ndo_register_rfs_filter = edma_register_rfs_filter,
        .ndo_get_default_vlan_tag = edma_get_default_vlan_tag,
#endif
        .ndo_get_stats          = edma_get_stats,
        .ndo_change_mtu         = edma_change_mtu,
};

control table

// struct ctl_table_header *edma_ctl_table_hdr;
// edma_cinfo->edma_ctl_table_hdr = register_net_sysctl(&init_net, "net/edma", edma_table);
static struct ctl_table edma_table[] = {
        {
                .procname       = "default_lan_tag",
                .data           = &edma_default_ltag,
                .maxlen         = sizeof(int),
                .mode           = 0644,
                .proc_handler   = edma_change_default_lan_vlan
        },
        {
                .procname       = "default_wan_tag",
                .data           = &edma_default_wtag,
                .maxlen         = sizeof(int),
                .mode           = 0644,
                .proc_handler   = edma_change_default_wan_vlan
        },
        {
                .procname       = "weight_assigned_to_queues",
                .data           = &edma_weight_assigned_to_q,
                .maxlen         = sizeof(int),
                .mode           = 0644,
                .proc_handler   = edma_weight_assigned_to_queues
        },
        {
                .procname       = "queue_to_virtual_queue_map",
                .data           = &edma_queue_to_virtual_q,
                .maxlen         = sizeof(int),
                .mode           = 0644,
                .proc_handler   = edma_queue_to_virtual_queue_map
        },
        {
                .procname       = "enable_stp_rstp",
                .data           = &edma_enable_rstp,
                .maxlen         = sizeof(int),
                .mode           = 0644,
                .proc_handler   = edma_enable_stp_rstp
        },
        {
                .procname       = "athr_hdr_eth_type",
                .data           = &edma_athr_hdr_eth_type,
                .maxlen         = sizeof(int),
                .mode           = 0644,
                .proc_handler   = edma_ath_hdr_eth_type
        },
        {}
};

Eth IRQ

  • 16 queues of eth tx
  • 4 queues of eth rx
root@OpenWrt:/# cat /proc/interrupts 
           CPU0       CPU1       CPU2       CPU3       
 20:   36300197     208695    2407496      78552       GIC  20  arch_timer
 35:          0          0          0          0       GIC  35  watchdog bark
 97:          0          0          0          0       GIC  97  edma_eth_tx0
 98:          0          0          0          0       GIC  98  edma_eth_tx1
 99:          0          0        738          0       GIC  99  edma_eth_tx2
100:          0          0          0          0       GIC 100  edma_eth_tx3
101:          0          0          0          0       GIC 101  edma_eth_tx4
102:          0          0          0          0       GIC 102  edma_eth_tx5
103:          0          0          0        797       GIC 103  edma_eth_tx6
104:          0          0          0          6       GIC 104  edma_eth_tx7
105:          0          0          0          0       GIC 105  edma_eth_tx8
106:          0          0          0          0       GIC 106  edma_eth_tx9
107:         73          0          0          0       GIC 107  edma_eth_tx10
108:          0          0          0          0       GIC 108  edma_eth_tx11
109:          0          0          0          0       GIC 109  edma_eth_tx12
110:          0          0          0          0       GIC 110  edma_eth_tx13
111:          0          9          0          0       GIC 111  edma_eth_tx14
112:          0          0          0          0       GIC 112  edma_eth_tx15
127:      52330          0          0          0       GIC 127  78b5000.spi
129:         24          0          0          0       GIC 129  i2c-msm-v2-irq
133:          0          0          0          0       GIC 133  sps
139:       1148          0          0          0       GIC 139  msm_serial_hsl0
155:          0          0          0          0       GIC 155  mmc0
164:          0          0          0          0       GIC 164  xhci-hcd:usb1
168:         29          0          0          0       GIC 168  xhci-hcd:usb3
173:          0          0          0          0       GIC 173  int_msi
174:       2103      48023          0          0       GIC 174  wifi2
183:          0          0          0          0       GIC 183  int_pls_link_down
200:       3367          0    2982303          0       GIC 200  wifi0
201:       3244          0          0      24028       GIC 201  wifi1
239:          0          0          0          0       GIC 239  sps
270:      15220          0          0          0       GIC 270  sps
272:      23697          0          0          0       GIC 272  edma_eth_rx0
274:          0       5607          0          0       GIC 274  edma_eth_rx2
276:          0          0       5424          0       GIC 276  edma_eth_rx4
278:          0          0          0       5911       GIC 278  edma_eth_rx6
306:          0          0          0          0   msmgpio  18  wps
310:          0          0          0          0   msmgpio  22  7824900.sdhci cd
IPI0:          0          0          0          0  CPU wakeup interrupts
IPI1:          0          0          0          0  Timer broadcast interrupts
IPI2:      63422      52002      19887      46408  Rescheduling interrupts
IPI3:          5          7          7          5  Function call interrupts
IPI4:          4          2          4          2  Single function call interrupts
IPI5:          0          0          0          0  CPU stop interrupts
IPI6:      12161      14746       7912       6997  IRQ work interrupts
IPI7:          0          0          0          0  completion interrupts
Err:          0

cc


Author: Shi Shougang

Created: 2019-04-03 Wed 10:53

Emacs 24.3.1 (Org mode 8.2.10)

Validate