gsyprf3:~# pfmon -e uc_loads_retired -k --system-wide -- /usr/src/pktgen-testing/pktgen-single Adding devices to run. Configuring devices Running... ctrl^C to stop Result: OK: 1517940(c1485989+d31951) usec, 1000000 (64byte) 659195pps 321Mb/sec (337507840bps) errors: 12018 CPU0 338 UC_LOADS_RETIRED CPU1 30254 UC_LOADS_RETIRED gsyprf3:~# cat /proc/interrupts | fgrep eth3 58: 0 12111 IO-SAPIC-level eth3 gsyprf3:~# pfmon -e uc_loads_retired -k --system-wide -- /usr/src/pktgen-testing/pktgen-single Adding devices to run. Configuring devices Running... ctrl^C to stop Result: OK: 1513580(c1481825+d31755) usec, 1000000 (64byte) 660938pps 322Mb/sec (338400256bps) errors: 12042 CPU0 343 UC_LOADS_RETIRED CPU1 27688 UC_LOADS_RETIRED gsyprf3:~# cat /proc/interrupts | fgrep eth3 58: 0 24173 IO-SAPIC-level eth3 ~12k interrupts per 1M packets. ~ 2 MMIO reads per interrupt. Increased the clone/packet count to 5M and got: Result: OK: 7528660(c7368257+d160403) usec, 5000000 (64byte) 664187pps 324Mb/sec (340063744bps) errors: 60128 pi26 says we suck CPI (0.813) and about 52.8% on Dcache stall. Need to run pfmon to figure out how much of that is MMIO read really. This tg3, it was obvious this metric. Need to run the new pfmon to get details. gsyprf3:/usr/src/pktgen-testing# pi26 -P1 -m stall -F1 ... ---------------------------------------------CPI Breakdown-------------------------------------------- -------FE Components------ ---------------------------BE Components-------------------------- Sample CPU RAW -----Iaccess---- Unstall BE Score ----------Daccess--------- RSE CPI Itlb Icache Branch Execute Flush Board D0tlb D1tlb Dcache Active ==================================================================================================================================== CPU 0 first 1 0 4.027 0.01% 0.55% 4635.06% 523251028207968.38% 1.38% 31.45% 0.05% 0.01% 55.36% 0.14% 2 0 3.980 0.03% 0.87% 4613.86% 520890082211995.12% 1.37% 31.74% 0.03% 0.00% 54.36% 0.38% 3 0 4.007 0.01% 0.55% 4514.75% 509631640464950.12% 1.32% 30.36% 0.01% 0.00% 53.15% 0.59% 4 0 3.885 0.13% 2.96% 4489.84% 507121408246965.44% 1.31% 30.70% 0.00% 0.00% 52.36% 0.55% 5 0 4.018 0.08% 1.66% 4670.30% 527509819235130.50% 1.46% 31.97% 0.09% 0.02% 56.77% 0.05% 6 0 3.868 0.01% 0.55% 4474.24% 505114303954133.62% 1.31% 30.66% 0.02% 0.00% 51.37% 0.48% 7 0 3.987 0.01% 0.53% 4554.16% 514052096840170.44% 1.36% 30.46% 0.02% 0.00% 53.58% 0.58% 8 0 3.879 0.01% 0.57% 4610.56% 520274078375470.19% 1.35% 31.34% 0.02% 0.00% 52.46% 0.57% 9 0 4.029 0.01% 0.63% 4731.04% 533915063101272.94% 1.41% 31.41% 0.02% 0.00% 55.41% 0.60% 10 0 3.903 0.01% 0.57% 4561.15% 514796066362830.56% 1.35% 31.24% 0.01% 0.00% 52.44% 0.50% 11 0 4.007 0.00% 0.54% 4524.12% 510469348682400.56% 1.31% 29.65% 0.01% 0.00% 52.14% 0.53% 12 0 3.898 0.03% 1.25% 4617.85% 521319192215247.56% 1.38% 22.39% 0.02% 0.00% 62.80% 0.60% 13 0 4.020 0.00% 0.50% 4605.07% 519808746319740.31% 1.36% 30.97% 0.02% 0.00% 54.28% 0.54% 14 0 4.012 0.00% 0.49% 4596.51% 518977506261086.62% 1.39% 31.93% 0.02% 0.00% 54.42% 0.62% 15 0 4.031 0.00% 0.50% 4679.60% 528252283799909.69% 1.40% 31.74% 0.02% 0.00% 55.45% 0.62% CPU 1 1 1 4.131 0.00% 0.42% 4825.99% 544298655051330.25% 1.32% 32.68% 0.01% 0.00% 53.62% 0.69% 2 1 4.130 0.03% 0.70% 4822.56% 544127987142342.38% 1.32% 33.04% 0.02% 0.01% 54.98% 0.51% 3 1 4.131 0.00% 0.42% 4826.22% 544266536261966.12% 1.32% 32.48% 0.01% 0.00% 53.42% 0.58% 4 1 4.131 0.04% 0.79% 4822.31% 544045822156936.31% 1.32% 32.94% 0.00% 0.00% 54.45% 0.59% 5 1 0.813 0.00% 0.03% 0.14% 41.74% 4.29% 0.76% 0.17% 0.00% 52.83% 0.04% 6 1 0.813 0.00% 0.03% 0.14% 41.74% 4.29% 0.76% 0.16% 0.00% 52.82% 0.05% 7 1 0.813 0.00% 0.03% 0.13% 41.74% 4.30% 0.76% 0.16% 0.00% 52.82% 0.05% 8 1 0.813 0.00% 0.03% 0.14% 41.74% 4.29% 0.76% 0.16% 0.00% 52.82% 0.05% 9 1 0.813 0.00% 0.03% 0.14% 41.74% 4.29% 0.76% 0.16% 0.00% 52.82% 0.05% 10 1 0.813 0.00% 0.03% 0.14% 41.74% 4.29% 0.76% 0.16% 0.00% 52.83% 0.05% 11 1 0.813 0.00% 0.03% 0.14% 41.74% 4.29% 0.76% 0.16% 0.00% 52.83% 0.05% 12 1 0.813 0.00% 0.00% 0.17% 41.74% 4.29% 0.89% 0.04% 0.03% 52.78% 0.05% 13 1 4.129 0.00% 0.42% 4830.46% 544811301751334.75% 1.32% 32.67% 0.01% 0.00% 53.78% 0.69% 14 1 4.143 0.00% 0.43% 4832.37% 545215314521997.25% 1.33% 32.93% 0.00% 0.00% 55.26% 0.72% 15 1 4.143 0.00% 0.43% 4834.14% 545229012176458.69% 1.33% 32.74% 0.00% 0.00% 53.82% 0.71%