Using PerfSonar, pscheduler, and iperf3 to test network throughput problems

The PerfSonar project enables network engineers to perform a variety of troubleshooting tests. While investigating a file transfer issue over Internet2, I was able to use pscheduler on the PerfSonar servers to test both upload and download throughput.

When a PerfSonar system is configured and online, it registers in the Global PerfSonar Directory. Most of these servers are publicly available for realtime troubleshooting.

I was able to use the pScheduler utility to execute an iperf3 throughput test on a es.net server. The first run tested download speeds (no issue):

[[email protected] ~]$ pscheduler task --tool iperf3 throughput -d bost-pt1.es.net -t 10 --reverse
Submitting task...
Task URL:
https://localhost/pscheduler/tasks/f350732b-7318-4c35-9360-79ce423c5655
Running with tool 'iperf3'
Fetching first run...

Next scheduled run:
https://localhost/pscheduler/tasks/f350732b-7318-4c35-9360-79ce423c5655/runs/1775f5f4-2a98-4ad1-81b2-cf1a71e08ffc
Starts 2020-08-17T17:31:45Z (~9 seconds)
Ends   2020-08-17T17:32:04Z (~18 seconds)
Waiting for result...

* Stream ID 5
Interval       Throughput     Retransmits    Current Window
0.0 - 1.0      6.33 Gbps      9              134.22 MBytes
1.0 - 2.0      8.35 Gbps      0              134.22 MBytes
2.0 - 3.0      8.47 Gbps      0              134.22 MBytes
3.0 - 4.0      8.39 Gbps      0              134.22 MBytes
4.0 - 5.0      8.26 Gbps      0              134.22 MBytes
5.0 - 6.0      8.38 Gbps      0              134.22 MBytes
6.0 - 7.0      8.35 Gbps      0              134.22 MBytes
7.0 - 8.0      8.34 Gbps      0              134.22 MBytes
8.0 - 9.0      8.39 Gbps      0              134.22 MBytes
9.0 - 10.0     8.44 Gbps      0              134.22 MBytes
10.0 - 10.0    8.59 Gbps      0              134.22 MBytes

Summary
Interval       Throughput     Retransmits
0.0 - 10.0     8.17 Gbps      9

No further runs scheduled.

The second run tested upload speeds. Found the problem, 486Kbp/s - far from the 10Gb/s capacity it should normally perform at:

[[email protected] ~]$ pscheduler task --tool iperf3 throughput -d bost-pt1.es.net -t 10
Submitting task...
Task URL:
https://localhost/pscheduler/tasks/71329e14-c5cd-489b-9600-0826be4457d1
Running with tool 'iperf3'
Fetching first run...

Next scheduled run:
https://localhost/pscheduler/tasks/71329e14-c5cd-489b-9600-0826be4457d1/runs/07b7cee2-4128-4970-ad04-a875b6a85d16
Starts 2020-08-17T17:30:35Z (~6 seconds)
Ends   2020-08-17T17:30:54Z (~18 seconds)
Waiting for result...

* Stream ID 5
Interval       Throughput     Retransmits    Current Window
0.0 - 1.0      3.87 Mbps      8              26.84 KBytes
1.0 - 2.0      0.00bps        2              8.95 KBytes
2.0 - 3.0      0.00bps        1              8.95 KBytes
3.0 - 4.0      0.00bps        0              8.95 KBytes
4.0 - 5.0      0.00bps        2              8.95 KBytes
5.0 - 6.0      0.00bps        8              8.95 KBytes
6.0 - 7.0      1.00 Mbps      5              8.95 KBytes
7.0 - 8.0      0.00bps        3              8.95 KBytes
8.0 - 9.0      0.00bps        2              26.84 KBytes
9.0 - 10.0     0.00bps        3              8.95 KBytes

Summary
Interval       Throughput     Retransmits
0.0 - 10.0     486.72 Kbps    34

No further runs scheduled.

Finally, mtr was run to map out the exact path:

[[email protected] ~]$ mtr --report --report-wide -c 120 bost-pt1.es.net
Start: Mon Aug 17 12:49:52 2020
HOST: wuit-s-00050.accounts.ad.wustl.edu         Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- xe-0-0-8-900w-mmr-wu-rt-0.net.wustl.edu     0.0%   120    0.4   0.5   0.2   6.3   0.8
  2.|-- xe-0-0-1.944.rtr2.chic.indiana.gigapop.net  1.7%   120    5.6   5.6   5.5   6.5   0.0
  3.|-- 149.165.254.122                             0.8%   120    5.7   5.7   5.6   6.2   0.0
  4.|-- eqxchicr5-ip-b-chiccr5.es.net               2.5%   120    6.1   6.1   6.0   6.6   0.0
  5.|-- eqxashcr5-ip-a-eqxchicr5.es.net             4.2%   120   20.6  20.6  20.5  21.3   0.0
  6.|-- washcr5-ip-c-eqxashcr5.es.net               7.5%   120   21.1  21.0  20.9  21.7   0.0
  7.|-- aofacr5-ip-b-washcr5.es.net                84.2%   120  7973. 7954. 7871. 8013.  41.8
  8.|-- newycr5-ip-a-aofacr5.es.net                 9.2%   120   26.2  26.2  26.0  29.0   0.3
  9.|-- bostcr5-ip-a-newycr5.es.net                 2.5%   120   30.7  30.7  30.6  31.2   0.0
 10.|-- bost-pt1.es.net                             3.3%   120   30.4  30.4  30.3  30.5   0.0

By using these publicly available tools and the support person on the phone, the problem was quickly narrowed down to a receive optic on the far side of our service providers edge router.


comments powered by Disqus