Author Topic: Expected 10gbe speed?  (Read 6063 times)

2020-06-03, 19:03:56

danio1011

  • Active Users
  • **
  • Posts: 361
    • View Profile
Hi all,

We just moved to 10gbe.  It's super simple right now, a Belkin switch and some short 6a cables.  Right now I'm seeing copy speeds around 450-500 MB a second.  This is copying large files like CXRs just as a test.

When I copy these same files locally I get about 1.2GB a second.  Shouldn't I be seeing higher than 450-500 over the network, something closer to 1GB?  There's no other load and both computers are otherwise idle, and everything is nVME. 

Right now it's just using windows 10 shared folders, so I'm wondering if there's some sort of caching or encryption going on.

Cheers and thanks for any input!
Daniel

2020-06-05, 11:46:08
Reply #1

1equals2

  • Active Users
  • **
  • Posts: 77
    • View Profile
Couple of things to check:
- drivers and firmwares - updating them to latest and finding them from the "official" site not aways works. For instance despite my lan cards are branded as Asus, and Asus does have drivers, I am currently using the Aquantia ones, from another website.
- try with other cables as well, cat 6a or more. Have heard a lot of stories about bad cables which are claimed to sustain higher speed but deliver poor performance
- Try checking you switch and lan card settings. That's crucial. For instance had to manually adjust some settings in Asus XG-C100C to boost performance and have it somehow stable - Jumbo packet is disabled here  and boost in speed is around 50%.
- Check Your Belkin settings and play with some of them  if the switch is manageble or try looking up for any complaints about lower speeds on Belkin forums....

As a side story, a friend of mine who has access to bigger market, when on the 10gbe route, but with local NAS- Qnap.
After exchanging couple of devices - mostly lan cards, and proper drivers, he managed to sustain almost same speed on his 10gb network with fiber optic, regardless of having mixed software environment - win 7, win 10, linux, etc.

Good luck!
Other places to find the work of Angel Kostov
https://www.instagram.com/1equals2/
https://www.behance.net/ARCHO

2020-06-07, 00:10:18
Reply #2

danio1011

  • Active Users
  • **
  • Posts: 361
    • View Profile
Couple of things to check:
- drivers and firmwares - updating them to latest and finding them from the "official" site not aways works. For instance despite my lan cards are branded as Asus, and Asus does have drivers, I am currently using the Aquantia ones, from another website.
- try with other cables as well, cat 6a or more. Have heard a lot of stories about bad cables which are claimed to sustain higher speed but deliver poor performance
- Try checking you switch and lan card settings. That's crucial. For instance had to manually adjust some settings in Asus XG-C100C to boost performance and have it somehow stable - Jumbo packet is disabled here  and boost in speed is around 50%.
- Check Your Belkin settings and play with some of them  if the switch is manageble or try looking up for any complaints about lower speeds on Belkin forums....

As a side story, a friend of mine who has access to bigger market, when on the 10gbe route, but with local NAS- Qnap.
After exchanging couple of devices - mostly lan cards, and proper drivers, he managed to sustain almost same speed on his 10gb network with fiber optic, regardless of having mixed software environment - win 7, win 10, linux, etc.

Good luck!

Wow, thank you for the checklist of ideas!  That's extremely helpful.  I'll start ticking things off and see if my transfer rates vary.  I need to find a good benchmarking utility so I'm not just watching windows explorer, but even still I would imagine those speeds should match local transfer speeds.

Thanks again for the helpful response.
Daniel

2020-06-08, 00:19:27
Reply #3

danio1011

  • Active Users
  • **
  • Posts: 361
    • View Profile
Couple of things to check:
- drivers and firmwares - updating them to latest and finding them from the "official" site not aways works. For instance despite my lan cards are branded as Asus, and Asus does have drivers, I am currently using the Aquantia ones, from another website.
- try with other cables as well, cat 6a or more. Have heard a lot of stories about bad cables which are claimed to sustain higher speed but deliver poor performance
- Try checking you switch and lan card settings. That's crucial. For instance had to manually adjust some settings in Asus XG-C100C to boost performance and have it somehow stable - Jumbo packet is disabled here  and boost in speed is around 50%.
- Check Your Belkin settings and play with some of them  if the switch is manageble or try looking up for any complaints about lower speeds on Belkin forums....

As a side story, a friend of mine who has access to bigger market, when on the 10gbe route, but with local NAS- Qnap.
After exchanging couple of devices - mostly lan cards, and proper drivers, he managed to sustain almost same speed on his 10gb network with fiber optic, regardless of having mixed software environment - win 7, win 10, linux, etc.

Good luck!

Just a quick follow up question:  You mention jumbo packets are disabled?  Normally I would think Jumbo is enabled and MTU is set to 9000 for 10gbe right?  Have you had better luck with it disabled?

Edit:  Right now I've had a massive jump in speed by disabling 'Recv Segment Coalescing (IPv4)' on my Aquantia chip which came with the Asus.  It went from looking like it was dropping packets like crazy to actually behaving well.  But now one computer sends at 10gbe and one sends at more or less 5gbe.  If you flip to the other computer and run the same benchmark then you get the mirror image (write is 10, read is 5...the other is the opposite).  So both are pretty fast, but one seems to be about half the speed it should be.  Wondering if there might be a system variable in play like encryption or something...as in, perhaps the explanation is Windows is setup differently rather than the NIC is behaving differently.  I've been moving cables around but am going to check that one more time.
« Last Edit: 2020-06-08, 07:08:54 by danio1011 »

2020-06-09, 07:47:11
Reply #4

1equals2

  • Active Users
  • **
  • Posts: 77
    • View Profile
Hi there,

The Aquantia chip does the work but it is far from the optimal solution, as it tends to overheat, due to the passive cooling, which is not enough.
I personally ended playing so much with settings that eventually got lost inside all the variables, so with each change got different results.  Make sure You restart PC, after each change.
The option You mentioned is Turned On here, everywhere - have two Asus cards(aka Aquantia) and the main server has  integrated intel X552/X557-AT 10GBASE-T.
To maintain the speeds over the network what worked for me was to increase "Transmit Buffers" to the max supported number available and disable Jumbo Packets , everywhere.
Strangely, disabling  the TCP/IP v6 Internet Protocol  from the "Ethernet Porperties" inside win 10 helped a bit as well.

Guy whom found on Amazon, suggested having the following options activated/disbaled:

ARP off
Energy-Efficient Ethernet off
Flow control off
Jumbo Packets off
Large Send Offload (all 3) off
Maximum number of RSS Queues - 8 queues (only set to number of threads of your processor or lower as this puts more load on CPU)
NS Offload off (only applicable to some Aquantia chipsets)
Transmit Buffers 8184

In regards to the NVNe drives, can not suggest much. I am still using old-school high capacity SATA Hdd in raid10 here, as PC here does not support NVMe.
Do you have by chance the "Trim " option enabled  on any of the disks ( if "Samsung")?
Is any of the disks "Evo" by chance? Test it with a disk benchmarking program and then compare the results with the Network benchmarking program as a start.
Last thing which could think of - bit obvious but try checking if You have the disk plugged into  PCIe Gen 3.0 x4.

Good luck!


 



Couple of things to check:
- drivers and firmwares - updating them to latest and finding them from the "official" site not aways works. For instance despite my lan cards are branded as Asus, and Asus does have drivers, I am currently using the Aquantia ones, from another website.
- try with other cables as well, cat 6a or more. Have heard a lot of stories about bad cables which are claimed to sustain higher speed but deliver poor performance
- Try checking you switch and lan card settings. That's crucial. For instance had to manually adjust some settings in Asus XG-C100C to boost performance and have it somehow stable - Jumbo packet is disabled here  and boost in speed is around 50%.
- Check Your Belkin settings and play with some of them  if the switch is manageble or try looking up for any complaints about lower speeds on Belkin forums....

As a side story, a friend of mine who has access to bigger market, when on the 10gbe route, but with local NAS- Qnap.
After exchanging couple of devices - mostly lan cards, and proper drivers, he managed to sustain almost same speed on his 10gb network with fiber optic, regardless of having mixed software environment - win 7, win 10, linux, etc.

Good luck!

Just a quick follow up question:  You mention jumbo packets are disabled?  Normally I would think Jumbo is enabled and MTU is set to 9000 for 10gbe right?  Have you had better luck with it disabled?

Edit:  Right now I've had a massive jump in speed by disabling 'Recv Segment Coalescing (IPv4)' on my Aquantia chip which came with the Asus.  It went from looking like it was dropping packets like crazy to actually behaving well.  But now one computer sends at 10gbe and one sends at more or less 5gbe.  If you flip to the other computer and run the same benchmark then you get the mirror image (write is 10, read is 5...the other is the opposite).  So both are pretty fast, but one seems to be about half the speed it should be.  Wondering if there might be a system variable in play like encryption or something...as in, perhaps the explanation is Windows is setup differently rather than the NIC is behaving differently.  I've been moving cables around but am going to check that one more time.
« Last Edit: 2020-06-09, 07:58:13 by 1equals2 »
Other places to find the work of Angel Kostov
https://www.instagram.com/1equals2/
https://www.behance.net/ARCHO

2020-06-10, 23:00:58
Reply #5

danio1011

  • Active Users
  • **
  • Posts: 361
    • View Profile
Thank you for the additional list of options and things to try.  Right now the biggest change that has helped me is to turn off Receive Side Coalescing on the Aquantia chip (Asus board.)  That had a massive impact on my Aquantia chip.  Nothing else has had any impact, I tried all the ones you mentioned and have tried jumbo on\off (including on switch), etc.  My disk drives all come out to about 2000-3000 MB/s when tested locally with the same software so I don't think they're the bottleneck.  The network right now is 900MB one way (good!) and 400MB the other way.  And if I goto the other computer it's always the mirror image.  I even flashed the firmware on the Aquantia and am running the very latest drivers, all yielding zero difference.

Long story short, I'm starting to suspect the Intel X550 10gbe chip on my Gigabyte Aorus xTreme.  I'm thinking I'll buy an independent NIC, probably with an Aquantia chipset on it that would eventually be compatible with my NAS (Synology 1819+ that has the expansion slot) and use it as a test NIC, moving it from machine to machine.  Then either buy another one for the NAS later if it solves the problem, or if it doesn't solve the problem it's no loss because I'll move it to the NAS immediately.  Right now my guess is the X550 just isn't capable of transmitting at 10gbe because that's what every single test is showing me right now.  Almost seems like a duplex problem, but it happens even when just doing simple copies...as in, I'm not sending in the other direction at the same time.  Unless I'm misunderstanding duplex.

Anyway, cheers and thanks again.  Didn't know 10gbe would be such a wild west!

2020-06-11, 12:01:20
Reply #6

clemens_at

  • Active Users
  • **
  • Posts: 142
    • View Profile
also go try iperf: https://iperf.fr/
Its a tool to measure network bandwidth.
With this you should be able to rule out networking bottlenecks pretty easily.
good luck.




2020-06-17, 15:39:57
Reply #7

Juraj

  • Moderator
  • Active Users
  • ***
  • Posts: 4743
    • View Profile
    • studio website
Intel X550 is in fact vastly superior to Aquantia, I have a lot of both chips, both integrated to boards (Asus with Aquantia chips, Aorus with Intel), and as separate add-on cards.
They both seriously overheat so take that into account :- ). The X550 card I have in my fileserver, I strapped a Noctua fan on top.

I have no issue attaining 1.2GB/s under all conditions.

I had lot of issues setting up Aquantia chips, the Rec Coalescing I know was recent issue on Macs, but I also have something else (Checksum offload?) that helped.
I never had to adjust packets or anything though.

Yeah 10Gbit can be issue because of poor driver support in Windows. But with Intel NICs I never had a single issue.
Please follow my new Instagram for latest projects, tips&tricks, short video tutorials and free models
Behance  Probably best updated portfolio of my work
lysfaere.com Please check the new stuff!

2020-06-19, 17:51:27
Reply #8

danio1011

  • Active Users
  • **
  • Posts: 361
    • View Profile
Thanks Juraj and Clemens!  With my new adjusted settings iperf3 gives me good results if I'm understanding them correctly.  But file transfers in one direction still start at like 500, then droop to 400...300...back to 400... etc.  Meanwhile local file transferring those exact same files might be more like 2000-3000.  So I'm almost starting to wonder if it's something with Windows file sharing setup.  Everything has the same gateway, subnet mask, no ip conflict, IPV6 is disabled, etc.

Now that I know x550 is superior I might get one of those off Amazon and start moving that to various machines just to see if perhaps one of the controllers is just inherently slow or overheating or something.

Cheers all,
Daniel

2020-09-04, 20:49:08
Reply #9

danio1011

  • Active Users
  • **
  • Posts: 361
    • View Profile
Hi Juraj,

I just got an Intel X550-T2 directly from Amazon (sold\shipped by Amazon).  They listed it as made in Mexico but this clearly says 'Made in Thailand' and has a mfgr date of 6/2020. 

It has a sticker to 'VerifyIntel.bradyid.com' and when I enter the code and confirms it's made in Thailand, but the Mac ID doesn't match.  The first few digits match, but not the last half.

Think it's possibly a fake or am I being neurotic?  I guess two things can be true at once...

Thanks!
Daniel