Why is something so slow over a VPN connection?
I’ve been asked many times why some applications are slow over a VPN connection.
In order to understand why this is the case, we must first understand what type of slow is normal, and what isn’t.
The most popular problem with VPN connections are MTU issues. Try a ping -l 1500 internal-ip. If that doesn’t work, you have an MTU issue.
The next step is to test the raw speed – copying a file with FTP or SCP to and from a server. Is the speed okay? If the speed is according to the lines used, your problem probably isn’t with the VPN connection.
The main problem with VPN connections is latency. The average latency for a roundtrip from ADSL to ADSL here in Switzerland is about 80ms. This is a lot of time, especially if you’re taking multiple round trips. If we look at other connectivity methods like UMTS (with about 500ms Latency), it gets a lot worse. 10 Roundtrips für ADSL are a second, but 10 Roundtrips for UMTS are 5 seconds.
You can use Wireshark to look at the traffic on the interface in order to see if the application that is slow uses multiple round trips to achieve it’s goal.
There are several well known applications that have problem with using multiple roundtrips:
The most popular one is SMB – it uses 5 – 10 Roundtrips for a directory list under certain circumstances. This means 0.5 – 1 second wasted on network latency (and this doesn’t even include the transmission time). The solution for this problem varies – if you need access from a site, use DFS-R and a server local to that site, if you want road warriors to have faster access, use Sharepoint.
Another thing causing many problems are ERP applications built onto the “Fat Client” design principle (Business Logic in Client), which queries the database directly – this ensures many, many roundtrips for every bit of information displayed. (Note that DIAS-iS doesn’t have this problem because it’s a Thin Client application, with the Business Logic in the server). Another remedy is an ERP application which can have servers in multiple sites, but this is probably not SMB software anymore.
There’s not really a solution for this, because it means switching ERP applications, or getting your software provider to start supporting high latency links.. A bit drastic for latency problems.
An interesting application regarding this is Outlook – Outlook is a hybrid. With Cached Exchange mode enabled, it behaves like a Thin Client, because it has a local replica of the database. So, the solution for Outlook is to always use Cached Exchange mode – a good idea anyway because it reduces load on the Exchange server.
Ihsan Dogan:
I’ve got over ADSL a latency of 40ms and over UMTS 200-400ms (depends on the signal quality). How do you get to thos values?
21. March, 2007, 21:09Lukas Beeler:
They are ADSL/ADSL or ADSL/UMTS roundtrip values – not ADSL-Well Connected site.
They also assume a connection which is already in use, not an empty one.
22. March, 2007, 07:47Jakob Magun:
Can you comment on the latency of SDSL, superSDSL compared do ADSL, VSDL? We currently thinking on moving from ADSL to VDSL or SDSL. Our main application used over the will be a VPN.
26. May, 2007, 07:37Lukas Beeler:
We have no customers with superSDSL or VDSL links, so i can only comment on SDSL.
SDSL seems to be a bit faster, and if both SDSL links belong to the same ISP, latency is around 30ms. This is a lot better than ADSL, and even fat client applications should work with that.
But this could be just for those links (Line Length, Equipment, Backbone are all factors here), i really lack enough input there because we don’t have many customers with SDSL links.
26. May, 2007, 07:45James:
Hi, thank your for the information.
10. October, 2007, 15:07Can you say a little more about MTU issue ? I understand what is MTU but I don’t understand how I can modify the MTU when I’m on VPN over UMTS.
thx
Pravin:
Hi,
I’m an IT manager of a small company in Mauritius and we are connecting via VPN to a company in Belgium. The company has asked to installed Oracle and also a GIS software. We are using an ADSL 1MB (UTM) and ping gives 500ms, is that normal because we are having a lot of trouble working, PC freezing, Oracle timeout.
Thx for your help.
6. June, 2008, 09:16John:
We’ve got remote offices and 500ms ping can ruin remote applications. The first thing I realized when troubleshooting a particular case was that his upload speed was 125kbps which is really impossible for any corporate vpn with M$ products.
Please elaborate on how MTU can improve vpn service? I’m using openvpn as an alternative but I’m not really sure if the problem is in the vpn software or the actual connection (could a particular router hardware vpn brand improve performance?)
9. January, 2009, 13:18Simon Bannister:
Can you explain a little more about the MTU issue as well. Tried pinging and did not get a response but in normal operation I did.
Thanks in advance.
S.
17. November, 2009, 16:36Jeff Kamungu:
hi,
8. December, 2009, 14:19according to my internet provider am on a corporate line with a minimum speed of 64kbps.can vpn work properly with such a speed?
VPN Latency:
[...] Originally Posted by nico34 can anybody help me, or give me the gonfiguration to reduce my vpn latency, we are testing our new vpn connection and the latency gets up to 1000 to 2000 everytime we try to transfer files! I really need your help Does that help? http://projectdream.org/wordpress/20…pn-connection/ [...]
11. April, 2010, 04:17Alexandre Gauthier:
Hi, just landed here while searching for a solution to a rather weird OpenVPN issue I have on Windows 7 x64, which has been driving me nuts. The answer is not in here, sadly, but since the OpenVPN documentation is uh… well, scarce, ever since they went commercial, I thought I might be able to shed some light on some questions raised in the comments so far.
If you’re using OpenVPN and are having terrible latency issues, I’d suspect the transport.
OpenVPN can be configured to tunnel over TCP or UDP. In fact this also applies to any VPN solution that involves TCP as the transport layer. OpenVPS AS apparently defaults to TCP in the generated client configuration files, or at least in my case it did. Encapsulating TCP over TCP usually results in horrible performance the second some mild packet loss starts to occur — or worse, was already occurring before you established a TCP connection through the VPN — due to TCP’s congestion control mechanisms. In fact this is most likely to occur whenever you really start to /use/ the VPN instead of just pinging hosts and going OH IT WORKS, MISSION ACCOMPLISHED. (I’d hardly call 30 bytes in an icmp echo request a stress test).
Since the encapsulated TCP (your vpn link) and the transport TCP (your actual connection) are both blissfully unaware of each other, there may (and in fact almost always will) be some… well, dimorphism between them, in regard to control timers and retransmission rates.
It sort of creates a chain reaction where the moment your transport connection or some upstream router coughs a bit mid sentence or trips on its shoelace — woops — and starts dropping packets. TCP notices the sudden change in rate and increases its timeout values as to not suddenly sever the connection. Which is the entire point of TCP — ensuring delivery at all costs.
Meanwhile the encapsulated TCP, who has no clue that it is being forced into a payload against its will, just notices a sudden eerie silence where its not receiving jack squat, since the transport TCP is waiting /longer/ than before to adapt to the packet loss. This results in the encapsulated TCP /also/ worrying about the well being of its ACKs and increasing its timeout values. However, it will most likely lower them slightly /less/ so than the transport TCP, because, after all, that connection was just established from its perspective, and it is unaware of the fact that your transport TCP has been downloading your mail for quite a while before you moved on to downloading all these files off the CIFS server at work.
This effectively means that the encapsulated TCP layer will end up queuing retransmissions faster than the transport layer is willing to retransmit them, which makes the encapsulated layer increase its timeout values even more so every time, which results in everything stalling rather abruptly ever five seconds. It often just snowballs up from there and eventually results in you throwing your laptop against the wall.
Long story short — if you’re using TCP as a transport protocol for your VPN, switch over to UDP.
UDP is stateless, and does no error checking. It’s like sending a postcard instead of a registered letter. But that doesn’t matter, since the encapsulated TCP inside will take care of all that. This results in a tremendous performance boost.
Now, regarding the MTU (which as you probably know is the maximum length of a data unit a protocol can send in one shot, without fragmenting), default MTU for Ethernet is 1500. However you must account for the overhead encapsulation generates. VPNs most often send their transport packets with the do not fragment bit set, it makes things simpler for the encapsulated TCP, and prevents needless retransmission. This means that the MTU of your virtual interface must be smaller than the MTU of your ethernet card, to make room for the outer packet. In fact, optimally, it should be exactly
Max Payload Size without headers – encryption overhead = target MTU.
*Most* VPNs account for this and set it dynamically. Depending on the OS and available system calls, they might use Path MTU Discovery to figure it out and enforce the DNF bit. However some misinformed firwall admins who think ICMP is dangerous sometimes disable that, which can result in horrors. That or you are running TCP/IP over some exotic layer 2 physical media, like carrier pigeon, firewire, CSU/DSU Serial or /some/ DSL lines and something is configured wrong. If you just went “Ooooh” and suddenly realized this explains why you couldn’t reach /some/ sites, well, yes. Fix your MTU. You’re not alone, we all went there once or twice. Less so today, because of PMTU. But as a general rule, if you can’t send 1500 bytes with ping through your link, and all the computers are ethernet based, things will probably get ugly unless you configure it properly.
So, often, tweaking the MTU values for the VPN interface and/or in the VPN itself (this is very software-dependent) can bring a tremendous performance boost as well. Larger MTU is more efficient — but if your link is unreliable, a single bit that is wrong in a packet means the entire thing must be sent all over again. Packet loss also affects this for the same reason. In such cases, a lower MTU would bring out a better performance.
The main problem usually lands when you start working with various ATM encapsulated transports (namely xDSL lines) that use various breeds of PPPoE, PPPoA or (rarely) something else entirely, which are all various layers of encapsulation (packets within datgrams within packets within frames within…) which all result in some overhead and a rather exotic looking MTU, often closer to 1400 than 1500 to account for the padding. Normally, good routers handle all of this automagically for machines behind NATs, but it still results in some overhead. If you add the VPN on top of that, it is often going to be slow — which brings this blog post and the questions asked in the comments together.
Modern, non-crappy VPNs handle this dynamically with PMTU, to determine what’s the maximum transmission unit they can send without it barfing all over the place, but if it’s broken somehow, or your vpn doesn’t do PMTU, and your weird ISP specific software PPPoE DSL link has a MTU of 1452 while the virtual interface is configured to do 1500, and your VPN defaults at 1300 to account for the overhead… well, you will encounter trouble.
As a test you might want to tweak these values and slowly increase them as high as they will go without breaking, or switch to a better VPN that does PMTU. That will clear most issues but it will most likely still result in less than stellar performance. You might just want to get a better link if it becomes a constant problem even with tweaking.
I hope this at least helped someone, somehow. Now back to trying to figure out why I get horrible latency (>500ms) on OpenVPN over TCP on a 30 megabits link to a 10 megabits link /but only in Windows 7/. Everything else works fine and I’m grasping at straws. I’ll pester the local admin at this one workplace to switch to UDP but it still doesn’t _explain_ it, and it will bug be forever. Urgh.
1. July, 2010, 09:27