At Shunra, we have helped many customers discover how the mix of users accessing a site can have a significant impact on overall application performance.
The findings we deliver are not intuitive. It would seem that users accessing an application from a slower network would of course have poorer overall performance, but would stay out of the way of users with high-speed network connections.
Not so!
As it turns out, web servers have a finite number of resources assigned to handling client (end user) requests. As long as the requests can be handled rapidly within the server, each user’s experience is largely independent of one another.
What happens when a user on a slower link accesses the application? Well, responses will naturally take longer to be transmitted back to the client. Because of the way TCP operates (guaranteed delivery), there are back and forth acknowledgments that need to occur. Many of them can require a “round trip” which results in a delay caused by the latency that is in effect between the client and server. For example, an easy test I did tonight in a few minutes, using Shunra’s emulation technology shows that a 125K response will take 50% more time when accessed over a 200ms network link than a 100ms network link, and 10 times longer than a 20ms network link. This is assuming “infinite” bandwidth, too! The time for downloading the data over a 200ms link is 1.2 seconds. The performance of pages containing multiple graphics, javascript, stylesheets, and other elements will be even more severely impacted.
What happens to the application when a response takes longer to be sent back to the client workstation? The data being generated at the server (for example in a JSP page or ASP.NET page) has to be stored somewhere. What happens in fact is that the server reserves an output buffer for the response. If more data is being generated than fits in the buffer, the server thread needs to be suspended (wait) until enough has been transmitted over the network to allow the buffer to empty. If enough threads are tied up in this manner, new requests coming into the server will wait in an input queue for an available thread.
Furthermore, if the application logic is connected to the database while generating the output, bottlenecks can appear in the DB connection pool or even in locks and contention within the database.
The net effect can surface as sporadic and hard to troubleshoot performance variations that can take weeks or months to pinpoint. This is why Shunra offers tight integration with Industry-leading performance testing tools. A load test that is performed without including the impact of the network can miss important scalability issues that will impact end user experience and impact bottom-line business results.
NOTE: The default buffer size is 8kb for Java Servlet engines such as Tomcat, and 4MB for IIS 6.
December 11th, 2008 at 5:07 pm
Good article, but what can be done if you find problems? i.e., app works well locally, but when my remote users start to use it, they yell about performance issues. Then the local users start to see performance and availability problems b/c server resources are being drained.
December 12th, 2008 at 10:23 am
Great question!
There are too many variables to provide a single “cook book” answer, but here are my thoughts.
Any testing process should result in timely and actionable information. When testing using WAN emulation shows that slower users will be impacting overall application performance, there are several things that can be done.
It will be important to first identify where the bottleneck is actually originating. Key statistics such as the web server utilization (e.g. mod_status on Apache), and the DB connection pool usage should help determine where queuing is occurring. DB monitoring will tell you if transaction throughput is being impacted due to lock contention. Deeper monitoring that instruments the transaction code path and profiles individual transactions may also be needed.
If the problem is in the web server itself, where slow users are tying up worker threads or connection pool entries, it is likely that adding more web server threads or connection pool entries could provide enough capacity to support the volume of slower users.
The more proactive approach is to test the application during development, and identify in advance those transactions that will either (A) perform unacceptably for some remote users, or (B) result in consuming excessive server-side resources. Sometimes simple changes such as ensuring that compression and caching is being done properly will have a substantial difference. A simple change to the way that application page output is buffered, or the size of the page buffer can resolve many problems as well.
January 1st, 2009 at 12:58 pm
I would like to add to Les’ comments some of my suggestions:
During a load test that introduces “slow” users next to “fast” ones, I always use SiteScope and other monitoring tools to monitor various server resources. I also schedule the test in a way that first ramps up the “fast” local users and runs for 5 minutes in “steady state” (when all local users are up and running) and only then gradually introduces the “slow” remote users (by remote I mean generating load over an emulated network). So any impact that the slow users have on server performance can be easily identified.
Hope this helps,
Amichai
July 29th, 2009 at 8:42 pm
Ouch! Applications accessed by users with a slower network link can impact everyone http://tinyurl.com/nw8tk5