With two WMS running off identical data on the same server, I thought it would be interesting to compare speeds and the map output. So I lined up a map request with identical parameters to both services and ran them through Fiddler a few times (to get an idea of the response times).

Output quality and performance

	GeoServer	MapServer
Response time	0.8 to 1.0 seconds with USE_JAI_IMAGEREAD=true 0.6 to 1.3 seconds with USE_JAI_IMAGEREAD=false	0.4 to 0.6 seconds
Image size	63,574 bytes	78,327 bytes
The map
Quality	Adequate, but quite fractured	Lovely
Comments	A significantly slower response than MapServer, and a worse image. I couldn’t find any difference between the different re-sampling methods, perhaps suggesting that the settings were not taking effect (I’ve seen GeoServer GUI issues like this a couple of times now, requiring diving into the XML config files). Later – yes the settings were present in the XML config, but seem to make no difference to the output.	Faster and much nicer image. If we turn off bilinear re-sampling, then the response time improves to around 0.3 seconds and the image deteriorates slightly as shown below (becoming similar in quality to the GeoServer map).

Oh dear this is not looking good for GeoServer. But is it the whole story? Next I compared them in a more realistic scenario, as base maps in OpenLayers, trying both single tile and multi-tile modes. I configured the map to view the 1:50,000 mapping at a scale where no re-sampling could occur.

	GeoServer	MapServer
Response time multi-tile (30 tiles each 256x256 pixels)	0.2 to 1.2 seconds, but on average a shade faster	0.3 to 0.7 seconds
Response time single-tile (1725 x 1173 pixels)	4 to 6 seconds	3 to 5 seconds
Image size (typical 256x256 tile)	30,447 bytes	22,674 seconds
Image size (1725 x 1173 pixels)	646,681 bytes	690,494 bytes
The map (256x256 tile)
Quality	Very good	Very good (pretty much identical)

OK so now it is looking better for GeoServer; when there is no re-sampling it can return identical images to MapServer, though slightly larger, and performance is pretty similar and possibly quicker.

Output quality and performance when re-projecting

Now time to see how they cope with serving the maps into a different coordinate system, and in particular the global spherical Mercator used in Google and Bing Maps.

	GeoServer	MapServer
Response time multi-tile (30 tiles each 256x256 pixels)	0.2 to 1.2 seconds	0.1 to 0.5 seconds
Response time single-tile (1725 x 1173 pixels)	10 seconds	6 seconds
Image size (typical 256x256 tile)	29,169 bytes	37,261 seconds
Image size (1725 x 1173 pixels)	1,041,098 bytes	982,050 bytes
The map (256x256 tile)
Quality	Pretty horrible	Very good

With default settings, GeoServer produces horrible output when re-projecting the raster maps. But this does need more investigation when I found out how to make it use different re-sampling algorithms.

Performance under load

Now time to see how they perform under load using multi-mechanize. I chose to compare MapServer, GeoServer and a tilecache at returning a 256x256 tile in EPSG:900913 (i.e. re-projecting on-the-fly). Each test ran for 100 seconds, starting with 10 virtual users, increasing by 10 every 10 seconds. The tilecache image was of course cached and simply being retrieved from disk. All tests were using the same web server, with files on the same drive.

Note: the vertical scale differs in each graph to fit the observed values.

	GeoServer	MapServer	Tilecache
Transactions	381	556	2188
Errors	57	0	0
Average response time	6.1	6.5	2.3
95% response time	26.8	17.6	5.1
Response graph
Transactions per second
Comments	Completely broken after about 80 seconds – had to re-start GeoServer	No errors, but significantly disrupted performance	Slight slow-down under high load, but very reassuring performance

GeoServer clearly could not stand up to more than about 30 virtual users when re-projecting the maps, and ultimately the service stalled completely requiring a re-start.

I tried the tests again requesting the maps in OSGB so that the service did not involve re-projection. I also reduced the number of virtual users to 50. The results were quite surprising…

	GeoServer	MapServer	Tilecache
Transactions	2018	1114	2197
Errors	0	0	0
Average response time	1.3	2.4	1.1
95% response time	2.4	9.1	2.5
Response graph
Transactions per second
Comments	Impeccable!	Much better than when re-projecting, but only half as good as GeoServer.	Only fractionally better than GeoServer

This is a hugely impressive result for GeoServer. My guess is that it is employing server-side in-memory caching, and because all my requests were identical the responses were very fast. MapServer on the other hand has to start a CGI process of each request and presumably cannot benefit from an in-memory cache.

When re-projecting on the other hand, perhaps it does not use a cache. And because all the GeoServer operations are running as one process, it is restricted to one virtual processor on our server whereas each MapServer exe can grab one for itself. Certainly when running these tests the server was running at about 40-80% CPU with MapServer, compared to 7% with GeoServer.

I ran this test again with the number of virtual users doubled back to 100, and GeoServer came through almost unscathed, this time processing 2072 transactions though with 3 errors.

	GeoServer	MapServer	Tilecache
Transactions	2072	907	2234
Errors	3	0	0
Average response time	2.4	5.9	2.3
95% response time	5.9	25.6	4.8
Response graph
Transactions per second
Comments	Wow	Suffering, but no errors	The best as expected

I suspect this test may not predict real-world behaviour with lots of users requesting different WMS maps, where I suspect GeoServer would lose its advantage, but it is interesting anyway.

NB I had GeoServer running with AllowMultithreading=true

Summary

If the maps need to be re-projected then use MapServer – both performance and output quality are vastly superior. In a site that is going to get any significant load it is unwise to re-project raster mapping on-the-fly anyway, and it should definitely be cached. In this case the WMS performance doesn’t matter so much but output quality is paramount, so MapServer it is.  If I find a way of making GeoServer produce nice looking output I’ll come back and revise this post!

If no re-projection is required, then simple observation has the two roughly equal under light load. The maps looked almost identical, and performance was close. GeoServer had the advantage for me in that it didn’t suffer from MapServer’s tendency to choke on some requests; in practice this meant I went over my entire OSGB cache again with a GeoServer WMS as the source, to plug the gaps left by the MapServer service.

Under heavy (albeit contrived) load, the crucial factor was whether or not there was any re-projection involved. Without re-projection GeoServer produced astonishing performance, thrashing MapServer and nearly matching a tilecache. Introduce re-projection however, and GeoServer collapses to the extent that it can crash entirely – not good when it is running all your mapping services. MapServer never broke, and despite some gaps and slow responses, it managed to struggle through all tests without a single error.

So out of this experience I think we need to be using both, depending on the purpose.

Next to test performance on vector layers, including file based GIS formats and spatial databases…