Unscientific Comparison Between Router Firmwares

Before you continue to read, this piece is written by a OpenWRT user so you should expect some degree of bias towards OpenWRT and my Mods but I backed this up with data. It should be reproducible.
LAN to WAN methodology is described here.

This post is about my testing of various firmware available for TL-WR1043ND Router.
Since the Router is TP-Link branded, all comparisons will be using TP-Link factory firmware as a standard.
The performance comparison would be LAN to WAN over Gigabit Ethernet Static NAT Routing.

I tested a total of 5 firmwares:

1. Stock TP-Link firmware
2. DD-WRT
3. Gargoyle
4. OpenWRT
5. OpenWRT with tweaks

Observations

 TP-Link firmware

My assessment on this firmware is that it is targeted as a standard multimedia Router, offering MediaServer, Samba and FTP on a Single Router.

It is a firmware with fixed flash partitions with a small sector for storing your Router settings persistently.The webserver uses a frame layout to reload only a single page, the setting page in the center. There is not much I can say about it except the Routing Performance.

It averages below 140Mbps. Performance may vary based on the running processes but by default there is very little running processes to tune so you can estimate the max to be 150Mbps. It uses the standard GCC mips cross compiler to generate the firmware which partly explains the Mediocre performance. It would be suitable for users with a subscription line of 100Mbps.

DD-WRT firmware

Now DD-WRT is a very interesting software. Yes I first started on 3rd Party firmware on DD-WRT which I later moved on to OpenWRT. Because the firmware was so interesting I took a closer look at it.

DD-WRT is structured very much like TP-Link firmware it has only 64KB fixed partition for storing user settings, although there are ways to enable optware.

All the features seen on the webserver is all the features you can access.

1 thing is for sure, it is jam packed with features. 1 feature which stock I could not find on DD-WRT but could find on stock firmware is the media server. This however does not come as a surprise, the biggest package to enable mediaserver feature is the ffmpeg codec which is impossible to fix into DD-WRT which is already jam packed with features.

When I telnet into router it shows that the router uses GCC linaro 4.8 compiler to optimize the firmware as the same as OpenWRT.

It uses some of OpenWRT's components and also a updated kernel version 3.5.7.

The web interface while it looks nice seems messy to someone using OpenWRT for a long time. By default all additional features are disabled so the Routing test below is probably the best you can get. Performance isn't bad.

It maxes at below 250Mbps so it will be good for 200Mbps lines.

Gargoyle

Gargoyle is based on OpenWRT with its own javascript based user interface on top. It prides on its control over QoS Services. It uses linaro gcc 4.6 like OpenWRT. Technically speaking it should perform around the same as OpenWRT however this is not the case as I have seen later. Some differentiating features not found by default on other firmware is the Tor routing function. It allows installation of addition software as like OpenWRT JJFS persistent storage is mounted on top of the read only squashfs but the software repository is limited. This Gargoyle firmware which I tested is based on OpenWRT Attitude Adjustment. After disabling MSS Fix a optional iptable feature which slows down routing the results are as shown.

As shown in the chart routing performance averages above 170Mbps. If a bandwidth margin is considered it would be good for 150Mbps connection.

OpenWRT Attitude Adjustment

This is the standard OpenWRT Attitude Adjustment release, it comes bare with very little features out of box. After disabling MSS fix the following result is obtained.

It averages below 250Mbps like DD-WRT so it should be good for 200Mbps line.

OpenWRT Attitude Adjustment Optimized

This is my build of OpenWRT Attitude Adjusment for WR1043ND.

Basically I added features functionally equivalent to stock firmware, added a couple of additional features like L7 filter(also found in DD-WRT but disabled by default). The build is platform specific optimized and targeted at MIP24Kc Processor. I also included Web Optimizations and Fixes and stripped out all debugging features I can find. You can either have debug or performance but not both. Basically this is OpenWRT on steroids. After many trial and error I finalised it when there is consistent observable improvement.

To make the comparison fair I disabled L7 filter and MSS fix(Actually it outperforms the rest even without it at 270Mbps!).

So below is the result of it without L7 filter and MSS fix.

So this goes to show how optimization can improve performance on embedded system. This post might invite flaming between 3rd party firmware die hard fans but I think the graphs speak for themselves.