Web filtering with RouterOS web proxy leveraging domain blacklists – how to.
Web filtering with a RouterOS enabled device leveraging domain blacklists from Squidblacklist.org is not difficult to achieve. By simply converting our existing blacklists that we publish for other platforms, into a Mikrotik *.rsc script , we can then reliably filter web traffic using a RouterOS enabled device, as well as many low cost RouterBoard devices.
What does a RouterOS web proxy blacklist look like?
I will quickly walk you through a brief illustrated description of how one would upload a blacklist script via winbox and schedule its execution on an RB750gl. Download and test drive a copy of this blacklist from the links we have provided at the bottom of this article.
Upload the blacklist via winbox (drag n drop)
Of course in production you could and likely would be using automated procedures to do this, using ftp, ssh, or whatever you like for file transfer, but for this demonstration we are just going to drop and drop.
Open the system scheduler and create a task.
Here we create a task in system scheduler and pick a time to execute it. In this case, we are running the following ‘import tik-piracy.rsc’ command.
RB750gl CPU load during import and duration.
It is important for me to point out that the cpu on this RB750GL was under 100% load during this import process, and lasted approximately 2-3 minutes in duration. Around 14,000 domains were loaded from this particular blacklist.
System log after blacklist script run completion.
RouterOS Web Proxy access list after blacklist import.
Here we can see over 14,000 domains successfully imported into the RB750gl’s web proxy access list, which is now ready to filter against piracy related domains.
System Requirements – Memory Limitations
It is important for us to mention that the system memory, ram usage, is very high using our blacklists, in this example we used one of our smaller size lists, and most of our blacklists are comparable in size and line count, however. The most sought after lists that we publish will not run on a low cost Routerboard device such as the RB750gl, and would result in memory exhaustion, a kernel panic and thereafter a resulting watchdog timer reboot. If you attempted to load more than one of these blacklists on a small device such as an RB411/532/133/112/950x type device, this failure would likely be the result. Therefore, we recommend that an x86/x64 PC based platform be used for serious web filtering purposes using RouterOS web proxy. Alternatively, one of the more sophisticated, higher end Mikrotik RouterBoard platforms might also suffice. RouterOS has been proven to have an inherent flaw, or set of flaws that make running our larger lists, porn, proxies, malicious, and prime difficult. A minimum of 6gb ram is required before attempting to run these lists, and a recommended 8gb. Running these four blacklists on a RouterOS device should be considered experimental until Mikrotik fixes the problems challenging RouterOS.
Obtaining blacklists for RouterOS Web proxy.
These blacklists and more are available for download to our members immediately. A subscription to squidblacklist.org is required.
- Blacklist immediate availability from squidblacklist.org for all subscribed members.
- A Routerboard compatibility chart can be found here.
- A sample RouterOS web proxy blacklist is available for download here.
- Mikrotik blacklist conversion tool for windows is available for download gratis.
- Update script examples are available for download.
- If you have any questions contact us.