Web filtering with RouterOS web proxy leveraging domain blacklists – how to.

 

Web filtering with a RouterOS enabled device leveraging domain blacklists from Squidblacklist.org is not difficult to achieve.  By simply converting our existing blacklists that we publish for other platforms, into a Mikrotik *.rsc script , we can then reliably filter web traffic using a RouterOS enabled device, as well as many low cost RouterBoard devices.

 

 

What does a RouterOS web proxy blacklist look like?

Piracy rsc blacklist.
Piracy rsc blacklist script.

 

I will quickly walk you through a brief illustrated description of how one would upload a blacklist script via winbox and schedule its execution on an RB750gl. Download  and test drive a copy of this  blacklist from the links we have provided at the bottom of this article.

 

Upload the blacklist via winbox (drag n drop)

Upload blacklist rsc
Uploading the piracy blacklist rsc via winbox.

 

Of course in production you could and likely would be using automated procedures to do this, using ftp, ssh, or whatever you like for file transfer,  but for this demonstration we are just going to drop and drop.

 

 Open the system scheduler and create a task.

Create task in system scheduler
Create a task.

Here we create a task in system scheduler and pick a time to execute it.  In this case, we are running the following  ‘import tik-piracy.rsc’  command.

 

RB750gl CPU load during import and duration.

Cpu load during run, RB750g
Cpu load during tik-piracy run, RB750g.

 

It is important for me to point out that the cpu on this RB750GL was under 100% load during this import process, and lasted approximately 2-3 minutes in duration.  Around 14,000 domains were loaded from this particular blacklist.

 

System log after blacklist script run completion.

RouterOS Syslog after loading blacklist
RouterOS syslog after loading piracy blacklist.

 

RouterOS Web Proxy access list after blacklist import.

Access list after Loading.
RouterOS web proxy access list after loading piracy blacklist.

Here we can see over 14,000 domains successfully imported into the RB750gl’s web proxy access list, which is now ready to filter against piracy related domains.

System Requirements – Memory Limitations

tikchart
Blacklist compatibility chart.

It is important for us to mention that the system memory, ram usage, is very high using our blacklists, in this example we used one of our smaller size lists, and most of our blacklists are comparable in size and line count, however. The most sought after lists that we publish will not run on a low cost Routerboard device such as the RB750gl, and would result in memory exhaustion, a kernel panic  and thereafter a resulting watchdog timer reboot.  If you attempted to load more than one of these blacklists on a small device such as an RB411/532/133/112/950x type device, this failure would likely be the result. Therefore, we recommend that an x86/x64 PC based platform be used for serious web filtering purposes using RouterOS web proxy.  Alternatively,  one of the more sophisticated, higher end Mikrotik RouterBoard platforms might also suffice.  RouterOS has been proven to have an inherent flaw, or set of flaws that make running our larger lists, porn, proxies, malicious, and prime difficult.  A minimum of 6gb ram is required before attempting to run these lists, and a recommended 8gb. Running these four blacklists on a RouterOS device should be considered experimental until Mikrotik fixes the problems challenging RouterOS.

Obtaining blacklists for RouterOS Web proxy.

These blacklists and more are available for download to our members immediately. A subscription to squidblacklist.org is required.

 Footnotes:


 

  • Blacklist immediate availability from squidblacklist.org for all subscribed members.
  • A Routerboard compatibility chart can be found here.
  • A sample RouterOS web proxy blacklist is available for download here.
  • Mikrotik blacklist conversion tool for windows is available for download gratis.
  • Update script examples are available for download.
  • If you have any questions contact us.

Subscribe Today – Paypal or Credit Card Accepted.

Flat rate subscription. Select a membership option & subscribe.



Select Payment Option



  • You will be issued a username and password.
  • You will be granted access to our member area.
  • 5 Year Membership Option now vvailable.
  • For lifetime membership options click here.clipart
  • Contact us if you would like a pre order invoice.

Disclaimer: All sales are final, we do not issue refunds. Cancel your subscription anytime.

Your feedback. What would you like to see change.

What would you like to see, if anything change about the service, do you have an idea for a new blacklist that you would like to see implemented? Is there something about the service you find lacking? Your feedback is important to us and we would like to know, please, feel free to comment on this blog entry and share your perspective with us on the points that matter to you.

 

 

Thanks!

 

Signed,

Benjamin E. Nichols

http://blog.squidblacklist.org


Subscribe Today – Paypal or Credit Card Accepted.

Flat rate subscription. Select a membership option & subscribe.



Select Payment Option



  • You will be issued a username and password.
  • You will be granted access to our member area.
  • 5 Year Membership Option now vvailable.
  • For lifetime membership options click here.clipart
  • Contact us if you would like a pre order invoice.

Disclaimer: All sales are final, we do not issue refunds. Cancel your subscription anytime.

New list published – racism domain blacklist

We are announcing that we have recently published the all new racism blacklist tailored specifically for web filtering and Squid proxy.  This blacklist contains domain names belonging to websites that publish racist content.

This blacklist is available to all Squidblacklist.org members and can be downloaded at the following url.

http://blog.squidblacklist.org/downloads.html


Subscribe Today – Paypal or Credit Card Accepted.

Flat rate subscription. Select a membership option & subscribe.



Select Payment Option



  • You will be issued a username and password.
  • You will be granted access to our member area.
  • 5 Year Membership Option now available.
  • For lifetime membership options click here.clipart
  • Contact us if you would like a pre order invoice.

Disclaimer: All sales are final, we do not issue refunds. Cancel your subscription anytime.

Worlds largest porn blacklist – adult blacklist.

 

Our adult blacklist has grown significantly over the last couple of years, the db now contains well over 1.3xx,xxx adult domains. As you may know, our objective is to identify every single porn domain in existence, aggressively and by any means necessary.

Our list of pornography domains is  extensive,  and it is growing with every update. It is the largest list of pornography domains available. When combined with a web filtering solution, we are confident that our porn blacklist is the strongest, most effective list for blocking porn at this price point. We have tested the options, we can confidently conclude that none of our competitors come close to matching what we have done with our porn blacklist.

If you are looking for the best adult blacklist you can possibly find, well you just found it. Be aware that approx 750,000+ of these porn domains do not resolve, and therefore put into a temporary delay pool for re query. The actual production porn blacklist you will download is approx 600k live domains. We will give you a diff file with the dead domains upon request.

 

http://blog.squidblacklist.org


Subscribe Today – Paypal or Credit Card Accepted.

Flat rate subscription. Select a membership option & subscribe.



Select Payment Option



  • You will be issued a username and password.
  • You will be granted access to our member area.
  • 5 Year Membership Option now vvailable.
  • For lifetime membership options click here.clipart
  • Contact us if you would like a pre order invoice.

Disclaimer: All sales are final, we do not issue refunds. Cancel your subscription anytime.

Updating pfSense and Squidguard Blacklists with Squidblacklist.org

 

pfSense Logo

For a while we have been informing the public that yes, our lists are compatible with pfSense, DansGuardian, SquidGuard, HAProxy and just about a jillion other web filtering platforms you could possiblely think of. Its as easy as subscribing, and inserting the link to our shalla/urlblacklist compatible format archive found here http://www.squidblacklist.org/downloads/squidblacklists/squidblacklist.tar.gz

Untitled2

Authentication:
You will be required to authenticate with a valid username and password to download our works, which is no big deal really, most of these platforms are using wget or fetch to pull the blacklists, and both of these programs support basic authentication. Figuring it out should be trivial for the average user who knows how to google.


Alternatively, if you really wanted to, enabling automated updates for our standard acl formatted blacklists can also be done, but does require a little additional work, just some simple scripting really.  One of our members recently shared a link and a series of scripts with the intention that his works be shared with the community.

If you are interested in pursuing this method, below is a link to his original blog entry describing his efforts as well as direct links to download the scripts he has drafted. But for most people using the shalla/urlblacklist compatible formatted archive will be much easier because many of these programs and platforms are coded to accept that particular file structure.

Thank you to Paul Goreman for submitting his time and research.

http://paulgorman.org/blog/1395259268
http://legoclan.com/pfsense/
We have also hosted a backup copy of his update script.
http://blog.squidblacklist.org/downloads/sblorg_updater.tar.gz


Subscribe Today – Paypal or Credit Card Accepted.

Flat rate subscription. Select a membership option & subscribe.



Select Payment Option



  • You will be issued a username and password.
  • You will be granted access to our member area.
  • 5 Year Membership Option now available.
  • For lifetime membership options click here.clipart
  • Contact us if you would like a pre order invoice.

Disclaimer: All sales are final, we do not issue refunds. Cancel your subscription anytime.

WordPress is up.

I and some associates decided it would be in all of our best, mutual interests to finally setup a wordpress for squidblacklist.org so that we might begin to engage in public dialogue regarding issues of mutual interest. so, here it is. feel free to register. And we will sort the bugs as we go along.  Any issues with this blog just email me.

 

Signed,

Benjamin E. Nichols

http://blog.squidblacklist.org


Subscribe Today – Paypal or Credit Card Accepted.

Flat rate subscription. Select a membership option & subscribe.



Select Payment Option



  • You will be issued a username and password.
  • You will be granted access to our member area.
  • 5 Year Membership Option now vvailable.
  • For lifetime membership options click here.clipart
  • Contact us if you would like a pre order invoice.

Disclaimer: All sales are final, we do not issue refunds. Cancel your subscription anytime.