Problems & Solutions with Mikrotik RouterOS DNS Domain Blacklists.

Article by Benjamin E. Nichols http://www.squidblacklist.org
Introduction.

As a publisher of domain blacklist data, I thought it would be appropriate the share some recent challenges and resultant experiences with Mikrotik RouterOS DNS Domain name blacklists, especially considering that we charge a service fee for access to these blacklists. But first, we would like to thank Mikrotik for their fine products and timely support and we hope to continue to co-operate in the future. The following article describes recent issues, historical problems, and current fixes, patches and workarounds for categorized domain blacklisting utilizing the static dns entry features of RouterOS.

Recently we had some issues with these lists that was directly related to four individual problems all of which have been resolved.

1. Painfully Slow Import of Large Domain Blacklists ( Resolved )

Importing large domain blacklists has been an excruciatingly, absurdly slow process for many years, even with the most expensive high end Mikrotik RouterOS devices, which I believe the general consensus would be that, if one were to pay a premium, one would anticipate premium performance.

This issue has been recently resolved by Mikrotik with a new patch for RouterOS included in the latest release candidate as of October 2016. This was a very overdue fix and a welcomed change that will undoubtedly bring us closer to making domain based web filtering using standalone Mikrotik RouterOS devices that much closer to be practical for most people. We suggest you test the latest release candidate for yourself. The fix has been included in 6.38rc15 (Release candidate) available from http://www.mikrotik.com/download.

2. Recent Changes to RouterOS Static DNS ( Resolved )

Another issue that we faced with static dns entries in particular, is that with a recent change in RouterOS, a change that was made some time towards the end of summer 2016. They seemed to have changed the way that Static DNS entries are handled by the OS, forcing us to change our format in order to retain operability. Which actually, ended up being a very good thing that forced us to make dramatic improvements to our static dns format, a change which, in and of itself is something that was admittedly also long overdue. And the new format is perfectly suited for RouterOS Static DNS Entries. And we owe a debt of gratitude to the generous folks over at Mikrotik’s forums for helping us resolve those issues in a timely manner.

3. The 60 character limitation: ( Resolved )

Then we found out after consultation with Mikrotik via ticket submission and email discussions with their support staff, that RouterOS has a 60 character limitation that prevents domain names beyond a certain complexity to be loaded. This prompted us to open a support ticket, which began the dialogue. We were getting the following message when loading blacklists “error regex too compex”.

Also note the misspelling of the word complex, "compex".

Also note the misspelling of the word complex, “compex”.

( The spelling of complex has been fixed after our discussion with support staff. ) We decided an easy work around for this was to simply remove all domain names with more than 60 characters, and while we dont like throwing away domain data, after careful analysis of the data removed, most of these domains were junk so its not too big of an issue, and its a fix which seemed to work to address the problem. ( Mikrotik has informed us they do not plan to fix this any time soon as the work involved doing so is prohibitive at this time . )

4. Static DNS Blacklist – Script Failure at “Error duplicate entry detected” ( RESOLVED )

Now, this is where we hit a roadblock with Mikrotik Static DNS Entries. You see, loading an individual static dns blacklist from Squidblacklist.org into a RouterOS device works just fine, given that you must have adequate resources, memory, storage and cpu power. ( see our compatibility chart )

The problem however is with the fact that the way RouterOS handles duplicate entries, causing the import process to abort. Allow me to elaborate. The reason why this is an unacceptable end is that some domains will inevitably exist in multiple blacklists and/or blacklist categories, for example, a pornography website may also be malicious, and therefore the domain name will be present in both the adult, and the malicious blacklists. A network administrator may decide to load both of these two blacklists, which should work, however, it wont. We have no way of predicting which combination of blacklists somebody would opt to load so we cannot create some code to willy nilly remove domains on th fly..

Further Expounding. A solution is required to address the problem of loading multiple blacklists with overlapping domain entries.

Here is the solution.

We add on-error={} to the end of the line, this seems to be a great workaround and has eliminated the issue.

dsdfsdsdf

As a result of publishing this article and the work we have done here, our ADS blacklist for blocking ads using Mikrotik RouterOS Static DNS will now be free for everybody to download and use to show our appreciation for contributions.

It can be downloaded at the following url. http://www.squidblacklist.org/downloads/tik-dns-ads.rsc

A huge thank you to the developers, and to the volunteers who spent countless hours resolving issues, and creating a better future.

Thank you to Jonas Carlsson of remote24.se for contacting us with scripting support on the resolution of issue #4

Thank you for your time, and we hope that making this information public will help somebody out there.

Respectfully,

Benjamin E. Nichols
http://www.squidblacklist.org


If you like what we are doing here and want to support our efforts, please consider subscribing to download all of our blacklists.

Flat rate subscription. For full access to all of our works, select a membership option & subscribe today.



Select Payment Option



  • You will be issued a username and password.
  • You will be granted access to our member area.
  • 3 Year Membership Option now available.
  • Contact us if you would like a pre order invoice.

Disclaimer: All sales are final, we do not issue refunds. Cancel your subscription anytime.


2 Replies to “Problems & Solutions with Mikrotik RouterOS DNS Domain Blacklists.”

  1. Hi!

    I read your post regarding the problem with RouterOS not handling adding an entry that already exists.
    This is a real mess and not only affecting list imports but other things as well.

    The biggest problem is that it quits the script processing, unlike any other script processing engine that I know of.

    Our “solution” is to do a try-catch approach.
    For instance:

    / do {/system scheduler remove remote24} on-error={}

    This handles the case if the schedule “remote24” does not exist.

    Just throw whatever you want between {}.

    Keep up the good work!

    Take care


    Jonas Carlsson

Leave a Reply

Your email address will not be published. Required fields are marked *

*