Something smells funny at urlblacklist.com

Urlblacklist.com is horrible service provider whos website was recently down for over 3 months in the 2016 calendar year. As a provider of services to paying clients, this is simply a disgrace. We have been monitoring urlblacklist.com, watched multiple outages, and we know that their domain name changed hands recently, something they did not announce on their website, so we will do it here.

Competition is great, it drives innovation and gives companies the incentive to improve and the drive needed to continue to improve services and products. And in the spirit of competition sometimes a little dirt gets slung. But it isn’t only serving our personal interests to outline the failures of one of our competitors today. In this case, I believe that it is in fact doing the public a service. And by this I mean spreading awareness, to warn unsuspecting customers of what they are spending their money on, and of course, to convince you. with a little sound reasoning, to purchase our services instead of the competition.

There aren’t many suppliers of domain blacklist data for web filtering platforms and applications, this is one of the primary motivating reasons why we created Squidblacklist.org in 2012. We knew it could be done better. There are however a small handful of other providers remaining. One of these websites is urlblacklist.com, and it is them whom we have chosen to single out for scrutiny in this blog post.

The website urlblacklist.com was down for nearly an entire month in October 2016. Then, a second outage occured that lasted nearly 2 months, beginning on or around Nov 10, lasting through until Dec 30, 2016. Further scrutiny reveals that in Nov of 2016, the owner of urlblacklist.com irresponsibly allowed his domain name to expire, which resulted in another 2 month long outage, in which another company took the opportunity to purchase the domain name, which is now registered under an entity named “Dr Guardian” who one can only assume has taken ownership and reopened the website, and is actively processing payments by unsuspecting customers.

They also do not make any mention of ownership changing hands anywhere on their website, a courtesy any respectable business would extend to its existing clients and the general public. Instead of even acknowledging the domain name ownership change. they choose to deceptively place blame for the outage on a billing issue with their registrar.

And I doubt if anybody knows who is really even operating the website. The owner has never responded to emails, doesn’t seem too care if his website goes down until months later, and I suggest that you should seriously consider switching to Squidblacklist.org if you are a current urlblacklist.com subscriber.

A brief visit to urlblacklist.com shows that the owner would like you to believe the second extended outage of 2016 was brief, a deception which is evident in a recent “news” message claiming that “over Christmas vacation” there was an outage, I guess Christmas vacation begins around Nov 10 and ends December 30th for the owner.

One can safely assume by looking at the ancient and truly aweful web design of urlblacklist.com and then research its track record of unreliability, and make some general conclusions about its owner and or operators which are to say, generally not favorable at all. And this lack of integrity can also be found in the quality of their blacklists, or lack thereof, which is of course, what really matters.

Urlblacklist.com is an aging website. I would encourage you to use their lists, and monitor the daily changes, you will know first hand that nearly 60% of the domains in their blacklists do not even resolve, a good indicator that there is very poor technology behind the update processes going on behind the scenes. It becomes clear rather quickly that they are pushing old, recycled domain data by systematically removing a set number of domains and re-adding them back using some crude scripts or something, rotating this data in and out in a way which gives the customer the illusion that updates are taking place.

With manual additions and removals being performed occasionally presumably by human hands. Beyond this however, there is obviously zero innovation taking place at urlblacklist.com. Which is evidently ran by an incompetent individual who is doing the world a disservice by continuing to accept payments for what is, in our opinion, hardly passable even as a purely free service.

Shalla and any other websites referring people to this website are also guilty of doing the general public a disservice by linking to urlblacklist.com because clearly anybody who has actually observed or used the lists from these people should come to the same conclusions that we have, and that is urlblacklist.com is an unreliable provider of services that needs to go away.

Also make note that shady clones of urlblacklist.com are also in existence, these domains are registered by totally different owners.
http://schwela.com is one of them.

Problems & Solutions with Mikrotik RouterOS DNS Domain Blacklists.

Article by Benjamin E. Nichols http://www.squidblacklist.org
Introduction.

As a publisher of domain blacklist data, I thought it would be appropriate the share some recent challenged and resultant experiences with Mikrotik RouterOS DNS Domain name blacklists, especially considering that we charge a service fee for access to these blacklists. But first, we would like to thank Mikrotik for their fine products and timely support and we hope to continue to co-operate in the future. The following article describes recent issues, historical problems, and current fixes, patches and workarounds for categorized domain blacklisting utilizing the static dns entry features of RouterOS.

Recently we had some issues with these lists that was directly related to four individual problems all of which have been resolved.

1. Painfully Slow Import of Large Domain Blacklists ( Resolved )

Importing large domain blacklists has been an excruciatingly, absurdly slow process for many years, even with the most expensive high end Mikrotik RouterOS devices, which I believe the general consensus would be that, if one were to pay a premium, one would anticipate premium performance.

This issue has been recently resolved by Mikrotik with a new patch for RouterOS included in the latest release candidate as of October 2016. This was a very overdue fix and a welcomed change that will undoubtedly bring us closer to making domain based web filtering using standalone Mikrotik RouterOS devices that much closer to be practical for most people. We suggest you test the latest release candidate for yourself. The fix has been included in 6.38rc15 (Release candidate) available from http://www.mikrotik.com/download.

2. Recent Changes to RouterOS Static DNS ( Resolved )

Another issue that we faced with static dns entries in particular, is that with a recent change in RouterOS, a change that was made some time towards the end of summer 2016. They seemed to have changed the way that Static DNS entries are handled by the OS, forcing us to change our format in order to retain operability. Which actually, ended up being a very good thing that forced us to make dramatic improvements to our static dns format, a change which, in and of itself is something that was admittedly also long overdue. And the new format is perfectly suited for RouterOS Static DNS Entries. And we owe a debt of gratitude to the generous folks over at Mikrotik’s forums for helping us resolve those issues in a timely manner.

3. The 60 character limitation: ( Resolved )

Then we found out after consultation with Mikrotik via ticket submission and email discussions with their support staff, that RouterOS has a 60 character limitation that prevents domain names beyond a certain complexity to be loaded. This prompted us to open a support ticket, which began the dialogue. We were getting the following message when loading blacklists “error regex too compex”.

Also note the misspelling of the word complex, "compex".
Also note the misspelling of the word complex, “compex”.
( The spelling of complex has been fixed after our discussion with support staff. ) We decided an easy work around for this was to simply remove all domain names with more than 60 characters, and while we dont like throwing away domain data, after careful analysis of the data removed, most of these domains were junk so its not too big of an issue, and its a fix which seemed to work to address the problem. ( Mikrotik has informed us they do not plan to fix this any time soon as the work involved doing so is prohibitive at this time . )

4. Static DNS Blacklist – Script Failure at “Error duplicate entry detected” ( RESOLVED )

Now, this is where we hit a roadblock with Mikrotik Static DNS Entries. You see, loading an individual static dns blacklist from Squidblacklist.org into a RouterOS device works just fine, given that you must have adequate resources, memory, storage and cpu power. ( see our compatibility chart )

The problem however is with the fact that the way RouterOS handles duplicate entries, causing the import process to abort. Allow me to elaborate. The reason why this is an unacceptable end is that some domains will inevitably exist in multiple blacklists and/or blacklist categories, for example, a pornography website may also be malicious, and therefore the domain name will be present in both the adult, and the malicious blacklists. A network administrator may decide to load both of these two blacklists, which should work, however, it wont. We have no way of predicting which combination of blacklists somebody would opt to load so we cannot create some code to willy nilly remove domains on th fly..

Further Expounding. A solution is required to address the problem of loading multiple blacklists with overlapping domain entries.

Here is the solution.

We add on-error={} to the end of the line, this seems to be a great workaround and has eliminated the issue.

dsdfsdsdf

As a result of publishing this article and the work we have done here, our ADS blacklist for blocking ads using Mikrotik RouterOS Static DNS will now be free for everybody to download and use to show our appreciation for contributions.

It can be downloaded at the following url. http://www.squidblacklist.org/downloads/tik-dns-ads.rsc

A huge thank you to the developers, and to the volunteers who spent countless hours resolving issues, and creating a better future.

Thank you to Jonas Carlsson of remote24.se for contacting us with scripting support on the resolution of issue #4

Thank you for your time, and we hope that making this information public will help somebody out there.

Respectfully,

Benjamin E. Nichols
http://www.squidblacklist.org



If you like what we are doing here and want to support our efforts, please consider subscribing to download all of our blacklists.

Flat rate subscription. For full access to all of our works, select a membership option & subscribe today.



Select Payment Option



  • You will be issued a username and password.
  • You will be granted access to our member area.
  • 5 Year Membership Option now available.
  • For lifetime membership options click here.clipart
  • Contact us if you would like a pre order invoice.

Disclaimer: All sales are final, we do not issue refunds. Cancel your subscription anytime.


OPNSense developers add support for Authentication – Enabling Blacklists from Squidblacklist.org

OPNsense Firewall Distribution
OPNsense Firewall Distribution

After a brief exchange via email, one of the developers of OPNsense has contributed a modification that will allow for an option to authenticate using credentials for downloading blacklists, something which is essential for Squidblacklist.org members to use and update blacklists with OPNsense. This change will be available in the upcoming 16.7.7 release of OPNsense.

In the coming days and weeks further testing and collaboration should follow, to ensure full functionality, such as automated updates with scheduling options and features. And standard testing for any issues that may arise, but we are very excited to see progress being made. And would like to express our gratitude to the developers who are donating their time and effort to make the world a better place for the end user.

I will post a full review once final testing has concluded and the resultant product is solid, we will also update our materials to reflect the latest changes when we reach this point and confirmation compatibility and stability are achieved.

New Blacklist: A List of Terrorist & Extremist Domains.

We have been working to compile the initial release of a new blacklist, which we have now released in the members area of Squidblacklist.org. Available for immediate download, our new and unique “Terrorism and Extremism” Domain Blacklist. Available now for all current and new subscribers. As is always, we have formatted this blacklist for use with multiple platforms.

DNS Zone format, RouterOS DNS and Web Proxy *RSC import script, Squid Proxy Native ACL format, as well as plain text Dansguardian/SquidGuard Compatible format for use with all other web filtering platforms and applications.

Subscribe today and find out why so many have switched. Shouldnt you be using a higher class blacklist?

We are committed to continually evolve the technology we are using on the backend to produce ever increasingly better publications, We are proud to consistently bring ever more refined and simply the highest quality blacklists available.


Subscribe Now For Zone Access.

Flat rate subscription. For full access to all of our works, select a membership option & subscribe today.



Select Payment Option



  • You will be issued a username and password.
  • You will be granted access to our member area.
  • 5 Year Membership Option now available.
  • For lifetime membership options click here.clipart
  • Contact us if you would like a pre order invoice.

Disclaimer: All sales are final, we do not issue refunds. Cancel your subscription anytime.


Update: RouterOS and Large Static DNS Blacklists

We would like to extend a huge debt of gratitude for Mikrotik’s rapid and appropriate response to the issues we recently had related to a new updated version of RouterOS and changes that were made to the way the OS handles static DNS entries.

We were a bit surprised when we were forced to make changes to the format, and in doing so, had discovered that RouterOS needed some optimizations, which Mikrotik software engineers were happy to work with us, and roll out an update to RouterOS that optimizes the import speed of large blacklists on Mikrotik RouterOS.

They also were generous and advised us on how to appropriately eliminate an issue we were having with our dns blacklist format. I would like to thank the members of the Mikrotik public forum for help getting the regex format corrected.

Thank you all!

Updates and Changes to Mikrotik RouterOS Blacklists

There have been some changes in the latest version of Mikrotik RouterOS, which meant we really had no choice but to make some minor changes, otherwise the old format simply would fail to work when you tried to load them into a current version of Mikrotik RouterOS ( version v6.37 or newer ).

It seems Mikrotik decided, for whatever reason, to change the way static dns entries are handled.

RouterOS DNS Static Entry Change - Side by Side Comparison
RouterOS DNS Static Entry Change – Side by Side Comparison

As you can see in the image above , the changes were significant enough to force us to make the changes, if you are havin any issues loading our blacklists then you should update to the latest version of RouterOS as soon as possible.

We also decided that it would be best to add a single line to the headers included in each blacklist, to remove old entries befoe loading the new ones. Of course any knowledgeable admin would know to do this, but we felt is was something that should already be included in the blacklists for your convenience.

NEW FORMAT:

# TiK-DNS-Ads: Blacklist compiled by SquidBlacklist.org 10-01-2016. -MADE IN USA-
:log info "tik dns ads blacklist script import started"
:local redirectIP "127.0.0.1"
/ip dns static remove [find comment="sbl ads"]
/ip dns static
add regexp="^(.*\\.)\?004\\.frnl\\.de\$" address="$redirectIP" comment="sbl ads"
add regexp="^(.*\\.)\?01s\\.net\$" address="$redirectIP" comment="sbl ads"
add regexp="^(.*\\.)\?01viral\\.com\$" address="$redirectIP" comment="sbl ads"
add regexp="^(.*\\.)\?0427d7\\.se\$" address="$redirectIP" comment="sbl ads"
add regexp="^(.*\\.)\?0702\\.de\$" address="$redirectIP" comment="sbl ads"
add regexp="^(.*\\.)\?0ca\\.net\$" address="$redirectIP" comment="sbl ads"

I hope this will help to clarify for those of you who are scratching your heads about the sudden changes.

Thank you for your support.

Signed,

Benjamin E. Nichols
http://www.squidblacklist.org


Blacklisting has Evolved. Subscribe Now!

Flat rate subscription. Select a membership option & subscribe.



Select Payment Option



  • You will be issued a username and password.
  • You will be granted access to our member area.
  • 5 Year Membership Option now available.
  • For lifetime membership options click here.clipart
  • Contact us if you would like a pre order invoice.

Disclaimer: All sales are final, we do not issue refunds. Cancel your subscription anytime.


Blacklist Processing – Automated Domain Addition & Removal – A.D.A.R. System – Delay Pool Recheck Function

A little peek at some of whats going on behind the scenes.

One of the things that we do when we process the data that we publish, is to add domains, but also to remove unwanted, or delisted domains that match a certain criteria. But its never a good idea to simply delete the data, instead, we place these removed domains into the delay pools, domains that are flagged as no longer resolving, or being redirected to place holders, which is common with domains that are parked or suspended. Once this data is added to a delay pool, the data is then rechecked again and again, if the domains ever come back online, they get added back to the blacklists again before being white washed through filters again later during our daily update processes.

At Squidblacklist.org, we are working to bring you a higher class of blacklist through logic and innovation.

DNS Zone Blacklists – Now Available For Download

DNS Filtering with Zone Blacklists
DNS Filtering with Zone Blacklists

Blacklists by Squidblacklist.org are now immediately available for download in Bind Zone compatible blacklist format. We have fully ported our domain blacklists into zone format for use with all major dns servers compatible with standard zone files. Providing a viable alternative to OpenDNS at a competitive price point.

These zone blacklist downloads are available to all Squidblacklist.org subscribers in the members area of our website.


// DNS-Ads: Blacklist compiled by Squidblacklist.org 09-06-2016. -MADE IN USA-
// Blacklists licensed under Creative Commons Attribution-NoDerivatives 4.0 License.
zone "0day.kiev.ua" {type master; file "/etc/namedb/blockeddomain.hosts";};
zone "100-100.ru" {type master; file "/etc/namedb/blockeddomain.hosts";};
zone "100free.com" {type master; file "/etc/namedb/blockeddomain.hosts";};
zone "100free.nl" {type master; file "/etc/namedb/blockeddomain.hosts";};
zone "100suelle.com" {type master; file "/etc/namedb/blockeddomain.hosts";};
zone "101com.com" {type master; file "/etc/namedb/blockeddomain.hosts";};
zone "101order.com" {type master; file "/etc/namedb/blockeddomain.hosts";};
zone "1100ad.de" {type master; file "/etc/namedb/blockeddomain.hosts";};
zone "123found.com" {type master; file "/etc/namedb/blockeddomain.hosts";};
zone "123go.com" {type master; file "/etc/namedb/blockeddomain.hosts";};
zone "123presta.com" {type master; file "/etc/namedb/blockeddomain.hosts";};


Subscribe Now For Zone Access.

Flat rate subscription. Select a membership option & subscribe.



Select Payment Option



  • You will be issued a username and password.
  • You will be granted access to our member area.
  • 5 Year Membership Option now available.
  • For lifetime membership options click here.clipart
  • Contact us if you would like a pre order invoice.

Disclaimer: All sales are final, we do not issue refunds. Cancel your subscription anytime.


Competitor website urlblacklist.com down for over a week.

urlblacklist.com is down
urlblacklist.com not loading

Well we hate to kick sand in the face of a competitor when they are down, ( actually no, we love it ) but seriously, we have been receiving reports that they have been down for over a week now, and lets be honest, downtime is a sin, and an unforgivable one. Being a provider of critical web filtering services requires a high degree of commitment and entails a responsibility to those whom depend on you to continuously produce quality updates in a timely basis with nearly 100 percent uptime and adequate bandwidth to ensure that you provide your clients, subscribers, and members systems and applications with the level of reliability that is not only expected, but required in 2016 for such a service provider.

The good news is that we do know how to keep our website up and running, and now have available all of our blacklists compressed into a single archive, with identical directory structure for users of urlblacklist to switch to seemlessly. Squidblacklist.org is bringing to market an evolved blacklist generation method, enhanced filters and automated domain removal and addition tools, enforced whitewashing and more, multiple updates daily with bleeding edge malicious updates from multiple sources and partners, we are here to raise the standard and serve you with a higher class of blacklist, and of course, we know how to keep a webserver running.

Sign up today and find out why Fortune 500 Companies, US DoD, Governments, Universities, and Municipalities worldwide are all using blacklists from Squidblacklist.org to enhance a wide range of web filtering applications and platforms.


Blacklisting has Evolved. Subscribe Now!

Flat rate subscription. Select a membership option & subscribe.



Select Payment Option



  • You will be issued a username and password.
  • You will be granted access to our member area.
  • 5 Year Membership Option now available.
  • For lifetime membership options click here.clipart
  • Contact us if you would like a pre order invoice.

Disclaimer: All sales are final, we do not issue refunds. Cancel your subscription anytime.


New Option Available for Urlblacklist/Shallalist Compatible Platforms!


We heard you loud and clear, you wanted our enhanced blacklists in a similar archive/file structure as shallalist & urlblacklist for your web filtering platform, so we finally did it!

pfSense Blacklist Update
pfSense Blacklist Update

Available now to all squidblacklist.org members is the new “Universal Archive Structure Format” for any platform coded for shallalist or urlblacklist file structured archives!
Untitled

This new option is available for immediate download to all Squidblacklist.org visitors in the members area of the website, a subscription is required.

Untitled2

Enhance your web filtering strategy with enhanced blacklists and subscribe today.


Blacklisting has Evolved. Subscribe Now!

Flat rate subscription. Select a membership option & subscribe.



Select Payment Option



  • You will be issued a username and password.
  • You will be granted access to our member area.
  • 5 Year Membership Option now available.
  • For lifetime membership options click here.clipart
  • Contact us if you would like a pre order invoice.

Disclaimer: All sales are final, we do not issue refunds. Cancel your subscription anytime.