Category Archives: Domain Database

Something smells funny at urlblacklist.com

Competition is great, it drives innovation and gives companies the incentive to improve and the drive needed to continue to improve services and products. And in the spirit of competition sometimes a little dirt gets slung. But it isn’t only serving our personal interests to outline the failures of one of our competitors today. In this case, I believe that it is in fact doing the public a service. And by this I mean spreading awareness, to warn unsuspecting customers of what they are spending their money on, and of course, to convince you. with a little sound reasoning, to purchase our services instead of the competition.

There aren’t many suppliers of domain blacklist data for web filtering platforms and applications, this is one of the primary motivating reasons why we created Squidblacklist.org in 2012. We knew it could be done better. There are however a small handful of other providers remaining. One of these websites is urlblacklist.com, and it is them whom we have chosen to single out for scrutiny in this blog post.

In summary, Urlblacklist.com is a horrible service provider. One whos website was recently down for over 3 months in the 2016 calendar year. As a provider of services to paying clients, this is simply a disgrace. We have been monitoring urlblacklist.com, watched multiple outages, and we know that their domain name changed hands recently, something they did not announce on their website, so we will do it for them, here.

The website urlblacklist.com was down for nearly an entire month in October 2016. Then, a second outage occured that lasted nearly 2 months, beginning on or around Nov 10, lasting through until Dec 30, 2016. Further scrutiny reveals that in Nov of 2016, the owner of urlblacklist.com irresponsibly allowed his domain name to expire, which resulted in another 2 month long outage, in which another company took the opportunity to purchase the domain name, which is now registered under an entity named “Dr Guardian” who one can only assume has taken ownership and reopened the website, and is actively processing payments by unsuspecting customers.

They also do not make any mention of ownership changing hands anywhere on their website, a courtesy any respectable business would extend to its existing clients and the general public. Instead of even acknowledging the domain name ownership change. they choose to deceptively place blame for the outage on a billing issue with their registrar.

And I doubt if anybody knows who is really even operating the website. The owner has never responded to emails, doesn’t seem too care if his website goes down until months later, and I suggest that you should seriously consider switching to Squidblacklist.org if you are a current urlblacklist.com subscriber.

A brief visit to urlblacklist.com shows that the owner would like you to believe the second extended outage of 2016 was brief, a deception which is evident in a recent “news” message claiming that “over Christmas vacation” there was an outage, I guess Christmas vacation begins around Nov 10 and ends December 30th.

One can safely assume by looking at the ancient and truly aweful web design of urlblacklist.com and then research its track record of unreliability, and make some general conclusions about its owner and or operators which are to say, generally not favorable at all. And this lack of integrity can also be found in the quality of their blacklists, or lack thereof, which is of course, what really matters.

Urlblacklist.com is an aging website. I would encourage you to use their lists, and monitor the daily changes, you will know first hand that nearly 60% of the domains in their blacklists do not even resolve, a good indicator that there is very poor technology behind the update processes going on behind the scenes. It becomes clear rather quickly that they are pushing old, recycled domain data by systematically removing a set number of domains and re-adding them back using some crude scripts or something, rotating this data in and out in a way which gives the customer the illusion that updates are taking place.

With manual additions and removals being performed occasionally presumably by human hands. Beyond this however, there is obviously zero innovation taking place at urlblacklist.com. Which is evidently ran by an incompetent individual who is doing the world a disservice by continuing to accept payments for what is, in our opinion, hardly passable even as a purely free service.

Shalla and any other websites referring people to this website are also guilty of doing the general public a disservice by linking to urlblacklist.com because clearly anybody who has actually observed or used the lists from these people should come to the same conclusions that we have, and that is urlblacklist.com is an unreliable provider of services that needs to go away.

Also make note that shady clones of urlblacklist.com are also in existence, these domains are registered by totally different owners.
http://schwela.com is one of them.


If you like what we are doing here and want to support our efforts, please consider subscribing to download all of our blacklists.

Flat rate subscription. For full access to all of our works, select a membership option & subscribe today.



Select Payment Option




Blacklist Processing – Automated Domain Addition & Removal – A.D.A.R. System – Delay Pool Recheck Function

A little peek at some of whats going on behind the scenes.

One of the things that we do when we process the data that we publish, is to add domains, but also to remove unwanted, or delisted domains that match a certain criteria. But its never a good idea to simply delete the data, instead, we place these removed domains into the delay pools, domains that are flagged as no longer resolving, or being redirected to place holders, which is common with domains that are parked or suspended. Once this data is added to a delay pool, the data is then rechecked again and again, if the domains ever come back online, they get added back to the blacklists again before being white washed through filters again later during our daily update processes.

At Squidblacklist.org, we are working to bring you a higher class of blacklist through logic and innovation.


If you like what we are doing here and want to support our efforts, please consider subscribing to download all of our blacklists.

Flat rate subscription. For full access to all of our works, select a membership option & subscribe today.



Select Payment Option



  • You will be issued a username and password.
  • You will be granted access to our member area.
  • 5 Year Membership Option now available.
  • For lifetime membership options click here.clipart
  • Contact us if you would like a pre order invoice.

Disclaimer: All sales are final, we do not issue refunds. Cancel your subscription anytime.


Competitor website urlblacklist.com down for over a week.

urlblacklist.com is down
urlblacklist.com not loading

Well we hate to kick sand in the face of a competitor when they are down, ( actually no, we love it ) but seriously, we have been receiving reports that they have been down for over a week now, and lets be honest, downtime is a sin, and an unforgivable one. Being a provider of critical web filtering services requires a high degree of commitment and entails a responsibility to those whom depend on you to continuously produce quality updates in a timely basis with nearly 100 percent uptime and adequate bandwidth to ensure that you provide your clients, subscribers, and members systems and applications with the level of reliability that is not only expected, but required in 2016 for such a service provider.

The good news is that we do know how to keep our website up and running, and now have available all of our blacklists compressed into a single archive, with identical directory structure for users of urlblacklist to switch to seemlessly. Squidblacklist.org is bringing to market an evolved blacklist generation method, enhanced filters and automated domain removal and addition tools, enforced whitewashing and more, multiple updates daily with bleeding edge malicious updates from multiple sources and partners, we are here to raise the standard and serve you with a higher class of blacklist, and of course, we know how to keep a webserver running.

Sign up today and find out why Fortune 500 Companies, US DoD, Governments, Universities, and Municipalities worldwide are all using blacklists from Squidblacklist.org to enhance a wide range of web filtering applications and platforms.


Blacklisting has Evolved. Subscribe Now!

Flat rate subscription. Select a membership option & subscribe.



Select Payment Option



  • You will be issued a username and password.
  • You will be granted access to our member area.
  • 5 Year Membership Option now available.
  • For lifetime membership options click here.clipart
  • Contact us if you would like a pre order invoice.

Disclaimer: All sales are final, we do not issue refunds. Cancel your subscription anytime.


New Option Available for Urlblacklist/Shallalist Compatible Platforms!


We heard you loud and clear, you wanted our enhanced blacklists in a similar archive/file structure as shallalist & urlblacklist for your web filtering platform, so we finally did it!

pfSense Blacklist Update
pfSense Blacklist Update

Available now to all squidblacklist.org members is the new “Universal Archive Structure Format” for any platform coded for shallalist or urlblacklist file structured archives!
Untitled

This new option is available for immediate download to all Squidblacklist.org visitors in the members area of the website, a subscription is required.

Untitled2

Enhance your web filtering strategy with enhanced blacklists and subscribe today.


Blacklisting has Evolved. Subscribe Now!

Flat rate subscription. Select a membership option & subscribe.



Select Payment Option



  • You will be issued a username and password.
  • You will be granted access to our member area.
  • 5 Year Membership Option now available.
  • For lifetime membership options click here.clipart
  • Contact us if you would like a pre order invoice.

Disclaimer: All sales are final, we do not issue refunds. Cancel your subscription anytime.


Music website/domain blacklist for content filtering published.

We have had a video, and an image blacklist for a while now, so it only seemed right to include a music blacklist. This became clear as it was requested by one of our current members who is responsible for content filtering at an educational facility.

The new Music domain blacklist for web filtering purposes has been added to our existing line up and is available in the ‘all’ archive as well as standalone download, just as you would expect any other blacklist that we offer, this blacklist is also available in multiple formats for broad compatibility with most content filters and url filtering platforms and applications.

Enhance your web filtering strategy with enhanced blacklists and subscribe today.


Blacklisting has Evolved. Subscribe Now!

Flat rate subscription. Select a membership option & subscribe.



Select Payment Option



  • You will be issued a username and password.
  • You will be granted access to our member area.
  • 5 Year Membership Option now available.
  • For lifetime membership options click here.clipart
  • Contact us if you would like a pre order invoice.

Disclaimer: All sales are final, we do not issue refunds. Cancel your subscription anytime.


Case Study – Web Filtering & Blacklist Quality Put To The Test.

DSC00728
Web filtering is an important consideration for any enterprise – it is one of the best-known and most efficient front line defenses against hacker attacks and malicious software. One of Squidblacklist.org customers was using a solution from another vendor which had reached end of life and needed to be replaced.

The system had not been performing to the customer’s satisfaction – it had proven difficult to manage, was not cost-effective, and its limited reporting capacity required an additional application to fill in the gaps in its functionality.

The client carried out an independent evaluation and selected two blacklist providers for deployment on separate Internet links to test the varying degrees of effective filtering. Filtering policies were created based on group membership rather than individual user rules as in the previous installation, and were integrated into Active Directory.This allowed existing support staff to grant Internet access by moving users into relevant Active Directory groups rather than modifying the proxy server configuration.

Improved Web Filtering Performance

Not only did the new Blacklists from Squidblacklist.org enhance the effective application of these appliances and the performance of web filtering for the customer, they also identified a number of websites which had been previously been mistakenly blocked or likewise, websites that should have been block, not blocked at all. The client was thus able to advise the relevant organizations – which included their customers and partner organizations – that their web filtering solutions had been compromised by poor quality blacklists from websites like shalla “secure services” and urlblacklist. These issues were then easily resolved by converting to blacklists by Squidblacklist.org.

The enhanced blacklists also introduced Weaknetlabs Technology which combines the best of conventional tools with new intelligent identification algorithms. ADR automatically tracks and adds or removes different domains. More effectively producing higher class of blacklists, than first generation blacklists from other providers. It also removes the inherent weaknesses in using human-only classification to give you the most up-to-date URL blocking and control.

The customer has since found that this new setup meets their requirements to an infinitely higher degree than their previous setup.


Blacklisting has Evolved. Subscribe Now!

Flat rate subscription. Select a membership option & subscribe.



Select Payment Option



  • You will be issued a username and password.
  • You will be granted access to our member area.
  • 5 Year Membership Option now available.
  • For lifetime membership options click here.clipart
  • Contact us if you would like a pre order invoice.

Disclaimer: All sales are final, we do not issue refunds. Cancel your subscription anytime.


Using Squidguard and Pfsense to Url Filter with Domain Blacklists from Squidblacklist.org

Using Squidguard and Pfsense to Url Filter with Domain Blacklists.

pfsense Logo

URL filtering is one strategy used to filter access to websites based the domain name and/or url. There are several commercial products available for URL or domain content filtering, but you could easily build a very reliable system on your own using SquidGuard and pfSense. SquidGuard is a useful add on package for the Squid proxy server and can be used to filter or redirect web requests on the network.

SquidGuard has a long list of features that can be tailored to fit your needs. It’s also rather fast and does’nt slow down the internet for your clients. If you do need to block access to a list of unwanted websites or only allow access to a whitelist of specific web sites, SquidGuard can certainly assist with this.

SquidGuard is also very flexible, and it is easy to adapt to different applications. If you intend to do basic URL filtering on your home network or if you need to create some complicated rules for a large private or public network SquidGuard can do it.

Before you can put a web filtering proxy under pfSense into production, some configuraation is required. If you are new to pfSense I might recommend reading through the instructions that shit with pfSense.

Install the package SquidGuard Package

SquidGuard & Squid proxy can both be installed using the pfSense package manager. To access the pfSense package manager, click packs on the system menu. Select the tab available packages and scroll down where you will find SquidGuard and Squid proxy individually, click the plus sign next to each item to begin the installation.

Once the installations are complete you will have a new menu item called proxy services/filter.

Blacklists

To set up domain blacklist, open the general settings page ‘Filter Services & Proxy’. Click the checkbox to activate the domain blacklist.

You can use one of several different domain black lists publicly available on the web. You can also find a list of several blacklists from http://www.squidblacklist.org. We have our blacklists available in multiple formats, but likely, youll want the standard directory formatted archives located at the following url. http://www.squidblacklist.org/downloads/squidblacklists/squidblacklist.tar.gz


Subscribe Today – Paypal or Credit Card Accepted.

Flat rate subscription. Select a membership option & subscribe.



Select Payment Option



  • You will be issued a username and password.
  • You will be granted access to our member area.
  • 5 Year Membership Option now available.
  • For lifetime membership options click here.clipart
  • Contact us if you would like a pre order invoice.

Disclaimer: All sales are final, we do not issue refunds. Cancel your subscription anytime.


Excluding URL blacklist

There may be some places that you need to allow your users to access. To prevent these sites from locking can create a new category of destination and add a list of domains or URLs that should not be blocked.

To do this click the target categories tab, and then click the plus sign to add a new category. You must assign a unique name to the new category, the name you choose can not contain spaces.

The target category can filter by domain name, URL, or an expression. Add a domain site will grant access to the main site and all its sub-pages. Entering a URL allows access only to that exact website. Expressions allow you to grant based on certain keywords access.

When finished, click Save, and then back to the common ACL tab or group (wherever that created the rule) and select and action whitelist for your new category.

You can also use this same method to add additional sites to its blacklist.

Filtering by Expression

In addition to the domain and URL filtering SquidGuard can create filters using regular expressions. These types of filters are great when you want to search for specific text strings in a URL to make a decision for this search. If you are unfamiliar with regular expressions can be a bit confusing at first, but there are many online resources on the subject, so I will not go into much detail about them in this article.

To create a filter that uses an expression, click the target categories tab, or create a new category or edit an existing one. Enter the expression you want to filter in the expression box and then click Save. Then go back to the common or group ACL tab and select the action (deny, permit, etc.) for your target category.

Here are some examples of filter expressions are presented. These can be edited according to what to filter. For more useful information about filtering regular expressions http://www.squidguard.org/Doc/Examples review.

Downloads based on file extension block

(* \ /.* \ (Zip | .. Rar | exe | msi | mpeg | avi))

Block certain TLDs

(.gov | .xxx | Mil | .net)

Block search “bypass proxy” on Google and Yahoo

(.*(google|yahoo).*(search_query|keywords|search|query|q|p)=.*(\+|\%20)*(proxy|bypass).*(\-|\+|\%20).*(proxy|bypass).*)

Programming rules & Time-based rules

SquidGuard also allows you to apply URL filtering based on schedules. Times are useful for applying rules at different times during the day, or only on certain days of the week.

For example, you could apply URL filtering rules strict office hours and automatically disable the rules after 17:00. If you are filtering your home network you may not want the children to visit certain sites during the school week, this is another example in which a time-based rule would be used.

To create a rule-based time, click the time tab and then click the plus sign to create a new schedule. You can create as many different times as you need.

Schedules can be applied using the ACL Groups tab. Create a new ACL or edit an existing group, then click the “time” drop-down box select the schedule you created.

Do not forget to click Apply on the General tab for the settings to take effect.

Conclusion

Commercial Web filtering devices can be very expensive and difficult to handle. PfSense SquidGuard and are completely free and very powerful. SquidGuard offers many other features that are not covered in this center. For more detailed information, visit SquidGuard.org and check out the documentation section. Also be sure to check out some of my other centers to learn about more ways to use pfSense on your network.
Guidelines pfSense

pfSense Bandwidth – Setting Traffic Shaping
Heavy users wide band can slow the entire network. This center will show you how to use pfSense to set traffic shaping to prioritize Internet traffic.
Dual Wan Router – How to load balance with pfSense
Dual WAN Routers allow you to increase the bandwidth of the Internet on your network by combining two Internet connections. Using pfSense can turn an old computer into a powerful multi WAN router.
How to set up a transparent proxy using squid pfSense
Proxy servers can be very useful for improving the speed of an Internet connection by caching, log Internet usage, or filter traffic. Learn how to set up a transparent proxy using pfSense.

WPA Wordlist Created fom our Master Domain Database.

WPA WORDLIST IS NOW AVAILABLE AGAIN. WOOOOOHOOOo!

13043338_10201884285744369_9022510362039297071_n

I was feeling nostalgic tonight and converted our master domain db into a HUGE wordlist for WPA cracking purposes, all lines less than 9 characters or greater than 32 removed. Should be a nice addition to any wireless crackers wpa wordlist collection.

WPA Wordlist contains 144,203,232 lines.
Filesize – 548MB compressed. 2.24GB decompressed.

We give you. SBL-WPA

It can be downloaded gratis from the following link. Please link to us or feel free to redistribute this file as you wish.

http://www.squidblacklist.org/downloads/SBL-WPA.tar.gz

Signed,

Benjamin E. Nichols
http://www.squidblacklist.org

Master Domain Database Now Available For Download

Today we are announcing the immediate availability of our master domain database, it is a huge domain list we are making available for download. The current total domain count which is currently above 200 Million top level domains is increasing every day with every daily update.

We do charge a small fee for access to this file, but at $29.99, we are confident that you will agree, it is a bargain compared to the other options out there, websites who charge $199-$499 for comparable databases.

Our Master Domain-DB can be purchased NOW!

Links to other sources for big domain lists are as follows.

Our rate smashes the competitions offerings. And it would be our pleasure to serve you.

domains-index.com
www.whoisxmlapi.com
www.yougetsignal.com
http://www.whythink.co.uk
viewdns.info

Web filtering with RouterOS web proxy leveraging domain blacklists – how to.

 

Web filtering with a RouterOS enabled device leveraging domain blacklists from Squidblacklist.org is not difficult to achieve.  By simply converting our existing blacklists that we publish for other platforms, into a Mikrotik *.rsc script , we can then reliably filter web traffic using a RouterOS enabled device, as well as many low cost RouterBoard devices.

 

 

What does a RouterOS web proxy blacklist look like?

Piracy rsc blacklist.
Piracy rsc blacklist script.

 

I will quickly walk you through a brief illustrated description of how one would upload a blacklist script via winbox and schedule its execution on an RB750gl. Download  and test drive a copy of this  blacklist from the links we have provided at the bottom of this article.

 

Upload the blacklist via winbox (drag n drop)

Upload blacklist rsc
Uploading the piracy blacklist rsc via winbox.

 

Of course in production you could and likely would be using automated procedures to do this, using ftp, ssh, or whatever you like for file transfer,  but for this demonstration we are just going to drop and drop.

 

 Open the system scheduler and create a task.

Create task in system scheduler
Create a task.

Here we create a task in system scheduler and pick a time to execute it.  In this case, we are running the following  ‘import tik-piracy.rsc’  command.

 

RB750gl CPU load during import and duration.

Cpu load during run, RB750g
Cpu load during tik-piracy run, RB750g.

 

It is important for me to point out that the cpu on this RB750GL was under 100% load during this import process, and lasted approximately 2-3 minutes in duration.  Around 14,000 domains were loaded from this particular blacklist.

 

System log after blacklist script run completion.

RouterOS Syslog after loading blacklist
RouterOS syslog after loading piracy blacklist.

 

RouterOS Web Proxy access list after blacklist import.

Access list after Loading.
RouterOS web proxy access list after loading piracy blacklist.

Here we can see over 14,000 domains successfully imported into the RB750gl’s web proxy access list, which is now ready to filter against piracy related domains.

System Requirements – Memory Limitations

tikchart
Blacklist compatibility chart.

It is important for us to mention that the system memory, ram usage, is very high using our blacklists, in this example we used one of our smaller size lists, and most of our blacklists are comparable in size and line count, however. The most sought after lists that we publish will not run on a low cost Routerboard device such as the RB750gl, and would result in memory exhaustion, a kernel panic  and thereafter a resulting watchdog timer reboot.  If you attempted to load more than one of these blacklists on a small device such as an RB411/532/133/112/950x type device, this failure would likely be the result. Therefore, we recommend that an x86/x64 PC based platform be used for serious web filtering purposes using RouterOS web proxy.  Alternatively,  one of the more sophisticated, higher end Mikrotik RouterBoard platforms might also suffice.  RouterOS has been proven to have an inherent flaw, or set of flaws that make running our larger lists, porn, proxies, malicious, and prime difficult.  A minimum of 6gb ram is required before attempting to run these lists, and a recommended 8gb. Running these four blacklists on a RouterOS device should be considered experimental until Mikrotik fixes the problems challenging RouterOS.

Obtaining blacklists for RouterOS Web proxy.

These blacklists and more are available for download to our members immediately. A subscription to squidblacklist.org is required.

 Footnotes:


 

  • Blacklist immediate availability from squidblacklist.org for all subscribed members.
  • A Routerboard compatibility chart can be found here.
  • A sample RouterOS web proxy blacklist is available for download here.
  • Mikrotik blacklist conversion tool for windows is available for download gratis.
  • Update script examples are available for download.
  • If you have any questions contact us.

Subscribe Today – Paypal or Credit Card Accepted.

Flat rate subscription. Select a membership option & subscribe.



Select Payment Option



  • You will be issued a username and password.
  • You will be granted access to our member area.
  • 5 Year Membership Option now vvailable.
  • For lifetime membership options click here.clipart
  • Contact us if you would like a pre order invoice.

Disclaimer: All sales are final, we do not issue refunds. Cancel your subscription anytime.