Tech News

Mining Website Logs For Hidden Robots.

We worked on a project recently, where we had found that a client’s entire site was screen scraped and put up at a different location without their permission. This of course led to a drop in search engine traffic as the new site was considered duplicate content.

The client receives a ton of traffic each day and none of the traditional visitor analytic tools (Google Analytics/Piwik) were providing any real insight into what and who was crawling their site in general.

We decided to bring out our old tool, Perl, to help data mine for the robots.

First, let me state, that by no means will this ever stop or detect every robot that is nefariously crawling your website. Most really crafty developers would mask their robot in their code, by stating that it is actually a Chrome browser running on Windows 7. It would take its time and slowly/carefully/randomly download pages to grab the content.

The purpose of our exercise below was to see what other reported robots where mining our site for links and/or possible competitive link ranking.

Read More

The more we have gained experience on the OpenStack platform, the more we began wondering how large of a Rackspace cloud we could build on re-sold enterprise level hardware. This would be perfect for POC (proof of concept) labs, R&D computing development, or even a small office computing cloud. The goal was to get a cloud up and running with little cost, without sacrificing enterprise server features, but also not expecting a full NOC center. We would assume that UPS’s or networking equipment are already provided and/or not necessary.

The only place we have found to purchase reliable second-hand equipment, is eBay. So we logged on to find out how big of a Rackspace OpenStack Private Cloud we could build on a $10,000 budget using just “Buy it now” pricing and flat shipping.

Read More

Recently we were working on a software integration project, whereby we were migrating a php based application onto a CentOS system from an Ubuntu system. The platform requires the ability to call out to GPG and decrypt certain text elements as part of its “data at rest” encryption policy.

Prior to the move, the system used the “PECL” based GNUPG module and it worked farily efficiently  Moving to the CentOS platform presented a few challenges, one of which being that the underlying GPG software was upgraded to the latest version (2). This version does NOT work with the existing PECL module, as it does not know how to send the passphrases to the new GPG pinentry interface.

Thankfully, there is another PEAR based module that was purported to work with GPG2, called Crypt GPG. The install and setup was extremely simple and we used the latest beta version (1.4.0b4) because it had the necessary pass-phrase hand-off that we needed to get working again.

Read More

Apparently there was some sort of global attack against a variety of hosting providers last night (4/11/2013) targetting users of WordPress.  The massive botnet attack targeted  WordPress accounts and went directly after the login screen.

The performance degradation you experienced today was caused by a global attack against websites running the WordPress content managment system. The attack was directed at the login page for these sites. We have blocked external access to the login pages and that has corrected the performance issues.

One of our providers gave us that update this morning.  We are still continuing to research the issue and see how Cloudflare was able to mitigate the attacks.

Read More