simple.
In my previous article "How to track visitors to your site in ASP.NET & C#", I discussed how you can monitor traffic on your ASP.NET website by registering the IP address and Host name with every request.
In this article, I will show how you can write ASP.NET code to use in your site to monitor certain types of visitors, specifically, Google's spider. Simply, all need to do is catch HTTP_USER_AGENT server request parameter with each request, then check if it is named "Googlebot". Here is how to do it:
if (HttpContext.Current.Request.ServerVariables["HTTP_USER_AGENT"].Contains("Googlebot"))
{
// Welcome Google's bot, I'm a big fan! I'm gonna tell
// my boss you are here :)
}
The question now is Where to put the above code?
As I recommended in my previous article about tracking site visitors, there are 2 options. One is to put the code in the Load event of your main site master page. The other option is to put it in the Application_BeginRequest event. The Application_BeginRequest event can be caught in the Global.asax, which you can add to your site by choosing Add New Item from Visual Web Developer, then choosing Global.asax option. You will also need to attach the Application_BeginRequest event handler in the Application_Start. So now, here is how your code would look like:
public void Application_Start(object sender, EventArgs e)
{
BeginRequest += new System.EventHandler(
(Application_BeginRequest);
}
public void Application_BeginRequest(object sender,
EventArgs e)
{
if (HttpContext.Current.Request.ServerVariables
["HTTP_USER_AGENT"].Contains("Googlebot"))
{
// Welcome Google bot!
}
}
Finally, You want to celebrate Google's bot visit by having your server emailing you about this great event, maybe even sending you a text message on your phone, or even calling you directly (that's also doable too, but I'll discuss it in a another article). For now, we will only email, so here is what we'll add to the code:
new SmtpClient().Send("MySite@xxx.com",
"MyEmail@xxx.com",
"Guess who came to visit!",
"Google is crawling in my Site!");
If you want to get more info about emailing from a web application, or a windows application, check out this article I wrote about how to utilize Google's free gmail server to send emails programmatically from your web application.
Now if you are too worried that some hacker spider out there might call themselves Googlebot in hops to play a joke on you, or maybe you've been telling all your friends about how sad you are because Google bot won't come crawl your site, so one of your very good friends decides to cheer you up by sending their own spider to your site and call it Googlebot... You'd probably want to spoil their attempts by checking the requester's ip address in that case, and comparing it to Google’s IP range, this way you can tell if the spider Google's bot, or one of your good hacker buddies trying to cheer you up! (Or for that matter, some hacker spider trying to spoof your site)
From what I gathered out there, Google's IP range is 64.68.80.0 - 64.68.87.255. To find out how to get the requester's IP address in .NET, go to this article.
So now, the final code to put in your site would look like this:
public void Application_Start(object sender, EventArgs e)
{
BeginRequest += new System.EventHandler(
(Application_BeginRequest);
}
public void Application_BeginRequest(object sender,
EventArgs e)
{
if (HttpContext.Current.Request.ServerVariables
["HTTP_USER_AGENT"].Contains("Googlebot"))
{
// Welcome Google bot! I'm going to let my
// webmaster know that you're here!
new SmtpClient().Send("MySite@xxx.com",
"MyEmail@xxx.com",
"Guess who came to visit!",
"Google is crawling in my Site!");
}
}
And now you can be the first to know when Google's bot visits you! :)
P.S. For all of you PHP programmers, I've seen a PHP article similar to mine above here.