Sometimes all you need is a quick crawl within a site, which requires to log in first. There are two main techniques I can think about: One is to POST the login form with your script, and get the necessary cookie setting. The second is to login manually with a browser, and then hand over the web cookies to your script. Let’s start with the first (traditional?) method:

You could use WWW::Mechanize for that (not that I’ve tried), or use the good old LWP. Something like:

#!/usr/bin/perl

use warnings;
use HTTP::Request::Common qw(POST);
use HTTP::Cookies;
use LWP::UserAgent;

$basedir = 'http://www.somesite.com/';

# Create a cookie jar and log into the server

$ua = LWP::UserAgent->new;

$ua->agent("Mozilla/5.0"); # pretend we are very capable browser
$jar = HTTP::Cookies->new();
$ua->cookie_jar($jar);

my $req = POST $basedir.'login.php',
  [ username => 'dracula',
    password => 'bloodisgood'
  ];
print "Now logging in...\n";
$res = $ua->request($req);

# We're not really interested in the result.
# This was only a cookie thing.

die "Error: " . $res->status_line . "\n"
  unless ($res->is_success);

# And now we continue to whatever we wanted to do

The problem is that sometimes the login form is complicated. At times it’s obfuscated intentionally, and uses several tricks to make it difficult to automate the login. Sniffing a successful login (your own, I hope) may be helpful, since the correct POST data is there. If the login is through https, just go through the web page, replace all “https” with “http” and make a fake login. It may not login for real (it usually does), but at least you have the dump info.

But the bottom line is that it may be difficult. In some cases, it’s easier to do the login manually, and continue with your script from there.

Cookie stealing basics

So the plan is to login manually, and then give away the web cookies to your script. The target server can’t tell the difference. In extreme cases, you may need to set up the HTTP headers, so that your script’s and the browser send the same ones exactly. I suggest making your script identify itself with exactly the same user agent header as your browser. Some sites check that, and reject your login if there’s no match. Believe that.

There are several examples for this trick. One is using wget and its –load-cookies flag. It’s quick and dirty, and loads cookies from a cookie file in good old Netscape format. Some browsers can export their cookies to such a file (Firefox uses another format internally, for exampe). But there is still one major problem, and that’s the transient cookies.

Who ate my (transient) cookie?

Every cookie, which is sent from the server to the browser (or whatever you have there) has an expiration date. Some cookies are marked to be erased when the browser (ha!) quits. These are transient cookies.

The thing is, that the browser has no reason to write these cookies to the cookie file on the disk. Why write something that will be erased anyhow? So stealing cookies from the cookie file doesn’t help very much, if the crucial cookies are transient. If you can’t stay logged in to a site after shutting down your browser and getting it back on, that site may be using transient cookies for its session.

The only simple way I know to get a hand on those transient cookies, is to dump them into a file while the browser is alive and kicking. The Export Cookies add-on for Firefox does exactly that.

I suppose that wget works properly with the add-on’s output. I haven’t tried. I wanted to do this with Perl.

Importing Netscape cookies to LWP

It was supposed to be simple. The HTTP::Cookies::Netscape should have slurped the cookie file with joy, and taken things from there. But it didn’t. The module, if I may say so, has a problematic programming interface, which is miles away from the common Perl spirit.

The worst thing about it, is that if no cookies are imported because it didn’t like the cookie file, or didn’t find it at all, there is no notification. An empty cookie jar is silently created. I think that any Perl programmer would expect a noisy die() on that event. I mean, if the cookie file wasn’t read, there’s no point going on in 99% of the cases.

A second problem is with transient cookies. Their expiration time in the cookie file is set to 0 (surprise, surprise, they’re not supposed to survive at all), and the module simply discards them. I don’t blame the module for that, since transient cookies aren’t supposed to be found in a cookie file.

I’ve made the necessary changes for making it to work with the Export Cookies add-on, and got a new module, Exported.pm (click to download). I suggest to copy it next to where you find Netscape.pm in your Perl distribution.

Bottom line, the script looks like this:

#!/usr/bin/perl

use strict;
use warnings;
use LWP::UserAgent;
use HTTP::Cookies::Exported;

my $baseurl = 'http://www.somesite.com/juicydata.php';
my $ua = LWP::UserAgent->new;
# A user agent string matching your browser is a good idea.
$ua->agent('Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7');

# Not loading file in new(), because we don't want writeback
my $cookie_jar = HTTP::Cookies::Exported->new();
$cookie_jar->load('cookies.txt');
$ua->cookie_jar($cookie_jar);

my $req = HTTP::Request->new(GET => $baseurl);

my $res = $ua->request($req);
if ($res->is_success) {
  my $data = $res->content;

  # Here we do something with the data
}
else {
  die "Fatal error: $res->status_line\n";
}

These are just some notes I wrote down for myself, in case I’ll ever want to repeat this mess.  Microsoft Visual Studio 2003 was used for developing the C++ class as well as the setup project. For testing, IE 6 was used.

A button running a script or executable

The simple way is described here. The interesting thing is that it all boils down to setting up some registry keys and values, put a couple of files somewhere, which contain the icon and whatever you want to execute, and off you are. The execution target can be some EXE or a script, including Javascript (!) which is pretty cool. What is less cool, is that the script is pretty crippled. In particular, it can’t manipulate the browser (as of IE6) and I’m not sure about its capabilities in manipulating the current document. So it’s easy, but not very useful.

Now, seriously

I didn’t want to face the facts, but I had no choice: There is no easy way to write a Toolbar button that actually does something useful. A terrible Microsoft document (“the guide” henceforth) offers some clues about how to make a COM DLL for this purpose.

I’ve seen plenty of web sites offering extensions for Firefox, but not for Internet Explorer. I thought the reason was that people with brain prefer Firefox. After writing an extension for IE and Firefox, I realize that the huge difference in difficulty is the probable reason.

Making a “Hello, world” toolbar button

• In Visual Studio, create a “regular” ATL project. Keep it as DLL, uncheck “Attributed” and then check “Allow merging of proxy/stub code“. Otherwise a separate stub/proxy DLL is created, and the IID/CLSID/LIBID symbols aren’t resolved in the h-file. I’m sure there’s a better way to solve this. I’m sure it would take me days to find out how.
• Right-clicking the “Source Files”, add a class. Pick ATL Simple Object, and be sure to set the Options: Aggregation is “No” and IObjectWithSite checked.
• (Build it and see that it is OK. Just so you know it’s possible)
• Now open the .rc file in the solution explorer. Just walk through its properties and make sure that they make sense. The language may be set to something unnecessarily local. In particular, fix the Version->VS_VERSION_INFO so that Company Name and such say something more respectable than TODO-something.

At this point, we sort-of follow Microsoft’s disastrous guide. The first changes are in the .h-file.

• The guide tells us to add the IOleCommandTarget interface. This boils down to adding only two lines (the public declaration and COM_INTERFACE_ENTRY), which are those mentioning IOleCommandTarget explicitly. All the rest is already there, courtesy of Visual Studio.
• Add an #include <atlctl.h> in the beginning.
• And immediately after END_COM_MAP:

  public:
      STDMETHOD(Exec)(const GUID *pguidCmdGroup, DWORD nCmdID,
          DWORD nCmdExecOpt, VARIANTARG *pvaIn, VARIANTARG *pvaOut);
      STDMETHOD(QueryStatus)(const GUID *pguidCmdGroup, ULONG cCmds,
          OLECMD *prgCmds, OLECMDTEXT *pCmdText);

• These methods need to be implemented, of course. For an “Hello, world” application, this is enough (put in .cpp file):

  STDMETHODIMP Cjunkie::Exec(const GUID *pguidCmdGroup, DWORD nCmdID,
      DWORD nCmdExecOpt, VARIANTARG *pvaIn, VARIANTARG *pvaOut)
  {
       MessageBox(NULL, _T("Hello, world"), _T("It works!"), 0);
  
      return S_OK;
  }
  
  STDMETHODIMP Cjunkie::QueryStatus(const GUID* pguidCmdGroup, ULONG cCmds,
      OLECMD prgCmds[], OLECMDTEXT* pCmdText)
  {
  	int i;
  
  	// Indicate that we can do everything!
  
  	for (i=0; i<((int) cCmds); i++)
  		prgCmds[i].cmdf = OLECMDF_SUPPORTED | OLECMDF_ENABLED;
  
      return  S_OK;
  }

• Just a word about the QueryStatus method implemented above: Microsoft describes what this function should do, but I found almost no sample implementation of it. Basically, the purpose of this function is to tell the world what the module is ready to do and what not. I went for an I-can-do-all approach, since any call to a toolbar button means “do your thing”. I’m not sure if this is the right thing to do, but given the promises regarding how narrowminded the calls are expected to be, I think this approach wins. I mean, ask a silly question, get a silly answer.
• At this point, believe it or not, the project should build. The source code up to this stage is listed at the end of this post.

Setup project

• Create a new Setup project. Give it a nice name (it will be the MSI file’s name)
• Put its configuration as Release (as opposed to Debug) and check its “build” checkbox in the Configuration Manager if necessary. So it gets compiled…
• Create a special folder (Windows Folder) to put the files in (too little to open an application folder for)
• Make a subfolder in the Windows folder.
• Put all files there: The DLL (Add->Project Output…->Primary output) and the icon file (read its format here).
• Set the “Register” property of “Primary Output” to vsdrpCOMSelfReg (explained below).
• Open a properties window, and set up the Setup project’s properties.
• Open the Setup project’s Registry Editor and set up the entries. A sample screenshot below.
• Make sure that the ‘DeleteAtUninstall’ property of the extension’s GUID is  ‘True’ (but none of the others’!)

Visual Studio's Setup project: The Registry Editor

Note that the path to the Windows Folder is given as [WindowsFolder]. This makes the value point to where the file was actually installed. A list of such variables can be found here.

And of course, the ‘{4B19…}’ -thing is the button’s class ID (in GUID form). Put your own instead.

• Next, I went for the User Interface Editor. That’s a great opportunity to make the installation process neater. First I removed the “Installation Folder” and “Confirm Installation” steps. The only folder used is the Windows folder anyhow, and with nothing to choose there is nothing to confirm.
• Then a 500x70 BMP file was added to the target directory. This is used as a banner on the installation dialogs by setting the BannerBitmap property for each installation dialog. Since the banner is overlaid with black text, it makes sense to put the logo at the bottom right corner and keep the banner bright.

A note about registration

This was a really bad one. The DLL has to be registered as the owner of the GUID, so that when that GUID is mentioned in the Explorer’s extension list, Explorer knows what DLL to fetch and run “Exec” on. (I suppose the important part is an entry with the key HKEY_CLASSES_ROOT\CLSID\{here comes the GUID}. Or maybe HKEY_LOCAL_MACHINE\SOFTWARE\Classes\CLSID\{here comes the GUID}?)

I wasn’t ready to think about pinpointing the keys to be set up (what do I know about Windows?). Neither was I ready to run Regsvr32 at installation for that (a great opportunity to fail the installation on a hostile computer).

The solution was proposed here: You go to the setup project, pick the item which marks the placement of the DLL (appears as “Primary Output from …”), right-click it and open the Properties page. There you change the “Register” property from vsdrpDoNotRegister (the default) to vsdrpCOMSelfReg.

Now, the project has an RGS file, which it seems wasn’t respected at all, but since the DLL’s registration is now secured, I don’t mind setting up the rest in the Setup project (the “Registry Editor” within a Setup project comes handy for this).

Just a word of caution: In the Setup project’s Registry Editor, you need to line up some of the existing keys as if they should be added, so to bring you to the desired path in the Registry (that is, ‘Microsoft’, ‘Internet Explorer’ and ‘Extensions’). Be sure that the ‘DeleteAtUninstall’ property of these is ‘False’, or you will cause some serious damage to the registry during uninstallation. Also, it’s a good idea to back up the complete registry before starting to play with the Setup project.

To make things a bit more complicated, the property of your GUID key should have the ‘DeleteAtUninstall’ property ‘True’, so that Explorer won’t look for your button after uninstalling.

Interaction with the browser

The “Hello world” application could have been written in Javascript. For some real action, just follow that horrible guide. At this stage, things actually get pretty easy.

• To get a hold of the browser, we need to implement the SetSite method. Copied it right off the guide to the .cpp file.
• The private property declaration, as well as the prototype of Setsite were copied into the .h-file
• At this point, the project built and run (and I could verify that Setsite had been executed once, before the first call to Exec)
• Then I switched to the Exec() method they offered. Basically, I changed nothing except the class name, and put zero instead of  navOpenInNewTab (not supported in my environment, which hasn’t heard about IE7).

Making a POST request

At some point, I decided that I needed to implement a POST request. This was more or the less the stage, at which I realized, that I was actually writing Visual Basic, only in C++. The lesson learned was that maybe I should have started with Visual Basic (YUCK!) to begin with.  And of course, I confirmed an old rule in Microsoft programming: “Prepare to spend a crazy amount of time to implement a trivial feature.”

Implementing POST forces the use of Navigate2, which is a quicksand of SAFEARRAYs and VARIANTs. The available examples show you how to get it done with code that makes you puke and looks like it depends on luck more than some solid API.

To my surprise and delight, I managed to narrow the whole thing down to this relatively-elegant code:

char postdata[] = "postdata=yeah";
CComVariant RequestUrl(_T("http://my.site.com")); 

VARIANT noArg;
noArg.vt = VT_EMPTY;

VARIANT flags;
flags.vt = VT_I4;
flags.lVal = 0;

CComSafeArray<byte> pSar(strlen(postdata), 0);

for (int x=0; x<strlen(postdata); x++)
  pSar.SetAt(x, postdata[x]);

CComVariant postdata(pSar); // Make this an array

m_spWebBrowser->Navigate2(&RequestUrl, &flags, &noArg, &postdata, &noArg);

Sources of Hello World application

Since the most difficult part was to get the application open a dialog box when the button was clicked, here’s the code for it. The main attempt here is to keep it simple.

And by the way, opening dialog boxes seems to be a bad idea. Explorer crashed a few times when the button was clicked before the dialog box was closed. It seems like the response to the Exec() call should be swift.

Header file:

// junkie.h : Declaration of the Cjunkie

#pragma once
#include "resource.h"       // main symbols

#include "myproject.h"
#include
#include  // For handling BSTRs
#include  // For handling BSTRs

// Cjunkie

class ATL_NO_VTABLE Cjunkie :
public CComObjectRootEx,
  public CComCoClass,
  public IObjectWithSiteImpl,
  public IDispatchImpl,
  public IOleCommandTarget
{
 public:
  Cjunkie()
    {
    }

  DECLARE_REGISTRY_RESOURCEID(IDR_JUNKIE)

    DECLARE_NOT_AGGREGATABLE(Cjunkie)

    BEGIN_COM_MAP(Cjunkie)
    COM_INTERFACE_ENTRY(Ijunkie)
    COM_INTERFACE_ENTRY(IDispatch)
    COM_INTERFACE_ENTRY(IObjectWithSite)
    COM_INTERFACE_ENTRY(IOleCommandTarget)
    END_COM_MAP()

    public:
  STDMETHOD(Exec)(const GUID *pguidCmdGroup, DWORD nCmdID,
		  DWORD nCmdExecOpt, VARIANTARG *pvaIn, VARIANTARG *pvaOut);
  STDMETHOD(QueryStatus)(const GUID *pguidCmdGroup, ULONG cCmds,
			 OLECMD *prgCmds, OLECMDTEXT *pCmdText);

  DECLARE_PROTECT_FINAL_CONSTRUCT()

    HRESULT FinalConstruct()
    {
      return S_OK;
    }

  void FinalRelease()
    {
    }

};

OBJECT_ENTRY_AUTO(__uuidof(junkie), Cjunkie)

And application file:

// junkie.cpp : Implementation of Cjunkie

#include "stdafx.h"
#include "junkie.h"

STDMETHODIMP Cjunkie::Exec(const GUID *pguidCmdGroup, DWORD nCmdID,
			   DWORD nCmdExecOpt, VARIANTARG *pvaIn,
			   VARIANTARG *pvaOut)
{
  MessageBox(NULL, _T("Hello, world"), _T("It works!"), 0);

  return S_OK;
}

STDMETHODIMP Cjunkie::QueryStatus(const GUID* pguidCmdGroup, ULONG cCmds,
				  OLECMD prgCmds[], OLECMDTEXT* pCmdText)
{
  int i;

  // Indicate that we can do everything!

  for (i=0; i<((int) cCmds); i++)
    prgCmds[i].cmdf = OLECMDF_SUPPORTED | OLECMDF_ENABLED;

  return  S_OK;
}