This post examines what the Microsoft’s compiler does in response to a variety of special functions that implement atomic operations and memory barriers. If you program like a civilized human being, that is with spinlocks and mutexes, this is a lot of things you should never need to care about.

I’ve written a similar post regarding Linux, and it’s recommended to read it before this one (even though it repeats some of the stuff here).

To make a long story short:

• The Interlocked-something functions do not just guarantee atomicity, but also function as a memory barrier to the compiler
• Memory fences are unnecessary for the sake of synchronizing between processors
• The conclusion is hence that those Interlocked functions also work as multiprocessor memory barriers.

Trying it out

The following code:

LONG atomic_thingy;

__declspec(noinline) LONG do_it(LONG *p) {
  LONG x = 0;
  LONG y;
  x += *p;
  y = InterlockedExchangeAdd(&atomic_thingy, 0x1234);
  x += *p;

  return x + y;
}

When compiling this as “free” (i.e. optimized), this yields:

_do_it@4:
  00000000: 8B FF              mov         edi,edi
  00000002: 55                 push        ebp
  00000003: 8B EC              mov         ebp,esp
  00000005: 8B 45 08           mov         eax,dword ptr [ebp+8]
  00000008: 8B 10              mov         edx,dword ptr [eax]
  0000000A: 56                 push        esi
  0000000B: B9 34 12 00 00     mov         ecx,1234h
  00000010: BE 00 00 00 00     mov         esi,offset _atomic_thingy
  00000015: F0 0F C1 0E        lock xadd   dword ptr [esi],ecx
  00000019: 8B 00              mov         eax,dword ptr [eax]
  0000001B: 03 C1              add         eax,ecx
  0000001D: 03 C2              add         eax,edx
  0000001F: 5E                 pop         esi
  00000020: 5D                 pop         ebp
  00000021: C2 04 00           ret         4

First thing to note is that InterlockedExchangeAdd() has been translated into a “LOCK XADD”, which indeed fetches the previous value into ECX and stores the updated value into memory. The previous value is stored in ECX after this operation, which is @y in the C code — InterlockedExchangeAdd() returns the previous value.

XADD by itself isn’t atomic, which is why the LOCK prefix is added. More about LOCK below.

What is crucially important to note, is that putting InterlockedExchangeAdd() between the two reads of *p prevents the optimizations of these two reads into one. @p isn’t defined as volatile, and yet it’s read from twice (ptr [eax]).

Another variant, now trying InterlockedOr():

LONG atomic;

__declspec(noinline) LONG do_it(LONG *p) {
  LONG x = 0;
  LONG y;
  x += *p;
  y = InterlockedOr(&atomic, 0x1234);
  x += *p;

  return x + y;
}

Once again, compiled as “free”, turns into this:

_do_it@4:
  00000000: 8B FF              mov         edi,edi
  00000002: 55                 push        ebp
  00000003: 8B EC              mov         ebp,esp
  00000005: 8B 4D 08           mov         ecx,dword ptr [ebp+8]
  00000008: 8B 11              mov         edx,dword ptr [ecx]
  0000000A: 53                 push        ebx
  0000000B: 56                 push        esi
  0000000C: 57                 push        edi
  0000000D: BE 34 12 00 00     mov         esi,1234h
  00000012: BF 00 00 00 00     mov         edi,offset _atomic
  00000017: 8B 07              mov         eax,dword ptr [edi]
  00000019: 8B D8              mov         ebx,eax
  0000001B: 0B DE              or          ebx,esi
  0000001D: F0 0F B1 1F        lock cmpxchg dword ptr [edi],ebx
  00000021: 75 F6              jne         00000019
  00000023: 8B F0              mov         esi,eax
  00000025: 8B 01              mov         eax,dword ptr [ecx]
  00000027: 5F                 pop         edi
  00000028: 03 C6              add         eax,esi
  0000002A: 5E                 pop         esi
  0000002B: 03 C2              add         eax,edx
  0000002D: 5B                 pop         ebx
  0000002E: 5D                 pop         ebp
  0000002F: C2 04 00           ret         4

This is quite amazing: The OR isn’t implemented as an atomic operation, but rather it goes like this: The previous value of @atomic is fetched into EAX and then moved to EBX. It’s ORed with the constant 0x1234, and then the cmpxchg instruction writes the result (in EBX) into @atomic only if its previous value was the same as EAX. If not, the previous value is stored in EAX instead. In the latter case, the JNE loops back to try again.

I should mention that cmpxchg compares with EAX and stores the previous value to it if the comparison fails, even though this register isn’t mentioned explicitly in the instruction. It’s just a thing that cmpxchg works with EAX. EBX is the register to compare with, and it therefore appears in the instruction. Confusing, yes.

Also here, *p is read twice.

It’s also worth noting that using InterlockedOr() with the value 0 as a common way to grab the current value yields basically the same code (only the constant is generated with a “xor esi,esi” instead).

So if you want to use an Interlocked function just to read from a variable, InterlockedExchangeAdd() with zero is probably better, because it doesn’t create all that loop code around it.

Another function I’d like to look at is InterlockedExchange(), as it’s used a lot. Spoiler: No surprises are expected.

LONG atomic_thingy;

__declspec(noinline) LONG do_it(LONG *p) {
  LONG x = 0;
  LONG y;
  x += *p;
  y = InterlockedExchange(&atomic_thingy, 0);
  x += *p;

  return x + y;
}

and this is what it compiles into:

_do_it@4:
  00000000: 8B FF              mov         edi,edi
  00000002: 55                 push        ebp
  00000003: 8B EC              mov         ebp,esp
  00000005: 8B 45 08           mov         eax,dword ptr [ebp+8]
  00000008: 8B 10              mov         edx,dword ptr [eax]
  0000000A: 56                 push        esi
  0000000B: 33 C9              xor         ecx,ecx
  0000000D: BE 00 00 00 00     mov         esi,offset _atomic_thingy
  00000012: 87 0E              xchg        ecx,dword ptr [esi]
  00000014: 8B 00              mov         eax,dword ptr [eax]
  00000016: 03 C1              add         eax,ecx
  00000018: 03 C2              add         eax,edx
  0000001A: 5E                 pop         esi
  0000001B: 5D                 pop         ebp
  0000001C: C2 04 00           ret         4

And finally, what about writing twice to the same place?

LONG atomic_thingy;

__declspec(noinline) LONG do_it(LONG *p) {
  LONG y;
  *p = 0;
  y = InterlockedExchangeAdd(&atomic_thingy, 0);
  *p = 0;

  return y;
}

Writing the same constant value twice to a non-volatile variable. This calls for an optimization. But the Interlocked function prevents it, as expected:

_do_it@4:
  00000000: 8B FF              mov         edi,edi
  00000002: 55                 push        ebp
  00000003: 8B EC              mov         ebp,esp
  00000005: 8B 4D 08           mov         ecx,dword ptr [ebp+8]
  00000008: 83 21 00           and         dword ptr [ecx],0
  0000000B: 33 C0              xor         eax,eax
  0000000D: BA 00 00 00 00     mov         edx,offset _atomic_thingy
  00000012: F0 0F C1 02        lock xadd   dword ptr [edx],eax
  00000016: 83 21 00           and         dword ptr [ecx],0
  00000019: 5D                 pop         ebp
  0000001A: C2 04 00           ret         4

Writing a zero is implemented by ANDing with zero, so it’s a bit confusing. But it’s done twice, before and after the “lock xadd”. Needless to say, these two writes are fused into one by the compiler without the Interlocked statement in the middle (I’ve verified it with disassembly, 32 and 64 bit).

Volatile?

In Microsoft’s definition for the InterlockedExchangeAdd() function (and all other similar ones) is that the first operand is a pointer to a LONG volatile. Why volatile? Does the variable really need to be?

The answer is no, if all accesses to the variable is made with Interlocked-something functions. There will be no compiler optimizations, not on the call itself, and the call itself is a compiler memory barrier.

And it’s a good habit to stick to these functions, because of this implicit compiler memory barrier: That’s usually what we want and need, even if we’re not fully aware of it. Accessing a shared variable almost always has a “before” and “after” thinking around it. The volatile keyword doesn’t protect against reordering optimizations by the compiler (or otherwise).

But if the variable is accessed without these functions in some places, the volatile keyword should definitely be used when defining that variable.

More about LOCK

It’s a prefix that is added to an instruction in order to ensure that it’s performed atomically. In many cases, it’s superfluous and sometimes ignored, as the atomic operation is ensured anyhow.

From Intel’s 64 and IA-32 Architectures Software Developer’s Manual, Volume 2 (instruction set reference) vol. 2A page 3-537, on the “LOCK” prefix for instructions:

Causes the processor’s LOCK# signal to be asserted during execution of the accompanying instruction (turns the instruction into an atomic instruction). In a multiprocessor environment, the LOCK# signal ensures that the processor has exclusive use of any shared memory while the signal is asserted.

The manual elaborates further on the LOCK prefix, but says nothing about memory barriers / fences. This is implemented with the MFENCE, SFENCE and LFENCE instructions.

The LOCK prefix is encoded with an 0xf0 coming before the instruction in the binary code, by the way.

Linux counterparts

For x86 only, of course:

• atomic_set() turns into a plain “mov”
• atomic_add() turns into “lock add”
• atomic_sub() turns into “lock sub”

Is a memory barrier (fences) required?

Spoiler: Not in x86 kernel code, including 32 and 64 bits. Surprise. Unless you really yearn for a headache, this is the right place to stop reading this post.

The theoretical problem is that each processor core might reorder the storing or fetching of data between registers, cache and memory in any possible way to increase performance. So if one one processor writes to X and then Y, and it’s crucial that the other processor sees the change in Y only when it also sees X updated, a memory barrier is often used. In the Linux kernel, smp_wmb() and smp_rbm() are used in conjunction to ensure this.

For example, if X is some data buffer, and Y is a flag saying that the data is valid. One processor fills the buffer X and then sets Y to “valid”. The other processor reads Y first, and if it’s valid, it accesses the buffer X. But what if the other processor sees Y as valid before it sees the data in X correctly? A store memory barrier before writing to Y and a read memory barrier before reading from X ensures this.

The thing is, that the Linux kernel’s implementation of smp_wmb() and smp_rbm() for x86 is a NOP. Note that it’s only the smp_*() versions that are NOPs. The non-smp fences turn into opcodes that implement fences. Assuming that the Linux guys know what they’re doing (which is a pretty safe assumption in this respect) they’re telling us that the view of ordering is kept intact across processor cores. In other words, if I can assure that X is written before Y on one processor, then Intel promises me that on another processor X will be read with the updated value before Y is seen updated.

Looking at how Microsoft’s examples solve certain multithreading issues, it’s quite evident that they trust the processor to retain the ordering of operations.

Memory fences are hence only necessary to ensure the ordering on a certain processor on x86. On different architectures (e.g. ARM v7) smp_wmb() and smp_rbm() actually do produce some code.

When are these fences really needed? From Intel’s 64 and IA-32 Architectures Software Developer’s Manual, Volume 2 (instruction set reference) vol. 2A page 4-22, on the “MFENCE” instruction:

Performs a serializing operation on all load-from-memory and store-to-memory instructions that were issued prior the MFENCE instruction. This serializing operation guarantees that every load and store instruction that precedes the MFENCE instruction in program order becomes globally visible before any load or store instruction that follows the MFENCE instruction. The MFENCE instruction is ordered with respect to all load and store instructions, other MFENCE instructions, any LFENCE and SFENCE instructions, and any serializing instructions (such as the CPUID instruction).

That didn’t answer much. I searched for fence instructions in the disassembly of a Linux kernel compiled for x86_64. A lot of fence instructions are used in the initial CPU bringup, in particular after setting CPU registers. Makes sense. Then there are several other fence instructions in drivers, which aren’t necessarily needed, but who has the guts to remove them?

Most interesting is where I didn’t find a single fence instruction: In any of the object files generated in kernel/locking/. In other words, Linux’ mutexes and spinlocks are all implemented without any fence. So this is most likely a good proof that they aren’t needed for anything else but things related to the CPU state itself. I guess. For a 64-bit x86, that is.

Going back to Microsoft, it’s interesting that their docs for userspace Interlocked functions say “This function generates a full memory barrier (or fence) to ensure that memory operations are completed in order”, but not those for kernel space. Compare, for example InterlockedOr() for applications vs. the same function for kernel. Truth is I didn’t bother to do the same disassembly test for application code.

Some barriers functions

(or: A collection of functions you probably don’t need, even if you think you do)

• KeFlushWriteBuffer(): Undocumented and rarely mentioned, intended for internal kernel use. Probably just makes sure that the cache has been flushed (?).
• KeMemoryBarrier(): Calls _KeMemoryBarrier(). But in wdm.h, there’s an implementation of this function, calling FastFence() and LoadFence(). But these are just macros for __faststorefence and _mm_lfence. Looked at next.
• _mm_lfence() : Turns into an lfence opcode. Same as rmb() in Linux.
• _mm_sfence(): Turns into an sfence opcode. Same as wmb() in Linux.
• _mm_mfence(): Turns into an mfence opcode.

I’ve verified that the _mm_*fence() builtins generated the said opcodes when compiled for x86 and amd64 alike. See some experiments on this matter below.

The deprecated _ReadBarrier(), _WriteBarrier() and _ReadWriteBarrier() produce no code at all. MemoryBarrier() ends up as a call to _MemoryBarrier().

Experimenting with fence instructions

(or: A major waste of time)

This is the code compiled:

__declspec(noinline) LONG do_it(LONG *p) {
  LONG x = 0;
  x += *p;
  _mm_lfence();
  x += *p;

  return x;
}

With a “checked compiation” this turns into:

_do_it@4:
  00000000: 8B FF              mov         edi,edi
  00000002: 55                 push        ebp
  00000003: 8B EC              mov         ebp,esp
  00000005: 51                 push        ecx
  00000006: C7 45 FC 00 00 00  mov         dword ptr [ebp-4],0
            00
  0000000D: 8B 45 08           mov         eax,dword ptr [ebp+8]
  00000010: 8B 4D FC           mov         ecx,dword ptr [ebp-4]
  00000013: 03 08              add         ecx,dword ptr [eax]
  00000015: 89 4D FC           mov         dword ptr [ebp-4],ecx
  00000018: 0F AE E8           lfence
  0000001B: 8B 55 08           mov         edx,dword ptr [ebp+8]
  0000001E: 8B 45 FC           mov         eax,dword ptr [ebp-4]
  00000021: 03 02              add         eax,dword ptr [edx]
  00000023: 89 45 FC           mov         dword ptr [ebp-4],eax
  00000026: 8B 45 FC           mov         eax,dword ptr [ebp-4]
  00000029: 8B E5              mov         esp,ebp
  0000002B: 5D                 pop         ebp
  0000002C: C2 04 00           ret         4

OK, this is too much. There is no ptimization at all. So let’s look at the “free” compilation instead:

_do_it@4:
  00000000: 8B FF              mov         edi,edi
  00000002: 55                 push        ebp
  00000003: 8B EC              mov         ebp,esp
  00000005: 8B 45 08           mov         eax,dword ptr [ebp+8]
  00000008: 8B 08              mov         ecx,dword ptr [eax]
  0000000A: 0F AE E8           lfence
  0000000D: 8B 00              mov         eax,dword ptr [eax]
  0000000F: 03 C1              add         eax,ecx
  00000011: 5D                 pop         ebp
  00000012: C2 04 00           ret         4

So clearly, the fence command made the compiler read the value from memory twice, as opposed to optimizing the second read away. Note that there’s no volatile keyword involved. So except for the fence, there’s no reason to read from *p twice.

The exact same result is obtained with _mm_mfence().

Trying with _mm_sfence() yields an interesting result however:

_do_it@4:
  00000000: 8B FF              mov         edi,edi
  00000002: 55                 push        ebp
  00000003: 8B EC              mov         ebp,esp
  00000005: 8B 45 08           mov         eax,dword ptr [ebp+8]
  00000008: 8B 00              mov         eax,dword ptr [eax]
  0000000A: 0F AE F8           sfence
  0000000D: 03 C0              add         eax,eax
  0000000F: 5D                 pop         ebp
  00000010: C2 04 00           ret         4

*p is read into eax once, then the fence, and then it’s added by itself. As opposed to above, where it was read into eax before the fence, then read again into ecx, and then added eax with ecx.

So the compiler felt free to optimize the two reads into one, because the store fence deals only with writes into memory, not reads. Given that there’s no volatile keyword used, it’s fine to optimize reads, which is exactly what it did.

The same optimization occurs if the fence command is removed completely, of course.

For the record, I’ve verified the equivalent behavior on the amd64 target (I’ll spare you the assembly code).

This is my best effort to summarize the steps to attestation signing for Windows drivers (see Microsoft’s main page on this). I’m mostly a Linux guy with no connections inside Microsoft, so everything written below is based upon public sources, trial and (a lot of) error, some reverse engineering, and speculations. This couldn’t be further away from the horse’s mouth, and I may definitely be wrong occasionally (that is, more than usual).

Also, the whole topic of attestation signature seems to be changing all the time, so it’s important to keep in mind that this reflects the situation of May 10th 2021. Please comment below as things change or whenever I got things wrong to begin with.

Attestation signing replaces the method that was available until April 2021, which was signing the driver locally by its author, just like any code signing. With attestation signing, Microsoft’s own digital signature is used to sign the driver. To achieve that, the driver’s .inf and .sys files are packed in a .cab file, signed by the author, and submitted to Microsoft. Typically 10 minutes later, the driver is signed by Microsoft, and can be downloaded back by the author.

Unfortunately, the signature obtained this way is recognized by Windows 10 only. In order to obtain a signature that works with Windows 7 and 8, the driver needs to get through an HLK test.

Signing up to the Hardware Program

This seemingly simple first step can be quite confusing and daunting, so let’s begin with the most important point: The only piece of information that I found present in Microsoft’s output (i.e. their signature add-ons), which wasn’t about Microsoft itself, was the company’s name, as I stated during the enrollment. In other words, what happens during the sign-up process doesn’t matter so much, as long as it’s completed.

This is Microsoft’s general how-to page for attestation signing in general, and this one about joining the hardware program. It wasn’t clear to me from these what I was supposed to do, so I’ll try to offer some hints.

The subscription to the Hardware Program can begin when two conditions are met:

• You have the capability to sign a file with an Extended Validation (EV) code signing certificate.
• You have an Azure Active Directory Domain Services managed domain (“Azure AD”).

Obtaining an EV certificate is a bureaucratic process, and it’s not cheap. But at least the other side tells you what to do, once you’ve paid. I went for ssl.com as their price was lowest, and working with them I got the impression that the company has hired people who actually know what they do. In short, recommended. Actually, read this first if you plan working with them.

So what’s the Domain Services thing? Well, this is the explanation from inside the Partner web interface (once it has already been set up): “Partner Center uses Azure Active Directory for identity and access management”. That’s the best I managed to find on why this is necessary.

For a single user scenario, this boils down to obtaining a domain name like something.onmicrosoft.com from Microsoft. It doesn’t matter if the name turns out long and daunting: It doesn’t appear anywhere else, and you’re not going to type it manually.

So here’s what to do: First thing first, create a fresh Microsoft account. Not really necessary if you already have one, but there’s going to be quite some mail going its way (some of which is promotional, unless you’re good at opting out).

Being logged into that account, start off on Azure’s main page. Join the 12-month free trial. It’s free, and yet you’ll need to supply a valid credit card number in the process. As of writing this, I don’t know what happens after 12 months (but see “Emails from Azure calling for an upgrade” below on developments).

The next step is to create that domain service. I believe this is Microsoft’s page on the topic, and this is the moment one wonders why there’s talk about DNSes and virtual networks. Remember that the only goal is to obtain the domain name, not to actually use it.

And here comes the fuzzy part, where I’m not sure I didn’t waste time with useless operations. So you may try following this, as it worked for me. But I can’t say I understand why these seemingly pointless actions did any good. I suspect that the bullets in italics below can be skipped — maybe it’s just about creating an Azure account, and not necessarily allocate resources?

So here are the steps that got me going:

• Log in to your (new?) Azure account.
• Go to Azure’s Portal (or click the “Portal” link at the top bar on Azure’s main page)

• Click “Create a resource” (at the top left) and pick Azure AD Domain Services.
• For Resource Group I created a new one, “the_resource_group”. I guess the name doesn’t matter.
• The DNS name doesn’t matter, apparently. yourcompany.onmicrosoft.com or something. It’s not going to appear anywhere.
• I set SKU set to Standard, as it appeared to be the least expensive one.
• After finishing the setup, it took about an hour for Azure to finish the configuration. Time for a long and well deserved coffee break.
• But then it complained that I need to set up DNSes or something. So I went along with the automatic fix.

(end of possibly useless steps)

• There’s this thing on the Register for the Hardware Program page saying that one should log in with the Global administrator account. This page defines Azure AD Global administrator as “This administrator role is automatically assigned to whomever created the Azure AD tenant”. So apparently for a fresh Azure account, it’s OK as is.
• At this point, you’re hopefully set to register to the Hardware Developer Program. After clicking “Next” on the landing page, you’ll be given the choice of “Sign in to Azure AD” or “Create a new directory for free”. The Azure AD is already set up, so log in with the account just created.
• A word about that “Create a new directory for free” option. To make things even more confusing, this appears to be a quick and painless shortcut, however in my case I got “This domain name is not available” for any domain name I tried with. Maybe I missed something, but this was a dead end for me. This is the page I didn’t manage to get through. Maybe your luck is better than mine. So this is why I created the Azure AD first, and then went for registration.
• Going on with the registration, you’re given a file to sign with your EV certificate. I got a .bin file, but in fact it had .exe or .sys format. So it can be renamed to .exe and used with cloud signature services (I used eSigner). BUT do this only if you’re going to sign the .cab files with the same machinery, or you’ll waste a few hours wondering what’s going on. Or read below (“When the signature isn’t validated”) why it was wrong in my case.
• And this is the really odd thing: Inside the Microsoft Partner Center, clicking the “your account” button (at the top right) it shows the default directory in use. At some point during the enrollment procedure, the link with the Azure AD I created was made (?), but for some reason, the default directory shown was something like microsoftazuremycompany.onmicrosoft.com instead of mycompany.onmicrosoft.com, which is the domain I created before. This didn’t stop me from signing a driver. So if another directory was used, why did I create one earlier?

After all this, I was set to submit drivers for signature: From this moment on, the entry point for signing drivers is the Microsoft Partner Center dashboard’s main page.

Emails from Azure calling for an upgrade

To make a long story short, quite a few emails arrived on behalf of Microsoft Azure, urging me to “upgrade” my account, i.e. to allow charging my credit card for Azure services. I ignored them all, and had no issues continuing to sign drivers.

And now to the details.

A day after signing up to Azure, I discovered that $20 had been reduced from my promotional free trial credit. Apparently, I had enabled stuff that would have cost real money. So I deleted the resources I had allocated in Azure. This includes deleting the mycompany.onmicrosoft.com domain, which was obviously ignored by the Partner Center. It was about deleting the the resource group (which contained 7 elements, with the domain included): Just clicking on the resource group in the main portal page, and then Delete Resource Group at the top. It took several minutes for Azure to digest that.

About a month later, I got a notification from Azure urging me to upgrade my account: It went

You’re receiving this email because your free credit has expired. Because of this, your Azure subscription and services have been disabled. To restore your services, upgrade to pay-as-you-go pricing.

Scary, heh? Does “services have been disabled” mean that I’m about to lose the ability to sign drivers?

Once again, “upgrade” is a cute expression for giving permission to charge the credit card that I had to give during submission. The details of which can’t be deleted from the account, unless I submit another, valid one, instead.

As a side note, it turned out that I had a Network Watcher group activated. Maybe I missed it earlier, and maybe it was somehow added. So I deleted it as well. But it’s not clear if this was related to the fact that the credits expired, whatever that means.

A few days on, another mail from Azure, basically the same, urging me to upgrade. One day after that, came an invoice. How come? I haven’t approved any payment. So it was an invoice on 0.00 USD. Zero. Why it was sent to me, is still unclear.

And finally, roughly two months after the initial subscription, I got a “We’re sorry to see you go” email from Azure, saying “Your free credit expired on (this and this date), and because of this we’ve deleted your subscription and any associated data and services”. Uhhm. What about driver signing? Well, I’ve spoiled the suspension above.

Two weeks after getting this last email, I deleted all cookies on my browser that were related to Microsoft, logged into my account at the Partner Center and submitted a driver for signature. The process went through smoothly.

Checking my Azure cloud account, it seemed to had been reset to its starting state, even with a suggestion to start another $200 free trial credit round. Detaching my credit card was however still impossible.

So apparently, there’s no problem just ignoring these emails, and continue signing forever. Emphasis on “apparently”.

Overview of the signature process

To make a long story short, you prepare a .cab file with the driver’s files, sign it with your EV Certificate, upload it to the Hardware Dashboard, and get it back after 10 minutes with Microsoft’s digital signatures all over the place.

So instead of signing the driver yourself, you pack the whole thing neatly, and send it to Microsoft for adding the .cat file and signing the drivers. And yet, you must sign the .cab file to prove that you’re the one taking responsibility for it. It’s Microsoft’s signature on the driver in the end, but they know who to bash if something goes wrong.

.cab files are exactly like .zip files, in the sense that they contain a directory tree, not just a bunch of files. Unfortunately, when looking at .cab files with Windows’ built-in utilities, the directory structure isn’t presented, and it looks like a heap of files. This holds true both when double clicking a .cab file and when using expand -D, from Windows XP all the way to Windows 10. Ironically enough, double-clicking a .cab file with Linux desktop GUI opens it correctly as a directory tree.

It’s important to consider .cab files like .zip, with hierarchy, because the way the driver is submitted is by organizing the files in directories exactly as they appear in the driver package for release, minus the .cat file. So what really happens is that Microsoft uncompresses the .cab file like a .zip, adds the .cat file and then performs the digital signing. It then compresses it all back into a .zip file and returns it back to you. The files remain in the same positions all through.

I guess the only reason .zip files aren’t uploaded instead of .cab, is that signtool doesn’t sign zips.

Some people out there, who missed this point, got the impression that the signing is done for each architecture separately. That’s possible, but there’s no reason to go that way. It’s just a matter of preparing the file hierarchy properly.

Preparing the .cab file

For reference, this is Microsoft’s short page on makecab and very long page on .cab files (which begins with cabarc, but goes on with makecab).

First, set up a .ddf file, looking something like this:

.Set CabinetFileCountThreshold=0
.Set FolderFileCountThreshold=0
.Set FolderSizeThreshold=0
.Set MaxCabinetSize=0
.Set MaxDiskFileCount=0
.Set MaxDiskSize=0
.Set CompressionType=MSZIP
.Set Cabinet=on
.Set Compress=on

;Specify file name for new cab file
.Set CabinetNameTemplate=thedriver.cab
.Set DiskDirectoryTemplate= ; Output .cab files into current directory

.Define pathtodriver=thedriver-dir

.Set DestinationDir=thedriver
%pathtodriver%\thedriver.inf
.Set DestinationDir=thedriver\i386
%pathtodriver%\i386\thedriver.sys
.Set DestinationDir=thedriver\amd64
%pathtodriver%\amd64\thedriver.sys

The .cab file is then generated with something like

> makecab /f thedriver.ddf

“makecab” is in Window’s execution path by default.

In my case of transitioning from self-signed drivers to attestation signature, there was already a script that generated the directory ready for releasing the driver. So the change I made was not to copy the .cat file into that directory, and instead of signing the .cat file, create a .cab.

The .ddf file above relates to a driver released for Intel architecture, 32 and 64 bits. The subdirectories in the driver package are i386 and amd64, respectively, as defined in the .inf file.

Changes you should make to the .ddf file:

• Replace all “thedriver” with the name of your driver (i.e. the name of the .inf and .sys files).
• Set “pathtodriver” to where the driver package is. Note that makecab’s /d flag allows setting variables, so the Define directive can be removed, and instead go something like

  > makecab /d pathtodriver=..\driverpackage thedriver.ddf

• Then possibly modify the files to be included. Each DestinationDir assignment tells makecab the directory position to place the file(s) that appear after it. This should match the structure of your release package’s directory structure.
• If the line doesn’t start with a dot, it’s the path to a file to copy into the .cab file. The path can be absolute (yuck) or relative to the current directory.

All in all, the important thing is to form a directory tree of a driver for release in the .cab file.

The .ddf file shown above is a working example, and it includes only the .inf and .sys files. Including a .cat file is pointless, as Microsoft’s signature machinery generates one of its own.

As for .pdb files, it’s a bit more confusing: Microsoft’s main page includes .pdb files in the list of “typical CAB file submissions” (actually, .cat it listed too there), and then these files don’t appear in the .ddf file example on the same page. The graphics showing a tree for multiple package submissions is inconsistent with both.

A .pdb files contains the symbol map of the related .sys file, allowing the kernel debugger to display meaningful stack traces and disassemblies, in particular when analyzing a bugcheck. These files are not included in a driver release, not mentioned in the .inf file, not referenced in the .cat file and are therefore unrelated to the signature of the driver. Technically, Microsoft doesn’t need these files to complete an attestation signature.

Microsoft nevertheless encourages submitters of drivers to include .pdb files. When these file are missing in a driver submission, a popup shows up in the web interface saying “This submission does not include symbols. It is recommended to upload symbols within each driver folder”. This however doesn’t stop the process, and not even delay it, in case you’re slow on confirming it. So it’s up to you if you want to include .pdb’s.

Submitting the .cab file

The command for signing the .cab file is:

> signtool.exe sign /fd sha256 thedriver.cab

Note that timestamping is not required, but won’t hurt. The whole idea with timestamping is to make the signature valid after the certificates expire, but the .cab file is examined soon after the signature is made, and after that it has no more importance.

Note that ssl.com also offers an eSigner tool for signing the .cab file with a simple web interface. Just be sure to have registered with a signature made in eSigner as well, or things will go bad, see “When the signature isn’t validated” below. Or add eSigner’s certificate to the existing subscription.

November 2021 update: You probably don’t want to use eSigner at all. See this post.

Then the submission itself:

• In Microsoft Partner Center’s dashboard, click at “Drivers” on the left menubar. It might be necessary to click “Hardware” first to make this item appear.
• Click the “Submit new hardware” button at the top left to get started.
• Give the submission a name — it’s used just for your own reference, and surely won’t appear in the signed driver package.
• Drag the signed cab file to where it says to.
• The web interface requires selecting Windows releases in a lot of checkboxes. More on this just below.
• Click “Submit” to start the machinery. Once it finishes, successfully or not, it sends a notification mail (actually, three identical mails or so. Not clear why not only one).
• If and when the entire process is completed successfully, the driver can be downloaded: Under “Packages and signing properties”, there’s a “More” link. Click it, and a button saying “Download signed files” appears. So click it, obviously.

Now to the part about selecting Windows versions. It’s an array of checkboxes. This is a screenshot of this stage (click to enlarge):

First, the easy part: Don’t check the two at the top saying “Perform test-signing for X”. It says “Leave all checkboxes blank for Attestation Signing” in fine print above these.

Now we’re left with a whole lot of Windows 10 release numbers and architectures. From a pure technical point of view, there’s no need for this information to perform the signature, since the .inf file contains the information of which architectures are targeted.

Rather, this is the “attestation” part: Just above the “Submit” button, it says “You have completed quality testing of your driver for each of the Operating Systems selected above”. So this is where you testify which platforms you’ve tested the driver with. The deal is that instead of going through the via dolorosa of HLK tests, Microsoft signs the driver for you in exchange to this testimony. Or should I say, attestation.

Just to spell it out: The signature can’t and doesn’t limit itself to specific operating system builds, and it would be insane doing so, as it wouldn’t cover future Windows releases.

I have to admit that in the beginning I misunderstood this part, and tried to select as much as possible. And because my driver wasn’t compiled for arm64, and I had clicked versions saying “ARM64″, the submission was rejected with “thedriver.inf does not have NTARM64 decorated model sections” (in UniversalLog.txt). It was bit of a computer game to check the right boxes and avoid the wrong ones.

So no need to be greedy. Common sense is to test the driver on one operating system release for each architecture. In the example above, it’s for a driver released for Intel architecture, 32 and 64 bits. The checkbox selection reflects testing it with Windows 10 release 1607, 32- and 64-bit architecture. This is the proper way to go.

And yet, for the heck of it I tried submitting the same driver package with a single OS checked (1607 x64). To my surprise, the package was accepted and signed despite my declaration that it hadn’t been tested for the 32-bit version, even though a .sys file for that architecture was part of the package.

All in all, there must be a match between the architectures targeted by the driver (as listed in the .inf file) and those inferred by the selection of OSes. Nevertheless, it seems like Microsoft lets you get away with not testing all of them. In short, checking just one checkbox may be enough, even if the driver supports multiple architectures.

Looking at the signed zip

After receiving back the signed driver, I examined the files. My findings were:

• The .inf file is left completely intact (bytewise identical to the one in the .cab file).
• A signed .cat file was added.
• All .sys files were signed as well (contrary to what most of us do when releasing drivers). This makes the driver eligible for inclusion during boot.

Looking at the digital signatures with an ASN.1 dump utility, it’s appears like the only place there’s something not pointing at Microsoft, is an non-standard spcSpOpusInfo entry in the crypto blob, where the company’s name, appears in wide char format in the programName field (no, I’m not mistaken). This appears to be taken from the “Publisher display name” as it appears in the Account Settings in the Microsoft Partner Center dashboard.

So all in all, there are almost no traces to the fact that the driver’s origin isn’t Microsoft. Except for that entry in the crypto blob, which is most likely invisible unless the signature is analyzed as an ASN.1 file or string searched (with a tool that detects wide char strings). So it appears like all information, except for that “Publisher display name” remains between you and Microsoft.

When the signature isn’t validated

Sometimes, the process fails at the “Preparation” stage. As always on failures, the web interface suggest downloading a “full error report”. That report is a file named UniversalLog.txt file. If it says just “SignatureValidationFailed”, something went wrong with the signature validation.

The solution for this is to make sure that the certificate that was used for signing the .cab file is registered: Within Microsoft Partner Center, click the gear icon at the top right, select “Account Settings” and pick “Manage Certificates” at the left menu bar. That’s where the relevant certificate should be listed. The first time I got to this page, I saw the same certificate twice, and deleted one of those.

In my case the problem was that during the registration, I had made the signature with the cloud app (eSigner), but signed the driver with a local USB key dongle. As it turned out, these have different certificates.

So the solution was to delete the registered certificate from the account, and register the new one by signing a file with the local USB dongle. Doing this is a good idea in any case, because if something is wrong with the signature produced by signtool, it will fail the registration as well. So whether this renewed registration succeeds or fails, it brings you closer to the solution.

Sample certificate chains

For reference, these are examples of certificate chains: One properly signed .cab file and one for a the .cat file that has been attestation signed my Microsoft.

Note the /pa flag, meaning Default Authenticode Verification Policy is used. Or else verification may fail. Also note that the file isn’t timestamped, which is OK for submission of attestation signing.

> signtool verify /pa /v thefile.cab

Verifying: thefile.cab

Signature Index: 0 (Primary Signature)
Hash of file (sha256): 388D7AFB058FEAE3AEA48A2E712BCEFEB8F749F107C62ED7A41A131507891BD9

Signing Certificate Chain:
    Issued to: Certum Trusted Network CA
    Issued by: Certum Trusted Network CA
    Expires:   Mon Dec 31 05:07:37 2029
    SHA1 hash: 07E032E020B72C3F192F0628A2593A19A70F069E

        Issued to: SSL.com EV Root Certification Authority RSA R2
        Issued by: Certum Trusted Network CA
        Expires:   Mon Sep 11 02:28:20 2023
        SHA1 hash: 893E994B9C43100155AE310F34D8CC962096AE12

            Issued to: SSL.com EV Code Signing Intermediate CA RSA R3
            Issued by: SSL.com EV Root Certification Authority RSA R2
            Expires:   Wed Mar 22 10:44:23 2034
            SHA1 hash: D2953DBA95086FEB5805BEFC41283CA64C397DF5

                Issued to: THE COMPANY LTD
                Issued by: SSL.com EV Code Signing Intermediate CA RSA R3
                Expires:   Fri May 03 13:09:33 2024
                SHA1 hash: C15A6A7986AE67F1AE4B996C99F3A43F98029A54

File is not timestamped.

Successfully verified: thefile.cab

Number of files successfully Verified: 1
Number of warnings: 0
Number of errors: 0

One possibly confusing situation is to check if the root certificate exists before ever running this verification on a fresh Windows installation. It may not be there, but then the verification is successful, and the root certificate appears from nowhere. That rare situation is explained in this post.

Next up is the attestation signed .cat file:

> signtool.exe verify /kp /v thedriver.cat

Verifying: thedriver.cat

Signature Index: 0 (Primary Signature)
Hash of file (sha256): ED5231781724DEA1C8DE2B1C97AC55922F4F85736132B36660FE375B44C42370

Signing Certificate Chain:
    Issued to: Microsoft Root Certificate Authority 2010
    Issued by: Microsoft Root Certificate Authority 2010
    Expires:   Sat Jun 23 15:04:01 2035
    SHA1 hash: 3B1EFD3A66EA28B16697394703A72CA340A05BD5

        Issued to: Microsoft Windows Third Party Component CA 2014
        Issued by: Microsoft Root Certificate Authority 2010
        Expires:   Mon Oct 15 13:41:27 2029
        SHA1 hash: 1906DCF62629B563252C826FDD874EFCEB6856C6

            Issued to: Microsoft Windows Hardware Compatibility Publisher
            Issued by: Microsoft Windows Third Party Component CA 2014
            Expires:   Thu Dec 02 15:25:28 2021
            SHA1 hash: 984E03B613E8C2AE9C692F0DB2C031BF3EE3A0FA

The signature is timestamped: Mon May 10 03:10:15 2021
Timestamp Verified by:
    Issued to: Microsoft Root Certificate Authority 2010
    Issued by: Microsoft Root Certificate Authority 2010
    Expires:   Sat Jun 23 15:04:01 2035
    SHA1 hash: 3B1EFD3A66EA28B16697394703A72CA340A05BD5

        Issued to: Microsoft Time-Stamp PCA 2010
        Issued by: Microsoft Root Certificate Authority 2010
        Expires:   Tue Jul 01 14:46:55 2025
        SHA1 hash: 2AA752FE64C49ABE82913C463529CF10FF2F04EE

            Issued to: Microsoft Time-Stamp Service
            Issued by: Microsoft Time-Stamp PCA 2010
            Expires:   Wed Jan 12 10:28:27 2022
            SHA1 hash: AAE5BF29B50AAB88A1072BCE770BBE40F55A9503

Cross Certificate Chain:
    Issued to: Microsoft Root Certificate Authority 2010
    Issued by: Microsoft Root Certificate Authority 2010
    Expires:   Sat Jun 23 15:04:01 2035
    SHA1 hash: 3B1EFD3A66EA28B16697394703A72CA340A05BD5

        Issued to: Microsoft Windows Third Party Component CA 2014
        Issued by: Microsoft Root Certificate Authority 2010
        Expires:   Mon Oct 15 13:41:27 2029
        SHA1 hash: 1906DCF62629B563252C826FDD874EFCEB6856C6

            Issued to: Microsoft Windows Hardware Compatibility Publisher
            Issued by: Microsoft Windows Third Party Component CA 2014
            Expires:   Thu Dec 02 15:25:28 2021
            SHA1 hash: 984E03B613E8C2AE9C692F0DB2C031BF3EE3A0FA

Successfully verified: thedriver.cat

Number of files successfully Verified: 1
Number of warnings: 0
Number of errors: 0

Doing the same with the .sys file yields exactly the same result, with slight and meaningless differences in the timestamp.

Clearly, the certificate chain ends with “Microsoft Root Certificate Authority 2010″ rather than the well-known “Microsoft Code Verification Root”, which is the reason the attestation signature isn’t recognized by Windows 7 and 8.

Microsoft as a Certificate Authority, approving itself all through the chain. It’s quite odd this happened only now.

Crypto is not my expertise. This is a pile of jots I wrote down as I tried to figure out what the Microsoft catalogue file is all about. Not-so-surprising spoiler: It appears to be organized and elegant at first glance, but the more you look into it, it’s a mess. Of the kind that’s caused by someone making a quick hack to solve that little problem very urgently. And repeat.

Sources

• My own introduction to certificates.
• C code for parsing ASN.1 (dumpasn1.c) can be found on this page. Be sure to download dumpasn1.cfg as well.
• A JavaScript ASN.1 parser from this site.
• For the PKCS#7 syntax, see this page.
• The osslsigncode utility (written in plain C) is not just useful for analyzing signed CATs, but is also boilerplate code for manipulations based upon openSSL’s API.

ASN.1 basics

ASN.1 is a protocol for organizing (usually small) pieces of data into a file in a structured and hierarchical manner. For each type of container (e.g. an x.509 certificate), there’s a protocol describing its format, written in syntax somewhat similar to C struct definitions. Exactly like C struct definitions, it may contain sub-structures. But don’t take this analogy too far, because ASN.1 definitions have optional fields, and fields with unknown number of members.

So if you want to follow what everything means, you need the definition for each element that you encounter. These are sometimes defined in the protocol for the relevant container type. Just like C structs, just looking at the data tells you what’s an integer and what’s a string, but their meaning depends on the order they appeared.

And just like a struct may contain other structs, there are objects in ASN.1. These are the catch-all method for inserting elements with arbitrary form.

When an object is encountered, it always has an object ID (OID), which defines its class. In that case, the format and meaning of what’s encapsulated is defined in the object’s definition. It may not be published (e.g. OIDs specific to Microsoft). Note that an OID defines the format of the object, but not necessarily its meaning. Even though OIDs that are used in very specific cases also tell us what they contain.

A few random notes that may help:

• The common binary format is DER. Each item is encoded with a one-byte identifier (e.g 0x30 for SEQUENCE) followed by the length of the item (including everything in lower hierarchies): Below 0x80 it’s the length given as a single byte, otherwise it’s given in Big Endian format. The first byte is the number of bytes to define the length + 0x80, and then it’s the length. After this comes the data.
• Because the length of the item is given explicitly, there’s a lot of freedom for types like INTEGER: It can be a single byte or huge numbers.
• The SEQUENCE item, as its name implies, is a sequence of elements which encapsulates some kind of information. Some fields are optional. In that case, there’s a number in square brackets, e.g. [0], [1], [2] etc. in the format specification. These number in square brackets appear in the parsed output as well, to indicate which optional field is displayed (if there are any).
• Object identifiers are given in dotted format, e.g. 1.2.840.113549.1.7.2 for signedData. In dumasn1′s output, they appear with spaces, e.g.

  OBJECT IDENTIFIER signedData (1 2 840 113549 1 7 2)

  The translation from numeric OID to a meaningful name is possible if dumpasn1 happens to have that OID listed, which is sometimes not the case. Either way, when encountering these, just Google them up for more information.

• There are two ASN.1-defined types for describing time: UTCTime (tag 0x17) and GeneralizedTime (tag 0x18). They can be used interchangeably. Curiously enough, both are given as ASCII strings. Looking for these is helpful for finding certificates in a large blob, as well as time stamps.

DER and PEM formats

In practice, there are two formats out there for crypto data: DER, which is the native binary format, and PEM, which is a base64 representation of a DER binary. The reason I consider DER to be “native” is that when a digital signature is made on a chunk of data, it’s the DER representation of the a chunk of ASN.1 segment that is hashed.

Openssl can be used to convert between the two formats. As openssl’s default format is PEM, use -inform DER and -outform DER as necessary to convince it into playing ball.

For example, converting a certificate from PEM to DER

$ openssl x509 -in mycert.crt -outform DER -out mycert.der

or in the opposite direction:

$ openssl x509 -inform DER -in mycert.der -out mycert2.crt

As for a .cat file (or any other file in DER PKCS#7 format), the same goes

$ openssl pkcs7 -inform DER -in thedriver.cat -out thedriver.pem

and back from PEM to DER:

$ openssl pkcs7 -in thedriver.pem -outform DER -out thedriver.cat

It’s somewhat daunting that there’s no catch-all converter from DER to PEM, so there’s a need to know which kind of crypto creature is converted. The identification is however necessary, as there are headers indicating the type of data in both DER and PER. It would just have been nicer to have this done automagically.

Inspecting ASN.1 files

The absolute winner for dumping blobs is dumpasn1.c. Download it from the link given above, compile it and install it in /usr/local/bin/ or something. Be sure to have dumpasn1.cfg in the same directory as the executable, so that OBJECT IDENTIFIER items (OIDs) get a human-readable string attached, and not just those magic numbers.

Then go

$ dumpasn1 -t thedriver.cat | less

Note that dumpasn1 expects DER format. See above if you have a PEM.

Flags:

• -t for seeing the text of strings
• -hh for long hex dumps
• -u for seeing timestamps as they appear originally

For those looking for instant gratification, there’s an online JavaScript parser, asn1js. In fact, the site allows downloading the HTML and JavaScript sources, and point the browser local files.

And then there’s openssl’s own dumper, which produces data that is less useful for human interaction. Its only real advantage is that it’s most likely already installed. Go something like this (or drop the -inform DER for parsing a PEM file):

$ openssl asn1parse -inform DER -i -in thedriver.cat -dump

Attempting to parse a DER file without the -inform DER flag, the result may be “Error: offset out of range”. It’s a misleading error message, so don’t fall for this one.

The certificates in a .cat file

For a general tutorial on certificates, see this post.

To extract (hopefully all) certificates included in a .cat (or any other PKCS#7) file, in cleartext combined with PEM format, go

$ openssl pkcs7 -inform DER -print_certs -text -in thedriver.cat -out the-certs.txt

A Windows .cat file is just a standard PKCS#7 file, which is a container for signed and/or encrypted data of any sort. The idea behind this format apparently is to say: First, some information to apply the signature on. Next, here are a bunch of certificates that will help to convince the validator that the public key that is used for the validation should be trusted. This part is optional, but it typically contains all certificates that are necessary for the certificate validation chain, except for the root certificate (which validation software mustn’t accept even if it’s present, or else the validation is pointless). And after the (optional) certificate section comes the signature on the content of the first part — the data to sign.

In some situations, signtool improvises a bit on where to put the certificates for validation, in particular those needed for validating the timestamp, and if a second signature is appended. This is contrary to the straightforward approach of putting all and any certificate in the dedicated PKCS#7 section, as discussed below. The question is whether one is surprised that Microsoft diverged from the standard or that it adopted a standard format to begin with.

The consequence of stashing away certificates in less expected places is that openssl utilities that the command above for extracting certificates from a .cat file may miss some of those. The only real way to tell is to look at an ASN.1 dump.

Finding the digital signature in a non-cat file

Signtool creates digital signatures in a similar way even for non-cat files. The quick way to find it is by looking for the following hex sequence (with e.g. “hexdump -C”):

30 82 xx xx 06 09 2a 86 48 86 f7 0d 01 07 02

The part marked in read is the part saying “OBJECT IDENTIFIER 1.2.840.113549.1.7.2″ which means a SignedData object. Even though this data structure is supposed to contain the data it signs, signtool often appends it to the data for signature. Non-standard, but hey, this is Microsoft.

Pay attention to the last bytes of this sequence rather than the first ones. There are other similar OIDs, but the difference is in the last bytes.

The reason I’ve added the four bytes before is that these are the beginning of the SEQUENCE, which the signature always begins with. The 0x82 part means that the two following bytes contain the length of the current chunk (in big Endian). For snipping out the signature, include these four bytes, to conform the PKCS#7 format.

I should also mention that there might be a SignedData object inside the outer SignedData object, due to signtool’s obscure way of including timstamps and/or multiple signatures. In principle, the outer one is the one to process, but it might also make sense to look at the inner object separately, in particular for extracting all certificates that are included.

To create a file that begins with the ASN.1 data, go something like this (if the 30 82 starter appeared in the hexdump at 0xc408):

$ dd if=thedriver.sys of=theblob.bin skip=$((0xc408)) bs=1

.cat file dissection notes

The root object has signedData OID, meaning that it follows the following format:

SignedData ::= SEQUENCE {
  version           Version,
  digestAlgorithms  DigestAlgorithmIdentifiers,
  contentInfo       ContentInfo,
  certificates      [0]  CertificateSet OPTIONAL,
  crls              [1]  CertificateRevocationLists OPTIONAL,
  signerInfos       SignerInfos
}

I won’t go into the depth of each element. To make a long story short, there are three main elements:

• The contentInfo part, containing the data to be signed (a Microsoft catalogList item with file names, their hashes and more). If the CAT file isn’t signed (yet), this is the only part in the file. Note that catalogList contains the timestamp of the .cat file’s creation.
• The certificate part containing a list of certificates, which relate to the direct signature (as well as the timestamp on some versions of signtool). This is just a bunch of certificates that might become useful while evaluating the signature. As mentioned above and below, signtool sometimes puts them in signerInfos instead.
• The signerInfos part, containing a list of signatures on the data in the contentInfo part. But there’s always only one signature here. The timestamp is embedded into this signature. And even if a signature is “appended” with signtool’s /as flag, the additional signature isn’t added to this set, but obscurely shoved elsewhere. See below.

The signature is in the end, as a SignerInfos item.

SignerInfos ::= SET OF SignerInfo

SignerInfo ::= SEQUENCE {
  version                    Version,
  signerIdentifier           SignerIdentifier,
  digestAlgorithm            DigestAlgorithmIdentifier,
  authenticatedAttributes    [0]  Attributes OPTIONAL,
  digestEncryptionAlgorithm  DigestEncryptionAlgorithmIdentifier,
  encryptedDigest            EncryptedDigest,
  unauthenticatedAttributes  [1]  Attributes OPTIONAL
}

It’s easy to spot it in the dump as something like

8805  931:       SET {
8809  927:         SEQUENCE {
8813    1:           INTEGER 1
8816  135:           SEQUENCE {

towards the end.

Curiously enough, signerIdentifier is defined as

SignerIdentifier ::= CHOICE {
  issuerAndSerialNumber  IssuerAndSerialNumber,
  subjectKeyIdentifier   [2]  SubjectKeyIdentifier
}

and what it typically found is issuerAndSerialNumber. In other words, the details of the certificate which confirms the public key (i.e. its serial number) appear in this section, and not those of the signer. The only part that relates to the signer is the serial number.

So in essence, the textual parts in SignerIdentifier should essentially be ignored. To start the certificate chain, begin from the serial number and climb upwards.

The timestamp appears as unauthenticatedAttributes, and is identified as a countersignature (1.2.840.113549.1.9.6) or Ms-CounterSign (1.3.6.1.4.1.311.3.3.1):

9353  383:           [1] {
9357  379:             SEQUENCE {
9361    9:               OBJECT IDENTIFIER
         :                 countersignature (1 2 840 113549 1 9 6)

Just like the signature, it’s given in issuerAndSerialNumber form, so the textual info belongs to the issuer of the certificate. The only informative part is the serial number.

Notes:

• Public keys are typically identified by their serial numbers. This is the part that connects between the key and related certificates.
• Serial numbers appear just as “INTEGER” in the dump, but it’s easy to spot them as strings of hex numbers.
• In the ASN.1 data structure, certificates convey the information on the issuer first and then subject. It’s somewhat counterintuitve.

Looking at the data part of a .cat file

The truth is that dissecting a .cat file’s ASN.1 blob doesn’t reveal more than is visible from the utility that pops up when one clicks the .cat file in Windows. It’s just a list of items, one part consists of the files protected by the catalogue, and the second some additional information (which is superfluous).

For a Windows device driver, the files covered are the .sys and .inf files. In Window’s utility for inspecting .cat files, these files appear under the “Security Catalog” tab. Each file is represented with a “Tag” entry, which is (typically? Always?) the SHA1 sum of the file. Clicking on it reveals the attributes as they appear in the .cat file, among others the thumbprint algorithm (sha1) and the value (which coincides with the “Tag”, just with spaces). Even more interestingly, the File attribute is the file name without the path to it.

In other words, the catalogue doesn’t seem to protect the position of the files in the file hierarchy. The strategy for validating a file seems therefore to be to calculate its SHA1 sum, and look it up in the catalogue. If there’s a match, make sure that the file name matches. But there’s apparently no problem moving around the file in the file hierarchy.

Under the “General” tab of the same utility, there are hex dumps of the DER-formatted data each object, with the OID (all 1.3.6.1.4.1.311.12.2.1) given in the “Field” column for each. The information here is the operating system and the Plug & Play IDs (Vendor & Product IDs) that the driver covers. Which is redundant, since this information is written in the .inf file, which is protected anyhow. That may explain why the presentation of this info in the utility is done so horribly bad.

Multiple signatures

When an additional signature has been added by virtue of signtool’s /as flag (“append signature”), it’s added as an OID_NESTED_SIGNATURE (1.3.6.1.4.1.311.2.4.1) item in the unauthenticatedAttributes, after the timestamp signature of the original signature:

 9740  8031:             SEQUENCE {
 9744    10:               OBJECT IDENTIFIER '1 3 6 1 4 1 311 2 4 1'
 9756  8015:               SET {
 9760  8011:                 SEQUENCE {
 9764     9:                   OBJECT IDENTIFIER
           :                     signedData (1 2 840 113549 1 7 2)

Under this there’s a signedData item (i.e. the same OID as the one encapsulating the entire file), containing no data by itself, but does contain a bunch of certificates, a signature (on what?) and a timestamp, apparently with some Microsoft improvisations on the standard format.

So they basically said, hey, let’s just push another PKCS#7 blob, from beginning to end, minus the CAT data itself, in that place where anyone can do whatever he wants. The correct way would of course have been to add another SignerInfo item to the SignerInfos set, but hey, this is Microsoft.

The takeaway is that this is likely to cause problems, as hacks always do. And not just for us who want to analyze what’s in there. My response to this is to publish two separate driver files if needed, and stay away from these double signatures.

Checking Windows signatures in Linux

For this there’s opensslsigncode. I cloned a copy, and compiled at commit ID c0d9569c4f6768d9561978422befa4e44c5dfd34. It was basically:

$ ./autogen.sh
$ ./configure
$ make

It seemed to complain about curl not being installed, but it was this that was actually needed:

# apt install libcurl4-openssl-dev

Copied osslsigncode to /usr/local/bin/, and then I could check a Windows driver catalog file with

$ osslsigncode verify -in thedriver.cat

The important thing is that it prints out a neat summary of the certificates in the file. Less informative than using openssl to extract the certificates as shown above, and more descriptive than openssl’s output. However the version I tried crashed when faces with a driver with double signatures. Not sure who to blame.

Bought an Orico PCIe adapter for USB 3.0 for testing a USB device I’m developing (PVU3-5O2I). It has the VL805 chipset (1106/3483) which isn’t xHCI compliant. So it works only with the vendor’s own drivers for Windows, which you’ll have to struggle a bit to install.

Update: Be sure to read the comments at the bottom of this post. It appears like some people have had different experiences.

Attempt with Linux

That the device is detected by its class (xHCI), and not by its Vendor / Product IDs.

The following was found in the kernel log while booting:

[    0.227014] pci 0000:03:00.0: [1106:3483] type 00 class 0x0c0330
[    0.227042] pci 0000:03:00.0: reg 0x10: [mem 0xdf000000-0xdf000fff 64bit]
[    0.227104] pci 0000:03:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.227182] pci 0000:03:00.0: System wakeup disabled by ACPI

and

[    0.325254] pci 0000:03:00.0: xHCI HW did not halt within 16000 usec status = 0x14

and then

[    1.474178] xhci_hcd 0000:03:00.0: xHCI Host Controller
[    1.474421] xhci_hcd 0000:03:00.0: new USB bus registered, assigned bus number 3
[    1.505919] xhci_hcd 0000:03:00.0: Host not halted after 16000 microseconds.
[    1.506066] xhci_hcd 0000:03:00.0: can't setup: -110
[    1.506241] xhci_hcd 0000:03:00.0: USB bus 3 deregistered
[    1.506494] xhci_hcd 0000:03:00.0: init 0000:03:00.0 fail, -110
[    1.506640] xhci_hcd: probe of 0000:03:00.0 failed with error -110

The error message comes from xhci_halt() defined in drivers/usb/host/xhci.c, and doesn’t seem to indicate anything special, except that the hardware doesn’t behave as expected.

Update firmware, maybe?

The idea was to try updating the firmware on the card. Maybe that will help?

So I downloaded the driver from the manufacturer and the firmware fix tool from Station Drivers.

Ran the firmware fix tool before installing the driver on Windows. It went smooth. Then recycled power completely and booted Linux again (the instructions require that). Exactly the same error as above.

Went for Windows again, ran the firmware update tool, and this time read the firmware revision. It was indeed 013704, as it should be. So this doesn’t help.

Install driver on Windows 10

Checking in the Device Manager, the card was found as an xHCI controller, but with the “device cannot start (Code 10)”. In other words, Windows’ xHCI driver didn’t like it either.

Attempted installation of the driver. Failed with “Sorry, the install wizard can’t find the proper component for the current platform. Please press OK to terminate the install Wizard”. What it actually means is that the installation software (just downloaded from the hardware vendor) hasn’t heard about Windows 10, and could therefore not find an appropriate driver.

So I found the directory to which the files were extracted, somewhere under C:\Users\{myuser}\AppData\Local\Temp\is-VJVK5.tmp, and copied the USetup directory from there. Then selected xhcdrv.inf for driver installation. It’s intended for Windows 7, but it so happends, that generally drivers for Windows 7 and Windows 10 are the same. It’s the installer that was unnecessarily fussy.

After installing this driver, a “VIA USB eXtensible Host Controller” entry appeared in the USB devices list of the Device Manager, and it said it works properly.

After a reboot, there was “xHCI Root Hub 0″ under “Other Devices” of the Device Manager, with the error message “The drivers for this device are not installed”. It was available under the same USetup directory (ViaHub3.inf).

This added “VIA USB 2 Hub” and “VIA USB 3 Root Hub” to the list of USB devices, and believe it or not, the card started working.

Bottom line: It does work with its own very special drivers for Windows, with a very broken setup procedure.

Apr 2021 update: It turns out that the driver can be downloaded from Gigabyte’s site, as the device is apparently on their GA-78LMT-USB3 motherboard. The file name is mb_driver_via-usb3.exe, and this is the only one I managed to install, and the card works great with it. Other driver packages failed with the installer saying that it “can’t find the proper component”, whatever that meant.

General notes while playing around

• To start Windows 10 allowing installation of unsigned drivers, hold Shift while selecting “Restart” in the Power menu. Windows will appear to restart normally, but enters a menu before shutting down. Go Troubleshoot > Startup Settings The Windows 7-like warning appears during installation, and the driver installation survives the reboot it regular mode. However text saying “Test Mode” appears at the bottom right of the desktop on following boots. Windows may then randomly refuse to use the driver on the basis that it isn’t signed on following reboots.
• It seems like Windows restarts immediately after installing a driver if the installation fails. Message in event log: “The process C:\WINDOWS\system32\mmc.exe (DESKTOP-xxxxx) has initiated the restart of computer DESKTOP-xxxxx on behalf of user DESKTOP-xxxxx\theuser for the following reason: Hardware: Installation (Planned)
• “Read the source”, WDK-style: Read the inc\ddk\wdm.h include file, which supplies a lot of information and hints. Sometimes it’s much better than the docs. There are also sources for the implementation of basic functions such as open(), printf() etc. in Visual Studio’s installation directory.
• A list of status codes can be found here. Possibly relevant in particular: STATUS_UNSUCCESSFUL,  STATUS_NOT_IMPLEMENTED, STATUS_INVALID_PARAMETER, STATUS_NO_MEMORY, STATUS_INVALID_DEVICE_REQUEST, STATUS_DEVICE_BUSY, STATUS_END_OF_FILE, STATUS_ACCESS_DENIED, STATUS_DATA_ERROR, STATUS_NO_SUCH_DEVICE, STATUS_NO_SUCH_FILE, STATUS_OBJECT_NAME_NOT_FOUND,  STATUS_INSUFFICIENT_RESOURCES, STATUS_IO_TIMEOUT, STATUS_FILE_FORCED_CLOSED,  STATUS_CANCELLED, STATUS_NOT_FOUND, STATUS_RETRY, STATUS_RECEIVE_PARTIAL
• IRP’s major determine the dispatch routine launched (as set by DriverEntry). The minor is interpreted by the dispatch routine itself.
• A list of safe string functions: In MSDN’s page
• How to use symbolic links to create a name space: here.
• To see the internal object structure (and symbolic links), download WinObj from somewhere.
• Sniff IRPs in the system: Download IrpTracker.
• MSDN’s explanation on resource rebalancing, why, how and what to do about it.
• On a typical open for write, DesiredAccess=0x00120196 (FILE_GENERIC_WRITE | READ_CONTROL). On an open for read, DesiredAccess=0x00120089 (FILE_GENERIC_READ | READ_CONTROL). This is based upon command line > and <. Trying to “dir” the file gives DesiredAccess=0x00000080.
• The difference between IRP_MJ_CLEANUP and IRP_MJ_CLOSE: IRP_MJ_CLEANUP is called when the file is officially closed (all owners of the file handle have closed it). IRP_MJ_CLOSE is called when the file handle can be removed, which is when all outstanding IRPs have been completed as well. Note that IRP_MJ_CLEANUP dispatch routine must walk through the queue and cancel all requests (Why doesn’t the system do this?). Note that non-I/O IRPs for the file handler may still arrive for the file handler.
• Do implement a handler for IRP_MJ_SET_INFORMATION, so that such an IRP doesn’t fail. Even a yes-yes handler doing nothing but returning success. In particular, Cygwin sends this IRP with FileEndOfFileInformation when writing to a regular file. Go figure.
• Using MmGetMdlByteCount(irp->MdlAddress) to get the number of bytes to read or write is bad: It causes a bugcheck if the number of bytes to read or write is zero. Which is bad behavior from the application’s side, and still.
• Not directly relevant, but user space applications with POSIX functions such as open(), read() close() and friends should include <io.h> rather than <unistd.h> which doesn’t exist on VC++.
• Setting the DeviceType parameter is utterly important, since the native fopen() and open() calls will fail with an “Invalid Argument” error otherwise. The parameter in the call to IoCreateDevice() is ignored since the actual type is taken from the PDO. Hence a line in the INF file for setting up a registry value is necessary.
• When seeking a file, be either with POSIX-style _seek() and friends, or with native SetFilePointer(), this merely updates CurrentByteOffset, but no IRP is issued, so the device has no idea it happened at all. In particular, no IRP_MJ_SET_INFORMATION IRP with class FilePositionInformation is issued, despite what one could expect. At least so it works in Windows 7.
• pnpdtest (which is part of the WDK, and tge wtllog.dll is also somewhere in its directories) should be run with the verifier (standard Windows utility) configured for toughest tests specifically on the driver. Just type “verifier” at command prompt, reboot, and run pnpdtest. To run verifier with a good set of tests, go
  > verifier /volatile /flags 0xbfb /adddriver mydriver.sys
• Unlike Linux, the allocation of PCI resources (BAR addresses and interrupts) is not exclusive. As a result, if a device driver doesn’t deallocate the interrupt resource at unload, and hence the pointer to the ISR remains, attempting to reload the driver will cause a jump to the unloaded ISR pointer and a bugcheck. In Linux this wouldn’t happen, because the newly loaded instance of the driver wouldn’t pass the stage of getting the resources.
• devcon is the old tool for installing and looking at driver information, with source code in the Windows Driver Samples git repo. pnputil should be used instead, according to Microsoft.

Runlevels (IRQLs)

A very short summary (see Microsoft’s page for more):

• PASSIVE_LEVEL: The level of user space applications, system threads and work items specially generated to run in this level. May block, but in most cases it’s not allowed because of arbitrary thread context.
• APC_LEVEL: The special level for delivering data to user applications. Rarely seen, but often considered.
• DISPATCH_LEVEL: Used in DPC (Deferred Procedure calls) queued by ISR or by a custom call. Also, when acquiring a spinlock, the runlevel is raised to DISPATCH_LEVEL.
• Higher levels: When serving interrupts. No kind of mutex can be acquired (neither spinlocks). Some simple memory operations and queuing DPCs is more or less what’s left to do.

Notes:

• Paged code runs below DISPATCH_LEVEL. So when PAGED_CODE appears in the beginning of a function, it’s at most at APC_LEVEL.
• DPCs run at DISPATCH_LEVEL. When requested, a single instance is queued and executed when possible. The DPC instance is dequeued just before execution, so if another DPC request occurs immediately after launching a DPC, another run will take place, possibly simultaneously on another CPU. On the other hand, if several requests are made before execution is possible, the DPC is run only once on behalf of these requests.
• Custom DPCs can be queued from any runlevel. If the “background routine” needs to be queued from DISPATCH_LEVEL and down, use a work item instead.
• Confusingly enough, most (almost all) IRP dispatch calls are made in passive level. To make things more difficult, read, write and ioctl IRPs can go up to DISPATCH_LEVEL.

Bug check handling

What to do when the Blue Screen (BSOD) pays a visit. Or more precisely, how to catch exactly where your code wanted to use a zero pointer or something.

It’s important to save the entire binaries directory, including .obj files, for the driver distributed, since those files are necessary for bug check dissection.

The files are typically something like:
C:\Windows\Minidump\030312-17503-01.dmp
C:\Users\theuser\AppData\Local\Temp\WER-33571-0.sysdata.xml

However the XML file isn’t necessary for the bugcheck analysys, only the dump file.

To enable generation of a dump file, go to “System” from the Start menu, pick “Advanced system settings”  and click “Settings…” in the “Startup and Recovery” section. In the “System Failure” section, select “Small memory dump (128 kB)” in the drop-down menu. This option is possible only if there is swap space enabled. An alternative directory for the dump file can be selected there as well (but why?).

Analyze a bugcheck dump (change last file, of course). Be patient. Each step takes time.

> C:\WinDDK\7600.16385.1\Debuggers\kd.exe -n -y srv*c:\symbols*http://msdl.microsoft.com/download/symbols -i C:\devel\objchk_win7_x86\i386 -z C:\copies\022512-24414-01.dmp

The symbols which are downloaded are stored in C:\symbols (as required). These relate to a given version of the Windows kernel (and loaded drivers), so keep a different directory for each platform (?), or delete the directory altogether before invoking the debugger if unsure. It’s a cache, after all.

When those downloads are finished, go

0: kd> !analyze -v

Quit with “q”

To get a disassembly of the relevant code, go something like:

1: kd> u nt!PnpCallDriverEntry nt!PnpCallDriverEntry+0x4c

This prints out the disassembly from the beginning of PnpCallDriverEntry() to offset 0x4c. If the latter offset is the one appearing in the stack trace, the disassembly goes until the command after the call to the function above in the trace (because the address in the stack is the one to return to, not the caller’s address). The interesting disassemblies are of course those on the driver itself, not the system code as shown in this example.

On a 32-bit architecture, it’s also possible to disassemble the object file (but why?). Linux’ objdump -d works on Windows’ object files, surprisingly enough, and then there’s Microsoft’s counterpart:

> dumpbin.exe /disasm \path\to\objchk_win7_x86\i386\project.obj > project.asm

Note that we disassemble the object file to which the code belongs. That alone is a good reason to maintain an exact snapshot of released versions. The .sys file can be disassembled to verify that nothing has changed, but that disassembly doesn’t contain symbols names.

Unfortunately, the build for 64 bit (amd64) creates .obj files that can’t be disassembled, so only the .sys files can be used, and once again, they don’t contain symbols.

dumpbin.exe is part of the Visual C++ package. The necessary file bunlde are in my resumption directory, containing dumpbin.exe, link.exe, msdis140.dll and mspdb71.dll

April 2021 update: After another round of work on Windows drivers (still WDM), I’ve added a few more notes:

Another look at IRP handling

While the do’s and don’ts of IRP processing with the WDM model are well-documented, somehow I’ve never seen the idea behind the API explained. And it’s a fascinating one (not clear what kind of fascinating, though): It mimics a subroutine call, and implements the call stack as an array going with the IRP.

This is easiest explained by comparing with Linux’ driver API: For example, a read() system calls ends up as a call to the related function presented by the driver. That driver may call other functions in the kernel to fulfill its request. It may sleep waiting for data, causing the user-space process to sleep. It may call functions belonging to drivers at a lower level, i.e. closer to the hardware. Once it’s done, it returns, and the user space program gets its data.

Microsoft took an asynchronous approach, meaning that there is no direct connection between the user space program that initiated the read() call and the threads in the kernel that handles it. Rather, a data structure (the IRP) containing details on the read() request is set up, and then the execution flow goes event-driven style. Sometimes, the IRP is fulfilled by a single call to a handler functions, but in general, it’s juggled among several such. The fulfillment is cut into fragments, each running in a function of its own, and each needing to discover the overall state of the progress, not to mention tackle race conditions with the possibility that the IRP is canceled.

But the interesting point is the stack. And it has two meanings, actually: One is that drivers are stacked up, so there’s a low-level driver (say, the USB hub driver, which processes URBs) and higher-level drivers (e.g. for a sound card). Obviously, the intended usage was that if the user-space process makes a read() call for X bytes of data, an IRP is set up for that request, and handed over to the sound card’s driver for fetching X bytes of sound data. This driver goes on by setting up a request which is tied to the same IRP, for a data read URB of X bytes, and passing the IRP to the driver below it. And so it goes down the stack until some driver does the actual I/O. So the IRP represents the task to be fulfilled, and each driver in the stack presents its interpretation of the request.

And this brings me to the second meaning of “stack”: An execution stack. If the same thing was to run Linux kernel style, this would take the form of the upper driver literally calling a function belonging to a lower level driver, which could go on deeper and deeper until the ultimate driver is reached.

Apparently, Microsoft decided to mimic this execution stack by attaching an array of IO_STACK_LOCATION structures to the IRP’s main data structure, which is, well, the IRP stack . One way to look at these structures is that they contain the arguments of the function call to the driver beneath. So each of these IO_STACK_LOCATION structs contain the stuff that would have been pushed into the stack, had this lower-driver call been done Linux style. Which would have been the case, had Microsoft decided to give each IRP an execution thread that could sleep.

But wait, you say. There’s a return address too in the stack as well when a function is called. Well, I’m just getting to that.

So the idea is like this: The handling of an IRP consists of three parts: Things to do before calling the driver beneath, calling the driver beneath, and things doing after that. In Linux style it would have been something rhyming with

int stuff_handler( ... arguments ...)
{
   do_some_stuff();

   call_lower_driver( ... arguments ... );

   do_stuff_afterwards();
}

But this doesn’t work with Window’s model, because in most interesting cases, the call to the lower driver can be pended. The do_stuff_afterwards() needs to be done after the lower drivers have finished their business. That’s where the completion routine comes in: It contains the code for do_stuff_afterwards().

So effectively, the call to IoSetCompletionRoutine() takes the role of pushing the address to return to into the execution stack, in a regular function call. The completion routine is sort-of the return-from-function address for the IoCallDriver() call.

The completion routine can return one of two statuses:

• STATUS_CONTINUE_COMPLETION, which has the effect of returning from stuff_handler(), i.e. giving control to the function that called stuff_handler(). In Window’s stack model, this results in calling the completer function of some driver above. That is, the do-stuff-afterwards part of the upper driver.
• STATUS_MORE_PROCESSING_REQUIRED, meaning nope, don’t return from stuff_handler. Because all execution is done with functions, the completion function has to return when there’s no more useful things to do, but in this game of faking an execution thread, this status means “let’s pretend I never returned”.

So finally, we have IoCompleteRequest(). It simply means “stuff_handler returns now”. It’s the opposite to STATUS_MORE_PROCESSING_REQUIRED. It’s not (necessarily) called from a completer function, but it has the effect of returning from it with a STATUS_CONTINUE_COMPLETION.

The completion routine doesn’t have to be set. This is like finishing a Linux-style handler with

  return call_lower_driver( ... arguments ... );

which is typically translated into a JMP.

Looking at the IRP framework as a mimic of an execution thread is quite counterintuitive, in particular the idea that returning from that faked function is done by calling a function in the actual C code. Nevertheless, it helps somewhat when trying to make sense of some of the whole IRP framework.

Disassembling amd64 files

At some point, I wanted to disassemble the obj files created in an amd64 compilation. Dumpbin gave me:

Microsoft (R) COFF/PE Dumper Version 7.10.3077
Copyright (C) Microsoft Corporation.  All rights reserved.

Dump of file thefile.obj

FileType: ANONYMOUS OBJECT

and indeed, it’s not a COFF file, but rather an Anonymous Object file, with a rather odd magic word:

00000000  00 00 ff ff 01 00 64 86  a5 68 78 60 38 fe b3 0c  |......d..hx`8...|
00000010  a5 d9 ab 4d ac 9b d6 b6  22 26 53 c2 9b 81 01 00  |...M...."&S.....|
00000020  13 0c 07 00 a5 68 78 60  89 80 01 00 0f 00 00 00  |.....hx`........|
00000030  00 00 00 00 2e 64 72 65  63 74 76 65 00 00 00 00  |.....drectve....|

After the first four bytes, there a version number (0x0001) and then an 0x8664 signature for amd64. This is most likely a result of compiling with the /GL option for a whole program optimization. Hence there is nothing to disassemble here: Unlike a COFF file, which is just relocated into the final output, it appears like the content of this kind of object file is up for some additional mangling. Hence a disassembly of this file would have been useless, if at all possible.

So bottom line: Don’t even try. The .sys file can be disassembled, but it’s without symbols. So the sane way seems to be to use the kd debugger when disassembly is needed.

Bugcheck for memory leak

This is what it looks like at !analyze -v (minus lots of mumbo-jumbo)

Loading unloaded module list
.................
3: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

DRIVER_VERIFIER_DETECTED_VIOLATION (c4)
A device driver attempting to corrupt the system has been caught.  This is
because the driver was specified in the registry as being suspect (by the
administrator) and the kernel has enabled substantial checking of this driver.
If the driver attempts to corrupt the system, bugchecks 0xC4, 0xC1 and 0xA will
be among the most commonly seen crashes.
Arguments:
Arg1: 00000062, A driver has forgotten to free its pool allocations prior to unloading.
Arg2: 84caebd4, name of the driver having the issue.
Arg3: 84b9b418, verifier internal structure with driver information.
Arg4: 00000001, total # of (paged+nonpaged) allocations that weren't freed.
        Type !verifier 3 drivername.sys for info on the allocations
        that were leaked that caused the bugcheck.

[ ... ]

BUGCHECK_STR:  0xc4_62

IMAGE_NAME:  thedriver.sys

DEBUG_FLR_IMAGE_TIMESTAMP:  60635055

MODULE_NAME: thedriver

FAULTING_MODULE: 99a9b000 thedriver

VERIFIER_DRIVER_ENTRY: dt nt!_MI_VERIFIER_DRIVER_ENTRY ffffffff84b9b418
Symbol nt!_MI_VERIFIER_DRIVER_ENTRY not found.

CUSTOMER_CRASH_COUNT:  1

DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT

PROCESS_NAME:  System

CURRENT_IRQL:  2

LAST_CONTROL_TRANSFER:  from 82d51f03 to 82af9d10

STACK_TEXT:
8afa8a28 82d51f03 000000c4 00000062 84caebd4 nt!KeBugCheckEx+0x1e
8afa8a48 82d565eb 84caebd4 84b9b418 99a9b000 nt!VerifierBugCheckIfAppropriate+0x30
8afa8a58 82a29e8a 84caeb78 99a9b000 40000000 nt!VfPoolCheckForLeaks+0x33
8afa8a94 82bae69f 84caeb78 99a9b000 40000000 nt!VfTargetDriversRemove+0x66
8afa8aa8 82bae338 82b657e0 849a4a70 00000000 nt!VfDriverUnloadImage+0x5e
8afa8ae0 82baf58d 84caeb78 ffffffff 00000000 nt!MiUnloadSystemImage+0x1c6
8afa8b04 82cd8517 84caeb78 849c09c8 84c59668 nt!MmUnloadSystemImage+0x36
8afa8b1c 82c3e6f4 84c59680 84c59680 84c59668 nt!IopDeleteDriver+0x38
8afa8b34 82a85f60 00000000 84ab9818 84ab9768 nt!ObpRemoveObjectRoutine+0x59
8afa8b48 82a85ed0 84c59680 82bd6cb2 849c4838 nt!ObfDereferenceObjectWithTag+0x88

8afa8b50 82bd6cb2 849c4838 84ab9750 84ab9768 nt!ObfDereferenceObject+0xd

[ ... ]

STACK_COMMAND:  kb

FOLLOWUP_NAME:  MachineOwner

FAILURE_BUCKET_ID:  0xc4_62_LEAKED_POOL_IMAGE_thedriver.sys

BUCKET_ID:  0xc4_62_LEAKED_POOL_IMAGE_thedriver.sys

Followup: MachineOwner
---------

Just in case I’ll see one of these one day.

The WDK arrives with sample code for a PCI driver, known as PCIDRV. It demonstrates the recommended old-school methods for avoiding mishaps when maintaining your own IRP queue.

Frankly speaking, I’ve written this post as I learned the different corners of IRP juggling, so pretty much like certain operating systems, it’s a bit of a patchwork with possible inconsistencies. The written below should be taken as hints, not as a reference manual.

So this post consists of my own understanding of the example’s underlying logic. It’s definitely not an introduction to anything. This blob won’t make any sense unless you’re pretty familiar with the PCIDRV.C source and understand the API for handling (and canceling) IRPs.

To be fair, I’ll mention that Cancel-Safe Queues exist (but are not used in PCIDRV), and that the framework suggested by Microsoft for new drivers is KMDF (again, PCIDRV follows the older WDM). The choice is yours.

If I missed something, comments are welcome.

The possible bads

Above anything else, the goal is to avoid a nasty BSOD (Blue Screen of Death, bug check, you name it). In this context, this would happen because some code segment attempts to access memory a region which is either freed (unmapped virtual memory = bug check on the spot) or has a new function (freed and realocated, resulting in memory corruption). The following situations can cause this to happen:

• Attempting to access the device’s I/O, but the resources has been released (so nothing is mapped on that virtual address)
• Attempting to access device object data (the device extension region in particular, a.k.a. FdoData) after that has been freed
• Attempting to access IRP data structure which has been freed
• Writing to the process’ data buffer (in data I/O mode) after the process has shut down and its memory has been freed.

Slightly less catastrophic is not responding to cancel requests. Or more accurately put: Not responding with an IoCompleteRequest() to the fact that the Cancel entry of the IRP has gone TRUE (and the cancel routine, if registered, had been called) within a time period not perceived by humans. This is bad because the userspace application behind the IRP will not terminate until all its IRPs are gone. No matter how many times the user attempts to kill it with various weapons. And then the system will not be able to shut down, because the driver will refuse to unload, having outstanding IRP requests.

As for the possibility of trying to interact with hardware that isn’t physically there anymore, that’s a driver-dependent function. For example, trying to read from a PCIe device that doesn’t respond (possibly because it has been unplugged) should initiate a timeout mechanism, and finish the transaction with a peaceful all-1′s for data. What will actually happen depends on several factors (how the hardware actually behaves, and how Windows responds to that).

Carrying out the essence task of an IRP that has been canceled isn’t a problem in itself, as long as no illegal memory accesses take place.

Things guaranteed

To tackle the issues above, the Windows kernel promises a few things:

• The IRP entry’s memory will remain allocated at least until IoCompleteRequest() has been called, by the Cancel Routine or the normal completion (but IoFreeIrp() can be called any time afterwards).
• The calling application will not terminate until all its outstanding IRPs has undergone IoCompleteRequest().
• The driver’s object will not be freed until the dispatch call for IRP_MN_REMOVE_DEVICE has returned (as a matter of fact, the dispatch routine for this IRP does most of the release explicitly, e.g. with IoDeleteDevice).

As for freeing the I/O resources, that’s done by the driver itself, so it’s up to the driver not to release them (responding to  STOP or REMOVE  requests and/or queries) while they are being used.

Stop and Remove IRPs

To make things even trickier, every driver is required to hold any incoming IRP requests after receiving an IRP_MN_QUERY_STOP_DEVICE request, if it confirms its readiness for an IRP_MN_STOP_DEVICE (actually, the real requirement is to pause the device after either of these two, and PCIDRV chose to pause on the earlier request). This is because some other driver in the stack may refuse this request, leading to a cancellation of the intention to stop the device, in which case everything should go on as usual. The API assures that IRP_MN_STOP_DEVICE will not be issued if IRP_MN_QUERY_STOP_DEVICE failed by any of the targets. The IRP_MN_STOP_DEVICE  request, on the other hand, must not fail.

By the way, IRP_MN_REMOVE_DEVICEs don’t fall out of the blue either: They are preceded either by an IRP_MN_QUERY_REMOVE_DEVICE (when the removal is optional) or by an IRP_MN_SURPRISE_REMOVAL (when the removal is imminent, like an unplugged USB device).

So all in all, the trick for avoiding blue screens boils down to holding the return from those “game over” kind of calls just long enough to let the currently running operations finish, and make sure new ones don’t start.

Accessing I/O resources or the device object’s data

So the goal is not to release the I/O resources or the device object structure while some other thread expects them to be there. This is done by calling PciDrvReleaseAndWait() before releasing any precious stuff. In particular, this function is called with the STOP flag on receipt of an IRP_MN_QUERY_STOP_DEVICE. Since this IRP always precedes IRP_MN_STOP_DEVICE, the handler of the latter safely calls PciDrvReturnResources(), which in turn gets rid of the I/O resources with no worries. The exact same call to PciDrvReleaseAndWait() is issued when IRP_MN_QUERY_REMOVE_DEVICE is received.  If a IRP_MN_REMOVE_DEVICE arrives, on the other hand, the call is made with the REMOVE flag.

So what is this PciDrvReleaseAndWait() function about? It merely waits for a situation in which no IRPs are queued for processing and no such processing is ongoing. To skip the gory details, it waits for FdoData->OutstandingIO, which functions as a sort-of reference counter, to hit a value saying nothing is going on and won’t go on. This counter is incremented every time an IRP is queued in the receive or send queue. It’s also incremented when an IRP is handled directly by the dispatch routine, e.g. on PciDrvCreate(). It’s decremented in opposite situations: When the IRPs are dequeued and have finished processing, or when they have gone “off the hook” (that is, canceled or moved to an offline queue).

But waiting for the reference counter to hit a no-IRP value is not enough. IRPs can continue arriving after an IRP_MN_QUERY_STOP_DEVICE, so these IRPs must be prevented from execution while the device’s future is unknown. To cope with this, there’s a state variable, fdoData->QueueState, which reflects what to do with incoming IRPs. It can take three values, AllowRequests, HoldRequests and FailRequests.

Before calling PciDrvReleaseAndWait(), the handler of the stop/remove IRPs sets QueueState to HoldRequests or FailRequests, depending on whether there is hope to resume normal operation (power management routines change QueueState as well, by the way).

The HoldRequest state causes PciDrvDispatchIO() (the dispatch routine for read, write and ioctl IRPs) not to handle them in the normal manner, which would be to queue them on the RecvQueue, for example. Instead, it queues the IRP on the NewRequestsQueue (the “offline queue”) by calling PciDrvQueueRequest(). This queue is just for storage, so OutstandingIO is not incremented. The IRPs are just kept there in case the queuing state changes back to AllowRequests.

When the device is restarted, or when an IRP_MN_CANCEL_STOP_DEVICE arrives, the handler calls PciDrvProcessQueuedRequests(), which literally replays the IRPs in NewRequestsQueue by calling PciDrvDispatchIO() with each. An exception is those canceled (or in the middle of being canceled). Also, if the queue state happens to be FailRequest, all IRPs are completed with a juicy STATUS_NO_SUCH_DEVICE. As a matter of fact, if the queue state happens to turn back to HoldRequests while this IRP replaying is going on, IRP which was just about to be replayed is queued back to NewRequestsQueue, and the replay process is halted.

In fact, the way the IRP is queued back in this rare condition of the state going back to QueueState, may cause an odd situation. The thing is that the IRP which was the next one to be processed, and hence first in the queue, was pushed back to the last position in the same queue. So the IRPs’ order was changed as a result of this wizardry. This is not a bugcheck kind of problem, but if any application relies on the IRPs arriving in a certain order (are there any?), this could cause a very rare bug. The question is whether applications should rely on IRPs arriving in a certain order.

So much for the IRPs arriving in the future. What about those already queued? They are counted by OutstandingIO, but since these IRPs can remain pending for an indefinite time, their existence in the queues can cause PciDrvReleaseAndWait() to block forever.

To solve this, there’s yet another thing the handler of those stop/remove IRPs does before calling PciDrvReleaseAndWait(). That’s a call to PciDrvWithdrawIrps(), which does what its name implies: Moves all IRPs in the read and ioctl queues to the NewRequestsQueue, decrementing OutstandingIO for each one moved. Well, if you look in the code carefully, it’s PciDrvQueueRequest() which does the actual decrementing. But the job is done, anyhow.

And yet again, we have an IRP reordering issue: Since the state switches to HoldRequests before the call to PciDrvWithdrawIrps(), new IRPs will be queued before those in the ready-to-run queues, so they will be replayed in the wrong order. And again, I’m not sure if this matters.

Finally, just to have this mentioned: The initial state of the queue is HoldRequests, given by PciDrvAddDevice(). It’s only when the device is started, as an indirect result of an IRP_MN_START_DEVICE, that the state changes to AllowRequests. So the NewRequestsQueue isn’t just for helping the OutstandingIO reach a no-IRP value.

Handling canceled IRPs

In this respect, PCIDRV follows a pretty established paradigm. The cancel routine releases the global cancel lock, takes the dedicated queue’s lock, removes itself from the queue, and releases the queue lock. Then it marks the IRP’s status as STATUS_CANCELLED, zeroes the Information field and calls IoCompleteRequest(). If the queue affects the OutstandingIO reference counter, it’s decremented as well.

This is a good point to remind ourselves, that prior to calling the cancel routine, the Windows kernel first acquires the global Cancel spin lock, then sets the IRP’s Cancel entry to TRUE, after which it calls IoSetCancelRoutine() in order to get the address of the Cancel routine from the IRP’s structure and nullify the written value in an atomic operation. This use of IoSetCancelRoutine() makes the cancel routine’s pointer a safe indicator of the IRP’s state: If it’s NULL, the IRP isn’t cancelable, so it’s either in the process of being carried out or in the middle of cancellation.

The inline comments in PciDrvProcessQueuedRequests() explain how this is taken advantage of when removing an entry from a queue. The core is in calling IoSetCancelRoutine(nextIrp, NULL), so the cancel routine entry is read from and nullified before checking the Cancel entry.

As for carrying out the IRP, NICServiceReadIrps() in nic_recv.c demonstrates a similar process. It relies solely on IoSetCancelRoutine(irp, NULL) to either indicate that the cancel routine will run, is running or has run. And if that’s not the case, the atomic nullification makes sure it won’t run, so the IRP is executed and completed normally. It’s interesting to note, that the IRP Cancel flag isn’t even checked. In other words, if IoCancelIrp() just marked this flag, but was a nanosecond too late in grabbing and nullifying the cancel routine, this function will fail, and the IRP will be executed anyhow. In particular, any layers above the function driver will have their completion routine called with the Cancel flag set. Which they should be able to handle, of course.

This way or another, the IRP’s data structure will remain allocated and valid as a result of IoCancelIrp() failing, so there’s no memory corruption risk here.

Writing to a data buffer of a cleaned up process

In its NICServiceReadIrps() function (see nic_recv.c) it calls MmGetSystemAddressForMdlSafe() for a pointer to the application’s buffer memory, after making sure the IRP isn’t already canceled. It then releases the spinlock.

If the requesting application is in the middle of croaking, it’s possible that a cancel request will be issued for this IRP.  Even worse, it wants to release all application memory, including the buffer. But on the other hand, before it released the spinlock, the driver code nullified the pointer to the cancel routine, so any cancel request would fail.

First, I wasn’t sure if  Microsoft promises that the process will go on living as long as it has outstanding IRPs. Or maybe, is it OK to rely on MmGetSystemAddressForMdlSafe() returning a non-NULL value, indicating that some virtual space was allocated? After all, the physical memory is allocated in kernel virtual space, so if the process dies and frees its virtual memory resources, the driver’s pointer goes on pointing at real, nonpaged memory. The question remaining is whether the Windows kernel handles this race condition gracefully or not.

So I wasn’t 100% sure why the reference driver is so confident, until I found this page, which finally said it black on white: No user space application will terminate before all its IRP requests have been completed. As a matter of fact, the underlying assumption is that an uncompleted IRP may have a user application buffer mapped for DMA, so unless the driver confirms that the IRP is done, hardware could write directly to the physical memory, for all Windows knows. Quite amusingly, the page’s purpose was to urge driver programmers to allow for a quick cancellation, and not assure me that it’s safe to access the physical memory until completion is performed.

Summary

Not surprisingly, the PCIDRV sample, which has been inspected and imitated by quite a few programmers, seems to have it all covered. Why all this plumbing should be done by each and every WDM driver is a different question. I know. There’s new API for this. Let’s hope it’s better. For those brave enough to use it, that is.

On Windows 7, the file is C:\Windows\inf\setupapi.dev.log which can look like this when an unknown PCI device is detected by the system:

>>>  [Device Install (Hardware initiated) - pci\ven_10ee&dev_ebeb&subsys_ebeb10ee&rev_00\4&27574d66&0&0008]
>>>  Section start 2012/02/08 20:55:17.486
     ump: Creating Install Process: DrvInst.exe 20:55:17.496
     ndv: Retrieving device info...
     ndv: Setting device parameters...
     ndv: Searching Driver Store and Device Path...
     dvi: {Build Driver List} 20:55:17.506
     dvi:      Searching for hardware ID(s):
     dvi:           pci\ven_10ee&dev_ebeb&subsys_ebeb10ee&rev_00
     dvi:           pci\ven_10ee&dev_ebeb&subsys_ebeb10ee
     dvi:           pci\ven_10ee&dev_ebeb&cc_ff0000
     dvi:           pci\ven_10ee&dev_ebeb&cc_ff00
     dvi:      Searching for compatible ID(s):
     dvi:           pci\ven_10ee&dev_ebeb&rev_00
     dvi:           pci\ven_10ee&dev_ebeb
     dvi:           pci\ven_10ee&cc_ff0000
     dvi:           pci\ven_10ee&cc_ff00
     dvi:           pci\ven_10ee
     dvi:           pci\cc_ff0000
     dvi:           pci\cc_ff00
     cpy:      Policy is set to make all digital signatures equal.
     dvi:      Enumerating INFs from path list 'C:\Windows\inf'
     inf:      Searched 0 potential matches in published INF directory
     inf:      Searched 35 INFs in directory: 'C:\Windows\inf'
     dvi: {Build Driver List - exit(0x00000000)} 20:55:17.686
     ndv: Selecting best match from Driver Store (including Device Path)...
     dvi: {DIF_SELECTBESTCOMPATDRV} 20:55:17.686
     dvi:      No class installer for 'PCI Device'
     dvi:      No CoInstallers found
     dvi:      Default installer: Enter 20:55:17.696
     dvi:           {Select Best Driver}
!    dvi:                Selecting driver failed(0xe0000228)
     dvi:           {Select Best Driver - exit(0xe0000228)}
!    dvi:      Default installer: failed!
!    dvi:      Error 0xe0000228: There are no compatible drivers for this device.
     dvi: {DIF_SELECTBESTCOMPATDRV - exit(0xe0000228)} 20:55:17.716
     ndv: {Core Device Install} 20:55:17.716
!    ndv:      Installing NULL driver!
     dvi:      Set selected driver complete.
     dvi:      {DIF_ALLOW_INSTALL} 20:55:17.716
     dvi:           No class installer for 'PCI Device'
     dvi:           Default installer: Enter 20:55:17.716
     dvi:           Default installer: Exit
     dvi:      {DIF_ALLOW_INSTALL - exit(0xe000020e)} 20:55:17.716
     dvi:      {DIF_INSTALLDEVICE} 20:55:17.716
     dvi:           No class installer for 'PCI Device'
     dvi:           Default installer: Enter 20:55:17.716
!    dvi:                Installing NULL driver!
     dvi:                Writing common driver property settings.
     dvi:                {Restarting Devices} 20:55:17.736
     dvi:                     Restart: PCI\VEN_10EE&DEV_EBEB&SUBSYS_EBEB10EE&REV_00\4&27574D66&0&0008
     dvi:                     Restart complete.
     dvi:                {Restarting Devices exit} 20:55:17.906
     dvi:           Default installer: Exit
     dvi:      {DIF_INSTALLDEVICE - exit(0x00000000)} 20:55:17.906
     ndv:      Device install status=0xe0000203
     ndv:      Performing device install final cleanup...
!    ndv:      Queueing up error report since device installation failed...
     ndv: {Core Device Install - exit(0xe0000203)} 20:55:17.906
     ump: Server install process exited with code 0xe0000203 20:55:17.916
<<<  Section end 2012/02/08 20:55:17.916
<<<  [Exit status: FAILURE(0xe0000203)]

A successful installation for the same device could look like this:

>>>  [Device Install (DiShowUpdateDevice) - PCI\VEN_10EE&DEV_EBEB&SUBSYS_EBEB10EE&REV_00\4&27574D66&0&0008]
>>>  Section start 2012/02/08 23:22:31.407
 cmd: "C:\Windows\system32\mmc.exe" C:\Windows\system32\devmgmt.msc
 dvi: {DIF_UPDATEDRIVER_UI} 23:22:31.407
 dvi:      No class installer for 'PCI Device'
 dvi:      Default installer: Enter 23:22:31.423
 dvi:      Default installer: Exit
 dvi: {DIF_UPDATEDRIVER_UI - exit(0xe000020e)} 23:22:31.423
 ndv: {Update Driver Software Wizard for PCI\VEN_10EE&DEV_EBEB&SUBSYS_EBEB10EE&REV_00\4&27574D66&0&0008}
 inf:      Opened INF: 'c:\driver\xillybus.inf' ([strings])
 inf:      {SetupCopyOEMInf: c:\driver\xillybus.inf} 23:22:44.652
 sto:           {Import Driver Package: c:\driver\xillybus.inf} 23:22:44.683
 sto:                Importing driver package into Driver Store:
 sto:                     Driver Store   = C:\Windows\System32\DriverStore (Online | 6.1.7600)
 sto:                     Driver Package = c:\driver\xillybus.inf
 sto:                     Architecture   = x86
 sto:                     Locale Name    = neutral
 sto:                     Flags          = 0x00000000
 sto:                Copying driver package files to 'C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}'.
 inf:                Opened INF: 'c:\driver\xillybus.inf' ([strings])
 inf:                Opened INF: 'c:\driver\xillybus.inf' ([strings])
 flq:                {FILE_QUEUE_COPY}
 flq:                     CopyStyle      - 0x00000000
 flq:                     SourceRootPath - 'c:\driver\i386'
 flq:                     SourceFilename - 'xillybus.sys'
 flq:                     TargetDirectory- 'C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}\i386'
 flq:                {FILE_QUEUE_COPY exit(0x00000000)}
 flq:                {FILE_QUEUE_COPY}
 flq:                     CopyStyle      - 0x00000000
 flq:                     SourceRootPath - 'c:\driver'
 flq:                     SourceFilename - 'xillybus.inf'
 flq:                     TargetDirectory- 'C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}'
 flq:                {FILE_QUEUE_COPY exit(0x00000000)}
 flq:                {_commit_file_queue}
 flq:                     CommitQ DelNodes=0 RenNodes=0 CopyNodes=2
 flq:                     {_commit_copy_subqueue}
 flq:                          subqueue count=2
 flq:                          source media:
 flq:                               SourcePath   - [c:\driver\i386]
 flq:                               SourceFile   - [xillybus.sys]
 flq:                               Flags        - 0x00000000
 flq:                          {_commit_copyfile}
 flq:                               CopyFile: 'c:\driver\i386\xillybus.sys'
 flq:                                     to: 'C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}\i386\SET89B8.tmp'
 flq:                               MoveFile: 'C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}\i386\SET89B8.tmp'
 flq:                                     to: 'C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}\i386\xillybus.sys'
 flq:                          {_commit_copyfile exit OK}
 flq:                          source media:
 flq:                               SourcePath   - [c:\driver]
 flq:                               SourceFile   - [xillybus.inf]
 flq:                               Flags        - 0x00000000
 flq:                          {_commit_copyfile}
 flq:                               CopyFile: 'c:\driver\xillybus.inf'
 flq:                                     to: 'C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}\SET89D8.tmp'
 flq:                               MoveFile: 'C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}\SET89D8.tmp'
 flq:                                     to: 'C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}\xillybus.inf'
 flq:                          {_commit_copyfile exit OK}
 flq:                     {_commit_copy_subqueue exit OK}
 flq:                {_commit_file_queue exit OK}
 pol:                {Driver package policy check} 23:22:44.870
 pol:                {Driver package policy check - exit(0x00000000)} 23:22:44.870
 sto:                {Stage Driver Package: C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}\xillybus.inf} 23:22:44.870
 inf:                     Opened INF: 'C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}\xillybus.inf' ([strings])
 inf:                     Opened INF: 'C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}\xillybus.inf' ([strings])
 sto:                     Copying driver package files:
 sto:                          Source Path      = C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}
 sto:                          Destination Path = C:\Windows\System32\DriverStore\Temp\{296beaae-939f-1376-4169-f771e772a021}
 flq:                     {FILE_QUEUE_COPY}
 flq:                          CopyStyle      - 0x00000010
 flq:                          SourceRootPath - 'C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}\i386'
 flq:                          SourceFilename - 'xillybus.sys'
 flq:                          TargetDirectory- 'C:\Windows\System32\DriverStore\Temp\{296beaae-939f-1376-4169-f771e772a021}\i386'
 flq:                     {FILE_QUEUE_COPY exit(0x00000000)}
 flq:                     {FILE_QUEUE_COPY}
 flq:                          CopyStyle      - 0x00000010
 flq:                          SourceRootPath - 'C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}'
 flq:                          SourceFilename - 'xillybus.inf'
 flq:                          TargetDirectory- 'C:\Windows\System32\DriverStore\Temp\{296beaae-939f-1376-4169-f771e772a021}'
 flq:                     {FILE_QUEUE_COPY exit(0x00000000)}
 flq:                     {_commit_file_queue}
 flq:                          CommitQ DelNodes=0 RenNodes=0 CopyNodes=2
 flq:                          {_commit_copy_subqueue}
 flq:                               subqueue count=2
 flq:                               source media:
 flq:                                    SourcePath   - [C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}\i386]
 flq:                                    SourceFile   - [xillybus.sys]
 flq:                                    Flags        - 0x00000000
 flq:                               {_commit_copyfile}
 flq:                                    CopyFile: 'C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}\i386\xillybus.sys'
 flq:                                          to: 'C:\Windows\System32\DriverStore\Temp\{296beaae-939f-1376-4169-f771e772a021}\i386\SET8A54.tmp'
 flq:                                    MoveFile: 'C:\Windows\System32\DriverStore\Temp\{296beaae-939f-1376-4169-f771e772a021}\i386\SET8A54.tmp'
 flq:                                          to: 'C:\Windows\System32\DriverStore\Temp\{296beaae-939f-1376-4169-f771e772a021}\i386\xillybus.sys'
 flq:                               {_commit_copyfile exit OK}
 flq:                               source media:
 flq:                                    SourcePath   - [C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}]
 flq:                                    SourceFile   - [xillybus.inf]
 flq:                                    Flags        - 0x00000000
 flq:                               {_commit_copyfile}
 flq:                                    CopyFile: 'C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}\xillybus.inf'
 flq:                                          to: 'C:\Windows\System32\DriverStore\Temp\{296beaae-939f-1376-4169-f771e772a021}\SET8A55.tmp'
 flq:                                    MoveFile: 'C:\Windows\System32\DriverStore\Temp\{296beaae-939f-1376-4169-f771e772a021}\SET8A55.tmp'
 flq:                                          to: 'C:\Windows\System32\DriverStore\Temp\{296beaae-939f-1376-4169-f771e772a021}\xillybus.inf'
 flq:                               {_commit_copyfile exit OK}
 flq:                          {_commit_copy_subqueue exit OK}
 flq:                     {_commit_file_queue exit OK}
 sto:                     {DRIVERSTORE_IMPORT_NOTIFY_VALIDATE} 23:22:44.901
!    sto:                          Driver package does not contain a catalog file, but user wants to install anyway.
 sto:                     {DRIVERSTORE_IMPORT_NOTIFY_VALIDATE exit(0x00000000)} 23:22:46.196
 sto:                     Verified driver package signature:
 sto:                          Digital Signer Score = 0xFF000000
 sto:                          Digital Signer Name  = <unknown>
 sto:                     {DRIVERSTORE_IMPORT_NOTIFY_BEGIN} 23:22:46.196
 inf:                          Opened INF: 'C:\Windows\System32\DriverStore\Temp\{296beaae-939f-1376-4169-f771e772a021}\xillybus.inf' ([strings])
 sto:                          Create system restore point:
 sto:                               Description = Device Driver Package Install: Xillybus Ltd
 sto:                               Time        = 7394ms
 sto:                               Status      = 0x00000000 (SUCCESS)
 sto:                     {DRIVERSTORE_IMPORT_NOTIFY_BEGIN: exit(0x00000000)} 23:22:53.606
 sto:                     Importing driver package files:
 sto:                          Source Path      = C:\Windows\System32\DriverStore\Temp\{296beaae-939f-1376-4169-f771e772a021}
 sto:                          Destination Path = C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176
 sto:                     {Copy Directory: C:\Windows\System32\DriverStore\Temp\{296beaae-939f-1376-4169-f771e772a021}} 23:22:53.606
 sto:                          Target Path = C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176
 sto:                          {Copy Directory: C:\Windows\System32\DriverStore\Temp\{296beaae-939f-1376-4169-f771e772a021}\i386} 23:22:53.622
 sto:                               Target Path = C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176\i386
 sto:                          {Copy Directory: exit(0x00000000)} 23:22:53.622
 sto:                     {Copy Directory: exit(0x00000000)} 23:22:53.622
 sto:                     {Index Driver Package: C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf} 23:22:53.622
 idb:                          Registered driver store entry 'xillybus.inf_x86_neutral_c6abbde6e922f176'.
 idb:                          Published 'xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf' to 'C:\Windows\INF\oem2.inf'
 idb:                          Published driver store entry 'xillybus.inf_x86_neutral_c6abbde6e922f176'.
 sto:                          Published driver package INF 'oem2.inf' was changed.
 sto:                          Active published driver package is 'xillybus.inf_x86_neutral_c6abbde6e922f176'.
 sto:                     {Index Driver Package: exit(0x00000000)} 23:22:54.495
 sto:                     {DRIVERSTORE_IMPORT_NOTIFY_END} 23:22:54.495
 sto:                          Commit system restore point:
 sto:                               Description = Device Driver Package Install: Xillybus Ltd
 sto:                               Time        = 16ms
 sto:                               Status      = 0x00000000 (SUCCESS)
 sto:                     {DRIVERSTORE_IMPORT_NOTIFY_END: exit(0x00000000)} 23:22:54.511
 sto:                {Stage Driver Package: exit(0x00000000)} 23:22:54.511
 ndv:                Doing device matching lookup!
 inf:                Opened INF: 'C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf' ([strings])
 inf:                Saved PNF: 'C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.PNF' (Language = 0409)
 ndv:                Found device that matches new INF!
 sto:                Driver package was staged to Driver Store. Time = 9766 ms
 sto:                Imported driver package into Driver Store:
 sto:                     Filename = C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf
 sto:                     Time     = 9875 ms
 sto:           {Import Driver Package: exit(0x00000000)} 23:22:54.558
 inf:           Opened INF: 'c:\driver\xillybus.inf' ([strings])
 inf:           Driver Store location: C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf
 inf:           Published Inf Path: C:\Windows\INF\oem2.inf
 inf:           Opened INF: 'c:\driver\xillybus.inf' ([strings])
 inf:           OEM source media location: c:\driver\
 inf:      {SetupCopyOEMInf exit (0x00000000)} 23:22:54.729
 dvi:      Searching for hardware ID(s):
 dvi:           pci\ven_10ee&dev_ebeb&subsys_ebeb10ee&rev_00
 dvi:           pci\ven_10ee&dev_ebeb&subsys_ebeb10ee
 dvi:           pci\ven_10ee&dev_ebeb&cc_ff0000
 dvi:           pci\ven_10ee&dev_ebeb&cc_ff00
 dvi:      Searching for compatible ID(s):
 dvi:           pci\ven_10ee&dev_ebeb&rev_00
 dvi:           pci\ven_10ee&dev_ebeb
 dvi:           pci\ven_10ee&cc_ff0000
 dvi:           pci\ven_10ee&cc_ff00
 dvi:           pci\ven_10ee
 dvi:           pci\cc_ff0000
 dvi:           pci\cc_ff00
 inf:      Opened PNF: 'C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf' ([strings])
 sig:      {_VERIFY_FILE_SIGNATURE} 23:22:54.745
 sig:           Key      = xillybus.inf
 sig:           FilePath = C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf
!    sig:           No installed catalogs matching catalog name '' were found that validated the file.
!    sig:           Error 1168: Element not found.
 sig:      {_VERIFY_FILE_SIGNATURE exit(0x00000490)} 23:22:54.761
 dvi:      Selected driver installs from section [Xillybus_Inst] in 'c:\windows\system32\driverstore\filerepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf'.
 dvi:      Class GUID of device changed to: {eb32c6d5-2d4d-4d8c-a83b-4db9621fdb07}.
 dvi:      Set selected driver complete.
 dvi:      {Plug and Play Service: Device Install for PCI\VEN_10EE&DEV_EBEB&SUBSYS_EBEB10EE&REV_00\4&27574D66&0&0008}
 ump:           Creating Install Process: DrvInst.exe 23:22:54.761
 ndv:           Infpath=C:\Windows\INF\oem2.inf
 ndv:           DriverNodeName=xillybus.inf:MSFT.NTx86:Xillybus_Inst:1.0.0.0:pci\ven_10ee&dev_ebeb
 ndv:           DriverStorepath=C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf
 ndv:           Building driver list from driver node strong name...
 dvi:           Searching for hardware ID(s):
 dvi:                pci\ven_10ee&dev_ebeb&subsys_ebeb10ee&rev_00
 dvi:                pci\ven_10ee&dev_ebeb&subsys_ebeb10ee
 dvi:                pci\ven_10ee&dev_ebeb&cc_ff0000
 dvi:                pci\ven_10ee&dev_ebeb&cc_ff00
 dvi:           Searching for compatible ID(s):
 dvi:                pci\ven_10ee&dev_ebeb&rev_00
 dvi:                pci\ven_10ee&dev_ebeb
 dvi:                pci\ven_10ee&cc_ff0000
 dvi:                pci\ven_10ee&cc_ff00
 dvi:                pci\ven_10ee
 dvi:                pci\cc_ff0000
 dvi:                pci\cc_ff00
 inf:           Opened PNF: 'C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf' ([strings])
 sig:           {_VERIFY_FILE_SIGNATURE} 23:22:54.807
 sig:                Key      = xillybus.inf
 sig:                FilePath = C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf
!    sig:                No installed catalogs matching catalog name '' were found that validated the file.
!    sig:                Error 1168: Element not found.
 sig:           {_VERIFY_FILE_SIGNATURE exit(0x00000490)} 23:22:54.823
 dvi:           Selected driver installs from section [Xillybus_Inst] in 'c:\windows\system32\driverstore\filerepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf'.
 dvi:           Class GUID of device changed to: {eb32c6d5-2d4d-4d8c-a83b-4db9621fdb07}.
 dvi:           Set selected driver complete.
 ndv:           {Core Device Install} 23:22:54.823
 inf:                Opened INF: 'C:\Windows\INF\oem2.inf' ([strings])
 inf:                Saved PNF: 'C:\Windows\INF\oem2.PNF' (Language = 0409)
 ndv:                Class {eb32c6d5-2d4d-4d8c-a83b-4db9621fdb07} does not exist.
 dvi:                {Installing Class}
 inf:                     Opened PNF: 'C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf' ([strings])
 dvi:                     Installing device class: 'Xillybus' {EB32C6D5-2D4D-4d8c-A83B-4DB9621FDB07}.
 dvi:                     {Install Class Inf Section [ClassInstall32]}
 dvi:                     {Install Class Inf Section [ClassInstall32] exit (0x00000000)}
 dvi:                     {_SCAN_FILE_QUEUE}
 flq:                          ScanQ flags=22
 flq:                               SPQ_SCAN_FILE_VALIDITY
 flq:                               SPQ_SCAN_PRUNE_COPY_QUEUE
 flq:                          ScanQ number of copy nodes=0
 flq:                          ScanQ action=2 DoPruning=32
 flq:                          ScanQ end Validity flags=22 CopyNodes=0
 dvi:                     {_SCAN_FILE_QUEUE exit(0, 0x00000000)}
 dvi:                     {Install Class Inf Section [ClassInstall32]}
 inf:                          Addreg=XillybusClassReg  (xillybus.inf line 14)
 dvi:                     {Install Class Inf Section [ClassInstall32] exit (0x00000000)}
 dvi:                     {Install Class Inf Section [ClassInstall32.Services]}
 dvi:                     {Install Class Inf Section [ClassInstall32.Services] exit (0x00000000)}
 dvi:                     Class install completed with no errors.
 dvi:                {Installing Class exit(0x00000000)}
 dvi:                {DIF_ALLOW_INSTALL} 23:22:54.854
 dvi:                     No class installer for 'Xillybus driver for generic FPGA interface'
 dvi:                     No CoInstallers found
 dvi:                     Default installer: Enter 23:22:54.854
 dvi:                     Default installer: Exit
 dvi:                {DIF_ALLOW_INSTALL - exit(0xe000020e)} 23:22:54.854
 ndv:                Installing files...
 dvi:                {DIF_INSTALLDEVICEFILES} 23:22:54.854
 dvi:                     No class installer for 'Xillybus driver for generic FPGA interface'
 dvi:                     Default installer: Enter 23:22:54.854
 dvi:                          {Install FILES}
 inf:                               Opened PNF: 'c:\windows\system32\driverstore\filerepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf' ([strings])
 inf:                               {Install Inf Section [Xillybus_Inst.NT]}
 inf:                                    CopyFiles=Xillybus.CopyFiles  (xillybus.inf line 53)
 cpy:                                    Open PnpLockdownPolicy: Err=2. This is OK. Use LockDownPolicyDefault
 flq:                                    QueueSingleCopy...
 flq:                                    Inf     : 'c:\windows\system32\driverstore\filerepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf'
 flq:                                    SourceInf: 'c:\windows\system32\driverstore\filerepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf'
 flq:                                    SourceSection: [sourcedisksfiles]
 flq:                                    Source root path based on SourceInf
 flq:                                    SourceRootPath: 'C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176'
 flq:                                    {FILE_QUEUE_COPY}
 flq:                                         CopyStyle      - 0x00000000
 flq:                                         {FILE_QUEUE_COPY}
 flq:                                              CopyStyle      - 0x00000000
 flq:                                              SourceRootPath - 'C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176'
 flq:                                              SourcePath     - '\i386'
 flq:                                              SourceFilename - 'xillybus.sys'
 flq:                                              TargetDirectory- 'C:\Windows\system32\DRIVERS'
 flq:                                              TargetFilename - 'xillybus.sys'
 flq:                                              SourceDesc     - 'Xillybus install disk'
 flq:                                         {FILE_QUEUE_COPY exit(0x00000000)}
 flq:                                    {FILE_QUEUE_COPY exit(0x00000000)}
 inf:                               {Install Inf Section [Xillybus_Inst.NT] exit (0x00000000)}
 dvi:                               Processing co-installer registration section [Xillybus_Inst.NT.CoInstallers].
 inf:                               {Install Inf Section [Xillybus_Inst.NT.CoInstallers]}
 inf:                               {Install Inf Section [Xillybus_Inst.NT.CoInstallers] exit (0x00000000)}
 dvi:                               Co-installers registered.
 dvi:                               {Install INTERFACES}
 dvi:                                    Installing section [Xillybus_Inst.NT.Interfaces]
 dvi:                               {Install INTERFACES exit 00000000}
 dvi:                          {Install FILES exit (0x00000000)}
 dvi:                     Default installer: Exit
 dvi:                {DIF_INSTALLDEVICEFILES - exit(0x00000000)} 23:22:54.885
 ndv:                Pruning file queue...
 dvi:                {_SCAN_FILE_QUEUE}
 flq:                     ScanQ flags=620
 flq:                          SPQ_SCAN_PRUNE_COPY_QUEUE
 flq:                          SPQ_SCAN_FILE_COMPARISON
 flq:                          SPQ_SCAN_ACTIVATE_DRP
 flq:                     ScanQ number of copy nodes=1
 flq:                     ScanQ action=200 DoPruning=32
 flq:                     ScanQ end Validity flags=620 CopyNodes=1
 dvi:                {_SCAN_FILE_QUEUE exit(0, 0x00000000)}
 ndv:                Committing file queue...
 flq:                {_commit_file_queue}
 flq:                     CommitQ DelNodes=0 RenNodes=0 CopyNodes=1
 flq:                     {SPFILENOTIFY_STARTQUEUE}
 flq:                     {SPFILENOTIFY_STARTQUEUE - exit(0x00000001)}
 flq:                     {_commit_copy_subqueue}
 flq:                          subqueue count=1
 flq:                          {SPFILENOTIFY_STARTSUBQUEUE}
 flq:                          {SPFILENOTIFY_STARTSUBQUEUE - exit(0x00000001)}
 flq:                          source media:
 flq:                               Description  - [Xillybus install disk]
 flq:                               SourcePath   - [C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176\i386]
 flq:                               SourceFile   - [xillybus.sys]
 flq:                               Flags        - 0x00000000
 flq:                          {SPFQNOTIFY_NEEDMEDIA}
 flq:                               {SPFILENOTIFY_NEEDMEDIA}
 flq:                               {SPFILENOTIFY_NEEDMEDIA - exit(0x00000001)}
 flq:                          {SPFQNOTIFY_NEEDMEDIA - returned 0x00000001}
 flq:                          source media: SPFQOPERATION_DOIT
 flq:                          {_commit_copyfile}
 flq:                               {SPFILENOTIFY_STARTCOPY}
 ndv:                                    Saving LastKnownGood file C:\Windows\system32\DRIVERS\xillybus.sys (copy)
 flq:                               {SPFILENOTIFY_STARTCOPY - exit(0x00000001)}
 flq:                               CopyFile: 'C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176\i386\xillybus.sys'
 flq:                                     to: 'C:\Windows\system32\DRIVERS\SETB164.tmp'
 cpy:                               CopyFile Drp is active
 cpy:                               Source File 'C:\Windows\system32\DRIVERS\SETB164.tmp' is NOT signed NT5 Crypto.
 cpy:                               DrpGetFileProt Status=2 dwClass=0
 flq:                               MoveFile: 'C:\Windows\system32\DRIVERS\SETB164.tmp'
 flq:                                     to: 'C:\Windows\system32\DRIVERS\xillybus.sys'
 cpy:                               DrpSetRegFileProt 'C:\Windows\system32\DRIVERS\xillybus.sys' Status=0 Legacy
 flq:                               Caller applied security to file 'C:\Windows\system32\DRIVERS\xillybus.sys'.
 flq:                               {SPFILENOTIFY_ENDCOPY}
 flq:                               {SPFILENOTIFY_ENDCOPY - exit(0x00000001)}
 flq:                          {_commit_copyfile exit OK}
 flq:                          {SPFILENOTIFY_ENDSUBQUEUE}
 flq:                          {SPFILENOTIFY_ENDSUBQUEUE - exit(0x00000001)}
 flq:                     {_commit_copy_subqueue exit OK}
 flq:                     {SPFILENOTIFY_ENDQUEUE}
 flq:                     {SPFILENOTIFY_ENDQUEUE - exit(0x00000001)}
 flq:                {_commit_file_queue exit OK}
 ndv:                Registering CoInstallers...
 dvi:                {DIF_REGISTER_COINSTALLERS} 23:22:54.979
 dvi:                     No class installer for 'Xillybus driver for generic FPGA interface'
 dvi:                     Default installer: Enter 23:22:54.995
 inf:                          Opened PNF: 'c:\windows\system32\driverstore\filerepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf' ([strings])
 inf:                          {Install Inf Section [Xillybus_Inst.NT.CoInstallers]}
 inf:                          {Install Inf Section [Xillybus_Inst.NT.CoInstallers] exit (0x00000000)}
 dvi:                          Co-installers registered.
 dvi:                     Default installer: Exit
 dvi:                {DIF_REGISTER_COINSTALLERS - exit(0x00000000)} 23:22:54.995
 ndv:                Installing interfaces...
 dvi:                {DIF_INSTALLINTERFACES} 23:22:54.995
 dvi:                     No class installer for 'Xillybus driver for generic FPGA interface'
 dvi:                     No CoInstallers found
 dvi:                     Default installer: Enter 23:22:54.995
 dvi:                          {Install INTERFACES}
 inf:                               Opened PNF: 'c:\windows\system32\driverstore\filerepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf' ([strings])
 dvi:                               Installing section [Xillybus_Inst.NT.Interfaces]
 dvi:                          {Install INTERFACES exit 00000000}
 dvi:                     Default installer: Exit
 dvi:                {DIF_INSTALLINTERFACES - exit(0x00000000)} 23:22:54.995
 ndv:                Installing device...
 dvi:                {DIF_INSTALLDEVICE} 23:22:54.995
 dvi:                     No class installer for 'Xillybus driver for generic FPGA interface'
 dvi:                     Default installer: Enter 23:22:54.995
 dvi:                          {Install DEVICE}
 inf:                               Opened PNF: 'c:\windows\system32\driverstore\filerepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf' ([strings])
 dvi:                               Processing Registry/Property directives...
 inf:                               {Install Inf Section [Xillybus_Inst.NT]}
 inf:                               {Install Inf Section [Xillybus_Inst.NT] exit (0x00000000)}
 inf:                               {Install Inf Section [Xillybus_Inst.NT.Hw]}
 inf:                                    Empty section
 inf:                               {Install Inf Section [Xillybus_Inst.NT.Hw] exit (0x00000000)}
 dvi:                               {Writing Device Properties}
 dvi:                                    Provider name=Xillybus Ltd
 dvi:                                    DriverDate 02/01/2012
 dvi:                                    DriverVersion=1.0.0.0
 dvi:                                    Class name=Xillybus
 dvi:                                    Manufacturer=Xillybus Ltd
 dvi:                                    Matching DeviceID=pci\ven_10ee&dev_ebeb
 dvi:                                    Strong Name=oem2.inf:MSFT.NTx86:Xillybus_Inst:1.0.0.0:pci\ven_10ee&dev_ebeb
 dvi:                               {Writing Device Properties - Complete}
 inf:                               {Install Inf Section [Xillybus_Inst.NT.Services]}
 inf:                                    AddService=Xillybus,0x00000002,Xillybus_Service,Xillybus_LogInst  (xillybus.inf line 73)
 inf:                                    ServiceType=1  (xillybus.inf line 77)
 inf:                                    StartType=3  (xillybus.inf line 78)
 inf:                                    ErrorControl=1  (xillybus.inf line 79)
 inf:                                    ServiceBinary=C:\Windows\system32\DRIVERS\xillybus.sys  (xillybus.inf line 80)
 inf:                                    DisplayName="Xillybus driver service for generic FPGA interface"  (xillybus.inf line 76)
 dvi:                                    Add Service: Created service 'Xillybus'.
 inf:                                    AddReg=Xillybus_Log_Addreg  (xillybus.inf line 83)
 inf:                               {Install Inf Section [Xillybus_Inst.NT.Services] exit(0x00000000)}
 dvi:                               Updated reflected section names for: oem2.inf
 dvi:                          {Install DEVICE exit (0x00000000)}
 dvi:                          Writing common driver property settings.
 dvi:                               DriverDescription=Xillybus driver for generic FPGA interface
 dvi:                               DeviceDisplayName=Xillybus driver for generic FPGA interface
 dvi:                          {Restarting Devices} 23:22:57.319
 dvi:                               Restart: PCI\VEN_10EE&DEV_EBEB&SUBSYS_EBEB10EE&REV_00\4&27574D66&0&0008
 dvi:                               Restart complete.
 dvi:                          {Restarting Devices exit} 23:23:08.208
 dvi:                     Default installer: Exit
 dvi:                {DIF_INSTALLDEVICE - exit(0x00000000)} 23:23:08.208
 dvi:                {DIF_NEWDEVICEWIZARD_FINISHINSTALL} 23:23:08.208
 dvi:                     No class installer for 'Xillybus driver for generic FPGA interface'
 dvi:                     Default installer: Enter 23:23:08.208
 dvi:                     Default installer: Exit
 dvi:                {DIF_NEWDEVICEWIZARD_FINISHINSTALL - exit(0xe000020e)} 23:23:08.208
 ndv:                Device install status=0x00000000
 ndv:                Performing device install final cleanup...
 ndv:           {Core Device Install - exit(0x00000000)} 23:23:08.208
 ump:           Server install process exited with code 0x00000000 23:23:08.224
 ump:      {Plug and Play Service: Device Install exit(00000000)}
 dvi:      {DIF_NEWDEVICEWIZARD_FINISHINSTALL} 23:23:08.239
 dvi:           No class installer for 'Xillybus driver for generic FPGA interface'
 dvi:           No CoInstallers found
 dvi:           Default installer: Enter 23:23:08.239
 dvi:           Default installer: Exit
 dvi:      {DIF_NEWDEVICEWIZARD_FINISHINSTALL - exit(0xe000020e)} 23:23:08.239
 ndv: {Update Driver Software Wizard exit(00000000)}
<<<  Section end 2012/02/08 23:23:11.281
<<<  [Exit status: SUCCESS

The helpful information here is mostly which parts of the INF file were read and interpreted, and what was actually installed (a new class in this case).

It’s also likely that you used this function to support MSI (Message Signaling Interrupt). You may have attempted to follow the horribly misleading example given by Microsoft itself.

The thing is, that there’s a parameter they omitted in that example, and that’s params.MessageBased.ConnectionContext.InterruptMessageTable. That’s a pointer to a PIO_INTERRUPT_MESSAGE_INFO. If you don’t know what I’m talking about, I won’t bother to explain that: It already means I’ve just solved your problem.

I’ll just say, that IoConnectInterruptEx() uses that parameter as a pointer to write the value of another pointer of some all-so-cute and most likely useless structure of information. So if this parameter isn’t initialized, the kernel attempts to write to a null pointer, and there we are with a blue screen.

To get the example wrong in an official page is somewhat understandable. Using a pointer without checking it for being null in a kernel function is completely amazing.