This post examines what the Microsoft’s compiler does in response to a variety of special functions that implement atomic operations and memory barriers. If you program like a civilized human being, that is with spinlocks and mutexes, this is a lot of things you should never need to care about.

I’ve written a similar post regarding Linux, and it’s recommended to read it before this one (even though it repeats some of the stuff here).

To make a long story short:

• The Interlocked-something functions do not just guarantee atomicity, but also function as a memory barrier to the compiler
• Memory fences are unnecessary for the sake of synchronizing between processors
• The conclusion is hence that those Interlocked functions also work as multiprocessor memory barriers.

Trying it out

The following code:

LONG atomic_thingy;

__declspec(noinline) LONG do_it(LONG *p) {
  LONG x = 0;
  LONG y;
  x += *p;
  y = InterlockedExchangeAdd(&atomic_thingy, 0x1234);
  x += *p;

  return x + y;
}

When compiling this as “free” (i.e. optimized), this yields:

_do_it@4:
  00000000: 8B FF              mov         edi,edi
  00000002: 55                 push        ebp
  00000003: 8B EC              mov         ebp,esp
  00000005: 8B 45 08           mov         eax,dword ptr [ebp+8]
  00000008: 8B 10              mov         edx,dword ptr [eax]
  0000000A: 56                 push        esi
  0000000B: B9 34 12 00 00     mov         ecx,1234h
  00000010: BE 00 00 00 00     mov         esi,offset _atomic_thingy
  00000015: F0 0F C1 0E        lock xadd   dword ptr [esi],ecx
  00000019: 8B 00              mov         eax,dword ptr [eax]
  0000001B: 03 C1              add         eax,ecx
  0000001D: 03 C2              add         eax,edx
  0000001F: 5E                 pop         esi
  00000020: 5D                 pop         ebp
  00000021: C2 04 00           ret         4

First thing to note is that InterlockedExchangeAdd() has been translated into a “LOCK XADD”, which indeed fetches the previous value into ECX and stores the updated value into memory. The previous value is stored in ECX after this operation, which is @y in the C code — InterlockedExchangeAdd() returns the previous value.

XADD by itself isn’t atomic, which is why the LOCK prefix is added. More about LOCK below.

What is crucially important to note, is that putting InterlockedExchangeAdd() between the two reads of *p prevents the optimizations of these two reads into one. @p isn’t defined as volatile, and yet it’s read from twice (ptr [eax]).

Another variant, now trying InterlockedOr():

LONG atomic;

__declspec(noinline) LONG do_it(LONG *p) {
  LONG x = 0;
  LONG y;
  x += *p;
  y = InterlockedOr(&atomic, 0x1234);
  x += *p;

  return x + y;
}

Once again, compiled as “free”, turns into this:

_do_it@4:
  00000000: 8B FF              mov         edi,edi
  00000002: 55                 push        ebp
  00000003: 8B EC              mov         ebp,esp
  00000005: 8B 4D 08           mov         ecx,dword ptr [ebp+8]
  00000008: 8B 11              mov         edx,dword ptr [ecx]
  0000000A: 53                 push        ebx
  0000000B: 56                 push        esi
  0000000C: 57                 push        edi
  0000000D: BE 34 12 00 00     mov         esi,1234h
  00000012: BF 00 00 00 00     mov         edi,offset _atomic
  00000017: 8B 07              mov         eax,dword ptr [edi]
  00000019: 8B D8              mov         ebx,eax
  0000001B: 0B DE              or          ebx,esi
  0000001D: F0 0F B1 1F        lock cmpxchg dword ptr [edi],ebx
  00000021: 75 F6              jne         00000019
  00000023: 8B F0              mov         esi,eax
  00000025: 8B 01              mov         eax,dword ptr [ecx]
  00000027: 5F                 pop         edi
  00000028: 03 C6              add         eax,esi
  0000002A: 5E                 pop         esi
  0000002B: 03 C2              add         eax,edx
  0000002D: 5B                 pop         ebx
  0000002E: 5D                 pop         ebp
  0000002F: C2 04 00           ret         4

This is quite amazing: The OR isn’t implemented as an atomic operation, but rather it goes like this: The previous value of @atomic is fetched into EAX and then moved to EBX. It’s ORed with the constant 0x1234, and then the cmpxchg instruction writes the result (in EBX) into @atomic only if its previous value was the same as EAX. If not, the previous value is stored in EAX instead. In the latter case, the JNE loops back to try again.

I should mention that cmpxchg compares with EAX and stores the previous value to it if the comparison fails, even though this register isn’t mentioned explicitly in the instruction. It’s just a thing that cmpxchg works with EAX. EBX is the register to compare with, and it therefore appears in the instruction. Confusing, yes.

Also here, *p is read twice.

It’s also worth noting that using InterlockedOr() with the value 0 as a common way to grab the current value yields basically the same code (only the constant is generated with a “xor esi,esi” instead).

So if you want to use an Interlocked function just to read from a variable, InterlockedExchangeAdd() with zero is probably better, because it doesn’t create all that loop code around it.

Another function I’d like to look at is InterlockedExchange(), as it’s used a lot. Spoiler: No surprises are expected.

LONG atomic_thingy;

__declspec(noinline) LONG do_it(LONG *p) {
  LONG x = 0;
  LONG y;
  x += *p;
  y = InterlockedExchange(&atomic_thingy, 0);
  x += *p;

  return x + y;
}

and this is what it compiles into:

_do_it@4:
  00000000: 8B FF              mov         edi,edi
  00000002: 55                 push        ebp
  00000003: 8B EC              mov         ebp,esp
  00000005: 8B 45 08           mov         eax,dword ptr [ebp+8]
  00000008: 8B 10              mov         edx,dword ptr [eax]
  0000000A: 56                 push        esi
  0000000B: 33 C9              xor         ecx,ecx
  0000000D: BE 00 00 00 00     mov         esi,offset _atomic_thingy
  00000012: 87 0E              xchg        ecx,dword ptr [esi]
  00000014: 8B 00              mov         eax,dword ptr [eax]
  00000016: 03 C1              add         eax,ecx
  00000018: 03 C2              add         eax,edx
  0000001A: 5E                 pop         esi
  0000001B: 5D                 pop         ebp
  0000001C: C2 04 00           ret         4

And finally, what about writing twice to the same place?

LONG atomic_thingy;

__declspec(noinline) LONG do_it(LONG *p) {
  LONG y;
  *p = 0;
  y = InterlockedExchangeAdd(&atomic_thingy, 0);
  *p = 0;

  return y;
}

Writing the same constant value twice to a non-volatile variable. This calls for an optimization. But the Interlocked function prevents it, as expected:

_do_it@4:
  00000000: 8B FF              mov         edi,edi
  00000002: 55                 push        ebp
  00000003: 8B EC              mov         ebp,esp
  00000005: 8B 4D 08           mov         ecx,dword ptr [ebp+8]
  00000008: 83 21 00           and         dword ptr [ecx],0
  0000000B: 33 C0              xor         eax,eax
  0000000D: BA 00 00 00 00     mov         edx,offset _atomic_thingy
  00000012: F0 0F C1 02        lock xadd   dword ptr [edx],eax
  00000016: 83 21 00           and         dword ptr [ecx],0
  00000019: 5D                 pop         ebp
  0000001A: C2 04 00           ret         4

Writing a zero is implemented by ANDing with zero, so it’s a bit confusing. But it’s done twice, before and after the “lock xadd”. Needless to say, these two writes are fused into one by the compiler without the Interlocked statement in the middle (I’ve verified it with disassembly, 32 and 64 bit).

Volatile?

In Microsoft’s definition for the InterlockedExchangeAdd() function (and all other similar ones) is that the first operand is a pointer to a LONG volatile. Why volatile? Does the variable really need to be?

The answer is no, if all accesses to the variable is made with Interlocked-something functions. There will be no compiler optimizations, not on the call itself, and the call itself is a compiler memory barrier.

And it’s a good habit to stick to these functions, because of this implicit compiler memory barrier: That’s usually what we want and need, even if we’re not fully aware of it. Accessing a shared variable almost always has a “before” and “after” thinking around it. The volatile keyword doesn’t protect against reordering optimizations by the compiler (or otherwise).

But if the variable is accessed without these functions in some places, the volatile keyword should definitely be used when defining that variable.

More about LOCK

It’s a prefix that is added to an instruction in order to ensure that it’s performed atomically. In many cases, it’s superfluous and sometimes ignored, as the atomic operation is ensured anyhow.

From Intel’s 64 and IA-32 Architectures Software Developer’s Manual, Volume 2 (instruction set reference) vol. 2A page 3-537, on the “LOCK” prefix for instructions:

Causes the processor’s LOCK# signal to be asserted during execution of the accompanying instruction (turns the instruction into an atomic instruction). In a multiprocessor environment, the LOCK# signal ensures that the processor has exclusive use of any shared memory while the signal is asserted.

The manual elaborates further on the LOCK prefix, but says nothing about memory barriers / fences. This is implemented with the MFENCE, SFENCE and LFENCE instructions.

The LOCK prefix is encoded with an 0xf0 coming before the instruction in the binary code, by the way.

Linux counterparts

For x86 only, of course:

• atomic_set() turns into a plain “mov”
• atomic_add() turns into “lock add”
• atomic_sub() turns into “lock sub”

Is a memory barrier (fences) required?

Spoiler: Not in x86 kernel code, including 32 and 64 bits. Surprise. Unless you really yearn for a headache, this is the right place to stop reading this post.

The theoretical problem is that each processor core might reorder the storing or fetching of data between registers, cache and memory in any possible way to increase performance. So if one one processor writes to X and then Y, and it’s crucial that the other processor sees the change in Y only when it also sees X updated, a memory barrier is often used. In the Linux kernel, smp_wmb() and smp_rbm() are used in conjunction to ensure this.

For example, if X is some data buffer, and Y is a flag saying that the data is valid. One processor fills the buffer X and then sets Y to “valid”. The other processor reads Y first, and if it’s valid, it accesses the buffer X. But what if the other processor sees Y as valid before it sees the data in X correctly? A store memory barrier before writing to Y and a read memory barrier before reading from X ensures this.

The thing is, that the Linux kernel’s implementation of smp_wmb() and smp_rbm() for x86 is a NOP. Note that it’s only the smp_*() versions that are NOPs. The non-smp fences turn into opcodes that implement fences. Assuming that the Linux guys know what they’re doing (which is a pretty safe assumption in this respect) they’re telling us that the view of ordering is kept intact across processor cores. In other words, if I can assure that X is written before Y on one processor, then Intel promises me that on another processor X will be read with the updated value before Y is seen updated.

Looking at how Microsoft’s examples solve certain multithreading issues, it’s quite evident that they trust the processor to retain the ordering of operations.

Memory fences are hence only necessary to ensure the ordering on a certain processor on x86. On different architectures (e.g. ARM v7) smp_wmb() and smp_rbm() actually do produce some code.

When are these fences really needed? From Intel’s 64 and IA-32 Architectures Software Developer’s Manual, Volume 2 (instruction set reference) vol. 2A page 4-22, on the “MFENCE” instruction:

Performs a serializing operation on all load-from-memory and store-to-memory instructions that were issued prior the MFENCE instruction. This serializing operation guarantees that every load and store instruction that precedes the MFENCE instruction in program order becomes globally visible before any load or store instruction that follows the MFENCE instruction. The MFENCE instruction is ordered with respect to all load and store instructions, other MFENCE instructions, any LFENCE and SFENCE instructions, and any serializing instructions (such as the CPUID instruction).

That didn’t answer much. I searched for fence instructions in the disassembly of a Linux kernel compiled for x86_64. A lot of fence instructions are used in the initial CPU bringup, in particular after setting CPU registers. Makes sense. Then there are several other fence instructions in drivers, which aren’t necessarily needed, but who has the guts to remove them?

Most interesting is where I didn’t find a single fence instruction: In any of the object files generated in kernel/locking/. In other words, Linux’ mutexes and spinlocks are all implemented without any fence. So this is most likely a good proof that they aren’t needed for anything else but things related to the CPU state itself. I guess. For a 64-bit x86, that is.

Going back to Microsoft, it’s interesting that their docs for userspace Interlocked functions say “This function generates a full memory barrier (or fence) to ensure that memory operations are completed in order”, but not those for kernel space. Compare, for example InterlockedOr() for applications vs. the same function for kernel. Truth is I didn’t bother to do the same disassembly test for application code.

Some barriers functions

(or: A collection of functions you probably don’t need, even if you think you do)

• KeFlushWriteBuffer(): Undocumented and rarely mentioned, intended for internal kernel use. Probably just makes sure that the cache has been flushed (?).
• KeMemoryBarrier(): Calls _KeMemoryBarrier(). But in wdm.h, there’s an implementation of this function, calling FastFence() and LoadFence(). But these are just macros for __faststorefence and _mm_lfence. Looked at next.
• _mm_lfence() : Turns into an lfence opcode. Same as rmb() in Linux.
• _mm_sfence(): Turns into an sfence opcode. Same as wmb() in Linux.
• _mm_mfence(): Turns into an mfence opcode.

I’ve verified that the _mm_*fence() builtins generated the said opcodes when compiled for x86 and amd64 alike. See some experiments on this matter below.

The deprecated _ReadBarrier(), _WriteBarrier() and _ReadWriteBarrier() produce no code at all. MemoryBarrier() ends up as a call to _MemoryBarrier().

Experimenting with fence instructions

(or: A major waste of time)

This is the code compiled:

__declspec(noinline) LONG do_it(LONG *p) {
  LONG x = 0;
  x += *p;
  _mm_lfence();
  x += *p;

  return x;
}

With a “checked compiation” this turns into:

_do_it@4:
  00000000: 8B FF              mov         edi,edi
  00000002: 55                 push        ebp
  00000003: 8B EC              mov         ebp,esp
  00000005: 51                 push        ecx
  00000006: C7 45 FC 00 00 00  mov         dword ptr [ebp-4],0
            00
  0000000D: 8B 45 08           mov         eax,dword ptr [ebp+8]
  00000010: 8B 4D FC           mov         ecx,dword ptr [ebp-4]
  00000013: 03 08              add         ecx,dword ptr [eax]
  00000015: 89 4D FC           mov         dword ptr [ebp-4],ecx
  00000018: 0F AE E8           lfence
  0000001B: 8B 55 08           mov         edx,dword ptr [ebp+8]
  0000001E: 8B 45 FC           mov         eax,dword ptr [ebp-4]
  00000021: 03 02              add         eax,dword ptr [edx]
  00000023: 89 45 FC           mov         dword ptr [ebp-4],eax
  00000026: 8B 45 FC           mov         eax,dword ptr [ebp-4]
  00000029: 8B E5              mov         esp,ebp
  0000002B: 5D                 pop         ebp
  0000002C: C2 04 00           ret         4

OK, this is too much. There is no ptimization at all. So let’s look at the “free” compilation instead:

_do_it@4:
  00000000: 8B FF              mov         edi,edi
  00000002: 55                 push        ebp
  00000003: 8B EC              mov         ebp,esp
  00000005: 8B 45 08           mov         eax,dword ptr [ebp+8]
  00000008: 8B 08              mov         ecx,dword ptr [eax]
  0000000A: 0F AE E8           lfence
  0000000D: 8B 00              mov         eax,dword ptr [eax]
  0000000F: 03 C1              add         eax,ecx
  00000011: 5D                 pop         ebp
  00000012: C2 04 00           ret         4

So clearly, the fence command made the compiler read the value from memory twice, as opposed to optimizing the second read away. Note that there’s no volatile keyword involved. So except for the fence, there’s no reason to read from *p twice.

The exact same result is obtained with _mm_mfence().

Trying with _mm_sfence() yields an interesting result however:

_do_it@4:
  00000000: 8B FF              mov         edi,edi
  00000002: 55                 push        ebp
  00000003: 8B EC              mov         ebp,esp
  00000005: 8B 45 08           mov         eax,dword ptr [ebp+8]
  00000008: 8B 00              mov         eax,dword ptr [eax]
  0000000A: 0F AE F8           sfence
  0000000D: 03 C0              add         eax,eax
  0000000F: 5D                 pop         ebp
  00000010: C2 04 00           ret         4

*p is read into eax once, then the fence, and then it’s added by itself. As opposed to above, where it was read into eax before the fence, then read again into ecx, and then added eax with ecx.

So the compiler felt free to optimize the two reads into one, because the store fence deals only with writes into memory, not reads. Given that there’s no volatile keyword used, it’s fine to optimize reads, which is exactly what it did.

The same optimization occurs if the fence command is removed completely, of course.

For the record, I’ve verified the equivalent behavior on the amd64 target (I’ll spare you the assembly code).

Here’s a quick demonstration. This is the “before” screenshot of the Certificate Manager window (click to enlarge):

Looking at the registry, I found 11 certificates in HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SystemCertificates\AuthRoot\Certificates\ and 12 certificates in HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SystemCertificates\ROOT\Certificates\, so it matches exactly certmgr’s view of 23 root certificates.

And so I had a .cab file with a certificate that requires Certum’s root certificate for validation. Clear from the screenshot above, it’s not installed.

Then I right-clicked that .cab file, selected Properties, then the “Digital Signature Tab”, selected the certificate and clicked Details, and boom! A new root certificate was automatically installed (click to enlarge):

And suddenly there are 12 certificates in the AuthRoot part of the registry instead of 11. Magic.

And here’s the behind the scenes of that trick.

Microsoft publishes a Certificate Trust List (CTL), which every computer downloads automatially every now and then (once a week, typically). It contains the list of root authorities that the computer should trust, however apparently they are imported into the registry only as needed. This page describes the concept of CTL in general.

I don’t know where this is stored on the disk, however one can download the list and create an .sst file, which opens certmgr when double-clicked. That lists all certificates of the downloaded CTL. 425 of them, as of May 2021, including Certum of course:

> certutil -generateSSTFromWU auth.sst

So it seems like Windows installs certificates from the CRL as necessary to validate certificate chains. This includes the GUI for examining certificates, verifying with signtool, as well as requiring the certificate for something actually useful.

There’s also a utility called CTLInfo out there, which I haven’t tried. It apparently displays the CTL currently loaded in the system, but I haven’t tried it out.

There’s another post in Stackexchange on this matter.

Besides, I’ve written a general post on certificates, if all this sounded like Chinese.

This is my best effort to summarize the steps to attestation signing for Windows drivers (see Microsoft’s main page on this). I’m mostly a Linux guy with no connections inside Microsoft, so everything written below is based upon public sources, trial and (a lot of) error, some reverse engineering, and speculations. This couldn’t be further away from the horse’s mouth, and I may definitely be wrong occasionally (that is, more than usual).

Also, the whole topic of attestation signature seems to be changing all the time, so it’s important to keep in mind that this reflects the situation of May 10th 2021. Please comment below as things change or whenever I got things wrong to begin with.

Attestation signing replaces the method that was available until April 2021, which was signing the driver locally by its author, just like any code signing. With attestation signing, Microsoft’s own digital signature is used to sign the driver. To achieve that, the driver’s .inf and .sys files are packed in a .cab file, signed by the author, and submitted to Microsoft. Typically 10 minutes later, the driver is signed by Microsoft, and can be downloaded back by the author.

Unfortunately, the signature obtained this way is recognized by Windows 10 only. In order to obtain a signature that works with Windows 7 and 8, the driver needs to get through an HLK test.

Signing up to the Hardware Program

This seemingly simple first step can be quite confusing and daunting, so let’s begin with the most important point: The only piece of information that I found present in Microsoft’s output (i.e. their signature add-ons), which wasn’t about Microsoft itself, was the company’s name, as I stated during the enrollment. In other words, what happens during the sign-up process doesn’t matter so much, as long as it’s completed.

This is Microsoft’s general how-to page for attestation signing in general, and this one about joining the hardware program. It wasn’t clear to me from these what I was supposed to do, so I’ll try to offer some hints.

The subscription to the Hardware Program can begin when two conditions are met:

• You have the capability to sign a file with an Extended Validation (EV) code signing certificate.
• You have an Azure Active Directory Domain Services managed domain (“Azure AD”).

Obtaining an EV certificate is a bureaucratic process, and it’s not cheap. But at least the other side tells you what to do, once you’ve paid. I went for ssl.com as their price was lowest, and working with them I got the impression that the company has hired people who actually know what they do. In short, recommended. Actually, read this first if you plan working with them.

So what’s the Domain Services thing? Well, this is the explanation from inside the Partner web interface (once it has already been set up): “Partner Center uses Azure Active Directory for identity and access management”. That’s the best I managed to find on why this is necessary.

For a single user scenario, this boils down to obtaining a domain name like something.onmicrosoft.com from Microsoft. It doesn’t matter if the name turns out long and daunting: It doesn’t appear anywhere else, and you’re not going to type it manually.

So here’s what to do: First thing first, create a fresh Microsoft account. Not really necessary if you already have one, but there’s going to be quite some mail going its way (some of which is promotional, unless you’re good at opting out).

Being logged into that account, start off on Azure’s main page. Join the 12-month free trial. It’s free, and yet you’ll need to supply a valid credit card number in the process. As of writing this, I don’t know what happens after 12 months (but see “Emails from Azure calling for an upgrade” below on developments).

The next step is to create that domain service. I believe this is Microsoft’s page on the topic, and this is the moment one wonders why there’s talk about DNSes and virtual networks. Remember that the only goal is to obtain the domain name, not to actually use it.

And here comes the fuzzy part, where I’m not sure I didn’t waste time with useless operations. So you may try following this, as it worked for me. But I can’t say I understand why these seemingly pointless actions did any good. I suspect that the bullets in italics below can be skipped — maybe it’s just about creating an Azure account, and not necessarily allocate resources?

So here are the steps that got me going:

• Log in to your (new?) Azure account.
• Go to Azure’s Portal (or click the “Portal” link at the top bar on Azure’s main page)

• Click “Create a resource” (at the top left) and pick Azure AD Domain Services.
• For Resource Group I created a new one, “the_resource_group”. I guess the name doesn’t matter.
• The DNS name doesn’t matter, apparently. yourcompany.onmicrosoft.com or something. It’s not going to appear anywhere.
• I set SKU set to Standard, as it appeared to be the least expensive one.
• After finishing the setup, it took about an hour for Azure to finish the configuration. Time for a long and well deserved coffee break.
• But then it complained that I need to set up DNSes or something. So I went along with the automatic fix.

(end of possibly useless steps)

• There’s this thing on the Register for the Hardware Program page saying that one should log in with the Global administrator account. This page defines Azure AD Global administrator as “This administrator role is automatically assigned to whomever created the Azure AD tenant”. So apparently for a fresh Azure account, it’s OK as is.
• At this point, you’re hopefully set to register to the Hardware Developer Program. After clicking “Next” on the landing page, you’ll be given the choice of “Sign in to Azure AD” or “Create a new directory for free”. The Azure AD is already set up, so log in with the account just created.
• A word about that “Create a new directory for free” option. To make things even more confusing, this appears to be a quick and painless shortcut, however in my case I got “This domain name is not available” for any domain name I tried with. Maybe I missed something, but this was a dead end for me. This is the page I didn’t manage to get through. Maybe your luck is better than mine. So this is why I created the Azure AD first, and then went for registration.
• Going on with the registration, you’re given a file to sign with your EV certificate. I got a .bin file, but in fact it had .exe or .sys format. So it can be renamed to .exe and used with cloud signature services (I used eSigner). BUT do this only if you’re going to sign the .cab files with the same machinery, or you’ll waste a few hours wondering what’s going on. Or read below (“When the signature isn’t validated”) why it was wrong in my case.
• And this is the really odd thing: Inside the Microsoft Partner Center, clicking the “your account” button (at the top right) it shows the default directory in use. At some point during the enrollment procedure, the link with the Azure AD I created was made (?), but for some reason, the default directory shown was something like microsoftazuremycompany.onmicrosoft.com instead of mycompany.onmicrosoft.com, which is the domain I created before. This didn’t stop me from signing a driver. So if another directory was used, why did I create one earlier?

After all this, I was set to submit drivers for signature: From this moment on, the entry point for signing drivers is the Microsoft Partner Center dashboard’s main page.

Emails from Azure calling for an upgrade

To make a long story short, quite a few emails arrived on behalf of Microsoft Azure, urging me to “upgrade” my account, i.e. to allow charging my credit card for Azure services. I ignored them all, and had no issues continuing to sign drivers.

And now to the details.

A day after signing up to Azure, I discovered that $20 had been reduced from my promotional free trial credit. Apparently, I had enabled stuff that would have cost real money. So I deleted the resources I had allocated in Azure. This includes deleting the mycompany.onmicrosoft.com domain, which was obviously ignored by the Partner Center. It was about deleting the the resource group (which contained 7 elements, with the domain included): Just clicking on the resource group in the main portal page, and then Delete Resource Group at the top. It took several minutes for Azure to digest that.

About a month later, I got a notification from Azure urging me to upgrade my account: It went

You’re receiving this email because your free credit has expired. Because of this, your Azure subscription and services have been disabled. To restore your services, upgrade to pay-as-you-go pricing.

Scary, heh? Does “services have been disabled” mean that I’m about to lose the ability to sign drivers?

Once again, “upgrade” is a cute expression for giving permission to charge the credit card that I had to give during submission. The details of which can’t be deleted from the account, unless I submit another, valid one, instead.

As a side note, it turned out that I had a Network Watcher group activated. Maybe I missed it earlier, and maybe it was somehow added. So I deleted it as well. But it’s not clear if this was related to the fact that the credits expired, whatever that means.

A few days on, another mail from Azure, basically the same, urging me to upgrade. One day after that, came an invoice. How come? I haven’t approved any payment. So it was an invoice on 0.00 USD. Zero. Why it was sent to me, is still unclear.

And finally, roughly two months after the initial subscription, I got a “We’re sorry to see you go” email from Azure, saying “Your free credit expired on (this and this date), and because of this we’ve deleted your subscription and any associated data and services”. Uhhm. What about driver signing? Well, I’ve spoiled the suspension above.

Two weeks after getting this last email, I deleted all cookies on my browser that were related to Microsoft, logged into my account at the Partner Center and submitted a driver for signature. The process went through smoothly.

Checking my Azure cloud account, it seemed to had been reset to its starting state, even with a suggestion to start another $200 free trial credit round. Detaching my credit card was however still impossible.

So apparently, there’s no problem just ignoring these emails, and continue signing forever. Emphasis on “apparently”.

Overview of the signature process

To make a long story short, you prepare a .cab file with the driver’s files, sign it with your EV Certificate, upload it to the Hardware Dashboard, and get it back after 10 minutes with Microsoft’s digital signatures all over the place.

So instead of signing the driver yourself, you pack the whole thing neatly, and send it to Microsoft for adding the .cat file and signing the drivers. And yet, you must sign the .cab file to prove that you’re the one taking responsibility for it. It’s Microsoft’s signature on the driver in the end, but they know who to bash if something goes wrong.

.cab files are exactly like .zip files, in the sense that they contain a directory tree, not just a bunch of files. Unfortunately, when looking at .cab files with Windows’ built-in utilities, the directory structure isn’t presented, and it looks like a heap of files. This holds true both when double clicking a .cab file and when using expand -D, from Windows XP all the way to Windows 10. Ironically enough, double-clicking a .cab file with Linux desktop GUI opens it correctly as a directory tree.

It’s important to consider .cab files like .zip, with hierarchy, because the way the driver is submitted is by organizing the files in directories exactly as they appear in the driver package for release, minus the .cat file. So what really happens is that Microsoft uncompresses the .cab file like a .zip, adds the .cat file and then performs the digital signing. It then compresses it all back into a .zip file and returns it back to you. The files remain in the same positions all through.

I guess the only reason .zip files aren’t uploaded instead of .cab, is that signtool doesn’t sign zips.

Some people out there, who missed this point, got the impression that the signing is done for each architecture separately. That’s possible, but there’s no reason to go that way. It’s just a matter of preparing the file hierarchy properly.

Preparing the .cab file

For reference, this is Microsoft’s short page on makecab and very long page on .cab files (which begins with cabarc, but goes on with makecab).

First, set up a .ddf file, looking something like this:

.Set CabinetFileCountThreshold=0
.Set FolderFileCountThreshold=0
.Set FolderSizeThreshold=0
.Set MaxCabinetSize=0
.Set MaxDiskFileCount=0
.Set MaxDiskSize=0
.Set CompressionType=MSZIP
.Set Cabinet=on
.Set Compress=on

;Specify file name for new cab file
.Set CabinetNameTemplate=thedriver.cab
.Set DiskDirectoryTemplate= ; Output .cab files into current directory

.Define pathtodriver=thedriver-dir

.Set DestinationDir=thedriver
%pathtodriver%\thedriver.inf
.Set DestinationDir=thedriver\i386
%pathtodriver%\i386\thedriver.sys
.Set DestinationDir=thedriver\amd64
%pathtodriver%\amd64\thedriver.sys

The .cab file is then generated with something like

> makecab /f thedriver.ddf

“makecab” is in Window’s execution path by default.

In my case of transitioning from self-signed drivers to attestation signature, there was already a script that generated the directory ready for releasing the driver. So the change I made was not to copy the .cat file into that directory, and instead of signing the .cat file, create a .cab.

The .ddf file above relates to a driver released for Intel architecture, 32 and 64 bits. The subdirectories in the driver package are i386 and amd64, respectively, as defined in the .inf file.

Changes you should make to the .ddf file:

• Replace all “thedriver” with the name of your driver (i.e. the name of the .inf and .sys files).
• Set “pathtodriver” to where the driver package is. Note that makecab’s /d flag allows setting variables, so the Define directive can be removed, and instead go something like

  > makecab /d pathtodriver=..\driverpackage thedriver.ddf

• Then possibly modify the files to be included. Each DestinationDir assignment tells makecab the directory position to place the file(s) that appear after it. This should match the structure of your release package’s directory structure.
• If the line doesn’t start with a dot, it’s the path to a file to copy into the .cab file. The path can be absolute (yuck) or relative to the current directory.

All in all, the important thing is to form a directory tree of a driver for release in the .cab file.

The .ddf file shown above is a working example, and it includes only the .inf and .sys files. Including a .cat file is pointless, as Microsoft’s signature machinery generates one of its own.

As for .pdb files, it’s a bit more confusing: Microsoft’s main page includes .pdb files in the list of “typical CAB file submissions” (actually, .cat it listed too there), and then these files don’t appear in the .ddf file example on the same page. The graphics showing a tree for multiple package submissions is inconsistent with both.

A .pdb files contains the symbol map of the related .sys file, allowing the kernel debugger to display meaningful stack traces and disassemblies, in particular when analyzing a bugcheck. These files are not included in a driver release, not mentioned in the .inf file, not referenced in the .cat file and are therefore unrelated to the signature of the driver. Technically, Microsoft doesn’t need these files to complete an attestation signature.

Microsoft nevertheless encourages submitters of drivers to include .pdb files. When these file are missing in a driver submission, a popup shows up in the web interface saying “This submission does not include symbols. It is recommended to upload symbols within each driver folder”. This however doesn’t stop the process, and not even delay it, in case you’re slow on confirming it. So it’s up to you if you want to include .pdb’s.

Submitting the .cab file

The command for signing the .cab file is:

> signtool.exe sign /fd sha256 thedriver.cab

Note that timestamping is not required, but won’t hurt. The whole idea with timestamping is to make the signature valid after the certificates expire, but the .cab file is examined soon after the signature is made, and after that it has no more importance.

Note that ssl.com also offers an eSigner tool for signing the .cab file with a simple web interface. Just be sure to have registered with a signature made in eSigner as well, or things will go bad, see “When the signature isn’t validated” below. Or add eSigner’s certificate to the existing subscription.

November 2021 update: You probably don’t want to use eSigner at all. See this post.

Then the submission itself:

• In Microsoft Partner Center’s dashboard, click at “Drivers” on the left menubar. It might be necessary to click “Hardware” first to make this item appear.
• Click the “Submit new hardware” button at the top left to get started.
• Give the submission a name — it’s used just for your own reference, and surely won’t appear in the signed driver package.
• Drag the signed cab file to where it says to.
• The web interface requires selecting Windows releases in a lot of checkboxes. More on this just below.
• Click “Submit” to start the machinery. Once it finishes, successfully or not, it sends a notification mail (actually, three identical mails or so. Not clear why not only one).
• If and when the entire process is completed successfully, the driver can be downloaded: Under “Packages and signing properties”, there’s a “More” link. Click it, and a button saying “Download signed files” appears. So click it, obviously.

Now to the part about selecting Windows versions. It’s an array of checkboxes. This is a screenshot of this stage (click to enlarge):

First, the easy part: Don’t check the two at the top saying “Perform test-signing for X”. It says “Leave all checkboxes blank for Attestation Signing” in fine print above these.

Now we’re left with a whole lot of Windows 10 release numbers and architectures. From a pure technical point of view, there’s no need for this information to perform the signature, since the .inf file contains the information of which architectures are targeted.

Rather, this is the “attestation” part: Just above the “Submit” button, it says “You have completed quality testing of your driver for each of the Operating Systems selected above”. So this is where you testify which platforms you’ve tested the driver with. The deal is that instead of going through the via dolorosa of HLK tests, Microsoft signs the driver for you in exchange to this testimony. Or should I say, attestation.

Just to spell it out: The signature can’t and doesn’t limit itself to specific operating system builds, and it would be insane doing so, as it wouldn’t cover future Windows releases.

I have to admit that in the beginning I misunderstood this part, and tried to select as much as possible. And because my driver wasn’t compiled for arm64, and I had clicked versions saying “ARM64″, the submission was rejected with “thedriver.inf does not have NTARM64 decorated model sections” (in UniversalLog.txt). It was bit of a computer game to check the right boxes and avoid the wrong ones.

So no need to be greedy. Common sense is to test the driver on one operating system release for each architecture. In the example above, it’s for a driver released for Intel architecture, 32 and 64 bits. The checkbox selection reflects testing it with Windows 10 release 1607, 32- and 64-bit architecture. This is the proper way to go.

And yet, for the heck of it I tried submitting the same driver package with a single OS checked (1607 x64). To my surprise, the package was accepted and signed despite my declaration that it hadn’t been tested for the 32-bit version, even though a .sys file for that architecture was part of the package.

All in all, there must be a match between the architectures targeted by the driver (as listed in the .inf file) and those inferred by the selection of OSes. Nevertheless, it seems like Microsoft lets you get away with not testing all of them. In short, checking just one checkbox may be enough, even if the driver supports multiple architectures.

Looking at the signed zip

After receiving back the signed driver, I examined the files. My findings were:

• The .inf file is left completely intact (bytewise identical to the one in the .cab file).
• A signed .cat file was added.
• All .sys files were signed as well (contrary to what most of us do when releasing drivers). This makes the driver eligible for inclusion during boot.

Looking at the digital signatures with an ASN.1 dump utility, it’s appears like the only place there’s something not pointing at Microsoft, is an non-standard spcSpOpusInfo entry in the crypto blob, where the company’s name, appears in wide char format in the programName field (no, I’m not mistaken). This appears to be taken from the “Publisher display name” as it appears in the Account Settings in the Microsoft Partner Center dashboard.

So all in all, there are almost no traces to the fact that the driver’s origin isn’t Microsoft. Except for that entry in the crypto blob, which is most likely invisible unless the signature is analyzed as an ASN.1 file or string searched (with a tool that detects wide char strings). So it appears like all information, except for that “Publisher display name” remains between you and Microsoft.

When the signature isn’t validated

Sometimes, the process fails at the “Preparation” stage. As always on failures, the web interface suggest downloading a “full error report”. That report is a file named UniversalLog.txt file. If it says just “SignatureValidationFailed”, something went wrong with the signature validation.

The solution for this is to make sure that the certificate that was used for signing the .cab file is registered: Within Microsoft Partner Center, click the gear icon at the top right, select “Account Settings” and pick “Manage Certificates” at the left menu bar. That’s where the relevant certificate should be listed. The first time I got to this page, I saw the same certificate twice, and deleted one of those.

In my case the problem was that during the registration, I had made the signature with the cloud app (eSigner), but signed the driver with a local USB key dongle. As it turned out, these have different certificates.

So the solution was to delete the registered certificate from the account, and register the new one by signing a file with the local USB dongle. Doing this is a good idea in any case, because if something is wrong with the signature produced by signtool, it will fail the registration as well. So whether this renewed registration succeeds or fails, it brings you closer to the solution.

Sample certificate chains

For reference, these are examples of certificate chains: One properly signed .cab file and one for a the .cat file that has been attestation signed my Microsoft.

Note the /pa flag, meaning Default Authenticode Verification Policy is used. Or else verification may fail. Also note that the file isn’t timestamped, which is OK for submission of attestation signing.

> signtool verify /pa /v thefile.cab

Verifying: thefile.cab

Signature Index: 0 (Primary Signature)
Hash of file (sha256): 388D7AFB058FEAE3AEA48A2E712BCEFEB8F749F107C62ED7A41A131507891BD9

Signing Certificate Chain:
    Issued to: Certum Trusted Network CA
    Issued by: Certum Trusted Network CA
    Expires:   Mon Dec 31 05:07:37 2029
    SHA1 hash: 07E032E020B72C3F192F0628A2593A19A70F069E

        Issued to: SSL.com EV Root Certification Authority RSA R2
        Issued by: Certum Trusted Network CA
        Expires:   Mon Sep 11 02:28:20 2023
        SHA1 hash: 893E994B9C43100155AE310F34D8CC962096AE12

            Issued to: SSL.com EV Code Signing Intermediate CA RSA R3
            Issued by: SSL.com EV Root Certification Authority RSA R2
            Expires:   Wed Mar 22 10:44:23 2034
            SHA1 hash: D2953DBA95086FEB5805BEFC41283CA64C397DF5

                Issued to: THE COMPANY LTD
                Issued by: SSL.com EV Code Signing Intermediate CA RSA R3
                Expires:   Fri May 03 13:09:33 2024
                SHA1 hash: C15A6A7986AE67F1AE4B996C99F3A43F98029A54

File is not timestamped.

Successfully verified: thefile.cab

Number of files successfully Verified: 1
Number of warnings: 0
Number of errors: 0

One possibly confusing situation is to check if the root certificate exists before ever running this verification on a fresh Windows installation. It may not be there, but then the verification is successful, and the root certificate appears from nowhere. That rare situation is explained in this post.

Next up is the attestation signed .cat file:

> signtool.exe verify /kp /v thedriver.cat

Verifying: thedriver.cat

Signature Index: 0 (Primary Signature)
Hash of file (sha256): ED5231781724DEA1C8DE2B1C97AC55922F4F85736132B36660FE375B44C42370

Signing Certificate Chain:
    Issued to: Microsoft Root Certificate Authority 2010
    Issued by: Microsoft Root Certificate Authority 2010
    Expires:   Sat Jun 23 15:04:01 2035
    SHA1 hash: 3B1EFD3A66EA28B16697394703A72CA340A05BD5

        Issued to: Microsoft Windows Third Party Component CA 2014
        Issued by: Microsoft Root Certificate Authority 2010
        Expires:   Mon Oct 15 13:41:27 2029
        SHA1 hash: 1906DCF62629B563252C826FDD874EFCEB6856C6

            Issued to: Microsoft Windows Hardware Compatibility Publisher
            Issued by: Microsoft Windows Third Party Component CA 2014
            Expires:   Thu Dec 02 15:25:28 2021
            SHA1 hash: 984E03B613E8C2AE9C692F0DB2C031BF3EE3A0FA

The signature is timestamped: Mon May 10 03:10:15 2021
Timestamp Verified by:
    Issued to: Microsoft Root Certificate Authority 2010
    Issued by: Microsoft Root Certificate Authority 2010
    Expires:   Sat Jun 23 15:04:01 2035
    SHA1 hash: 3B1EFD3A66EA28B16697394703A72CA340A05BD5

        Issued to: Microsoft Time-Stamp PCA 2010
        Issued by: Microsoft Root Certificate Authority 2010
        Expires:   Tue Jul 01 14:46:55 2025
        SHA1 hash: 2AA752FE64C49ABE82913C463529CF10FF2F04EE

            Issued to: Microsoft Time-Stamp Service
            Issued by: Microsoft Time-Stamp PCA 2010
            Expires:   Wed Jan 12 10:28:27 2022
            SHA1 hash: AAE5BF29B50AAB88A1072BCE770BBE40F55A9503

Cross Certificate Chain:
    Issued to: Microsoft Root Certificate Authority 2010
    Issued by: Microsoft Root Certificate Authority 2010
    Expires:   Sat Jun 23 15:04:01 2035
    SHA1 hash: 3B1EFD3A66EA28B16697394703A72CA340A05BD5

        Issued to: Microsoft Windows Third Party Component CA 2014
        Issued by: Microsoft Root Certificate Authority 2010
        Expires:   Mon Oct 15 13:41:27 2029
        SHA1 hash: 1906DCF62629B563252C826FDD874EFCEB6856C6

            Issued to: Microsoft Windows Hardware Compatibility Publisher
            Issued by: Microsoft Windows Third Party Component CA 2014
            Expires:   Thu Dec 02 15:25:28 2021
            SHA1 hash: 984E03B613E8C2AE9C692F0DB2C031BF3EE3A0FA

Successfully verified: thedriver.cat

Number of files successfully Verified: 1
Number of warnings: 0
Number of errors: 0

Doing the same with the .sys file yields exactly the same result, with slight and meaningless differences in the timestamp.

Clearly, the certificate chain ends with “Microsoft Root Certificate Authority 2010″ rather than the well-known “Microsoft Code Verification Root”, which is the reason the attestation signature isn’t recognized by Windows 7 and 8.

Microsoft as a Certificate Authority, approving itself all through the chain. It’s quite odd this happened only now.

These are just a few jots on Wine. I guess this post will evolve over time.

I’m running Wine version 4.0 on Linux Mint 19, running on an x86_64.

First run

Every time Wine is run on a blank (or absent) directory given by WINEPREFIX, it installs a Windows environment. Which Windows version an several other attributes can be set with Wine Configuration:

$ WINEPREFIX=/path/to/winedir /opt/wine-stable/bin/winecfg

Note that Wine installs a 64-bit environment by default. Add WINEARCH=win32 after the WINEPREFIX setting on the first run of Wine for the relevant directory to install a 32-bit environment instead, which is recommended for its better support unless 64-bit applications are going to be used. No need for WINEARCH afterwards.

It often suggests to install Wine Mono and Wine Gecko. I usually tend to agree.

This installation downloads three files into .cache/wine/: wine_gecko-2.47-x86_64.msi, wine_gecko-2.47-x86.msi and wine-mono-4.7.5.msi. This is why Wine doesn’t ask for permission to install these when setting up new Windows environments after the first time.

Install and use Winetricks

It’s a good idea in general, and it allows installation of Microsoft runtime environment easily:

# apt install winetricks
# apt install wine32-development

And now to install Virtual Studio 6 runtime environment, for example (solving some error message on not being able to import isskin.dll or isskinu.dll)

$ WINEPREFIX=/path/to/winedir winetricks vcrun6sp6

For a list of all packages available, go

$ WINEPREFIX=/path/to/winedir winetricks list-all | less

Prevent browser popup

Wine has this thing that it opens a browser when so requested by the Windows application. That can be annoying at times, and get the program stuck when run inside a firejail. To prevent this altogether, just delete two files:

• drive_c/windows/syswow64/winebrowser.exe
• drive_c/windows/system32/winebrowser.exe

but that didn’t work with Picasa, because it opened the browser through its own xdg-open located at /opt/picasa/bin/xdg-utils-1.0.2/scripts/xdg-open. So I replaced it with the following lame script

#!/bin/bash
/usr/bin/konqueror "$1"

so at least it doesn’t mix in the real browser (which caused a mess at times).

Open explorer

The simplest way to start: Open the file explorer:

$ WINEPREFIX=/path/to/winedir /opt/wine-stable/bin/wine explorer

DOS command line

$ WINEPREFIX=/path/to/winedir /opt/wine-stable/bin/wine cmd

This is better than expected: The command session is done directly in the console (no new window opened). Like invoking a shell.

$ firejail --profile=~/my.profile --env=WINEPREFIX=/path/to/winedir /opt/wine-stable/bin/wine 'C:\Program Files\Malsoft\Malsoft.exe' &

include /etc/firejail/disable-common.inc
include /etc/firejail/disable-passwdmgr.inc
private-tmp
private-dev

# All relevant directories are read-only by default, not /opt. So add it.
read-only /opt
#
# This whitelisting protects the entire home directory.
# .cache/wine is where the Gecko + Mono installation files are kept.
# They can't be downloaded, because of "net none" below
mkdir ~/sandboxed/
mkdir ~/.cache/wine
whitelist ~/sandboxed/
whitelist ~/.cache/wine

net none
nonewprivs
caps.drop all
noroot
# blacklist everything that can be harmed
#
blacklist /mnt
blacklist /cdrom
blacklist /media
blacklist /boot

$ WINEPREFIX=/path/to/winedir /opt/wine-stable/bin/winecfg

Still in 2020, it seems like Microsoft lives up to its reputation: Being arrogant, thinking that anyone in business must be a huge corporate, and in particular ending up completely ridiculous. Microsoft’s mail servers, which accept on behalf of Hotmail, MSN, Office 365, Outlook.com, or Live.com users are no exception. This also affects companies and other entities which use their own domain names, but use Microsoft’s services for handling mail.

This post summarizes my personal experience and accumulated knowledge with delivering mail to their servers. I use a simple Linux sendmail SMTP MTA on a virtual server for handling the delivery of my own private mails as well as a very low traffic of transactional mails from a web server. All in all, it’s about 100 mails / month coming out from that server to all destinations.

So one server, one IP address with a perfect reputation on all open spam reputation trackers, with SPF, DKIM and DMARC records all in place properly.

One may ask why I’m not relying on existing mail delivery services or my ISP. Answer is simple: Any commercial mail delivery server is likely to have its reputation contaminated by some spammer, no matter what protection measures they take. When that happens, odds are that emails will just disappear, because the ISP has little interest in forwarding the bounce message saying that delivery failed. On a good day, they will be handling the problem quickly, and yet the sender of the lost mail won’t be aware that the correspondence is broken.

For this reason, it’s quite likely that small businesses will go on keeping their own, small, email delivery servers, maintaining their own reputation. So when Outlook’s servers are nasty with a single-IP server, they’re not just arrogant, but they are causing delivery issues with small to medium businesses.

To do when setting up the server

For starter info, go here. Microsoft is pretty upfront about not being friendly to new IP addresses (see troubleshooting page for postmasters).

So it’s a very good idea to create a Microsoft account to log into their services, and then join their Smart Network Data Service (SDNS) and Junk Mail Reporting Program. This is the start page for both of these services.

SDNS allows the owner of a mail server to register its IP address range (“Request Access“), so its status can be monitored (“View IP Status”) over time. When all is fine, the IP Status page says “All of the specified IPs have normal status”, and when they don’t like this or other IP address, it’s more like this (click to enlarge):

The Junk Mail Reporting Program (JMRP) allows the owner of the mail server to receive notifications (by email) when a mail message is delivered however deemed suspicious, either by an end-user (marking it as spam) or by automatic means. So it’s a good idea to create a special email address for this purpose and fill in the JMRP form. Even for the sake of claiming that you got no complaints when contacting support later on.

Note that this is important for delivery of mail to any institution relies on Microsoft’s mail infrastructure. A proper IP address blacklist delisting takes you from

Mar 11 20:18:23 sm-mta[5817]: x2BKIL2H005815: to=<xxxxxxx@mit.edu>, delay=00:00:02, xdelay=00:00:02, mailer=esmtp, pri=121914, relay=mit-edu.mail.protection.outlook.com. [104.47.42.36], dsn=5.7.606, stat=User unknown

(but the bounce message indicated that it’s not an unknown user, but a blacklisted IP number) to

Mar 11 21:15:12 sm-mta[6170]: x2BLF8rT006168: to=<xxxxxxx@mit.edu>, delay=00:00:03, xdelay=00:00:03, mailer=esmtp, pri=121915, relay=mit-edu.mail.protection.outlook.com. [104.47.42.36], dsn=2.0.0, stat=Sent (<5C86CFDC.6000206@example.com> [InternalId=11420318042095, Hostname=DM5PR01MB2345.prod.exchangelabs.com] 11012 bytes in 0.191, 56.057 KB/sec Queued mail for delivery)

Note that the session response said nothing about a blacklisted IP, however the bounce message (not shown here) did.

Finally, Microsoft suggest getting a certification from Return Path. A paid-for service, clearly intended for large companies and in particular mass mailers to get their spam delivered. Microsoftish irony at its best.

To do when things go wrong

First thing first, read the bounce message. If it says that it’s on Microsoft’s IP blacklist, go to the Office 365 Anti-Spam IP Delist Portal and delist it.

Then check the IP’s status (requires logging in). If you’re blocked, contact support. This doesn’t require a Microsoft login account, by the way. I’m not sure if this link to the support page is valid in the long run, so it’s on SNDS’ main page (“contact sender support”) as well as Troubleshooting page.

My own ridiculous experience

I kicked off my mail server a bit more than a year ago. There was some trouble in the beginning, but that was no surprise. Then things got settled and working for a year, and only then, suddenly & out of the blue, a mail to a Hotmail address bounced with:

Action: failed
Status: 5.7.1
Diagnostic-Code: SMTP; 550 5.7.1 Unfortunately, messages from [193.29.56.92] weren't sent. Please contact your Internet service provider since part of their network is on our block list (S3140). You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors. [VE1EUR01FT021.eop-EUR01.prod.protection.outlook.com]

And indeed, checking the IP status indicated that is was blocked “because of user complaints or other evidence of spamming”.

So first I went to the mail logs. Low traffic. No indication that the server has been tricked into sending a lot of mails. No indication that it has been compromised in any way. And when a server has been compromised, you know it.

No chance that there were user complaints, because I got nothing from JMRP. So what the “evidence of spamming”?

My best guess: A handful transactional mail messages (at most) to their servers for authenticating email addresses that were marker suspicious by their super software. Putting these messages in quarantine for a few hours is the common solution when that happens. Spam is about volume. If all you got was 4-5 messages, how could that be a spam server? Only if you look at percentage. 100% suspicious. Silly or what?

So I filled in the contact support form, and soon enough I got a message saying a ticket has been opened, and 30 minutes later saying

We have completed reviewing the IP(s) you submitted. The following table contains the results of our investigation.

Not qualified for mitigation
193.29.56.92
Our investigation has determined that the above IP(s) do not qualify for mitigation. These IP(s) have previously received mitigations from deliverability support, and have failed to maintain patterns within our guidelines, so they are ineligible for additional mitigation at this time.

Cute, heh? And that is followed by a lot of general advice, basically copied from the website, recommending to join JMRP and SDNS. Which I had a year earlier, of course. The script that responded didn’t even bother to check that.

But it also said:

To have Deliverability Support investigate further, please reply to this email with a detailed description of the problem you are having, including specific error messages, and an agent will contact you.

And so I did. I wrote that I had joined those two programs a year ago, that the mail volume is low and so on. I doubt it really made a difference. After sending the reply, I got a somewhat automated response rather quickly, now with a more human touch:

Hello,

My name is Ayesha and I work with the Outlook.com Deliverability Support Team.

IP: 193.29.56.92

We will be looking into this issue along with the Escalations Team. We understand the urgency of this issue and will provide an update as soon as this is available. Rest assured that this ticket is being tracked and we will get back to you as soon as we have more information to offer.

Thank you for your patience.

Sincerely,
Ayesha

Outlook.com Deliverability Support

And then, a few days later, another mail:

Hello,

My name is Yaqub and I work with the Outlook.com Deliverability Support Team.

Recent activity coming from your IP(s): ( 193.29.56.92) has been flagged by our system as suspicious, causing your IP to become blocked. I have conducted an investigation into the emails originating from your IP space and have implemented mitigation for your deliverability problem. This process may take 24 – 48 hours to replicate completely throughout our system.

Please note that lifting the block does not guarantee that your email will be delivered to a user’s inbox. However, here are some things that can help you with delivery:

(and here came the same suggestions on JMRP and SDNS)

And about 24 hours later, the IP status went back to OK again. And my emails went through normally.

Well, almost. A few days even further down, I attempted to send an email to a live.co.uk destination, and once again, I got the same rejection message (in block list, S3140). The only difference was that the mail server on the other side was hotmail-com.olc.protection.outlook.com (residing in the US), and now eur.olc.protection.outlook.com (somewhere in Europe).

I checked the IP’s status in SDNS and it was fine. So updating the Europeans on the updated IP status takes a bit time, or what?

So I replied to last email I got from Microsoft’s support, saying it failed with live.co.uk. I didn’t get any reply, but a few hours later I tried again, and the mail went through. Coincidence or not.

This time I also caught the related messaged from the mail log. It’s

May 01 15:10:28 sm-mta[2239]: 041FASMh002237: to=<xxxxx@live.co.uk>, ctladdr=<eli@billauer.se> (510/500), delay=00:00:00, xdelay=00:00:00, mailer=esmtp, pri=121816, relay=eur.olc.protection.outlook.com. [104.47.1.33], dsn=5.0.0, stat=Service unavailable
May 01 15:10:28 sm-mta[2239]: 041FASMh002237: 041FASMh002239: DSN: Service unavailable

for a failure, and

May 02 06:23:00 sm-mta[4024]: 0426Mx1I004021: to=<xxxxx@live.co.uk>, ctladdr=<eli@billauer.se> (510/500), delay=00:00:01, xdelay=00:00:01, mailer=esmtp, pri=121808, relay=eur.olc.protection.outlook.com. [104.47.18.97], dsn=2.0.0, stat=Sent (<5EAD11C3.20105@billauer.se> [InternalId=21887153366859, Hostname=AM6EUR05HT060.eop-eur05.prod.protection.outlook.com] 10627 bytes in 0.246, 42.064 KB/sec Queued mail for delivery -> 250 2.1.5)

for success.

Lesson learned: Contact support and insist.

And the lesson to all those using Microsoft’s mail services: Your provider cuts off your email contacts arbitrarily. Because they are Microsoft.

These are pretty random jots that I made while evaluating named pipes as a solution for project. I eventually went for a driver in the kernel for various reasons, so I never got to verify that anything written below is actually correct.

I’ve also written a small post on epoll with named pipes (in Linux, of course) along with a tiny test program.

Linux named pipes (FIFOs)

Section 3.162 of IEEE Std 1003.1-2001, Base Definitions, defines FIFO Special File (or FIFO) trivially, and refers to POSIX IEEE Std 1003.1-2001 for “Other characteristics” related to lseek( ), open( ), read( ), and write( ). In section 3.273 it defines Pipe, and says that it “behaves identically to a FIFO special file”.

From POSIX IEEE Std 1003.1-2001, System Interfaces, Issue 6, line 27153: “When opening a FIFO with O_RDONLY or O_WRONLY set: (…) If O_NONBLOCK is clear, an open( ) for reading-only shall block the calling thread until a thread opens the file for writing. An open( ) for writing-only shall block the calling thread until a thread opens the file for reading.”

Even though the POSIX standard is available for download at a cost of a few hundred dollars, there’s the Open Group Base Specification, which matches it quite well: Base Definitions and System Interfaces.

This is a good source on pipes.

• The buffer for each pipe is 64 kB by default on Linux
• An anonymous pipe (created with pipe() ) is just like a named pipe (i.e. a FIFO special file), only that the node is in pipefs, which only the kernel has access to. There are slight differences in how they’re handled.

Reading the source code (fs/pipe.c)

• This source clearly handles both named and anonymous pipes (the behavior differs slightly).
• There’s F_SETPIPE_SZ and F_GETPIPE_SZ fcntl calls, which clearly allow setting the buffer size. Added in patch 35f3d14dbbc58 from 2010 (v2.6.35).
• fifo_open(): The is_pipe boolean is true if the FIFO is a pipe() (has a magic of PIPE_FS). In other words, if it’s false, it’s a named pipe.
• If a FIFO is opened for read or write (not both), the open() call blocks until there’s a partner, unless O_NONBLOCK is set, in which case the behavior is somewhat messy. The comments imply that this is required by POSIX.
• It’s perfectly fine that a FIFO is opened multiple times both for read and write. Only one reader gets each piece of data, and quite randomly. Writers contribute to the stream independently.
• FIFOs and pipes implement the FIONREAD ioctl() command for telling how many bytes are immediately ready for reading. This is the only ioctl() command implemented however.
• The flags returned by epoll_wait() are as understood from pipe_poll(). Not clear what purpose the list of EPOLL* flags in those calls to wake_up_interruptible_sync_poll() has.

Things to look at:

• What if a file descriptor open for read is closed and there’s no write() blocking on the other side? Is there a way to get a software notification? A zero-length write always succeeds, even if the other side is closed (this case is handled explicitly before the part that produces the EPIPE / SIGPIPE).
• Same in the other direction: A descriptor open for write closes, but there’s no read() to get the EOF. Likewise, a zero-length read() always succeeds, even if the other side is closed (which makes sense, because even if it’s closed, reading should continue until the end).
• Is there a way to wait for the partner without blocking, or must a thread block on it?
• What happens on a close() followed immediately by an open()?
• What about user opening the file in the wrong direction?

open() with O_NONBLOCK? (not)

It’s tempting to open a FIFO with O_NONBLOCK, so there needs not to be a thread blocking while waiting for the other side to be opened. POSIX IEEE Std 1003.1-2001, System Interfaces, Issue 6,says in the part defining open(), page 836:

When opening a FIFO with O_RDONLY or O_WRONLY set:

• If O_NONBLOCK is set, an open( ) for reading-only shall return without delay. An open( ) for writing-only shall return an error if no process currently has the file open for reading.
• If O_NONBLOCK is clear, an open( ) for reading-only shall block the calling thread until a thread opens the file for writing. An open( ) for writing-only shall block the calling thread until a thread opens the file for reading.

In the list of error codes for open(), ENXIO is bound (along with another irrelevant sceniario) to: O_NONBLOCK is set, the named file is a FIFO, O_WRONLY is set, and no process has the file open for reading.

Linux’ implementation of FIFOs follows this exactly.

This rules out using O_NONBLOCK for opening a file for write — it will simply not work. As for opening a file for read, it will work, but the epoll() call won’t wake up the process before there is data to read. Opening the other side only doesn’t generate any event.

Windows named pipes

The basic reading: Microsoft’s introduction and a list of relevant API functions.

General notes:

• The concept of named pipes in Windows seems to resemble UNIX domain sockets in that it’s formed with a client / server terminology. Unlike domain sockets, the client may “connect” with a plain file open() (actually CreateFile and variants). Hence Windows’ named pipes can be truly full duplex between two processes.
• But named pipes can also be accessed remotely. The permissions need to be set explicitly to avoid that.
• The client’s opening of a named pipe is known by the return of ConnectNamedPipe (or a respective event in OVERLAPPED mode).
• Pipe names are not case-sensitive.
• A pipe is created (by the server) with access mode PIPE_ACCESS_INBOUND, PIPE_ACCESS_OUTBOUND or PIPE_ACCESS_DUPLEX, indicating its direction, as well as the number of instances — the number of times the pipe can be opened by clients. There’s PIPE_TYPE_BYTE and PIPE_TYPE_MESSAGE types, the former creating a UNIX-like stream, and the second treats writes to the pipe as atomic messages.
• A named pipe ceases to exist when its number of instances goes to zero. It’s an object, not a file. Hence for a sensible application where a single process generates the named pipe, they vanish when the process terminates.
• Use PIPE_WAIT even when non-blocking behavior is required. Don’t use PIPE_NOWAIT flag on creation for achieving non-blocking use of the pipe. Overlapping access is the correct tool. The former is available to allow compatibility with some Microsoft software accident.
• If a client needs to connect to a pipe which has all its instances already connected, it may wait for its turn with WaitNamedPipe.
• When finishing a connection with a client, FlushFileBuffers() should be called to flush pending written data (if the client didn’t close the connection first) and then DisconnectNamedPipe().
• The suitable mode for working is overlapped I/O. This is the official example.
• There’s a scary remark on this page and this claiming that named pipes are effectively useless for IPC, and that the object’s name has changed, and this is also discussed here. It seems however that this remark doesn’t relate to anything else than UWP apps. Or else it wouldn’t have cause the vulnerability mentioned here, and how could Docker use it this way?

Windows named pipes: Detecting disconnection of client

The word out there is that a disconnection can’t be detected unless there’s an outstanding ReadFile or WriteFile request. This holds true in particular for TCP sockets. So first try this: Make sure there’s only one instance, and let the server call WaitNamedPipeA as soon as a connection is made. This call will hopefully return when the real client disconnects. This will not work if Windows wants the server to disconnect as well before considering the pipe instance vacant enough. It might, because the client is allowed to connect before the server. It all depends on when Windows decrements the reference count and cleans up.

Hmmm… So the MBR was overwritten by GRUB, and now I need to get it back to Windowish. One can use Microsoft’s rescue disk-on-key, or the quick hack: Download ms-sys, compile with plain “make”, don’t bother to install, and just go from the bin/ directory:

# ./ms-sys -7 /dev/sda

and Windows 10 boots like a charm.

Either way, it’s the regular Windows 10 you want to download. Not the N or KN or something.

Wanting to be sure that some driver I’ve released will work with Windows 10, I upgraded from Windows 7, where the driver was installed, to Windows 10.

To my great surprise, Windows 10 started with the same desktop, including program shortcuts, all running as before. Only a new look and feel, which resembles Windows 8, just slightly less annoying.

I should mention that at the “Get going fast” stage of the installation, I went for Customize Settings and turned off basically everything. That’s where all the “Send Microsoft X and Y” goes.

The real surprise was that my own driver was already installed and running on the upgraded Windows 10. If I was looking for a sign that everything is the same under the hood, an automatic adoption of already installed driver is a good one. I don’t think Microsoft would risk doing that unless there was really nothing new.

Needless to say, removing the driver and reinstalling it went as smooth as always. Same device manager, same everything.

IMPORTANT: For a bare-metal install, boot the USB stick with the ISO image (possibly generated with winusb under Ubuntu) in non-UEFI mode, or the installer refuses to use existing MBR partitions (unless the partition table is GPT anyhow).

VirtualBox installation notes

Jots while installing a fresh Windows 10 on VirtualBox v4.3.12 under 64-bit Linux. Correction: I eventually upgraded to v5.0.12, which lists Windows 10 as a target OS. This was required to make the Windows Addons work.

• Set OS to Other Windows 32 bit (I suppose other Microsoft selections will work as well)
• Under processor, enable PAE/NX
• Attempting a 64-bit installation got stuck on the initial Windows splash image, no matter what I tried (maybe this was solved with 5.0.12, didn’t try this)
• Turn off screen saver after installation
• The installation will ask for the product key in two different occasions during the installation. Just skip.
• Didn’t work:
  In order to install VirtualBox Windows Additions, pick Devices > Insert Guest Additions CD Image… on the hosts’s VirtualBox menu. Then start the virtual machine. VirtualBox v4.3.12 doesn’t support Windows 10, so refuse the automatic run of the CD. Instead, browse the disc’s content and right-click VBoxWindowsAdditions-x86.exe. Choose Properties. Pick the Compatibility tab, check “Run this program in compatibility mode” and pick Windows 8 (as suggested on this post). Then run this program, which will then install the drivers properly. Windows will complain that the display adapter doesn’t work, but that’s fine. Just reboot.

In short, before switching to a new hardware, it’s essential to prepare it, or an 0x0000007b blue screen will occur on the new hardware.

The trick is to run sysprep.exe (under windows\system32\sysprep\) before the transition. Have “Generalize” checked, and choose “shutdown” at the end of the operation (“Shutdown Options”).

Once the computer shuts down, move the hard disk to the new computer. Windows should boot smoothly, and start a series of installation stages, including feeding the license key and language settings. Also, an account needs to be created. This account can be deleted afterwards, as the old account is kept. Quite silly, as a matter of fact.

Spoiler: Eventually, an accidental attempt brought things back to normal.

What installation is this?

I had some trouble telling if the installation was 32 or 64 bit. “systeminfo” didn’t work on command prompt, so I had to guess based upon the existing files. Be sure to look in C: and not in X: (are they the same?).

The ways I could tell it’s a 64-bit installation:

• The presence of C:\Program Files (x86)  as well as C:\Program Files (only the latter is present in a 32-bit installation).
• The presence of C:\Windows\SysWOW64, which is intended for running 32-bit programs under a 64-bit OS.

Note that C:\Windows\System32 is present in both 32- and 64-bit versions.

Random ideas

Some things one might want to try out:

• Run Linux on a LiveUSB stick and run Applications > System Tools > Disk Utility (or similar path) to get some S.M.A.R.T. info from the hard disk.
• Run chkdsk /R on Windows’ command prompt to hopefully fix the disk issues
• Create installation media on Window’s site (Google for it), as the self-fixing tools ask for it. The exact version of Windows is required for that, so “systeminfo” on command prompt should be helpful (if it says X86-based PC it’s 32 bit Windows, otherwise X64-based PC). In principle, looking at the computer’s properties is easier, but in recovery mode only command prompt is available.
• Choose “Refresh your PC” under “Troubleshoot” in the set of menus that appear when the computer fails to boot properly. The plug in the installation media when requested (don’t do it beforehand, as it won’t count). Not that it helped. I got “The media inserted is not valid. Try again.” So much for a descriptive error message, after preparing that silly USB stick for an hour or so.

So I went for booting the computer from USB, and got the “The drive where Windows is installed is locked. Unlock the drive and try again” error when trying to repair the OS. Following this page, I ran diskpart at command prompt and typed “list volume” which indeed printed out the disk partitions. This was done to make sure none appears as “RAW”, which would indicate that I don’t want to touch anything before the disk has been restored to a sane condition.

I also tried

bootrec /fixmbr
bootrec /fixboot
bootrec /rebuildbcd

but in vain. There was no difference.

sfc

It’s supposed to be the savior, isn’t it?

Looking for corrupted system files (System File Checker): “sfc /scannow” completed (verification 100% complete) but said that “Windows Resource Protection could not perform the requested operation”. The log file was found in X:\WINDOWS\LOGS\CBS\CBS.LOG (note the X:, it was put in the boot volume, not C:) with notepad (don’t forget to look for “All Files” and not just *.txt).

The reason it did nothing was that I didn’t run it as an Administrator. So going again, as an Administrator, I got “There is a system repair pending which requires reboot to complete. Restart Windows and run sfc again.” So I rebooted, and got the same message again. Very helpful. This page suggests looking what is pending in c:\windows\winsxs\pending.xml, and indeed such file existed. A long XML file, full with info about things that were about to happen.

Following this page I went for

dism.exe /image:C:\ /cleanup-image /revertpendingactions

which is an extremely annoying utility in that it claims that it doesn’t recognize the /cleanup-image nor /revertpendingactions options unless the line is typed exactly as above. Did I say something about helpful error messages?

So eventually it finished, and claimed to have done that successfully, but sfc still said there was pending system repair. Rebooting didn’t help. The pending.xml file was still there.

Trying dism again, it claimed having an error reverting an image, and sending me to a log file. Which, as one would expect, contained tons of rubbish and not much to go with. The reported error was 0x800f082f, which seems to be an undocumented error code. This post supplied a hack for working it around, but it wasn’t required in my case — the problem was the pending.xml file

As it previously complained about not having “sufficient scratch space” I also supplied the /scratchdir:c:\delme option the following time (with c:\delme being just an empty directory).

At this point I decided to rename pending.xml to was-pending.xml using notepad, which wasn’t all that simple (Remember to use the C: path, and not the X: default, remember to view all files and not just .txt, and note that changes are not updated in the GUI until I leave the directory and view it again. Things are weird when running in rescue mode). It would make more sense to use some File Manager, but “explorer” on command prompt wasn’t recognized.

sfc claimed not have completed the operation successfully (with a status_not_implemented error in the log). But that didn’t make any difference either.

Boot logging

This can be triggered off with F8 in theory, but probably only when things are relatively OK: Instead, at the Automatic Repair opening screen, go Advanced Options > Troubleshoot > Advanced Options > Startup Settings > Restart. So Windows reboots, but this time with a menu. Pick option 2, Enable boot logging. And boot, which fails again, of course with the same Automatic Repair Screen.

So go Advanced Options > Troubleshoot > Advanced Options > Command Prompt, enter as admin, and open type “notepad”.  Well, the file was supposed to be there as C:\windows\ntbtlog.txt, but no such file was there. Microsoft has a rather useless possible explanation.

The breakthrough

Based upon the same menu for enabling Boot logging, I picked (8) “Disable early-launch anti-malware protection”. And after a little while, the computer was suddenly up and running!

From a running Windows position, I was offered to run System Restore to a known, recent configurations, and agreed.

The computer started munching and crunching, after which it restarted, and brought me back to the Automatic Repair screen. But now it mentioned a log file on this initial screen: C:\Windows\System32\Logfiles\Srt\SrtTrail.txt. And there it said: Boot critical file C:\Windows\system32\drivers\mfeelamk.sys is corrupt. And also that C:\tbs.sys is corrupt.

So turning off that malware option again, a successful boot was completed again, and again with a message that System Restore had failed.

Indeed there was no C:\tbs.sys file at all, but it was found on C:\Windows\system32\drivers\ with zero lenght. The mfeelamk.sys file turns out to be McAfee’s anti-malware driver, and let’s believe that it was problematic indeed. But that explains why turning off early malware check solved the issue.

So I renamed the tbs.sys  so that Windows won’t find it (requires changing ownership and then permissions first) and ran sfc /scannow (from a running system this time, which is much slower). It ended up saying that it found some corrupt files, but was unable to fix some of them.

And then the computer booted up as usual. As simple as that.

So in hindsight, the problem was with one or two driver files which failed to load. Instead of saying that, Windows went “I’m sorry, you’re too stupid to be exposed to that information” and left me to guessing. And most people don’t complain. Just reinstall everything. Or even better, buy a new computer.

Lessons learned

• Always run chkdsk /r before trying to mess with the computer
• sfc /scannow in rescue mode is worthless
• Don’t try to be clever when fixing a Windows computer. The breakthrough step can’t be figured out logically on a senseless system. Just try things at random.
• Prepare a recovery USB stick, while all is fairly OK