Why is my default timezone Monrovia, Reykjavik?

I recently sent a friend an email from work and he pointed out my email timezone is set to Monrovia, Reykjavik and that I should refer to http://support.microsoft.com/kb/2298834.

Here is my response...

Oh, I gave up running Outlook and I'm accessing email via the Office365 interface.

Outlook had (has) some nasty bug where it takes a UI lock in a thread and then posts blocking web calls from the same thread. When the blocking calls actually take a bit of time (because the company isn't good at running data centers?) my dev machine would lose the ability to CHANGE WINDOW FOCUS or otherwise process windows messages.

I tried debugging this issue and after attaching windbg to Outlook.exe and discovering ~150 CLR threads I decided this effort was hopeless. I'm still unclear why they use the UI thread to post blocking web calls in an environment where they already have 149 OTHER threads they could use for this. Thus, I decided to use the https://outlook.com/owa/XXXX.com site instead for corporate email.

HOWEVER, more comedy ensued since MSFT apparently isn't very good with cookie management and continually tried to use my XXXX.X.XXXX@live.com tokens to access my corporate email -- the concept that I'd want to be logged in as XXXX.X.XXXX@live.com and NNNN@XXXX.com at the SAME TIME to different Microsoft website properties is completely alien.

Thus I've settled on my current solution, a shortcut: "iexplore.exe -private https://outlook.com/owa/XXXX.com"

This works very well since it ignores any persistent cookies on the system via InPrivate mode. However, I guess persistent settings like my actual timezone cannot be accounted for and I'm not even prompted for them (some UX designer probably focus grouped this and convinced themselves it was the optimal solution to just use the null timezone setting).

I thought about using Chrome exclusively for my Outlook.com needs but alas, Google seems to think the key to a successful browser is launching FIVE Chrome.exe processes utilizing ~75MB OF MEMORY to provide access to their useless "Hang Outs" feature. Which I hear now supports (THANKFULLY, the world was empty without this feature) Mustaches and Hats so I guess 75MB of RAM is a small price to pay for MUSTACHES AND HATS in a failing social media landing point that replaced the highly effective and useful googletalk.

Gone are the days of a trim gtalk.exe client and Outlook.exe (not .com) just working (for various levels of "working" exceedingly greater than the current implementation). I realize I'm becoming a crusty fossil in the industry; (already?!) clinging to concepts like trim code, easy to understand and debug projects, and my generate-the-object-code-I-intend C coding methodology...

Dispatch Objects

It seems like the next step in my OS development is going to be laying down some kernel work. I thought I'd be able to delay this, but doing so would only result in less useful work being done. A kernel is responsible for many things. I'm sure many more things than I even know about right now (such is the nature of this whole undertaking). However, my experience writing Windows drivers has offered a decent skeleton to start with.

One piece of this skeleton is dispatch objects. Basically, this is any sort of thing that can be waited upon. A timer, an event, a lock, a thread (joining a thread, etc.). I figure this is as decent a place to start as any. The way a dispatch object will work in my kernel is it will be a light weight structure containing a list of waiting threads. Of course, this means I need to start defining a thread structure as well. I imagine I'll continue to flesh out the guts of the kernel (or at least their structures) by jumping off on this one spot.

Since a dispatch object is essentially a list, the first structure I need to create is a list structure. There are many ways to create a list. You can create a linked list, a vector, or various other structures. Again, leaning on my Windows background I'll choose something very similar to the LIST_ENTRY structure in the NT kernel. In more data structure books I've seen a doubly-linked list defined as:

typedef struct _LIST_NODE
{
    struct _LIST_NODE* pPrev;
    struct _LIST_NODE* pNext;
    void* pData;
} LIST_NODE;

Here, I'll rely on something clever that Windows does with it's list structure -- there's no "data" pointer. The structure is intended to be embedded directly into any other data structure:

typedef struct _LIST_NODE
{
    struct _LIST_NODE* pPrev;
    struct _LIST_NODE* pNext;
} LIST_NODE;

typedef struct
{
    LIST_NODE  ThreadListHead;
    ...
} DISPATCH_HEADER;

typedef struct
{
    UINT32     ThreadId;
    ...
    LIST_NODE  DispatchListNode;
    ...
} THREAD_OBJECT;

The way this works is the list pointers are offset from to find the original object the list points to. In the DISPATH_HEADER above, this is simply the same address. However, say I had additional members before the ThreadListHead member, you'd then SUBTRACT back to get the original object pointer. This is done using a member offset macro, where the offset of a member into a structure is computed and then subtracted off the list pointer. While this may appear clunky at first it has one very nice advantage - you don't need to dynamically allocate small chunks of memory. In the example above, you would simply have a list insertion function that you'd pass a pointer to the THREAD_OBJECT::DispatchListNode member variable to.

There are multiple reasons this is a good thing. First, the previously mentioned avoidance of small memory allocations. Second, by not relying on any sort of memory allocation this data structure can be used in the kernel before memory allocation is even possible -- obviously, you need to write an allocator before you can use one. Third (but related to the first), the absence of smaller allocations for what is a widely used construct can add up to some pretty substantial memory savings. Any memory allocation will require a chunk of memory for bookkeeping. This is usually going to be at least the size of a few pointers. For small allocations of only 3 pointers (in the first list structure definition) the overhead of the allocation may very well be larger than the actual data usage. Considering how widely used the LIST_NODE structure will be, this is a substantial savings in memory as well as access speed (there isn't an indirect memory access to get to the actual structure, if you have the pointer to the list, it's only a subtraction to get the actual pointer versus another read).

So, there it is, the beginnings of a DISPATCH_HEADER, a THREAD_OBJECT, and a LIST_NODE. This kernel is practically writing itself...

Rolling Your Own Operating System

I started an extremely ambitious project a while back -- working on my own operating system. This isn't one that I ever intend to gain any ground or even boot on real hardware. (I'm targeting VirtualBox, VirtualPC/HyperV, and Bochs emulator environments right now.) It's just a place for me to play around with OS concepts. At work I write driver code for windows, but this code relies on a system of interfaces and mechanisms that have been designed and implemented by people much smarter than I. I'd like to start working on those more basic concepts. Sort of like peeling back the layers of an onion by recreating it. I anticipate I will learn much not only about how my OS works and can work, but about the tradeoffs that most OS designers have encountered along the way.

My current "OS" is very limited. It uses the standard DOS (well, Windows 7 -- because there are differences...) bootloader from the partition table. I created my own volume boot record to receive control from there. I'm able to load from the beginning of the partition. I'm still working on the loader. This is not be choice, but actually some unexpected first bit of learning. Here's the deal...

The BIOS loads and calls into the MBR code. The MBR code loads and calls into my volume boot record code. Now the fun starts. There are numerous extremely important interfaces the BIOS provides to early boot code. Before protected mode, paging, multi-tasking, and all the other things that really make an operating system an operating system you could just happily keep calling these BIOS interfaces to get this functionality. However, the BIOS code is legacy... It's so legacy that it isn't all that compatible with the way a modern OS sets up the CPU. (There are a few more compatible extensions -- VESA graphics BIOS stuff for example, but even this is severely limited)

So, it's already decision time. A lot sooner than I thought it would be. How do I start laying the ground work for the more modern features of my OS when I need to use the BIOS routines to talk to the hardware? Well, the short answer is I need to stop at some point, and probably the sooner the better. I don't want to litter my kernel code with a bunch of BIOS dependencies -- anything I write that requires this cannot really be reused anywhere else. Ideally, I would switch over from BIOS interfaces to my own all at once, however that will be difficult. Instead, I've decided a better approach is to rely on the BIOS interfaces until I'm ready to create my own. So, what does the BIOS give me that I need to reproduce?

1. VGA/VESA support -- in order to know my OS is doing something, I should probably have a way to see it working. ¹
2. Keyboard support.
3. Disk IO support.
4. ...

I'm sure there are many more but these are the interfaces I'm immediately aware of. The first item, VGA support was easiest. This is probably because it's also the only one that's moved past legacy support. I am able to select a video mode which allows me to draw 24/32 bit RGB directly into a flat buffer space. This buffer space is directly mapped to the screen buffer, so I'm set as far as rudimentary drawing is concerned. I don't get any hardware acceleration and I'm stuck having to implement all my own line drawing, bit-blitting, and other APIs, but at least it's doable. The most annoying thing about this is I cannot adjust for changing the monitor resolution, but if I look at my above goals of not really caring if this runs on real hardware this is acceptable.

Now comes disk IO. This is a bit sticky, since there is no easy way to go about this. I'm probably going to have to support at least IDE and AHCI. I'm not exactly sure if I can get away with IDE only and rely on hard disks to support the legacy commands... I think so, but not sure. I also cannot keep thunking back and forth between BIOS code for this support -- every time I do that I'll probably have to tear down the things I really wanted to learn about -- memory paging, scheduling, etc. I see some work on a storage driver in my near future... It does indeed look like I can just go with IDE. Hopefully, IDE will be pretty easy to work with. It looks like the specs are open as well -- http://www.t10.org/t13/technical/d98120r0.pdf

Here's what my volume boot record code looks like so far: http://www.darkautomata.com/blog/os/vbr_2013_11_29.asm.txt

---

¹ I can just use a COM port interface for logging what is actually happening in the OS, but there is going to come a time relatively early in the OS where I'll actually want to SEE what I've created. Maybe this isn't as big of a requirement as I think it is right now...

Screen DPI vs Remoting issues

It seems to me the user experience for high DPI displays like the Apple Retina display will be horrible for certain scenarios.

If I'm remoting into a windows machine using Remote Desktop or displaying an X application using X11 remoting then the host DPI settings are probably going to result in a bad experience. I already see this in when using my work laptop to remote into my desktop.

At work I have multiple 24" 1920x1080 monitors. When I RDP into the machine from my 14" 1920x1080 work laptop I get text in the code editor that is barely legible. I can't imagine how bad this would be on the Retina display!

The Remote Desktop and other remote display clients should enable a pass-through mechanism for client DPI. This way the host can probably format the display for remoting.

To expand on this idea, if all UI settings relating to size were stored in a device independent way (for example: twips, 1/1440 of an inch) then a translation could take place to always match the physical dimensions of the UI. You could even have a constant scaling factor so you can change this for a device. For example, I may not want my cell phone displaying at full physical dimensions, maybe it's okay to go with 50% or 25% and I know I have to squint at it, but I want my laptop or tablet to use 80-90% physical dimensions, etc...

If no one has thought of this yet well it's here first... prior art recorded.

Native or Managed?

I want to start a project with a friend to upgrade the UI of windbg. It's not too bad right now, but more importantly I think it could be so much more.

So here's the problem: Making an "awesome" UI is much easier in C# than in C++ using native code. However, the APIs for the debugging engine are all COM based and there aren't bindings for C#. There are a few unofficial bindings, but nothing really available. So, we'd probably have to roll our own bindings, which is not something I want to do.

I think a hybrid option would be to use C++/CLI. At least we'd be able to use the COM methods pretty easily and maybe even use the header files without any modification. We could also then use the managed API for the GUI. I think I'm going to suggest this as the route to take.

There's some concerns that .Net wouldn't be installed on all the environments you'd want to use the new UI but I don't think that should stop us. It's pretty easy to install and a version of .Net ships with modern Windows distributions already.

A bit to think about, but I think the C++/CLI approach is going to be the way to go. It will get me writing code faster and having SOMETHING working much faster than a native solution. I've recently worked on native GDI+ code and although it's nicer than GDI it's still not something I'd want to force myself on for a GUI intensive project.

ERROR_WORKING_SET_QUOTA and IO Completion Ports

While working on a streaming engine I came across an interesting little hole in the MSDN documentation for Completion Ports. Completion Ports allow for extremely efficient throughput of data. The way this is accomplished is by queuing IO to a "Completion Port" and then associating the Completion Port with one or more threads.

The reason this is so fast is because Windows can then chose which thread will complete the IO operation. Using a thread pool allows Windows to always pick the last executed thread in LIFO order. This greatly reduces TLB thrashing and other issues associated with a context switch on the CPU. When the threads aren't processing IO they are in a wait state. The IO is processed in a FIFO order.

The usual Completion Port architecture looks something like this:

1. Create a Completion Port using CreateIoCompletionPort.
2. Create the threads for the thread pool and call GetQueuedCompletionStatus to associate the threads with the Completion Port.
3. Associate file HANDLEs (opened in Overlapped IO mode) to the Completion Port.
4. Issue IO operations using ReadFile/WriteFile.
5. Process the IO operations in the thread pool threads.

When any IO operation completes Windows will smartly choose a thread waiting on GetQueuedCompletionStatus to wake up and send the IO result. The call to GetQueueCompletionStatus will return and data processing can begin. Ideally, an application would probably only have one Completion Port and perform all IO processing on this port/thread pool pair.

Everything about this is awesome, except... The documentation is really vague about how to handle ReadFile/WriteFile operations returning success (and thus not being queued). You need to make sure you call GetOverlappedResult (and probably with the Wait parameter set to FALSE) or you will start getting strange errors.

After a few of these immediate IO completions my streaming engine started hitting ReadFile failures described by "ERROR_WORKING_SET_QUOTA." Nowhere in the documentation for Completion Ports or GetOverlappedResult does it indicate this should be called in the Completion Port case. I suppose it's implied by the fact that you're using Overlapped IO, but still an explicit indication on MSDN would probably be useful.

This may be obvious to some but I wasted about an hour on this, so hopefully this post will shorten that time for someone else.

"R6025 - pure virtual function call" Uh oh...

A while ago I stumbled upon a "purecall" crash. The actual fix is pretty boring, but it is interesting to think about why pure-call crashes can happen at all. Since I like looking at disassembly and seeing what the compiler actually does with my code I'll take this approach here.

I recently spoke with a smart friend who's going through the process of learning C++ coding at a university. We started talking about abstract classes and one thing led to another until I brought up pure-call exceptions. More discussion ensued and I posed the question, "How can you actually cause a pure-call exception?"

If you think about it a bit, this should be impossible. There's no way to instantiate an abstract class. The compiler just won't let you. For any real subclasses of the abstract class the compiler will fill in an appropriate function pointer table. So what's the deal? How does this happen?

First, let's have a look at the memory structure of a typical C++ object containing virtual functions.

0:000:x86> ?? tmp
class BaseReal * 0x004a49a0
   +0x000 __VFN_table      : 0x01312110
   +0x004 m_data           : 0x1337beef

0:000:x86> dps 0x01312110 L5
01312110  013110c0 cppstuff!BaseReal::`scalar deleting destructor'
01312114  01311120 cppstuff!BaseReal::Get 
01312118  01311140 cppstuff!BaseReal::Sum 
0131211c  00000000
01312120  00000048

Here's an object tmp which contains a virtual function table pointer __VFN_table and then a single data element called m_data. I can see from dumping pointer-sized chunks of __VFN_table with symbol matching turned on that it's actually got the function pointers for the class called BaseReal.

This corresponds to the following code listing:

#include <stdio.h>
#include <stdlib.h>

class BaseAbstract
{
public:
    BaseAbstract ();
    virtual ~BaseAbstract ();
    
    virtual unsigned int Get () = 0;
    virtual unsigned int Sum ();
};

class BaseReal : public BaseAbstract
{
public:
    BaseReal ();
    virtual ~BaseReal ();
    
    virtual unsigned int Get ();
    virtual unsigned int Sum ();
    
    unsigned int m_data;
};

BaseAbstract::BaseAbstract ()
{
    //Sum ();  // BOOM.
}

BaseAbstract::~BaseAbstract ()
{
}

unsigned int BaseAbstract::Sum ()
{
    return Get () + 0;
}

BaseReal::BaseReal () : BaseAbstract ()
{
    m_data = 0x1337BEEF;
}

BaseReal::~BaseReal ()
{
    m_data = 0xDEADBEEF;
}

unsigned int BaseReal::Get ()
{
    return m_data;
}

unsigned int BaseReal::Sum ()
{
    return Get () + m_data;
}

int main (int argc, char* argv[])
{
    BaseReal *tmp = new BaseReal ();
    
    unsigned int value = tmp->Sum ();   // <<--- break point here.

    delete tmp;
    return 0;
}

If I uncomment the "BOOM" line and run again the application will crash before it gets to the break point. The interesting part is what happens before the crash. Let's have a look at the constructors disassembly. First, the BaseReal constructor:

0:000:x86> uf cppstuff!BaseReal::BaseReal
cppstuff!BaseReal::BaseReal :

   // Function prologue...
   41 00df1090 55              push    ebp
   41 00df1091 8bec            mov     ebp,esp

   // Setting up the "this" pointer (ecx usually contains 'this') and then
   // calling the BaseAbstract constructor.
   41 00df1093 51              push    ecx
   41 00df1094 894dfc          mov     dword ptr [ebp-4],ecx
   41 00df1097 8b4dfc          mov     ecx,dword ptr [ebp-4]
   41 00df109a e861ffffff      call    cppstuff!BaseAbstract::BaseAbstract (00df1000)

   // Loading eax with pointer to 'this' and then storing the virtual function
   // table for BaseReal (cppstuff!BaseReal::`vftable' (00df2120)
   41 00df109f 8b45fc          mov     eax,dword ptr [ebp-4]
   41 00df10a2 c7002021df00    mov     dword ptr [eax],
            offset cppstuff!BaseReal::`vftable' (00df2120)

   // Saving 0x1337BEEF to m_data.
   42 00df10a8 8b4dfc          mov     ecx,dword ptr [ebp-4]
   42 00df10ab c74104efbe3713  mov     dword ptr [ecx+4],1337BEEFh

   // Function epilogue...
   43 00df10b2 8b45fc          mov     eax,dword ptr [ebp-4]
   43 00df10b5 8be5            mov     esp,ebp
   43 00df10b7 5d              pop     ebp
   43 00df10b8 c3              ret


// Dumping the function table...
0:000:x86> dps cppstuff!BaseReal::`vftable' L3
00df2120  00df10c0 cppstuff!BaseReal::`scalar deleting destructor'
00df2124  00df1120 cppstuff!BaseReal::Get 
00df2128  00df1140 cppstuff!BaseReal::Sum 

This looks pretty reasonable. First there's the function prologue and then we do some C++ "this pointer" setup to make all that work. After that we immediately jump into the constructor for BaseAbstract. Once that work is done the m_data member is initialized. And now a look at the BaseAbstract constructor:

0:000:x86> uf cppstuff!BaseAbstract::BaseAbstract
cppstuff!BaseAbstract::BaseAbstract :

   // Function prologue...
   27 00df1000 55              push    ebp
   27 00df1001 8bec            mov     ebp,esp

   // Setting up the "this" pointer (ecx usually contains 'this') and 
   // saving it on the stack as a local in preparation for calling
   // the "Sum" function.
   27 00df1003 51              push    ecx
   27 00df1004 894dfc          mov     dword ptr [ebp-4],ecx

   // Loading eax with pointer to 'this' and then storing the virtual function
   // table for BaseReal (cppstuff!BaseAbstract::`vftable' (00df2110)
   27 00df1007 8b45fc          mov     eax,dword ptr [ebp-4]
   27 00df100a c7001021df00    mov     dword ptr [eax],offset 
            cppstuff!BaseAbstract::`vftable' (00df2110)
   28 00df1010 8b4dfc          mov     ecx,dword ptr [ebp-4]

   // Calling Sum -- which will fail.
   28 00df1013 e858000000      call    cppstuff!BaseAbstract::Sum (00df1070)

   // Function epilogue...
   29 00df1018 8b45fc          mov     eax,dword ptr [ebp-4]
   29 00df101b 8be5            mov     esp,ebp
   29 00df101d 5d              pop     ebp
   29 00df101e c3              ret


// Dumping the function table...
0:000:x86> dps cppstuff!BaseAbstract::`vftable' L3
00df2110  00df1020 cppstuff!BaseAbstract::`scalar deleting destructor'
00df2114  00df1224 cppstuff!purecall
00df2118  00df1070 cppstuff!BaseAbstract::Sum 


// Disassembly for BaseAbstract::Sum -- called in the constructor.
0:000:x86> uf cppstuff!BaseAbstract::Sum
cppstuff!BaseAbstract::Sum :

   // Prologue...
   36 00df1070 55              push    ebp
   36 00df1071 8bec            mov     ebp,esp

   // Saving ecx.  This is somewhat important.  Note: the "this call"
   // calling convention requires the "this pointer" to be in ecx.  The code
   // is using ebp-4 to stash the "this" pointer.
   36 00df1073 51              push    ecx

   // Copying "this" (ecx) to local storage -- anything with negative
   // ebp references is a local / spill location.
   36 00df1074 894dfc          mov     dword ptr [ebp-4],ecx

   // Load the address of the function table into eax.
   37 00df1077 8b45fc          mov     eax,dword ptr [ebp-4]

   // Dereference eax into edx -- now we have the function table.
   37 00df107a 8b10            mov     edx,dword ptr [eax]

   // Set ecx to "this" for the call, per calling convention.
   37 00df107c 8b4dfc          mov     ecx,dword ptr [ebp-4]

   // Deference the 2nd function table entry (the one for "Get").
   37 00df107f 8b4204          mov     eax,dword ptr [edx+4]
   37 00df1082 ffd0            call    eax

   // Epilogue...
   38 00df1084 8be5            mov     esp,ebp
   38 00df1086 5d              pop     ebp
   38 00df1087 c3              ret

There's the same "this" pointer initialization (although this was probably already done, these constructors have to work in a vacuum, so they may duplicate a little work). Next, the setup of the virtual function table and then the call to Sum. I think the compiler took a nice optimization here and didn't use the virtual function table to get the address of Sum. If this were code anywhere other than the constructor I imagine it would have used the function table pointer instead.

So now this brings us to the Sum code, which I also dumped. You can see it dereferences the virtual function table for the Get function call and then calls it. The problem is this is a pure virtual function so the table entry is for cppstuff!purecall; which is a function added by the compiler as a placeholder to indicate failure.

What are the lessons learned? You should never call virtual functions (or functions that call virtual functions) in the constructor or destructor. I didn't show the destructor code, but the whole process of loading the proper function table pointer and setting it is reversed.

Clear as mud?