I noticed I haven't written much for quite a long time. So, let's see. While working out the beginnings of my own operating system I came across the realization that every user mode thread pretty much needs to start with an IRET.
What's an IRET? It's an Interrupt RETurn instruction. It tells the CPU that it should be returning from an interrupt and as such, needs to pop some additional state off the stack. The funny thing is part of this state is which ring protection level to resume. You can transition to higher (and thus lower privilege) RPLs using IRET. It's also apparently the only way to perform this transition.
So... something even cooler (at least to me). Every single time a thread is reactivated from a context switch it is actually resuming via an IRET instruction. The symmetry of this made me happy. If a user mode thread issues a system call (Sleep, WaitForSingleObject, ReadFile, WriteFile, etc.) that's an interrupt! If a thread runs out of quantum, it will get kicked out by a timer interrupt.
The tricky part is getting the stack set up in such a way that the IRET pops off all the register context just how you need it to. Luckily, this is actually pretty easy (the instruction which triggered the interrupt did that for us). It's quite a bit harder to actually start a thread, because all those values need to be properly initialized by the kernel code.
Because I know everyone is curious, here's some additional resources:
Wednesday, December 7, 2016
Tuesday, April 28, 2015
Tabs! No, SPACES!! NO!!! TABS!!!!11111one..dot..
A recent discussion with my mentee touched on something I've been thinking about for a while. What's the difference between a scientist and an engineer? Scientists and engineers employ many of the same techniques and skills in their work. However, it's important to remember what a scientist does and what an engineer does. As far as I'm concerned, the simplest description is that engineers turn discoveries made by scientists into "something" useful. Scientists don't usually ship products. An extremely successful career in science may involve never shipping a single thing or perhaps one or two epic discoveries. The usual terms for an engineer who doesn't care about shipping a product are homeless, broke, or professor.
So, where are the major arguments in software engineering? Here's a few I've encountered and how I feel about them. (I'm not a journalist; I can offer up my opinion. Most journalists seem to offer up their opinion as well these days so whatever.) Of course, I try to temper my own opinions when discussing this with my mentee.
Tabs! No, SPACES! No, TABS!
This isn't as prevalent as it used to be. Other than at my university over 10 years ago I've only encountered it once in actual industry work. An engineer absolutely insisted on using tabs because one tab character took up less space on disk than four space characters. I prefer to be able to print code, open it in ANY text editor, or various other things that pretty much require uniform spacing -- which means spaces rather than tabs. The engineer who argued vehemently for tabs was also the kind of guy you'd probably expect to find hand-tuning assembly to optimize the E_UNIVERSE_HEAT_DEATH error handler mentioned below. In my opinion, one of those kind of engineers you wish would just stay in his corner mumbling about punch cards, FORTRAN, and the origin of the term "core dump."
Crystal palace coding
The idea your code (and everyone else's) must be absolutely perfect. If there's a trailing space at the end of a line it's not a NIT1 it's a giant catastrophe and "people could literally DIE if you don't fix this space at the end of the line." There's also the issue where braces go, how case statements are formatted, CamelCase vs. under_scores and other naming conventions, and numerous other horrible things that are completely swallowed by the lexical parsing stage of the compiler. Don't get me wrong though, there are good reasons to have coding standards and the dev team should also stick to the coding standards. However, since the end customer doesn't care about the coding standards at all there are probably better areas to invest which will surprise and delight the customer. The best part about these coding standards is once you switch teams or companies there will be an entirely new set of them and arguments why the new ones are the best...
E_UNIVERSE_HEAT_DEATH
Handling every single possible error case no matter how ridiculous. Sometimes I'll hear someone ask how code will handle something completely insane. My favorite is out of memory issues. "You should throw an exception if this allocation fails." -- Really? We should try to allocate an exception object after an allocation failed? How about if we just crash instead? Is it really all that bad to crash during a catastrophic error? Is someone going to catch that OOM exception and do something insanely clever with it so we can continue proper execution again? The much more likely case is the error will bubble up a few callstacks below and then someone will spend hours binary-search debugging the originator of the error code. That's an unbelievable waste of time just to adhere to someone's (I'm looking at you Bjarne Stroustrup!) ideal method of error handling. Sometimes I want to ask in response, "How do we detect if someone jams their penis into the CD-ROM tray? Should we define E_PENIS_NOT_EXPECTED and provide an appropriate message file to display helpful error text to the user? Can we DCR the CD-ROM firmware to detect illegal penis insertions?"
In summary...
I'll probably come back to this post and update it with new insights. It'll be fun to see how my views change over the years.
So, where are the major arguments in software engineering? Here's a few I've encountered and how I feel about them. (I'm not a journalist; I can offer up my opinion. Most journalists seem to offer up their opinion as well these days so whatever.) Of course, I try to temper my own opinions when discussing this with my mentee.
Tabs! No, SPACES! No, TABS!
This isn't as prevalent as it used to be. Other than at my university over 10 years ago I've only encountered it once in actual industry work. An engineer absolutely insisted on using tabs because one tab character took up less space on disk than four space characters. I prefer to be able to print code, open it in ANY text editor, or various other things that pretty much require uniform spacing -- which means spaces rather than tabs. The engineer who argued vehemently for tabs was also the kind of guy you'd probably expect to find hand-tuning assembly to optimize the E_UNIVERSE_HEAT_DEATH error handler mentioned below. In my opinion, one of those kind of engineers you wish would just stay in his corner mumbling about punch cards, FORTRAN, and the origin of the term "core dump."
Crystal palace coding
The idea your code (and everyone else's) must be absolutely perfect. If there's a trailing space at the end of a line it's not a NIT1 it's a giant catastrophe and "people could literally DIE if you don't fix this space at the end of the line." There's also the issue where braces go, how case statements are formatted, CamelCase vs. under_scores and other naming conventions, and numerous other horrible things that are completely swallowed by the lexical parsing stage of the compiler. Don't get me wrong though, there are good reasons to have coding standards and the dev team should also stick to the coding standards. However, since the end customer doesn't care about the coding standards at all there are probably better areas to invest which will surprise and delight the customer. The best part about these coding standards is once you switch teams or companies there will be an entirely new set of them and arguments why the new ones are the best...
E_UNIVERSE_HEAT_DEATH
Handling every single possible error case no matter how ridiculous. Sometimes I'll hear someone ask how code will handle something completely insane. My favorite is out of memory issues. "You should throw an exception if this allocation fails." -- Really? We should try to allocate an exception object after an allocation failed? How about if we just crash instead? Is it really all that bad to crash during a catastrophic error? Is someone going to catch that OOM exception and do something insanely clever with it so we can continue proper execution again? The much more likely case is the error will bubble up a few callstacks below and then someone will spend hours binary-search debugging the originator of the error code. That's an unbelievable waste of time just to adhere to someone's (I'm looking at you Bjarne Stroustrup!) ideal method of error handling. Sometimes I want to ask in response, "How do we detect if someone jams their penis into the CD-ROM tray? Should we define E_PENIS_NOT_EXPECTED and provide an appropriate message file to display helpful error text to the user? Can we DCR the CD-ROM firmware to detect illegal penis insertions?"
In summary...
I'll probably come back to this post and update it with new insights. It'll be fun to see how my views change over the years.
- NIT is short for "nitpick" in a code review. Something that certainly doesn't need to be changed unless you're churning the code again. Basically, I prefer it this way but I don't really care.
Monday, March 16, 2015
Amazon: Why not outsource?
I got this email from a recruiter. Not very many people I know have had a good experience working at Amazon. I wonder if this will get me on their "do-not-email" blacklist...
From: Jon To: XXXXXX@amazon.com Subject: RE: Amazon's XXXXXXXXXXXXXXXXXXXX team is hiring! Date: Mon, 16 Mar 2015 21:40:49 -0700 Hi XXXXXX, [TL;DR: Perhaps Amazon should outsource their engineering work.] Thanks for contacting me. I think it's wise to always be on the lookout for new opportunities. However, I think Amazon has some great challenges with respect to staffing. Here's a few reasons why I won't consider a position there at this time: First and foremost, I LOVE my current job and the company where I work. It would take quite a bit to pull me away. Not just money, but something interesting, challenging, and fun. I'm challenged daily, I get to work on things I genuinely believe help customers daily, and I'm passionate about what I do. Amazon doesn't have paid parental leave. At least it didn't when I asked ~12 months ago. This speaks a tremendous amount about the culture of Amazon. An employee told me most managers are willing to "let you work from home for a few weeks." The fact that paid parental leave isn't an official benefit at Amazon gives me the impression that Amazon doesn't care or acknowledge I may actually have a life outside of work. It's also pretty much standard at the top end of the industry to offer at least 4 weeks... If it's not a company wide policy, then the company wide policy is that it doesn't exist. Policy dictates culture; much the same way it dictates results. I've heard that Amazon pays so well these other benefits aren't needed. I should take my larger salary and apply it to the benefits I want. I feel the culture this policy dictates is one much more suited to an outsourcing contract than a long-term employee/employer relationship. Amazon's attrition is insane. I remember reading an article which reported the average employee tenure at Amazon is 12 months. This doesn't bode well for anyone at the company. For new hires, it indicates they might as well start their next job search the first day they are hired. For veterans it means they will constantly be ramping up new hires and imparting all that tribal knowledge which inevitably grows up around a code base. Once the veterans are lost all that tribal knowledge may very well be lost. Nobody will know about the easiest way to plug in new feature ABC into the existing architecture or fix bug XYZ. So now even the company doesn't make out well in the end because the new hires are grafting features or bug fixes onto a code base with no guidance. I don't envy anyone who must maintain the patchwork disaster of a system that environment eventually fosters. The only way this isn't an issue is if the work you're doing is so thoroughly uninteresting that no real engineering is required and cookie cutter templates apply to all the work. In which case Amazon should outsource. When I asked a manager what the deal was with the attrition I was told that "Amazon hires such good engineers that other companies constantly steal them away." Amazon wants the very best in the industry but the compensation and/or work environment is such that it sends them fleeing in the arms of another? I understand Amazon pays quite well, so it must be the work environment. This statement is either part of an HR script or someone has their head stuck somewhere it doesn't belong -- hopefully in the sand since the alternative is most unpleasant. I ran a quick search of my LinkedIn connections for those having worked at Amazon or currently working there. Ten of my contacts have worked there and only one is still working there. I'll be generous and say that's a 10% success rate. Or perhaps I just have sub-par LinkedIn connections -- they are after all, connected to me... Amazon's leadership. The one and only redeeming thing about Amazon is the focus on customers. Amazon loves their customers. It's clear in the service provided. It's clear in the leadership principals published on the careers page. It's even clear in the story I've heard about those "banged" emails Mr. Bezos will forward when a customer complains. However, if Amazon has a "1 out of 11 found this career acceptable" review. Where's the banged email for that? How vocally self critical is the senior leadership team of their atrocious employee retention? Or is that not really what the senior leadership team cares about? Do they actually want a workforce that churns every 12 months? Do they really care little for the work-life balance? If so, why not outsource the work? Amazon doesn't seem to want long-term employees. If cogs in the wheel are good enough why not just outsource the work? Thanks, -- Jon -------------------------------------------------------------------------- From: XXXXXXX@amazon.com To: Jon Subject: Amazon's XXXXXXXXXXXXXXXXXX team is hiring! Date: Wed, 11 Mar 2015 22:19:18 +0000 Hi Jon- I came across your resume today and was impressed with your experience and your education. I am looking for senior talent such as yourself to staff a new initiative for Amazon’s XXXXXXXXXXXXXX team. The position is located in Seattle. If you want to be where decisions are made and solve problems for customers I think this is a challenge you would really enjoy. Amazon's culture has been a great fit for me and one I think you would like. I would love to talk with you further. Can you suggest a time to connect? XXXX XXXXXX | Technical Sourcer | Amazon E: XXXXXX@amazon.com Phone: (206) XXX-XXXX Work hard. Have fun. Make history.
Wednesday, February 11, 2015
I'd rather...
While chatting with a coworker about C vs. C++ he pointed out I can read a helpful whitepaper to help code C++ in a manner which makes it "sane." My definition of "sane" when applied to a programming language is that no hidden and/or unexpected behavior occurs from a statement. Most of my day-to-day work involves kernel mode coding. In this case I want the most "sane" behavior possible. Unfortunately, now that I'm used to logical things occurring as a result of my coding I even prefer C to C++ for user mode or other utility coding.
For example:
Well the jokes on me because when I finally opened the whitepaper it was 72 pages long.
Also, I'm in pretty good company: http://harmful.cat-v.org/software/c++/
For example:
a++;I'd rather fashion a knife out of a salt block with an open wound and then stab myself repeatedly in the eye until brain matter leaks out than read up on how C++ can finally be made sane if we only follow these 40 simple rules.
Well the jokes on me because when I finally opened the whitepaper it was 72 pages long.
Also, I'm in pretty good company: http://harmful.cat-v.org/software/c++/
Saturday, September 6, 2014
Side projects... what next?
Lately I've been having a hard time deciding which side project to work on at home. Of course, having a seven month old has severely limited how much time I have for side projects, but I find it's still important to work out some "work-suppressed" creativity. A coworker from my IGT days has been wanting to work on a game for a few weeks now. He's been working with Unity and probably wants to work out some of his own work-suppressed creativity. This guy is an awesome 3D artist / modeler. I really enjoyed working with him at IGT. We worked together on the last large project I did at IGT before joining Microsoft and I had a lot of fun. He taught me about skeletal animation and we worked together to extend an existing toolchain to support it. That was some of the most fun I've had in a job.
However, I'm also becoming extremely interested in operations systems. I've taken my "OS" to the point where I really need to start making decisions about early boot environment and handoff to the kernel. Lately, I've been working on a FAT32 format utility specific to my OS. I need a custom format utility so I can add my own boot sector code and lay out the supplemental sectors accordingly. FAT32 is surprisingly simple, but also not all that interesting so the project has stalled here for a bit. Perhaps I'll just skip this and use the same 10MB "chunk" for booting as I have been. The problem is I want to start adding executable image and loader support, along with other things, so I figure I might as well have a real filesystem to work against.
There's a few other smaller projects I've been thinking about as well. The diff utility based on a "rolling hash" function. I had a beta of this working in C# with WPF rendering and it worked well. My next step is to turn it into a C library / DLL which I can use in multiple other utilities I'd like to create. I've also looked at making a code editor a few times. I'll usually get hung up on Unicode support because it's one of those things you really kind of need to plan for up front. While this may not have resulted in much of a code editor, I certainly learned a bunch about Unicode and also thought up the concept of an index tree. I've also thought about starting up work on my fractal terrain generation and perlin noise algorithms again. I was using XNA and there was some limitation on the type of integer calculations I perform efficiently in the shader. I can't remember the details since it was a couple years ago, but I'd definitely like to revisit using newer DirectX or OpenGL shaders which may not have the same limitation I encountered.
Yep... A lot of choices. I also had the beginnings of 3D graphics rendering engine in the works -- it was using Blender as the creation framework and I had python scripts to export into the engine formats. It was coming along. What I'd really like would be to have a job where I could work on one of these things to the point it's useful.
Friday, September 5, 2014
Microtip: Stack corruption? windbg dps to the rescue.
Everyone encounters it sooner or later. A crash with a corrupted callstack. You know what the stack pointer is, but it is clearly not unwinding properly. With a little elbow grease I've found the WinDbg command "dps" to be extremely helpful in this situation.
With this, you're telling the debugger to dump data, in "pointer size" chunks, and also try to match against loaded symbols. You'll quickly start seeing *some* sort of callstack with this display. Now the trick is to sort through the various locations to find the actual callstack. Windbg can help with this as well, you can tell it where the stack actually starts using the kn = [base address] command to verify if what you think is a good stack really is one. The trick is to look at address offsets. Pretty much all real function return addresses will have a non-zero offset. Usually, there are a small enough number of these that you can just brute force it -- like you see me doing (although I skipped a few for brevity). If there are a ton of these potential addresses, it may require additional detective work. Also, one important gotcha -- there may appear to be more than one valid stack, so at this point you need to read the source code or disassembly, and determine what makes the most sense. Also, knowing the ABI of the platform you're working on is invaluable.
This is a little tedious, but the worst bugs only rarely repro and can hold up a product release...
0:000> kn
# Child-SP RetAddr Call Site
00 0000006d`6dcdf310 0000006d`6dd80000 ntdll!RtlEnterCriticalSection+0x22
01 0000006d`6dcdf318 6f6e6d6c`402c0062 0x0000006d`6dd80000
02 0000006d`6dcdf320 00000000`00000470 0x6f6e6d6c`402c0062
03 0000006d`6dcdf328 00000000`00000480 0x470
04 0000006d`6dcdf330 00000000`00000000 0x480
0:000> * Oh noooooooo!!!!
0:000> dps 0000006d`6dcdf310
0000006d`6dcdf310 0000006d`6dd80000
0000006d`6dcdf318 6f6e6d6c`402c0062
0000006d`6dcdf320 00000000`00000470
0000006d`6dcdf328 00000000`00000480
0000006d`6dcdf330 00000000`00000000
0000006d`6dcdf338 0000006d`6dcdf428
0000006d`6dcdf340 77767574`73727170
0000006d`6dcdf348 00007fff`ba9738d8 KERNELBASE!VirtualQuery+0x28
0000006d`6dcdf350 87868584`83828180
0000006d`6dcdf358 8f9e8d9c`8b9a8988
0000006d`6dcdf360 97969594`93929190
0000006d`6dcdf368 ff9e9d9c`9b9a9998
0000006d`6dcdf370 00000000`00000030
0000006d`6dcdf378 0000006d`6dcdf3a8
0000006d`6dcdf380 0000006d`6dcdf440
0000006d`6dcdf388 00007fff`ba983552 KERNELBASE!SetUnhandledExceptionFilter+0x24a
0:000> dps
0000006d`6dcdf390 e7e6e5e4`e3e2e1e0
0000006d`6dcdf398 efeeedec`ebeae9e8
0000006d`6dcdf3a0 d7f6f5f4`f3f2f1f0
0000006d`6dcdf3a8 00000000`00000030
0000006d`6dcdf3b0 00000000`0000021a
0000006d`6dcdf3b8 00000000`00000000
0000006d`6dcdf3c0 00007ff6`d6290000 MicroLogUtil!__ImageBase
0000006d`6dcdf3c8 01000000`00000080
0000006d`6dcdf3d0 00000000`00095000
0000006d`6dcdf3d8 00007fff`bd24dcb7 ntdll!RtlDecodePointer+0x27
0000006d`6dcdf3e0 00000000`00000000
0000006d`6dcdf3e8 00000000`00000000
0000006d`6dcdf3f0 00000000`00000000
0000006d`6dcdf3f8 00000000`00000000
0000006d`6dcdf400 00000000`00000000
0000006d`6dcdf408 00007ff6`d62a5ec0 MicroLogUtil!__CxxUnhandledExceptionFilter
0:000> dps
0000006d`6dcdf410 000012b3`e0966000
0000006d`6dcdf418 00007fff`ba983476 KERNELBASE!SetUnhandledExceptionFilter+0x16e
0000006d`6dcdf420 00000000`00000000
0000006d`6dcdf428 00000000`959f04b3
0000006d`6dcdf430 0000006d`6dd86480
0000006d`6dcdf438 00007ff6`d6295a90 MicroLogUtil!_lock+0x50
0000006d`6dcdf440 00007ff6`d6290000 MicroLogUtil!__ImageBase
0000006d`6dcdf448 00000000`00095000
0000006d`6dcdf450 03810381`01000000
0000006d`6dcdf458 00000000`005a0058
0000006d`6dcdf460 0000006d`6dcdf468
0000006d`6dcdf468 00007ff6`d6299232 MicroLogUtil!_heap_alloc_dbg_impl+0x32
0000006d`6dcdf470 00000000`00000004
0000006d`6dcdf478 00007fff`bd2e1ebc ntdll!RtlZeroHeap+0x6e8
0000006d`6dcdf480 006b0073`00690064
0000006d`6dcdf488 0075006c`006f0056
0:000> dps
0000006d`6dcdf490 005c0032`0065006d
0000006d`6dcdf498 005c0070`006d0074
0000006d`6dcdf4a0 00720063`0069004d
0000006d`6dcdf4a8 0067006f`10070017
0000006d`6dcdf4b0 00000000`6dd86310
0000006d`6dcdf4b8 00007fff`bd2be5e7 ntdll!memset+0x1fa27
0000006d`6dcdf4c0 00000000`00000000
0000006d`6dcdf4c8 00000000`00000001
0000006d`6dcdf4d0 0000006d`6dd80000
0000006d`6dcdf4d8 00007ff6`d6299ab9 MicroLogUtil!_nh_malloc_dbg_impl+0x39
0000006d`6dcdf4e0 00000000`00000030
0000006d`6dcdf4e8 00000000`00000001
0000006d`6dcdf4f0 00000000`00000000
0000006d`6dcdf4f8 00007fff`00000000
0000006d`6dcdf500 0000006d`6dcdf560
0000006d`6dcdf508 00000000`00000000
0:000> dps
0000006d`6dcdf510 00000000`00000000
0000006d`6dcdf518 02100210`02100210
0000006d`6dcdf520 00000000`00000101
0000006d`6dcdf528 00007ff6`d6299a49 MicroLogUtil!_nh_malloc_dbg+0x49
0000006d`6dcdf530 00000000`00000030
0000006d`6dcdf538 00007ff6`00000000
0000006d`6dcdf540 00000000`00000001
0000006d`6dcdf548 00000000`00000000
0000006d`6dcdf550 00000000`00000000
0000006d`6dcdf558 0000006d`6dcdf560
0000006d`6dcdf560 00000000`00000000
0000006d`6dcdf568 00000000`00000000
0000006d`6dcdf570 00000000`00000000
0000006d`6dcdf578 00007ff6`d6294b7a MicroLogUtil!malloc+0x2a
0000006d`6dcdf580 00000000`00000030
0000006d`6dcdf588 0000006d`00000000
0:000> dps
0000006d`6dcdf590 0000006d`00000001
0000006d`6dcdf598 00000000`00000000
0000006d`6dcdf5a0 00000000`00000000
0000006d`6dcdf5a8 00007ff6`d6295db2 MicroLogUtil!_unlock+0x22
0000006d`6dcdf5b0 00000000`00000000
0000006d`6dcdf5b8 02100302`00000030
0000006d`6dcdf5c0 00007ff6`00000000
0000006d`6dcdf5c8 00007ff6`d62915e6 MicroLogUtil!AstNodeCreate+0x26
0000006d`6dcdf5d0 00000000`00000030
0000006d`6dcdf5d8 00007ff6`d62999b4 MicroLogUtil!_msize_dbg+0x234
0000006d`6dcdf5e0 0000006d`00000004
0000006d`6dcdf5e8 00007ff6`d62aa677 MicroLogUtil!_setmbcp_nolock+0x447
0000006d`6dcdf5f0 00000000`00000000
0000006d`6dcdf5f8 00007fff`bd265a63 ntdll!RtlEncodePointer+0x27
0000006d`6dcdf600 00000000`00000100
0000006d`6dcdf608 00007ff6`d62919e8 MicroLogUtil!AstStateInit+0x48
0:000> dps
0000006d`6dcdf610 00000000`00000000
0000006d`6dcdf618 00007ff6`d631bec0 MicroLogUtil!pairNode `RTTI Type Descriptor'+0x5a0
0000006d`6dcdf620 00000000`00000001
0000006d`6dcdf628 0000006d`6dcdf680
0000006d`6dcdf630 00000000`00000000
0000006d`6dcdf638 00007ff6`d6291115 MicroLogUtil!ProcessFile+0x85
0000006d`6dcdf640 0000006d`6dcdf6c0
0000006d`6dcdf648 0000006d`6dd827c8
0000006d`6dcdf650 00000000`00000000
0000006d`6dcdf658 00007fff`bd265a63 ntdll!RtlEncodePointer+0x27
0000006d`6dcdf660 00000000`00000002
0000006d`6dcdf668 00007ff6`d6307020 MicroLogUtil!`string'
0000006d`6dcdf670 0000006d`00000000
0000006d`6dcdf678 00000000`00000000
0000006d`6dcdf680 00000000`00000000
0000006d`6dcdf688 00007ff6`d62a5414 MicroLogUtil!__crtSetUnhandledExceptionFilter+0x14
0:000> dps
0000006d`6dcdf690 00000000`00000000
0000006d`6dcdf698 00000000`00000000
0000006d`6dcdf6a0 00000000`00000000
0000006d`6dcdf6a8 00000000`00000000
0000006d`6dcdf6b0 00007ff6`d62a7810 MicroLogUtil!_RTC_Terminate
0000006d`6dcdf6b8 00007ff6`d62a5f50 MicroLogUtil!__CxxSetUnhandledExceptionFilter+0x10
0000006d`6dcdf6c0 00000000`00000000
0000006d`6dcdf6c8 00000000`00000000
0000006d`6dcdf6d0 00000000`00000000
0000006d`6dcdf6d8 00000000`00000000
0000006d`6dcdf6e0 00000000`00000000
0000006d`6dcdf6e8 00000000`00000000
0000006d`6dcdf6f0 00000000`00000000
0000006d`6dcdf6f8 00000000`00000000
0000006d`6dcdf700 00000000`00000000
0000006d`6dcdf708 00000000`00000000
0:000> * Hmmm, that looks a little more promising...
0:000> * Looking for offsets that make sense... is it the VirtualQuery+0x28???
0:000> kn = 0000006d`6dcdf348
# Child-SP RetAddr Call Site
00 0000006d`6dcdf348 00007fff`ba9738d8 ntdll!RtlEnterCriticalSection+0x22
01 0000006d`6dcdf350 00007fff`ba983552 KERNELBASE!VirtualQuery+0x28
02 0000006d`6dcdf390 00007fff`ba983476 KERNELBASE!SetUnhandledExceptionFilter+0x24a
03 0000006d`6dcdf420 00007ff6`d62a5414 KERNELBASE!SetUnhandledExceptionFilter+0x16e
04 0000006d`6dcdf690 00007ff6`d62a5f50 MicroLogUtil!__crtSetUnhandledExceptionFilter+0x14
05 0000006d`6dcdf6c0 00000000`00000000 MicroLogUtil!__CxxSetUnhandledExceptionFilter+0x10
0:000> * Hmm, doesn't look like it. How about KERNELBASE!SetUnhandledExceptionFilter+0x24a?
0:000> kn = 0000006d`6dcdf388
# Child-SP RetAddr Call Site
00 0000006d`6dcdf388 00007fff`ba983552 ntdll!RtlEnterCriticalSection+0x22
01 0000006d`6dcdf390 00007fff`ba983476 KERNELBASE!SetUnhandledExceptionFilter+0x24a
02 0000006d`6dcdf420 00007ff6`d62a5414 KERNELBASE!SetUnhandledExceptionFilter+0x16e
03 0000006d`6dcdf690 00007ff6`d62a5f50 MicroLogUtil!__crtSetUnhandledExceptionFilter+0x14
04 0000006d`6dcdf6c0 00000000`00000000 MicroLogUtil!__CxxSetUnhandledExceptionFilter+0x10
0:000> * Nope. Okay... skipping ahead to the correct one: MicroLogUtil!_lock+0x50
0:000> kn = 0000006d`6dcdf438
# Child-SP RetAddr Call Site
00 0000006d`6dcdf438 00007ff6`d6295a90 ntdll!RtlEnterCriticalSection+0x22
01 0000006d`6dcdf440 00007ff6`d6299232 MicroLogUtil!_lock+0x50
02 0000006d`6dcdf470 00007ff6`d6299ab9 MicroLogUtil!_heap_alloc_dbg_impl+0x32
03 0000006d`6dcdf4e0 00007ff6`d6299a49 MicroLogUtil!_nh_malloc_dbg_impl+0x39
04 0000006d`6dcdf530 00007ff6`d6294b7a MicroLogUtil!_nh_malloc_dbg+0x49
05 0000006d`6dcdf580 00007ff6`d62915e6 MicroLogUtil!malloc+0x2a
06 0000006d`6dcdf5d0 00007ff6`d62919e8 MicroLogUtil!AstNodeCreate+0x26
07 0000006d`6dcdf610 00007ff6`d6291115 MicroLogUtil!AstStateInit+0x48
08 0000006d`6dcdf640 00007ff6`d6291473 MicroLogUtil!ProcessFile+0x85
09 0000006d`6dcdf730 00007ff6`d629549c MicroLogUtil!wmain+0x33
0a 0000006d`6dcdf770 00007ff6`d62955de MicroLogUtil!__tmainCRTStartup+0xec
0b 0000006d`6dcdf7c0 00007fff`bcc916ad MicroLogUtil!wmainCRTStartup+0xe
0c 0000006d`6dcdf7f0 00007fff`bd2734a5 KERNEL32!BaseThreadInitThunk+0xd
0d 0000006d`6dcdf820 00000000`00000000 ntdll!RtlUserThreadStart+0x1d
0:000> * Hey yo! That's looking like a correct stack!
Friday, August 29, 2014
What makes a good job?
Lately I've been reflecting on previous positions I've held and thought about the good and bad of each. It's probably a good idea to think about what job traits lead to satisfaction or dissatisfaction. There seem to be a few common properties which effect me most.
Good teammates turn into family friends and lifelong contacts. I recently had lunch with a guy I worked with over 3 years ago. It was the highlight of my week and I'd love to work with him again. If you want to know if you're in a cohesive team you only need to count the number of times team members have done things together away from work.
In my opinion, even worse than the employee who enters management because company culture doesn't permit IC growth is the employee who enters management for the prestige of it. These "prestige managers" seem to think being a boss is a more important role than being an IC. The reason this is the worst possible situation is these people don't understand a fundamental rule of leading: The leader works for the team.
Someone looking for prestige in a leadership role is under the impression the team works for the leader. Paradoxically, the only time the team truly will work for the leader is when the leader has won the respect and admiration of the team members. This does not happen through silly mandates, make-work meetings and tasks, managing appearances rather than results, or overbearing "guidance." It does not come from business buzzwords or MBA degrees. It comes from facilitating the team members. It comes from getting out of the way when you're a hindrance, proactively removing obstacles, and internalizing team member requirements. A bad leader will say, "I hope you're empowered to fix this." A good leader will not need to talk about empowerment because their ICs already are.
At one point I naively thought these misguided managers simply never had the benefit of an excellent manager mentor. I don't believe that's the case anymore. I've personally observed multiple examples of ICs with solid examples of good leadership turn into some of the worst prestige managers. They are incapable of observing what makes the solid leaders excellent. Since they are blind to these traits they are also incapable of applying the principles in their team. It's easy to spot these managers. Their teams bleed talent and whither. Team members who stick around are demoralized, have low job satisfaction, and through apathy eventually give up trying to make any positive change. A bad leader can poison a team or organization the way cancer moves through the body.
If a team, organization, or company has poor planning the quality of the final product will eventually suffer. Quality will suffer because bugs are not caught (or are caught too late to fix and reset testing to make a release date). Quality will also eventually suffer because the best employees will eventually leave. Common "fire drills" point to a process or cultural issue which the team, organization, or company cannot or is not motivated enough to change.
If a team, organization, or company has a staffing issue one must ask why. Do they suffer from high attrition? Do they not compensate properly? Do they expect too much of ICs? Can they not find qualified candidates? All of these are related to the simple fact that it's not a good place to work. Most of the issues behind the questions above are obvious, but what about the lack of qualified candidates? I believe there are quite a few extremely smart people out there looking for something new. If you can't entice quality candidates how do you expect to keep current talent?
Teammates
I've pretty much always stayed in a team well after the expiration date because of great team members. I often find myself missing peers more than any other positive aspects of a job. Because of this I feel my peers are an extremely important factor in job satisfaction.Good teammates turn into family friends and lifelong contacts. I recently had lunch with a guy I worked with over 3 years ago. It was the highlight of my week and I'd love to work with him again. If you want to know if you're in a cohesive team you only need to count the number of times team members have done things together away from work.
Technology
If I'm honest with myself I think I have to admit something: What I'm working on seems to be much more of a deciding factor when choosing the next job than contributing to actual happiness once there. I get caught up looking for the next shiny thing. However, there is some practicality in allowing technology to influence my next role. Usually, I try to choose a technology area that I believe will be most useful for my long term career. This is when I try to make sure I don't get stuck or pigeon holed into one area. I remember an engineer I worked with who was getting progressively bitter about always working on the same thing. He hated what he was being asked to work on but didn't seem to want to change positions. I suppose this is the most important reason to consider technology when choosing a job. You don't want to whither away as an expert of a dead-end technology.Manager
My manager is probably the single most important influence on job satisfaction. Unfortunately, companies (and their managers) don't seem to understand this fact very often. I've worked in company cultures where the only way to grow was to enter management. This is horrible. It takes a company's most productive employees and mashes them into a completely different area of expertise -- one in which they most likely are not qualified. One great thing about Microsoft is the dual path they support. You can be a "people manager" or you can be an "individual contributor" (IC) and obtain the same compensation.In my opinion, even worse than the employee who enters management because company culture doesn't permit IC growth is the employee who enters management for the prestige of it. These "prestige managers" seem to think being a boss is a more important role than being an IC. The reason this is the worst possible situation is these people don't understand a fundamental rule of leading: The leader works for the team.
Someone looking for prestige in a leadership role is under the impression the team works for the leader. Paradoxically, the only time the team truly will work for the leader is when the leader has won the respect and admiration of the team members. This does not happen through silly mandates, make-work meetings and tasks, managing appearances rather than results, or overbearing "guidance." It does not come from business buzzwords or MBA degrees. It comes from facilitating the team members. It comes from getting out of the way when you're a hindrance, proactively removing obstacles, and internalizing team member requirements. A bad leader will say, "I hope you're empowered to fix this." A good leader will not need to talk about empowerment because their ICs already are.
At one point I naively thought these misguided managers simply never had the benefit of an excellent manager mentor. I don't believe that's the case anymore. I've personally observed multiple examples of ICs with solid examples of good leadership turn into some of the worst prestige managers. They are incapable of observing what makes the solid leaders excellent. Since they are blind to these traits they are also incapable of applying the principles in their team. It's easy to spot these managers. Their teams bleed talent and whither. Team members who stick around are demoralized, have low job satisfaction, and through apathy eventually give up trying to make any positive change. A bad leader can poison a team or organization the way cancer moves through the body.
Partner teams
Much like the culture of a team, the culture of partner teams is extremely important. For the same reasons a bad manager introduces a cancer to the team partner teams do the same. Partner teams are by definition peers or customers. Nobody wants to work with a jerk and nobody wants a jerk for a customer. Talented ICs will eventually leave a team forced to deal with horrible partner teams.Work-Life balance
"I don't live to work, I work to live." I believe a balanced personal life creates many more motivated employees and directly impacts the success of a company, organization, or team. For me, when work cuts into my personal time I end up despising work. More importantly, when work-life balance shifts too far toward work it's an indicator the team, organization, or company will eventually fail. ICs are asked to work too much because of scheduling / planning issues or staffing issues.If a team, organization, or company has poor planning the quality of the final product will eventually suffer. Quality will suffer because bugs are not caught (or are caught too late to fix and reset testing to make a release date). Quality will also eventually suffer because the best employees will eventually leave. Common "fire drills" point to a process or cultural issue which the team, organization, or company cannot or is not motivated enough to change.
If a team, organization, or company has a staffing issue one must ask why. Do they suffer from high attrition? Do they not compensate properly? Do they expect too much of ICs? Can they not find qualified candidates? All of these are related to the simple fact that it's not a good place to work. Most of the issues behind the questions above are obvious, but what about the lack of qualified candidates? I believe there are quite a few extremely smart people out there looking for something new. If you can't entice quality candidates how do you expect to keep current talent?
Impact
I want to change the world. I want to work on something wonderful that makes the world a better place. I believe life is too short to not try to make the world better while we are here. There are many ways to make the world better. Entertainment, creating tools for others, basically anything that helps someone else. Someone also pointed out to me that developers typically want to work on things that ship. I fully agree -- it's pretty hard to have much impact on the world if your project doesn't ship.Conclusion
I started writing this to help me decide what I value the most should I look for a change. It appears leadership is a top influence for me at this time. However, I think leadership is perhaps a proxy for general team, organization, or company culture. The leaders will dictate the culture of a workplace.Tuesday, July 8, 2014
DarkOS: Hobbyist OS architecture taking shape.
I've been tinkering with some things in my "hobbyist OS" for a bit now. There was a bit of hiatus when my son was born but he's getting into a sleeping schedule more conducive to projects.
First, I've been messing around with a few emulators. VirtualBox seems to be the best for my work. A few key bits of functionality make it favorable:
Bochs is still pretty useful, but it started throwing odd crashes during protected-to-real transitions (more on this in a bit).
http://wiki.osdev.org/ has been very helpful in all my development.
I've got a plan of attack for the OS now:
Key components and TODO list:
Once I get the basic kernel booting I hope to crib quite a few module tidbits from a free *nix distribution like FreeBSD or NetBSD. I'll have to evaluate the licenses I'm choosing for my project and the ones provided by FreeBSD and/or NetBSD.
Some modules will be pretty straight forward. For example, the VESA module will mainly just be a set of helper functions for drawing and outputting text to the linear frame buffer. Other modules like EHCI/XHCI will be much more in-depth and probably won't every be tackled.
Ideally I'd like to be able to boot from USB thumb drive, access the network card, and if the POSIX environment works out, perhaps get enough libraries and components running in user mode for a web browser and self hosting a build of the OS.
Well, that's the plan anyway. I hope to make gradual progress with a few planned breaks in the next few months (moving houses... maybe a vacation...).
First, I've been messing around with a few emulators. VirtualBox seems to be the best for my work. A few key bits of functionality make it favorable:
- It doesn't require some of the more advanced CPU virtualization extensions needed by HyperV.
- It does run using some virtualization, so it's much faster than something like Bochs.
- It has a "debugger" mode with windbg style commands to dump registers, gdt, etc.
Bochs is still pretty useful, but it started throwing odd crashes during protected-to-real transitions (more on this in a bit).
http://wiki.osdev.org/ has been very helpful in all my development.
I've got a plan of attack for the OS now:
- Using long mode (amd64).
- Using ELF x86-amd64 Linux ABI. Cross compiler running in Cygwin has been built.
- No real/protected mode ping-ponging.
- The real/protected transitions were proving more annoying (and variable) between Bochs and VirtualBox. I'd get one working and then the other would fail. Since this whole idea of switching back and forth is full of issues I'm just not doing it.
- This requires slightly odd bootstrapping of components (loading kernel, drivers from disk but not processing the ELF format at first).
- Doing this, I hope to spend more time focused on the OS and less on the quirkiness of various BIOS implementations.
- POSIX user mode environment with a Windows NT -style kernel.
Key components and TODO list:
- Bootloader
- Enable A20 support.
- Get RAM configuration from BIOS.
- Read kernel boot options from disk or perhaps keyboard.
- Load kernel stub, kernel, and all modules indicated as required for boot. (The kernel and modules are streamed into higher address RAM directly from disk, no ELF image loading done yet).
- Switch video to VESA linear frame buffer mode (preferably 1024x768 24/32bit color.
- Fill out a table of boot and machine parameters for use by the kernel stub.
- Setup basic identity paging for long mode (amd64).
- Enter long mode and jump to the kernel stub.
- Kernel Stub
- Load kernel from raw file images the Bootloader copied from disk to full on ELF relocated modules.
- Convert the Bootloader "boot and machine parameters" to a table suitable for calling KeMain(...) with.
- Calling KeMain(...) to initiate kernel startup.
- Kernel
- Initialize a very basic set of kernel debugger stubs accessed through a serial port.
- Initialize the memory manager based on Kernel Stub provided parameters (used memory, etc.)
- Initialize the object manager.
- Create a "system" process (virtual address space, etc.)
- Call the loader functions in the kernel for each of the Bootloader provided boot critical modules.
- Call ModuleInit(...) for each boot critical modules.
- Call ModuleEnter(...) for each of the boot critical modules to initiate PCI/PnP/Hardware enumeration.
- Using file system and storage modules loaded as part of the boot critical modules group, load and start additional modules required by the system.
- Create a user mode process ("init") to bootstrap the rest of the system from user mode startup scripts.
Once I get the basic kernel booting I hope to crib quite a few module tidbits from a free *nix distribution like FreeBSD or NetBSD. I'll have to evaluate the licenses I'm choosing for my project and the ones provided by FreeBSD and/or NetBSD.
Some modules will be pretty straight forward. For example, the VESA module will mainly just be a set of helper functions for drawing and outputting text to the linear frame buffer. Other modules like EHCI/XHCI will be much more in-depth and probably won't every be tackled.
Ideally I'd like to be able to boot from USB thumb drive, access the network card, and if the POSIX environment works out, perhaps get enough libraries and components running in user mode for a web browser and self hosting a build of the OS.
Well, that's the plan anyway. I hope to make gradual progress with a few planned breaks in the next few months (moving houses... maybe a vacation...).
Thursday, March 6, 2014
Why is my default timezone Monrovia, Reykjavik?
I recently sent a friend an email from work and he pointed out my email timezone is set to Monrovia, Reykjavik and that I should refer to http://support.microsoft.com/kb/2298834.
Here is my response...
Here is my response...
Oh, I gave up running Outlook and I'm accessing email via the Office365 interface.
Outlook had (has) some nasty bug where it takes a UI lock in a thread and then posts blocking web calls from the same thread. When the blocking calls actually take a bit of time (because the company isn't good at running data centers?) my dev machine would lose the ability to CHANGE WINDOW FOCUS or otherwise process windows messages.
I tried debugging this issue and after attaching windbg to Outlook.exe and discovering ~150 CLR threads I decided this effort was hopeless. I'm still unclear why they use the UI thread to post blocking web calls in an environment where they already have 149 OTHER threads they could use for this. Thus, I decided to use the https://outlook.com/owa/XXXX.com site instead for corporate email.
HOWEVER, more comedy ensued since MSFT apparently isn't very good with cookie management and continually tried to use my XXXX.X.XXXX@live.com tokens to access my corporate email -- the concept that I'd want to be logged in as XXXX.X.XXXX@live.com and NNNN@XXXX.com at the SAME TIME to different Microsoft website properties is completely alien.
Thus I've settled on my current solution, a shortcut: "iexplore.exe -private https://outlook.com/owa/XXXX.com"
This works very well since it ignores any persistent cookies on the system via InPrivate mode. However, I guess persistent settings like my actual timezone cannot be accounted for and I'm not even prompted for them (some UX designer probably focus grouped this and convinced themselves it was the optimal solution to just use the null timezone setting).
I thought about using Chrome exclusively for my Outlook.com needs but alas, Google seems to think the key to a successful browser is launching FIVE Chrome.exe processes utilizing ~75MB OF MEMORY to provide access to their useless "Hang Outs" feature. Which I hear now supports (THANKFULLY, the world was empty without this feature) Mustaches and Hats so I guess 75MB of RAM is a small price to pay for MUSTACHES AND HATS in a failing social media landing point that replaced the highly effective and useful googletalk.
Gone are the days of a trim gtalk.exe client and Outlook.exe (not .com) just working (for various levels of "working" exceedingly greater than the current implementation). I realize I'm becoming a crusty fossil in the industry; (already?!) clinging to concepts like trim code, easy to understand and debug projects, and my generate-the-object-code-I-intend C coding methodology...
Outlook had (has) some nasty bug where it takes a UI lock in a thread and then posts blocking web calls from the same thread. When the blocking calls actually take a bit of time (because the company isn't good at running data centers?) my dev machine would lose the ability to CHANGE WINDOW FOCUS or otherwise process windows messages.
I tried debugging this issue and after attaching windbg to Outlook.exe and discovering ~150 CLR threads I decided this effort was hopeless. I'm still unclear why they use the UI thread to post blocking web calls in an environment where they already have 149 OTHER threads they could use for this. Thus, I decided to use the https://outlook.com/owa/XXXX.com site instead for corporate email.
HOWEVER, more comedy ensued since MSFT apparently isn't very good with cookie management and continually tried to use my XXXX.X.XXXX@live.com tokens to access my corporate email -- the concept that I'd want to be logged in as XXXX.X.XXXX@live.com and NNNN@XXXX.com at the SAME TIME to different Microsoft website properties is completely alien.
Thus I've settled on my current solution, a shortcut: "iexplore.exe -private https://outlook.com/owa/XXXX.com"
This works very well since it ignores any persistent cookies on the system via InPrivate mode. However, I guess persistent settings like my actual timezone cannot be accounted for and I'm not even prompted for them (some UX designer probably focus grouped this and convinced themselves it was the optimal solution to just use the null timezone setting).
I thought about using Chrome exclusively for my Outlook.com needs but alas, Google seems to think the key to a successful browser is launching FIVE Chrome.exe processes utilizing ~75MB OF MEMORY to provide access to their useless "Hang Outs" feature. Which I hear now supports (THANKFULLY, the world was empty without this feature) Mustaches and Hats so I guess 75MB of RAM is a small price to pay for MUSTACHES AND HATS in a failing social media landing point that replaced the highly effective and useful googletalk.
Gone are the days of a trim gtalk.exe client and Outlook.exe (not .com) just working (for various levels of "working" exceedingly greater than the current implementation). I realize I'm becoming a crusty fossil in the industry; (already?!) clinging to concepts like trim code, easy to understand and debug projects, and my generate-the-object-code-I-intend C coding methodology...
Wednesday, December 11, 2013
Dispatch Objects
It seems like the next step in my OS development is going to be laying down some kernel work. I thought I'd be able to delay this, but doing so would only result in less useful work being done. A kernel is responsible for many things. I'm sure many more things than I even know about right now (such is the nature of this whole undertaking). However, my experience writing Windows drivers has offered a decent skeleton to start with.
One piece of this skeleton is dispatch objects. Basically, this is any sort of thing that can be waited upon. A timer, an event, a lock, a thread (joining a thread, etc.). I figure this is as decent a place to start as any. The way a dispatch object will work in my kernel is it will be a light weight structure containing a list of waiting threads. Of course, this means I need to start defining a thread structure as well. I imagine I'll continue to flesh out the guts of the kernel (or at least their structures) by jumping off on this one spot.
Since a dispatch object is essentially a list, the first structure I need to create is a list structure. There are many ways to create a list. You can create a linked list, a vector, or various other structures. Again, leaning on my Windows background I'll choose something very similar to the LIST_ENTRY structure in the NT kernel. In more data structure books I've seen a doubly-linked list defined as:
Here, I'll rely on something clever that Windows does with it's list structure -- there's no "data" pointer. The structure is intended to be embedded directly into any other data structure:
The way this works is the list pointers are offset from to find the original object the list points to. In the
There are multiple reasons this is a good thing. First, the previously mentioned avoidance of small memory allocations. Second, by not relying on any sort of memory allocation this data structure can be used in the kernel before memory allocation is even possible -- obviously, you need to write an allocator before you can use one. Third (but related to the first), the absence of smaller allocations for what is a widely used construct can add up to some pretty substantial memory savings. Any memory allocation will require a chunk of memory for bookkeeping. This is usually going to be at least the size of a few pointers. For small allocations of only 3 pointers (in the first list structure definition) the overhead of the allocation may very well be larger than the actual data usage. Considering how widely used the
So, there it is, the beginnings of a DISPATCH_HEADER, a THREAD_OBJECT, and a LIST_NODE. This kernel is practically writing itself...
One piece of this skeleton is dispatch objects. Basically, this is any sort of thing that can be waited upon. A timer, an event, a lock, a thread (joining a thread, etc.). I figure this is as decent a place to start as any. The way a dispatch object will work in my kernel is it will be a light weight structure containing a list of waiting threads. Of course, this means I need to start defining a thread structure as well. I imagine I'll continue to flesh out the guts of the kernel (or at least their structures) by jumping off on this one spot.
Since a dispatch object is essentially a list, the first structure I need to create is a list structure. There are many ways to create a list. You can create a linked list, a vector, or various other structures. Again, leaning on my Windows background I'll choose something very similar to the LIST_ENTRY structure in the NT kernel. In more data structure books I've seen a doubly-linked list defined as:
typedef struct _LIST_NODE
{
struct _LIST_NODE* pPrev;
struct _LIST_NODE* pNext;
void* pData;
} LIST_NODE;
Here, I'll rely on something clever that Windows does with it's list structure -- there's no "data" pointer. The structure is intended to be embedded directly into any other data structure:
typedef struct _LIST_NODE
{
struct _LIST_NODE* pPrev;
struct _LIST_NODE* pNext;
} LIST_NODE;
typedef struct
{
LIST_NODE ThreadListHead;
...
} DISPATCH_HEADER;
typedef struct
{
UINT32 ThreadId;
...
LIST_NODE DispatchListNode;
...
} THREAD_OBJECT;
The way this works is the list pointers are offset from to find the original object the list points to. In the
DISPATH_HEADER above, this is simply the same address. However, say I had additional members before the ThreadListHead member, you'd then SUBTRACT back to get the original object pointer. This is done using a member offset macro, where the offset of a member into a structure is computed and then subtracted off the list pointer. While this may appear clunky at first it has one very nice advantage - you don't need to dynamically allocate small chunks of memory. In the example above, you would simply have a list insertion function that you'd pass a pointer to the THREAD_OBJECT::DispatchListNode member variable to.There are multiple reasons this is a good thing. First, the previously mentioned avoidance of small memory allocations. Second, by not relying on any sort of memory allocation this data structure can be used in the kernel before memory allocation is even possible -- obviously, you need to write an allocator before you can use one. Third (but related to the first), the absence of smaller allocations for what is a widely used construct can add up to some pretty substantial memory savings. Any memory allocation will require a chunk of memory for bookkeeping. This is usually going to be at least the size of a few pointers. For small allocations of only 3 pointers (in the first list structure definition) the overhead of the allocation may very well be larger than the actual data usage. Considering how widely used the
LIST_NODE structure will be, this is a substantial savings in memory as well as access speed (there isn't an indirect memory access to get to the actual structure, if you have the pointer to the list, it's only a subtraction to get the actual pointer versus another read).So, there it is, the beginnings of a DISPATCH_HEADER, a THREAD_OBJECT, and a LIST_NODE. This kernel is practically writing itself...
Friday, November 29, 2013
Rolling Your Own Operating System
I started an extremely ambitious project a while back -- working on my own operating system. This isn't one that I ever intend to gain any ground or even boot on real hardware. (I'm targeting VirtualBox, VirtualPC/HyperV, and Bochs emulator environments right now.) It's just a place for me to play around with OS concepts. At work I write driver code for windows, but this code relies on a system of interfaces and mechanisms that have been designed and implemented by people much smarter than I. I'd like to start working on those more basic concepts. Sort of like peeling back the layers of an onion by recreating it. I anticipate I will learn much not only about how my OS works and can work, but about the tradeoffs that most OS designers have encountered along the way.
My current "OS" is very limited. It uses the standard DOS (well, Windows 7 -- because there are differences...) bootloader from the partition table. I created my own volume boot record to receive control from there. I'm able to load from the beginning of the partition. I'm still working on the loader. This is not be choice, but actually some unexpected first bit of learning. Here's the deal...
The BIOS loads and calls into the MBR code. The MBR code loads and calls into my volume boot record code. Now the fun starts. There are numerous extremely important interfaces the BIOS provides to early boot code. Before protected mode, paging, multi-tasking, and all the other things that really make an operating system an operating system you could just happily keep calling these BIOS interfaces to get this functionality. However, the BIOS code is legacy... It's so legacy that it isn't all that compatible with the way a modern OS sets up the CPU. (There are a few more compatible extensions -- VESA graphics BIOS stuff for example, but even this is severely limited)
So, it's already decision time. A lot sooner than I thought it would be. How do I start laying the ground work for the more modern features of my OS when I need to use the BIOS routines to talk to the hardware? Well, the short answer is I need to stop at some point, and probably the sooner the better. I don't want to litter my kernel code with a bunch of BIOS dependencies -- anything I write that requires this cannot really be reused anywhere else. Ideally, I would switch over from BIOS interfaces to my own all at once, however that will be difficult. Instead, I've decided a better approach is to rely on the BIOS interfaces until I'm ready to create my own. So, what does the BIOS give me that I need to reproduce?
I'm sure there are many more but these are the interfaces I'm immediately aware of. The first item, VGA support was easiest. This is probably because it's also the only one that's moved past legacy support. I am able to select a video mode which allows me to draw 24/32 bit RGB directly into a flat buffer space. This buffer space is directly mapped to the screen buffer, so I'm set as far as rudimentary drawing is concerned. I don't get any hardware acceleration and I'm stuck having to implement all my own line drawing, bit-blitting, and other APIs, but at least it's doable. The most annoying thing about this is I cannot adjust for changing the monitor resolution, but if I look at my above goals of not really caring if this runs on real hardware this is acceptable.
Now comes disk IO. This is a bit sticky, since there is no easy way to go about this. I'm probably going to have to support at least IDE and AHCI. I'm not exactly sure if I can get away with IDE only and rely on hard disks to support the legacy commands... I think so, but not sure. I also cannot keep thunking back and forth between BIOS code for this support -- every time I do that I'll probably have to tear down the things I really wanted to learn about -- memory paging, scheduling, etc. I see some work on a storage driver in my near future... It does indeed look like I can just go with IDE. Hopefully, IDE will be pretty easy to work with. It looks like the specs are open as well -- http://www.t10.org/t13/technical/d98120r0.pdf
Here's what my volume boot record code looks like so far: http://www.darkautomata.com/blog/os/vbr_2013_11_29.asm.txt
1 I can just use a COM port interface for logging what is actually happening in the OS, but there is going to come a time relatively early in the OS where I'll actually want to SEE what I've created. Maybe this isn't as big of a requirement as I think it is right now...
My current "OS" is very limited. It uses the standard DOS (well, Windows 7 -- because there are differences...) bootloader from the partition table. I created my own volume boot record to receive control from there. I'm able to load from the beginning of the partition. I'm still working on the loader. This is not be choice, but actually some unexpected first bit of learning. Here's the deal...
The BIOS loads and calls into the MBR code. The MBR code loads and calls into my volume boot record code. Now the fun starts. There are numerous extremely important interfaces the BIOS provides to early boot code. Before protected mode, paging, multi-tasking, and all the other things that really make an operating system an operating system you could just happily keep calling these BIOS interfaces to get this functionality. However, the BIOS code is legacy... It's so legacy that it isn't all that compatible with the way a modern OS sets up the CPU. (There are a few more compatible extensions -- VESA graphics BIOS stuff for example, but even this is severely limited)
So, it's already decision time. A lot sooner than I thought it would be. How do I start laying the ground work for the more modern features of my OS when I need to use the BIOS routines to talk to the hardware? Well, the short answer is I need to stop at some point, and probably the sooner the better. I don't want to litter my kernel code with a bunch of BIOS dependencies -- anything I write that requires this cannot really be reused anywhere else. Ideally, I would switch over from BIOS interfaces to my own all at once, however that will be difficult. Instead, I've decided a better approach is to rely on the BIOS interfaces until I'm ready to create my own. So, what does the BIOS give me that I need to reproduce?
- VGA/VESA support -- in order to know my OS is doing something, I should probably have a way to see it working. 1
- Keyboard support.
- Disk IO support.
- ...
I'm sure there are many more but these are the interfaces I'm immediately aware of. The first item, VGA support was easiest. This is probably because it's also the only one that's moved past legacy support. I am able to select a video mode which allows me to draw 24/32 bit RGB directly into a flat buffer space. This buffer space is directly mapped to the screen buffer, so I'm set as far as rudimentary drawing is concerned. I don't get any hardware acceleration and I'm stuck having to implement all my own line drawing, bit-blitting, and other APIs, but at least it's doable. The most annoying thing about this is I cannot adjust for changing the monitor resolution, but if I look at my above goals of not really caring if this runs on real hardware this is acceptable.
Now comes disk IO. This is a bit sticky, since there is no easy way to go about this. I'm probably going to have to support at least IDE and AHCI. I'm not exactly sure if I can get away with IDE only and rely on hard disks to support the legacy commands... I think so, but not sure. I also cannot keep thunking back and forth between BIOS code for this support -- every time I do that I'll probably have to tear down the things I really wanted to learn about -- memory paging, scheduling, etc. I see some work on a storage driver in my near future... It does indeed look like I can just go with IDE. Hopefully, IDE will be pretty easy to work with. It looks like the specs are open as well -- http://www.t10.org/t13/technical/d98120r0.pdf
Here's what my volume boot record code looks like so far: http://www.darkautomata.com/blog/os/vbr_2013_11_29.asm.txt
1 I can just use a COM port interface for logging what is actually happening in the OS, but there is going to come a time relatively early in the OS where I'll actually want to SEE what I've created. Maybe this isn't as big of a requirement as I think it is right now...
Monday, July 23, 2012
Screen DPI vs Remoting issues
It seems to me the user experience for high DPI displays like the Apple Retina display will be horrible for certain scenarios.
If I'm remoting into a windows machine using Remote Desktop or displaying an X application using X11 remoting then the host DPI settings are probably going to result in a bad experience. I already see this in when using my work laptop to remote into my desktop.
At work I have multiple 24" 1920x1080 monitors. When I RDP into the machine from my 14" 1920x1080 work laptop I get text in the code editor that is barely legible. I can't imagine how bad this would be on the Retina display!
The Remote Desktop and other remote display clients should enable a pass-through mechanism for client DPI. This way the host can probably format the display for remoting.
To expand on this idea, if all UI settings relating to size were stored in a device independent way (for example: twips, 1/1440 of an inch) then a translation could take place to always match the physical dimensions of the UI. You could even have a constant scaling factor so you can change this for a device. For example, I may not want my cell phone displaying at full physical dimensions, maybe it's okay to go with 50% or 25% and I know I have to squint at it, but I want my laptop or tablet to use 80-90% physical dimensions, etc...
If no one has thought of this yet well it's here first... prior art recorded.
If I'm remoting into a windows machine using Remote Desktop or displaying an X application using X11 remoting then the host DPI settings are probably going to result in a bad experience. I already see this in when using my work laptop to remote into my desktop.
At work I have multiple 24" 1920x1080 monitors. When I RDP into the machine from my 14" 1920x1080 work laptop I get text in the code editor that is barely legible. I can't imagine how bad this would be on the Retina display!
The Remote Desktop and other remote display clients should enable a pass-through mechanism for client DPI. This way the host can probably format the display for remoting.
To expand on this idea, if all UI settings relating to size were stored in a device independent way (for example: twips, 1/1440 of an inch) then a translation could take place to always match the physical dimensions of the UI. You could even have a constant scaling factor so you can change this for a device. For example, I may not want my cell phone displaying at full physical dimensions, maybe it's okay to go with 50% or 25% and I know I have to squint at it, but I want my laptop or tablet to use 80-90% physical dimensions, etc...
If no one has thought of this yet well it's here first... prior art recorded.
Thursday, February 23, 2012
Native or Managed?
I want to start a project with a friend to upgrade the UI of windbg. It's not too bad right now, but more importantly I think it could be so much more.
So here's the problem: Making an "awesome" UI is much easier in C# than in C++ using native code. However, the APIs for the debugging engine are all COM based and there aren't bindings for C#. There are a few unofficial bindings, but nothing really available. So, we'd probably have to roll our own bindings, which is not something I want to do.
I think a hybrid option would be to use C++/CLI. At least we'd be able to use the COM methods pretty easily and maybe even use the header files without any modification. We could also then use the managed API for the GUI. I think I'm going to suggest this as the route to take.
There's some concerns that .Net wouldn't be installed on all the environments you'd want to use the new UI but I don't think that should stop us. It's pretty easy to install and a version of .Net ships with modern Windows distributions already.
A bit to think about, but I think the C++/CLI approach is going to be the way to go. It will get me writing code faster and having SOMETHING working much faster than a native solution. I've recently worked on native GDI+ code and although it's nicer than GDI it's still not something I'd want to force myself on for a GUI intensive project.
So here's the problem: Making an "awesome" UI is much easier in C# than in C++ using native code. However, the APIs for the debugging engine are all COM based and there aren't bindings for C#. There are a few unofficial bindings, but nothing really available. So, we'd probably have to roll our own bindings, which is not something I want to do.
I think a hybrid option would be to use C++/CLI. At least we'd be able to use the COM methods pretty easily and maybe even use the header files without any modification. We could also then use the managed API for the GUI. I think I'm going to suggest this as the route to take.
There's some concerns that .Net wouldn't be installed on all the environments you'd want to use the new UI but I don't think that should stop us. It's pretty easy to install and a version of .Net ships with modern Windows distributions already.
A bit to think about, but I think the C++/CLI approach is going to be the way to go. It will get me writing code faster and having SOMETHING working much faster than a native solution. I've recently worked on native GDI+ code and although it's nicer than GDI it's still not something I'd want to force myself on for a GUI intensive project.
Monday, January 30, 2012
ERROR_WORKING_SET_QUOTA and IO Completion Ports
While working on a streaming engine I came across an interesting little hole in the MSDN documentation for Completion Ports. Completion Ports allow for extremely efficient throughput of data. The way this is accomplished is by queuing IO to a "Completion Port" and then associating the Completion Port with one or more threads.
The reason this is so fast is because Windows can then chose which thread will complete the IO operation. Using a thread pool allows Windows to always pick the last executed thread in LIFO order. This greatly reduces TLB thrashing and other issues associated with a context switch on the CPU. When the threads aren't processing IO they are in a wait state. The IO is processed in a FIFO order.
The usual Completion Port architecture looks something like this:
When any IO operation completes Windows will smartly choose a thread waiting on GetQueuedCompletionStatus to wake up and send the IO result. The call to GetQueueCompletionStatus will return and data processing can begin. Ideally, an application would probably only have one Completion Port and perform all IO processing on this port/thread pool pair.
Everything about this is awesome, except... The documentation is really vague about how to handle ReadFile/WriteFile operations returning success (and thus not being queued). You need to make sure you call GetOverlappedResult (and probably with the Wait parameter set to FALSE) or you will start getting strange errors.
After a few of these immediate IO completions my streaming engine started hitting ReadFile failures described by "ERROR_WORKING_SET_QUOTA." Nowhere in the documentation for Completion Ports or GetOverlappedResult does it indicate this should be called in the Completion Port case. I suppose it's implied by the fact that you're using Overlapped IO, but still an explicit indication on MSDN would probably be useful.
This may be obvious to some but I wasted about an hour on this, so hopefully this post will shorten that time for someone else.
The reason this is so fast is because Windows can then chose which thread will complete the IO operation. Using a thread pool allows Windows to always pick the last executed thread in LIFO order. This greatly reduces TLB thrashing and other issues associated with a context switch on the CPU. When the threads aren't processing IO they are in a wait state. The IO is processed in a FIFO order.
The usual Completion Port architecture looks something like this:
- Create a Completion Port using CreateIoCompletionPort.
- Create the threads for the thread pool and call GetQueuedCompletionStatus to associate the threads with the Completion Port.
- Associate file HANDLEs (opened in Overlapped IO mode) to the Completion Port.
- Issue IO operations using ReadFile/WriteFile.
- Process the IO operations in the thread pool threads.
When any IO operation completes Windows will smartly choose a thread waiting on GetQueuedCompletionStatus to wake up and send the IO result. The call to GetQueueCompletionStatus will return and data processing can begin. Ideally, an application would probably only have one Completion Port and perform all IO processing on this port/thread pool pair.
Everything about this is awesome, except... The documentation is really vague about how to handle ReadFile/WriteFile operations returning success (and thus not being queued). You need to make sure you call GetOverlappedResult (and probably with the Wait parameter set to FALSE) or you will start getting strange errors.
After a few of these immediate IO completions my streaming engine started hitting ReadFile failures described by "ERROR_WORKING_SET_QUOTA." Nowhere in the documentation for Completion Ports or GetOverlappedResult does it indicate this should be called in the Completion Port case. I suppose it's implied by the fact that you're using Overlapped IO, but still an explicit indication on MSDN would probably be useful.
This may be obvious to some but I wasted about an hour on this, so hopefully this post will shorten that time for someone else.
Thursday, September 22, 2011
"R6025 - pure virtual function call" Uh oh...
A while ago I stumbled upon a "purecall" crash. The actual fix is pretty boring, but it is interesting to think about why pure-call crashes can happen at all. Since I like looking at disassembly and seeing what the compiler actually does with my code I'll take this approach here.
I recently spoke with a smart friend who's going through the process of learning C++ coding at a university. We started talking about abstract classes and one thing led to another until I brought up pure-call exceptions. More discussion ensued and I posed the question, "How can you actually cause a pure-call exception?"
If you think about it a bit, this should be impossible. There's no way to instantiate an abstract class. The compiler just won't let you. For any real subclasses of the abstract class the compiler will fill in an appropriate function pointer table. So what's the deal? How does this happen?
First, let's have a look at the memory structure of a typical C++ object containing virtual functions.
Here's an object
This corresponds to the following code listing:
If I uncomment the "BOOM" line and run again the application will crash before it gets to the break point. The interesting part is what happens before the crash. Let's have a look at the constructors disassembly. First, the BaseReal constructor:
This looks pretty reasonable. First there's the function prologue and then we do some C++ "this pointer" setup to make all that work. After that we immediately jump into the constructor for
There's the same "this" pointer initialization (although this was probably already done, these constructors have to work in a vacuum, so they may duplicate a little work). Next, the setup of the virtual function table and then the call to
So now this brings us to the
What are the lessons learned? You should never call virtual functions (or functions that call virtual functions) in the constructor or destructor. I didn't show the destructor code, but the whole process of loading the proper function table pointer and setting it is reversed.
Clear as mud?
I recently spoke with a smart friend who's going through the process of learning C++ coding at a university. We started talking about abstract classes and one thing led to another until I brought up pure-call exceptions. More discussion ensued and I posed the question, "How can you actually cause a pure-call exception?"
If you think about it a bit, this should be impossible. There's no way to instantiate an abstract class. The compiler just won't let you. For any real subclasses of the abstract class the compiler will fill in an appropriate function pointer table. So what's the deal? How does this happen?
First, let's have a look at the memory structure of a typical C++ object containing virtual functions.
0:000:x86> ?? tmp
class BaseReal * 0x004a49a0
+0x000 __VFN_table : 0x01312110
+0x004 m_data : 0x1337beef
0:000:x86> dps 0x01312110 L5
01312110 013110c0 cppstuff!BaseReal::`scalar deleting destructor'
01312114 01311120 cppstuff!BaseReal::Get
01312118 01311140 cppstuff!BaseReal::Sum
0131211c 00000000
01312120 00000048
Here's an object
tmp which contains a virtual function table pointer __VFN_table and then a single data element called m_data. I can see from dumping pointer-sized chunks of __VFN_table with symbol matching turned on that it's actually got the function pointers for the class called BaseReal.
This corresponds to the following code listing:
#include <stdio.h>
#include <stdlib.h>
class BaseAbstract
{
public:
BaseAbstract ();
virtual ~BaseAbstract ();
virtual unsigned int Get () = 0;
virtual unsigned int Sum ();
};
class BaseReal : public BaseAbstract
{
public:
BaseReal ();
virtual ~BaseReal ();
virtual unsigned int Get ();
virtual unsigned int Sum ();
unsigned int m_data;
};
BaseAbstract::BaseAbstract ()
{
//Sum (); // BOOM.
}
BaseAbstract::~BaseAbstract ()
{
}
unsigned int BaseAbstract::Sum ()
{
return Get () + 0;
}
BaseReal::BaseReal () : BaseAbstract ()
{
m_data = 0x1337BEEF;
}
BaseReal::~BaseReal ()
{
m_data = 0xDEADBEEF;
}
unsigned int BaseReal::Get ()
{
return m_data;
}
unsigned int BaseReal::Sum ()
{
return Get () + m_data;
}
int main (int argc, char* argv[])
{
BaseReal *tmp = new BaseReal ();
unsigned int value = tmp->Sum (); // <<--- break point here.
delete tmp;
return 0;
}
If I uncomment the "BOOM" line and run again the application will crash before it gets to the break point. The interesting part is what happens before the crash. Let's have a look at the constructors disassembly. First, the BaseReal constructor:
0:000:x86> uf cppstuff!BaseReal::BaseReal
cppstuff!BaseReal::BaseReal :
// Function prologue...
41 00df1090 55 push ebp
41 00df1091 8bec mov ebp,esp
// Setting up the "this" pointer (ecx usually contains 'this') and then
// calling the BaseAbstract constructor.
41 00df1093 51 push ecx
41 00df1094 894dfc mov dword ptr [ebp-4],ecx
41 00df1097 8b4dfc mov ecx,dword ptr [ebp-4]
41 00df109a e861ffffff call cppstuff!BaseAbstract::BaseAbstract (00df1000)
// Loading eax with pointer to 'this' and then storing the virtual function
// table for BaseReal (cppstuff!BaseReal::`vftable' (00df2120)
41 00df109f 8b45fc mov eax,dword ptr [ebp-4]
41 00df10a2 c7002021df00 mov dword ptr [eax],
offset cppstuff!BaseReal::`vftable' (00df2120)
// Saving 0x1337BEEF to m_data.
42 00df10a8 8b4dfc mov ecx,dword ptr [ebp-4]
42 00df10ab c74104efbe3713 mov dword ptr [ecx+4],1337BEEFh
// Function epilogue...
43 00df10b2 8b45fc mov eax,dword ptr [ebp-4]
43 00df10b5 8be5 mov esp,ebp
43 00df10b7 5d pop ebp
43 00df10b8 c3 ret
// Dumping the function table...
0:000:x86> dps cppstuff!BaseReal::`vftable' L3
00df2120 00df10c0 cppstuff!BaseReal::`scalar deleting destructor'
00df2124 00df1120 cppstuff!BaseReal::Get
00df2128 00df1140 cppstuff!BaseReal::Sum
This looks pretty reasonable. First there's the function prologue and then we do some C++ "this pointer" setup to make all that work. After that we immediately jump into the constructor for
BaseAbstract. Once that work is done the m_data member is initialized.
And now a look at the BaseAbstract constructor:
0:000:x86> uf cppstuff!BaseAbstract::BaseAbstract
cppstuff!BaseAbstract::BaseAbstract :
// Function prologue...
27 00df1000 55 push ebp
27 00df1001 8bec mov ebp,esp
// Setting up the "this" pointer (ecx usually contains 'this') and
// saving it on the stack as a local in preparation for calling
// the "Sum" function.
27 00df1003 51 push ecx
27 00df1004 894dfc mov dword ptr [ebp-4],ecx
// Loading eax with pointer to 'this' and then storing the virtual function
// table for BaseReal (cppstuff!BaseAbstract::`vftable' (00df2110)
27 00df1007 8b45fc mov eax,dword ptr [ebp-4]
27 00df100a c7001021df00 mov dword ptr [eax],offset
cppstuff!BaseAbstract::`vftable' (00df2110)
28 00df1010 8b4dfc mov ecx,dword ptr [ebp-4]
// Calling Sum -- which will fail.
28 00df1013 e858000000 call cppstuff!BaseAbstract::Sum (00df1070)
// Function epilogue...
29 00df1018 8b45fc mov eax,dword ptr [ebp-4]
29 00df101b 8be5 mov esp,ebp
29 00df101d 5d pop ebp
29 00df101e c3 ret
// Dumping the function table...
0:000:x86> dps cppstuff!BaseAbstract::`vftable' L3
00df2110 00df1020 cppstuff!BaseAbstract::`scalar deleting destructor'
00df2114 00df1224 cppstuff!purecall
00df2118 00df1070 cppstuff!BaseAbstract::Sum
// Disassembly for BaseAbstract::Sum -- called in the constructor.
0:000:x86> uf cppstuff!BaseAbstract::Sum
cppstuff!BaseAbstract::Sum :
// Prologue...
36 00df1070 55 push ebp
36 00df1071 8bec mov ebp,esp
// Saving ecx. This is somewhat important. Note: the "this call"
// calling convention requires the "this pointer" to be in ecx. The code
// is using ebp-4 to stash the "this" pointer.
36 00df1073 51 push ecx
// Copying "this" (ecx) to local storage -- anything with negative
// ebp references is a local / spill location.
36 00df1074 894dfc mov dword ptr [ebp-4],ecx
// Load the address of the function table into eax.
37 00df1077 8b45fc mov eax,dword ptr [ebp-4]
// Dereference eax into edx -- now we have the function table.
37 00df107a 8b10 mov edx,dword ptr [eax]
// Set ecx to "this" for the call, per calling convention.
37 00df107c 8b4dfc mov ecx,dword ptr [ebp-4]
// Deference the 2nd function table entry (the one for "Get").
37 00df107f 8b4204 mov eax,dword ptr [edx+4]
37 00df1082 ffd0 call eax
// Epilogue...
38 00df1084 8be5 mov esp,ebp
38 00df1086 5d pop ebp
38 00df1087 c3 ret
There's the same "this" pointer initialization (although this was probably already done, these constructors have to work in a vacuum, so they may duplicate a little work). Next, the setup of the virtual function table and then the call to
Sum. I think the compiler took a nice optimization here and didn't use the virtual function table to get the address of Sum. If this were code anywhere other than the constructor I imagine it would have used the function table pointer instead.
So now this brings us to the
Sum code, which I also dumped. You can see it dereferences the virtual function table for the Get function call and then calls it. The problem is this is a pure virtual function so the table entry is for cppstuff!purecall; which is a function added by the compiler as a placeholder to indicate failure.
What are the lessons learned? You should never call virtual functions (or functions that call virtual functions) in the constructor or destructor. I didn't show the destructor code, but the whole process of loading the proper function table pointer and setting it is reversed.
Clear as mud?
Wednesday, September 14, 2011
Header Files and Named Parameters
If you're as much of a geek as I am you may hang out in IRC channels chatting it up with your other coder friends. Occasionally, you may also get into a heated debate about one topic or another.
About a week ago I got into just such a debate (and the geekness of it is a little embarrassing). We were debating whether you should name your parameters in header files or not. As a "software engineer" it's my belief that you should make the actual act of coding as easy as possible. There are enough ways to shoot yourself in the foot and it's always best to minimize complexity and leave that intellectual horsepower to the actual problem at hand.
I find that most seasoned "software engineers" feel this way as well. I've run into a few "programming is just hard" coders, but I believe a real "engineer" would care a little more about their users and also the interesting and unexpected problems that can arise from using other software packages. Given all this, of course I was on the "please, for the love of God, name those parameters" side of the discussion. Well, turns out I was wrong. My decision to include named parameters can actually make programming a LOT harder. Here's why:
Suppose I have 3 files, defined here:
This all seems like it's going to work perfectly. If I compile and run this, I get "20" for my output. Everything is perfect.
Now then, suppose someone changes my main.c like so:
Now my output is the most unexpected: "10"
I know you may wonder why exactly someone would define
Hmmm, so what happened here? Let's have a look at some disassembly to see.
Huh, so that's weird. Looks like if int_arg is defined as a long then the compiler seems to think it should pass an extra 32-bits to the "sum" function. You can also see this in the "stack cleanup" code directly after the function call -- there's an extra 4 bytes being pulled off the stack. The extra 4 bytes are coming from that extra push onto the stack before the function call. So why is this?
Well, the compiler is converting
Well, no. The reason this whole thing compiles and links without warnings is because these are C functions. There is no name decoration. The linker is simply looking for a function called
So there's a case where I was advocating for making things easier only to possibly introduce some really hard-to-find bugs. How would I fix this? I'd add some nice comments to these functions in the header file, which is a common convention. Additionally, I know understand why this convention is used.
Thanks!
About a week ago I got into just such a debate (and the geekness of it is a little embarrassing). We were debating whether you should name your parameters in header files or not. As a "software engineer" it's my belief that you should make the actual act of coding as easy as possible. There are enough ways to shoot yourself in the foot and it's always best to minimize complexity and leave that intellectual horsepower to the actual problem at hand.
I find that most seasoned "software engineers" feel this way as well. I've run into a few "programming is just hard" coders, but I believe a real "engineer" would care a little more about their users and also the interesting and unexpected problems that can arise from using other software packages. Given all this, of course I was on the "please, for the love of God, name those parameters" side of the discussion. Well, turns out I was wrong. My decision to include named parameters can actually make programming a LOT harder. Here's why:
Suppose I have 3 files, defined here:
// lib.h
#ifndef __LIB_H__
#define __LIB_H__
long sum (long int_arg, long int_arg2);
#endif
// lib.c
#include "lib.h"
long sum (long int_arg, long int_arg2)
{
return (int_arg + int_arg2);
}
// main.c
#include <stdio.h>
#include <stdlib.h>
#include "lib.h"
int main (int argc, char *argv[])
{
fprintf (stdout, "%d\n", sum (10, 10));
return 0;
}
This all seems like it's going to work perfectly. If I compile and run this, I get "20" for my output. Everything is perfect.
Now then, suppose someone changes my main.c like so:
// main.c
#define int_arg long
#include <stdio.h>
#include <stdlib.h>
#include "lib.h"
int main (int argc, char *argv[])
{
fprintf (stdout, "%d\n", sum (10, 10));
return 0;
}
Now my output is the most unexpected: "10"
I know you may wonder why exactly someone would define
int_arg to be something like long and I agree this is somewhat contrived. But, it can happen. This was the simplest plausible case I could come up with. Perhaps the coder is using a library that allows you to change the certain behavior by defining what an "int_arg" should be. Perhaps the library is using the "int_arg" type to indicate what type of integers it should read from a file. I know all of this is a stretch, but the important thing to remember is that this can happen. As all good engineers know, if it can happen it most likely will happen eventually.
Hmmm, so what happened here? Let's have a look at some disassembly to see.
// outputs "20"
_main:
004113E0: 55 push ebp
004113E1: 8B EC mov ebp,esp
004113E3: 81 EC C0 00 00 00 sub esp,0C0h
004113E9: 53 push ebx
004113EA: 56 push esi
004113EB: 57 push edi
004113EC: 8D BD 40 FF FF FF lea edi,[ebp-0C0h]
004113F2: B9 30 00 00 00 mov ecx,30h
004113F7: B8 CC CC CC CC mov eax,0CCCCCCCCh
004113FC: F3 AB rep stos dword ptr es:[edi]
004113FE: 6A 0A push 0Ah
00411400: 6A 0A push 0Ah
00411402: E8 EE FC FF FF call @ILT+240(_sum)
00411407: 83 C4 08 add esp,8
0041140A: 8B F4 mov esi,esp
// outputs "10"
_main:
004113E0: 55 push ebp
004113E1: 8B EC mov ebp,esp
004113E3: 81 EC C0 00 00 00 sub esp,0C0h
004113E9: 53 push ebx
004113EA: 56 push esi
004113EB: 57 push edi
004113EC: 8D BD 40 FF FF FF lea edi,[ebp-0C0h]
004113F2: B9 30 00 00 00 mov ecx,30h
004113F7: B8 CC CC CC CC mov eax,0CCCCCCCCh
004113FC: F3 AB rep stos dword ptr es:[edi]
004113FE: 6A 0A push 0Ah
00411400: 6A 00 push 0
00411402: 6A 0A push 0Ah
00411404: E8 EC FC FF FF call @ILT+240(_sum)
00411409: 83 C4 0C add esp,0Ch
0041140C: 8B F4 mov esi,esp
Huh, so that's weird. Looks like if int_arg is defined as a long then the compiler seems to think it should pass an extra 32-bits to the "sum" function. You can also see this in the "stack cleanup" code directly after the function call -- there's an extra 4 bytes being pulled off the stack. The extra 4 bytes are coming from that extra push onto the stack before the function call. So why is this?
Well, the compiler is converting
long int_arg to be long long and then assuming it's an unnamed parameter. Later on, the actual implementation doesn't have the define and is compiled as expecting a long, not a long long. Well, that's odd, but you may ask, "It's going to throw some linker warnings, right?"
Well, no. The reason this whole thing compiles and links without warnings is because these are C functions. There is no name decoration. The linker is simply looking for a function called
sum that it can use. It finds one and no linker-time type-checking is performed. The only reason C++ has any sort of name declaration is because functions can be overloaded with different parameter types. In C++ this would be a linker error but in C it's just a horrible horrible bug.
So there's a case where I was advocating for making things easier only to possibly introduce some really hard-to-find bugs. How would I fix this? I'd add some nice comments to these functions in the header file, which is a common convention. Additionally, I know understand why this convention is used.
// lib.h
#ifndef __LIB_H__
#define __LIB_H__
long sum (long /* int_arg */, long /* int_arg2 */);
#endif
Thanks!
Friday, September 2, 2011
WinDBG microtip
I constantly find myself reading variables and structures the hard way in WinDBG... Why don't I ever just use the "??" command!?
Thursday, August 25, 2011
DBH: How the Debugger Help utility can help.
My work in servicing Windows usually results in a bunch of "This component is crashing, please investigate it" questions. One of the first things I always need to do is locate the source code for that component. Lately, I've been using a nice little utility, dbh.exe to accomplish this.
dbh.exe is included with the Windows Debugging tools and it basically provides a command line interface to the dbghelp.dll. For code-finding purposes I'll use the "src" option.
Here's the output on a small utility I'm working on (it's much more useful for large source bases (like Windows) where the code is spread out in many separate source code depots):
I had to cut off some more of the display because this is on a development machine with private Windows symbols -- it displays all the source paths for the Visual Studio CRT, Windows source files that define various GUIDs used by my application, and other miscellaneous junk.
The most important thing I tell people to answer the "Where does the code for XYZ live?" question is: "Check out dbh.exe in the debugging tools." If you have the symbols to your code it's amazing what you can do with the default tools. There are a few features I'd like to see (disassembly of a function with line data, etc...) and perhaps I'll be making my own version of dbh.exe shortly.
There's so much to dbh.exe and I haven't even scratched the surface with this post. I hope you find it as useful as I do.
dbh.exe is included with the Windows Debugging tools and it basically provides a command line interface to the dbghelp.dll. For code-finding purposes I'll use the "src" option.
Here's the output on a small utility I'm working on (it's much more useful for large source bases (like Windows) where the code is spread out in many separate source code depots):
c:\>dbh d:\data\projects\base\windd\Debug\windd.exe src *.c*
d:\data\projects\base\windd\windd\main.cpp
d:\data\projects\base\windd\windd\globals.cpp
d:\data\projects\base\windd\windd\datastream.cpp
d:\data\projects\base\windd\windd\commands.cpp
...
The most important thing I tell people to answer the "Where does the code for XYZ live?" question is: "Check out dbh.exe in the debugging tools." If you have the symbols to your code it's amazing what you can do with the default tools. There are a few features I'd like to see (disassembly of a function with line data, etc...) and perhaps I'll be making my own version of dbh.exe shortly.
There's so much to dbh.exe and I haven't even scratched the surface with this post. I hope you find it as useful as I do.
Wednesday, August 3, 2011
No More Memory Leaks!
Introduction
One of the most common problems in coding is tracking down memory leaks. I remember working at IGT and trying to track down a really nasty problem -- we had some code that seemed to leak a very small amount of memory. On most products you may actually just yell, "Ship It!" and be done with it but this wasn't an option. The code in question had to run for a very long time -- multiple years depending on the quality of power at the location. A slot machine handles money and having software dealing with money slowly consume all the available memory on a system is not acceptable.Tracking down the offending memory allocation was tedious and time consuming. Reviewing source code and making changes then setting up tests to stress the environment and monitor the memory growth using various system APIs to query available memory. I worked with a great electrical engineer who even spent the extra time to load the data into Excel and come up with graphs. (This didn't really increase our productivity, but it made it look like we were doing something substantial.)
Fast-forward about five years to my work on Microsoft Windows in the Windows Sustained Engineering group. Obviously, memory leaks are still an issue. More importantly, they are trouble for many of our customers. Whenever leaks are reported by customers we need to investigate, find the leak, and fix it. Code review is out of the question for something as substantial as Windows with literally gigabytes of source code. Luckily, there's some great tools available for use on Windows.
User Mode Stack Traces & UMDH
User Mode Stack Traces can be enabled by the gflags.exe utility available in the Debugging Tools for Windows package.
I won't provide a real link here for fear the tag will eventually expire, instead use your favorite search engine to find the latest version. NOTE: Gflags.exe is a powerful and terribly dangerous tool. I don't have time to review all the options available so please be careful.
Gflags is just a utility to enable the appropriate options at the OS level. Windows has plenty of diagnostic information available, this is one of the reasons I consider Windows so "developer friendly." The User Mode Stack Trace options can be found Image File tab of GFlags. You'll need to enter the application name in the image name dialog and hit TAB. Once that's done you can select Create user mode stack trace database to configure tracking for the executable. Make sure you click Apply and OK when done.
Another option is to use the command-line interface of gflags.
gflags.exe -i notepad.exe +ustNow that Windows will track the allocations, you can start using UMDH to capture and analyze "dumps." UMDH stands for User Mode Dump Heap. UMDH.exe is a handy little utility provided with the Debugging Tools for Windows package which operates in two modes. The first mode creates dumps by accessing the user mode stack trace database windows creates for properly configured processes. The second mode analyzes the differences between any two dumps.
I like to think of UMDH dumps as allocation-state snapshots in time. UMDH allows you to compare two snapshots and view the differences between them. Not only will you see memory allocations but you will also see memory de-allocations.
Putting It Together
With the tools introduction done I can finally get to where the magic happens. Here's an example session using the tools:
--- Setup leaky.exe to track user mode stack traces.
C:\test>c:\debuggers\gflags.exe -i leaky.exe +ust
--- Launch leaky.exe (in a different window)
--- Take the first trace.
C:\test>c:\debuggers\umdh.exe -pn:leaky.exe -f:Dump0.txt
--- Take a second trace after executing "memory leaking" operations.
C:\test>c:\debuggers\umdh.exe -pn:leaky.exe -f:Dump1.txt
--- Use the second mode of UMDH to diff the traces.
C:\test>c:\debuggers\umdh.exe Dump0.txt Dump1.txt > Dump_output.txt
--- View the output.
C:\test>type Dump_output.txt
// _NT_SYMBOL_PATH set by default to C:\Windows\symbols
// Debug library initialized ...
DBGHELP: leaky - private symbols & lines
.\leaky.pdb
DBGHELP: ntdll - export symbols
DBGHELP: kernel32 - export symbols
DBGHELP: KERNELBASE - export symbols
//
// Each log entry has the following syntax:
//
// + BYTES_DELTA (NEW_BYTES - OLD_BYTES) NEW_COUNT allocs BackTrace TRACEID
// + COUNT_DELTA (NEW_COUNT - OLD_COUNT) BackTrace TRACEID allocations
// ... stack trace ...
//
// where:
//
// BYTES_DELTA - increase in bytes between before and after log
// NEW_BYTES - bytes in after log
// OLD_BYTES - bytes in before log
// COUNT_DELTA - increase in allocations between before and after log
// NEW_COUNT - number of allocations in after log
// OLD_COUNT - number of allocations in before log
// TRACEID - decimal index of the stack trace in the trace database
// (can be used to search for allocation instances in the original
// UMDH logs).
//
+ 1000 ( 1c00 - c00) 7 allocs BackTrace74F40
+ 4 ( 7 - 3) BackTrace74F40 allocations
ntdll!MD5Final+0000B3DD
leaky!malloc+0000005B (f:\dd\vctools\crt_bld\self_64_amd64\crt\src\malloc.c, 89)
leaky!operator new+0000001F (f:\dd\vctools\crt_bld\self_64_amd64\crt\src\new.cpp, 59)
leaky!Leaky+00000015 (d:\data\projects\base\leaky\leaky\main.cpp, 6)
leaky!main+0000005C (d:\data\projects\base\leaky\leaky\main.cpp, 18)
leaky!__tmainCRTStartup+0000013B (f:\dd\vctools\crt_bld\self_64_amd64\crt\src\crt0.c, 278)
kernel32!BaseThreadInitThunk+0000000D
ntdll!RtlUserThreadStart+00000021
Total increase == 1000 requested + d0 overhead = 10d0
Well, that's a very simple example but I've used this same technique for much more complicated memory leaks in the past -- there's just a few more entries in the call stacks list :). Now you know how to track down (and hopefully prevent) all memory leaks in your code. You can learn a lot when interpreting these UMDH logs. I've included the MSDN article for UMDH log interpretation in the references section. Also, because most system resources are backed by some sort of dynamically allocated memory this technique will also work for other resources leaks as well.
Happy coding!
References
http://support.microsoft.com/kb/268343http://msdn.microsoft.com/en-us/Library/ff551046(v=VS.85).aspx
Subscribe to:
Posts (Atom)