Tuesday, July 8, 2014

DarkOS: Hobbyist OS architecture taking shape.

I've been tinkering with some things in my "hobbyist OS" for a bit now. There was a bit of hiatus when my son was born but he's getting into a sleeping schedule more conducive to projects.

First, I've been messing around with a few emulators. VirtualBox seems to be the best for my work. A few key bits of functionality make it favorable:

  • It doesn't require some of the more advanced CPU virtualization extensions needed by HyperV.
  • It does run using some virtualization, so it's much faster than something like Bochs.
  • It has a "debugger" mode with windbg style commands to dump registers, gdt, etc.

Bochs is still pretty useful, but it started throwing odd crashes during protected-to-real transitions (more on this in a bit).

http://wiki.osdev.org/ has been very helpful in all my development.

I've got a plan of attack for the OS now:

  • Using long mode (amd64).
  • Using ELF x86-amd64 Linux ABI. Cross compiler running in Cygwin has been built.
  • No real/protected mode ping-ponging.
    • The real/protected transitions were proving more annoying (and variable) between Bochs and VirtualBox. I'd get one working and then the other would fail. Since this whole idea of switching back and forth is full of issues I'm just not doing it.
    • This requires slightly odd bootstrapping of components (loading kernel, drivers from disk but not processing the ELF format at first).
    • Doing this, I hope to spend more time focused on the OS and less on the quirkiness of various BIOS implementations.
  • POSIX user mode environment with a Windows NT -style kernel.

Key components and TODO list:

  • Bootloader
    • Enable A20 support.
    • Get RAM configuration from BIOS.
    • Read kernel boot options from disk or perhaps keyboard.
    • Load kernel stub, kernel, and all modules indicated as required for boot. (The kernel and modules are streamed into higher address RAM directly from disk, no ELF image loading done yet).
    • Switch video to VESA linear frame buffer mode (preferably 1024x768 24/32bit color.
    • Fill out a table of boot and machine parameters for use by the kernel stub.
    • Setup basic identity paging for long mode (amd64).
    • Enter long mode and jump to the kernel stub.
  • Kernel Stub
    • Load kernel from raw file images the Bootloader copied from disk to full on ELF relocated modules.
    • Convert the Bootloader "boot and machine parameters" to a table suitable for calling KeMain(...) with.
    • Calling KeMain(...) to initiate kernel startup.
  • Kernel
    • Initialize a very basic set of kernel debugger stubs accessed through a serial port.
    • Initialize the memory manager based on Kernel Stub provided parameters (used memory, etc.)
    • Initialize the object manager.
    • Create a "system" process (virtual address space, etc.)
    • Call the loader functions in the kernel for each of the Bootloader provided boot critical modules.
    • Call ModuleInit(...) for each boot critical modules.
    • Call ModuleEnter(...) for each of the boot critical modules to initiate PCI/PnP/Hardware enumeration.
    • Using file system and storage modules loaded as part of the boot critical modules group, load and start additional modules required by the system.
    • Create a user mode process ("init") to bootstrap the rest of the system from user mode startup scripts.

Once I get the basic kernel booting I hope to crib quite a few module tidbits from a free *nix distribution like FreeBSD or NetBSD. I'll have to evaluate the licenses I'm choosing for my project and the ones provided by FreeBSD and/or NetBSD.

Some modules will be pretty straight forward. For example, the VESA module will mainly just be a set of helper functions for drawing and outputting text to the linear frame buffer. Other modules like EHCI/XHCI will be much more in-depth and probably won't every be tackled.

Ideally I'd like to be able to boot from USB thumb drive, access the network card, and if the POSIX environment works out, perhaps get enough libraries and components running in user mode for a web browser and self hosting a build of the OS.

Well, that's the plan anyway. I hope to make gradual progress with a few planned breaks in the next few months (moving houses... maybe a vacation...).