vore microcomputers logo

VapUFS Implementation Hell: Week 2 of VoreStation Development


written by Niko Chow-Stuart (Lead Developer)

drawing of the vap mascot looking at a journal, with a pencil held up on their ear.

while unfortunately, as i expected, i wasn't able to do as much work on the filesystem the week before last; after managing to get somewhat back on track this week, i've finished nearly all aspects of VapUFS's implementation!

regarding the work i've done, i should probably go more in-depth about this image that i posted on my personal social media accounts a bit ago. checklist of things implemented for the filesystem, including direct access, creation and flushing of journal entries, and wrappers for journaled writes. the last entry, labeled directories, is unchecked.

starting at the section labeled "direct access", the implementation was started by writing some basic functions for retrieving the core structures of the filesystem off of the disk (using a generic abstraction layer for the disk, not with any actual disk drivers). this was an important step to take, as very early on, i decided to make it as clear as possible that all structs would be stored on-disk in big-endian values. because of this, i was able to add in a step in the retrieval process that (depending on architecture) converts the big-endian values to the cpu's native byte-ordering, and back again when storing.

as the filesystem is journaled, i then began the work of implementing the creation of journal entries as well as the somewhat more complex step of "flushing" the journal entries (i.e. performing them if they are not corrupt, and then marking them as complete). this section of implementation took much longer than the previous (which i'd say maybe took like an hour at most, including the time i spent procrastinating). the source code (which i'll get to later in this post!) of the filesystem goes much more in-depth about how everything works.

regarding journaling, there is one major aspect listed in the image, that being the classification of single block writes and multi block writes. one thing that i believe i have yet to clarify in the source code is that the defined disk block size for the filesystem should be at least large enough to be able to store the largest struct for the file system (which i believe is the superblock, spanning around 156 bytes). this means that all metadata writes (which are journaled) can be performed using a single block. i was originally planning on using this strategy for all writes, where they would be broken up into single block writes in the journal; however, there is one area where this would be a bad idea: directories. directories, in the typical UFS fashion, are stored in an inode's data blocks; however, they could theoretically span multiple blocks in length. because of this, as well as the fact that they would be stored in a potentially fragile b-tree structure, i decided to add a multi block journaled write. this way, if the system is shut down during a directory contents update (e.g. new folder or file), the directory shouldn't become corrupted.

another potential confusion in the original image would be the difference between "journaling" and "journaled access" (which i admit is probably not the best way to label it). "journaling" here refers to directly creating and executing journal entries, whereas "journaled access" refers to higher-level functions that manage creating journaled writes for various metadata structs as well as file data. notably, file data journaled writes will automatically decide between single block and multi block operations depending on the amount of data being written. this section of implementation also took quite a bit, as it required extra thought into the planning of journal executions and failure states (one of which i realised while writing this that i forgot to implement, that being the deallocation of data blocks upon source block corruption).

other than that, hopefully the only thing left is to write a serialisable b-tree implementation for directory contents. i'm planning on also storing a pure array of file and directory names along-side the b-tree in order to assist with recovery upon corruption, however none of that is written yet so i'll see how things go when the time comes!

regarding the release of the VapUFS source code, it should be up by the time this goes public! and by that i mean, you can view it here: link to git repo. if anyone has trouble viewing it, just shoot me an email at nikocs@voremicrocomputers.com and i'll try to fix it asap!

that's it for the progress this week! hopefully i'll actually start releasing these weekly now that i have a bit more time on my hands.
i will note that donations to our LiberaPay are very much appreciated, especially as i'm currently bouncing between this and some other stuff to try to pay for everything! :D

other than that, have a good week and happy voring!

image of xenia, an anthropomorphic fox who was a contender for the linux mascot image made by @cathodegaytube!