All our corporate files where suddenly inaccessible! This was caused by a malfunctioning harddrive in our main NAS box.

Fortunately we had bought a device with a two-disk RAID-1 configuration. To temporarily resolve the problem we simply had to remove the broken harddrive and reboot the device. Nice and simple! I love RAID!

Henriette Weber (my good friend) didn’t care much for this fact and sent me her new book “Return of Involvement“.

In her own personal way she puts her heart and soul into explaining the marketing and corporate screw-ups and supplying a road ahead. Involvement! I felt closer to her after reading the book, and even though I’m not personally involved in anything that could be connected to the content of the book it set some stuff straight for me regarding my life as a consumer.

I can only recommend everybody to go get the book!

After reading the book myself I talked with some of the people at work which had no second thoughts immediately asking me to borrow the book so the could read this splendid work! They accepted the book straight away mainly because of my recommendation – I know that Henriette will love this – this is one of the key issues of her book!

After (almost) completing the book I decided I absolutely had to go to her reception – and so I did. There I met Henriette who was welcoming everyone with her always shiny happy presence and stunning looks.

Just go get her book!

Last night I found – as I always do the first time around – that my communication wasn’t coming around to the server. I tried, I tried some more, I failed…. And went to bed upset.

I woke up this morning – went to do my morning business and BAM it hit me! Flush!! Oh my… ! I booted the computer, added a “stream.Flush()” and we where off! It is certainly NOT the first time I’ve missed this part! I blame the high level stream handlers – did I tell them to buffer? No!

Anyway – remember to flush people!

Since the program was first introduced to the marked there had been a desire to have some sort of database underneath it all to hold data along with calculation results and overall program state. The nature of the program only allowed users to import and export files which in many cases was far too little. Upon my arrival to the company one of my first comments (easy made because I’m not really there to make up sellable buzz words) where something like “I don’t think you mean database – you simply just want a save function”. Besides that I probably had a lot of other comments as well but the essence here being what mostly describes my work: finding out what people want. This is a classical example of users focusing on how they want it instead of what they want. Needless to say that ABC-tool can now store the program state in project files with absolutely no use of databases.

Which leads me to the wheel-inventing part!

The development team where set on a quest to create a file format from scratch to support the numerous functions that our program holds. Furthermore we where very aware that the requirements to this format could/can change over night – nothing is certain. Last but not least we needed the format to be easy to debug and human readable (shortening development time) and make it easy to implement readers for other programming languages.

Numb from thinking big thoughts I reached into the back of my head and thought: XML! Now there is a format that keeps the human readability in serialized form. Furthermore XML supports all the gibberish that we would ever think to stuff into it – and it is very expandable. The only problems would be that we need to stuff all our data into one single document – fairly easy to accomplish – and then the size of the document would be an issue when dealing with large data files. The last point would be a problem not easily solved by developers (us) but very annoying to users. Imagine a file containing what seems to be a small amount of data craving 100+ MB of storage space – ouch! The fact is that at many similar records – especially when record node names are long – XML sucks! A lot! This takes up way too much disk space due to the fact that node names are repeated for every record.

So – our natural thought was: How do we solve this problem during development and make this the best solution mankind has every seen? The answer was short: Compression! (a fairly long word to be honest!). The compression solution actually also solves the other problem we had and lets us distribute our data into different files. So – as you probably have guessed – the file format that we use to store data, calculations and program state is simply a bunch of XML files organized in one single compressed ZIP-file. Simple and neat. Now where have I head about such a solution before? Oh yeah – the new Microsoft Office formats! I call it: The wheel – re-invented.

Usually it has been the content of World of Warcraft addons – which generally doesn’t contain that many files. This time it is the win32 source code of the Apache HTTPD server.

The ZIP-file downloaded had a size of 9.4MB and the total download time was about 30 seconds to a minute. Windows Vistas current estimate is 1 hour and 46 minutes to complete. I really can’t see why this should be the case. Yes it is a lot of small files. Yes it is a whole lot – but come on – give me a break! Windows XP performed better than this!

If someone can shed some light over this please do so – I’m dying to find out why it is so freakin’ slow!