I am using Visual Studio 2005 so this might have been resolved in 2008 or 2010 – but non-mandatory program suites call for previous version updates! (This is why our rather limited team of developers solve this by supplying mandatory updates via subscription)

Anyway – My solution has a few projects (a main .exe and a few DLL’s) and one of these holds some settings and resources which is accessed through the ‘standard’ interface. We’ve simply double clicked the right place and added settings and resources. It works! It’s magic!!

The problem occurred when I wanted to change the default namespace for the project containing the settings and resources. VS2005 refactor my settings and resources to the new namespace – which is great, that was what I wanted – but in EVERY place where these are referred I need to change the reference manually! That was a LOT of work! Luckily it required no thinking at all!

I find it very ridiculous that the namespace is changed for the settings/resources but isn’t changed throughout the rest of the code – this is actually a feature that the normal ‘refactor’ function handles nicely without much trouble and something that a simple search and replace can solve…

The file format that our software uses for saving and loading projects are basicly a ZIP-archive containing XML. We started noticing some strange error reports that all pointed in the direction of the save-method. This is one part of the program that have never really caused any problems before. The exception being thrown said something about “surrogate character”… More specificly invalid high surrogate… This was the first time I’d heard that term so I quickly googled it and found the answer: Something in the string being saved in the XML-document was in an invalid byte range. How nice…

This is particular funny because the input data came from Excel 2007 files, which is also ZIP-archives containing XML… Which means that the .NET conversion from XML to unicode string and back to XML was failing… ! The solution was to sanitize the strings before putting them in the XML-document for output – something that seems like a nasty hack I didn’t really need if the XML-output code in .NET would encode the strings properly.

My work on ABC-tool keeps looping back to optimizing memory usage. The ABC-tool application runs on Windows PCs which means that it has a 2 GB memory limit (typically). The largest datasets I have every run across is about 150 MB in size – but this was XML-files and the data part was significantly smaller. Even though – we come across the OutOfMemoryException waaaaay too often!

How can that be? What are we doing wrong? How can we make it better?

First off I would like to make some points clear. We write our main product ABC-tool in C#.NET. The program runs in all managed code with a few exceptions. The main exit from managed code is when we send data to Microsoft Excel. The data transmitted could include as much as the entire set of data imported into our program. Beside that we have a small brake from managed code to handle various license specific features – nothing much; probably not even worth mentioning.

Running an entire program in managed code means that we generally don’t manage memory allocation and certainly not de-allocation by ourselves! A few weeks back I wrote a little tool to manage memory consumption evolving over time by different processes and the result was very upsetting! When importing data to ABC-tool the memory consumption peaks due to the internal processing of all text into numbers, dates etc. The troubling part is that right when the data import is complete the data is applied to the views of our program which uses a fair amount of memory in itself. By inserting different pauses and delays I have concluded that the memory consumption level won’t drop until after at least 2-3 seconds of runtime. This means that there is a short delay between releasing the memory and it actually being available again. I have observed that when I start allocating more memory or even do heavy computations the garbage collector refuse to run! My conclusion was that it does nothing until idle! This means that we can have out of memory errors occurring simply because the garbage collector is waiting for us to stop allocating memory – that isn’t right!

One thing that we have certainly done wrong is assuming that memory allocating and releasing was being handled for us. Actually it was – just not while we were doing other things. This means that we will have to change the way we think of our managed code and memory.

During the last few weeks I’ve tried to identify areas of the code that makes the memory consumption peak. I’ve also tried to identify strings of events each resulting in memory peaks and forced garbage collection between them. Last but not least I feel the way I think change every time I write managed code. Memory management is still an issue – I didn’t dodge it and probably never will!

Lately the releases has contained data and presentation oriented additions to the framework which would allow us to save many lines of code if used properly. The only problem is – our users need the framework!

It is safe to say that 95% of our users have .NET version 1.0 and 1.1 installed. When it comes to version 2.0 (which our program is linked upon) the number drops to about 85-90% and framework versions 3.0 and 3.5 is installed on under 5% of the user systems. The 10-15% of our users that lack the required framework version 2.0 is bad enough! For various reasons I won’t go into detail with here – we can’t use the usual approach and bundle the installer in an MSI-pack that automatically installs the needed framework – we have to do this manually! This means that two developers spent most of a day writing some script for our installation script that checked for the proper .NET version and downloads + installs version 2.0 if needed.

I really think that Microsoft could help us out a lot on this by simply making all the versions of the .NET Framework mandatory updates through Windows Update. This would help the installation rate among our users to around 95% or even higher – they simply do as told.

This problem is actually one of the main reasons that the Java platform was discarded during the planning phase of the project. With Java the install base drops below the 5% marker which means that every install would have a bundled framework install. The impact on the users would be enormous. They would see a 150+ MB disk space used and have highly increased download times. Not acceptable.