Category → Programming
Ready, set, Go!
I’m currently experimenting with Go.
I would really recommend anyone who takes programming seriously to have a look at it. What I’ve seen so far is very nice. I mean VERY nice.
The concept and idea behind the language (we need a new systems programming language) is very good – I’m sure this will be big! And those of you caring about size and speed – well – go manage your own memory!
Microsoft XNA
After a slow December which actually only featured a major release of ABC-tool I’m back with a rather simple post…
The past few days I’ve been playing around with Microsoft XNA – yes – it is game development!
I must say – the time I’ve had with XNA had been full of positive experiences! I really think that Microsoft has done a great job with this release – it brings the power of game development to the masses.
In the past I’ve done a very small amount of game programming and the complexity that was needed before I had a simple ‘Hello world’ was enormous! It was very stressful. But now – ‘Game’ is a project type in the programming environment (Visual Studio) that I’m usually working with and they are written in the language (C#) that I’ve got the most experience with… Yay!
Note: My first game won’t be stating “Hello world”… Now that I’m the father of a baby girl it will be “Hello kitty”.
Minimizing memory footprint
My work on ABC-tool keeps looping back to optimizing memory usage. The ABC-tool application runs on Windows PCs which means that it has a 2 GB memory limit (typically). The largest datasets I have every run across is about 150 MB in size – but this was XML-files and the data part was significantly smaller. Even though – we come across the OutOfMemoryException waaaaay too often!
How can that be? What are we doing wrong? How can we make it better?
First off I would like to make some points clear. We write our main product ABC-tool in C#.NET. The program runs in all managed code with a few exceptions. The main exit from managed code is when we send data to Microsoft Excel. The data transmitted could include as much as the entire set of data imported into our program. Beside that we have a small brake from managed code to handle various license specific features – nothing much; probably not even worth mentioning.
Running an entire program in managed code means that we generally don’t manage memory allocation and certainly not de-allocation by ourselves! A few weeks back I wrote a little tool to manage memory consumption evolving over time by different processes and the result was very upsetting! When importing data to ABC-tool the memory consumption peaks due to the internal processing of all text into numbers, dates etc. The troubling part is that right when the data import is complete the data is applied to the views of our program which uses a fair amount of memory in itself. By inserting different pauses and delays I have concluded that the memory consumption level won’t drop until after at least 2-3 seconds of runtime. This means that there is a short delay between releasing the memory and it actually being available again. I have observed that when I start allocating more memory or even do heavy computations the garbage collector refuse to run! My conclusion was that it does nothing until idle! This means that we can have out of memory errors occurring simply because the garbage collector is waiting for us to stop allocating memory – that isn’t right!
One thing that we have certainly done wrong is assuming that memory allocating and releasing was being handled for us. Actually it was – just not while we were doing other things. This means that we will have to change the way we think of our managed code and memory.
During the last few weeks I’ve tried to identify areas of the code that makes the memory consumption peak. I’ve also tried to identify strings of events each resulting in memory peaks and forced garbage collection between them. Last but not least I feel the way I think change every time I write managed code. Memory management is still an issue – I didn’t dodge it and probably never will!
Visual Studio – Again!
Once again Visual Studio has proven troubleish!
I’ve spent the last hour trying to find out why our project at work wouldn’t compile (complaining about missing files) on other machines but would on mine! I went through every file in the repository and confirmed that it was up to date. I confirmed that the same files where present and up-to-date on the other machines… !!!
I turned out that I had been moving files around. I had made a sub-namespace to one of our project and moved some files into it. This can be done with no worries – except I didn’t close VS2005 before I did a commit. BIG mistake!! Apparently the project-file only gets updated with the new file locations when I close VS2005 and not when I hit the save-button…
Another hour spent learning the hard truth about Visual Studio.
Office 2007 formats
I haven’t followed the great format wars over the last year or so – and I must admit that it had slipped out of my consciousness for a bit. Until I saw Mortens post the other day (warning: the link is in danish).
After reading this post my mind spent 5 seconds looping through all the data it has passively consumes over the last year – about formats and stuff. It was hard! For those of you who have seen the movie Butterfly Effect – I image it was like those loop backs…
Well… These days I do a large amount of programming and interaction with the Microsoft Office formats. This means that I do a LOT of COM interfacing from my C# code – and I generally hate it!
The only thing on my mind right now is the possibility of doing XML-only based input/output. And if I am lucky – I can add Open Office support in that process!
Bayesian filtering
Currently I’m thinking a lot about Bayesian filtering and new ways to apply this to my every day software.
For those who aren’t that informed on the subject I would recommend reading A Plan for Spam by Paul Graham. Wikipedia has a few articles of interest as well. I’d recommend the ones on Bayes’ theorem and Bayesian spam filtering.
Currently I see a lot of areas where this technique can be applied – generally everything evolve around e-mails, RSS-feeds and automatic web-site crawling.
I hope that I’ll have the time in the near future to actually implement something.
Memory consumption
I’m currently devoting some of my spare time to developing a tool that can track the memory consumption of individual running processes and display the data on a graph.
To narrow this description down a bit – the tool let’s you select a process and shows a graph of used memory over the last few minutes. Pretty nifty.
Windows XP/Vista lets me track these values numerical and with no history – the application of this feature is rather big during my every day work.
At the moment I’m trying to narrow down some performance and memory consumtion issues in our product which – if solved properly – would allow some very large data sets to be imported.
Double keyed tables
Do you ever find yourself in the need of double keyed tables? Well I do!
If you’re now quite sure what I mean about this I’ll try and sum it up. Imagine a Hashtable where you supply a key-pair instead of one key.
myHashtable.Put(key1, key2, value);
Or if you like that flavor
myHashtable[key1, key2] = value;
So far I haven’t found a good implementation of this scenario which leads me to the only solution: Do it yourself!
Edit: During my work yesterday I made a generic class for double keyed tables. The overall time it took was about 30 minutes and was really worth it! As a natural consequence I can’t reveal the source code here.
Updating .NET Versions
When Microsoft release new versions of their .NET Framework there is no doubt that there will be some nifty new features that we can all benefit from.
Lately the releases has contained data and presentation oriented additions to the framework which would allow us to save many lines of code if used properly. The only problem is – our users need the framework!
It is safe to say that 95% of our users have .NET version 1.0 and 1.1 installed. When it comes to version 2.0 (which our program is linked upon) the number drops to about 85-90% and framework versions 3.0 and 3.5 is installed on under 5% of the user systems. The 10-15% of our users that lack the required framework version 2.0 is bad enough! For various reasons I won’t go into detail with here – we can’t use the usual approach and bundle the installer in an MSI-pack that automatically installs the needed framework – we have to do this manually! This means that two developers spent most of a day writing some script for our installation script that checked for the proper .NET version and downloads + installs version 2.0 if needed.
I really think that Microsoft could help us out a lot on this by simply making all the versions of the .NET Framework mandatory updates through Windows Update. This would help the installation rate among our users to around 95% or even higher – they simply do as told.
This problem is actually one of the main reasons that the Java platform was discarded during the planning phase of the project. With Java the install base drops below the 5% marker which means that every install would have a bundled framework install. The impact on the users would be enormous. They would see a 150+ MB disk space used and have highly increased download times. Not acceptable.
Annoying merge
At work we use a Subversion server to store, branch and merge all our code. Generally this works out great! The life quality of the programming staff has risen quite a few nods after introducing this. Even as a single developer I can only recommend it – the posibility of branching and merging whilst hopping back and forth between different revisions is great! Simply great!
Anyway – one thing that constantly annoys me is the lack of code knowledge by the merge tool. This means that often when two developers has been editing the same file and lines there is a conflict that the merge tool can’t solve properly. Often when files are merged the result can’t compile! This means more management – which means an unhappy developer. Me!
To the best of my knowledge the merge tool could do a much better job if it simply knew what it was merging. A merge tool re-written specifically for C# code would in my oppinion do a much better job in these special cases. I’m not even going to start on the subject of designer generated GUI code merging. We do our very best to avoid this – it is always a source of great failure!
I’ll conclude this rant by noting that my irritation obviously hasn’t peaked yet. If this was the case I would’ve put some effort into a solution.
