Even the smallest amount of programming I do go in my repository, and I wouldn’t want it any other way – the luxury of being able to roll back and forth, branching, merging and suddenly sharing with other developers if the need arise is very nice!

One of the main issues though – especially when working in a team – is how often and how much to commit. During the past few years I’ve realized a few key points:

• You can never commit too often! (Given that:)
• You should never commit code that doesn’t compile!

Following these points is easy, important and gives you very high flexibility. I bend the  second rule a bit, and add the following:

• Branches are for experiments and long hard deviations, and
• Only trunk needs to be compileable at all times

This comes from the (well known) trunk/branches/tags-model which we established when we started using Subversion. We reserve trunk for the place from where we can always release a new version. Code in here should always be compile-ready – no exceptions to this rule at all!

Our branches are used for experimental development, new versions etc. and the compile-issues in here shouldn’t be to the annoyance of the other developers but generally no more than one day’s work should go in each commit (see the first rule)!

Needless to say I follow these rules very precisely all other projects, even the ones where I’m alone and working at night.

The problem with the whole idea is that Google App Engine don’t allow me to handle data requests larger that 1 MB without going through the Blobstore interface. This would be now big problem for me except the fact that to use the Blobstore API my account should enable billing. The… Setup… Process… Of… Billing… And… Quotas… Is… Long… And… Hard… Geez! There are various (personal as well as non-personal) reasons that I just don’t feel like enabling billing on my Google App Engine account just yet…

So for now – this project has been shelved – at least for the Google App Engine platform – I am (as you would guess) still keen on the whole idea and will look into a more traditional development environment ASAP.

The file format that our software uses for saving and loading projects are basicly a ZIP-archive containing XML. We started noticing some strange error reports that all pointed in the direction of the save-method. This is one part of the program that have never really caused any problems before. The exception being thrown said something about “surrogate character”… More specificly invalid high surrogate… This was the first time I’d heard that term so I quickly googled it and found the answer: Something in the string being saved in the XML-document was in an invalid byte range. How nice…

This is particular funny because the input data came from Excel 2007 files, which is also ZIP-archives containing XML… Which means that the .NET conversion from XML to unicode string and back to XML was failing… ! The solution was to sanitize the strings before putting them in the XML-document for output – something that seems like a nasty hack I didn’t really need if the XML-output code in .NET would encode the strings properly.

Or to put it in other words: Silverlight – I don’t get what the fuzz is about!

We already have applets (Java) and flash… To be honest – the only real reason to add another technology to the list of shockingly overlapping solutions is that it is powered by .NET and backed by Miscrosoft…

(ok ok – I know – Silverlight isn’t new – not even by a long shot – but I really still find it kind of redundant)

I have been researching web technologies and programming languages for a future project – and the overwhelming array of redundant technologies left we with a slightly confused feeling. Anyway – I know what road to take – I suspect that I actually did before doing my research.

I would really recommend anyone who takes programming seriously to have a look at it. What I’ve seen so far is very nice. I mean VERY nice.

The concept and idea behind the language (we need a new systems programming language) is very good – I’m sure this will be big! And those of you caring about size and speed – well – go manage your own memory!

Last night I found – as I always do the first time around – that my communication wasn’t coming around to the server. I tried, I tried some more, I failed…. And went to bed upset.

I woke up this morning – went to do my morning business and BAM it hit me! Flush!! Oh my… ! I booted the computer, added a “stream.Flush()” and we where off! It is certainly NOT the first time I’ve missed this part! I blame the high level stream handlers – did I tell them to buffer? No!

Anyway – remember to flush people!

My work on ABC-tool keeps looping back to optimizing memory usage. The ABC-tool application runs on Windows PCs which means that it has a 2 GB memory limit (typically). The largest datasets I have every run across is about 150 MB in size – but this was XML-files and the data part was significantly smaller. Even though – we come across the OutOfMemoryException waaaaay too often!

How can that be? What are we doing wrong? How can we make it better?

First off I would like to make some points clear. We write our main product ABC-tool in C#.NET. The program runs in all managed code with a few exceptions. The main exit from managed code is when we send data to Microsoft Excel. The data transmitted could include as much as the entire set of data imported into our program. Beside that we have a small brake from managed code to handle various license specific features – nothing much; probably not even worth mentioning.

Running an entire program in managed code means that we generally don’t manage memory allocation and certainly not de-allocation by ourselves! A few weeks back I wrote a little tool to manage memory consumption evolving over time by different processes and the result was very upsetting! When importing data to ABC-tool the memory consumption peaks due to the internal processing of all text into numbers, dates etc. The troubling part is that right when the data import is complete the data is applied to the views of our program which uses a fair amount of memory in itself. By inserting different pauses and delays I have concluded that the memory consumption level won’t drop until after at least 2-3 seconds of runtime. This means that there is a short delay between releasing the memory and it actually being available again. I have observed that when I start allocating more memory or even do heavy computations the garbage collector refuse to run! My conclusion was that it does nothing until idle! This means that we can have out of memory errors occurring simply because the garbage collector is waiting for us to stop allocating memory – that isn’t right!

One thing that we have certainly done wrong is assuming that memory allocating and releasing was being handled for us. Actually it was – just not while we were doing other things. This means that we will have to change the way we think of our managed code and memory.

During the last few weeks I’ve tried to identify areas of the code that makes the memory consumption peak. I’ve also tried to identify strings of events each resulting in memory peaks and forced garbage collection between them. Last but not least I feel the way I think change every time I write managed code. Memory management is still an issue – I didn’t dodge it and probably never will!

I’ve spent the last hour trying to find out why our project at work wouldn’t compile (complaining about missing files) on other machines but would on mine! I went through every file in the repository and confirmed that it was up to date. I confirmed that the same files where present and up-to-date on the other machines… !!!

I turned out that I had been moving files around. I had made a sub-namespace to one of our project and moved some files into it. This can be done with no worries – except I didn’t close VS2005 before I did a commit. BIG mistake!! Apparently the project-file only gets updated with the new file locations when I close VS2005 and not when I hit the save-button…

Another hour spent learning the hard truth about Visual Studio.

After reading this post my mind spent 5 seconds looping through all the data it has passively consumes over the last year – about formats and stuff. It was hard! For those of you who have seen the movie Butterfly Effect – I image it was like those loop backs…

Well… These days I do a large amount of programming and interaction with the Microsoft Office formats. This means that I do a LOT of COM interfacing from my C# code – and I generally hate it!

The only thing on my mind right now is the possibility of doing XML-only based input/output. And if I am lucky – I can add Open Office support in that process!

For those who aren’t that informed on the subject I would recommend reading A Plan for Spam by Paul Graham. Wikipedia has a few articles of interest as well. I’d recommend the ones on Bayes’ theorem and Bayesian spam filtering.

Currently I see a lot of areas where this technique can be applied – generally everything evolve around e-mails, RSS-feeds and automatic web-site crawling.

I hope that I’ll have the time in the near future to actually implement something.