May 8th, 2009
Last night I was doing some network/socket programming. Just for the fun of it. This was done as I always do (it’s not the first time I’m doing this kind of programming) by building up the socket connection and running it through the stream handlers that the programming language library (Java/C#) provides.
Last night I found - as I always do the first time around - that my communication wasn’t coming around to the server. I tried, I tried some more, I failed…. And went to bed upset.
I woke up this morning - went to do my morning business and BAM it hit me! Flush!! Oh my… ! I booted the computer, added a “stream.Flush()” and we where off! It is certainly NOT the first time I’ve missed this part! I blame the high level stream handlers - did I tell them to buffer? No!
Anyway - remember to flush people!
Tags: C#, Programming
No Comments yet »
May 2nd, 2009
Starting 2009 ABC-tool faced some challenges regarding development. The program has been undergoing some enormous changes since I joined the company and one of the biggest was yet to be seen.
Since the program was first introduced to the marked there had been a desire to have some sort of database underneath it all to hold data along with calculation results and overall program state. The nature of the program only allowed users to import and export files which in many cases was far too little. Upon my arrival to the company one of my first comments (easy made because I’m not really there to make up sellable buzz words) where something like “I don’t think you mean database – you simply just want a save function”. Besides that I probably had a lot of other comments as well but the essence here being what mostly describes my work: finding out what people want. This is a classical example of users focusing on how they want it instead of what they want. Needless to say that ABC-tool can now store the program state in project files with absolutely no use of databases.
Which leads me to the wheel-inventing part!
The development team where set on a quest to create a file format from scratch to support the numerous functions that our program holds. Furthermore we where very aware that the requirements to this format could/can change over night – nothing is certain. Last but not least we needed the format to be easy to debug and human readable (shortening development time) and make it easy to implement readers for other programming languages.
Numb from thinking big thoughts I reached into the back of my head and thought: XML! Now there is a format that keeps the human readability in serialized form. Furthermore XML supports all the gibberish that we would ever think to stuff into it – and it is very expandable. The only problems would be that we need to stuff all our data into one single document – fairly easy to accomplish – and then the size of the document would be an issue when dealing with large data files. The last point would be a problem not easily solved by developers (us) but very annoying to users. Imagine a file containing what seems to be a small amount of data craving 100+ MB of storage space – ouch! The fact is that at many similar records – especially when record node names are long – XML sucks! A lot! This takes up way too much disk space due to the fact that node names are repeated for every record.
So – our natural thought was: How do we solve this problem during development and make this the best solution mankind has every seen? The answer was short: Compression! (a fairly long word to be honest!). The compression solution actually also solves the other problem we had and lets us distribute our data into different files. So – as you probably have guessed – the file format that we use to store data, calculations and program state is simply a bunch of XML files organized in one single compressed ZIP-file. Simple and neat. Now where have I head about such a solution before? Oh yeah – the new Microsoft Office formats! I call it: The wheel – re-invented.
No Comments yet »
January 25th, 2009
After a slow December which actually only featured a major release of ABC-tool I’m back with a rather simple post…
The past few days I’ve been playing around with Microsoft XNA - yes - it is game development!
I must say - the time I’ve had with XNA had been full of positive experiences! I really think that Microsoft has done a great job with this release - it brings the power of game development to the masses.
In the past I’ve done a very small amount of game programming and the complexity that was needed before I had a simple ‘Hello world’ was enormous! It was very stressful. But now - ‘Game’ is a project type in the programming environment (Visual Studio) that I’m usually working with and they are written in the language (C#) that I’ve got the most experience with… Yay!
Note: My first game won’t be stating “Hello world”… Now that I’m the father of a baby girl it will be “Hello kitty”.
No Comments yet »
October 27th, 2008
My work on ABC-tool keeps looping back to optimizing memory usage. The ABC-tool application runs on Windows PCs which means that it has a 2 GB memory limit (typically). The largest datasets I have every run across is about 150 MB in size – but this was XML-files and the data part was significantly smaller. Even though – we come across the OutOfMemoryException waaaaay too often!
How can that be? What are we doing wrong? How can we make it better?
First off I would like to make some points clear. We write our main product ABC-tool in C#.NET. The program runs in all managed code with a few exceptions. The main exit from managed code is when we send data to Microsoft Excel. The data transmitted could include as much as the entire set of data imported into our program. Beside that we have a small brake from managed code to handle various license specific features – nothing much; probably not even worth mentioning.
Running an entire program in managed code means that we generally don’t manage memory allocation and certainly not de-allocation by ourselves! A few weeks back I wrote a little tool to manage memory consumption evolving over time by different processes and the result was very upsetting! When importing data to ABC-tool the memory consumption peaks due to the internal processing of all text into numbers, dates etc. The troubling part is that right when the data import is complete the data is applied to the views of our program which uses a fair amount of memory in itself. By inserting different pauses and delays I have concluded that the memory consumption level won’t drop until after at least 2-3 seconds of runtime. This means that there is a short delay between releasing the memory and it actually being available again. I have observed that when I start allocating more memory or even do heavy computations the garbage collector refuse to run! My conclusion was that it does nothing until idle! This means that we can have out of memory errors occurring simply because the garbage collector is waiting for us to stop allocating memory – that isn’t right!
One thing that we have certainly done wrong is assuming that memory allocating and releasing was being handled for us. Actually it was – just not while we were doing other things. This means that we will have to change the way we think of our managed code and memory.
During the last few weeks I’ve tried to identify areas of the code that makes the memory consumption peak. I’ve also tried to identify strings of events each resulting in memory peaks and forced garbage collection between them. Last but not least I feel the way I think change every time I write managed code. Memory management is still an issue – I didn’t dodge it and probably never will!
Tags: .NET, C#, Programming
No Comments yet »
October 22nd, 2008
Once again Visual Studio has proven troubleish!
I’ve spent the last hour trying to find out why our project at work wouldn’t compile (complaining about missing files) on other machines but would on mine! I went through every file in the repository and confirmed that it was up to date. I confirmed that the same files where present and up-to-date on the other machines… !!!
I turned out that I had been moving files around. I had made a sub-namespace to one of our project and moved some files into it. This can be done with no worries - except I didn’t close VS2005 before I did a commit. BIG mistake!! Apparently the project-file only gets updated with the new file locations when I close VS2005 and not when I hit the save-button…
Another hour spent learning the hard truth about Visual Studio.
Tags: C#, Programming, vs2005
No Comments yet »
October 10th, 2008
I haven’t followed the great format wars over the last year or so - and I must admit that it had slipped out of my consciousness for a bit. Until I saw Mortens post the other day (warning: the link is in danish).
After reading this post my mind spent 5 seconds looping through all the data it has passively consumes over the last year - about formats and stuff. It was hard! For those of you who have seen the movie Butterfly Effect - I image it was like those loop backs…
Well… These days I do a large amount of programming and interaction with the Microsoft Office formats. This means that I do a LOT of COM interfacing from my C# code - and I generally hate it!
The only thing on my mind right now is the possibility of doing XML-only based input/output. And if I am lucky - I can add Open Office support in that process!
Tags: C#, excel, office, Programming
No Comments yet »
September 13th, 2008
At the moment I have some spare time to write something. For the silliest of reasons! I’m waiting for Windows Vista to extract the content of a ZIP-file.
Usually it has been the content of World of Warcraft addons - which generally doesn’t contain that many files. This time it is the win32 source code of the Apache HTTPD server.
The ZIP-file downloaded had a size of 9.4MB and the total download time was about 30 seconds to a minute. Windows Vistas current estimate is 1 hour and 46 minutes to complete. I really can’t see why this should be the case. Yes it is a lot of small files. Yes it is a whole lot - but come on - give me a break! Windows XP performed better than this!
If someone can shed some light over this please do so - I’m dying to find out why it is so freakin’ slow!
No Comments yet »
August 19th, 2008
I’m currently devoting some of my spare time to developing a tool that can track the memory consumption of individual running processes and display the data on a graph.
To narrow this description down a bit - the tool let’s you select a process and shows a graph of used memory over the last few minutes. Pretty nifty.
Windows XP/Vista lets me track these values numerical and with no history - the application of this feature is rather big during my every day work.
At the moment I’m trying to narrow down some performance and memory consumtion issues in our product which - if solved properly - would allow some very large data sets to be imported.
Tags: memory, Programming
No Comments yet »
August 13th, 2008
Do you ever find yourself in the need of double keyed tables? Well I do!
If you’re now quite sure what I mean about this I’ll try and sum it up. Imagine a Hashtable where you supply a key-pair instead of one key.
myHashtable.Put(key1, key2, value);
Or if you like that flavor
myHashtable[key1, key2] = value;
So far I haven’t found a good implementation of this scenario which leads me to the only solution: Do it yourself!
Edit: During my work yesterday I made a generic class for double keyed tables. The overall time it took was about 30 minutes and was really worth it! As a natural consequence I can’t reveal the source code here.
Tags: Programming
No Comments yet »