2 Years Old!

Another year has gone, and I’ve updated the blog look for a bit of a change. 2008 was pretty productive – three main technologies (NME, for low level rendering, Neash for flash interfacing and HXCPP for generating C++ code) have progressed to the “very useful” stage. I have one more technology to go, “gm2d” – a reasonably light weight API to abstract the platform quirks required for super fast rendering on flash, software NME or OpenGL. This will also contain some stuff for making games more easily. And of course, I hope to get the first three technologies to production quality.

HXCPP 0.3 Released

I have put together a new version of hxcpp, the c++ backend for haxe. New features include improved coverage of language features. All the unit tests except remoting pass now. I have also cleaned up the ocaml code a bit and improved the output consistency. Still a bit to do here, but not that much.

The code now contains a dependency system that allows for incremental compiling, greatly improving the speed.

Dlls are now all in one directory – by adding this to the exe path no dll copying should be required. This still needs a little thought :- I tried to delay load the dll, thereby giving greater control over locating it, but it seems the rtti system brings the dll in before I can change the load path. Apart from this, most of the 0.1 TODO list is finished. There are still a few little language features required – such as “break” from within a return block but it’s 99% there, and the external libraries are pretty much untested.

I have also hijacked the neko code to provide OS libraries. This means that I had to allow “neko” class paths in the cpp target – this seems a little odd – I will have to think about a solution here.

Strings are implemented with wchar_t, rather than utf8 bytes, so some neko functions that took “string” actually take “byte array” in cpp. On the plus side, multi-byte characters are “native” in the c++ target.

The source and demos are in hxcpp-03.zip.

HXCPP 0.2 – Huge performance increase.

I have switched hxcpp over from using ref-counting to using Boehm garbage collection. I have also added some additional perfromance improvements, such as integer-index field names to make interaction with neko more efficient.

The overall result is that for the Physaxe demo, the frame rate went from 24 fps to 82 fps (in opengl mode). The swf file runs at about 35 fps, and neko at about 8 fps. This is about what I was hoping for from the first round, but I got there in the end.

You can download the updated files here.

Boehm GC, virtual inheritance and finalizers.

I’m trying to get a speedup for the cpp backend for haxe by using garbage collection. Initial results are very promising – potentially about twice as fast. Howerver I spent a good few hours getting the the bottom of a little problem. Boehm garbage collection is a very impressive piece of work – it has all sorts of magic that does magical things, such as deal with virtual inheritance. This was a bit of a surprise because you do not always get “real” pointers, when you store an object pointer, you get one with an offset. However it seemed to work. Until I added finalizers to the external draw objectes used by the renderer. Apparently, you can only add finalizers to the “real” pointers (ie, those returned from “GC\_MALLOC” et al), rather than a pointer to the same object related by virtual inheritance. The symptom was that the object gets finalized in the first “gc_collect”, even though it was still “used” as far as I was concerned. I guess this is not too surpising, and the fix was pretty easy, but the fact that everything else worked so well lulled be into not suspecting this initially.

C++ backend for haXe

I have just completed an alpha release of a c++ backend for haxe. This means that you can complile haxe code into a 100% compiled executable. You can download the demo file in hxcpp-01.zip. Sorry, windows only at this stage.

The distribution contains a new cpp backend for haxe. It has been based on a 2.0 version of haxe, which may be a tiny bit out of date. Most of the changes are in the new “gencpp.ml”, and to the standard library files, with a few little extra bits here and there. You can re-compile the
haxe compiler if you have ocaml by using the supplied install.ml script.

To try this version for yourself, first backup your haxe distro and copy then supplied “compile/bin/haxe.exe” and “compiler/std/*” files over the top. Use the “-cpp cpp_directory” command line to generate a directory that contains src, include and nmake files. You can then compile these using the microsoft visual studio “nmake” utility. The build system requires the library, include, make and dlls from the “hxcpp” directory. To access these, you should set the environment variable “HXCPP” to point to hxcpp directory extracted from this distribution. This can be done from right-click-“My Computer”/Properties/Advanced/Environment Variables, or from the commandline before compiling.
These resulting “exe” file also needs the hxcpp.dll file from the hxcpp/dll directory. The should be in your “path”, or simply copy it next to your exe.

You can recompile the hxcpp.dll using the nmake file in the directory. You can change the compile flags from the $HXCPP/nmake.setup file (eg, turn on debug).

Demos

Two demos have been included – “perf”, a small benchmark program I found on the net
and a “Physaxe” demo. The source is included (slightly modified), and so are the binaries.
The cpp src and include directories have been included to give you taste of the
output if you can’t be bothered setting up the compiler yourself.
The binaries can be found in demos/bin, and are compiled for neko, swf and cpp.
The neko version can be run with “neko phx.n” or “neko TestRunner.n”. You do not
need a very recent version of neko, but you do need the included “nme.ndll” findable
by neko (next to it will work).

The cpp version of Physaxe uses the cpp verion of NME. This was compiled from
the same code base as the neko version, except it uses the “neko.h” file found
in the hxcpp directoty instead of the one that comes with neko. The nme.dll should
be next to the compiled exe.

If you want to compile the nme versions yourself, you will need the latest nme and neash
versions from code.google.com:
http://code.google.com/p/nekonme/source/checkout
http://code.google.com/p/neash/source/checkout

Performance
————
The flash version of physaxe runs the fastest, with the cpp version about 70% of the
speed (when using the opengl version), and neko about 20% of the speed.

One of the problems is that the cpp version uses the neko api, which required fields
to be looked up by name, which is quite slow in this implementation. A faster version
could link directly to the hxcpp objects – but then it could not use the same API.
This problem is made far worse by the fact the physaxe re-renders each point in each
object every frame, rather then simply adjusting the matrix of existing objects.

I think the most significant loss of perfromance is coming from the reference counting
housekeeping. I will look into a garbage collected system soon.

The results from the “TestRunner” are mixed with flash being faster for stings, but
cpp faster for maths and looping. Neko is fastest for the sting sort in this case,
but this is unusual because the stings are already sorted. When they are not, neko
is very slow. The cpp string code is very simple, so there is scope for improvement there.

TODO
—-
There is still plenty to do.

  • A lot of the operators (eg, “*=”) have not been looked at.
  • The actual formatting of the generated code needs a complete overhaul.
  • The ml code needs some simplifying/cleaning.
  • The standard libraries (eg, xml,regex)
  • Need some way of locating the various dlls etc.
  • Splitup/refactor the HObject.h et al files.
  • Returning values from blocks/swithes.
  • Complete neko.h
  • Look at GC.

Plenty more, I’m sure.

Neash/NME 0.8 released

While this blog may have been quiet, I have been busy. The next version of Neash and NME has been released on haxelib, making it very easy to upgrade. Some cool features include:

  • Fonts
  • Filters (some)
  • Bitmap Caching
  • Improved opengl speed
  • SWF reading/playing
  • Scroll rects (axis aligned)
  • Haxe 2.0 upgrade
  • But perhaps an introduction is in order. The purpose of Neash is to allow you to create programs that run on both flash and also natively, say as a downloadable program. You start with a simple (or complex) haxe program, targetting flash.


    import flash.display.Sprite;
    import flash.display.Shape;

    class Simple extends Sprite
    {

    public function new()
    {
    super();
    flash.Lib.current.addChild(this);

    // creating a new shape instance
    var circle:Shape = new Shape( );
    // starting color filling
    circle.graphics.beginFill( 0xff9933 , 1 );
    // drawing circle
    circle.graphics.drawCircle( 0 , 0 , 40 );
    // repositioning shape
    circle.x = 80;
    circle.y = 80;

    // adding displayobject to the display list
    addChild( circle );
    }

    static public function main()
    {
    new Simple();
    }
    }

    And you get a SWF that renders something like this. This works well in a browser, but what if you wanted to distrubute this as a stand-alone exe? You would of course use one of the may flash-to-exe tools around. These all, one way or another, involve packaging up the flash runtime, and this has licensing implications. Also, it can be difficult to add DRM or other native extensions to the code. So the alternative offered here is to compile it to neko!

    There are 3 simple steps for compiling to neko. 1. Get the libraries, 2. create the compiler command-line and 3. some very minor source code mods.

    1. Getting the libraries. Simply use the “haxelib” tool that comes with haxe to download and install the “NME” and “Neash” libraries. From the command (shell) prompt, type: haxelib install nme followed by haxelib install neash
    2. Create the compiler command. Rather that typing haxe -main …… every time, you can create a “.hxml” file that contains the commands, then you can simply use haxe file.hxml. The hxml file contains flags or key-value pairs of command-line arguements. You can also use the “–next” to compile to more than one target from the single invovation of haxe.

      To use neash, you will need the nme and neash libraries. To add these, you can use the command-line options “-lib neash” and “-lib nme”. For the neko target, you will also need to redirect the “flash” code to use “neash” instead. This is easily done with the “–remap flash:neash” command.

      So a hxml file that targets both flash and neko looks something like this:

      -main Simple
      -swf Simple.swf
      -swf-version 9
      -swf-header 640:480:100:334433
      -lib neash
      -cmd echo SWF done
      
      --next
      -main Simple
      -neko Simple.n
      --remap flash:neash
      -lib nme
      -lib neash
      -cmd echo Neko done
      

    3. Source code mod. You need to do some very minor modifications to run the neko version using neash. Specifically, you need to call “Init” and “Run” and the first and last things you do in your main routine. eg:

         static public function main()
         {
            neash.Lib.Init("Simple",640,480);
            neash.Lib.SetBackgroundColour(0x334433);
      
            new Simple();
      
            neash.Lib.Run();
         }
      


      Currently there is no way to get the command-line flash header data into the neko programme. The neash calls are perfectly safe under flash, so it is safe to include these in both flash and neko projects. However, you will then need “-lib neash” when compiling your flash version. The alternative is to have some “#if neko” directives in the static main routine, and carry on normally from there.

    So running haxe on this hxml file will produce both “Simple.swf” and a “Simple.n” files. You can run neko Simple.n to run the neko program, producing much the same result. You can use neko Simple.n -opengl to run with opengl acceleration – although that will no be much use in the simple case.

    All these project files, along with many others, can be found in the “samples” area of the neash library that you get when you use haxelib to install neash.

    Neash/NME first release

    clocks.png

    I am pleased to announce the initial version of neash 0.1, that works with the latest version on NME, 0.3.2.
    Neash allows the same code to be compiled for both the flash virtual machine and the neko virtual machine. NME provides the advanced vector rendering routines to emulate the flash9 API. You can download each of these packages using “haxelib install nme” and “haxelib install neash”.

    The image here shows the same code running the the flash player, as well as a native win32 window. The code is in the samples area, and is as follows:

    import neash.display.Sprite;
    import neash.display.Shape;
    import neash.geom.Matrix;
    
    import neash.Timer;
    
    class Clock extends Sprite
    {
       var mCx:Float;
       var mCy:Float;
       var mRad:Float;
    
       var mMinuteHand:Shape;
       var mSecondHand:Shape;
       var mHourHand:Shape;
    
       public function new(inCx:Float, inCy:Float, inRad:Float)
       {
          super();
    
          neash.Lib.current.addChild(this);
    
          mCx = inCx;
          mCy = inCy;
          mRad = inRad;
    
          var gfx = graphics;
          for(i in 0...12)
          {
              gfx.beginFill(0x603030);
              gfx.lineStyle(0x000000);
              var theta = i * Math.PI * 2.0 / 12.0;
              var cos1 = Math.cos(theta-0.01);
              var sin1 = Math.sin(theta-0.01);
              var cos2 = Math.cos(theta+0.01);
              var sin2 = Math.sin(theta+0.01);
    
              gfx.moveTo( mCx + cos1*mRad,     mCy + sin1*mRad );
              gfx.lineTo( mCx + cos2*mRad,     mCy + sin2*mRad );
              gfx.lineTo( mCx + cos2*mRad*1.1, mCy + sin2*mRad*1.1 );
              gfx.lineTo( mCx + cos1*mRad*1.1, mCy + sin1*mRad*1.1 );
              gfx.lineTo( mCx + cos1*mRad,     mCy + sin1*mRad );
              gfx.endFill();
          }
    
          mSecondHand = CreateHand(mRad*0.02,mRad);
          mMinuteHand = CreateHand(mRad*0.05,mRad*0.75);
          mHourHand = CreateHand(mRad*0.1,mRad*0.5);
    
          var timer = new Timer(1000);
          var me = this;
          timer.run = function() { me.SetHandPositions(); }
          SetHandPositions();
       }
    
       function SetHandPositions()
       {
          var date = Date.now();
    
          var hour = date.getHours() % 12;
          var minute = date.getMinutes();
          var seconds = date.getSeconds();
    
          mSecondHand.rotation = seconds *360/60;
          mMinuteHand.rotation = (minute + seconds/60.0)  * 360/60;
          mHourHand.rotation =   (hour + minute/60.0 + seconds/3600.0) * 360/12;
       }
    
       function CreateHand(inWidth:Float, inLength:Float) : Shape
       {
          var result = new Shape();
          addChild(result);
    
          var gfx = result.graphics;
    
          gfx.beginFill(0x303060);
          gfx.moveTo(0,0);
          gfx.lineTo(-inWidth/2,-inWidth);
          gfx.lineTo(0,-inLength);
          gfx.lineTo(inWidth/2,-inWidth);
          gfx.lineTo(0,0);
    
          result.x = mCx;
          result.y = mCy;
    
          return result;
       }
    
       static public function main()
       {
          neash.Lib.Init("Clock",640,480);
          neash.Lib.SetBackgroundColour(0xffffff);
    
          new Clock(320,240,200);
    
          neash.Lib.Run();
       }
    }
    
    

    Porting this from a pure flash program involves:

    1. Changing the “import” commands to replace “flash” with “neash”
    2. Also changing from the “haxe.Timer” to the “neash.Timer”, which supports additional options.
    3. The “main” routine must call “neash.Lib.Init( )” very early and at some point must call “neash.Lib.Run()”.
    4. That’s it!

    I’m looking at ways to remove point 1 using some compiler options, but this may not happen. Also note that the command-line for compiling requires “-lib neash” and, on the native platform, also requires “-lib nme”.

    The clock example is really just to show timers working. NME runs the game loop as fast as it can at the moment, so timers are a bit pointless to some extent. This may change in the future.

    The flash9 api coverage is a little thin at the moment. I have concentrated on flash.display.Graphics, including linear and radial gradients, bitmaps and line styles. However, some supporting things such as events and TextFields, are not well covered. Also note that some things (so far using a colour-alpha as a 32-bit int, eg #ffffffff) are not supported under neko.

    Neash is currently only for flash9/neko compiling. Flash8 and JavaScript will only be supported if there is big need for it (basically if someone else wants to do it).

    Neash does not have a binary component, so it works on platforms currently supported by NME. Namely windows and linux. Mac support will follow when NME is ported.

    shapes.png
    Neash was designed initally for games, and so is quite graphics-oriented. Here is an example showing the neash port of the “plotex” haxe module (you can get it with “haxelib install plotex”) running on windows.
    I’m not sure if this necessarily useful, but it is fun in a dorky sort of a way. One of the porting problems was due to the fact that plotex already supports 3 out of the 4 haxe platforms, so it contains conditional compiles. As a rule of thumb, use neash where you would use flash9. So “#if flash9” becomes “#if (flash9||neko)”. Another issue was parameters passed from the html to the swf object. These are emulated in neash using command-line arguements.
    The source code changes can be found here. Note:copyright retained by original author, see [plotex website](http://code.google.com/p/plotex/) for more details.

    Has It Really Been A Year?

    Well, one year on and I’ve had some fun, but still no games! The technology side of things has certainly come into focus in the last 12 months. I’ve seen haXe double and redouble performance to become my preferred option for web game development. I’ve been actively working on the NME to get the downloadable side of things going, and I’m almost done (just add one more feature…). Once that is done, I will pull my finger out and actually try to release a game – even if it is a little one.

    I’m really enjoying the haXe/flash development, and looking back at my original ideas about trying java, I think I will have to change my game-plan (or “mission statement”) to just look at haXe/as3 based programming. The technology is pretty good these days, so it just comes down to productivity, and having fun.

    Porting NME to linux

    Linux Port
    So, I decided to try porting NME to Linux. The first thing you do is find an empty hard drive, download an ISO, and off you go, right? Wrong. I did this – but I wanted to keep my windows partition, without risk of rooting it over, so I didn’t over write my MBR. Then, I could not actually boot my linux partition. Eventually I got around this by booting from a rescue CD (I do not, and never will again, own a working floppy driver) and specifying my /dev/sd partition in a bit of a round about sort of way. Anyhow, I got Linux booting (with a little bit of pain per boot – but I could live with this), but then the wireless network did not work. This I could not live with, so I though about resurrecting some crappy old hardware I have lying around, but in the end I sought, and found, an infinitely better solution.

    The answer is virtualization! I can’t overstate how much easier this was than actually getting the hardware together. First, I downloaded the free version of [vmware’s “player” product](http://vmware.com/products/player/), and then the Ubuntu 7.1 virtual machine, that included a dev environment [http://jars.de/linux/ubuntu-710-vmware-image-download-english](http://jars.de/linux/ubuntu-710-vmware-image-download-english). The reference to “jars” seems to be a java reference, but we wont hold that against them. It “just worked”. I edited the xorg.conf file to change the keyboard to US, (so I could use the ‘|’ key), and that was about it. With only mimimal clicking, the network was fully working, *and* I could switch to my windows emails etc. This was what I wanted – much better than dual boot. Compiling NME was just a matter of adding the “dev” version of packages (vim, subversion, SDL, SDL\_miver, SDL\_image, TTF) with the GUI “Synaptic Package Manager”, which also “just worked”.

    The download from code.google, via svn, worked easily once I added the “subversion” package.

    Then a little [makefile](http://nekonme.googlecode.com/svn/trunk/project/makefile) to bring it all together. I had to make some code hacks. I don’t know if this is a gcc “bug” or “feature”, but when I tried to call a class member function, that was templated on an int, eg:

    int x = Class.Member<4>(10);

    the gcc compiler thought I was using “operator<", which is a bit of a shame. I also had a quick go at static linking, via the ".a"s, rather than the ".so"s, but the linker told me I needed a "version section" for one of the SDL symbols, so I gave up. The image you see is in the top left is of a window, on my windows desktop, running a virtual machine, running ubuntu, running the neko virtual machine, running a neko script linking to the Linux version of NME. And running it very well, I might add. The only problems at the moment, are text, when a font is not supplied, the "mod" music in the "blox" demo, and OpenGL on my setup. The PNG, JPG, WAV and OGGs stuff all worked first time. I think I might be able to get the music going, if I get the right plugin dso. Also, I'm not sure about what ".so" files need to be distributed with the program to resolve all the dependencies. Any SDL linux guru have any comments on this?

    Compile NME.ndll From Source.

    Neko NME now compiles into a single, statically linked ndll (actually, it still uses plugins for mp3 and tiff). This means you can’t use the standard development distributions for building it, so I’ve made some libraries to use. You can download them from [nme-dev-0.3.tgz](/wp-content/uploads/2007/12/nme-dev-0.3.tgz).

    To compile, first checkout the svn version as described below, then extract these to create a “thirdparty” directory next to the nekonme-read-only directory. The “NME.sln” project should then refere to the libraries and includes in this thirdparty directory.

    This has been build with Visual Studio Express 2005. Currently, this is a windows-only version.