return(benchmark)

Finally, enough of the engine has been reimplemented such that I was able to perform benchmarking again. The results don’t look good.

Last time I benchmarked Copper, I used the following test:


benchmark = {
	m = MSec_ClockTime()
	i = int(0)
	loop {
		if ( int_equal(i: 1000000) ) { stop }
		i = int+(i: 1)
	}
	mm = MSec_ClockTime()
	print(MSecTime_rdc(mm: m:))
}
benchmark()

The times I got using the newest branch of Copper at that time were as follows:

Times in milliseconds:

  • 19811
  • 19675
  • 19711
  • 19645

With optimizations, the times were cut in half:

  • 10819
  • 10780
  • 10909
  • 10968

Using an incrementor (++), times were significantly faster:

  • 8532
  • 8312
  • 8390
  • 8619

Overall, these were still disappointing, so I decided to rewrite much of the engine, including both the parsing and running components.

The new engine has been tested with a new, similar test, but with the functions renamed for shortness. My other tests seemed to suggest that the name length was not too big of a deal (though, yes, it is a significant slow-down factor because this language isn’t compiled).


benchmark = {
	m = MSec_ClockTime()
	i = int(0)
	loop {
		if ( int_equal(i: 1000000) ) { stop }
		i = int+(i: 1)
	}
	mm = MSec_ClockTime()
	print(MSec-(mm: m:))
}

Times in milliseconds for unoptimized code:

  • 23626
  • 24379
  • 24079
  • 23890

Times for optimized code:

  • 12823
  • 12602
  • 12997
  • 12972

Incrementor test:


benchmark = {
	m = MSec_ClockTime()
	i = int(0)
	loop {
		if ( int_equal(i: 1000000) ) { stop }
		int++(i:)
	}
	mm = MSec_ClockTime()
	print(MSec-(mm: m:))
}

Times in milliseconds for unoptimized code:

  • 21379
  • 21401
  • 20909
  • 21216

Times in milliseconds for optimized code:

  • 12187
  • 12060
  • 11646
  • 12057

It should be obvious to the observer that the new system appears to be doing significantly worse, even slower. But where is the bottleneck? For this, we need to do some profiling, which is much easier in the new version of the engine than the old one.
Running the first benchmark code again (adding 1 to increment the number) with profiling code, I was able to see that all of the time was being spent in the execute() function rather than the parse() function, which suggests that there is nothing wrong with the parsing function. That being the case, I’m quite certain most of the issue is in the function calls themselves – the most commonly-used feature in Copper – rather than the loop structure. To be certain, all I needed was some profiling of the opcode for function access and function calls. The profile returned 0 time, but since there were so many calls, it’s easy to see how it all added up.
However, that got me wondering. What exactly is the optimization bonus of languages like Python? If you recall, my speed profiling of Python led to a very short time. If you recall, the actual time for appending to a list was between 258 and 273 milliseconds. But one has to realize that the interpreter isn’t dumb, and it’s likely that a number of things I did were optimized, so I decided to rerun the following code, but adding a print() statement to force the compiler to show me everything:


#! /bin/env/Python3
import time

t1 = time.process_time()
a = []
for i in range(1000000):
	a.append(i)
	print(i, " ")
t2 = time.process_time()
print((t2 - t1) * 1000)

Time in milliseconds:
6588.125144000001

That’s a far cry from the 258 that it was optimized to. Removing the list creation and appending, the time was 6628.6869, which shows that the actual list creation was really cheap. Printing is a time-consuming task, so to be fair, I needed to check the same thing on Copper:


benchmark = {
	m = MSec_ClockTime()
	i = int(0)
	loop {
		if ( int_equal(i: 1000000) ) { stop }
		int++(i:)
		print(i: " \n")
	}
	mm = MSec_ClockTime()
	print(MSec-(mm: m:))
}

Time: 24511

Obviously, Copper is the slow-bird here.

Conclusion

The new version is worse in performance, sadly, although I did learn that the slowest aspect is entirely execution – not even a scrap was parsing. Perhaps my profiling was incorrect, but I doubt it.
Python performs some nice optimizations that save alot of time, as expected, although I’m sure that lists are fast to append onto when their type is built-in (enumerable, rather than identified by string).
I knew beforehand that function calls would definitely slow down Copper, but I was hoping the simplicity of the language itself might compensate for that in part. It didn’t help as much as I had hoped, and thus the Copper VM (and perhaps the language itself) is clearly a poor choice for a language where speed is critical.
On the bright side, the foreign function interface is much easier to target than before, so while the code may be slow, at least it’s nice and pretty on both the scripting side and the C++ side.
If I want things to improve significantly, the next approach to optimization would need to be just-in-time compiling (down to a fast bytecode, like assembly), probably through LLVM (because there’s no way I want to try this myself), and that’s only if I decide to continue in that direction. I had done things the current way because I couldn’t think of another approach that could still perform Copper with all of its mechanisms. Copper has a number of things it does to maintain safety and ease-of-use, such as the automatic initialization of variables, the re-initialization of broken pointers, the automatic cleanup, and so-forth. However, the structure for all of this is complicated, lending itself to tons of function calls.

I don’t have time right now to go back and enhance the virtual machine, though it may be a fun venture in the future.
From now on, I’ll probably be posting documentation and finishing the last bit of work before finally releasing the source code.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s