Archive for September, 2006

Token Filters for Lua

Sunday, September 17th, 2006

In an attempt to at least get a chance at making those haughty Lisp fanatics shut up for a minute about their macros, people have been playing with a patch for Lua which will make some of the same tricks possible: token filters. The idea is that you register with the Lua interpreter your own Lua function which gets the chance to examine and modify the token stream for any subsequent pieces of Lua code that get to be executed. Your function gets called for each token and outputs one or many tokens, but thanks to the tricks made possible by Lua coroutines, you can pretend that you can simply get() any further tokens you need and put() them to the output stream at your own pace.

Of course, this is much more inconvenient than Lisp macros, because a token stream is an order of magnitude harder to examine and make intelligent decisions upon than the no-syntax S-expressions of Lisp, but still, it’s a nice hack. People have started doing truly interesting/useful things, like this function parameter checking facility.

The main problem I see is not with the awkwardness of the token stream mechanics; if this catches on, I think sensible frameworks which allow work with higher-level Lua constructs will emerge. The problem is with the uber-simplistic Lua compiler, which does virtually no optimizations. Most often than not, the idea of macros (and other metaprogramming techniques, such as template metaprogramming in C++) is to introduce simpler syntax for things which will be expanded/calculated, then most of the inefficiency will be optimized away, and all that will happen at compile time. With Lua, there’s nothing to optimize your code, so often you’ll have nothing to gain by running some calculation at compile time instead runtime. And the lack of optimization is a feature - it comes with the “tiny, simple, embeddable” part of the Lua bundle.


Wednesday, September 13th, 2006

I’ve been toying with file system optimization recently as we’re starting to implement a background loading system. At pdimov’s advice, I tried setting FILE_FLAG_SEQUENTIAL_SCAN to all Windows file open operations, and got an amazing 10% of absolutely automatic reduction in load time… an excellent case of money left on the table. (The load process has already been optimized in several iterations, so shaving off 10% by another means would not be trivial.)

It’s of great educational value to examine all kinds of logs and statistics produced by the file system. For example, I estimated that more than 70% of high-level file read operations would consume an entire file; the real number turned out to be around 10%. I found all sorts of weird leftover tiny reads, like skeletons being read one bone at a time, or scripts being loaded via a 512 byte buffer. We probably have a wristwatch programmer embedded undercover in our team…

The very existence of functions like fread() is some kind of lie-to-children, like the fact that the earth is round (it is not) or that the atom is a little planet. People who pride themselves of writing in a “to-the-metal” language like C or C++ to squeeze maximal performance out of the hardware, and still use fread (or, heaven forbid, iostreams), should take some time to read about low-level esoterica like FILE_FLAG_NO_BUFFERED with its sector-aligned reads bypassing the system cache, IO completion ports for file access, or even the scatter/gather API used to build your own cache. This thesis by Jan Wassenberg  quotes 10x improvement in effective reading throughput for a real-world loading sequence over the plain read().

The Case for 64-bit OSes

Friday, September 8th, 2006

I did a small experiment in memory allocation today. My work PC has 2 GB of physical RAM. I tried allocating:

  • 1 GB in one go: failed
  • 2 x 512 MB : failed
  • 1 x 900 MB: succeeded
  • 2048 x 1 MB: failed around the 1800th allocation

Clearly something is fragmenting the address space - a fixed loading address DLL, the AGP aperture, maybe some stubborn driver - I don’t know enough about the Windows internals to find out what it is. But I know I want to be able not to worry about whether it will be possible to allocate 1 GB on a machine with 2 GB of physical RAM!

Iron Lua, please!

Thursday, September 7th, 2006

Via Let’s Kill Dave come the news that Iron Python 1.0 is out. And you could even use it to build XNA games, as it compiles to the same MSIL that C# compiles to.

Python is sooo 2004, however. Lua’s where all the cool kids, sorry, responsible game developers are. Several years ago His Moonness, Roberto himself, had worked on a CLR incarnation of Lua; maybe it’s time to dust it off, bring it up to .NET 2.0 and Lua 5.1, and release it. For great justice.

ATI Tootle

Wednesday, September 6th, 2006

My list of subscribed blogs failed to notify me of the release of an interesting tool by ATI called Tootle, and the associated paper on which it is based. (Well, your ”blogroll” is better than mine, because now you’ve learned  about it from me.) What it does is to rearrange mesh indices to optimize both for vertex cache hit rate AND early-Z hit rate of the resulting pixels; they do it by splitting the mesh in several “roughly planar” patches, doing traditional vertex cache order optimization within each patch, and order the patches to minimize overdraw.

They do it, however, with what seems to be an atrocious preprocessing cost. For example, for a mesh split in 32 patches (which is the low end of the range they cite), they do on the order of 32*32*162 renders of portions of the mesh to determine patch vs. patch occlusion via hardware occlusion culling; it’s not surprising they don’t mention preprocessing times in their results - to render 160K frames should take on the order of minutes, even if you’re being clever.

No PS3 for the sophisticated

Wednesday, September 6th, 2006

Just as I was writing the previous post, whining about the high price of the HDTV+360 combo, in came the news that the Playstation 3 is being delayed by a full 4 months in Europe, until March 2007.  To add insult to injury (or was it the other way round? I can never remember), they’ll be launching in the US with just 400k units, and - get it - 100k units for Japan. If this is not a paper launch, I don’t know what is.

Interesting discussion in this Beyond3D thread:

I think their mentality is now that they don’t really care how many PS3s (as consoles) they sell compared to the 360. I think their main priority is now how many PS3s (as Blu-Ray players and media hubs) they will sell compared to HDDVD and Windows Media Hubs.

Remember that Sony will make a whole lot more money from Blur movie sales than anything else, if the format does kick off. It’s practically just the cost of the discs and packaging and nothing else since the content is already all done, ready to be re-re-resold for the Nth time on a different format at a higher price than DVDs.

So I think in their eyes PS3 might just sell 50M in its lifetime, which is not good compared to PS1 and PS2, but that will mean that there will be 50M Blur players on the market, and that is Good (TM) for them.

Sony might get 30% market share (and MS Unintended get another 30% each), but in their eyes, that might be around 90% of the HD-movie market, which would be GREAT business for them.

Microsoft have already secured next-gen exclusivity of the major soccer (sorry, football) titles for this Christmas season; they should move in for the kill and aggressively reduce the price of 360, at least recalculating the US prices by the exchange rate, e.g. ~250 EUR for the Core and ~300 EUR for the premium.

HDTV is for real

Wednesday, September 6th, 2006

After several months of occasional playing on a Xbox 360 on a old, cheap SDTV, yesterday I bought (as a present, alas, not for me) an “HD Ready” HDTV and had the chance to try the 360 on it.

I took the 360 to the store, to test the TV with it, because I wasn’t sure I could trust the “HD Ready” stickers. Well, they turned to be for real: HD Ready means 720p/1080i (vs. the “full” HDTV, which is supposed to be 1080p, or 1920×1080). I didn’t get around to testing with the 360 in-store, because the demo reels they started were convincing enough. After you’ve watched them, switching to a normal, broadcast TV channel is painful: you realize how desaturated, blocky, blurry and grainy the signal really is. If you’re into gaming, 720p is all you’ll need this (next?) generation - the 360 doesn’t even support 1080p, and PS3 supports it mostly nominally, as it doesn’t have the fillrate for it.

Later, we took the TV to its intended recipient, and tried several games, most notably Dead Rising, which has gotten some bad rap for being virtually unplayable on a SDTV due to its small points fonts. Well, I didn’t realize it even had text in such places :-)  All in all, the HDTV set allows the 360 to display a picture as rich in information as a typical PC game in a typical PC resolution. Definitely a must have if you’re into console gaming; the experience is much, much better.

Incidentally, this raises the price of “console gaming” quite a lot. Theoretically, the 360 is supposed to cost “$299″, but this is its cheaper, HDD-less SKU, and you need to already have a TV, and, of course, if you happen to live in the US. In practice, here in Sofia it goes for something like 500 EUR, and you need to buy a decent HDTV for e.g. 400 EUR (the 29″ Samsung I tried yesterday), which is more like $1200…

The Flow of Car Crashes

Wednesday, September 6th, 2006

I rarely happen to appreciate machinima, but this video of 1000 overlaid replays from PC racer TrackMania is simply beautiful. 

Three Apps

Monday, September 4th, 2006

I’ve been using for the past several weeks three cool little applications. The “past several weeks” seem to me an adequate test period - I easily get excited about new software, but rarely run it a second time if I don’t like it - so I can whole-heartedly recommend them.

The first of them is the Windows Live Writer, one of those desktop blog posting tools. It’s has a clean, nice UI, does the job with minimal fuss and is surprisingly powerful. You point it at your blog homepage, give your username and password and, if you’re lucky - I was - you start posting, as it autodetects your blog type and picks the proper posting API. You can drag an image over its window, resize it, set the flow of text around it and it generates a thumbnail which links to the original (which is also uploaded). It tries to present an editing environment that looks like your post will look when published. Basically, it Just Works. Maybe other desktop posting tools have evolved - last time I checked about an year ago, they were a complete mess.

The second tool is Dark Room, which, I hear, is a Windows reimplementation of something called Writeroom, which is for Mac OS X. It’s a sensory-deprivation-chamber text writing tool: it opens fullscreen, defaulting to green-monospace-on-black-background, and lets you write without distractions. I like writing in such an environment, as I’m certainly guilty of the cardinal let’s-pick-the-proper-style-for-this-heading sin when I write in Word, and have a hard time taking SciTE seriously. Nothing tells me “Just. Start. Writing.” like the dark screen of DarkRoom.

The third tool is called, errr, Console, and is just that, a cmd.exe wrapper with tabs and some more options for buffer size, appearance etc. I tend to have several console prompts open, so it’s handy.

By the way, both DarkRoom and Console support transparency, which seems to me an especially masochistic option. What’s the point of making your text unreadable on purpose?? I remember it was a big thing several years ago with *ix window managers, but I thought it would be a passing fad. I’m surprised to see it in recent applications. I certainly haven’t ever seen anyone use it.

Active vs. Passive Observer

Monday, September 4th, 2006

An interesting, and somewhat obvious in hindsight distinction made by John Carmack in his Quakecon keynote between active and passive observers of a game’s visuals. The passive observer is someone who is watching a screenshot, or watching another person play; the active observer is the user in control of the game. The passive observer is much more likely to notice bad texel/pixel ration, aliasing on the trees in the distance or wobbly shadows; for him, the image is the end. The active observer treats the image as a means to an end, which is the game itself. Essentially, we’re marketing games to passive observers, who’ll buy them and then experience them as active observers.