21st June 2001

Intro Blurb

Memory handling under the classic MacOS has always been a little arcane and shaky. Much of the impetus behind the move to a 'modern' OS (OSX) had its roots in the desire to provide much better memory management for Mac users - be that guard pages, efficient virtual memory schemes, protected memory etc. It seems that OSX has achieved many of these goals already, but that many Mac users in production environments are still using the Classic MacOS until OSX matures. This article is for those Mac users who are still running MacOS9.x and wish to squeeze every last ounce of resolution out of available memory. It may also help users with relatively low RAM resources overcome common memory-related issues. For high resolution renders, Newtek have repeatedly stated that OSX is the way to go.

Immutable Facts

Under MacOS9.x, there are a few immutable facts about the OS and Lightwave which put an upper limit on output resolutions available.

1. Although it is now possible in 9.1 to allocate over 1Gig of RAM to an application, it's not clear whether the application itself can use allocations over 1Gig. Certainly, OS9.x versions before 9.1 simply couldn't access more than 1Gig of RAM (physical or virtual).

2. Virtual Memory under OS9.x will not operate if you have more than 1Gig of RAM.

3. The most significant RAM hog in the L6 architecture is the Final Frame Buffer. This is a chunk of memory that Lightwave uses to construct the whole image. There's no way of reducing the RAM required for this buffer if you want to achieve certain resolutions. It's hard and fast and a function of the 32bit render pipeline. Here's the figures (based on Chuck's formula which states that a 7k by 7k render requires 1.8Gig RAM).

As you can see, if you're capped at 1Gig of RAM then the maximum resolution you can achieve under OS9 (unless 9.1 has removed this ceiling) is @5000x5000 px. However, that's only the beginning of the story. This is where the Memory Tower comes in. There are a whole host of other RAM occupants that impact this upper figure - effectively pushing your resolution lower. The figures in table above are potential, but unachievable maximums.

The Memory Tower

OK. Crap name. Suggestions welcome. The diagram at the top of this page is an attempt to show the separate chunks of memory being used when you hit the render button in Lightwave. It's not complete and the topography and proportions will vary from scene to scene. Although there are a few of these chunks that the user has very little control over (Final Frame Buffer being the most obvious) there are many that, if altered, can help reduce memory requirements:

Segment Memory

Segment Memory is a good place to start if you're hitting RAM barriers. The old formula for working out Segment Memory in 5.x was simply to multiply width by height by 24. This would give you a figure, which, when entered into the Segment Memory field would enable you to render your images in one segment. For L6, the formula is more like width by height by 32. The table below shows the amount of RAM required to render in one segment (it seems to be very similar to the buffer table):

Rendering in one segment is good because a) it speeds up motion blur and b) some older plugins and image filters may require it. There's no definitive list of those plugins or Image Filters and effects that require a single segment but here's the only one (!) listed in the Lightwave 6.5b User Guide: anaglypyh stereo simulate.

This is a precis of some of the information I received from Arnie: the Image Filter problems with multi-segment renders were related to the Z-buffer and having to split the Z-buffer across segments. In the current version of Lightwave Image Filters have the option of maintaining a full size (unsegmented) Z buffer which holds the minimum Z for each pixel from all passes. If the Image Filter is coded to work in this way then it can impact the extent to which a reduction in Segment Memory can help, depending on the size of the full-size Z buffer. Normal rendering, Image Filters which don't use this option and pixel filters still use the segment-sized Z buffer for the current pass.

If users know of any specific Image Filters, Pixel Filters or Plugins that require 1 segment, please let me know and I'll list them here.

Arnie Cachelin, a Senior Lightwave Software Engineer, says:

"I definitely recommend tweaking segment memory to get renders working. It was designed for just that. Even if you end up with single scanline segments!"

As you can see, as soon as you attempt to render anything over 1000x1000 in one segment, your RAM requirements start to become high. Lowering your segment memory in certain situations to 8Mb, 4Mb or even 1Mb can liberate memory headroom but may sacrifice speed and may impact certain filters/effects/plugs (although the list is very small!).

Shadow Maps

Obvious really, but try and ensure your shadow maps are as low as possible without sacrificing quality - don't set them to ultra high levels without being conscious of their RAM demands.

Arnie Cachelin, a Senior Lightwave Software Engineer, says:

"If shadowmap memory is really a problem, one can switch to ray-traced shadows, making a time-vs.-memory tradeoff."

Polys and Program Code

Often, there's little more you can do after you've constructed your models to reduce polygons any further. However, poly count has a direct bearing on RAM used in Layout so the more efficient your model the happier Layout will be. More importantly, remember that keeping your render patch level as low as possible will greatly reduce the RAM required by Subdiv objects. It's no use having your render patch level at 20 when it looks fine at 4 - the difference in RAM usage is enormous. Similarly, objects with weight maps occupy much more RAM than the same object without. Using Maps on point selections rather than whole objects can help.

Frame Buffer

Ah. Frame Buffer. There's nothing you can do to impact this one other than reducing your output resolution!

Texture Maps

The classic memory hog. I won't go into this in too much detail because it's arcane and complex, except to say that if you can get away with using 8bit colour images and greyscale maps wherever possible you'll be saving memory. Similarly, never use a resolution that's unnecessarily high for the output resolution. Arnie has clearly stated on the Newtek Discussion Forums that one 4092x4092 texture map will take up 64Mb RAM in Lightwave (the formula for 32bit maps being width x height x 4). According to Arnie, if you turn on texture antialiasing (mipmapping) for that image, the RAM requirements will nearly double. Pixel blending does *not* invoke mipmapping. There's a section in the 6.5b pdf (Appendix 4: Page 988) that discusses the RAM overhead from mipmapping.

If you don't need texture antialiasing on for all of your image maps, then you can make substantial savings. Be aware that, by default, texture antialiasing is switched on for all image maps.

Arnie Cachelin, a Senior Lightwave Software Engineer, says:

"Mac users in particular equate image size with quality, (probably from their print/scanning background). With texture maps in 3D, huge amounts of processing are done on the image, both in the mipmap resampling steps and in the antialiased rendering of the surface itself, which combines the map with lighting/reflection effects, bumpmaps, etc. This tends to negate the benefit of large maps in almost all cases. Users should start with smaller maps, and only increase the res if it is necessary. Setting the OpenGL texture size smaller will also free some RAM, as well as boost real-time performance."

Image Viewer

If you're not intending to save out to an HDR format or to use some of the exposure options in Image Viewer FP then *don't* use Image Viewer FP - it's RAM requirements are substantially more than Image Viewer. Even if you're using Image Viewer non-FP be aware that it's RAM requirements will be fairly significant - holding both a 'blittable 24-bit version of the image as well as the original image (see Arnie's comments below). If RAM is short, turn off Image Viewer completely and use F10 to render your output to a file - you'll make substantial RAM savings, but be aware that some of the output plugins themselves require RAM.

Arnie Cachelin, a Senior Lightwave Software Engineer, says:

"Unfortunately the Image viewer requires a 24-bit version of the image in blittable display format, as well as the 24, 32, 96, or 128 bit original image. Again, this is great for test renders but can be eliminated for final (F10) renders. Some image savers used by F10 renders may duplicate some or all of the image, depending on the requirements of the format. I haven't done it, but a methodical test of memory consumed by the savers could also prove useful".

Show Rendering in Progress

Although it's a small window keeping it open requires more RAM.

Viper

Similarly, Viper adds to RAM requirements. If you're not using it, turn it off.

Arnie Cachelin, a Senior Lightwave Software Engineer, says:

"Viper storage requirements can get large if the viper preview is a larger resolution. It should definitely be disabled for actual final renders, since it also takes some amount of time to accumulate its spare copies of the various per-pixel data it requires."

Multithreading

As we get to Draft 3 of this document, it seems that multithreading may not deserve its position as a RAM chunk. There seems to be a very small RAM overhead in using multithreading (regardless of the number of threads used). Here's another precis of the prevailing view that Arnie forwarded to me: 'each thread gets a separate copy of the arrays that hold the current clipped polygon's vertex data. These are sized according to the polygon in the scene with the most vertices. It's just small change compared to the render buffers (which are shared among all threads). I wouldn't worry about the memory overhead of multithreading.'

What's Missing?

Obviously, this is a very simplistic view of a massively complex subject. There are many other factors - radiosity, caustics, specific buffers, shaders, lens flares etc. - that all occupy RAM and may merit a 'chunk' on the Memory Tower. There's much detailed work to be done on image map formats - some 8bit formats are expanded out in Lightwave to 24bit for example. Feel free to email me so that I can keep this area as accurate as possible.

So, How do I Do High Res Renders in OS9?

Use OSX! If you can't, or the plugins you require are not yet available, either use Limited Region to create thin horizontal strips (must be the full width) of your image and render each one separately or try Rich Helvey's unique tip here.