Render Farms

Projects range from small one person achievements to huge VFX productions where hundreds of artists contribute to the result. The thing they have in common is that they need computational power to finish certain steps in the process. Rendering is the first one that comes to mind but simulations and compositing take up their own share of CPU cycles.

It is possible to let your workstation chug away on those calculations and eventually it will get done but more often than not a tight deadline does nog give you this luxury.

The speed increase with a Render Farm

Render farms are all about speeding things up. Let's say you have only one workstation and you work 8 hours a day (Ok, you have to be lucky to work only 8 hour days in VFX, usually you need to do more to reach the deadline). That means that you have 16 hours left for rendering your sequence. Even with 10 minutes a frame, which is quite acceptable, it will give you 96 frames which is only 4 seconds of animation (at 24 fps that is) a day. You could try to simplify the scene to speed things up but that is not always possible.

The simplest form of a farm is to have second machine next to your workstation which can do calculations while you continue working. This will more than double your capacity and will get you 10 seconds of animation done per day for the same scene. When you have only one machine it is easy enough to manage the rendering of different scene files manually. No extra management software is needed at this point.

Let's move to a bigger setup where there are 5 artists in the shop and let's say they have a workstation each. With the same 10 minute a frame setup it means that they can render 20 seconds of animation each day when working 8 hour days. You can see that the math behind it is simple enough to make predictions once you got an idea how long a frame will take. I must agree that it can be tricky to predict render times when the contents of the scene and hence the render times per frame change a lot. But of course, an indication is better than nothing.

It becomes obvious: the more machines you get, the more complicated the managing of the jobs will become. In these circumstances it is wise to introduce a render manager into the pipeline.

The render manager

The render manager is a piece of software which automates the distribution of jobs to your render farm. It consists out of two parts. The server side program which is the actual manager and the client side program which activates the right renderer for the job. The server side program usually runs on a dedicated machine, very often the same machine as the license server for your software. The client program is installed on each machine which can handle a render job.

Instead of launching the job manually on the client machines it will be submitted to the render manager. When a job is accepted, the render manager has several tasks to do. First it checks the availability of the clients. It then will send a chunk of the job towards a client. The chunk size can be dictated by the user. It can go from part of a frame to a full frame to even a group of frames. If your frames render fast then it is usually better the group them otherwise one frame per client will do fine.

The render manager will now divide all the chunks between the clients. When a client is finished it will report back to the manager and a new job will be given automatically.

Another task of the render manager is to queue the jobs from the users. It allows several people to submit jobs without the fear that they will fail because the render farm might be too busy. The render managers usually have the option to prioritize jobs according to preset rules.

A good render manager will tell you when things go wrong. When a render fails it usually generates an error message which is passed on to the manager. It will then show you which jobs didn't render properly.

What is needed for running a render farm

• Computers: Ok, that is pretty obvious. Fast CPUs and a lot of memory are preferred. You do not want them to start swapping memory to disk. It will grind the render to a halt. Big hard drives are not important, do get server grade ones though. Rack machines are the preferred choice when not using workstations. They are space efficient and are usually server grade so they are made to run 24/7 at full capacity. Do not use the render manager server as a render box so you need one machine dedicated for the manager to run on.
• Operating system: A good rendering manager works cross platform. It doesn't matter if Maya was installed under Windows, MacOSX or Linux. Most render farms run Linux as you can install it without a GUI. This saves memory and makes the machine more efficient (oh, and Linux is free!). Some software packages do not have a Linux or Mac version and then the choice of OS becomes obvious.
• A render manager: If you are serious about rendering you need a render manager. It will automate everything, speed things up and take a lot of worry out of your hands. It will also make your farm very scalable. There are many different ones out there and all come at different prices so it is hard to recommend any of them.
• Software Licenses: That's right, software is usually not free and you need to buy the right amount of licenses. Most VFX packages have separate render licenses. They tend to be cheaper and are sometimes sold in bundles. For example Maya comes with 5 render licenses when you buy a floating license. Take note that you can only use packages which have a command line render options. A render manager can not start GUIs.
• Network: A fast network is a must. The bigger your farm becomes the more data you have to pull trough those wires. Yes, wires. Running a farm on WiFi is a bad idea.
• A storage server: Rendering images or baking out data can fill up disks pretty quick. A good server with a RAID system for redundancy and continuity will be ideal for storing data. You do not want to loose all that render time because a disk went bad. Remember: a RAID is not a backup system, it was designed to let you keep on working even when a disk dies on you. If you want to be sure that your render data is safe, copy it to a second server after the render job finishes.

Scalability

One of the things we notice in VFX is that render and simulation times never seem to go down. We would assume that with faster processors our render times would go down but instead we see another evolution where the render times stay the same but the complexity of the scene goes up. 

Lucky for us, render farms are to a point very scalable. Just add in more machines and licenses when needed. It only becomes a problem when the server or the network can not handle the traffic anymore. You can imagine that a farm at ILM or Pixar is a complex matter to maintain.

If you you need a quick boost in render capacity but don't have the cash to expand the farm,  you can always hire the services of online render farms. You could say that it is rendering in the cloud. It works very similar as your local render farm with the main difference that you need to upload your data to the cloud first. With a slow internet connection this may take a while but once in the cloud the renders go really fast.