Andrew Graham is creating openFirehawk
8
patrons
$72
per month

VFX Cloud Rendering Direct with Open Source Infrastructure As Code


If you need invoices for tax purposes after pledging I can setup automatic invoicing (for Gold sponsors and up). If you are not comfortable with automatic invoicing, you can send one off donations with paypal.me

For VFX artists and studios, cloud rendering costs more than necessary and creates limits to our craft.


OpenFirehawk allows self management and proposes to make the cloud the cheapest resource possible, with tools provided by open source code.


Your support will help me to make this independent resource available to others and improve it's user friendliness, compatibility, stability and documentation, and take it from a prototype to something that is a hardened and production proven tool that can work in more production environments.


What is openFirehawk?


openFirehawk is an open source framework for cloud rendering and virtual workstations.


The openFirehawk infrastructure as code project focuses on allowing visual effects artists and studios to use elastic compute for cloud rendering (or processing geometry) and provide direct control of render node configuration and data storage. Virtual workstations can also be used to handle data remotely reducing the transport cost of heavy data. openFirehawk seeks to change the viability of cloud computing for VFX artists and studios with cost, availability and customisation. It seeks to provide open source tools to self manage cloud rendering infrastructure.

The cost of rendering is available at a price point direct to the cloud provider, potentially 4-8 times cheaper than alternatives. This example shows cloud rendering for houdini. Simulation and rendered output in this video below was rendered with an AWS spot fleet for $0.01 per core hour, excluding the cost of software licenses for Houdini.
Video 2



Video 3

Improving this project requires considerable time and resources and I would like to continue to work on openFirehawk full time.

Why is it needed?

Cloud rendering promises a lower barrier to entry to high end rendering for startups and individual artists, but to actually use the base infrastructure at the cheapest possible price it's difficult to learn and implement.  Solutions exist with services that attempt to make it easier, but the cost of the service layers is high, they are proprietary closed source code, and control for the end user can be restrictive.  Rendering costs may also be tied to a single vendor, instead of tied to the lowest possible base compute price, which is a fluctuating commodity.  

As a Houdini FX Artist myself, I especially need a customised solution to interact with all data in a cost effective way, and the freedom to improve efficiency the same way I might be able to contribute to a pipeline at any stage of rendering.  This need is potentially shared by all types of CG roles.

How is it built?

openFirehawk uses Terraform to provision infrastructure. Terraform is an open source infrastructure as code language that has broad support for different cloud platforms. Terraform currently will not describe all your resources in the same way for each cloud provider, but it serves as a good foundation to describe resources with different providers since it provides the ability to share data between them and creation of resources. Terraform is the most capable language for interfacing with multiple cloud providers (You can read Terraform: Up and Running for a comparison with other tools). It allows openFirehawk to scale towards other cloud providers in the future.

Prior to this point It has taken me 9 months of research and work to get something working for my needs as an FX TD.  I'm happy to share this work and learning and hopefully enable others who want the same thing.  The test renders above were using Side FX Houdini and no other solution like this exists for the Houdini artist, but it will benefit all kinds of remote rendering needs.  The cost of producing this animation was many times cheaper than my alternatives, and provided the most granular level of control over those resources that I could wish for.

Although openFirehawk is far from being cloud vendor agnostic, it is possible to achieve with similar modules as the ones I've created in the current git repository.

OpenFirehawk Functionality so far...

Currently I have successfully tested in Amazon Web Services dynamic creation of a VPN connection, dynamic creation of highly performant NAS storage, and configuration of a machine image for render nodes to scale a fleet from. That fleet is run with Houdini (with floating licences) and Deadline (which can be used for free on AWS with their Usage Based Licencing).

Technical breakdown of what it does currently:
-It starts remote storage that you manage. It also starts remote render nodes that can automatically scale, and optionally spin up remote workstations to improve interaction with remote data.
-These remote compute resources can connect to an onsite virtual machine over a VPN. that connection is to your onsite license server, rendering management database and connection server for Thinkbox Deadline ( integration with Open Cue would be exciting to integrate too, for those wondering). Onsite resources can connect to the remote infrastructure if required, and still function locally when the remote resources are fully shutdown.
-As an artist I can submit scenes from onsite that generate heavy data remotely (like caches, assets, and simulation). With floating software licenses, I can remotely interact with the heavy data through a virtual workstation when required if that data is too heavy to transfer back on site (in the Terrabytes).
-Any stage of output, like final renders can be returned onsite if required, and any stage of output can be interacted with in artist software remotely.
-Any resource that exists remotely in a private network can be directly managed through a vpn, and can be isolated from the internet.
-The onsite connection can exist in a virtual machine and remain isolated from the rest of your on site network.

What are the benefits in doing this?

If we can interact with all cloud vendors as flexibly as this way, individuals and the biggest VFX houses alike will have access to the lowest possible base instance price for rendering, potentially paid per minute.

For rendering, access to all primary cloud providers can be achieved with an open source platform, and the collective benefit of turning compute into a true commodity will ensure the highest efficiency at the lowest possible price is available.


If you're an individual, or a company that agrees with this vision and would like to contribute -
your support for this open source work will allow it to continue to improve.  I believe for it to become more accessible for others to use would be an amazing thing for the visual effects industry as a whole and for individual artists who seek more creative freedom, or to be location free.  

It would give me great pleasure in being able to continue to advance this work that I am highly passionate about, which is why I have started this page to see just what might be possible with support from fellow artists and companies alike.

The renders must flow.

The benefits of the cloud for VFX are ever changing and debatable, but this point in particular is steady from my study of it;

- The cloud can help every 3D / CG / VFX content producer when used for renderfarm overspill to maintain output and keep images flowing for artists if it's available at an efficient price point.

Artist time is more expensive if they iterate less during crunch time.  Also though, if we are not using the cloud and depend entirely on owning the resources, then it is not cost effective to own so much renderfarm that some of those resources are idle beyond a certain proportion of their life, (or if they are used inefficiently). There's a certain point of usage for a onsite system where it no longer makes sense to own that system, if using the cloud is an option.

Until the cloud is cheaper - there's a certain capacity of resources that it makes sense to own on site.  There's also a flexible buffer for rendering that it makes sense to operate with the cloud - provided it can be managed efficiently.  If the cloud is cheaper, then the cloud buffer can be larger to offload renders on.  That provides more cash resource stability for production and less risk.

I would like to change the way we interact with cloud resources to do this and make solutions highly available.

The cloud will also enforce a need for higher efficiency and smarter workflows in VFX.  Tools for rapid review and  efficient handling of data are likely evolve naturally.  For example, there is no disk really that becomes full and everyone suddenly panics to free up some of it, instead, storage becomes progressively more expensive as it expands with data, without a likely limit we would encounter, except.  tools that provide feedback will evolve to a more efficient state.

It seems that when it comes to rendering on the cloud, you have to choose a vendor and stick with it, since so much of that software that is integrated is unique with each implementation.  The time and resources required to achieve that is high, especially for a complete solution to manage complicated shot work that may have many dependencies.  We cannot press the submit button and expect just the final frames to come back without being able to troubleshoot the big data in the middle, and we also can't afford to move all the heavy simulation data through the web - its inefficient.

It's a problem for VFX companies that want to use the cloud in some capacity will share, and it doesn't make sense to maintain that work multiplied by the number of vfx companies and number of cloud vendors that exist.  Proprietary solutions exist, but without open source code, we are restricted in the way we are allowed to interface with it, and also the systems we are allowed to use, plus the cost of the service layer which can be high.  There's a collective benefit that can be gained by shared infrastructure as code- it allows us to be flexible with cloud providers of choice based on the instantaneous price - which turns compute into a commodity.  

Google, AWS, Azure and others are not likely to initially give us this type of cross compatibility without heavy demand.  An open source VFX infrastructure as code solution has the potential to simplify vendor lock in problems that exist today, in the same way that Terraform has forged this path for cloud dev.

Terraform is a brilliant example - it is so good at what it does while being open source, that various cloud vendors, and individuals themselves contribute to the platform.  Indeed, if Terraform becomes somewhat of a gold standard, compatibility is necessary for those cloud
vendors to be competitive.

What happens to money raised?

The funds raised support me to develope openFirehawk and cover the costs for testing of infrastructure with cloud providers.

This is an evolving creation, and who knows what it may become.  I am coming at it from this approach: The integrity of this creation and what we are enabling collectively is most important to me. I am a true believer of the power of open source, we need more powerful and creative ways of allowing open source to provide for us all. I have previously contributed to Open Source Ecology - https://www.opensourceecology.org on projects like the open source microhouse, and open source Compressed Earth Brick Press. I've also worked on my own Open Source 3D printer with Michael Moon (he was earlier responsible for the first open source 3D printing firmware and also contributed to Smoothieboard 3D printing hardware).

I believe that if we are to survive on earth we will have to bring self awareness, collective good, business and technology all into alignment together - or we will stay in trouble like we are now.

With your support I would like to see openFirehawk become a high functioning solution to the problem of allowing artists to extend their capabilities with the cloud.  If the funds extend past my initial goals, I would also like to take that opportunity to pay other contributors to work on this project.  Other's experience would serve to create a better foundation for some areas than I could.

What is my experience in doing this?

2019 marks the 20th year I have worked in computer graphics. In that time I've contributed to Hollywood film, motion graphics and architectural visualization in the CG industry. I've worked for companies such as ILM, Weta Digital, Sony Pictures Imageworks, Rising Sun Pictures, Image Engine and Macrograph.

I have a deep interest in tools that liberate imagination into reality. As an FX TD, I want to interact with cloud compute resources more closely to enable the types of benefits possible when TD's and developers build pipelines, and I want it to be as cost effective as possible.  I've started to build something that can help studios and individuals offload 3D rendering / simulation workloads with infrastructure as code that they can have control over, and I hope to execute this vision to the best degree that I can.

If you decide to support this, thankyou for your help in making this a reality!
Tiers
openFirehawk Backer
$10 or more per month

Your name will be put in the backers.md in the openFirehawk repository.

openFirehawk Generous Backer
$50 or more per month

Your name will be placed at the top of backers.md in the openFirehawk repository.

Bronze Sponsor
$100 or more per month

Your name or company logo (small) will be put in backers.md in the openFirehawk repository.

Silver Sponsor
$250 or more per month

Your name or company logo (medium) will be put at

  • The top of backers.md in the openFirehawk respository
  • On a dedicated sponsors page on openFirehawk.com
Gold Sponsor
$500 or more per month

Your name or company logo (big) will be put at


  • The homepage of openFirehawk.com.
  • The top of the project repository's README.
Platinum Sponsor
$2,000 or more per month only 5 left

Your name and company logo will be put on:


  • The top left of every documentation page on openFirehawk.com
  • The homepage of openFirehawk.com (at the top of the sponsors section)
  • The top of the project repo's README
Goals
$72 of $4,000 per month
I will be be able to dedicate half of my working time to openFirehawk.
1 of 2

VFX Cloud Rendering Direct with Open Source Infrastructure As Code


If you need invoices for tax purposes after pledging I can setup automatic invoicing (for Gold sponsors and up). If you are not comfortable with automatic invoicing, you can send one off donations with paypal.me

For VFX artists and studios, cloud rendering costs more than necessary and creates limits to our craft.


OpenFirehawk allows self management and proposes to make the cloud the cheapest resource possible, with tools provided by open source code.


Your support will help me to make this independent resource available to others and improve it's user friendliness, compatibility, stability and documentation, and take it from a prototype to something that is a hardened and production proven tool that can work in more production environments.


What is openFirehawk?


openFirehawk is an open source framework for cloud rendering and virtual workstations.


The openFirehawk infrastructure as code project focuses on allowing visual effects artists and studios to use elastic compute for cloud rendering (or processing geometry) and provide direct control of render node configuration and data storage. Virtual workstations can also be used to handle data remotely reducing the transport cost of heavy data. openFirehawk seeks to change the viability of cloud computing for VFX artists and studios with cost, availability and customisation. It seeks to provide open source tools to self manage cloud rendering infrastructure.

The cost of rendering is available at a price point direct to the cloud provider, potentially 4-8 times cheaper than alternatives. This example shows cloud rendering for houdini. Simulation and rendered output in this video below was rendered with an AWS spot fleet for $0.01 per core hour, excluding the cost of software licenses for Houdini.
Video 2



Video 3

Improving this project requires considerable time and resources and I would like to continue to work on openFirehawk full time.

Why is it needed?

Cloud rendering promises a lower barrier to entry to high end rendering for startups and individual artists, but to actually use the base infrastructure at the cheapest possible price it's difficult to learn and implement.  Solutions exist with services that attempt to make it easier, but the cost of the service layers is high, they are proprietary closed source code, and control for the end user can be restrictive.  Rendering costs may also be tied to a single vendor, instead of tied to the lowest possible base compute price, which is a fluctuating commodity.  

As a Houdini FX Artist myself, I especially need a customised solution to interact with all data in a cost effective way, and the freedom to improve efficiency the same way I might be able to contribute to a pipeline at any stage of rendering.  This need is potentially shared by all types of CG roles.

How is it built?

openFirehawk uses Terraform to provision infrastructure. Terraform is an open source infrastructure as code language that has broad support for different cloud platforms. Terraform currently will not describe all your resources in the same way for each cloud provider, but it serves as a good foundation to describe resources with different providers since it provides the ability to share data between them and creation of resources. Terraform is the most capable language for interfacing with multiple cloud providers (You can read Terraform: Up and Running for a comparison with other tools). It allows openFirehawk to scale towards other cloud providers in the future.

Prior to this point It has taken me 9 months of research and work to get something working for my needs as an FX TD.  I'm happy to share this work and learning and hopefully enable others who want the same thing.  The test renders above were using Side FX Houdini and no other solution like this exists for the Houdini artist, but it will benefit all kinds of remote rendering needs.  The cost of producing this animation was many times cheaper than my alternatives, and provided the most granular level of control over those resources that I could wish for.

Although openFirehawk is far from being cloud vendor agnostic, it is possible to achieve with similar modules as the ones I've created in the current git repository.

OpenFirehawk Functionality so far...

Currently I have successfully tested in Amazon Web Services dynamic creation of a VPN connection, dynamic creation of highly performant NAS storage, and configuration of a machine image for render nodes to scale a fleet from. That fleet is run with Houdini (with floating licences) and Deadline (which can be used for free on AWS with their Usage Based Licencing).

Technical breakdown of what it does currently:
-It starts remote storage that you manage. It also starts remote render nodes that can automatically scale, and optionally spin up remote workstations to improve interaction with remote data.
-These remote compute resources can connect to an onsite virtual machine over a VPN. that connection is to your onsite license server, rendering management database and connection server for Thinkbox Deadline ( integration with Open Cue would be exciting to integrate too, for those wondering). Onsite resources can connect to the remote infrastructure if required, and still function locally when the remote resources are fully shutdown.
-As an artist I can submit scenes from onsite that generate heavy data remotely (like caches, assets, and simulation). With floating software licenses, I can remotely interact with the heavy data through a virtual workstation when required if that data is too heavy to transfer back on site (in the Terrabytes).
-Any stage of output, like final renders can be returned onsite if required, and any stage of output can be interacted with in artist software remotely.
-Any resource that exists remotely in a private network can be directly managed through a vpn, and can be isolated from the internet.
-The onsite connection can exist in a virtual machine and remain isolated from the rest of your on site network.

What are the benefits in doing this?

If we can interact with all cloud vendors as flexibly as this way, individuals and the biggest VFX houses alike will have access to the lowest possible base instance price for rendering, potentially paid per minute.

For rendering, access to all primary cloud providers can be achieved with an open source platform, and the collective benefit of turning compute into a true commodity will ensure the highest efficiency at the lowest possible price is available.


If you're an individual, or a company that agrees with this vision and would like to contribute -
your support for this open source work will allow it to continue to improve.  I believe for it to become more accessible for others to use would be an amazing thing for the visual effects industry as a whole and for individual artists who seek more creative freedom, or to be location free.  

It would give me great pleasure in being able to continue to advance this work that I am highly passionate about, which is why I have started this page to see just what might be possible with support from fellow artists and companies alike.

The renders must flow.

The benefits of the cloud for VFX are ever changing and debatable, but this point in particular is steady from my study of it;

- The cloud can help every 3D / CG / VFX content producer when used for renderfarm overspill to maintain output and keep images flowing for artists if it's available at an efficient price point.

Artist time is more expensive if they iterate less during crunch time.  Also though, if we are not using the cloud and depend entirely on owning the resources, then it is not cost effective to own so much renderfarm that some of those resources are idle beyond a certain proportion of their life, (or if they are used inefficiently). There's a certain point of usage for a onsite system where it no longer makes sense to own that system, if using the cloud is an option.

Until the cloud is cheaper - there's a certain capacity of resources that it makes sense to own on site.  There's also a flexible buffer for rendering that it makes sense to operate with the cloud - provided it can be managed efficiently.  If the cloud is cheaper, then the cloud buffer can be larger to offload renders on.  That provides more cash resource stability for production and less risk.

I would like to change the way we interact with cloud resources to do this and make solutions highly available.

The cloud will also enforce a need for higher efficiency and smarter workflows in VFX.  Tools for rapid review and  efficient handling of data are likely evolve naturally.  For example, there is no disk really that becomes full and everyone suddenly panics to free up some of it, instead, storage becomes progressively more expensive as it expands with data, without a likely limit we would encounter, except.  tools that provide feedback will evolve to a more efficient state.

It seems that when it comes to rendering on the cloud, you have to choose a vendor and stick with it, since so much of that software that is integrated is unique with each implementation.  The time and resources required to achieve that is high, especially for a complete solution to manage complicated shot work that may have many dependencies.  We cannot press the submit button and expect just the final frames to come back without being able to troubleshoot the big data in the middle, and we also can't afford to move all the heavy simulation data through the web - its inefficient.

It's a problem for VFX companies that want to use the cloud in some capacity will share, and it doesn't make sense to maintain that work multiplied by the number of vfx companies and number of cloud vendors that exist.  Proprietary solutions exist, but without open source code, we are restricted in the way we are allowed to interface with it, and also the systems we are allowed to use, plus the cost of the service layer which can be high.  There's a collective benefit that can be gained by shared infrastructure as code- it allows us to be flexible with cloud providers of choice based on the instantaneous price - which turns compute into a commodity.  

Google, AWS, Azure and others are not likely to initially give us this type of cross compatibility without heavy demand.  An open source VFX infrastructure as code solution has the potential to simplify vendor lock in problems that exist today, in the same way that Terraform has forged this path for cloud dev.

Terraform is a brilliant example - it is so good at what it does while being open source, that various cloud vendors, and individuals themselves contribute to the platform.  Indeed, if Terraform becomes somewhat of a gold standard, compatibility is necessary for those cloud
vendors to be competitive.

What happens to money raised?

The funds raised support me to develope openFirehawk and cover the costs for testing of infrastructure with cloud providers.

This is an evolving creation, and who knows what it may become.  I am coming at it from this approach: The integrity of this creation and what we are enabling collectively is most important to me. I am a true believer of the power of open source, we need more powerful and creative ways of allowing open source to provide for us all. I have previously contributed to Open Source Ecology - https://www.opensourceecology.org on projects like the open source microhouse, and open source Compressed Earth Brick Press. I've also worked on my own Open Source 3D printer with Michael Moon (he was earlier responsible for the first open source 3D printing firmware and also contributed to Smoothieboard 3D printing hardware).

I believe that if we are to survive on earth we will have to bring self awareness, collective good, business and technology all into alignment together - or we will stay in trouble like we are now.

With your support I would like to see openFirehawk become a high functioning solution to the problem of allowing artists to extend their capabilities with the cloud.  If the funds extend past my initial goals, I would also like to take that opportunity to pay other contributors to work on this project.  Other's experience would serve to create a better foundation for some areas than I could.

What is my experience in doing this?

2019 marks the 20th year I have worked in computer graphics. In that time I've contributed to Hollywood film, motion graphics and architectural visualization in the CG industry. I've worked for companies such as ILM, Weta Digital, Sony Pictures Imageworks, Rising Sun Pictures, Image Engine and Macrograph.

I have a deep interest in tools that liberate imagination into reality. As an FX TD, I want to interact with cloud compute resources more closely to enable the types of benefits possible when TD's and developers build pipelines, and I want it to be as cost effective as possible.  I've started to build something that can help studios and individuals offload 3D rendering / simulation workloads with infrastructure as code that they can have control over, and I hope to execute this vision to the best degree that I can.

If you decide to support this, thankyou for your help in making this a reality!

Recent posts by Andrew Graham

Tiers
openFirehawk Backer
$10 or more per month

Your name will be put in the backers.md in the openFirehawk repository.

openFirehawk Generous Backer
$50 or more per month

Your name will be placed at the top of backers.md in the openFirehawk repository.

Bronze Sponsor
$100 or more per month

Your name or company logo (small) will be put in backers.md in the openFirehawk repository.

Silver Sponsor
$250 or more per month

Your name or company logo (medium) will be put at

  • The top of backers.md in the openFirehawk respository
  • On a dedicated sponsors page on openFirehawk.com
Gold Sponsor
$500 or more per month

Your name or company logo (big) will be put at


  • The homepage of openFirehawk.com.
  • The top of the project repository's README.
Platinum Sponsor
$2,000 or more per month only 5 left

Your name and company logo will be put on:


  • The top left of every documentation page on openFirehawk.com
  • The homepage of openFirehawk.com (at the top of the sponsors section)
  • The top of the project repo's README