Olive - September 2020 Update

First of all, I'd really like to apologize for how long it's been since the last update. I never intended for it to be over a year like this.

Olive is still very much alive! If you joined the Discord server or watch the GitHub repository, you'll know that (apart from the last few weeks where I've had to take a step back for health reasons) commits are being pushed to the repo on a near-daily basis. If you haven't been watching those though, you may have understandably gotten the impression that it had been abandoned like so many other open source NLEs projects. For that, once again, I apologize.

Version 0.2 (also known as "The Rewrite") is still not quite ready, but it's growing extremely close to a new release. In this update, I'll discuss what's been accomplished, the challenges I've had along the way, how you can help get the next release out sooner, and why it's taken so long for another update post to come out.

What's taking so long?

Looking back, the time estimates I made for having this rewrite done seem laughable. I estimated anywhere from "about 4 weeks" to "maybe a few months", yet now we're well over a year into it. Turns out trying to write the best NLE you can conceive of tends to take a lot of time (who'd have guessed?)

There are a few big reasons it's all been so slow. One of the biggest is that Olive 0.2 is built around a number of ideas and technologies that kind of haven't been done before all in the same package. No other NLE - not even commercial - has tried to combine complete color management, a fast yet high-fidelity half-float/float-based render pipeline, node-based compositing and audio mixing, and a highly efficient automated disk cache all together in the one program. As a result, a lot of time has been spent simply prototyping and revising, trying to figure out the best ways to pull all of this off in a coherent yet performant way. Unfortunately there's no handbook or step-by-step guide for a lot of this, and while we wish we could have had all the answers and solutions a year ago, that just hasn't been the case.

Another big reason is Olive, like the majority of open source projects, has very few people working on it. The team in total is one lead developer, a handful of occasional contributors, and a few consultants. That's it. And while your donations and pledges are extremely generous and I'm very grateful for them, we still make barely enough money at the moment to support one person full-time - and that's one person attempting to pull off features that not even commercial NLEs with multi-million dollar budgets have implemented.

Obviously we'll need more resources as Olive continues to grow, and we do have plans and ideas for future growth, but while we're still in this experimental/rewrite phase, this is all we've got. But still, it's a lot better than nothing, so once again thank you for sticking through this with us.

What's been accomplished?

Regardless of the difficulties, we are actually super close to a release that boasts all of the aforementioned features (plus even more, such as native support for EXR image sequences and OpenTimelineIO).

The node compositor, while still not fully featured, is very much usable. Connecting nodes together works like visual programming to construct custom effects without needing to write a single line of code (or having to wait for us to implement it). Rather than a fixed pipeline of one effect always occurring after another, you get full control over the flow and how each node links to each other. For the uninitiated, it may seem unintuitive at first, but once you're accustomed to them, nodes are an extremely flexible way of working. I hope as Olive's node system continues to develop, people will be able to create effects that I can't even conceive of.

The disk cache has been rewritten probably half a dozen times to try and squeeze as much performance out as possible, and the current iteration is certainly the quickest yet. You'll remember from previous updates that I mentioned a big issue with 0.1: as soon as too many clips or effects are running at once for the computer to handle, there's nothing the user can do to watch the video back in realtime other than export it; there's no way to pre-render those frames in advance so they can be previewed in the editor. The disk cache is not just an implementation of said pre-rendering, it's also designed to be as fast and unintrusive as possible, working quickly in the background to pre-render frames and optimize scrubbing and playback without any manual intervention required.

The color management, thanks to OpenColorIO, is also very much functional and largely effortless to use. All footage gets transformed to a common reference space, allowing footage from several different cameras/color spaces to match seamlessly in the edit. Additionally, compositing in a reference space like "scene linear" ensures all compositing calculations are radiometrically accurate, essentially blended the same way light blends in the physical world (which most compositing software still doesn't do by default). The result can then be transformed to match any kind of display, or exported to match any delivery color space or a wide gamut color space for further work in another program. All of this helps to ensure your video color stays accurate throughout whatever pipeline you have.

A primitive form of masking also exists in the form of a "polygon" node that can be multiplied with a footage node. This allows users to cut footage into arbitrary shapes and feather the edges by passing the polygon through a blur node. There's still work to be done here, including implementing bezier points and improving the user interface for editing points, but this has been a heavily requested feature and it's very much underway.

These are some of the major features that have been accomplished, but many more have too, which you'll be able to see if you try one of the new builds. Which brings us to...

How you can help

New builds are routinely posted on this GitHub page and we're at a point where testing to find crashes and bugs would be extremely useful and help as get this new release done sooner. You may be aware that the issue tracker has been closed for quite some time now, but I plan to re-open it some time this week (after updating the website to start promoting 0.2). In the mean time, if you'd like to test and find bugs/crashes, feel free to send them either here on Patreon or through the Discord.

Future

The primary goal of 0.2 is to complete enough functionality so that users are able to edit and finish projects with complete color management, the node compositor, and the disk cache at your disposal. Particular emphasis is on the disk cache to ensure it's not only fast, but robust and reliable. Other features we would like to implement for 0.2 are support for OpenTimelineIO (for easy integration with other software, including DAWs and other NLEs) and linear timecode/LTC (for automated synchronization of multiple audio/video sources).

For 0.3 and beyond, we're looking into project management features to help sort through potentially hours of footage, including allowing users to pre-cache/proxy only the sections of footage you need. I would also like to extend functionality for integrating multiple projects together to allow for easier collaboration. We've also been looking into leveraging the render pipeline for network rendering, allowing for the possibility of multiple headless computers working together to render frames both for preview caching and for export.

Why have there been so few update posts here? (long)

Meanwhile, if you're wondering about the lack of updates and overall radio silence on this Patreon (and in general if we're being honest), I have to admit that's more of a personal thing.

The attention surrounding Olive was extremely sudden and honestly a little overwhelming. When it started to drum up, I'd been developing it for a year, and that whole time it was a completely unknown project - in fact it was my first C++ project and it was even my first large-scale programming project. I wasn't experienced by any means and I was learning along the way. I made a lot of mistakes, both from a programming perspective and from a video handling perspective, but very suddenly a lot of people were calling it the "the best" or even the "only good" open source NLE. While I of course appreciated the praise, I started to feel a lot of imposter syndrome because I knew I was still so inexperienced.

There was definitely a lot of good that came out of the attention. This Patreon for instance, as well as meeting the collaborators that are now helping to make Olive better. But in retrospect, I definitely wasn't prepared for it. A project I'd become emotionally invested in after putting together from scratch and that wasn't by any means finished yet was suddenly being scrutinized on what seemed like a global scale. 

The rewrite started not just because the new features I wanted to implement required a radically different foundation, but also because I knew (after a year and a half of learning C++ and people picking apart the code) that the Olive 0.1 codebase was full of problems that made it unsustainable. Part of me hoped that the rewrite would serve to both acknowledge and bury the old and flawed code.

But then the rewrite started to take a long time. A really, really long time. And the demand for Olive just seemed to grow, along with the users running into its various shortcomings. While I'd hoped to finish the rewrite and free both myself and its users from the old buggy code, instead the weight around my neck just grew and grew.

It felt like Olive was becoming what I'd hoped it never would: more empty promises from a buggy, unstable NLE.

I had two choices: I could take time away from the rewrite to refamiliarize myself with the old code and try to fix some of the problems with 0.1, or knuckle down and try to get 0.2 done faster. I always ended up choosing the latter, but to this day I'm not sure if it was the right choice. At the time, it felt reasonable since it often felt like we were almost done (the first iteration of the disk cache was finished all the way back in September 2019). But redesigns upon redesigns kept pushing it back in ways I couldn't predict, and eventually we ended up here: well over a year and still no official updates.

I started to feel like a failure. I'm the sort of person who tends to shut down when stressed, and that shutting down is what resulted in my general radio silence on here, Twitter, Reddit, and even the Discord. As a programmer, I've always been better at coding than at speaking, so rather than (what felt like) disappoint people by talking about how long 0.2 was taking, my singular focus became trying to get it done as fast as I could. It felt like everything would be okay if I just got it done.

But working that hard for that long has its consequences. Eventually, it all came to a head when I recently suffered a stress injury. I'll be fine but it forced me to re-evaluate what I was doing. I'm still committed to making Olive a reality, but there was nothing healthy about the way I was stressing myself out. So I've decided to relax and open up more to people like you, who despite everything are still supporting this project.

I also want to make it clear I don't blame any of the community for how I was feeling. I know it was all in my head and I'm of course thrilled by how much people have been supporting Olive for the past year. And I hope people are looking forward to the new release as much as I am.

Conclusion

So this is once again where I thank you all for your patience and support and announce that - while 0.2 is not yet a release and not yet stable - I would like to start encouraging people to try it out and report their findings. Shortly after this post, I'll be updating the website and re-opening the issue tracker. I might even post on Twitter for the first time in (oh god) almost two years... I admit social media doesn't come naturally to me; like I said, I've always been better at coding than talking.

Thank you all and I'll see you in a month! (for real this time)

Become a patron to

3
Unlock 3 exclusive posts
Be part of the community
Connect via private message