Two Guys and a Toolkit - Week 10: Pipeline Terminology & Series Wrap Up

This week we’re going to wrap up the series by talking a little bit about pipeline terminology. More specifically, we’re going to throw out some words and we’d like to know how you’re using them in your facility. I’ll provide a little bit of insight and opinion, but this is your opportunity to share your thoughts on what terms make the most sense for a given concept. 

It seems highly unlikely that there will ever be a day when everyone agrees on a common pipeline language. There are decades of history and culture at various facilities that definitely do not align and I know some people who feel very strongly that their way is the right way. But it would be great to get a discussion going to try and at least understand why we use certain words and whether it makes sense to use different words. Maybe there is some common ground we can meet on. What follows are a few key terms that I hope will jump start the conversation. 

I have to admit that when I started on the Toolkit team and started learning about Shotgun, the term Asset was confusing to me. When you spend 10+ years thinking of assets in the “digital asset” sense, moving into Shotgun land requires some synapse remapping. That said, the word I came to know and use when referring to a cg character/prop/environment was a totally made up word that doesn’t need to be mentioned here. But I’m still not convinced Asset is the right word to refer to this high level concept of a thing that needs to be created in various forms and pushed through a pipeline. What do you think? How does the word Asset sit with you when you think about what it represents in Shotgun. Do you have a better word?

Let’s assume for a minute that Asset, as it is used in Shotgun, is the best word for the job. If so, then I need a new word to describe production files and their associated metadata. In the Shotgun sense these are just files that may eventually be tracked as PublishedFiles. Maybe that’s sufficient, but I’ve worked on a couple of pipelines that have tracked metadata for files, and collections of files, before they were published for downstream consumption. In the last pipeline I worked on, we called these Products. I’m curious if your facility tracks files similarly or if you have a different word you use for this concept. 

In the Shotgun world, the Version entity represents reviewable media. This is another one that really confused me when I joined the team and, even more so than Asset, made me feel like people were speaking a totally different language. Every time I hear someone say they’ve “uploaded a new Version to Shotgun”, my immediate (internal) response is still “a version of what?”

I’ve heard a number of other words used to describe what Shotgun calls a Version including: Take, Clip, Media, Movie, Daily, and Reviewable. What terms do you use? If Version isn’t the best name in your opinion, what word would you use?  In my idealistic world, a Version is just a number, or string, that is tied to an iteration of something: ProductVersion, WorkAreaVersion, TaskVersion, etc. On its own, the word just seems too generic. 

We talked about this one early on in the series, but I’d like to get your input on it as well. In the Toolkit sense, an App is a piece of functionality or UI that runs within an engine. Usually the engine is tied to digital content creation (DCC) software, such as Maya or Nuke, which many of us are used to calling the application, or app for short. Now, if someone comes to me with a problem on production, and I ask them which app they were running when the error occurred, will they say they were using Maya? Or will they say they were in the Publisher? I’m guessing it depends on the user, but I personally think there’s an opportunity to clean up this terminology a bit and make the question a little less ambiguous.

Perhaps one obvious direction would would be to call these things Toolkit Tools instead of Apps. I guess you could argue that you don’t create tools with a toolkit. On the other hand, you could argue that a toolkit comes with a set of Tools (Publisher, Loader, Panel, etc). So which sounds better to you, Publisher App or Publisher Tool? It would be great to get your input on this. Are you fine with App? Do you like Tool? Or do you have another term that you like better? 

Here is a list we put together with a few other words that might be considered overused or confusing when talking to other pipeline folks. Have you had a conversation with someone from another studio whose definition of one of these was completely different from yours? Please let us know in the comments. We’d love to hear your stories regarding pipeline terminology and how it might be improved in Shotgun or across the industry in general. 

I’ve said it a couple of times throughout this series, but it’s worth saying again. Regardless of what terms we use to describe our pipelines, it’s consistency that is the key. The more your production teams speak the same production language, the more efficient your studio will be. I think that’s one place where the Toolkit platform has it right. The fact that each production department can use the same toolset to interact with the pipeline and its data makes communication and support much easier in the long run.

Hi everyone, Jeff here. We have reached the end of the series, so we thought we would take this opportunity to discuss what questions we’ve answered for ourselves.

I think we would need to answer yes to this question. The first issue that we ran up against was publishing look development data in the form of shader networks. This portion of a studio’s workflow will likely vary depending on what DCCs are being used to produce characters, props, and other assets, so the need to implement a custom solution feels mostly appropriate.

The second gap, and the one that is most significant, is the lack of a publish routine for rendered elements out of a 3D DCC (like Maya). This can be a challenging portion of the workflow to develop depending on the desired behavior, but the fact that Toolkit doesn’t come with any implementation at all, even just for the sake of providing an example, came as a surprise to us.

We would agree that figuring out how to work with pipeline configurations, and the masses of YAML data contained within them, presents the biggest roadblock to quick progress. There is a huge amount of information contained in the very deep and complex configuration system that drives the behavior of Toolkit. Tackling that portion of the learning curve is extremely daunting, even for experienced pipeline engineers like ourselves.

The great thing is that everyone else on the Toolkit team agrees with us, and one of the highest priorities is adding polish to this portion of Toolkit. The idea is that the YAML data will still be there, but we’ll be removing some of the need to edit it by hand. If Shotgun Desktop can provide an interface for managing certain portions of a configuration then everyone wins.

Again, we can answer yes depending on what features are being discussed. During the course of the series we outlined proof of concept implementations of things like PublishedFile grouping and PublishedFile statuses.

There are limitations in the configuration system that make multi-location or cloud-based workflows problematic, however. That is an area of Toolkit that is currently getting some attention, though. The “cloud config” project mentioned in this week’s webinar is currently in development, and will address many of these limitations.

So that wraps up our 10 week series on building a simple pipeline using Toolkit! We really enjoyed going through this process with you and we appreciate you allowing us to expand the topics outside of the initial scope of the series. We appreciate all of the feedback and comments we’ve received, and we hope you’ve enjoyed following along. As always, we encourage you to keep the conversation going and let us know if you have questions about Shotgun, Toolkit, or pipelines in general. We’re available via email or on the Shotgun support site anytime you need to reach us. 

In the coming weeks, we’ll be putting together a standalone tutorial for setting up a simple, end-to-end pipeline using Shotgun and Toolkit. This tutorial will take all of the things we learned throughout this process and centralize them into a document that can be used by folks as they begin their own exploration of Toolkit. We’ve gotten some great suggestions in the comments and in support tickets, but if you have any additional input on what you’d like to see in the tutorial, please do let us know.

Once again, thanks for following along!