ATTORNEY DOCKET NO. 5199.FACT.PR TITLE CLOUD BASED VIDEO ANIMATION RENDERING MANAGEMENT SYSTEM INVENTOR: Thomas Ryan Mikota, of Orem Utah 1
The operating system (or user interface) stores most different file types as data objects in a large database. Each data object has tags and hooks which associate the object with related objects, people, software, graphical interface elements, etc. The operating system does not need to store files in particular locations or maintain a particular file or folder structure because data objects are associated with projects, etc. by the tags and not by a file structure. Similarly, the operating system need not have codes and commands incorporated formally into the operating system because it associates the particular code with the program necessary to run the code, the GUI element asking the code to be run, etc. Data objects in the database can be any type of file. When data objects are uploaded into the database, tags are given to the object. Many tags are automatically provided, such as the file type, etc. Other tags are assigned to the object by the person who is loading the object into the database. The person loading the object into the data base can assign tags such as: the project or assignment to which the object belongs, the persons assigned to work on that project, the subject matter of the object, etc. The system utilizes contextual searching to manage and work with data objects stored in the database. Rather than looking for a particular file folder, a user would find needed files or data objects by linking to them from other associated data objects. The present invention is particularly suited for use in movie making and in digital animation. By way of example, a person working on animating a movie scene could work as follows. The person themself is saved as a data object within the database. The movies, scenes, etc. that the person is working on are tagged and thus associated with the person. In a user interface as shown herein, a person will typically have a screen portion identifying them and showing the projects that the person is working on. Clicking on a particular project will then show the different scenes, characters, or other data objects which are associated with that project. Each particular data object, such as a movie character, would also be associated with other related objects by similar tags. A character would be associated with other similar characters, related skin effects or lighting effects, etc. In this way, a person can easily and quickly find data objects which are necessary for working on a particular task. The person can easily utilize those related data objects (again, these being anyting such as skin, texture, or lighting attributes) to modify and work on a particular assignment. The operating system allows many different things to be tied to the different data objects. For example, budget amounts, scheduling and deadlines, associated employees and workers, necessary tasks, hours worked, number of revisions, total time from the project beginning, etc. can all be associated with an object. This allows a person to easily perform management and analysis tasks with any particular data object, whether that be the smaller individual objects such as a computer rendered scene or an entire movie. The operating system allows a person to easily program different graphical modules which become part of the user interface and which perform desired tasks. The operating system 2
has a number of preprogrammed dialog entry boxes or drop-down type selection boxes which a user can integrate into graphical program boxes. Each program can be constructed to analyze data and provide a desired result, locate desired data objects, etc. These graphical interface modules can even perform modifications to data objects such as applying textures and lighting to an animated character model. The operating system allows a person to quickly pull together the necessary animation characters, background models, surface characteristics, lighting, etc. into a scene to begin the final artistic work of making the scene ready. In modifying data objects or performing tasks, the operating system allows persons to create their own graphical interface modules by associating different commands and different pieces of programing code with the buttons or data entry boxes. The operating system does not need to have pieces of code, macros, etc. compiled into a particular type of system file, but can save each of these pieces of code individually in the database and link or associate these pieces of code with a desired graphical module. The operating system uses a API (application programming interface) which allows the operating system to control other pieces of software. The operating system thus acts as an intermediate between the user and the software, allowing the user to request data or modifications to a data object and controlling the fundamental software to implement those changes. Thus, where a user combines the animated characters and objects in a movie scene with available lighting sets for that animated scene, the operating system can automatically use a program such as MAYA to combine the 3D models and scene information with the lighting information and prepare the scene for the final artistic work to finish the scene. The tags and data hooks which can be associated with each data object in the database can also provide accounting and management information about the relevant project. The system can save information in the database to reflect the number of revisions to a project, the time spent on a project, the start/finish dates for the project, etc. This can allow a person to analyze the accounting information for the project and compare this to the budgeting or to forecast costs for a future project. Persons managing a project can easily see progress on the project and the time spent on the project and identify when difficulties are encountered and address these as they arise rather than trying to identify what when wrong after the project was completed. The system allows for easy reviewing and quality control of a project. With example to an animated movie scene, a person reviewing the scene can mark objects and place notes while viewing the scene. The system saves these notes as a data object, saving relevant information such as the time stamp of the scene, the part of the image which was problematic, and the notes or suggested changes. Because the system has tags identifying the person responsible for the scene, people working on the scene or the particular model, etc., the system can automatically send the revision notes to the necessary persons. This is a significant time savings compared to having a conventional meeting with the director and all project managers where a scene is run through sequentially to identify problems. 3
The operating system also facilitates communications between the people working on a particular scene, character, object, etc. The operating system allows a person to simply right click on a movie scene being watched, 3D model being viewed, data object being reviewed, etc. and a window opens showing the persons associated with that particular data object. Clicking or mouseing over any one of those persons will show the different ways to contact that person, such as by email, telephone, videoconferencing, etc. Clicking one of these options causes the system to automatically initiate the desired form of communications. The operating system, through its API, initiates the program necessary for that form of communication and facilitates the communication. The system provides a workflow solution where data storage, software, and computer resources are all shared. This saves considerable time and money. A central computing system is provides which includes a large number of processing units and data storage arrays. The system has all of the individual software programs used by the individual workers as well as all of the data objects stored thereon. Thus, an individual worker need not have the processing power and software necessary for each work task. The person s personal computer will run the operating system, organizing, managing and instructing changes to be made to the data objects stored on the system. When a person instructs changes to be made to a particular data object, the operating system runs the necessary piece of software on the computing system to make the changes and publishes the desired type of output file. This allows complex processing to be performed quickly as the cloud computing system has far more processing power available than the person s individual computer. Additionally, because all of the various software programs and data objects are stored on the cloud system, many individuals can collaborate on a project and the individual data objects can be shared between different persons, work groups and companies as is desired. An aspect of the invention is a User Interface ( UI ) that enables users to perform different functions from any location. The UI acts as interim OS between the user s data and the cloud. The cloud has not only stored data, but according to the present invention, also includes computing processors and individual application software. The UI enables users to perform their jobs easier and faster by minimizing the number of clicks that they have to make to perform their job. For example, you could from the same file send an email and check how much the project cost. Also, Users can access the UI from all over the world via internet connection. The UI is customizable based on each user s preference. Unlike the traditional method of executing different tasks, this UI will enable users to perform many different tasks from the same interface. For example, the user could view a pdf file, at the same time; the user could view the status of the project, send an email, make a call, and checks the cost see the example below. 4
The user could send message, instant message, make a voice call, video conference, or screen share from the same file. When you click on the people menu you instantly know who is involved with the process and at what stage. If you need to communicate with them you can simply choose one of the many options open to you through the system: Message, Instant Message, Voice Call, Video Conference, and Screen Share. We will utilize popular software such as skype for some of these features (like voice and video chat) The user can create a widget to perform certain function. The widgets are the center block to the UI interface. Widgets are very necessary to the software behind the Factory Symbiotic Operatic System. They form the building blocks for our web interfaces. A Widget is defined as any piece of visual programming that can be reused easily in other pieces of software. Here s some examples of widgets, both as UI elements and as right click menus. 5
The Info widget displays user defined meta tags for the selected object in our screen grab the user had happened to clicked on the latest published look dev for king_kong the information is displayed here. Managers have the ability to change which information is displayed at any time and publish those changes to their department or the entire company. The key is that flexibility is built in to the software to accommodate almost any workflow. The Search Filter Widget allows users to customize which data is displayed in the search window. This is very useful when you re trying to find all of the submissions from one department for the day 6
The UI will link this new widget with data from the database. Unlike traditional methods, the user will be only working with display and search interfaces rather than file browsing. The UI is novel in that it enables all users to create widgets to perform their jobs easier and faster without any knowledge of programing or programing languages. After creating the widget, the UI will convert the widget into programing codes that is compatible with the UI and link the data with new widget. Further, the UI will enable users to visually interface with the database and create widgets based on their needs. The user could grant access to his or her widgets anywhere. Software Breakdown: Step 1: ingestion Step 2: Contextual Search & Navigation Contextual search is the building block for all of our tools. This does not mean that a user will have to use traditional search methods to find everything, it means that our user interfaces (UI) are purpose built to display the information that matters. 7
UI Creation Software: Our Database Driven operating system provides ways for us to program something once and reuse it over and over again. As a companion to the robust database we are creating a graphical programming platform that will be able to output user interfaces into various programming languages to be used in different applications. Below is an example of how that graphical programming interface might look. In the above example a User Data Object is connected to the input of a line edit widget. What that means is when someone types in that line edit our search engines will be confining their search to data ojbects of type user. After the user is selected from the list, or typed the line edit widget will return a name, both on the screen and in the program itself. We have the output from the line edit connectd to the input of the film combo box. This will display objects of type film through the filter of the input which in this case will be the user information. We could have just as easily connected a widget that returned studio information to the 8
widget and have it show us films filtered by the studio that produced it. We follow this logic down to the output of the entire widget and see that it either returns a scene or a shot. This will be valuable to other widgets that are looking for a shot or scene input. There are many many ways this information can be useful. We ll get into some examples in a second, but I ll explain the component parts before then. Explanation of Component Parts: 1. Data Object: There will be many types of data ojbects within our system. All will be similar but have a different function and different attributes. 2. Widget: Can be as small as a combo box or line edit or as complex as the widget we ve created. In essence a widget takes some output from a data object or another widget and filters its results accordingly 3. Input/Output nodes: In our graphical mode it will be necessary to be able to define inputs and outputs of different widgets we ll do this in a similar way to how is defined abve. Examples: 9
The above example shows the dialog that may be used in the instance of building a shot for a film. 1. Artists opens the dialog and sees the shots that are assigned to him in the left panel 2. Clicking on one of those shots displays the assets involved in the shot. He will be importing all of them into his scene. 3. In the upper right hand corner we see some kind of light rig widget. This shows him light rigs that are published according to the search criteria he puts in 4. In the bottom right hand corner we see a render layer widget this widget takes geometry and light rigs that the user drags/drops and organizes them into render layers. 5. Having these custom widgets as a whole allows the user to create render layers with existing lighting files and assemble it all with the touch of a button, everything but the render layer widget simply shows items from the database in a way that makes sense contextually to the lighting artist. The render layer widget will have some custom python scripts attached to it that create it within Maya 6. Having information laid out this way turns a job that could take all day to one that takes seconds. (there were shots on Avatar that literally took about 6-8 hours just to get set up) Simple Model Import Example The example Below shows how models might be imported into scenes or edited. 1. There is a widget on the left that shows the assignments an artist has, 2. when a model is selected the central widget shows the different revisions. 3. When one of those revisions is selected a preview window shows a turntable of that model 4. The latest publish information for that model is shown as well so that the person importing it knows what the difference from one model to the next is. 5. These are all entries in the database and our robust widget creation and reuse software makes it easy to get at that information. 6. If for example we found that modellers often wanted to see previs for the models they were working on to understand context we could easily add a widget that accepts a data object as input and returns the previs shots where it is used. That widget could be attached to the UI by just about anyone and published for everyone to use. 7. The end result is that the modeller or whoever needed that model is able to import it easily and understand as much information as they need to about it.. 10
11
Executive Summary of Invention I. Major Aspects of Invention a. Give any user access to millions of dollars of computer hardware & software in an easy to use workflow b. New Symboitic Operatic System works within any existing operating system, but acts as an interim OS between your data and the cloud. c. Provides pathways from locally installed software to our databases on the cloud d. Introduces the idea of contextual ingestion & search, which replaces traditional file browsing methods used in all major operating systems. e. Contextual organization of data based on teams, jobs, projects, or any other qualifier. f. UI creation and management: If you can sketch out a User Interface with our design tools, you can be a programmer. Our own programmers use a sophisticated system of widgets to visually program all of our user interfaces. We will make this technology available to all of our users. They will be able to visually interface with the aspects of the database they need for their specific use cases and will be able to make these interfaces available to everyone on their team even the world. g. Introduces the ability for teams all over the world working with highly complex data to instantly share that data since everything in our systems that goes through a publish stage will be checked for compliance. II. Reasons software & hardware are different from existing hardware, software and systems a. We move completely away from the file browser for opening, saving, importing, and exporting data. Our system uses a contextual display and search paradigm that is completely unique. b. We have systems that allow non-programmers to program sophisticated User Interfaces c. We have systems that allow non-programmers to publish useful user interfaces and workflows to the entire community d. Our systems allow instant access to data and complete flexibility with how that data is used, even down to allowing the data to interact with itself. e. Our data center allows anyone with an internet connection to connect to purpose built visual effects & animation machines and thousands of rendering machines to support them. f. Our data center combined with our software systems allow for easy collaboration with hundreds of freelancers throughout the world. This same collaboration can be had between companies. 12
Flowcharts of the building blocks of the software 13
Software Path for Employees 14
Software Path for Managers 15
Software Breakdown: Step 1: ingestion 16
Step 2: Contextual Search & Navigation Contextual search is the building block for all of our tools. This does not mean that a user will have to use traditional search methods to find everything, it means that our user interfaces (UI) are purpose built to display the information that matters. This is a broad overview of the basic search window 17
Factory Widgets: Widgets are very necessary to the software behind the Factory Symbiotic Operatic System. They form the building blocks for our web interfaces as well as our QT interfaces. A Widget is defined as any piece of visual programming that can be reused easily in other pieces of software. Here s some examples of widgets, both as UI elements and as right click menus. The Info widget displays user defined meta tags for the selected object in our screen grab the user had happened to clicked on the latest published look dev for king_kong the information is displayed here. Managers have the ability to change which information is displayed at any time and publish those changes to their department or the entire company. The key is that flexibility is built in to the software to accommodate almost any workflow. The Search Filter Widget allows users to customize which data is displayed in the search window. This is very useful when you re trying to find all of the submissions from one department for the day 18
Many of the icons are right-clickable the people icon is a good one to show some extremely efficient functionality. When you click on the people menu you instantly know who is involved with the process and at what stage. If you need to communicate with them you can simply choose one of the many options open to you through the system: Message, Instant Message, Voice Call, Video Conference, and Screen Share. We will utilize popular software such as skype for some of theses features (like voice and video chat) Any Communication through these menus is published and connected to the asset so artists and supervisors can go back and review notes later. Often when watching a submission, or searching for an asset a user will want to revisit the file that created it. Our contextual search on the back end makes this possible through right click menus or through custom UI widgets. The idea is that because our databases track all relationships and our UI creation tools make all of that information available we can create context for just about anything even other software 19
The death of the Open & Save Menu Factory Contextual Workflows. Anyone who has worked on a computer in the last 20 years knows how hard it can often be to find a file. Our contextual browsers put an end to that. Asset Panel: This gives me a few options for how I want to view data. As shown I m seeing my assignments I ve selected King Kong Detail Panel: This panel is showing me contextual assets based on my assignments and my department. Shader writers need access to the models and textures, as well as the shading library which is large enough to have it s own widget to the right The Shading Widget creates complex shading networks for any render engine based on the settings in the specular object dragged over from the Specular widget. The Shading widget shows which file inputs the network is looking for and the user can either Auto Populate the list or drag and drop textures from the Detail Panel. There is nothing like this on the market today. The Specular Widget shows all the available shading templates in the libarary this will be incredibly vast and allow artists to set up complex shader networks in seconds simply by dragging to the shading widget 20