Nearly two and a half years ago I've started a project called oyProjectManager. It was a simple pipeline tool written in Python and it was using PyQt4 for UI. This tool helped us a lot to sort things out in this past two years. Before, it was a chaos to manage files and asset versions.
When I first started this project I was not very experienced with OOP, design patterns and abstraction concepts like MTV or MVC or methodologies like TDD etc. So I've ended up with a useful tool, which is designed and programmed not in the best way.
Seven months ago, I was trying to write an automated backup script, which simply finds all the nuke files in the project folder structure and copies all the inputs (by using rsync) to the filtered backup folder in the server.
Then I've realised that I can not go with the system any more because it was very hard to get some data which will be very simple if the system was designed correctly. I was trying to avoid adding big design changes to the system for a long time because I was already writing a new complete Production Asset Management System (ProdAM) for the last 2 years called Stalker, which is another open source project and deserves a couple of posts in this blog, and any code I will add to oyProjectManager will be meaningless cause I've already correctly designed Stalker and I am going to use it in the near future. But it seems that I still had time before starting to use Stalker, so I've started to write oyProjectManager from scratch this time by following TDD practices. I've fixed a lot of things, misconceptions, bad code led by bad design. But again I couldn't go too far (didn't add any Project Management capabilities).
Previous versions of oyProjectManager were using project specific XML files to store simple data and it was storing the metadata information in file names, thus it was limited in terms of how much and what kind of data will be stored in the system. The file naming convention was static (constant, fixed) and limited, and the resultant project structure was firmly tied to this naming convention and because of that the project structure was bad, the assets were spread around the whole project structure, models were there, rigs here, shots there etc (following Maya's own project structure). It was dirty. Also there was a misconception about what an asset is: I was calling every single version file an Asset which is obviously is false.
After v0.2.0 of oyProjectManager all the project structure, the placement of individual version files and the name of these files are all defined by Jinja2 templates. So you can easily create a complex project structure. And because the metadata is hold in a proper database (it can be any type of database that SQLAlchemy supports, the system is heavily tested for SQLite3 and loosely tested for PostgreSQL) it is much more easier to query the data and the system works lighting fast when compared to the previous implementation. For the last 4 months we were using oyProjectManager in our studio and everybody was very happy about it.
Anyway, the new version is up an running. And if you want to use it in your own facility visit oyProjectManager's PyPI page to download the source code and here is the documentation.
The documentation doesn't have a section which is explaining the installation yet. But it is simply as follows:
Automated install:
As a side note I need to state that even though I've run the installation code above in Linux we are using mostly Windows and OSX based workstations, there was no problem running SQLAlchemy nor Jinja2 in those systems. May be they were not able to use the C extensions those are supplied with SQLAlchemy and Jinja2 but the general speed is more than ok already.
Manual install:
Configuration
I hope you will like oyProjectManager and how it helps you to work more easily and organize your files, and drop me a line if you are having problems installing it or you have suggestions for the next releases.
I will do another post showing the scripting part of oyProjectManager and how to create data easily by using plain Python.
And again the real bomb is Stalker, wait for it.
When I first started this project I was not very experienced with OOP, design patterns and abstraction concepts like MTV or MVC or methodologies like TDD etc. So I've ended up with a useful tool, which is designed and programmed not in the best way.
Seven months ago, I was trying to write an automated backup script, which simply finds all the nuke files in the project folder structure and copies all the inputs (by using rsync) to the filtered backup folder in the server.
Then I've realised that I can not go with the system any more because it was very hard to get some data which will be very simple if the system was designed correctly. I was trying to avoid adding big design changes to the system for a long time because I was already writing a new complete Production Asset Management System (ProdAM) for the last 2 years called Stalker, which is another open source project and deserves a couple of posts in this blog, and any code I will add to oyProjectManager will be meaningless cause I've already correctly designed Stalker and I am going to use it in the near future. But it seems that I still had time before starting to use Stalker, so I've started to write oyProjectManager from scratch this time by following TDD practices. I've fixed a lot of things, misconceptions, bad code led by bad design. But again I couldn't go too far (didn't add any Project Management capabilities).
Previous versions of oyProjectManager were using project specific XML files to store simple data and it was storing the metadata information in file names, thus it was limited in terms of how much and what kind of data will be stored in the system. The file naming convention was static (constant, fixed) and limited, and the resultant project structure was firmly tied to this naming convention and because of that the project structure was bad, the assets were spread around the whole project structure, models were there, rigs here, shots there etc (following Maya's own project structure). It was dirty. Also there was a misconception about what an asset is: I was calling every single version file an Asset which is obviously is false.
After v0.2.0 of oyProjectManager all the project structure, the placement of individual version files and the name of these files are all defined by Jinja2 templates. So you can easily create a complex project structure. And because the metadata is hold in a proper database (it can be any type of database that SQLAlchemy supports, the system is heavily tested for SQLite3 and loosely tested for PostgreSQL) it is much more easier to query the data and the system works lighting fast when compared to the previous implementation. For the last 4 months we were using oyProjectManager in our studio and everybody was very happy about it.
Anyway, the new version is up an running. And if you want to use it in your own facility visit oyProjectManager's PyPI page to download the source code and here is the documentation.
The documentation doesn't have a section which is explaining the installation yet. But it is simply as follows:
Automated install:
- Download and install Python 2.6 (that is because most of the programs are still using 2.6)
- Download and install setup tools
- Run:
easy_install-2.6 oyProjectManager
This will install oyProjectManager to your pythons dist-packages folder with all its dependencies - Copy all the packages to a shared folder, which is visible to other computers in your studio
- Add this shared folder path to PYTHONPATH environment variable to let all the applications to see the modules.
As a side note I need to state that even though I've run the installation code above in Linux we are using mostly Windows and OSX based workstations, there was no problem running SQLAlchemy nor Jinja2 in those systems. May be they were not able to use the C extensions those are supplied with SQLAlchemy and Jinja2 but the general speed is more than ok already.
Manual install:
- Download oyProjectManager and extract it and copy the oyProjectManager folder to a shared folder which all of your workstations are able to see
- Download SQLAlchemy and again extract it to a shared folder
- Do the same for Jinja2
- Add this shared folder path to PYTHONPATH environment variable to let all the applications to see the modules.
- If you don't plan to use another interface library and do all that nasty coding by your self I also strongly suggest you to install PyQt4 or PySide for all to your applications (Maya, Nuke, Houdini etc.) and use the UI coming with oyProjectManager. Because installing PyQt4 and PySide is a little bit involving, I'm leaving you by your self about how to install it. For a quick tip search for MyQt4 for Maya, and you can use my post about compiling PyQt4.
- Newer versions of Nuke comes with PySide installed, to be able to use PySide instead of PyQt4, add a new environment variable to Nuke's startup script (init.py) called "PREFERRED_QT_MODULE" and assign the value "PySide"
Configuration
- oyProjectManager needs two environment variables to be defined, REPO and OYPROJECTMANAGER_PATH
REPO: set to a path where you want to save your projects. This is typically a mapped drive like "M:\Projects" for Windows and or a mount like "/mnt/M/Projects" in Linux and OSX.
OYPROJECTMANAGER_PATH: The path that contains the "config.py" file, which is the configuration file for the system (I also suggest to use this folder to place your SQLite3 database file)
See the documentation of the config file
for the details about config.py. - You should be ready to go, run maya, be sure that maya is able to see the directory you have copied the modules in to and run:
from oyProjectManager.environments import mayaEnv from oyProjectManager.ui import version_creator mEnv = mayaEnv.Maya() version_creator.UI(mEnv)
When you run the above script, you should be able to see the version_creator UI. Which as the name suggests creates new versions for some assets or shots.
The interface should explain itself. But because you didn't created any Project yet you will not be able to create any assets or shots or versions. - To bring up the project_manager interface use the following code:
from oyProjectManager.ui import project_manager project_manager.UI()
This will show up a new interface where you can create and edit Projects, Sequences and Shots.
You can also run the project_manager from the regular Python shell if you have successfully installed oyProjectManager, SQLAlchemy and Jinja2 to your Python.
I hope you will like oyProjectManager and how it helps you to work more easily and organize your files, and drop me a line if you are having problems installing it or you have suggestions for the next releases.
I will do another post showing the scripting part of oyProjectManager and how to create data easily by using plain Python.
And again the real bomb is Stalker, wait for it.
Comments