Repathing your assets in .max files without 3ds Max

loocas | 3ds Max,dotNET,maxscript,Python,technical | Friday, August 3rd, 2012

Asset Tracking

Ever since I read this blog post on the Area, I was intrigued to get this working in an IronPython environment (that’s pretty much all I know, in the “serious programming” area). Unfortunately for me, the article mentions C++, OLE and COM. Which are my least favourite technical subjects.

So, now when I finally really needed this solution (more on that some time later), I had to ask on the Autodesk forums.

Luckily I got an answer. But first off, huge thank you goes to Larry Minton, an Autodesk Engineer, without whom I wouldn’t have been able to get this thing going.

Now, about the problem. If your facility has a render farm and you happen to work off of your local storage, you have to point your assets to a UNC path where they’re stored so that all the machines on the network can find and load them when rendering. There are many ways of doing this and usually your pipeline TDs had figured this one out prior you even starting any work on the project. :) Unfortunately for me, I’m the only pipeline TD here. :D So I had to figure out a way of re-pathing my assets in 3ds Max scene files prior to rendering.

Up until now I’ve been dealing with this via MAXScript and Deadline. I wrote a handy dandy PostLoad script that gets executed right after a Max scene file gets loaded up, then the renderer fires off and renders correctly. This has worked flawlessly for me and probably will work forever. However, you always have to supply this script along every submitted file. So, this gets a little more problematic with more complex PostLoad scripts that you also need to run occassionally for specific jobs/shots.

Then, not very long ago, I bumped into these two commands in MAXScript:

getMAXFileAssetMetadata <filename>

and

setMAXFileAssetMetadata <filename> <array of AssetMetadata_StructDef instances>


These commands will allow you to source a special stream of data off of a given .max file and will allow you to overwrite it with whatever you supply the methods with. This is a perfect solution for a situation where you have to batch modify a ton of .max files in order to prepare them for the network rendering. Unfortunately, though, you have to have 3ds Max running in order to perform MAXScript operations. This is a bit prohibiting on a server side, where you usually don’t have a 3ds Max license available or you don’t want to use up a single seat of floating licenses.

Then there is the OLE stream. It has been possible, since 3ds Max 2010, to get such data out of the .max files without running 3ds Max and thus without needing a single license of Max to perform such pipeline tasks. It isn’t as trivial as the MAXScript approach, though, but definitely doable.

The problem with .NET languages, though, is that this was designed more for C++, than C# or even IronPython, or Python for that matter. But, luckily, there is a library, OpenMCDF, that allows you to manipulate OLE data streams in files. It is very simple to use the OpenMCDF objects and methods and works perfectly fine under IronPython (which is my case of using it). And so, the last piece of puzzle was to figure out how to access the meta data asset stream and, mainly, how to modify it – effectively re-pathing the assets outside of 3ds Max.

First of all, download and unpack the OpenMCDF assembly into the IronPython\DLLs folder.

Now, in IronPython, in order to get to the data stream, you have to Compound the .max file using the OpenMcdf.CompoundFile() method. If you want to modify it, supply a UpdateMode parameter and a couple of boolean variables denoting whether you want to recycle sectors and or erase free sectors.

from OpenMcdf import *
cFile = CompoundFile(r"pathToMaxFile.max")


You then have to access the RootStorage and its streams. This gets a little tricky. I had to look this up in the C++ SDK (kindly provided by Larry). The stream name you’re looking for is called FileAssetMetaData2, so with this in mind:

theStream = cFile.RootStorage.GetStream("FileAssetMetaData2")
buffer = theStream.GetData()


Now here it becomes a little cumbersome for high-level programmers like myself. Essentially, what we get is an array object full of Byte objects carrying data, like so:

Array[Byte]((<System.Byte object at 0x00000000000023BF [92]>,
<System.Byte object at 0x00000000000023C0 [108]>,
<System.Byte object at 0x00000000000023C1 [59]>))


Let’s take a look at the MAXScript output, to put things into perspective. If you run this command:

getMAXFileAssetMetadata @"pathToMaxFile.max"

you get something like this:

#((AssetMetadata_StructDef assetId:"{923B6C5C-1ACC-48D7-A879-FF472DC61589}"
filename:"D:\_SOME_PATH_\TEST\BEAUTY.exr" type:#bitmap))


It’s an array of AssetMetadata_StructDef objects. More importantly, the data it contains is: asset ID, what type of an asset it is and the file path, which is what we’re interested in modifying. The ID and type is not to be played with! You can get pretty nasty Max crashes if you do something wrong with these. Believe me. :)

Now that we know what data we’re looking for, we need to make some order out of the Byte array we received from the stream. To make things even more complicated, the Byte array contains not only the data we want, but also some descriptors (not really the best name, but at least that’s what I think of them). The descriptors tell you how long the next Byte stream is in order to retreive correct data. Here’s how the data is structured in the stream in order to make something out of it (again, huge thanks go to Larry for this):

  • 16 Bytes – the AssetID (don’t modify this!)
  • 4 Bytes – the length of the next Byte stream that contains the Asset Type name
  • (len+1)*2 Bytes – the amount of Bytes you need to read in order to retreive the Asset Type name (UTF-16)
  • 4 Bytes – the length of the next Byte stream that contains the Asset file path string
  • (len+1)*2 Bytes – the amount of Bytes you need to read in order to retreive the Asset file path string (UTF-16)

So, that’s basically it. This recipe loops for as long as there are available Bytes in the stream. This, of course, means that if you want to change the file path string, you will have to update the preceding 4 Bytes denoting the file path string’s length, which is a pain in the ass, if you ask me.

I’ve figured out a pretty dirty workaround, which I’m presenting here. I’ll present a way of reading the file paths, so that you can use them in your asset management or production management software, for example.

The process is as follows:

  1. Get the buffer Byte Array object
  2. Decode it using the UTF-16 table into a unicode string
  3. Split the string by using a special “\x00″ delimiter character
  4. The resulting list object stores all the data you need in a human-readable form (well, that applies only to the file path string and the asset type string)
  5. Filter out only every fourth item in the list, which is the asset file path

Unfortunately, there is no way (none that I know of) to tell what is the asset file path associated with. All you know is whether it is a Bitmap, or a Cache etc… but there is no way to tell whether it is a render element save file path, or an actual texture path etc.

But still, for repathing purposes, it’s all you need! Here’s the IronPython code showing the above process in action:

from System.Text import Encoding
assetFilePaths = (Encoding.Unicode.GetString(buffer)).split("\x00")[3::4]


That’s it! The resulting list contains all the Asset file path strings that are contained in the .max file you sourced. With a bit of imagination and some more programming, you can start re-pathing the file paths using these methods. Also make sure to read the OpenMCDF documentation for all the available methods and object properties.

Get the Flash Player to see this content.

I’ll post my scripts here as soon as they’re polished and thoroughly tested. However, I’m very thrilled to finally have a way of modifying asset paths in my .max files without having to run Max first, it’s such a time saver and it’s very helpful in a modern pipeline.

29 Comments »

  1. Thank you for the in-depth and thorough research!

    Comment by Sergo — August 3, 2012 @ 20:08

  2. Wow, super useful. Thank you!

    Comment by deko — August 5, 2012 @ 21:24

  3. Good stuff.
    However, I would urge you to consider syncing all your textures to a cloud storage provider like S3, then you never have to reside from any location. Also, it’s backed up, with potential of a perm VPN tunnel and static IP if required. Also, you will want to do this if you’re going to maximise your cloud rendering setup in Deadline v6 :-)
    Asperasoft for sync is too expensive, so you could use rsync. A Windows version is available if req.
    Mike

    Comment by Mike Owen — August 6, 2012 @ 01:29

  4. Hey, Mike, thanks for the tip.

    I’m a bit hesitant of syncing all the textures etc… to a remote location behind my internet connection. Sure I can get a 100mbit line at my company, but I’m at a 10mbit line at home, which isn’t really that great for syncing textures.

    Also, we are not talking about textures only, but also caches and other external files. But especially caches are just huge!

    Or am I missing your point here and you meant something else?

    Comment by loocas — August 30, 2012 @ 02:45

  5. any updates with this one.. how is the writing back to the process…

    Comment by hubert — September 5, 2012 @ 07:29

  6. Well, it works. Just follow my explanation. I’ve tested it successfully and I’m still about to integrate it into my pipeline. :)

    Comment by loocas — October 3, 2012 @ 17:15

  7. can I see your final script for this one, I created a program but I have a problem on the saving phase.

    Comment by hubert — October 10, 2012 @ 08:06

  8. Good stuff.
    When release the final script ?

    Comment by WTV3D — December 6, 2012 @ 15:54

  9. Hi, WTV3D. Unfortunately I haven’t had the time to work on that, but I’m planning to do it over the holidays. :) The final will, however, be just a Python script that you’ll pass in what you want to replace and what you want to replace it with and the script will take care of that for you. But, again, unfrotunately, there isn’t any way (none that I know of) to determine what is and actual render output and what are texture, cache, xref etc… inputs. So, everything will get repathed to one single location, unfortunately.

    Comment by loocas — December 10, 2012 @ 16:37

  10. I completed mine.. in C#.. but unfortunately.. am stucked with same problem same as loocas…

    Comment by hubert — December 13, 2012 @ 08:33

  11. if you want.. I can hand over the completed program.. there is no manual though…

    Comment by hubert — December 13, 2012 @ 08:35

  12. I’ve done some deeper research into the max file format, for replacing asset paths in older max files, in case you’re interested. It’s at http://blog.kaetemi.be/tag/max_file_format :)

    Comment by Kaetemi — January 17, 2013 @ 11:25

  13. Thanks a lot for the link, Kaetemi! I’ll look into it.

    Comment by loocas — January 20, 2013 @ 13:26

  14. Great post, although like others i’m running into problems re-saving the file once I’ve modified the file paths. Is there any possiblity you could post this part of the source code?

    Thanks,

    Comment by Tom — May 3, 2013 @ 15:30

  15. Hello loocas !
    I wanted to ask if you can share your Python script i see in the short movie you shown.
    Can you please mail it to me ?
    Thanks in advance,
    Oleg.

    Comment by Oleg — June 11, 2013 @ 23:03

  16. Hey, Oleg, sorry, I can’t share the code, yet, but you can get the free exe app that does all the hard work for you here: http://blog.duber.cz/3ds-max/repathing-3ds-max-assets-externally-command-line-utility

    Comment by loocas — June 17, 2013 @ 09:23

  17. Hello, thanks for sharing that utility !
    I’ve managed to get all the assets from .max file in short C# code , but how to change the paths it is still a mystery for me.. Can you share your knowledge in that ? You said that it is a pain in the ass and i believe you !
    Thanks,
    Oleg.

    Comment by Oleg — July 24, 2013 @ 21:49

  18. Hey, Oleg, once you get the stream data from the .max file, you need to decipher them. I don’t remember where I got the “formula” for that, possibly the SDK documentation, but you have to load up the bytes, read them sequentially and based on that decide how many bytes you need to read next in order to distinguish between the various data types stored in the stream. Sounds complicated, but once you wrap your head around it, it’s rather easy. But there are some issues with this approach to which I haven’t found a solution. Looks more like an undocumented behavior to me, because I’m doing exactly what the documentation suggests, but there are still problems with the correct data being written to the stream, for some reason.

    Comment by loocas — July 25, 2013 @ 10:34

  19. Thanks for the answer and i think i understand you.
    I’ve managed to get the stream and decode it and take out the assets. I just want to make simple utility which is crawling given paths and searching for missing assets , after that writing them in 3dsmax file without opening it. This could be very useful in checking files before opening. You can even make a simple collector based on that.

    Comment by Oleg — July 25, 2013 @ 21:31

  20. Hey, Oleg, you can use my cmd line utility for that as well :) http://blog.duber.cz/3ds-max/repathing-3ds-max-assets-externally-command-line-utility

    Comment by loocas — August 1, 2013 @ 15:43

  21. Hello
    Thanks, that’s a great thing ! I would love to use it though there is one thing which i think it lacks – i cannot point what type of asset where i want to repath. For example vrmeshes i want to place in VRMesh folder , textures in Maps , x-refs in Xref , and etc.
    Maybe you plan to update this utility ? It will be a really cool thing to use !
    Thanks for sharing !
    Oleg.

    Comment by Oleg — August 5, 2013 @ 14:20

  22. Hey, Oleg, unfortunately this cannot be done externally. Or at least I haven’t found a way to distinguish what asset is what. :( I mean, you can do that by sourcing the AssetType data, but there are only a few asset types available. For example there is no difference between Render Elements and Render Outputs. Etc…

    Comment by loocas — August 8, 2013 @ 00:56

  23. Hey !
    Yes it’s exactly like that , but that is enough for now i think , other stuff i am checking by extension of the file. By the way, have you tested your command line utility completely , it does not corrupt any .max file ?
    Thanks,
    Oleg.

    Comment by Oleg — August 8, 2013 @ 17:25

  24. Hey, Oleg, I have tested the cmd utility, not on Max 2014 scenes, though. But earlier versions didn’t show any corruption. I ran into issues with the paths not being correctly re-pathed, for some, to me unknown, reason and I can’t get anyone from Autodesk to help me with this, so… yeah, that’s about it. :)

    Comment by loocas — August 12, 2013 @ 12:36

  25. Great, thanks, loocas, i already try to use it , seems it is working pretty nice ! As i know for 2014 max they have changed stream name to “FileAssetMetaData3″ , isn’t it ?

    I am drilling the documentations towards repathing , actually i want to make an opensource library with all the functions necessary to work with .max files without 3dsmax. There is more interesting stuff in other streams , for example i can get name of all of the objects , materials , username of last user who modified file , name of PC , render resolution and much more. So i want to make separate function in library for all of these.

    Can you please tell me if this is correct way of thinking for re-pathing assets :

    1. finding a string with a path in .max file stream
    2. make an “A” byte array , in which we save found path from stream with all necessary delimiters and “00″ symbols between
    4. in main stream “B” byte array with all of the assets we find this “A” array.
    5. we convert new string to byte array “C” with all the delimiters and “00″ chars between
    6. we cut A-B and insert C from starting index of B in A

    If you have a bit of time , can you please tell me if this “algorithm” is correct ?
    Thanks so much for your help!

    Comment by Oleg — August 15, 2013 @ 17:38

  26. Hey Lukas,

    thanks for the in-depth research on that. I am not using IronPython tho and am hitting a few walls with encodings. It seems that the IDs do not always decode properly usng UTF16? Can you confirm that? I can decode fine if i set errors to ignore, but i guess then i cannot re-assemble a proper bytearray to write back into the file.

    Any pointers would be greatly appreciated!

    Thorsten

    Comment by Thorsten — November 22, 2013 @ 12:09

  27. Hey, Oleg, unfortunately I’m not familiar with all the streams available in the .max files. I had to get a lot of help from the Autodesk devs for my re-pathing tools.

    Thorsten, I used UTF-8 if I remember correctly and then filtered out special bytes from the string. They were only used where there was a special character or a space. I then re-assembled the string this way and it worked without issues.

    Just don’t mess up the encoding and decoding. :D

    Comment by loocas — November 27, 2013 @ 10:46

  28. Hi,
    I have been trying to write this for max 2014, with “FileAssetMetaData3″ my only Problem is i dont know how to save the new Datastream, (I cant see the code in the video ;D) , if anybody could help

    Comment by oliteto — July 8, 2015 @ 17:19

  29. Hey, oliteto, you can find everything you need in the OpenMCDF docs. :) Good luck.

    Comment by loocas — July 26, 2015 @ 15:25

RSS feed for comments on this post. TrackBack URI

Leave a comment

Powered by WordPress | Theme by Roy Tanck