Advanced Production (Summary)

Despite returning to Semester 2 later than expected (due to illness), I have made excellent progress, with major breakthroughs in overcoming many of the challenges my project was facing.

My programme of study takes a more visualisation based route, and I am working with the University mathematics division to visualise numerical data (gained from their mathematical models) which represents the growth and development of cancer cells (or solid tumour growth).

Working with numerical data has created it's own set of challenges, as Maya cannot read or interpret this in it's off-the-shelf form. As a result of this problem, I have developed skills in using Maya's own scripting language, MEL (Maya Embedded Language) and also a second scripting language (used more widely) called Python. Although this has been a difficult and time-consuming process (several months of learning, which is still ongoing) I have been able to harness this new understanding and create custom tools which can be used to read the mathematical data and create appropriate 3D geometry.

Throughout learning each scripting language, I referred primarily to online sources, and the 'official' documentation provided. I also used the Digital Tutors service, which was both informative and relevant, helping me overcome the hurdle of knowing where to start, and what to focus on.

Through creating my own custom tools, I was able to work with the numerical data-sets provided. Starting with simple tests, I was (eventually) able to construct fully animated scenes containing simple spheres, representing the individual cancer cells, and although time-consuming to process, these results opened the way for more advanced development of the data and it's visual attributes.

I spent some time experimenting with Maya's render layers, and compositing passes using Nuke. This was as a response to feedback given last semester - not everything has to be done in 3D, as it can often be quicker and easier to complete some work in 3D, but fine-tune the details in post-production. This helped me develop possible ideas for the look of cells, when considering texture on a microscopic basis, and I also experimented with 'faking' depth-of-field techniques to enhance this.

Although creating spheres was great, I wanted to push my skills further. I spent time working with RealFlow, Cinema 4D and Houdini - three great 3D packages. With the exception of RealFlow, these were entirely new to me, and although Cinema 4D was fairly straight-forward, Houdini had a very steep learning curve. Using my Python scripting skills, I created a large-scale metaball system, equivalent to the spheres created in Maya - this was a challenging task, forcing me to solve several problems, whilst trying to script in a language mostly new to myself. The result of this technical experimentation allowed me to create an interesting almost-organic 3D structure, which moved and behaved as one surface, instead of 1068 individual cells.

After completing these Python related tasks, I returned to Maya to work with additional data-sets, still showing the growth and development of cancer cells, but in a different format. This data would require not only animated movement of cells, but changing colours, and eventually, complex density 'clouds'.

The difficulty at this stage in my work, was the size of the data-sets... one of which contains over 30 million lines of information (a text file weighing in at around 800mb). Despite breaking this into smaller chunks of data, it was still computationally intense, and difficult to work with (often crashing high speed computers). I spent time streamlining and optimising my scripts, and the way they handled the data - as an example, my oxygen density script was originally taking around 350 seconds per frame to process, whereas now it takes around 90 seconds, about a quarter of the time. As I have learned more about MEL, and gained a better understanding of the language, I have been constantly improving and refining my scripts, to ensure the best efficiency when working with large data sets.

I am currently working with new additional data-sets, which again have constantly required me to think and problem solve... now that my scripts are optimised, I am confident that my scripting abilities are suited to a wide range of data types, across two very different scripting languages. This allows me to tackle upcoming problems with new knowledge, based on the experiences I have had over the last ~3 months, giving me a definite headstart.

In summary, my practice-based learning has developed the following;

• Scripting - both MEL and Python, for simple and complex tasks (including 3D visualisation)
• Improved software knowledge - Maya, RealFlow, Nuke
• New software knowledge - Cinema 4D, Houdini
• Advanced 3D skills development - particles/dynamics, data input/output, script optimisation, rendering
• Experience in 2D compositing with 3D-based image sequences

Due to the sensitive nature of the mathematical data I am working with (and as the source data is effectively the result of unpublished research) I am unable to post a large amount of my visual work online. However, this sensitive material will be made privately available during course presentations, and to module assessors as necessary.

Visible Progress!

Over the last couple of weeks, my role in the Going Live project increased significantly, and then stopped completely. All of the animated shots had lighting added, and then I added the render layers/passes and started feeding completed shots through the render farm (which was considerably faster than I expected it to be). Sound effects and music were then added by the sound team, creating our finished advert.

It took a long time to get there, and there were problems along the way, but I learned a lot (particularly about rendering and compositing) and I am glad we all got there in the end! Next week, we are due to meet with the company in their London studio and present our finished project - hopefully the feedback will be good!

As for my cell visualisation work, this has been making good progress since my role in Going Live has lessened.

The first data-set I was working with, which represented cells and fibres in 2D space, now has fully working scripts, which are streamlined to work efficiently (or to actually work at all!). I am currently awaiting feedback on the video outcome of this work, so that I can decide where to take this next.

The other data-sets (involving cells, blood vessels, and oxygen density maps) have made even better progress. Again, after optimising my MEL scripts, the amount of data (several million lines of information) has become manageable, although time-consuming to process. I am currently part of the way through 'translating' this data into Maya's 3D environment.

An example render from the cells file can be seen below. This example frame is approximately two thirds of the way through the cell data, and incorporates some 'noise' on the cell surfaces to break up the uniformity (an idea suggested by the mathematician who provided the data);

As for the oxygen density, I decided to continue using a single polygonal plane for this, with grid points in the data having a matching vertex on the 3D geometry. The data then lifts/lowers each grid point/vertex between 0 and 1, where 1 is the most dense area of the oxygen 'clouds'.

The 'look' of these clouds is then controlled using one of two shaders.

Shader 1 ("Clouds") is coloured white, and uses a vertically-aligned ramp shader for it's transparency value, where 0 is fully transparent and 1 is fully visible. This means that as points on the vertex grid are changed in the Y-axis, their transparency is also changed (as they are moved higher, they become more visible).

Shader 2 ("Bands") expands upon this idea, and uses a second ramp for the colour (from blue to red, low to high). The transparency ramp is also 'sliced' into bands which are evenly spaced vertically - this means that only the narrow bands are visible, giving us slices of colour (where the colour is defined by where the slice falls on the colour ramp, rather than a fixed colour). This gives a result similar to the high/low pressure bands which weather presenters often use, but with colour added.

I have included a video below, which better explains these shaders - the white 'cloud' is shader 1, and the coloured 'bands' are shader 2;

Although this video shows a top-down view of the scene, it is important to remember that these effects are generated in 3D - moving forwards, I could include moving camera or changing points of view to highlight particular events.

Also, the oxygen density visuals are considered another 'layer' which I can add to the cells and blood vessels, creating a more complete final output.

I am not sure as to how this final output will look at the moment, as I am still developing the visual elements of each of the data-sets, but progress is good and things are at least working now...

Needs more juice...

Similar to my last blog entry, my efforts are still divided between two main projects;

The Going Live project has been making excellent progress. The team and I have continued improving the visual outcome of the 3D elements. With a texture in place, lighting and rendering have started to progress rapidly. Animated shots are being lit, rendered and composited, so that final testing can take place.

Although I have taken on the role of CG Supervisor, I have been involved in other areas, and responsible for others. I was involved in tracking the camera footage and creating a 3D pre-visualisation of the sequence. I have monitored the 3D pipeline continuously, and offered advice and created fixes for problems - this includes working with others in modeling, texturing, rigging, animation, and lighting. I was responsible for creating a dynamic cloth system which would allow for easier animation, and I have managed the rendering process (involving creating render layers and using the render farm). Finally, I get to pass rendered scenes to the compositing team, who can work their magic... and before I know it, we will have finished the advert!

Returning to my cell visualisation work, I have reached a temporary plateau. I have developed my pipeline/workflow and have implemented new and improved scripts to manage the new data that I have received from my colleagues in the Mathematics division.

However, the vast quantity of data has been as proving difficult to process efficiently. After some development, I have speeded up the process, but this still takes a considerable amount of time to complete. Fortunately, I discussed this with my project supervisor/programme leader, and agreed the use of high-specification computers in the University, where I can process data quickly, making use of round-the-clock facilities. This week, I will begin using these facilities, to start working with this new data that I have received.

It is important that I begin this early, as the sooner the numerical data has been translated into 3D space, the sooner I can begin developing the visual qualities of the mathematical models. Completing this visualisation process becomes even more important, as the mathematical models will be presented as part of a conference in June - giving me a deadline to work towards...

More Data

Over the last couple of weeks, I have had to divide my attention between multiple projects.

The 'Going Live' project has ramped up into production, with modeling and rigging now complete. Animation has started, and texturing is currently underway. My role as CG Supervisor has been demanding, as all of these CG elements have been happening in quick succession. On top of this role, I was also responsible for implementing a customised nCloth dynamics system for our 'character'. This was created alongside the rigging process, to ensure that these components would work together happily, and after resolving a few technical problems, the system is now working nicely. My next task was to create and organise the appropriate render layers in Maya, ready for rendering and then compositing to take place (hopefully late this week or early next). Although I had worked with render layers before, this project requires more variants than I am used to working with, so has taken a bit of time to configure and setup properly. Despite all this work taking up more time than initially expected, the project has made good progress, and continues to do so.

After my meeting with mathematics last week (and several more since), my cell visualisation workload has increased also. I have received new data from both students, and I am currently in the process of writing scripts that will translate these into 3D scenes inside Maya.

The first new data set contains fibres (to be added to cells), which are based on xyz locations and xy rotations. I had not scripted rotation values yet, so this was a good opportunity to expand my knowledge of MEL. I am currently awaiting the full data-set for this part of the visualisation, so will continue to work on this moving forwards.

The other data set adds oxygen density to a scene containing cancer cells and blood vessels. This file contains over 30 million lines of information, and weighs in at around 800mb - making it rather difficult to work with. I have tried different approaches in visualising this data efficiently, such as adjusting transparency on cubes based on the density value or scaling particle clouds. Unfortunately, there are about 10,200 points per frame, so these methods take far too long to calculate. I am currently testing a new method, which creates a single polygonal plane, with the required number of vertices. The script then runs through each vertice, and moves it in the y-axis based on the density value (between 0 and 1). A ramp shader then adjusts the transparency of the plane based on the height (where 0 density is fully transparent). This creates white, cloudy patches where oxygen density is high. Although this still takes a long time to process, it is considerably faster than the other methods.

Most of this work is still on-going, and has 'arrived' at the same time, making it difficult to balance. Fortunately, I have been able to allow extra time in working on these projects, so hopefully the worst of it is over now...

On a more exciting note, three of my videos were used at an event in Dundee on Saturday 12th March. The videos are 3D visualisations of mathematical models which are being used to predict cancer growth and development, and were developed in collabroation with a PhD student in the University's mathematics division. They were shown at an event called "Sensational Women in Science" as part of the Women in Science Festival 2011.

Also, some of the other data I am currently working with will be presented at a large conference later this year (in June), so I have a deadline which I can work towards.

Snake In The Grass

...more specifically, a Python.

Continuing on from my Cinema 4D experimentation with metaballs, I abandoned that line of testing, as it wasn't going to be a viable option for the large amount of cells I was dealing with. Alongside Digital Tutors, I returned to Python scripting, trying to 'translate' my MEL script into something useable inside of Houdini.

After almost a week's worth of scripting, fixing things, and re-scripting, the Python/Houdini version of my data-loading script works, creating all of the necessary nodes in Houdini, and keyframing all of the animation. This allowed me to natively create metaballs, rather than try and convert existing scene information. Houdini handles metaballs exceptionally well, adjusting the viewport geometry on the fly (so it doesnt crash regularly like Cinema 4D). Using Mantra, I was able to render out the full 1900 frame image sequence, with frame 1900 shown below;

Although I am still learning to use Houdini, I have made breakthrough progress in using the software alongside Python, opening up a whole range of new opportunities. I have started trying to figure out shaders and lighting in Houdini (which seems more difficult than expected), and I created some example 'looks' for my meta-surface, shown below;

I am due to meet with the mathematics department this week, where I will present all of my research, ideas, and generated media - bearing in mind, that they have not seen any of the results so far. I am hoping that this meeting will help inform my next steps, and direct my technical understanding to a new visual solution.

Going To The Cinema

This morning, I stumbled across a fantastic website, called Molecular Movies. It describes itself as a 'portal to cell and molecular animation' and aims to provide scientists with tutorials on developing 3D visualisation skills.

Alongside tutorials, the website has a showcase section, with examples of 3D being used to visualise complex biological scenes - although these are interesting to watch, the scientific content is a bit beyond my own knowledge. Fortunately (and more importantly), it is useful to have a 'database' of the kind of work that is going on in my field.

Returning to the learning resources, I realised I had already covered the majority of content, as it was designed for those new to using 3D. That is, until I found an interesting article on using metaballs to create molecular surfaces, using Cinema 4D. Although I was not familiar with using C4D, I decided it was worth looking at it, as it was going to less technically challenging than learning Houdini (which is also reliant on Python), and should give a similar result.

Creating metaballs in C4D was fairly straight forward, and gave great results with very little input. It took a bit of time to figure out how to animate objects, and then render a scene, but after using the software, I would definately be confident using it again. The geometry was also significantly 'cheaper' than my testing in RealFlow, and could be used on a large scale. A video example of metaballs in action can be seen below;

The next step was to take my cell data-set and use metaballs to create a single organic structure - this proved substantially more difficult. Realising that I couldn't simply 'read' my data, as C4D also relies on Python, I opted to export both OBJ and FBX files from Maya, hoping that at least one would work.

These imported easily, and a metaball surface could be applied, but did not work properly (simply creating one large sphere). After vast amounts of experimentation, I scaled the imported objects... and success! A simple fix for what seemed like a complicated problem. This breakthrough meant that I could now take objects into Cinema 4D, use metaballs to create a surface, and either render, or export to Maya for render (as I am more familiar with the software package).

The only difficulty now, is that this workflow currently only works on single frames, and can't be applied to my animated data-set... this is the next problem to solve! A render of the results so far can be seen below (showing a before/after comparison);

After my previous post, I felt like I had reached a brick wall, but after discovering the Molecular Movies website, it has helped me hurdle these difficulties... only to find a new hurdle waiting on the other side!

Reaching Limits...

This week has proven difficult in terms of my ideas being restricted in their implementation, primarily by the software/technology I have available.

Scripting has made a reasonable amount of progress. I have added colour changes to my first data-set, as it was not clear when cells 'appeared'. New cells begin red, and fade to their regular colour over 25 frames (1 second), which is more informative, and better communicates the data visually. It is important to point out here, that this change was added to the MEL version of my script. A render taken from the Maya scene can be seen below;

I started developing skills in scripting using Python, as Houdini relies on this for data input. Python also crosses over with Maya as the shared 'language'. I began by trying to find the common links between MEL and Python, which accelerated my understanding of how to code effectively in Python and make things happen. The Maya-Python version of the script is now equivalent to the MEL script.

However, the Houdini-Python script currently only reads data, but does not yet generate or animate geometry - I am having difficulties finding useful information on the Houdini specific Python module (commands), so this part of scripting is on hold for now.

Due to these difficulties, I returned to Maya, and started work on trying to create an organic-looking cell surface - that is, a single surface which is created from all of the cells, and moves and acts as one (instead of 1067 individual cell 'spheres'). Initially, I wanted to generate a particle field in place of spheres, but, unfortunately, Maya does not allow transformation of individual particles within a particle object. There is no way around this, except by creating a separate particle object per cell - by doing this however, the particles can no longer merge together as one.

I started to consider other options to achieve this organic look, with RealFlow being top of my list. I generated a 'particle sphere' in Maya and exported this directy into RealFlow. After developing a complicated workflow of imports and exports, and a lot of experimentation, I was able to then use RealFlow to mesh this particle field. However, for one cell to be meshed fully, I required around 37,000 particles, generating approximately 110,000 faces for the mesh. With a scene containing 1067 cells, I realised that although giving a nice result, it was certainly not a viable option. A render of the style of cell-split can be seen below;

Beyond using RealFlow, my next efforts will involve using particles or metaballs in Houdini. Metaballs work very nicely (tested using 2-3 objects manually), and are 'clever' spheres which merge together when they are close to each other (without any configuration in Houdini, metaballs give a great looking result). Particles in Houdini will hopefully offer more flexibility than Maya, allowing me another technique to create the look that I want to achieve.

Although I have found this week frustrating, finding limitations in the software I am using, I am confident that I will be able to find a solution, allowing me to realise the full potential of the creative ideas I want to unleash...