Composited Sequence

As part of my cell visualisation project, I have been working on a second set of data, which also represents the growth and development of cancer cells. This dataset however, focuses on the stage of which the individual cells are in, with the intention of identifying the most effective time for treatment to take place.

With thanks to one of the mathematicians working in this area, I was provided with a large quantity of data (around 33 million lines of numerical information) which had been generated, and could be used to drive my 3D visualisation. I created a MEL script which could 'read' each section of this data and then generate/animate the appropriate objects in 3D space.

After completing this stage, I then setup the scenes for rendering - adding lighting, placing cameras, adjusting render settings and creating render layers. Using the VERL render farm, I generated approximately 33,000 renders which could be composited together to create a completed 3000 frame sequence (depicting 600 hours of cell growth and development).

Using After Effects, I combined all of the image sequences and created several individual compositions which would be layered to create the final output. Because almost all of the work had been completed in Maya, there was very little that had to be done in After Effects, except combine the appropriate layers. Rather than generate a QuickTime MOV, I rendered a master image sequence (after speaking to one of my classmates) which could then be used to create suitable video files as and when needed.

Because the source of my output comes from unpublished mathematical models, I am unable to post any of the video online (it would be unfair to the ongoing research still being carried out). Instead, I have opted to include an image of a single frame, which can be seen below;

Over the next few weeks, I will be meeting with the mathematician that I created this for, as there are several upcoming maths conferences and events where my work could be used to illustrate the research they are carrying out.

Advanced Production (Summary)

Despite returning to Semester 2 later than expected (due to illness), I have made excellent progress, with major breakthroughs in overcoming many of the challenges my project was facing.

My programme of study takes a more visualisation based route, and I am working with the University mathematics division to visualise numerical data (gained from their mathematical models) which represents the growth and development of cancer cells (or solid tumour growth).

Working with numerical data has created it's own set of challenges, as Maya cannot read or interpret this in it's off-the-shelf form. As a result of this problem, I have developed skills in using Maya's own scripting language, MEL (Maya Embedded Language) and also a second scripting language (used more widely) called Python. Although this has been a difficult and time-consuming process (several months of learning, which is still ongoing) I have been able to harness this new understanding and create custom tools which can be used to read the mathematical data and create appropriate 3D geometry.

Throughout learning each scripting language, I referred primarily to online sources, and the 'official' documentation provided. I also used the Digital Tutors service, which was both informative and relevant, helping me overcome the hurdle of knowing where to start, and what to focus on.

Through creating my own custom tools, I was able to work with the numerical data-sets provided. Starting with simple tests, I was (eventually) able to construct fully animated scenes containing simple spheres, representing the individual cancer cells, and although time-consuming to process, these results opened the way for more advanced development of the data and it's visual attributes.

I spent some time experimenting with Maya's render layers, and compositing passes using Nuke. This was as a response to feedback given last semester - not everything has to be done in 3D, as it can often be quicker and easier to complete some work in 3D, but fine-tune the details in post-production. This helped me develop possible ideas for the look of cells, when considering texture on a microscopic basis, and I also experimented with 'faking' depth-of-field techniques to enhance this.

Although creating spheres was great, I wanted to push my skills further. I spent time working with RealFlow, Cinema 4D and Houdini - three great 3D packages. With the exception of RealFlow, these were entirely new to me, and although Cinema 4D was fairly straight-forward, Houdini had a very steep learning curve. Using my Python scripting skills, I created a large-scale metaball system, equivalent to the spheres created in Maya - this was a challenging task, forcing me to solve several problems, whilst trying to script in a language mostly new to myself. The result of this technical experimentation allowed me to create an interesting almost-organic 3D structure, which moved and behaved as one surface, instead of 1068 individual cells.

After completing these Python related tasks, I returned to Maya to work with additional data-sets, still showing the growth and development of cancer cells, but in a different format. This data would require not only animated movement of cells, but changing colours, and eventually, complex density 'clouds'.

The difficulty at this stage in my work, was the size of the data-sets... one of which contains over 30 million lines of information (a text file weighing in at around 800mb). Despite breaking this into smaller chunks of data, it was still computationally intense, and difficult to work with (often crashing high speed computers). I spent time streamlining and optimising my scripts, and the way they handled the data - as an example, my oxygen density script was originally taking around 350 seconds per frame to process, whereas now it takes around 90 seconds, about a quarter of the time. As I have learned more about MEL, and gained a better understanding of the language, I have been constantly improving and refining my scripts, to ensure the best efficiency when working with large data sets.

I am currently working with new additional data-sets, which again have constantly required me to think and problem solve... now that my scripts are optimised, I am confident that my scripting abilities are suited to a wide range of data types, across two very different scripting languages. This allows me to tackle upcoming problems with new knowledge, based on the experiences I have had over the last ~3 months, giving me a definite headstart.

In summary, my practice-based learning has developed the following;

• Scripting - both MEL and Python, for simple and complex tasks (including 3D visualisation)
• Improved software knowledge - Maya, RealFlow, Nuke
• New software knowledge - Cinema 4D, Houdini
• Advanced 3D skills development - particles/dynamics, data input/output, script optimisation, rendering
• Experience in 2D compositing with 3D-based image sequences

Due to the sensitive nature of the mathematical data I am working with (and as the source data is effectively the result of unpublished research) I am unable to post a large amount of my visual work online. However, this sensitive material will be made privately available during course presentations, and to module assessors as necessary.

Visible Progress!

Over the last couple of weeks, my role in the Going Live project increased significantly, and then stopped completely. All of the animated shots had lighting added, and then I added the render layers/passes and started feeding completed shots through the render farm (which was considerably faster than I expected it to be). Sound effects and music were then added by the sound team, creating our finished advert.

It took a long time to get there, and there were problems along the way, but I learned a lot (particularly about rendering and compositing) and I am glad we all got there in the end! Next week, we are due to meet with the company in their London studio and present our finished project - hopefully the feedback will be good!

As for my cell visualisation work, this has been making good progress since my role in Going Live has lessened.

The first data-set I was working with, which represented cells and fibres in 2D space, now has fully working scripts, which are streamlined to work efficiently (or to actually work at all!). I am currently awaiting feedback on the video outcome of this work, so that I can decide where to take this next.

The other data-sets (involving cells, blood vessels, and oxygen density maps) have made even better progress. Again, after optimising my MEL scripts, the amount of data (several million lines of information) has become manageable, although time-consuming to process. I am currently part of the way through 'translating' this data into Maya's 3D environment.

An example render from the cells file can be seen below. This example frame is approximately two thirds of the way through the cell data, and incorporates some 'noise' on the cell surfaces to break up the uniformity (an idea suggested by the mathematician who provided the data);

As for the oxygen density, I decided to continue using a single polygonal plane for this, with grid points in the data having a matching vertex on the 3D geometry. The data then lifts/lowers each grid point/vertex between 0 and 1, where 1 is the most dense area of the oxygen 'clouds'.

The 'look' of these clouds is then controlled using one of two shaders.

Shader 1 ("Clouds") is coloured white, and uses a vertically-aligned ramp shader for it's transparency value, where 0 is fully transparent and 1 is fully visible. This means that as points on the vertex grid are changed in the Y-axis, their transparency is also changed (as they are moved higher, they become more visible).

Shader 2 ("Bands") expands upon this idea, and uses a second ramp for the colour (from blue to red, low to high). The transparency ramp is also 'sliced' into bands which are evenly spaced vertically - this means that only the narrow bands are visible, giving us slices of colour (where the colour is defined by where the slice falls on the colour ramp, rather than a fixed colour). This gives a result similar to the high/low pressure bands which weather presenters often use, but with colour added.

I have included a video below, which better explains these shaders - the white 'cloud' is shader 1, and the coloured 'bands' are shader 2;

Although this video shows a top-down view of the scene, it is important to remember that these effects are generated in 3D - moving forwards, I could include moving camera or changing points of view to highlight particular events.

Also, the oxygen density visuals are considered another 'layer' which I can add to the cells and blood vessels, creating a more complete final output.

I am not sure as to how this final output will look at the moment, as I am still developing the visual elements of each of the data-sets, but progress is good and things are at least working now...

Needs more juice...

Similar to my last blog entry, my efforts are still divided between two main projects;

The Going Live project has been making excellent progress. The team and I have continued improving the visual outcome of the 3D elements. With a texture in place, lighting and rendering have started to progress rapidly. Animated shots are being lit, rendered and composited, so that final testing can take place.

Although I have taken on the role of CG Supervisor, I have been involved in other areas, and responsible for others. I was involved in tracking the camera footage and creating a 3D pre-visualisation of the sequence. I have monitored the 3D pipeline continuously, and offered advice and created fixes for problems - this includes working with others in modeling, texturing, rigging, animation, and lighting. I was responsible for creating a dynamic cloth system which would allow for easier animation, and I have managed the rendering process (involving creating render layers and using the render farm). Finally, I get to pass rendered scenes to the compositing team, who can work their magic... and before I know it, we will have finished the advert!

Returning to my cell visualisation work, I have reached a temporary plateau. I have developed my pipeline/workflow and have implemented new and improved scripts to manage the new data that I have received from my colleagues in the Mathematics division.

However, the vast quantity of data has been as proving difficult to process efficiently. After some development, I have speeded up the process, but this still takes a considerable amount of time to complete. Fortunately, I discussed this with my project supervisor/programme leader, and agreed the use of high-specification computers in the University, where I can process data quickly, making use of round-the-clock facilities. This week, I will begin using these facilities, to start working with this new data that I have received.

It is important that I begin this early, as the sooner the numerical data has been translated into 3D space, the sooner I can begin developing the visual qualities of the mathematical models. Completing this visualisation process becomes even more important, as the mathematical models will be presented as part of a conference in June - giving me a deadline to work towards...

More Data

Over the last couple of weeks, I have had to divide my attention between multiple projects.

The 'Going Live' project has ramped up into production, with modeling and rigging now complete. Animation has started, and texturing is currently underway. My role as CG Supervisor has been demanding, as all of these CG elements have been happening in quick succession. On top of this role, I was also responsible for implementing a customised nCloth dynamics system for our 'character'. This was created alongside the rigging process, to ensure that these components would work together happily, and after resolving a few technical problems, the system is now working nicely. My next task was to create and organise the appropriate render layers in Maya, ready for rendering and then compositing to take place (hopefully late this week or early next). Although I had worked with render layers before, this project requires more variants than I am used to working with, so has taken a bit of time to configure and setup properly. Despite all this work taking up more time than initially expected, the project has made good progress, and continues to do so.

After my meeting with mathematics last week (and several more since), my cell visualisation workload has increased also. I have received new data from both students, and I am currently in the process of writing scripts that will translate these into 3D scenes inside Maya.

The first new data set contains fibres (to be added to cells), which are based on xyz locations and xy rotations. I had not scripted rotation values yet, so this was a good opportunity to expand my knowledge of MEL. I am currently awaiting the full data-set for this part of the visualisation, so will continue to work on this moving forwards.

The other data set adds oxygen density to a scene containing cancer cells and blood vessels. This file contains over 30 million lines of information, and weighs in at around 800mb - making it rather difficult to work with. I have tried different approaches in visualising this data efficiently, such as adjusting transparency on cubes based on the density value or scaling particle clouds. Unfortunately, there are about 10,200 points per frame, so these methods take far too long to calculate. I am currently testing a new method, which creates a single polygonal plane, with the required number of vertices. The script then runs through each vertice, and moves it in the y-axis based on the density value (between 0 and 1). A ramp shader then adjusts the transparency of the plane based on the height (where 0 density is fully transparent). This creates white, cloudy patches where oxygen density is high. Although this still takes a long time to process, it is considerably faster than the other methods.

Most of this work is still on-going, and has 'arrived' at the same time, making it difficult to balance. Fortunately, I have been able to allow extra time in working on these projects, so hopefully the worst of it is over now...

On a more exciting note, three of my videos were used at an event in Dundee on Saturday 12th March. The videos are 3D visualisations of mathematical models which are being used to predict cancer growth and development, and were developed in collabroation with a PhD student in the University's mathematics division. They were shown at an event called "Sensational Women in Science" as part of the Women in Science Festival 2011.

Also, some of the other data I am currently working with will be presented at a large conference later this year (in June), so I have a deadline which I can work towards.

Reaching Limits...

This week has proven difficult in terms of my ideas being restricted in their implementation, primarily by the software/technology I have available.

Scripting has made a reasonable amount of progress. I have added colour changes to my first data-set, as it was not clear when cells 'appeared'. New cells begin red, and fade to their regular colour over 25 frames (1 second), which is more informative, and better communicates the data visually. It is important to point out here, that this change was added to the MEL version of my script. A render taken from the Maya scene can be seen below;

I started developing skills in scripting using Python, as Houdini relies on this for data input. Python also crosses over with Maya as the shared 'language'. I began by trying to find the common links between MEL and Python, which accelerated my understanding of how to code effectively in Python and make things happen. The Maya-Python version of the script is now equivalent to the MEL script.

However, the Houdini-Python script currently only reads data, but does not yet generate or animate geometry - I am having difficulties finding useful information on the Houdini specific Python module (commands), so this part of scripting is on hold for now.

Due to these difficulties, I returned to Maya, and started work on trying to create an organic-looking cell surface - that is, a single surface which is created from all of the cells, and moves and acts as one (instead of 1067 individual cell 'spheres'). Initially, I wanted to generate a particle field in place of spheres, but, unfortunately, Maya does not allow transformation of individual particles within a particle object. There is no way around this, except by creating a separate particle object per cell - by doing this however, the particles can no longer merge together as one.

I started to consider other options to achieve this organic look, with RealFlow being top of my list. I generated a 'particle sphere' in Maya and exported this directy into RealFlow. After developing a complicated workflow of imports and exports, and a lot of experimentation, I was able to then use RealFlow to mesh this particle field. However, for one cell to be meshed fully, I required around 37,000 particles, generating approximately 110,000 faces for the mesh. With a scene containing 1067 cells, I realised that although giving a nice result, it was certainly not a viable option. A render of the style of cell-split can be seen below;

Beyond using RealFlow, my next efforts will involve using particles or metaballs in Houdini. Metaballs work very nicely (tested using 2-3 objects manually), and are 'clever' spheres which merge together when they are close to each other (without any configuration in Houdini, metaballs give a great looking result). Particles in Houdini will hopefully offer more flexibility than Maya, allowing me another technique to create the look that I want to achieve.

Although I have found this week frustrating, finding limitations in the software I am using, I am confident that I will be able to find a solution, allowing me to realise the full potential of the creative ideas I want to unleash...

Biomedical Visualisation

With a new year, comes new ideas and refreshed inspiration!

Over the last couple of weeks, I have been developing my MEL scripting abilities - something necessary if I want to work with importing numerical data into Maya. Although daunting at first, things have gradually started to make sense, showing the logical development of the stages involved in trying to achieve my goal.

Last week, I had a major breakthrough in using MEL, and was able to create a script which would read one of the mathematical data-sets. The script works on a line-by-line basis, reading comma separated values and placing those into individual variables, which are then used to create objects and keyframe animation. Upon running the script, all actions are automated, and require no input from the user - this simplifies the process, and speeds up creation of a Maya scene greatly. It also ensures that no mistakes are made, as long as the script is correct and data is formatted consistently.

An example of the type of data being used can be seen below;

The purpose of this script is to convert raw numerical data into something visual, built in 3 dimensions. Currently, the script uses simple polygonal spheres to represent cells, although in the future this could be changed to use particles, or something different entirely. An example render of the scripts process can be seen below - the results are dramatically different to looking at thousands of line of numbers;

At this stage, I was confident in my MEL writing abilities, so created 2 more (similar) scripts which would work with the other mathematician's dataset. This data is entirely different however, as the cells are placed in 2 dimensions, and instead of changing size/radius, they change colour based on a numerical value. This posed it's own problems, as I would need to have a new shader for every cell, if I wanted them to change colour individually. The scripts for this data work, but are still in early stages - no renders have been produced as of yet.

Beyond being able to get data into Maya, I was then free to experiment with the visual aspects of representing the data. I tested several techniques which would be useful later on, including using render layers (for alpha channels and depth passes), adjusting camera settings in Maya (to create depth of field) and compositing render layers using both Nuke and After Effects (using short image sequences). Although familiar with compositing techniques in Photoshop, I wanted to familiarise myself with these methods when working with videos and image sequences.

As the final part of my experimentation, I started working with the first data-set in 3D, and added a camera and some basic lighting. I rendered 3 passes - 'beauty', alpha and depth - and composited each of these layers using Photoshop. After some experimentation, I realised that I was happy with the result, and would be confident in replicating the visual style using finished image sequences. The completed composite can be seen below;

Although I am only 2 weeks into this semester, I have made tremendous breakthroughs in my own programme of study, particularly with using MEL to import numerical data. I hope to continue this progress throughout the semester!

Moving forwards, I would also like to continue developing my technical skills and abilities - this will ensure that I have the best opportunities to create high quality work, with no restrictions on the software I can use to achieve this. I hope to continue working with RealFlow, and hope to have some visual examples soon. I also plan on developing skills in using Houdini, another 3D package, with a more technical focus.

Finally, Scripting...

After avoiding MEL scripting for far too long, I started working through "Introduction to MEL" by Digital Tutors, in the hopes that I could finally make sense of the more complicated material presented in In Silico.

Full information on the lessons can be found here, as they covered a wide variety of topics. This included a look at MEL syntax, creating primitive objects, editing attributes, using WHILE/FOR/IF commands and then combining all of the taught content into a single project, which resulted in the ability to create a textured flower with randomised attributes, simply by clicking a single button. The image below shows some of these random flowers;

After completing these lessons, I feel much more confident in using simple MEL expressions, and I intend on continuing this development, which will hopefully allow me to create a more complicated script which will import large amounts of numerical data and automate the creation of objects and animation.

For those who may be interested (and I realise it may not be many!), here is the complete script used to create the flowers...

Research Continues...

Today was a day for research. Or at least that's what I thought before I started.

Continuing on with 'In Silico', I worked through Chapters 12-13. This introduced MEL scripting, and also looked at importing numerical data into Maya. These chapters seemed to ramp up the complexity significantly on previous material, and were difficult to follow. Although I worked through all of the examples, nothing has stuck in my mind and I feel like I haven't gained a huge amount... except the mess of expressions and terms floating around in my mind.

After struggling with the In Silico material, I chose to shift focus to something more creative (and let's say more interesting). Instead of using the Gnomon material to explore the use of dynamics, I have started working on a new book, on loan from the University Library. 'Maya Studio Projects: Dynamics' is written by Todd Palamar and published by Sybex, and deals with creating realistic earth, wind, fire and water effects in Maya. Although my interest lies within creating fire, I decided to look at the particle stuff and see how it compared to using the older Gnomon material.

Within a page of starting, I was already working with simple dynamics setups in Maya. The book was clear, and easy to follow. It made use of high-quality images to complement the textual instructions - something the Gnomon DVDs struggled with.

The first chapter of the Studio Projects book deals with using particles, and has you creating and customising emitters, whilst playing with different types of particles and changing the appearance of the outcome. It uses simple expressions alongside the normal Maya interface. Having completed a couple of examples, I felt better about today's work.

Who knows, after time, the complicated approach of In Silico may make sense. In the meantime I'll walk before I can run, and give this new book a chance.

The first video example shows a comet 'flying' along a path. This was created by applying a lifespan to the particles which 'trailed' from a sphere. Ramp shaders were also applied to colour and transparency values.

The second video example shows an asteroid belt 'orbiting' around a planet (pre-supplied). This was an experiment in using a NURBS circle to emit particles, and then containing them with a cylindrical volume. A ramp was also applied based on the size of the spheres, which was randomised using a simple expression.

Gnomon Dynamics 1 : Galaxy Spiral

Today I decided to approach the Gnomon material, giving 'In Silico' a rest before starting on the more complicated MEL scripting chapters.

Gnomon's Dynamics 1 disc is designed as a basic introduction to dynamics, and according to the package, introduces the following tools;

Particle Tool, Create Emitter, Directional Emitters, Volume Emitters, Various Fields, Hardware Rendering Techniques

Quite an extensive list for an introduction, especially with a runtime of only 110 minutes. I say only... after getting about 9 minutes in, my imagination was already running wild and I had been well and truly distracted by the possibilites of 'painting' particles directly into a scene.

With the addition of two fields (Turbulence and Vortex) I had created a swirling galaxy system using only particles. I knew that I wanted to develop this scene further and see what I could do with it. I created a basic ramp shader and using a simple expression, coloured the particles based upon their distance from the scene's origin point.

After an hours worth of tinkering and playing around with these simple tools in Maya, I remembered the Gnomon Dynamics tutorial I should have been watching - not that it felt immediately important, I had created something really spectacular all by myself...

In Silico : Chapters 1-7

So far this week, I have been reading 'In Silico', and working through the tutorials provided. I had previously completed a lot of work within Maya, so the majority of material covered was not new to me... until I reached the section on dynamics, something I haven't worked with before.

As part of this new material, I created a simple dynamics system - a cylinder filled with tiny little spheres, which would 'float' around and collide with the cylinder and each other. This was created using a particle emitter, and although somewhat basic, has already opened my eyes to the benefits of using the dynamic system. The ability to create a fully animated scene by simply just setting up some object properties is amazing. Certainly much easier than trying to animate each object individually (there were 200 spheres) and far more realistic!

In addition to realising the potential of using dynamics, I was also introduced to my first piece of MEL scripting. MEL stands for Maya Embedded Language and is a scripting language within Maya. I used MEL to apply colour to each of the spheres based on their proximity to other spheres. They are normally black, and become more red as they get closer to other spheres, becoming fully red upon collision. Using colour made the simulation clearer - something which will be very important in later work that I undertake.

Finally, I created a quick playblast of the simulation. Nothing fancy, or complicated, but it shows the start of my research into using dynamics and scripting within Maya.