The Art of Dominant Color in Film

In a recent post to the nettime list, Florian Cramer gives a slightly tongue-in-cheek description of his recent work on his Floppy Films project
In my own attempt of compressing this year's Oscar-nominated films to full-length 1.44 MB files as part of my Floppy Films project, I also crunched "Slumdog Millionaire" to a 7x3 pixel/8fps/128 colors animated GIF. In such a stripped-to-the-bones rendering, the dominant color palette of the production design becomes quite visible. In the case of "Slumdog Millionaire", the dominance of red-brown-yellow 'curry' colors aptly reveals the whole exoticism of the film. [In contrast, "Milk" uses a bright pastel - milky - color palette while dark sepia tones dominate the 1940s/50s period setting of "The Reader", and a frequent combination of olive greens and magenta violets sets the white trash tone for "The Wrestler". Since everything else seems predictable enough, I didn't bother watching higher resolution versions of these films.]

Watching the Floppy Films version of "Slumdog Millionaire" in a web browser may produce upscaling artifacts that create the illusion of there being more pixel information that there actually is, but it really is just 7x3. Here is a zoomed look at the actual information opening frame of the film:



At the heart of the artist's playfully irreverent abstraction of the Oscar lineup is a good question. Can we describe cultural objects like feature films by characteristics such as "dominant color palette"? Can we meaningfully compare them? Is such knowledge practical and useful, and will we learn anything we didn't already know?

We can, but the methods and outcomes are not always obvious. Here is one quick cultural analytics approach to Florian's Floppy Film take on "Slumdog Millionaire". Rather than impressionistically assessing a "red-brown-yellow 'curry'" theme, we can visualize the film's appearance, average states, and histogram distribution.

What does "Slumdog Millionaire" look like overall?


A montage of the entire film reveals a dark image dominated by unsaturated light tans, greens, and blues, with a few bright oranges and blues at the beginning and end.

What is the "Slumdog Millionaire" dominant color and composition?


Taking the mean of the entire film produces an olive green image with slightly brighter center-shot.

What is the "Slumdog Millionaire" dominant color pallete?


Black and grey are main colors, as in most realistic cinema - this perhaps goes without saying, but maybe it shouldn't go unsaid, as it probably isn't true in things like cartoons, ads, or music videos. Besides black, a quantization of the top 12 colors shows greens, blues, and tan, with a little bit of orange. Reds and browns aren't anywhere near the top, and yellow is a minor note.


This olive-blue "Slumdog Millionaire" revealed above seems very different from the red-brown-yellow one in the artist's tongue-in-cheek description, however we can find that palette in the film. Here is the same process applied only to the opening two minutes of the film:

What does the opening look like overall?


What is the opening's dominant color and composition?


What is the opening's dominant color pallete?



Here in the opening we can clearly see the red-brown-yellow 'curry' colors that the artist described, and perhaps begin to guess at how a general aesthetic assessment arose out of this strong opening impression.

This process of exploring "Slumdog Millionaire" suggests several things:

  1. We can characterize the palettes of films as a whole, and it can be useful to do so.

  2. We can characterize shots and sequences as differing from or typifying the character of a film as a whole.

  3. We can examine the individual or group aesthetic impression of what typifies the "essential" character of a film and identifies which shots or scenes best correspond to that assessment.


Cultural analytics approaches to film can be a useful tool for cutting through confirmation or selection bias and seeing the character of the object clearly -- however, it is not simply a method for trumping aesthetic impression with quantified data. Inquiry can flow just a easily in the other direction, beginning with the fact of the impression, and asking what in the film best corresponds to or typifies that perceived essence. For example, if the impression of a critic or a group was that red-brown-yellow typifies the essential look of "Slumdog Millionaire," we could begin with the opening and closing shots and ask how / why / if they serve as an essential representation of the film as a whole. Or we could take the histogram of each frame (or of a rolling average for a short window of frames) and search it.

This brief exploration is fairly far afield from the original provocation of the Floppy Films project, however the process of engaging Cramer's off-hand comment on color has helped me understand that work's method (and others like it) a bit better. Many forms of artistic and visual abstraction (e.g. "unfocused / pixelated video" type projects) may not actually help us come to grips with how color changes over time, even though their abstract idiom gives us a strong impression that they are doing so. Histogrammatic visualizations are direct and explicit, while simple frame montages (after the style of projects such as Cinema Redux by Brendan Awes) is both visceral and has the added virtue of Tuftian high information density. By contrast, Floppy Films: Slumdog Millionaire is much more effective at communicating the "dominant color palette" of the moment -- which is an important kind of knowledge of a very different order.

BettyBoop.viz

Cultural Analytics method allows us see the diffirences between cultural artifacts which at first may appear to be identical - and to reveal the similarities between artifacts which may appear to have nothing in common. The following quick "analytical sketch" (think of "sketches" done by artists @processing.org) done with 5 Betty Boop cartoons from the 1930s shows how by experimenting with alternative ways of graphing the data we make visible the relationships between the artifacts from the same set (i.e., cartoons made by Max Fleischer animation studio over a period of a few years.)

The five cartoons are available at archive.org. They are Betty Boop: A Song a Day (1936), Betty Boop: Is My Pam Read (1932), Betty Boop: More Pep (1936), Betty Boop's Ker-Choo (1932), Betty Boop: The Candid Candidate (1937).



A sequence from Betty Boop is My Palm Read, 1932 (sampled at 10 frame intervals).


In order to reveal the similarities and diffirences between these five works done by the same animation studio, we have also included the data from a artifact produced in another era and with radically diffirent technology - a music video by Björk (which we analyze in more detail in MotionGraphics.viz.)



5_films_vs_Biork_mean_PART2
X: time | Y: mean value (i.e., average gray scale value for each frame).
Thik red line: Björk video. Other lines: Betty Boop cartoons.
The graph does shows that the "temporal signatures" of Björk video - which in its techniques exemplifies motion graphics of the 2000s - is quite distinct from the cartoons of the 1930s drawn by hand and shot on animation stand. This is not a big surpise; but the graph does not tell us much about the relationships of the five cartoons to each other.


5_films_vs_Biork_mean_sorted
Same data but represented as a histogram-like form better reveals the patterns. Again, it is not surprising to see that the visual temporal structure of the five cartoons is quite different from that of Biork video; however, what we did not expect to find is how remarkably similar the cartoons are to each other. Produced over the course of 5 years by different people, it is though they are parts of the same film. (Adorno's fans, take notice.)
X: mean value (average gray scale value for each frame).
Y: number of frames which have a particular mean value.


5_films_vs_Biork_centerofmass_sorted
Having discovered the technique of graphing the measurements of all the frames of a film sorted in value, we decided to apply it to other measurements of the same artifacts. Here we plot sorted center of mass data. (Center of mass is defined as "he brightness-weighted average of the x and y coordinates all pixels in the image.") Although a single measurement of the center of mass for one frame may not be very informative, a successive set of measurements indicates (very aproximately - but quick to produce) how the action is moving in X,Y over the time. The similarity of the five cartoons stands out even more clearly.


5_films_vs_Biork_x_centerofmass_overlay_B
In this graph we are looking at all frames in five cartoons and all the frames from Biork video - projected on top of each other. The data plotted again is center of mass measurements. Now we discover something new. Althogh worked over with software, Biork's video is based on live footage of the singer, so in this way is typical of the universe of moving images which show people - who tend to walk on the ground. However, the cartoon universe is almost completely isotropic (invariant with respect to direction): the distribution of enter of mass measurements is symmetrical, were the characters are not bound by gravity.


5_films_vs_Biork_y_centerofmass_overlay_PART
Here we plot center of mass measurements (Y value) over time. When data is plotted in this way, the opposition between the isotropic movement universe of the cartoons vs. non-isotropic universe of the humans is also clearly visible.

PLAYPOWER no Brasil e no mundo

A seção Gadget Lab do Blog da revista Wired fez uma cobertura completa sobre o projeto Playpower, que foi apresentado no dia 10 de março de 2009 no O'Reilly Emerging Technology Conference, in San Jose, Califórnia. O projeto vem cada vez mais ganhando espaço nas discussões sobre soluções educacionais através do uso de computadores com baixo custo e fácil acessibilidade.
O projeto lançou recentemente a sua página em língua portuguesa, para os interessados em colaborar na produção de jogos com preços acessíveis à maioria da população carente. Só para lembrar, o computador custa, nos Estados Unidos, 12 dólares.
O site do projeto em português é: www.playpower.org/br

Exploring Time in Structured Video

How can we characterize mass quantities of video in ways that make it more tractable to data exploration without losing aesthetic richness? Three new techniques are briefly discussed here:


  1. Time Frames, in which the time-axis is collapsed to create a single combined image (much like a long-exposure photograph)
  2. Time Slices, in which the video is "sliced" along the time-axis to reveal recurrent patterns not visible on the surface (much like counting the rings in a tree trunk)
  3. Time Structures, in which the video is arranged as a stack of images and then sculpted in 3d (much like a 3d medical image such as a brain scan)

The resulting combinations, cross-sections, and 3d models can all be used:
  • as thumbnails to characterize complex video clips
  • as analytic images to suggest regions and segments for further analysis
  • as classifiers for clustering large video clip data sets


Like common techniques of representing video such as film strips or montages of key frames, the above representations all work to make time simultaneously apprehensible. Unlike film strips and montages, however, these frames, slices, and structures all emphasize variation and continuity over time.

Examples below deal exclusively with video gameplay recordings, however experiments applying these techniques to other types of software-generated video and video in general are underway.


Time Frames: collapsing the time axis of video


Asteroids (Atari 2600): 3 sample frames


Asteroids (Atari 2600): mean frame of 10 mins
Each pixel represents the arithmetic mean of that location over the entire 10 minute recording of gameplay. The player


Asteroids (Atari 2600): standard deviation frame of 10 mins

Each pixel represents the amount of change over the course of the 10 minute recording of gameplay. Dangerous asteroids spawn thickly at the perimeter, and thin out as they approach the clear center column protected by the spinning player.


Orbient (Nintendo Wii): 3 sample frames


Orbient (Nintendo Wii): mean frame of 10 levels


Orbient (Nintendo Wii): mean frame for each of 9 levels


Desktop Tower Defense 1.5 (Internet): mean frame for each of two players


Desktop Tower Defense 1.5 (Internet): mean frame of 40 players


Time Slices: exploring the time axis of video


Cubello (Nintendo Wii): sample frame


Cubello (Nintendo Wii): mean frame of 5 minutes of play.
Image indicates consistent HUD-style user interface elements as possible areas of interest along three edges of the screen.


Cubello (Nintendo Wii): time slices of 5 minutes of play
Slices are taken from the ammo magazine column (left) and from the "bonus time" row (bottom). Each slice can be read as a graph.


Frogger (Atari 2600): 3 sample screenshots


Frogger (Atari 2600): mean frame of 5 minutes of play


Frogger (Atari 2600): "lily pads" time slice of 5 minutes of play

Reading from top, green bars indicate the duration the pad has been marked complete (filled with a Frogger face). Orange marks indicate the appearance of bonus flies.


Frogger (Atari 2600): "countdown timer" time slice of 5 minutes of play
Reading from top, orange bars indicate changing score, blue bar indicates countdown timer, which resets every time Frogger either fills a lily pad or loses a life.


Time Structures: scultping the space-time cube of video


Rockband (Nintendo Wii): sample frame


Rockband (Nintendo Wii): regions of interest in user interface



Rockband (Nintendo Wii): mean frame with two time slices.
Left: time slice of "Band meter" bar. Bottom: time slice of musical note sequence as tablature.




Rockband (Nintendo Wii): orthoganal views of band meter and musical note time slices.






Rockband (Nintendo Wii): 3-dimensional structures extracted from video regions of interest: tablature in 3d view, wheel-shaped meter (from center bottom of UI) and instrument marker (from inside left of UI)

Recently...