Image Processing and Software Epistemology

Turning everything into data, and using algorithms to process is analyze it has a number of major consequences for what it means to know something. It creates new strategies which together make up software epistemology. Epistemology is a branch of philosophy which asks questions such as what is knowledge, how it can acquired, and to what extent a subject can be known. Digital code, algorithms, machine learning, big data, social media, faster processors and larger storage, and other parts of modern techno-social universe introduce new ways of acquiring knowledge, in the process redefining what it means to know something.

if you have a dataset, its always possible to invent new algorithms (or new ways to scale existing algorithms to analyze big data faster) which can analyze the information in ways the previous algorithms could not. You can extract additional patterns and also generate new information from the existing data.

Algorithms and software applications which analyze images and video provide particularly striking examples of the ability to generate additional information from the data years or even decades after it was recorded.

In Blowup, a 1966 film directed by Michelangelo Antonioni, the photographer who takes pictures of a couple kissing in the park uses the strategy of printing progressive enlargements of one area of the photograph, until a grainy close-up reveals a body in the grass and a man with a gun hiding in the trees.

Blowup: Photographer examining the photo he took in the park.

Around the same time as this film was used, computer science researchers were already developing the new field of digital image processing, including the algorithms for image enhancement such as sharpening of edges, increasing contrast, noise reduction, and blur reduction. The early articles in the field show the blurry photographs taken by surveillance planes which were sharpened by the algorithms. Today many of these techniques are build into every image manipulation software such as Photoshop, as well as in the lens and firmware of digital cameras. They become essential to both consumer photography, and commercial visual media, as every published photograph first goes through some software manipulation.

Typical example of digital image processing taken from a textbook. Top: the original image captured by the camera mounted on a plane. Bottom: the image after it was automatically processed by software.

The still frame from the Blowup.

The same frame adjusted in Photoshop.

Contemporary digital cameras - both dSLR and high-end compacts - can record images both in JPEG and RAW formats. In the former, an image is compressed, which limits the possibilities for later extraction of additional information using software. Raw format contains the unprocessed data from the camera's image sensor. Raw format assumes that a photographer will later manipulate the photo in software to get best results from the millions of pixels recorded by camera. "Working with a Raw file, you’ll have a greater chance of recovering highlights in overexposed clouds, rescuing detail from areas of shadow or darkness, and correcting white balance problems. You’ll be able to minimize your images' noise and maximize their sharpness, with much greater finesse." (William Porter, Raw vs. JPEG: Which should you shoot and when?, 9/19/2012.)

This means that new software can generate new information from a photo in Raw format captured years early. The following example from Porter's article demonstrates this dramatically (go to the article link to see high resolution photos; the captions are from the article):

Porter: "JPEG straight from the camera, a high-end compact (Panasonic DMC LX3). It’s a nominally decent exposure, preserving highlight detail at the expense of detail in the shadows. But it’s an unusable photo."

Porter: "Fortunately, I was shooting Raw + JPEG at that moment. This is the best version I got from processing the Raw file myself in Adobe Lightroom 3."

Porter: "A year after I took the photo, I discovered the amazing, Mac-only Raw Photo Processor. Running the Raw file through RPP, I was able to recover detail that even Lightroom’s converter hadn’t found."

"Software epistemology" also includes ability to generate new information from old analog data. For example, years ago I saw a demo at SIGGRAPH where a few film shots from the 1940s Hollywood film were manipulated in software to generate the same shots as seen from a different point of view.

Another important type of software epistemology is fuzing data sources to create the new knowledge which is not explicitly contained in any of them. Using the web, it is possible to create a description of an individual by combining piece of information from his/her various social media profiles and making deductions form them.

Combining separate sources can give additional meanings to each of the sources. Consider the technique of stitching a number of separate photos into a single panorama available in many digital cameras. Strictly speaking, the underlying algorithms do not add any new information to each of the images (their pixels are not changed). But since each image can now becomes a part of the larger whole, its meanings for a human observer change.

The abilities to generate new information from the old data, fuse separate information sources together, and create new knowledge from analog sources are just some techniques of "software epistemology." In my future publications and notes I am hoping to gradually describe others - as I am mastering the appropriate algorithms and their applications.

1 comment:

  1. Dear Lev,

    Thanks for this wonderful post.

    In the Berkeley Folk Music Festival project I am working on (, I've been thinking a lot about the ways in which one might produce sonic information from visual materials once images are digitized. This could be everything from whimsical transformations of existing data into new outputs (along the lines of Voyant Bubbles, to, as you put it, fusing various kinds of data together—existing recordings, images of musicians performing, the acoustic qualities of particular spaces—to try to "hear" an image or "see" a song or performer or performance style in a new way. Is there a way, in other words, to put the rhythm into algorithms? Digital sonification studies?

    What your post made me think about is this: it strikes me that the key is algorithms and how humanities scholars might access, explore, probe, develop them. Is there a kind of open source algorithm resource needed for digital humanities scholars? Ways for DH scholars to probe, think through, use, theorize, experiment with algorithms in more robust ways? Some kind of platform or software or shared digital space for this?

    Thanks again. I look forward so much to your explorations of software epistemology, particularly in relation to the ways in which it can bring together humanities-shaped critical thinking with algorithmic thinking.



Note: Only a member of this blog may post a comment.