I pioneered the approach of modifying an image using abstract attributes or "semiotics." The approach enables for example, modifying an image to look more ominous, futuristic, cozy, etc. The approach included training a classifier on an abstract concept and using it to score and navigate samples from the latent space of a Generative Adversarial Network. Once OpenAI's CLIP was released, the approach was generalized to include any concept, and Paint by Word was born.
Words flow onto canvas like raindrops,
Her words like paint splatter,
Her words like rain, pitter patter,
Casting on canvas a moving image
Blurring the line between words and visual identity.
Inspired by Magritte’s "The Treachery of Images" (This is not a pipe) the speaker in the video reads a poem that challenges the relationship between representation and reality
Renders a recurrent portrait of the speaker in her spoken words. paint splatters:
Simple Setup + Code >
A camera captures the speaker's image film and audio.
Speech recognition converts their spoken words to text.
Processing to run a simple script to convert pixels to text.
An algorithm positions the words to trace the contours of their face
The portrait continuously evolves as she speaks.
2020, presented at MIT CSAIL Embodied Intelligence Workshop
I pioneered the approach of modifying an image by its "semiotics," for example, modifying an image to appear more formal, comfortable, clinical, etc. This was novel as it enabled image modification using abstract concepts rather than merely tangible ones. The approach included training a classifier on an abstract concept and using it to score and navigate samples from the latent space of a Generative Adversarial Network.
The Pesaran Lab at NYU’s Center for Neural Science is working to develop advanced brain-machine interfaces that enable dexterous control of robotic systems. As part of the lab’s research agenda, I am developing a real-time virtual avatar using data from a state-of-the-art motion capture system that tracks all the movements of the human arm and hand, comprising 27 joint angles during object reach and grasp.
The skeleton data is generated from an infrared system that captures motion data. This data is then processed in Cortex to calibrate bone segments and SIMM to resolve joint angles. I use this segmental model to define the degrees of freedom of a virtual avatar in 3DS Max. The result is presented in Vizard. The avatar responds to real-time arm and hand movements of a subject. As the subject moves in physical space, the avatar moves in tandem in the virtual space. The system renders the virtual arm in real-time at ~20 Hz.
Role : Producing the real-time graphic rendering of the arm by skinning the skeletal model to a 3D mesh. Defining degrees of freedom using a Hierarchical Translation Rotational model for joint angles. Ensuring system performance during live subject tests.
Describe lab
LISP based programming language in Urdu. This work investigates how programming in the Urdu language may yield different programming styles.
الف (pronounced A-lif) is a programming language written in Urdu, the national language of Pakistan.
You can try it out here.
الف is a tree structured LISP based interpreter written in JavsScript using PEG.js . Its tree structure allows it to support multiple nested arguments.
Currently, الف can multiply, divide, and subtract. It is able to compare two values and execute true cases of an an if statement. Using Processing.js, it can also create rectangles and ellipses.
The long term goal of this project is to explore how linguistic structures affect programmatic reasoning. In particular, I am interested in exploring how the grammar of a language can produce different structures in programmatic thinking.
IDE currently in development by Brian Clifton
