I pioneered the approach of modifying an image using abstract attributes or "semiotics." The approach enables for example, modifying an image to look more ominous, futuristic, cozy, etc. The approach included training a classifier on an abstract concept and using it to score and navigate samples from the latent space of a Generative Adversarial Network. Once OpenAI's CLIP was released, the approach was generalized to include any concept, and Paint by Word was born.
2020, presented at MIT CSAIL Embodied Intelligence Workshop
I pioneered the approach of modifying an image by its "semiotics," for example, modifying an image to appear more formal, comfortable, clinical, etc. This was novel as it enabled image modification using abstract concepts rather than merely tangible ones. The approach included training a classifier on an abstract concept and using it to score and navigate samples from the latent space of a Generative Adversarial Network.
The Pesaran Lab at NYU’s Center for Neural Science is working to develop advanced brain-machine interfaces that enable dexterous control of robotic systems. As part of the lab’s research agenda, I am developing a real-time virtual avatar using data from a state-of-the-art motion capture system that tracks all the movements of the human arm and hand, comprising 27 joint angles during object reach and grasp.
The skeleton data is generated from an infrared system that captures motion data. This data is then processed in Cortex to calibrate bone segments and SIMM to resolve joint angles. I use this segmental model to define the degrees of freedom of a virtual avatar in 3DS Max. The result is presented in Vizard. The avatar responds to real-time arm and hand movements of a subject. As the subject moves in physical space, the avatar moves in tandem in the virtual space. The system renders the virtual arm in real-time at ~20 Hz.
Role : Producing the real-time graphic rendering of the arm by skinning the skeletal model to a 3D mesh. Defining degrees of freedom using a Hierarchical Translation Rotational model for joint angles. Ensuring system performance during live subject tests.
Describe lab
LISP based programming language in Urdu. This work investigates how programming in the Urdu language may yield different programming styles.
الف (pronounced A-lif) is a programming language written in Urdu, the national language of Pakistan.
You can try it out here.
الف is a tree structured LISP based interpreter written in JavsScript using PEG.js . Its tree structure allows it to support multiple nested arguments.
Currently, الف can multiply, divide, and subtract. It is able to compare two values and execute true cases of an an if statement. Using Processing.js, it can also create rectangles and ellipses.
The long term goal of this project is to explore how linguistic structures affect programmatic reasoning. In particular, I am interested in exploring how the grammar of a language can produce different structures in programmatic thinking.
IDE currently in development by Brian Clifton
