ML Eyes - drawing rigs with machine learning
This is an idea I prototyped to control animation rigs with drawings! This is done with machine learning and convolutional neural nets using PyTorch. I'm starting small on purpose to establish a proof of concept, but hopefully I can make it do bigger things later on!
The technique I used is to generate a bunch of synthetic training data by randomizing a rig's controls and rendering images based on those. After that, I trained a CNN model using supervised learning on image/controls pairs.
WIP - Snake
Work in progress!
I'm experimenting with a ML workflow to render a character and post-process passes such as outlines and shading.
I modeled the snake in Blender and rigged it in Houdini. I'm still working out the exact workflow, but I'm using ComfyUI and writing custom nodes for it as a starting point.
USD was used for the Blender -> Houdini integration. This was nice because it allowed me to iterate on the mesh and to easily isolate parts of the geometry while rigging.
Math surface compiler
A work-in-progress tool I'm making to compile math expressions into multiple targets such as Houdini VEX, Blender Geometry nodes, WebGL, Cranelift, USD, numpy, and maybe others!
The primary language for this project is Rust. I use "pest" (a PEG parser library) to parse the Desmos/LaTeX-like expressions into an AST. A lowered version of the AST is sent to a VEX backend, which represents the expressions in static single assignment form.
The UI you're seeing is a web app that uses Mathquill for the input boxes (the same text boxes that Desmos uses).
Once this is in sturdier shape, I plan to open source it!
Rust cubes Demo
2024 update: WebAssembly port! www.otterstack.com/rust-cubes-demo/
My first real project in the programming language Rust! I was learning Rust while working on this project in 2014. The most recent update in 2024 compiles this project with Miniquad, which allows it to run in the browser!
It's a colorful and distracting cube, made possible with OpenGL 3, SDL2, GLSL shaders, and spotty linear algebra! (like quaternions, ooooo!)
Houdini to Desmos workflow
I created some art in Houdini and exported it to the Desmos 3D graphing calculator!
The idea is to turn a SOP graph into a Lisp-style intermediate graph, which is then translated into Desmos math expressions (parametric equations, sets of triangles).
I originally started working on this in Maya because I prefer animating in it, but realized that creating/parsing bespoke nodes with its own semantics was more straightforward to do in Houdini. Houdini has its own encapsulation mechanism called HDAs (Houdini Digital Assets) which makes this easier to do.
The custom HDAs I created conveys parametric surfaces such as spheres, bezier surfaces (which can be converted from NURBS), and triangle meshes. Animation uses cubic (hermite) keyframes, and are expressed as piecewise functions in Desmos.
Links to Desmos art created by this so far:
- Bill Cipher Animation. Disks, planes, triangle meshes, cubic (hermite) keyframes.
- OLD Bill Cipher Animation. Uses more parametric surfaces (e.g. the hat is cylinders). A lot slower.
- Utah Teapot. Entirely bezier surfaces.
- Suzanne. Entirely triangles.
- Moomintroll. Disks and bezier surfaces.
USD shell
I wanted to be able to grok, query and modify USD scenes in an interactive shell environment. It's largely inspired by bash and jq (JSON query language).
I made a bespoke DSL based on the .usda syntax. Some cool features include wildcards in prim paths and broadcast assignments (i.e. assign to many).
It's possible to perform an action like "double the radius of every Sphere" in a single line (with some caveats with regards to instancing).
Doom Adding Machine
I wrote an article about it here.
Gonna be real here, this was mostly an excuse to make the punchline: "Can Doom run it?"
I demonstrate that it is possible to run any bounded computation in Doom, minus constraints on level size. I have not proven that Doom is Turing complete.
This works with the vanilla MS-DOS release of Doom 2 (v1.9). No mods or anything!
I love projects like these. I was inspired by the esoteric machines in other games such as Minecraft and RollerCoaster Tycoon.
Piano Auto-Animator
This was my capstone project for the Technical Arts program at BCIT. The objective was to automate the hands of characters playing piano keys. A use case for this would be for shots in films where a character plays the piano for an extended duration of time.
This is provided as a user interface that runs in Autodesk Maya 2022.
The user provides a layout of joints from two hands, and a piano key rig. The user manually positions the tips of the 10 fingers, using the provided numerical curve controls.
The program then solves the positions of the wrists and fingers by keyframing the rotations of joints and translating the wrist. I implemented several approximation methods to get an effect that was visually pleasing.
The original scope was to animate using MIDI keys, but I later reduced the scope for time reasons. Other improvements such as interpolating the keyframes could be made, too.
Overall, it's pretty cool! I learned a surprising (and sometimes obnoxious) amount of detail regarding inverse kinematics and human hand anatomy!
Voxel plugins
This is a OpenUSD file format plugin that reads slab6 (.kvx) and MagicaVoxel (.vox) voxel files natively in OpenUSD.
When a voxel file is loaded, the plugin generates meshes and/or point instances. This makes for convenient iteration between a voxel editor (like MagicaVoxel) and a USD-based workflow (like Solaris).
Jupyter Notebooks for Maya
Brings Jupyter notebooks and a quicker scripting workflow into Maya (2023)! Works in Jupyter Lab and VS Code.
Jupyter notebooks are widely used in the Python data science ecosystem, and it's a workflow I sorely missed while scripting in Maya (let's face it, Maya's built-in script editor isn't that great).
The usual answer to happy scripting in Maya is to move to an IDE workflow (like PyCharm or VS Code), but in my opinion there's still a need for quick experimental scripting as opposed to the write->deploy->test workflow you get when working with IDEs.
Super Mario Bros NES decompilation project
For learning purposes, I used Ghidra to "decompile" the game "Super Mario Bros" into fully-working C code. (note: "decompile" is a loaded term as the output doesn't resemble the original source - the original game was written manually in 6502 assembly!).
This was quite an ambitious project that involved drastically changing the SLEIGH specs that Ghidra uses to interpret the 6502 machine code.
USD deep-dive project
This was a period over a few months of studying USD, both from a practical use-case and technical point of view.
I studied resources including the source code of OpenUSD, various openly-available USD scenes, and experimenting with the CLI tools and workflows.
Some outcomes included:
- Created USD-based tools: A tech demo for an "unravel mesh" tool (GitHub)
- Made railroad diagrams specifying the .usda syntax (.html page)
- Got the Animal Logic "ALab" demo to work with the "Cycles" (Blender) Hydra delegate. Required fixups to accomodate Cycles.
- Compiled 3D libraries such as OpenUSD, OSL, Cycles, and much of the VFX Platform 2023 on my Rocky Linux machine.
- Studied the .usdc file format (fun fact: the token ":-)" exists in every .usdc file as a placeholder for index 0, because it worksaround a bug!)
.usda syntax railroad diagrams
Part of my USD deep dive project.
Can be found here: https://www.otterstack.com/usda.html
I took the bison grammar file from OpenUSD, processed it a bit, and created diagrams!
Railroad diagrams are visual tools that describe the legal grammar of a language. .usda has specific rules for which tokens (words) are allowed to appear in order.
I've found this quite handy for verifying an understanding of the structure of USD files.
- As an example: Where are all the possible places an asset path can appear?
- How is an attribute and a relationship different?
Legacy 3D format to USD
Part of my USD deep dive project.
There was this old piece of shareware software from the 90's called "3Space Publisher" that I grew up using. I reverse-engineered the file format and wrote a program to convert it to USD.
The file format is parsed with Kaitai Struct, which I also used during the reverse-engineering process.
This is one of many of what I call "nostalgia projects". The actual software wasn't very good, but it piqued my interest in computer graphics back then (along with Blender and Lightwave 3D).
This could potentially be a file format plugin that resolves at the asset level, but I don't imagine there's a lot of demand for it! :p
Forced Perspective experiments in Houdini
This is a forced perspective trick I implemented with VEX and some manual math involving projection matrices.
Behind the scenes it transforms points into clip space. In a nutshell, it reshapes the frustum formed by the camera's FOV into a cube, then I squish the cube along Z to make it look flat. I borrowed OpenGL's method (gluPerspective) to calculate the projection matrix. I think you can also use Houdini's toNDC function, but if I recall it lacks the Z component for depth testing. It was also mostly an exercise for myself.
Undertale NES demo
An incomplete NES homebrew recreation of the Lesser Dog battle from Undertale.
I wrote it on-and-off over the course of a month. I used 6502 assembly, as well as a custom Python-based preprocessor to generate code before compilation.
Undertale (2015) is a well-known video game with an established 8-bit aesthetic, so I wanted to see how it could be applied to real NES hardware. As it turns out, it transfers well.
LlamaDB
I wrote an introductory article about it here.
A simple in-memory SQL database I wrote in Rust. This project was not a meaningful attempt to be a real database, but was rather a learning project. I learned about B+Trees and obscure SQL features, mostly.
The project required a confident command of data structures, Lisp-style evaluation, and lexers/parsers.
Mr. Scroll (Ludum Dare 31)
A game for the Ludum Dare 31 jam, written in Rust!
Game jam theme: Entire Game on One Screen
Ludum Dare is a tri-yearly competition where developers are required to develop a game from scratch in 48 or 72 hours. The gimmick for my entry is a single-screen platformer that scrolls across screen boundaries (like Pac-Man), and using the scrolling mechanism as a game mechanic.
All audio is synthesized in-game, and resembles NES-style audio (there are 4 channels after all: 2 pulses, 1 triangle, and 1 noise).
Overall, I got 52nd place out of 2640 in the jam, and 17th place for "Innovation".
NES Emulator
A bare-minimum Nintendo Entertainment System (NES) emulator that I wrote in Java. This was written in two weeks during a slow period at school.
For reference, writing a NES emulator requires:
- Implementing a 6502 microprocessor emulator
- Implementing the graphics chip, or the Picture Processing Unit (PPU)
- Reading lots of unofficial documentation, much of which outlines quirks of the original hardware
King 3D model
A rigged 3D model of the character King from the show "The Owl House". Modeled in Autodesk Maya 2022.
This was course work I did for the "Tech Arts 1" course at BCIT.
I learned a lot about character rigging with this assignment (the sole deliverable for the course).
Well
A 3D model of a well. Modeled in Houdini 18.5. Rendered with Maya and Arnold.
This was course work I did for the "Visual Scripting" course at BCIT.