Painting on Things, in VR (Part One)

A while ago, artist/engineer/cool-dude Barak and I made a little "Venice inspired" VR scene, using Unreal Engine 4. It had an interesting art style that I still don't quite know how to describe. I think "Black-And-White" is closest, but colloquially that often means "shades of grey", and this style didn't use grey. Just pure black, and pure white. Here's an example of the style, not set in Italy this time:

We certainly weren't the first to work with this "black-and-white" look (go play Antichamber), but it does look kinda cool, if I do say so myself. We showed the original scene to some friends and family (it was actually made as a gift for Barak's then-fiancé), and somewhere along the line, somebody said a very inspiring sentence:

"Haha, it looks just like a coloring book."

Man, it kind of does, huh? And wouldn't that be really fun, a coloring book in VR? We thought so. A coloring book that you can stand inside of. The only problem with the idea is that someone has to actually make it, and that someone is me.

I started in UE4, because I already had our black-and-white style working there, and because it has an easy API for drawing into render targets, getting UV coordinates from hit tests, etc:

So I spent a few hours getting a basic prototype working. In short, I made a special actor which represents a mesh that can be drawn on. While the user is drawing, the system runs a line trace every frame from the end of their brush. If the object can be drawn on, and is the same object as last frame, the system draws a line from the old to the new point in the render target. Pretty simple, especially since UE4 has a project setting to enable computing UV coordinates from a hit result.

There are a few disadvantages though. For one, this only supports fully opaque brushes, with no anti-aliasing. It also just sweeps a square in order to form the line, so the endpoints are uh, pointy. The biggest issue though is that it draws the line in UV space, which is not continuous in screen (brush?) space. That means that if you try to draw over a UV seam, it takes the shortest path through the object UVs, which is not necessarily the shortest path in world space. It also means that if you use a brush that is much larger than the geometry, painted strokes can sort of spill over into unrelated parts of the object.

I thought about trying to partially correct for this by world projecting the textures, but I do want this system to support concave objects. For now, I elected to ignore the problem; Just don't draw the line if the UV-space length is too long. This doesn't really fix the issue, and also doesn't solve the spill-over problem for large brushes, but it does mitigate the first issue a bit. The real, robust solution is a bit complicated, and involves a couple extra render targets to project from screen (brush?) space back into UV space. A good subject for another post.

So as mentioned above, instead of stopping to solve these problems, I ignored them, pushing forward with a nice rounded brush by drawing my own custom geometry instead of just "lines". Because I'm specifying the vertices of the brush stroke, I get to also supply texture coordinates and colors. In practice, this means that both brush size and color can be nicely interpolated along a brush stroke:

It feels better already. I imagine the next steps in a good feeling solid brush will be some sort of spline interpolation - maybe a cubic spline based on some history of points. There's also the problem of tracking jitter and hand shakiness at a distance, which I've alleviated in the past with the help of some low-pass filtering. My plan is to try out different combinations of size/location interpolation and filtering, and see what sorts of brushes (pencil, fountain pen, solid marker, etc) I can come up with.

Oh and I'll leave you with two more images. One is a first-pass at soft brushes, and the other is a scribbled-on version of the scene above, using this little painting prototype:

I definitely plan to keep working with this project. It's a ton of fun so far, both to develop and to play with. Things may take a bit of a detour into more heavy graphics programming, while I figure out how to make this stuff scale to a whole scene. Render Targets in UE4 obfuscate a lot of what's actually going on in terms of memory, plus I can't get runtime mipmap generation to work without making some engine changes. Expect some more posts about this, especially since there's also a fair bit of work done on a custom shading model for the engine that should help improve the outline shader, among other things.

So stay tuned!

Everything I Did in the Years Before 2020

Even though the date on this post says 2019, I'm actually compiling it here in early 2020. This is my solution to a sort of annoying problem, which is that I've gone and spent several years doing, learning, and making stuff, without having the foresight to put any of it up online. It's a double-edged sword - 80% of what I've done is now embarrassing to look at, and thankfully, I get to retroactively decide not to show it to you. On the other hand though, that last 20% is a huge catalog of dead, unfinished, or unremarkable projects, and they're doomed to be lost as soon as I run out of disk space.

Instead of trying to fill out a blog post for each of them, I've decided to dump a whole bunch of these projects into this one Super Duper MegaPost (tm). Not everything made the cut, obviously. To be honest, most of my projects get deleted after I give up on them, so some stuff really is lost forever. Again, as a disclaimer, many of these projects are still embarrassing to look at for all the usual reasons. They're old, and I didn't know what I was doing yet, and blah blah blah. They represent my first five-ish years of learning, messing up, and starting things I obviously couldn't finish. I figure that learning from your mistakes is cool and all, but learning from other peoples' mistakes is even cooler - so I'll post them here anyway.

Plus I didn't only make mistakes. Sometimes I made stuff that was almost sort of interesting.

Oh, and in addition to this mega-post, I'll be uploading my game jam submissions separately, since they at least represent "finished" projects. So keep scrolling down to see those.


And now, on with the show:

---

At one point in the somewhat recent past, I found out that I could re-live my childhood dream of beating Super Monkey Ball 2, with the help of an emulator. After about half an hour though, I gave up on the dream again, because I realized it would be more fun to implement the mechanics and screw around in UE4. We actually developed this into a fun little game concept, where you can morph between a few different shapes, each of which has different movement mechanics and physical properties.

Anyway the morphing uses basic blend shapes (morph targets in UE4), although the bouncing deformation in the second video is just a material effect. The movement itself, along with the bounce physics, were tricky to implement. I didn't want to rely on the physics system to move around, since it's inconsistent and a bit unreliable at high speeds; So I wrote a custom character controller in C++, and made a vague attempt to properly compute the bounce forces for the soft body. I keep promising myself that someday I will revisit the concept, becuase it really would be a cool game.

I've played a fair bit with splines in UE4 over the years, and there are two main challenges. The first is that a spline can have many points, but a spline mesh can only have two. You always have to hack together a system that skins a spline with an array of meshes, which is a bit of a pain. The second is that I swear somewhere in the implementation is a rotation normalization bug, because the only reliable way to control spline roll without introducing kinks as you cross coordinate axes is to compute the roll manually. Anyway, I was playing with an ability that would grow vines over stuff, and it worked reasonably well.

Coincidentally, for the same project, I got to dive in and play a lot with character control and movement abilities. This is easily my favorite part of gameplay programming; I'm fascinated by movement systems in platformers, and "skill-based" modes (like surf and bhop in Source Engine games). I tuned the movement to be pretty fun, extended the player controller with niceties like crouching/sliding/sprinting, and also added a couple of dash abilities:

At one point I decided to do ocean water, for some reason (The reason was that it is very fun). There are a few methods, of varying complexity, and I pretty much implemented them all. The prettiest one uses a compute shader for FFT, to generate textures for offset, gradient, and folding. I want to stress that I did not write the FFT shader, although I spent hours and hours in my university library trying to wrap my head around the math. I just adapted an old UE4 plugin (like, version 4.2 old) that handled the FFT. My actual goal was to read back the displacement texture to compute buoyancy on the game thread, but at the time I didn't know enough about the rendering pipeline to make that happen.

A little known fact is that since Minecraft came out, every novice engine programmer goes through a voxel phase. I was no exception, and at some point I got super into runtime terrain meshing. I used the Runtime Mesh Component plugin to do your basic metaballs example with marching cubes. That plugin was super interesting, and I was sad when it stopped being maintained. Thankfully though, it has since been resurrected, and now has made some seriously cool improvements; I really want to poke around with it again sometime. Oh and the holes in the mesh are just where it clips the simulation bounds, that's not a bug in the meshing.

Just like every engine programmer goes through a voxel phase, every novice tech artist goes through a toon shading phase. This one is a pretty standard post-process effect for outlines and cel shading. In hindsight, this sort of effect is extremely wasteful; UE4 does a tremendous amount of work (and uses a tremendous amount of video memory) for the PBR pipeline, and this effect throws most of that work away. I've since done the same effect with what I think is a better approach, involving a custom shading model for the engine. The new one also supports all light types (not just directional), and handles shadows more gracefully. Regardless, I actually made a YouTube tutorial for the older effect (on top of the pile of existing ones), which was full of bad, wrong information, but someone took it and made a short film with it, which was awesome!

In addition to all the UE4 stuff, I've started and then abandoned countless graphics engines, usually built around the Vulkan or D3D12 APIs. Invariably, I get far enough along to realize that actually I've structured things poorly, and then I start again. I do get a bit further every time though, so I've got that going for me, which is nice. Here's a couple random examples of old and dead engines, none of which do anything remotely remarkable:

OK this is getting far too long, and there are so many more dead projects to talk about, but they just get less and less polished from here. Instead of padding out this post any further, I'll drop a couple of unlisted YouTube links, which are mostly just me talking through some internal tools stuff, or showing other team members what I've been up to. These were never meant for public eyes, so consider it a "behind the scenes" look, from years ago when I didn't know what I was doing (I still don't, but hey).

• Talking about sculpting rocks
• Old ocean water
• Some character animation stuff (stock animations, but custom models/particles)

Finally, here are some fun extras presented without context: