Thursday, January 2, 2020

Painting on Things, in VR (Part One)

A while ago, artist/engineer/cool-dude Barak and I made a little "Venice inspired" VR scene, using Unreal Engine 4. It had an interesting art style that I still don't quite know how to describe. I think "Black-And-White" is closest, but colloquially that often means "shades of grey", and this style didn't use grey. Just pure black, and pure white. Here's an example of the style, not set in Italy this time:

An example of coloring book style art for VR


We certainly weren't the first to work with this "black-and-white" look (go play Antichamber), but it does look kinda cool, if I do say so myself. We showed the original scene to some friends and family (it was actually made as a gift for Barak's then-fiancé), and somewhere along the line, somebody said a very inspiring sentence:

"Haha, it looks just like a coloring book."

Man, it kind of does, huh? And wouldn't that be really fun, a coloring book in VR? We thought so. A coloring book that you can stand inside of. The only problem with the idea is that someone has to actually make it, and that someone is me.

I started in UE4, because I already had our black-and-white style working there, and because it has an easy API for drawing into render targets, getting UV coordinates from hit tests, etc:





So I spent a few hours getting a basic prototype working. In short, I made a special actor which represents a mesh that can be drawn on. While the user is drawing, the system runs a line trace every frame from the end of their brush. If the object can be drawn on, and is the same object as last frame, the system draws a line from the old to the new point in the render target. Pretty simple, especially since UE4 has a project setting to enable computing UV coordinates from a hit result.

There are a few disadvantages though. For one, this only supports fully opaque brushes, with no anti-aliasing. It also just sweeps a square in order to form the line, so the endpoints are uh, pointy. The biggest issue though is that it draws the line in UV space, which is not continuous in screen (brush?) space. That means that if you try to draw over a UV seam, it takes the shortest path through the object UVs, which is not necessarily the shortest path in world space. It also means that if you use a brush that is much larger than the geometry, painted strokes can sort of spill over into unrelated parts of the object.

I thought about trying to partially correct for this by world projecting the textures, but I do want this system to support concave objects. For now, I elected to ignore the problem; Just don't draw the line if the UV-space length is too long. This doesn't really fix the issue, and also doesn't solve the spill-over problem for large brushes, but it does mitigate the first issue a bit. The real, robust solution is a bit complicated, and involves a couple extra render targets to project from screen (brush?) space back into UV space. A good subject for another post.

So as mentioned above, instead of stopping to solve these problems, I ignored them, pushing forward with a nice rounded brush by drawing my own custom geometry instead of just "lines". Because I'm specifying the vertices of the brush stroke, I get to also supply texture coordinates and colors. In practice, this means that both brush size and color can be nicely interpolated along a brush stroke:




It feels better already. I imagine the next steps in a good feeling solid brush will be some sort of spline interpolation - maybe a cubic spline based on some history of points. There's also the problem of tracking jitter and hand shakiness at a distance, which I've alleviated in the past with the help of some low-pass filtering. My plan is to try out different combinations of size/location interpolation and filtering, and see what sorts of brushes (pencil, fountain pen, solid marker, etc) I can come up with.

Oh and I'll leave you with two more images. One is a first-pass at soft brushes, and the other is a scribbled-on version of the scene above, using this little painting prototype:


A simple cube painted on with a prototype soft brush
VR coloring book example, scribbled in a bit this time



I definitely plan to keep working with this project. It's a ton of fun so far, both to develop and to play with. Things may take a bit of a detour into more heavy graphics programming, while I figure out how to make this stuff scale to a whole scene. Render Targets in UE4 obfuscate a lot of what's actually going on in terms of memory, plus I can't get runtime mipmap generation to work without making some engine changes. Expect some more posts about this, especially since there's also a fair bit of work done on a custom shading model for the engine that should help improve the outline shader, among other things.

So stay tuned!

Tuesday, December 31, 2019

Everything I Did in the Years Before 2020

Even though the date on this post says 2019, I'm actually compiling it here in early 2020. This is my solution to a sort of annoying problem, which is that I've gone and spent several years doing, learning, and making stuff, without having the foresight to put any of it up online. It's a double-edged sword - 80% of what I've done is now embarrassing to look at, and thankfully, I get to retroactively decide not to show it to you. On the other hand though, that last 20% is a huge catalog of dead, unfinished, or unremarkable projects, and they're doomed to be lost as soon as I run out of disk space.

Instead of trying to fill out a blog post for each of them, I've decided to dump a whole bunch of these projects into this one Super Duper MegaPost (tm). Not everything made the cut, obviously. To be honest, most of my projects get deleted after I give up on them, so some stuff really is lost forever. Again, as a disclaimer, many of these projects are still embarrassing to look at for all the usual reasons. They're old, and I didn't know what I was doing yet, and blah blah blah. They represent my first five-ish years of learning, messing up, and starting things I obviously couldn't finish. I figure that learning from your mistakes is cool and all, but learning from other peoples' mistakes is even cooler - so I'll post them here anyway.

Plus I didn't only make mistakes. Sometimes I made stuff that was almost sort of interesting.

Oh, and in addition to this mega-post, I'll be uploading my game jam submissions separately, since they at least represent "finished" projects. So keep scrolling down to see those.


And now, on with the show:



At one point in the somewhat recent past, I found out that I could re-live my childhood dream of beating Super Monkey Ball 2, with the help of an emulator. After about half an hour though, I gave up on the dream again, because I realized it would be more fun to implement the mechanics and screw around in UE4. We actually developed this into a fun little game concept, where you can morph between a few different shapes, each of which has different movement mechanics and physical properties.

Anyway the morphing uses basic blend shapes (morph targets in UE4), although the bouncing deformation in the second video is just a material effect. The movement itself, along with the bounce physics, were tricky to implement. I didn't want to rely on the physics system to move around, since it's inconsistent and a bit unreliable at high speeds; So I wrote a custom character controller in C++, and made a vague attempt to properly compute the bounce forces for the soft body. I keep promising myself that someday I will revisit the concept, becuase it really would be a cool game.






I've played a fair bit with splines in UE4 over the years, and there are two main challenges. The first is that a spline can have many points, but a spline mesh can only have two. You always have to hack together a system that skins a spline with an array of meshes, which is a bit of a pain. The second is that I swear somewhere in the implementation is a rotation normalization bug, because the only reliable way to control spline roll without introducing kinks as you cross coordinate axes is to compute the roll manually. Anyway, I was playing with an ability that would grow vines over stuff, and it worked reasonably well.





Coincidentally, for the same project, I got to dive in and play a lot with character control and movement abilities. This is easily my favorite part of gameplay programming; I'm fascinated by movement systems in platformers, and "skill-based" modes (like surf and bhop in Source Engine games). I tuned the movement to be pretty fun, extended the player controller with niceties like crouching/sliding/sprinting, and also added a couple of dash abilities:





At one point I decided to do ocean water, for some reason (The reason was that it is very fun). There are a few methods, of varying complexity, and I pretty much implemented them all. The prettiest one uses a compute shader for FFT, to generate textures for offset, gradient, and folding. I want to stress that I did not write the FFT shader, although I spent hours and hours in my university library trying to wrap my head around the math. I just adapted an old UE4 plugin (like, version 4.2 old) that handled the FFT. My actual goal was to read back the displacement texture to compute buoyancy on the game thread, but at the time I didn't know enough about the rendering pipeline to make that happen.






A little known fact is that since Minecraft came out, every novice engine programmer goes through a voxel phase. I was no exception, and at some point I got super into runtime terrain meshing. I used the Runtime Mesh Component plugin to do your basic metaballs example with marching cubes. That plugin was super interesting, and I was sad when it stopped being maintained. Thankfully though, it has since been resurrected, and now has made some seriously cool improvements; I really want to poke around with it again sometime. Oh and the holes in the mesh are just where it clips the simulation bounds, that's not a bug in the meshing.





Just like every engine programmer goes through a voxel phase, every novice tech artist goes through a toon shading phase. This one is a pretty standard post-process effect for outlines and cel shading. In hindsight, this sort of effect is extremely wasteful; UE4 does a tremendous amount of work (and uses a tremendous amount of video memory) for the PBR pipeline, and this effect throws most of that work away. I've since done the same effect with what I think is a better approach, involving a custom shading model for the engine. The new one also supports all light types (not just directional), and handles shadows more gracefully. Regardless, I actually made a YouTube tutorial for the older effect (on top of the pile of existing ones), which was full of bad, wrong information, but someone took it and made a short film with it, which was awesome!





In addition to all the UE4 stuff, I've started and then abandoned countless graphics engines, usually built around the Vulkan or D3D12 APIs. Invariably, I get far enough along to realize that actually I've structured things poorly, and then I start again. I do get a bit further every time though, so I've got that going for me, which is nice. Here's a couple random examples of old and dead engines, none of which do anything remotely remarkable:






OK this is getting far too long, and there are so many more dead projects to talk about, but they just get less and less polished from here. Instead of padding out this post any further, I'll drop a couple of unlisted YouTube links, which are mostly just me talking through some internal tools stuff, or showing other team members what I've been up to. These were never meant for public eyes, so consider it a "behind the scenes" look, from years ago when I didn't know what I was doing (I still don't, but hey).


Finally, here are some fun extras presented without context:




Tuesday, February 19, 2019

UE4 Winter Jam 2019: Coup d'Amour

After having to miss the fall UE4 jam last year, we were determined to make something happen in the winter jam. School schedules still managed to get in the way, but in the end we found time for a fun little project.

I tend to wear a lot of hats for game jams; I primarily did gameplay programming, but also some UI stuff, sound, particles, and general tech art stuff. With school getting in the way I really only got to spend 3ish days out of the 5.

Andrew did sound and music, as usual, which turned out really well. He really only had the weekend (two days), and then had to take off back home. We had an emergency in the last 24 hours with music, and he had to compose something really fast using GarageBand on a  2011 iMac, which was quite impressive.

Barak did his usual amount of art, which is to say, "all of it". He also only had 2 or 3 days, because on top of school, he had to go attend a wedding in the middle of it all.

We also got Will in for this one. He did a lot of UI design and programming, and also did some great work with the trailer and submission. The trailer is mostly limited by my ability to supply him footage near the deadline. Like the rest of us, he only had around 3 days to pitch in.

The theme this time around was `All's Fair in Love and War`, and we decided to submit something in the `Friends and Foes` category (a shared-screen multiplayer game). Will came up with the game's name; It's a play on coup de' grâce, which is "killing blow" or "blow of mercy" in French. Coup d'Amour is "blow of love", which is fun considering the jam's theme (He also checked it with a native French speaker to make sure the wordplay is valid, so hopefully we aren't making fools of ourselves there).

So there you have it. Coup d'Amour is a local multiplayer game for four players:

controls for the game
screenshot of the game



It's a twin-stick shooter, and you all play on the same screen. The rules are pretty simple:

  • You play in teams of two, as king and queen. You're attached to each other with a rope (it's a metaphor for your love, or something).
  • To win, use your crossbow to sever the other team's rope.
  • There are two colors of crossbow bolts; You shoot one, your teammate shoots the other.
  • You have a shield, but it only blocks one color. Use it to protect your partner, and hopefully they'll protect you too.
  • Your shield will break if it takes too much damage. That wouldn't be good.
  • Blocking damage makes your teammate's shots more powerful. If you charge them up enough, they could break right through an enemy shield.
  • You can shield bash the enemy if they get too close. This costs some shield durability, but will stun your enemy.
  • Grab powerups to give you a shield-ignoring fireball, or regenerate your own shield's durability.

These rules make for a pretty fun kind of rock/paper/scissors style gameplay. You want to block shots to charge up your partner, but not so many shots that you lose your own shield. To win, you can out-play, out-smart, or just out-shoot the enemy team. Since you're attached to your partner though, you'll have to coordinate where and when you move or attack. Play too timidly, and you'll find yourself with a broken shield. Too aggressively, and the enemy will do too much damage for you to block.

Here's a quick video to give you a sense of the gameplay, and I apologize that it has no sound (we were in a rush near the deadline, as you might imagine):

We had a ton of fun working on the project - the logic is pretty complex, with a lot of moving parts, and all of the art save for some sounds was done by hand (although the far-off camera angle doesn't really do it justice, sorry Barak).

If you'd like to download and play it, and you have four gamepads and a Windows computer, you can check it out on my Itch.io page here. Unfortunately it does require four gamepads for you to get past the lobby screen, a fact which I fear may have made it difficult to judge for the jam.

Tuesday, August 21, 2018

UE4 Summer Jam 2018: Albina (VR Category Winner!)

A while back we did an informal, air-quotes "game jam" in my basement. It went well, and we had a ton of fun, so we decided to enter a "real" jam over the summer. We picked the UE4 Summer Jam, since we happen to be using UE4 anyway. The theme was `Well that escalated quickly`, and what's amazing to me is that we (and by we I mean Barak) had the core game idea an hour before the theme announcement. We were sitting around, setting up PCs and brainstorming random directions to go, and Barak pitched this great idea for a high pressure room-escape game. You play as one of the dogs sent in to space by the Soviet Union, trying to keep your spaceship from falling apart while you simultaneously try to open the hatch and escape.

We all loved the idea, but we sort of dismissed it. It was super specific, so we thought it was unlikely to fit the theme; A real shame, since it was a cool concept.

The announcement came though, and we all burst out laughing. We tried to sit and workshop some other ideas, but we knew pretty quickly which one was the winner, and we got to work. The game is called Albina, and other than some major packaging bugs in the last 8 hours of the jam, I'm pretty proud of it:

first-person view of the space capsule
view of one puzzle with the power shut off
alternate angle of the space capsule
some in-game hints for a puzzle

The setup is this: You play as Albina, the soon-to-be second dog in space. When the first (Laika) doesn't return, you realize it's a one way trip. Now you have to get the parachute and escape the capsule before it leaves the atmosphere.

In essence, it's an escape room, with a bunch of connected puzzles that you can complete in any order. You could work backwards from the goal, and try to figure out what you would need to open the hatch (You see it's connected to a keypad, but the keypad doesn't have power). Alternatively, you could just start using things that you find (you've got a hammer, now what objects are hammer-able?).

The twist on the escape room genre is that as you solve puzzles, things start going wrong. The radiator springs a leak, or the power cuts off, and you must solve those problems before they kill you. As you continue, the number of problems... well it escalates. There's a lot of escalation in fact; The rocket escalates, your heart rate escalates, the number of concurrent alarm sounds escalates, and the list goes on.

Here's the video we submitted as part of the jam. It's not the most polished, for obvious reasons, but it shows a bit of the gameplay:



Like most of them, this was a super fun project! As our first "real" game jam, we made a lot of mistakes, most of which didn't reveal themselves until the final day. We entered the final stretch only to find that 8+ hours of debugging the "Simon Says" logic still hadn't fixed it, packaging the game broke all UI functionality, and sometimes the game would just kill you for no reason. We almost couldn't even submit the game, because of two very strange problems in the packaged build:

  1. We somehow corrupted a couple of pretty important blueprint assets (primarily the GameState blueprint). This was totally our fault, because in our *incredible* wisdom, we didn't set up proper source control. Instead, we shuffled project files around a Google Drive folder, and manually ran the asset migration tool to move blueprints around. I don't recommend this strategy, just FYI. Whatever the problem was, it caused some packaging errors related to paths that didn't get normalized, and I had to spend a couple of hours just trying to get UE4 to output an executable. Oops.
  2.  There are some quirks with UE4's Widget Interaction Component, which is generally the way to handle menus in VR - you use a laser pointer. I personally prefer to use physical buttons and levers instead, when possible, but in a game jam setting there isn't always time for that. For some reason in the packaged build, the enable/disable rules for widget components weren't being respected, so you couldn't actually press any of the menu buttons. After 4 or 5 hours of trying to fix this, we settled on the clean and extremely non-hacky solution of teleporting all the menus that weren't interactable a mile underground until the player needed to see them. Hey, if it works it works.

Because of these slowdowns, we just had to strip some things out. I expected to be leaving features unimplemented, but in this case it hurt even more because we had to cut features that were already implemented. We had to scrap the seated mode, pause menu, and victory/loss UI elements, which is just a shame. Technically to beat the game you don't even need the parachute, because the "you jumped out without a chute" UI had to be cut. Oh and there's only 4 rounds of Simon Says instead of 5 (it's complicated).

Regardless, we're pretty proud of the game. The core logic, and all of the little polish elements really came together in the end, and it all makes for a ton of chaotic fun. If you'd like to play it, you'll need a VR capable Windows machine, and you can check it out on my Itch.io page.

Oh, and I almost forgot to mention - the game won the VR/AR category, which totally blew us away! The hurricane here in NC delayed the results a bit, so we almost forgot to even check! I guess it's not exactly an Oscar, especially since there were only around 15 VR submissions, but for a first entry I'm really pleased. We cheered for a bit after getting the results, and then sort of said "Huh, yeah, so I guess we just uh, go back to doing homework now". And then that's what we did.