Update: A Blind Legend is now on Steam!

Hello Everyone !

We can finally talk about the big Unity game project that took almost one and a half year to complete, A Blind Legend.

let’s start with the official game description :

A Blind Legend is a collaborative project of an audio-only action/adventure game for mobile phones. It’s based on a very innovative technology: binaural sound. In this game, the players are guided only by 3D sound and live the adventure by controlling their hero with multi-point tactile gestures. A Blind Legend” is above all a story: an adventure from the Age of Chivalry, to which we want to give epic scope.

In “A Blind Legend”, your eyes will be of no help.
So close them, sharpen your hearing and your blade…
and embark on an epic, perilous rite of passage.

The game is aimed not just at visually impaired (VI) people but also at anyone who’s hungry for an original sensory experience with a trailblazing video game. For us, the game achieves two objectives: giving VI people access to high-quality video games, but also raising public awareness of this type of disability.

What is “A Blind Legend” ? Well it’s a 3D audio game which you can try (and complete) for free. It was created by DOWINO who crowdfunded the project via a french kickstarter-like platform, here’s the link : https://en.ulule.com/a-blind-legend/

You can start downloading the app for iOS or for Android.

There’s even a desktop version of the first level (out of 5) , but that one’s only in french … still here’s the link.

The apps are quite small, but launch them as you need to download content, which is about 130Mb in total of audio content (so if you want to play after you’re done reading this, go ahead now 😀

The game being very rich in what we ended up calling “cinematics” and in dialogue and ambient sounds… The assets are heavy. We’re not working with textures here it’s all audio… And the audio must be of good enough quality.

Ok now that you’ve got the app, and you went in game to download the localized content, let’s go ahead and talk about the development process.

A project with a lot of unknowns :

Now, This is an audio/adventure game. It was presented to us though, as something that we could walk around in like a first person shooter, and also with “melee” combat which must convey a very satisfying feeling when ennemies are killed off – so shouldn’t be too hard. This didn’t sound too complex. I mean, a basic AI to follow the user, areas to launch game events and so on, a first person controller (why not the default controller unity comes with!)

Sure but we first had to prove that the key part of the game, the Audio part, would be ok.

Having an audio engineer in the team, specialized in what we can call binaural sound was very helpful. His being able to design his own signal processing patches in max/msp to show us the ropes and explain the basics of what binaural sound was, ambisonics, HRTF filters, would led us to understand basically what was going on, how we could actually set that up in code.

We first met with Alexandre Rocca (sound engineer that did all the mixing in game and was here to really QA the hell out of sound) , and talked about the entire process of actually going from a single mono source with a relative 3D position in space to the listener, and managing to get out a common stereo output for a user’s normal headphones.

By default, Unity would simply change the volume based on the distance to the object (a behaviour which you have control over with curves in unity AudioSources) and left and right panning is also done realtime by the audio engine by default.

What would binaural sound bring the table though?

Well clearly, as we heard demo’s of his work, there was something you could not get with the default “3D audio” of unity, close to true audio immersion. And there’s a reason why, this is not magic.

What happens is positional data is encoded into the signal of the point sources we place in the world, then at the listener level, with a filter basically, you get to decode this data and the sound, filter it with HRTF filters based on the angle of the sources to us.

HRTF is a set of data that will describe how sound reacts with your head and ears, and how any sound is transformed from its source to the left and right ear. with that data, the transformation can be then applied dynamically in our world.

To get a real description of what HRTF is, please read the wikipedia article – must be most interesting than my naïve description of it.

As I’m not an audio engineer myself, the vocabulary I’m using might be a bit weird, But yes speaking simply, on top of the mono audio signal will be encoded the x y z position relative to the listener.

The key parts of a true but basic complete binaural audio engine (or a plugin to process audio not in realtime) is the first encoder, and then a decoder. That’s a very rough description, actually there are more sub modules… I’m trying not to make this blog post long :p

We have a virtual set of ears, a virtual head through which sound can pass or on which sound can bounce, a virtual set of shoulders too… and our virtual microphones are in both virtual ears, the output of which is going directly to your headphones. That’s the very very dirty way of saying it but really yes, we’re trying to re-create how audio is filtered when bouncing off and going through you and entering your ear holes ;D

So what you are hearing, is as close to reality as possible. It can even be closer if that virtual set of ears (or the HRTF data set) were custom created from your own body/head/ears – which can be done in sound labs in fact by just putting you in the center of a room, mics in your ears, and data taken while sending sounds to the center of the room where you are.

Only for a public game, this couldn’t be done. It was found that with some time to adapt, and in a majority of case, your brain would accept this new set of ears as his own and roll with it.

It’s kind of the problem of presence in the VR world where you have to accept your new set of virtual hands. But clearly, your brain can handle it and quite fast.

So I hope that’s plenty for you to understand the concept of binaural audio and how that would be used in realtime in a game.

More info with Ambisonics – a technique which after application can then be “merged” to a stereo output using HRTF filters. And sound engineers reading this, please go ahead and correct me if the way I’m saying it is misinforming more than anything.

We needed to confirm that this could be done, and could be done fast, the main target of the game, would be mobile devices.

We found some crazy Nasa java applets , or even more obscure positional audio stuff we had difficulty adapting. Aymeric even took the time to look around into a git repository shared by a french research center in computer science and musical creation (CICM) , about High Order Ambisonics – which , to put it so simply that it’ll annoy engineeer : will allow better results. (I highly suggest, if you haven’t read anything so far, that last link to High Order Ambisonic, if you don’t nevermind, I hope I did my best so far).

We would be very late if this had been done from scratch completely, even if we had the team to do so, and an audio engineer who basically had the max/msp patches that would do the job. After that, optimization would need to be done, and as we’ve discussed with him recently too, it’s not only about optimizing the code, it’s also about finding the right equations to approximate stuff, for that a good ear is necessary… In fact, an audio programmer is necessary. And we are no audio programmers though we have now gained knowledge.

So he found something that was announced for unity, 3Dception by Two Big Ears which was amazing to try… and then to work with. They now have solutions for other platforms and other engines, try it out. really. Even for a visual game.

We did a little prototype game with it, saw its potential. Those first beta versions we tested were far from the quality 3Dception is now in fact (it was amazing but now the features coming in are mind blowing).

And Two Big Ears also had , I believe, real trouble working for their unity target as Unity is not really that open – And as we know… unity really just wanted to focus on shaders this last year. I felt like Audio was for them an after thought. During the development though, we saw promise in unity, the audio mixer came in mid-development, amazing news, and today, unity 5.2 which opens up their audio api much more (Sadly this was too late for an october 6 launch though ;D )

So Anyway. This cut down development time by a LOT. And I’d like to thank Varun Nair from the Two Big Ears team for being so patient with me and my problems during development. I never went further than creating a realtime (well it was very slow) lowpass filter… so you know, this was very new.

Now that we have our magical audio filters , everythings done right? I mean come on…

Audio only game play :

So the game has absolutely nothing to show. We just happened to show some last minute things on screen, just because sadly, something with a black screen just means “crash” or “freeze” for most people.

This cloud idea though, is really something that was there since the beginning, as were the screen flashes and visual sword swipes , from the crowdfunded project’s trailer and images, this was something that could not be done without. But it tells you nothing, its visual feedback to you have in audio form and also , the device should vibrate too. But it’s all optional (and can be deactivated).

DOWINO wanted to prove that sword combat would work without any visual hints, as combat was the main driver of the game, and the key part that made this game different. So we had to prototype a basic “move around / get attacked / hit back” scene.

First big mistake, going at it like the game was normal.

As the game was built within a WYSI(roughly)WYG editor, the first approach was to setup a first person controller, create areas where enemies would walk around (and stay in) and those enemies would implement a basic state machine to seek the user as he moves around… and hit him. It was fun… but impossible to play really.

Although it was already decided that only 4 enemies would be able to attack at once in all 4 directions so that the player would not have to be precise when it comes to detecting where the enemy is exactly (because it is really too hard) to swipe at the correct angle or roughly the correct angle, being attacked in 4 directions with each AI having its own state machine running independently, without any visual feedback this was hard. I mean Daredevil can handle that probably

but it was really confusing. And for someone who didn’t know the sounds that would be played, or even how the AI worked, it would be even harder.

A decision was made, that instead of combat being just street combat, everyone piling up on you at once, it would instead be epic battle against “gentlemen” or at least there would be a notion of fear, that enemies would stay a bit away from you waiting for their turn instead… I mean this was not verbalized like this by the game designer, it’s my interpretation, and I guess that’s solved a problem as well as giving us a better gameplay in this context anyway.

This was the first thing to show us that we just plainly had to drop everything we knew about video games, and what gameplay would be like. Good third person games nowadays, are kind of “sequential” (I’m not saying there’s no ‘AI’ but you just can’t find everyone at the same time) , with a bit of random too , sure, but also it’s third person, you get to see what’s going on, you can react with a counter an attack while another enemy is being destroyed by your high kick or whatever.

So with First person. it would be one thing at a time. And really, that still makes for incredibly difficult combat – even when you finally get the hang of where the enemy is, because then what becomes difficult, is timing. it becomes all about timing… and when you think of it, that’s just how… ok I saw a bit of daredevil during development (thank you Netflix France but please hurry up I don’t need the french voice over) so I’m going to reference that… daredevil fights as there’s the “observe” phase, and the “attack at the right moment”. That can make for an epic fight, instead of just basically swiping all around hoping an enemy would impale himself on your sword. This is in fact something that will cost you lives… as some enemies react and hit you if you plain hit them at the wrong time !

Anyway we got this out of the way. Exploring the game and introducing other kinds of interaction would be far easier. But still… Everything had to be thought differently… and we could even cheat on some things, as long as the audio was right. So sometimes its just a fake 3D world. sometimes its complete 3D but there are fewer things going on than you would expect. Let’s say it would be hard to turn into a visual experience now. But that’s great, we focused on audio, not on something where you just turn off the camera afterwards.

Creating special design tools for game, level and audio designers :

This game is linear of course. We have to get through several environment to reach the end. This gives us the opportunity to try new things both with audio or gameplay. But we would have to work with everyone at once. on top of that, team members were not all in the same city, not even necessarily in the same country though everyone was from France. So it was important for everyone to have the tools to work with, to try the game, to tweak his own variables, to give his own suggestions, without having to wait for the developers to create a build.

This being a very different game than a visual one, it was decided early that we would not stick to a perfect plan, as to design the perfect plan, the perfect design document, means we would have everything figured out, and that – since we were all sighted – everything can defintely be figured out WRONG.

So anyway, about the tools.

Unity’s Editor is great to extend. It’s an interesting challenge, having access to all elements of the project in scenes or assets themselves, and the api to access assets or serialized objects and so on , was fun to work with, though sometimes frustrating as it was like there was a game within a game (It’s kind of like that sometimes).

Actually the editor would run the game and the thing that would come up fast, is that if you have access to an object through the inspector, changing a property will change the property of that instance of the object in the editor – instance that no longer exists once you start the actual scene and a lot of times I had to be careful not to mistakenly process things on the instance, but to actually use SerializedObject so that at runtime a different instance would run but the correct data would be there.

It’s very important to carefully go over what a prefab is with the designers. unfortunately in this case, a prefab would not simply be a rock model or a player model in 3D. All prefabs we have are actually just data and saving them in a scene would break some other scenes – specially when audio parameters where changed.

And in the case of iterative design and coding, a re-haul of MonoBehaviours or how things are done could break a lot of things… Thankfully though, keeping the necessary data from one MonoBehaviour to a very different one is easy thanks to SerializedObject once again. In fact it’s likely that more than 50% of the code is actually just access to serialized objects.

Actually saying this might sound like I’m surprised that’s it makes for a lot of the project’s code. No really since almost everything had to be done from scratch or was just done from scratch, then obviously, working with the serialized data to save scene is obvious.

So That’s something you really want to learn to work with. And I made tools for myself as well, to make somethings automatic across all unity scenes, post process them and so on.

Audio/Level Designer Tools :

For the Audio Designer, Unity was fairly complete, each AudioSource had plenty of panning/volume control based on position, and basic reverb effects were used as well. in combat there’s a simple lowpass filter going over sounds that are not of interest (helping a lot to focus on the action) and the filter goes down more when we hide behind our shield.

At the time Development began, Unity’s current Audio system wasn’t there, I mean it wasn’t exposed as much so no mixers… we knew it would come ultimately as it was announced. Knowing that is was using FMod also helped knowing we would be able to hack in something at some point if necessary… But anyway, it came and everything was great. Each scene is running a mixer snapshot, each AudioSource is carefully placed, a MonoBehaviour controls the 3Dception filter to deactivate it based on distance if necessary, so we can have a full scene without all filters actually ruining the performance. Some sources can move around to come in from up,down or to represent something or someone in the scene.

Due to the amount of sound clips this project has, categorizing Audio Clips was very important. And the first thing we did was to create a more abstract way of using audio clips . A Sound object was created which would hold a name, a category… and more to classify sounds correctly. In the end, most of the time, the Audio or Level designer does not have to drag and drop clips on objects… He can , in the scene, create a library (a simple object which will show a list) manipulate that list, drag and drop a lot of clips at once from a single folder, if their default name (file name) doesn’t suit them, they can be renamed… And they can even have a default volume so that each Sound object knows which asset it represents, but also what volume it should play at.

We use AudioSource.PlayOneShot() a LOT, to have as few sources as possible… as our performance tests with the first versions of 3Dception and unity showed that some devices we were going to target wouldn’t tolerate more than 10 sources with 3Dception filters or so. In the end, Unity worked on their audio part, 3Dception greatly improved as well, we made so changes, and this limit of 10 or so sources went up. We probably can do better that what is now on the stores performance wise though – specially because now Unity opened up their sound API even more… thanks to VR too.

So with as many Audio library objects as designers want on each scene so they can actually manage their sound properly, it’s just a matter of having the right index to look through. With support for english and french, obviously a simple field for audioclips would mean at some point we’d have to build 2 version of the game. We didn’t have to.

Audio and Level designers can test their scenes with the language they decide to play in, which is important as not everything is timed the same, and they can store their changes for each language dynamically using ScriptableObjects that we store. Those objects stored as assets that describe timing, volume per clip, name of clips and so on, are then very important in loading the right clips at runtime.

In fact, the build itself is small, as every clips – almost – come from asset bundles that are loaded from the disk, same that were loaded from a server when you first run the app and choose your language.

All of this versatility would allow for quicker iteration of scenes, gameplay tests.

So Categorizing audio clips was an important tool to speed up the designer’s workflow , as with more than 5k of audioclips or something, a simple table wouldn’t really help.

Audio libraries were made with reorderable lists in unity – a sexy version of basic lists (yes, I could’ve used property drawers, I did sometimes). And sounds could be listened to without opening a new inspector, and the waveform even displayed as a texture in the inspector. Everything to ease the workflow and make it enjoyable too (I find enjoyability to be important too).

We have other tools that help control , visualize, game events, variable changes, checkpoints… and actually, a very key part of the game too was Louise. Louise is I guess the most important and rewritten piece of code of the entire game, as she is the guide. She sure walks on a given polygonal path, stops here and there or can follow you, but also knows where you are, and sometimes she can’t go past an obstacle, sometimes she waits, I thought I’d create her with a simple state machine but it was more complex than that. Believe me having a guide that doesn’t follow you but that YOU should follow is very different. She is not the sidekick, she is the lead character to me, you are the sidekick. That’s of course, my opinion… Though she is repetitive in her words and doesn’t clearly do much… Without her, and when she wasn’t coded well, it was very hard to follow. The level designer did a good job at setting her up on each scenes, using the tools created for that task, and even though we had two different job here, we had a common goal and could discuss the tool itself to move forward. That’s where I think it’s very powerful to share something with the team, where workflows meet. In fact Louise wouldn’t be so great without his feedback on the tools that’s for sure, or we’d have to add more dev time on the clock.

Extending the unity editor

I do hope I’ve convinced you that this is very important from small to big projects.
Unless your game consists of assets and scripts from the asset store this is something you’ll definitely want to do for , I guess two main reasons. 1. Workflow Enhancement and 2. Enjoying Development, 1 being the most important but 2 is constantly feeding the other imo.

Oh and if you’re an indie and have only a couple of team mates. Well bringing them along the development process … is important as well. Sure there are tools only the designer will have to use. But what if he can work better by actually having a game specific tool with him, without you having to change code here and there. Ok I’m probably repeating myself.

In fact, I realize now, that this being my first 20 people and more team project, I’m just realizing that team work is important… if you’re already with a team, you know that and I’m preaching to the choir.

Let’s go for what Unity can do now then.

If you can create a custom inspector for each important object, do so. Sometimes it’s just cleaner to not draw the default inspector of your script and have the only important bit stick out if you’re not the only one touching the game.

With A Blind Legend, there probably was a lot of instances where I decided to change the property of the target object in the inspector, and then use

if (GUI.changed)
            EditorUtility.SetDirty (target);

to make sure my object was automatically reserialized and saved… but sometimes I just went ahead and actually created a new SerializedObject and saved it myself to make sure. I guess in the beginning of the project, I was really not very aware of what unity was doing in its editor. I was hoping sometimes GUI.changed to be true but it wasn’t probably from me not using it correctly, the save thing was to create a serializedObject myself, and apply the modified properties directly.

Similarly the layout system was a mystery to me. When should I be using EditorGUILayout or GUILayout or GUI ? clearly when you go into designing inspectors, you will have to have experience with layout systems. With EditorGUILayout though, most of the time everything will be ok, all new elements will appear under the previous one, width should fit correctly too. There are lots of things to say about the editor.

For example Here’s a script I had to check whether the scene has changed, or has been saved, to prepare that scene for editing and data to be ready for the designers, something I had to do by hand myself without that tool and would’ve had to explain to everyone. A good way to get some errors as no one should be required (other than the devs) to understand what is going on behind the scenes, and as number of scene increased, and iteration of scenes as well, this just felt like the right thing to do even for me.

[InitializeOnLoad]
public class SceneSetupUtils : AssetModificationProcessor
{

    public static SceneSetupUtils instance;

    static SceneSetupUtils ()
    {
        if (instance == null)
            instance = new SceneSetupUtils();
        else
            return;

        EditorApplication.update += instance.Update;
    }

    public string currentScene;

    public SceneSetupUtils ()
    {
        Debug.Log("started with the editor or after script compilation.");
        currentScene = EditorApplication.currentScene;
    }

    public void Update()
    {
        checkSceneChange();
        //do something when the editor updates!
    }

    public void checkSceneChange()
    {
        if(EditorApplication.currentScene != currentScene)
        { 
            //Process scene after opening/save/change here.
        }
    }

Ok this came late in development where we just had to double check everything was right with the audio.

Also, the designers would be able to drag/drop audio clips in the scene or on scripts. But in the end, the final build should NOT contain any unwanted referenced audio clips as the build size should not be over 50Mb for Android and we wanted all audio to be outside. This means keeping things simple for the designer, but an horrible job by hand of removing all clip references on audiobehaviour for the developper, post processing scenes was the way to go here with the following script.

public class ScenePostProcess {

	[PostProcessSceneAttribute (2)]
	public static void OnPostprocessScene() {
            //Loop through necessary objects, get the wanted component, create a SerializedObject from them like so :
            SerializedObject so = new SerializedObject(component);
            SerializedProperty property1 = so.FindProperty("audioClipReference");
            property1 .objectReferenceValue = null;
            so.ApplyModifiedProperties();
        }
}

this is a very simplified post process script. It will run before each scenes are built (so right before going into play mode, or during the build process so your final scenes will definetely not have the objectReference to that property and therefore Unity will not build into it.

Anyway the purpose here, is to not have to create complex inspectors, I wanted Unity to look like unity and objects to still be drag&droppable, I didn’t want to have sound assets path to be written by designers all the time to know which sound to load where. I still had to fetch audio clips without having their reference in the end so this solution worked.

Hint : To get even deactivated objects (which will happen) use :

GameObject[] gameObjects = Resources.FindObjectsOfTypeAll();

While I’m there

    [MenuItem("Data/Explore Persistent Data...")]
    static void ExplorePersistentData()
    {
        EditorUtility.RevealInFinder(Application.persistentDataPath + "/");
    }
    [MenuItem("Data/Clear Persistent Data")]
    static void ClearPersistentData()
    {
        System.IO.Directory.Delete(Application.persistentDataPath +"/", true);
    }

Yes, because %AppData%/LocalLow/ (main dev machine was under Windows) is always just so many clicks away…

But anyway, that’s how you write Menu Items that will do things for you. From there you can load up gameObject in your opened scene and process them as well… so you could quickly disable all shadows… Or you can access your assets with AssetDatabase and quickly change the import settings of all sounds or all sounds in certain folders (note: That’s how we used to do it… Now Unity finally supports multiple AudioClip inspection).

Actually, Most of the tools are really game specific, I could not share them. But I think I made my point, After actually looking at tutorials, how Layouts , buttons , labels are created (unity docs and tutorials over the web are nice introductions) take good care of saving your data, understand that the object you are editing is not the object that will end up in the game at runtime but a totally separate instance, and you have to set it to Dirty or have a SerializedObject done and saved (SerializedObject.ApplyModifiedProperties() ).

You can setup your unity editor to save assets as text files (scenes, prefabs), this is recommended for versioning but in this discussion, to debug !

The YAML format is readable, and this greatly helps debugging issues with inspectors, custom editors, wizards and menu items that handle data and objects.

Do that with :
Edit > Project Settings > Editor > Asset Serialization mode: Forced Text

Anyway that’s it for me !
Sorry if the end is pretty sloppy though, we could go on forever with unity and actually start a tutorial, but I’m cutting it short, keeping it relevant to the project and our experience with this project as well and what we’ve learned and found to use most of the time.

Hopefully this was interesting 🙂