The sample is written in TensorFlow 1.0 API style. Gradient supports TensorFlow 1.15, which has newer Keras APIs. Try converting the old style code into the new style. Try on other environments. Try other Unity ML Agents environments, and see how actor-critic will perform there. You might need to tweak observation and action processing for that. Get the Android Tensorflow Classification package from Epiphany Digital and speed up your game development process. Find this & other AI options on the Unity Asset Store. Sprint into Spring Sale is on: get 50% off top assets and score extra savings with coupon code SPRING2021.
I’ve been thinking for a while about how best to combine machine learning knowledge I’ve built up and my other hobby - making video games. To this end I’ve been looking into using Tensorflow with Unity3d. I forsee a lot of issues around performance at runtime, along with cross platform issues issues down the road.
As a start here is a quick rundown of compiling Tensorflow to run a trained graph from a C# Unity script using the C++ api.
Unity Tensorflow Github
1. Getting a Graph to Use
The idea here is to keep it to a bare minimum for what you need. The goal here is to make sure we can get TF will run at all rather than spending time making it do something useful. For these first couple of steps I borrowed heavily from Jim Flemming’s excellent Medium post on the basics of using the C++ api, which we’ll need. Using a simple python script you can generate a protobuf file to store your graph, I used:
which does the very impressive task of finding the max of two numbers and storing it in a variable c.
2. Compiling Tensorflow to a shared libaray (.so)
We can use the C++ api for TF to read and excetue graphs we’ve already saved pretty easily. In addition the Google documentation for using their build tool, Bazel is pretty good so compiling the shared library isn’t really too much of an issue. Here I used Jim Flemming’s build script with a few changes.
Debugging
When we start using this with Unity it’s not the same as running a script internally, we have to recompile to add debug messages. There’s not a simple method of accessing the Unity editor console to print useful debug messages anyway since we’re constrained by the return type of our function. This is a problem even with this very simple example as we’ve hard coded the path to the protobuf file which will cause problems down the road. Unity isn’t handling the paths so it won’t put the files in sensible places when building for instance. To give us some useful output on errors I’ve added logging to a file instead of standard out and known return values so it’s obvious where in the code we hit an error.
Shared library
The function is incased in an extern 'C' {}. Since we’re not building an exceutable we can’t just have a main function that returns 0, we want to return our max number. The C++ compiler however doesn’t preserve function names so calling our function after compilation won’t work, we have to tell the compiler we want to declare our funtion with C linkage. The Unity documentation does a fine job of explaining why we need to do this.
Here’s the code put inside the Tensorflow repo at /tensorflow/loader/loader.cc
Unity Tensorflow Plugin
We’ve hard coded here that it should output 3, which is the max of 3.0 and 2.0 cast to an int
We can compile using Bazel to either an exceutable or a shared library. The former is good to make sure the above code works, you can just throw the function into a main function and compile it. Once that’s working we can make our shared library. The Bazel BUILD file for that looks like:
If you keep this inside the loader folder with the cc file we can build from that folder using bazel build :loader.so. The actual so we need will end up
3: Running in the Unity Editor
You can access the shared library using the [DllImport ...] statement for C#, it doesn’t matter this isn’t a dll! This works the same as any Unity plugin, I have it in assets/plugins Here’s my very simple unity script:
All this is doing is:
- Loading our library
- On scene start running the
runfunction from the library - Putting the resulting integer as the string on an attached UI element.
Because of the way we hardcoded the model loading in the C++ and how the Unity editor does paths the protobuf needs to be in the root of the Unity project itself, not the plugins folder, not /assets. Now we can run this from the editor…
… Very impressive Unity, 3 is the right answer…
4: Running in a build
The shared libaray we’ve made as far as I know will only work for Linux. We need to compile different plugins for other platforms. Within the Unity editor we can set plugins to be included in different builds from the inspector for the asset. By default everything gets included which is fine if we’re only bulding for Linux but our project size will get pretty out of control if we have 3 or 4 TF libraries always being included in each build.
The default settings will work for the build; however this is where our lazy hard coding of paths is a problem. In this case the root path is going to be whereever your excutable is run from, so the /models folder needs to go there, not anywhere within the /data folder!
Unity Tensorflowsharp
It’s not particularly impressive but it works - we can access Tensorflow from a built Unity3d project! Now to run some more exciting graphs…
Note: TensorFlowSharp plugin has been deprecated, instead Unity uses its new Barracuda inference engine. See the new example. It’s better in terms of performance and ease of use for certain models.
This is an example of using model trained with TensorFlow in Unity application for image classification and object detection. It’s a quick port of TF Classify and TF Detect examples from TensorFlow repo, using TensorFlowSharp for gluing it all together.
Classify results:
Detect results:
Note that performance is worse than in TensorFlow Android example and at this moment I’m not quite sure how to improve that. Hopefully this will be enough to get you started.
You’ll need Unity 2019.2 or above and Unity TensorFlow Plugin.
- Open the project in Unity.
- Install TensorFlow plugin.
- Open Classify or Detect scene in Assets folder.
- In
Edit -> Player Settings -> Other settingsaddENABLE_TENSORFLOWto theScripting Define Symbolsfor the target platform. - In
Other settingsalso setScripting runtime versionto.NET 4.6 Equivalent. - Build and run.
Important: in new versions of Unity you might see error “Multiple assemblies with equivalent identity have been imported…”. In that case, you’ll need to go into ‘Assets/ML-Agents/Plugins/Android’ folder and manually delete all .dll files that are specified in the error message.
“Unloading broken assembly…” error can be safely ignored.
For iOS, folow this additional instructions: ios-additional-instructions-for-building
More info can be found here.
To use your own model:
- Make sure your model trained with TensorFlow 1.4 if you use 0.3 version of the Unity plugin that I linked above. You can also try 0.4 version that uses TensorFlow 1.7.1.
- Change extension of your model from .pb to .bytes.
- Put your model and labels in Resources.
- Set Model file and Labels file to your model and labels in main camera object of the scene you chose.
- If neccesary, change
classifyImageSize,IMAGE_MEAN,IMAGE_STD,INPUT_NAMEandOUTPUT_NAMEto suit your model.
I’m not a Unity expert, so if you found any problems with this example feel free to open an issue.