[WIP] Inceptor -- The "change ANYTHING extender" for your mods

[earlzdotnet]

So, this is my first go at this. I wanted to announce it at a point that I feel is fairly early, in hopes that maybe some other people could help me make it better and actually start building some cool stuff with it.

I made a new project I've named Inceptor. Inceptor is a command-line application which will modify the core code of the game to allow you to inject your own method calls supplementing and/or replacing the core game logic. It uses Mono.Cecil to accomplish this and is basically the equivalent of decompiling the code, modifying the code a bit to call my assembly, and then recompiling it. I originally made this out of frustration with how limited the official modding API is and because I work a lot with IL and the lower level bits of .NET, and it seemed like a fun thing to mess with.

The extender is located at https://github.com/Earlz/Inceptor I'm still deciding on a license. It'll be either GPL or BSD, haven't decided yet. You need to run it across the core game DLL (Assembly-CSharp.dll) and replace that, and then place the Inceptor assembly in the same folder.

Afterwards, you can actually go to making some really awesome mods using stuff that is not even imaginable with the official API. For instance, if you want to set the pollution rate of all industrial builds to 0 (without affecting any other building) you can with this:

    public class InjectionPoint: LoadingExtensionBase
    {
        static string locker = "";
        public override void OnLevelLoaded(LoadMode mode)
        {
            ChirpLogger.ChirpLog.Debug("hmmm something");

            ChirpLogger.ChirpLog.Info("loaded");
            InceptorInterceptor.Add("System.Void IndustrialBuildingAI::GetPollutionRates(System.Int32,System.Int32&,System.Int32&)", new InceptorInterceptor.Interceptor((@this, name, parms) =>
            {
                //note: chirplogger is apparently broken when using from different threads or in some way can not handle this and will silently fail to log anything
                StringBuilder sb = new StringBuilder(); //we log the previous values here...
                sb.AppendLine("--begin--");
                int i=0;
                foreach (var p in parms)
                {
                    sb.AppendFormat("parms[{0}] = {1}\r\n", i, p == null ? "null" : p.ToString());
                    i+=1;
                }
                sb.AppendLine("--end--");
                lock(locker)
                {
                    File.AppendAllText("C:\\dev\\debug.txt", sb.ToString());
                }
                //when a parms element is set for reference types (ones ending in a &), then the value will be persisted back in the method
                parms[1] = 0; //ground pollution
                parms[2] = 10; //noise pollution
                return new object(); //return an object to tell the intercepted method to return immediately after this (return null to tell it to continue through with the body of the method
            }));

            base.OnLevelLoaded(mode);
        }
        public override void OnCreated(ILoading loading)
        {
            base.OnCreated(loading);
        }
    }

It's some pretty low level stuff and is hard to use, but it makes a ton of new things possible. The eventual goal is for the end-user to run an "installer" program once, and then mods can be loaded easily and even distributed from the workshop utilizing this extender.

Note: If you want to understand what method you actually need to inject into and how to modify the code and/or what the various parameters even mean, you should use a decompiler such as ILSpy to disassemble the Assembly-CSharp.dll file and read it's equivalent C# code (note some of it will be rough around the edges and there will of course be no comments)

Also, I've made an example mod using it at https://github.com/Earlz/CitiesTestMod

 

[eis_os]

Nice work. Btw. It's easier then my first CIM2PatchPlus Patch I wrote in MSIL Bytecodes CIM2 has no API at all.
Myself using dnlib for PatchPlus and wrote a small "Assembler" doing string lookups for types and pretty basic jump targets.
Actually only Patching the needed ones, and make all necessary changes to get access to private methods.

But I am not sure how this should work with Workshop Mods?

 

[nik1t0zz]

What's the point in going so low level? (Below is how you can easily make industrial buildings stop polluting, could be easier if they didn't actually calculate the pollution values)

namespace Test
{
	public class LoadingExtentionSkylines : LoadingExtensionBase
	{
		public override void OnLevelLoaded(LoadMode mode)
		{
			try
			{
				if (mode != LoadMode.LoadGame && mode != LoadMode.NewGame)
				{
					return;
				}
				
				int counter = 0;
				
				for (uint i = (uint)PrefabCollection<BuildingInfo>.PrefabCount(); i > 0; i--)
				{
					BuildingInfo infoPrefab = PrefabCollection<BuildingInfo>.GetPrefab(i);

					if (infoPrefab == null)
					{
						continue;
					}

					BuildingAI component = infoPrefab.GetComponent<BuildingAI>();

					if (component != null)
					{
						if (ReplaceBuildingPrefabs(component, found, infoPrefab))
						{
							counter++;
						}
					}
				}

				SkylineUtils.Debug.AddDebugMessage("Replaced " + counter + " building components");
				
			}
			
			catch (Exception ex)
			{
				SkylineUtils.Debug.AddDebugError(ex.ToString());
			}
		}
		
		private static bool ReplaceBuildingPrefabs(BuildingAI component, bool? found, BuildingInfo infoPrefab)
		{
			if (component.GetType() == typeof(IndustrialBuildingAI))
			{
				Type newType = typeof(Skylines.IndustrialBuildingAINoPollution);
				ReplaceBuildingAI(component, newType, infoPrefab);

				return true;
			}

			return false;
		}
		
		private static void ReplaceBuildingAI(BuildingAI component, Type newType, BuildingInfo infoPrefab)
		{
			UnityEngine.Object.Destroy(component);

			BuildingAI newAI = infoPrefab.gameObject.AddComponent(newType) as BuildingAI;

			newAI.m_info = infoPrefab;
			infoPrefab.m_buildingAI = newAI;
			newAI.InitializePrefab();
		}
	}
		
	class IndustrialBuildingAINoPollution : IndustrialBuildingAI
	{
		public override void GetPollutionRates(int productionRate, out int groundPollution, out int noisePollution)
		{
			groundPollution = 0;
			noisePollution = 0;
		}
	}
}

Switching managers is a bit harder, but there must be a way.
Modding API (the one in ICities) is useless (sort of) as it mostly provides entry points (like OnLevelLoaded()) so you could swap managers out through Singleton or change AI and other components in the prefabs through PrefabCollection.
I hope this helps you in some way, as I don't find going so low level of any use (and it is very damn hard to make any mods through IL patching).

 

[eis_os]

Well there may be reasons:

1. You don't want to copy whole code blocks because a method or many are private and can't be overridden or accessed.
2. If you can inject code blocks you may be able to add several mods side by side.

Example:
If Mod A wants to change Method 1, and Mod B Method 2. It may be technical possible if both runtime patch. If Mod A and B replaces the Citizen Manager then you get only one.

Technical I would like to only change runtime change method Attributes, however JIT can perform optimizations, so I didn't investigate much yet on it.
I changed some private to public members for CIM2PatchPlus that made life a lot easier.

 

[nik1t0zz]

Most private methods get called from public methods within that class, so it's not really a problem to write a new public copy of the private method, override public method and call the new one in it.
The mods that override the same class could be a problem, but authors could just make compatibility versions and detect if mod is active or class was already overridden and just hook in the compatible version, not really a problem, while (I think) any changes to .dll will be hard to pull off on a massive scale and just hard to make in the first place. Plus, if you'll think about how easy it is for users to download mods from workshop (and maintain and update them for authors) and how much hassle the .dll injections would be.

 

[mabako]

Most private methods get called from public methods within that class, so it's not really a problem to write a new public copy of the private method, override public method and call the new one in it.

There are -a lot- of public functions you can't override as they're neither marked as override nor virtual.

 

[ApocDev]

There are -a lot- of public functions you can't override as they're neither marked as override nor virtual.

Actually, you can override just about everything, even when it's not marked as vitual/abstract.

	public class CitizenAIMod : ResidentAI
	{
		// C# trix to override a non-virtual method.
		protected new bool StartPathFind(ushort instanceID, ref CitizenInstance citizenData, Vector3 startPos, Vector3 endPos, VehicleInfo vehicleInfo)

Also, since this is Mono (and mono is open-source), you can actually runtime-patch the methods without need to break things down with Mono.Cecil. I have some POC code being worked on right now (been having issues with threading, but that's more of a laziness issue, than an actual technical issue).

For example, in normal old .NET (x86 for this specific code, I have an x64 variant laying around somewhere) you can detour .NET methods with about 20 lines of code:

		public unsafe static void DetourMethod(MethodInfo source, MethodInfo patch)
		{
			var getDynamicHandle =
				Delegate.CreateDelegate(typeof(Func<DynamicMethod, RuntimeMethodHandle>), typeof(DynamicMethod).GetMethod("GetMethodDescriptor", BindingFlags.Instance | BindingFlags.NonPublic))
					as Func<DynamicMethod, RuntimeMethodHandle>;

			// Create a method that literally just jmps to our patch func. (The one we want actually executing code)
			var newMethod = new DynamicMethod("_" + Environment.TickCount + Environment.TickCount * 6, source.ReturnType, source.GetParameters().Select(p => p.ParameterType).ToArray());
			var body = newMethod.GetILGenerator();
			body.Emit(OpCodes.Jmp, patch);
			body.Emit(OpCodes.Ret);

			// Now make sure we JIT the "new" method
			// And then just overwrite the old method ptr, with the new ptr.
			var handle = getDynamicHandle(newMethod);
			RuntimeHelpers.PrepareMethod(handle);
			*((IntPtr*)new IntPtr(((IntPtr*)source.MethodHandle.Value.ToPointer() + 2)).ToPointer()) = handle.GetFunctionPointer();
		}

However, that won't work with Mono since they don't implement a few required things there (specifically, GetMethodDescriptor and GetDynamicILInfo)

There is also MethodRental.SwapMethodBody which is also not implemented in Mono. (See the running trend here...)

But, as I said, I have some code almost working to do runtime patching/detouring of methods.

 

[Kanimir]

But sounds good!

 

[earlzdotnet]

Actually, you can override just about everything, even when it's not marked as vitual/abstract.

	public class CitizenAIMod : ResidentAI
	{
		// C# trix to override a non-virtual method.
		protected new bool StartPathFind(ushort instanceID, ref CitizenInstance citizenData, Vector3 startPos, Vector3 endPos, VehicleInfo vehicleInfo)

Also, since this is Mono (and mono is open-source), you can actually runtime-patch the methods without need to break things down with Mono.Cecil. I have some POC code being worked on right now (been having issues with threading, but that's more of a laziness issue, than an actual technical issue).

For example, in normal old .NET (x86 for this specific code, I have an x64 variant laying around somewhere) you can detour .NET methods with about 20 lines of code:

		public unsafe static void DetourMethod(MethodInfo source, MethodInfo patch)
		{
			var getDynamicHandle =
				Delegate.CreateDelegate(typeof(Func<DynamicMethod, RuntimeMethodHandle>), typeof(DynamicMethod).GetMethod("GetMethodDescriptor", BindingFlags.Instance | BindingFlags.NonPublic))
					as Func<DynamicMethod, RuntimeMethodHandle>;

			// Create a method that literally just jmps to our patch func. (The one we want actually executing code)
			var newMethod = new DynamicMethod("_" + Environment.TickCount + Environment.TickCount * 6, source.ReturnType, source.GetParameters().Select(p => p.ParameterType).ToArray());
			var body = newMethod.GetILGenerator();
			body.Emit(OpCodes.Jmp, patch);
			body.Emit(OpCodes.Ret);

			// Now make sure we JIT the "new" method
			// And then just overwrite the old method ptr, with the new ptr.
			var handle = getDynamicHandle(newMethod);
			RuntimeHelpers.PrepareMethod(handle);
			*((IntPtr*)new IntPtr(((IntPtr*)source.MethodHandle.Value.ToPointer() + 2)).ToPointer()) = handle.GetFunctionPointer();
		}

However, that won't work with Mono since they don't implement a few required things there (specifically, GetMethodDescriptor and GetDynamicILInfo)

There is also MethodRental.SwapMethodBody which is also not implemented in Mono. (See the running trend here...)

But, as I said, I have some code almost working to do runtime patching/detouring of methods.

using your "protected new" trick won't work in this case. It overrides it only if the calling method is expecting the type to be CitizenAIMod, not ResidentAI. You can't just do some "trick" to emulate virtual method behavior for non-virtual methods

I was also thinking about making a new version of Mono for it with intercepting, but this sounds more complicated in the long run, though it would probably be the more powerful option.

I also need to make it so that it converts all private and protected fields and methods to be public for easy access.


And yea, I'm aware now that the pollution rate for industrial buildings could be overridden globally. That was just the fastest example I could think of. My actual intention is to make a mod where "ground pollution" is affected by wind speed in the area. I also want to add the notion of a water table and the amount of water int he ground only being capable of sustaining a certain "rate" of water. Also more interesting stuff like being capable of changing the individual lanes of each road to left, right, left/straight turns etc.

 

[ApocDev]

using your "protected new" trick won't work in this case. It overrides it only if the calling method is expecting the type to be CitizenAIMod, not ResidentAI. You can't just do some "trick" to emulate virtual method behavior for non-virtual methods

I was also thinking about making a new version of Mono for it with intercepting, but this sounds more complicated in the long run, though it would probably be the more powerful option.

A "new version of Mono" wouldn't really work all that well unless you have access to Unity's mono implementation. They do their own changes, and optimizations for game-related things.

 

[earlzdotnet]

Actually, that is the point of it. .NET (and Mono) search the class heirarchy starting at the class it has an instance of (CitizenAIMod in this case) and looks for a matching metadata method (which is found from StartPathFind in the CitzenAIMod class), both runtimes walk the inheritance tree from the instance class, up each base class. It doesn't do specific-class references in almost all cases. (Test it out yourself) https://msdn.microsoft.com/en-us/library/ms173153.aspx also has good docs for you to read through.

A "new version of Mono" wouldn't really work all that well unless you have access to Unity's mono implementation. They do their own changes, and optimizations for game-related things.

See, the problem is the core game code never knows what CitizenAIMod is, so it never actually calls CitizenAIMod::SomeMethod.. It only knows about ResidentAI and calls ResidentAI::SomeMethod, which for a non-virtual SomeMethod will result in it always actually executing ResidentAI::SomeMethod.

The new version of Mono bit is possible. Mono is LGPL licensed and so Unity has to provide source code for their version of Mono. Someone did something similar to this even to enable remote-debugging so that they could effectively debug their Mods from MonoDevelop. See https://github.com/angavrilov/ksp-devtools

 

[ApocDev]

See, the problem is the core game code never knows what CitizenAIMod is, so it never actually calls CitizenAIMod::SomeMethod.. It only knows about ResidentAI and calls ResidentAI::SomeMethod, which for a non-virtual SomeMethod will result in it always actually executing ResidentAI::SomeMethod.

The new version of Mono bit is possible. Mono is LGPL licensed and so Unity has to provide source code for their version of Mono. Someone did something similar to this even to enable remote-debugging so that they could effectively debug their Mods from MonoDevelop. See https://github.com/angavrilov/ksp-devtools

I'm pretty against pre-patching the assemblies, as that always leads to failure in most cases, especially when you start messing with scope modifiers (priv/prot -> public, etc). And in a lot of cases, games will checksum the assemblies to ensure it's "correct" (eg for updates and the like) and just not start the game at all. This is from experience in messing around with hundreds of games. I'm not saying CO/Paradox will end up doing that, but it's always something I avoid if possible.

Doing a body IL replacement is probably the safest way to handle method detouring. Flat out hooking every single method isn't really a good idea in terms of performance, and stability. Luckily, Mono is pretty simple in how it handles the post-JIT method stuff unlike the CLR which holds method refs in at least 6 different places. There's only 2 or 3 places (IIRC) that need replacements and you can fully "detour" a method call, or just modify the IL and re-JIT it to jmp to your callback somewhere. I have the IL modification "done" already, but doing the full detour is a bit more difficult. (I'd prefer not to just add a jmp <method> retn as the detour if at all possible, so debugging doesn't go all batshit crazy)

 

[Copernicus]

@ApocDev: I agree on not changing the assembly itself. I actually have done some simple experiments and find that straight forward patching works fine (to my surprise, I expected Mono to do much more patching post-JIT). I would be interested to know how you modify the IL code at runtime. It is simple to exchange it pre-JIT, but deploying code that actually works seems hard to do. Are you actually changing the metadata to contain new tokens for your new methods and classes etc.? Also, how would you force a re-JIT? Do you re-JIT the entire codebase? If not, how do you ensure that the newly compiled method is actually used in callers that have already been JIT-compiled?

@nik1t0zz: I do not see how this is related to DLL-injections. The game is loading our code by design, there is no need to inject anything.

 

[earlzdotnet]

I agree on not changing the assembly itself. I actually have done some simple experiments and find that straight forward patching works fine (to my surprise, I expected Mono to do much more patching post-JIT). I would be interested to know how you modify the IL code at runtime. It is simple to exchange it pre-JIT, but deploying code that actually works seems hard to do. Are you actually changing the metadata to contain new tokens for your new methods and classes etc.? Also, how would you force a re-JIT? Do you re-JIT the entire codebase? If not, how do you ensure that the newly compiled method is actually used in callers that have already been JIT-compiled?

@nik1t0zz: I do not see how this is related to DLL-injections. The game is loading our code by design, there is no need to inject anything.

See there's the difference. I'm not doing runtime modification of the assembly, nor runtime patching. I use Mono.Cecil to basically read in the IL, modify it in some ways so I can "detour" some methods, then write it back out to a file. You replace the Assembly-CSharp.dll file in the Managed folder of the game and that's it. Afterwards mods reliant on Inceptor just work. There is no need for each mod to require it's own Inceptor run, it only needs to reference the InceptorAssembly which is now placed in the Managed folder beside the Assembly-CSharp file.

This is the type of stuff I do in my day job so it's straight forward to me, so sorry if I'm not explaining it thoroughly enough. I don't know how to break it down any further than that.

 

[Copernicus]

What you are doing is perfectly clear to me, I was referring to ApocDev's claim to replace IL at runtime, because (for all I know) this is hard. Sorry if I did not make that clear enough, I have edited the post to clarify this. I have thought about using Cecil myself, but I consider exchanging the assembly with a rigged one cheating .

 

[ApocDev]

What you are doing is perfectly clear to me, I was referring to ApocDev's claim to replace IL at runtime, because (for all I know) this is hard. Sorry if I did not make that clear enough, I have edited the post to clarify this. I have thought about using Cecil myself, but I consider exchanging the assembly with a rigged one cheating .

The simplest way to handle it, is to replace the pointer to the "code" for the method (post-JIT code) with some prepared code. (This is easily done in normal .NET, but Mono requires a few extra "tweaks" to make it work) Again, it's something I've been tinkering with in my free time (not much of which I have lately. I've actually spent most of my recent time reversing the ColossalNative DLL) For instance, SharpDX uses runtime-replacements to keep the API incredibly fast, by replacing the calls with trampolines to the native DirectX API. The same basic thing I'm doing, except it trampolines to another .NET func. Also, you can force-JIT a method pretty trivially with RuntimeHelpers.PrepareMethod.

Pre-patching the assemblies has always led to issues, especially if you don't take care of the major issues that pop up (such as synchronization, generics, etc).

All that said, if CO is not going to be "breaking the game if the assemblies don't match", then pre-patching may be the best option to get a well-designed mod API together. (One that isn't so focused on forcing class overrides for pretty trivial changes)

 

[eis_os]

CIM2 screwed with Cecil (only loading/saving),dnlib worked fine. I don't know why. Types are evil, I can say I have a lot happy users with pre patching with CIM2* but it won't work for CSL. (it's not End-User Compatible)

Technical the Mods are loaded to late, otherwise my plan would be:

Find the Mono Runtime in Memory and patch it so loading/jiting the Game Code results
in public override able methods and attributes. Compiling Code still needs a pre-patched DLL.
Technical it is still possible that the Compiler generated Code with OpCode Call instead of Callvirt.

I the end I guess I will have a main mod that includes the patcher that changes Mono in memory,
all my other mods will depend on it and supply the patch infrastructure (search/replace/add/restore)
and provide the necessary rejit functionality.

I fear a different Problem, the complexity of modding.
TTDPatch had the problem that as soon as you want to modify a code piece there are x previous patches already there.
It was monolithic (but with user configuration) and end up in a dependency hell.



* CIM2PatchPlus needs two compile steps and is circular referenced, however it does change the IL Methods Body and inserts code in the middle of Methods.

 

[Copernicus]

@ApocDev: So you are doing post-JIT detours after all and are not replacing the IL code, right? (You were claiming otherwise in your last post, which is what got me interested in the first place .) Because that was my whole point; and I already have a working post-JIT detour. I spend the first day after release reversing ColossalNative; there is nothing interesting to be found there, it just registers some internal calls to Mono and has a bunch of OpenGL stuff in it. In Mono, RuntimeHelpers.PrepareMethod is a no-OP, you cannot use it to force-JIT methods. Use GetFunctionPointer on the method handle instead to force jitting. Furthermore, you were claiming to re-JIT the IL, but this approach does not allow you to re-JIT anything. It merely compiles methods which have not been jitted before; once it is jitted, it will not help you.

 

[ApocDev]

@ApocDev: So you are doing post-JIT detours after all and are not replacing the IL code, right? (You were claiming otherwise in your last post, which is what got me interested in the first place .) Because that was my whole point; and I already have a working post-JIT detour. I spend the first day after release reversing ColossalNative; there is nothing interesting to be found there, it just registers some internal calls to Mono and has a bunch of OpenGL stuff in it. In Mono, RuntimeHelpers.PrepareMethod is a no-OP, you cannot use it to force-JIT methods. Use GetFunctionPointer on the method handle instead to force jitting. Furthermore, you were claiming to re-JIT the IL, but this approach does not allow you to re-JIT anything. It merely compiles methods which have not been jitted before; once it is jitted, it will not help you.

I can do either really. The pre-JIT IL is stored pretty close the post-JIT native code, and making JIT re-run on a method is trivial (at least in normal .NET). Mono does require a bit of extra "thinking" since they do AOT compiling which is a bit painful in some cases.

 

[Copernicus]

I know of no method to re-JIT a method with Mono (or .NET), that's why I was asking. So if you have any concrete pointers, I'd appreciate them. For all I know it is nigh on impossible to re-JIT existing methods from managed code. Furthermore, re-JITting single methods would screw with all kinds of things, mostly because all call sites would have to be patched to call the new piece of code instead.
Concerning AOT: To my best knowledge, Mono supports AOT compilation, but it is not used unless it is explicitly asked for -- which is not the case here.

Now for IL: As I already said, I know where the IL is stored, how to replace it and all that stuff. I have already written a simple piece of code that replaces IL code (see the link in my first post in this thread). My point is that you cannot just replace the original IL with your IL across assembly boundaries, because its tokens will not match up and you get calls to bogus methods. That's why I was asking whether or not you edit the meta data and add in new tokens, and if so, how do you do it? (I know that it is possible, but the only way I can imagine it being done is very painful.)

I'd like to see some actual evidence for your claims, because if they are right, I must have missed something pretty obvious