Compile Python static from source with CMake

[Zakant]

This buildfile seems to work.
Needs changes to work with python 3.6 and remove options as they arent requiered if we build it just for our plugin.

[overfl0]

Are you sure you want to build python yourself? Why not use already existing embedded distributions:
https://docs.python.org/3.5/using/windows.html#embedded-distribution

You're gonna have to provide all this data to the users anyway so I don't see the need to compile python yourself.
Note that python is not only C code but also .py files (standard library) and precompiled libs (pyd, etc...)

[Zakant]

The problem is that the plugin will search for a pythonXX.dll to load the interpreter. In my opinion it would be the best case to link the interpreter static to our plugin. By doing so we can ensure everybody

• Has a python installation
• Has the right version of python
• Has a version that matches his system (UNIX/Windows x86/x64)

Many standard .pyd files can be compiled into the static library. We would only need to care about the standard library witch consist of .py files.
However that can be handled by your pbo loader so we ship them in a pbo as well.

[overfl0]

You're right with loading the dll. What's more, there will be an issue with loading the right python dll (32bit/64bit).

Unfortunately, from what I can see, there is lots of gotchas:
https://stackoverflow.com/questions/1150373/compile-the-python-interpreter-statically

You have to select the modules to use (what if you don't select a module and someone uses it?), then run the right command to compile (that's easy, actually).
Then, what will happen if someone uses a python module they wrote and that python module requires python to be dynamically linked (does it? I don't know).

Note: We will also have to figure out a way for people to add additional modules in their mods.

I would postpone all of this until we have a working prototype, that we currently don't have.

[overfl0]

Is there any way we could use the system PYTHON path variable and run the code in that interpreter, then we wouldn't have to compile it in our project and the end user would be able to run whichever version they choose.

TBH I'd go the exact opposite way because otherwise, you'll end with a mess where everyone has another python version with different modules installed (not sure if we want to enable pip or not yet).
Then you're guaranteed that everyone's code will never work the same on everyone's computers.

Anyway, I have currently solved this problem for Windows + Python3.5 + pip in Pythia - Python 3.6 is currently a little problematic (overfl0/Pythia#32).

I'm using the embeddable zip file (with tiny modifications) from https://www.python.org/downloads/release/python-354/ and they work fine. What I mean by that is that "they just work" - you don't even have to have python installed on your computer. You run Pythia and you have Python working; no steps required. It's something that I would like to see in Intercept-python too.

[dedmen]

Onto pybind11 http://pybind11.readthedocs.io/en/master/classes.html#creating-bindings-for-a-custom-type
It sounds easy to use. But our main problem is http://pybind11.readthedocs.io/en/master/advanced/embedding.html which.... doesn't look that hard. Though they only doc how it works on the surface. We are however interested in the backend.
But with the upcoming intercept dll directory thingymabob where you can specify a directory to load dependent dll's from.
Maybe I should implement that 🤔

[overfl0]

Regarding pybind11, I could theoretically test it by porting Pythia to it which, if successful, would mean that it can also easily be used with intercept-python.

Regarding the dependent dlls directory I think that there is currently no point in doing so. I have already solved this problem myself and have code that works the way it's all set up currently (I think).
What would help MUCH MORE is... documentation. Because right now, it's mostly tribal knowledge + asking Dedmen what and how.

Instead, I would much prefer some doc stating what exactly is needed and why and where to create a minimal working version of intercept in another language that works and can be tested from within the game.
Also, even the basic wiki page that explained how to build intercept and create a sample hello world plugin has always been broken AFAIR and has always been the source of my frustration and was the reason I was giving up on working on intercept-python each time I thought that I had free time.

Fixing those could benefit Intercept much more than implementing another new technical solution, right now.

Don't get me wrong: I can't promise that I will have the time to work full-time on intercept-python, but the things that I listed above were (partly, of course) the reasons why I have not done so yet.

[dedmen]

Instead, I would much prefer some doc stating what exactly is needed and why and where to create a minimal working version of intercept in another language that works and can be tested from within the game.

That depends on the language you are trying to implement and what you want to implement.

Yeah the broken tutorial is definetly an issue. Especially with the latest changes it got broken even more ^^

We also have the intercept-examples repo that is somewhat up-to-date and working. And other bigger example projects like my Script Profiler or Intercept CBA which are based on CMake and constantly updated with the latest changes because I actually use these to test.

I also muuuuuch prefer the Intercept CBA/script profiler approach of embedding the Intercept client sources in a submodule and directly compiling them in. It's much easier and faster than going via the client_lib route that Intercept had in place before I joined.

I'm gonna get the intercept-examples repo up to current master branch compatibility and look at the tutorial.