diff options
author | Kevin Cheng <kevcheng@google.com> | 2019-04-18 11:31:16 -0700 |
---|---|---|
committer | Kevin Cheng <kevcheng@google.com> | 2019-05-02 13:59:40 -0700 |
commit | 757c264bc10ebc71074ee3f5fb66d670667a09bc (patch) | |
tree | 26c7f7b74c752db99d9b0ac1f94fc592aca1e53a /doc/misc | |
parent | 99013222844839c42437f16eace25f4e6a8a8b20 (diff) | |
download | cffi-757c264bc10ebc71074ee3f5fb66d670667a09bc.tar.gz |
Add in cffi 1.12.2 (e0c7666)
Since this is a mercurial repo, d/led zip of src:
https://bitbucket.org/cffi/cffi/get/v1.12.2.zip
Also add in misc METADATA/NOTICE/Android.bp/etc files.
Bug: 122778810
Test: None
Change-Id: I36c58ed07a2cdd4d9d11831908175a5c988f33c1
Diffstat (limited to 'doc/misc')
-rw-r--r-- | doc/misc/design.rst | 51 | ||||
-rw-r--r-- | doc/misc/grant-cffi-1.0.rst | 124 | ||||
-rw-r--r-- | doc/misc/parse_c_type.rst | 72 |
3 files changed, 247 insertions, 0 deletions
diff --git a/doc/misc/design.rst b/doc/misc/design.rst new file mode 100644 index 0000000..390049c --- /dev/null +++ b/doc/misc/design.rst @@ -0,0 +1,51 @@ +================ +Design decisions +================ + +* Generally follow LuaJIT's ffi: http://luajit.org/ext_ffi.html + +* Be explicit: almost no automatic conversions. Here is the set + of automatic conversions: the various C integer types are + automatically wrapped and unwrapped to regular applevel integers. The + type ``char`` might correspond to single-character strings instead; + for integer correspondance you would use ``signed char`` or ``unsigned + char``. We might also decide that ``const char *`` automatically maps + to strings; for cases where you don't want that, use ``char *``. + +* Integers are not automatically converted when passed as vararg + arguments. You have to use explicitly ``ffi.new("int", 42)`` or + ``ffi.new("long", 42)`` to resolve the ambiguity. Floats would be + fine (varargs in C can only accept ``double``, not ``float``), but + there is again ambiguity between characters and strings. Even with + floats the result is a bit strange because passing a float works + but passing an integer not. I would fix this once and for all by + saying that varargs must *always* be a cdata (from ``ffi.new()``). + The possibly acceptable exception would be None (for ``NULL``). + +* The internal class ``blob`` is used for raw-malloced data. You only + get a class that has internally a ``blob`` instance (or maybe is a + subclass of ``blob``) by calling ``ffi.new(struct-or-array-type)``. + The other cases, namely the cases where the type is a pointer or a + primitive, don't need a blob because it's not possible to take their + raw address. + +* It would be possible to add a debug mode: when we cast ``struct foo`` + to ``struct foo *`` or store it in some other struct, then we would + additionally record a weakref to the original ``struct foo`` blob. + If later we try to access the ``struct foo *`` but the weakref shows + that the blob was freed, we complain. This is a difference with + ctypes, which in these cases would store a strong reference and + keep the blob alive. "Explicit is better than implicit", so we ask + the user to keep a reference to the original blob alive as long as + it may be used (instead of doing the right things in 90% of the cases + but still crashing in the remaining 10%). + +* LuaJIT uses ``struct foo &`` for a number of things, like for ``p[0]`` + if ``p`` is a ``struct foo *``. I suppose it's not a bad idea at least + to have internally such types, even if you can't specify them through + pycparser. Basically ``struct foo &`` is a type that doesn't own a + blob, whereas ``struct foo`` is the type that does. + +* LuaJIT uses ``int[?]`` which pycparser doesn't accept. I propose + instead to use ``int[]`` for the same purpose (its use is anyway quite + close to the C standard's use of ``int[]``). diff --git a/doc/misc/grant-cffi-1.0.rst b/doc/misc/grant-cffi-1.0.rst new file mode 100644 index 0000000..b026209 --- /dev/null +++ b/doc/misc/grant-cffi-1.0.rst @@ -0,0 +1,124 @@ + +=========================== +Grant Proposal for CFFI 1.0 +=========================== + +*Accepted by the PSF board on April 4, 2015* + +This Grant Proposal is to give a boost towards "CFFI 1.0". Two main +issues with the current CFFI need to be solved: the difficulties of +installation, and the potentially large time taken at import. + +1. The difficulties of installation can be seen from outside by looking +at various workarounds and 3rd-party documentation that have grown into +existence. For example, the `setup.py` of projects like cryptography, +PyNaCl and bcrypt deploys workarounds that are explicitly documented in +https://caremad.io/2014/11/distributing-a-cffi-project/. + +2. The time taken at import is excessive in some cases. For example, +importing `pygame-cffi` on a Raspberry Pi ARM board takes on the order +of 10 to 20 seconds (and this is the "fast" case where the compiler +doesn't need to be invoked any more). + + +Technical Overview +------------------ + +"CFFI" is an existing Python project which complements the ctypes, +SWIG and Cython approaches to ease writing C Extension Modules for +Python. It has several advantages over the previous approaches, which +are presented at the start of the documentation at +http://cffi.readthedocs.org/en/latest/ . It has been very successful +so far: http://pypi-ranking.info/alltime records almost 7 million +downloads (for comparison, the #1 of all packages has almost 36 +million downloads). CFFI works on any Python >= 2.6, including 3.x, +as well as on PyPy. + +One problem is that while getting started with CFFI is very easy, the +installation process of a package that uses CFFI has got its rough +edges. CFFI (at least in its "verify()" mode) is based on calling the +C compiler to get information about the exact C types, structures, +argument types to functions, and so on. The C compiler is invoked +transparently at run-time, and the results cached. A +correctly-installed package using CFFI should cache the results at +installation time, but it can be difficult to ensure that no more +run-time compiler invocation is needed; doing so requires following +some extra guidelines or understanding some internal details. (The +problem is particularly acute on Windows where a typical user might +not have a proper C compiler installed.) + +To fix this, we have in mind adding a different CFFI mode (replacing +"verify()"), while keeping the access to the underlying C library +unmodified. In this mode, the code containing the cdef() and verify() +invocations would be moved to a separate Python source file. Running +that Python file would produce a dynamically-linked library. There +would be no caching logic involved; you would need to run it +explicitly during development whenever you made changes to it, to +re-generate and re-compile the dynamically-linked library. + +When distributed, the same file would be run (once) during +installation. This can be fully automated in setuptools-based +setup.py files; alternatively, it can be done in distutils-based +setup.py files by requiring prior manual installation of CFFI itself. + +A major difference with the existing verify() approach would be that +the ``.so/.dll/.dylib`` file would not be immediately loaded into the +process; you would load it only from the installed program at +run-time, and get the ``ffi`` and ``lib`` objects in this way (these +are the two objects that you use so far to access a C library with +verify()). + +Additionally, this would solve another issue: every import of a large +CFFI-using package takes a while so far. This is caused by CFFI +needing to parse again the C source code given in the cdef() (adding a +run-time dependency to the ``pycparser`` and ``ply`` packages). CFFI +also computes a CRC to know if it can reuse its cache. In the +proposed change, all the cdef() code would be pre-parsed and stored in +the dynamically-linked library, and no CRC would be needed. This +would massively reduce the import times. + + +Grant objective +--------------- + +The objective is to give a boost towards "CFFI 1.0", which needs to have +the functionalities described above in order to solve the two main +issues with the current CFFI: the difficulties of installation, and the +time taken at import. + +Included in the objective: the internal refactorings of CFFI that are +needed to get it done cleanly. The goal is to avoid simply adding +another layer on top of the old unchanged CFFI. + +This work may happen eventually in any case, but support from the PSF +would help make it happen sooner rather than later. + + +Grant size +---------- + +2'500 US$ for supporting the development time. This would cover 2.5 +weeks of full-time work at the part-time cost of 25 US$ per hour. + +The estimated work time until the CFFI 1.0 release is a bit larger +than that (I estimate it at roughly 4 weeks), but 2.5 weeks should +cover all the basics. An extended grant size of 4'000 US$ would be +appreciated but not required ``:-)`` + + +Grant beneficiaries +------------------- + +Armin Rigo, main author of CFFI, committing 2.5 weeks of full-time +work. + + +Grant follow-up +--------------- + +I will report on the success of the grant on the CFFI mailing list and +on the blog I usually post to (the PyPy blog) and mention the PSF as +providing the grant. The PSF will receive an email pointing to these +postings once they are out. Moreover a full CFFI 1.0 release should +follow (likely starting with beta versions); the PSF will receive +another email pointing to it. diff --git a/doc/misc/parse_c_type.rst b/doc/misc/parse_c_type.rst new file mode 100644 index 0000000..1d1029d --- /dev/null +++ b/doc/misc/parse_c_type.rst @@ -0,0 +1,72 @@ +================================================== +CPython C extension module produced by recompile() +================================================== + +Global variable:: + + _cffi_opcode_t _cffi_types[]; + +Every _cffi_types entry is initially an odd integer. At runtime, it +is fixed to be a `CTypeDescrObject *` when the odd integer is +interpreted and turned into a real <ctype> object. + +The generated C functions are listed in _cffi_globals, a sorted array +of entries which get turned lazily into real <builtin function +objects>. Each entry in this array has an index in the _cffi_types +array, which describe the function type (OP_FUNCTION opcode, see +below). We turn the odd integers describing argument and return types +into real CTypeDescrObjects at the point where the entry is turned +into a real builtin function object. + +The odd integers are "opcodes" that contain a type info in the lowest +byte. The remaining high bytes of the integer is an "arg" that depends +on the type info: + +OP_PRIMITIVE + the arg tells which primitive type it is (an index in some list) + +OP_POINTER + the arg is the index of the item type in the _cffi_types array. + +OP_ARRAY + the arg is the index of the item type in the _cffi_types array. + followed by another opcode that contains (uintptr_t)length_of_array. + +OP_OPEN_ARRAY + for syntax like "int[]". same as OP_ARRAY but without the length + +OP_STRUCT_UNION + the arg is the index of the struct/union in _cffi_structs_unions + +OP_ENUM + the arg is the index of the enum in _cffi_enums + +OP_TYPENAME + the arg is the index of the typename in _cffi_typenames + +OP_FUNCTION + the arg is the index of the result type in _cffi_types. + followed by other opcodes for the arguments. + terminated by OP_FUNCTION_END. + +OP_FUNCTION_END + the arg's lowest bit is set if there is a "..." argument. + +OP_NOOP + simple indirection: the arg is the index to look further in + +There are other opcodes, used not inside _cffi_types but in other +individual ``type_op`` fields. Most importantly, these are used +on _cffi_globals entries: + +OP_CPYTHON_BLTN_* + declare a function + +OP_CONSTANT + declare a non-integral constant + +OP_CONSTANT_INT + declare an int constant + +OP_GLOBAL_VAR + declare a global var |