Lua (5.3) Language addition (#4804)

* starting Lua port of python implmention. Syncing commit

* Bulk of Lua module port from Python done. Not tested, only static analysis. Need to work on binary strings. Started work on flatc lua code generation

* Fixed all the basic errors to produced a binary output from the builder, don't know if it is generated correctly, but it contains data, so that must be good

* fixed binary set command that was extending the array improperly

* continued improvement

* Moved lua submodules down a directory so their names don't clash with potential other modules. Added compat module to provide Lua versioning logic

* Successful sample port from Python

* working on testing Lua code with formal tests

* continued to work on tests and fixes to code to make tests pass

* Added reading buffer test

* Changed binaryarray implmentation to use a temporary table for storing data, and then serialize it to a string when requested. This double the rate of building flatbuffers compared to the string approach.

* Didn't need encode module as it just added another layer of indirection that isn't need

* profiled reading buffers, optimizations to increase read performance of monster data to ~7 monster / millisecond

* Writing profiler improvments. Get about
~2 monsters/millisecond building rate

* removed Numpy generation from Lua (came from the Python port)

* math.pow is deprecated in Lua 5.3, so changed to ^ notation. Also added .bat script for starting Lua tests

* adding results of generate_code.bat

* simple edits for code review in PR.

* There was a buffer overflow in inserting the keywords into the unorder set for both the Lua and Python code gens. Changed insertion to use iterators.

* fixed spacing issue

* basic documenation/tutorial updates. Updated sample_binary.lua to reflect the tutorial better

* removed windows-specific build step in Lua tests

5 files changed, 281 insertions, 1 deletions

diff --git a/docs/source/FlatBuffers.md b/docs/source/FlatBuffers.md
index ff27476f..351c38ed 100644
--- a/docs/source/FlatBuffers.md
+++ b/docs/source/FlatBuffers.md
@@ -4,7 +4,7 @@ FlatBuffers    {#flatbuffers_index}
 # Overview {#flatbuffers_overview}
 
 [FlatBuffers](@ref flatbuffers_overview) is an efficient cross platform
-serialization library for C++, C#, C, Go, Java, JavaScript, TypeScript, PHP, and Python.
+serialization library for C++, C#, C, Go, Java, JavaScript, Lua, TypeScript, PHP, and Python.
 It was originally created at Google for game development and other
 performance-critical applications.
 
@@ -134,6 +134,8 @@ sections provide a more in-depth usage guide.
     in your own programs.
 -   How to [use the generated Go code](@ref flatbuffers_guide_use_go) in your
     own programs.
+-   How to [use the generated Lua code](@ref flatbuffers_guide_use_lua) in your
+    own programs.
 -   How to [use the generated JavaScript code](@ref flatbuffers_guide_use_javascript) in your
     own programs.
 -   How to [use the generated TypeScript code](@ref flatbuffers_guide_use_typescript) in your
diff --git a/docs/source/LuaUsage.md b/docs/source/LuaUsage.md
new file mode 100644
index 00000000..75b1f3b8
--- /dev/null
+++ b/docs/source/LuaUsage.md
@@ -0,0 +1,81 @@
+Use in Lua    {#flatbuffers_guide_use_lua}
+=============
+
+## Before you get started
+
+Before diving into the FlatBuffers usage in Lua, it should be noted that the
+[Tutorial](@ref flatbuffers_guide_tutorial) page has a complete guide to general
+FlatBuffers usage in all of the supported languages (including Lua). This
+page is designed to cover the nuances of FlatBuffers usage, specific to
+Lua.
+
+You should also have read the [Building](@ref flatbuffers_guide_building)
+documentation to build `flatc` and should be familiar with
+[Using the schema compiler](@ref flatbuffers_guide_using_schema_compiler) and
+[Writing a schema](@ref flatbuffers_guide_writing_schema).
+
+## FlatBuffers Lua library code location
+
+The code for the FlatBuffers Lua library can be found at
+`flatbuffers/lua`. You can browse the library code on the
+[FlatBuffers GitHub page](https://github.com/google/flatbuffers/tree/master/lua).
+
+## Testing the FlatBuffers Lua library
+
+The code to test the Lua library can be found at `flatbuffers/tests`.
+The test code itself is located in [luatest.lua](https://github.com/google/
+flatbuffers/blob/master/tests/luatest.lua).
+
+To run the tests, use the [LuaTest.sh](https://github.com/google/flatbuffers/
+blob/master/tests/LuaTest.sh) shell script.
+
+*Note: This script requires [Lua 5.3](https://www.lua.org/) to be
+installed.*
+
+## Using the FlatBuffers Lua library
+
+*Note: See [Tutorial](@ref flatbuffers_guide_tutorial) for a more in-depth
+example of how to use FlatBuffers in Lua.*
+
+There is support for both reading and writing FlatBuffers in Lua.
+
+To use FlatBuffers in your own code, first generate Lua classes from your
+schema with the `--lua` option to `flatc`. Then you can include both
+FlatBuffers and the generated code to read or write a FlatBuffer.
+
+For example, here is how you would read a FlatBuffer binary file in Lua:
+First, require the module and the generated code. Then read a FlatBuffer binary
+file into a `string`, which you pass to the `GetRootAsMonster` function:
+
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.lua}
+    -- require the library
+    local flatbuffers = require("flatbuffers")
+    
+    -- require the generated code
+    local monster = require("MyGame.Sample.Monster")
+
+    -- read the flatbuffer from a file into a string
+    local f = io.open('monster.dat', 'rb')
+    local buf = f:read('*a')
+    f:close()
+
+    -- parse the flatbuffer to get an instance to the root monster
+    local monster1 = monster.GetRootAsMonster(buf, 0)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Now you can access values like this:
+
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.lua}
+    -- use the : notation to access member data
+    local hp = monster1:Hp()
+    local pos = monster1:Pos()
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+
+## Text Parsing
+
+There currently is no support for parsing text (Schema's and JSON) directly
+from Lua, though you could use the C++ parser through SWIG or ctypes. Please
+see the C++ documentation for more on text parsing.
+
+<br>
diff --git a/docs/source/Tutorial.md b/docs/source/Tutorial.md
index 2d0e0d8e..11fe4c8c 100644
--- a/docs/source/Tutorial.md
+++ b/docs/source/Tutorial.md
@@ -31,6 +31,7 @@ Please select your desired language for our quest:
   <input type="radio" name="language" value="php">PHP</input>
   <input type="radio" name="language" value="c">C</input>
   <input type="radio" name="language" value="dart">Dart</input>
+  <input type="radio" name="language" value="lua">Lua</input>
 </form>
 \endhtmlonly
 
@@ -136,6 +137,10 @@ For your chosen language, please cross-reference with:
 <div class="language-dart">
 [example.dart](https://github.com/google/flatbuffers/blob/master/dart/example/example.dart)
 </div>
+<div class="language-lua">
+[sample_binary.lua](https://github.com/google/flatbuffers/blob/master/dart/example/sample_binary.lua)
+</div>
+
 
 ## Writing the Monsters' FlatBuffer Schema
 
@@ -322,6 +327,12 @@ Please be aware of the difference between `flatc` and `flatcc` tools.
   ./../flatc --dart monster.fbs
 ~~~
 </div>
+<div class="language-lua">
+~~~{.sh}
+  cd flatbuffers/sample
+  ./../flatc --lua monster.fbs
+~~~
+</div>
 
 For a more complete guide to using the `flatc` compiler, please read the
 [Using the schema compiler](@ref flatbuffers_guide_using_schema_compiler)
@@ -439,6 +450,19 @@ The first step is to import/include the library, generated files, etc.
   import 'monster_my_game.sample_generated.dart' as myGame;
 ~~~
 </div>
+<div class="language-lua">
+~~~{.lua}
+  -- require the flatbuffers module
+  local flatbuffers = require("flatbuffers")
+
+  -- require the generated files from `flatc`.
+  local color = require("MyGame.Sample.Color")
+  local equipment = require("MyGame.Sample.Equipment")
+  local monster = require("MyGame.Sample.Monster")
+  local vec3 = require("MyGame.Sample.Vec3")
+  local weapon = require("MyGame.Sample.Weapon")
+~~~
+</div>
 
 Now we are ready to start building some buffers. In order to start, we need
 to create an instance of the `FlatBufferBuilder`, which will contain the buffer
@@ -518,6 +542,12 @@ which will grow automatically if needed:
   var builder = new fb.Builder(initialSize: 1024);
 ~~~
 </div>
+<div class="language-lua">
+~~~{.lua}
+  -- get access to the builder, providing an array of size 1024
+  local builder = flatbuffers.Builder(1024)
+~~~
+</div>
 
 After creating the `builder`, we can start serializing our data. Before we make
 our `orc` Monster, lets create some `Weapon`s: a `Sword` and an `Axe`.
@@ -705,6 +735,24 @@ our `orc` Monster, lets create some `Weapon`s: a `Sword` and an `Axe`.
   );
 ~~~
 </div>
+<div class="language-lua">
+~~~{.lua}
+    local weaponOne = builder:CreateString("Sword")
+    local weaponTwo = builder:CreateString("Axe")
+
+    -- Create the first 'Weapon'
+    weapon.Start(builder)
+    weapon.AddName(builder, weaponOne)
+    weapon.AddDamage(builder, 3)
+    local sword = weapon.End(builder)
+
+    -- Create the second 'Weapon'
+    weapon.Start(builder)
+    weapon.AddName(builder, weaponTwo)
+    weapon.AddDamage(builder, 5)
+    local axe = weapon.End(builder)
+~~~
+</div>
 
 Now let's create our monster, the `orc`. For this `orc`, lets make him
 `red` with rage, positioned at `(1.0, 2.0, 3.0)`, and give him
@@ -852,6 +900,21 @@ traversal. This is generally easy to do on any tree structures.
   final List<int> treasure = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9];
 ~~~
 </div>
+<div class="language-lua">
+~~~{.py}
+    -- Serialize a name for our mosnter, called 'orc'
+    local name = builder:CreateString("Orc")
+
+    -- Create a `vector` representing the inventory of the Orc. Each number
+    -- could correspond to an item that can be claimed after he is slain.
+    -- Note: Since we prepend the bytes, this loop iterates in reverse.
+    monster.StartInventoryVector(builder, 10)
+    for i=10,1,-1 do
+        builder:PrependByte(i)
+    end
+    local inv = builder:EndVector(10)
+~~~
+</div>
 
 We serialized two built-in data types (`string` and `vector`) and captured
 their return values. These values are offsets into the serialized data,
@@ -964,6 +1027,16 @@ offsets.
   final List<myGame.WeaponBuilder> weaps = [sword, axe];
 ~~~
 </div>
+<div class="language-lua">
+~~~{.lua}
+    -- Create a FlatBuffer vector and prepend the weapons.
+    -- Note: Since we prepend the data, prepend them in reverse order.
+    monster.StartWeaponsVector(builder, 2)
+    builder:PrependUOffsetTRelative(axe)
+    builder:PrependUOffsetTRelative(sword)
+    local weapons = builder:EndVector(2)
+~~~
+</div>
 
 <div class="language-cpp">
 <br>
@@ -1063,6 +1136,16 @@ for the `path` field above:
   ];
 ~~~
 </div>
+<div class="language-lua">
+~~~{.lua}
+    -- Create a FlatBuffer vector and prepend the path locations.
+    -- Note: Since we prepend the data, prepend them in reverse order.
+    monster.StartPathVector(builder, 2)
+    vec3.CreateVec3(builder, 1.0, 2.0, 3.0)
+    vec3.CreateVec3(builder, 4.0, 5.0, 6.0)
+    local path = builder:EndVector(2)
+~~~
+</div>
 
 We have now serialized the non-scalar components of the orc, so we
 can serialize the monster itself:
@@ -1267,6 +1350,22 @@ can serialize the monster itself:
   final int orc = orcBuilder.finish(builder);
 ~~~
 </div>
+<div class="language-lua">
+~~~{.lua}
+    -- Create our monster by using Start() andEnd()
+    monster.Start(builder)
+    monster.AddPos(builder, vec3.CreateVec3(builder, 1.0, 2.0, 3.0))
+    monster.AddHp(builder, 300)
+    monster.AddName(builder, name)
+    monster.AddInventory(builder, inv)
+    monster.AddColor(builder, color.Red)
+    monster.AddWeapons(builder, weapons)
+    monster.AddEquippedType(builder, equipment.Weapon)
+    monster.AddEquipped(builder, axe)
+    monster.AddPath(builder, path)
+    local orc = monster.End(builder)
+~~~
+</div>
 
 Note how we create `Vec3` struct in-line in the table. Unlike tables, structs
 are simple combinations of scalars that are always stored inline, just like
@@ -1409,6 +1508,12 @@ Here is a repetition these lines, to help highlight them more clearly:
   equipped: axe,                                  // Union data
 ~~~
 </div>
+<div class="language-lua">
+~~~{.lua}
+    monster.AddEquippedType(builder, equipment.Weapon) -- Union type
+    monster.AddEquipped(builder, axe) -- Union data
+~~~
+</div>
 
 After you have created your buffer, you will have the offset to the root of the
 data in the `orc` variable, so you can finish the buffer by calling the
@@ -1480,6 +1585,13 @@ appropriate `finish` method.
   // See the next code section, as in Dart `finish` will also return the byte array.
 ~~~
 </div>
+<div class="language-lua">
+~~~{.lua}
+    -- Call 'Finish()' to instruct the builder that this monster is complete.
+    builder:Finish(orc)
+~~~
+</div>
+
 
 The buffer is now ready to be stored somewhere, sent over the network, be
 compressed, or whatever you'd like to do with it. You can access the buffer
@@ -1577,6 +1689,13 @@ like so:
   final Uint8List buf = builder.finish(orc);
 ~~~
 </div>
+<div class="language-lua">
+~~~{.lua}
+    -- Get the flatbuffer as a string containing the binary data
+    local bufAsString = builder:Output()
+~~~
+</div>
+
 
 Now you can write the bytes to a file, send them over the network..
 **Make sure your file mode (or tranfer protocol) is set to BINARY, not text.**
@@ -1690,6 +1809,19 @@ import 'package:flat_buffers/flat_buffers.dart' as fb;
 import './monster_my_game.sample_generated.dart' as myGame;
 ~~~
 </div>
+<div class="language-lua">
+~~~{.lua}
+  -- require the flatbuffers module
+  local flatbuffers = require("flatbuffers")
+
+  -- require the generated files from `flatc`.
+  local color = require("MyGame.Sample.Color")
+  local equipment = require("MyGame.Sample.Equipment")
+  local monster = require("MyGame.Sample.Monster")
+  local vec3 = require("MyGame.Sample.Vec3")
+  local weapon = require("MyGame.Sample.Weapon")
+~~~
+</div>
 
 Then, assuming you have a buffer of bytes received from disk,
 network, etc., you can create start accessing the buffer like so:
@@ -1798,6 +1930,17 @@ List<int> data = ... // the data, e.g. from file or network
 myGame.Monster monster = new myGame.Monster(data);
 ~~~
 </div>
+<div class="language-lua">
+~~~{.lua}
+    local bufAsString =   -- The data you just read in
+
+    -- Convert the string representation into binary array Lua structure
+    local buf = flatbuffers.binaryArray.New(bufAsString)
+
+    -- Get an accessor to the root object insert the buffer
+    local mon = monster.GetRootAsMonster(buf, 0)
+~~~
+</div>
 
 If you look in the generated files from the schema compiler, you will see it generated
 accessors for all non-`deprecated` fields. For example:
@@ -1876,6 +2019,13 @@ accessors for all non-`deprecated` fields. For example:
   var name = monster.name;
 ~~~
 </div>
+<div class="language-lua">
+~~~{.lua}
+  local hp = mon:Hp()
+  local mana = mon:Mana()
+  local name = mon:Name()
+~~~
+</div>
 
 These should hold `300`, `150`, and `"Orc"` respectively.
 
@@ -1969,6 +2119,15 @@ To access sub-objects, in the case of our `pos`, which is a `Vec3`:
   double z = pos.z;
 ~~~
 </div>
+<div class="language-lua">
+~~~{.lua}
+  local pos = mon:Pos()
+  local x = pos:X()
+  local y = pos:Y()
+  local z = pos:Z()
+~~~
+</div>
+
 
 `x`, `y`, and `z` will contain `1.0`, `2.0`, and `3.0`, respectively.
 
@@ -2041,6 +2200,12 @@ FlatBuffers `vector`.
   var thirdItem = monster.inventory[2];
 ~~~
 </div>
+<div class="language-lua">
+~~~{.lua}
+  local invLength = mon:InventoryLength()
+  local thirdItem = mon:Inventory(3) -- Lua is 1-based
+~~~
+</div>
 
 For `vector`s of `table`s, you can access the elements like any other vector,
 except your need to handle the result as a FlatBuffer `table`:
@@ -2122,6 +2287,13 @@ except your need to handle the result as a FlatBuffer `table`:
   var secondWeaponDamage = monster.Weapons[1].damage;
 ~~~
 </div>
+<div class="language-lua">
+~~~{.lua}
+  local weaponsLength = mon:WeaponsLength()
+  local secondWeaponName = mon:Weapon(2):Name()
+  local secondWeaponDamage = mon:Weapon(2):Damage()
+~~~
+</div>
 
 Last, we can access our `Equipped` FlatBuffer `union`. Just like when we created
 the `union`, we need to get both parts of the `union`: the type and the data.
@@ -2257,6 +2429,19 @@ We can access the type to dynamically cast the data as needed (since the
   }
 ~~~
 </div>
+<div class="language-lua">
+~~~{.lua}
+  local unionType = mon:EquippedType()
+
+  if unionType == equipment.Weapon then
+    local unionWeapon = weapon.New()
+    unionWeapon:Init(mon:Equipped().bytes, mon:Equipped().pos)
+
+    local weaponName = unionWeapon:Name()     -- 'Axe'
+    local weaponDamage = unionWeapon:Damage() -- 5
+  end
+~~~
+</div>
 
 ## Mutating FlatBuffers
 
@@ -2337,6 +2522,11 @@ mutators like so:
   <API for mutating FlatBuffers not yet available in Dart.>
 ~~~
 </div>
+<div class="language-lua">
+~~~{.lua}
+  <API for mutating FlatBuffers is not yet available in Lua.>
+~~~
+</div>
 
 We use the somewhat verbose term `mutate` instead of `set` to indicate that this
 is a special use case, not to be confused with the default way of constructing
@@ -2449,5 +2639,9 @@ For your chosen language, see:
 <div class="language-dart">
 [Use in Dart](@ref flatbuffers_guide_use_dart)
 </div>
+<div class="language-lua">
+[Use in Lua](@ref flatbuffers_guide_use_lua)
+</div>
+
 
 <br>
diff --git a/docs/source/doxyfile b/docs/source/doxyfile
index b4e84fa9..8cc46cc6 100644
--- a/docs/source/doxyfile
+++ b/docs/source/doxyfile
@@ -758,6 +758,7 @@ INPUT = "FlatBuffers.md" \
         "TypeScriptUsage.md" \
         "PHPUsage.md" \
         "PythonUsage.md" \
+        "LuaUsage.md" \
         "Support.md" \
         "Benchmarks.md" \
         "WhitePaper.md" \
diff --git a/docs/source/doxygen_layout.xml b/docs/source/doxygen_layout.xml
index c7233143..c0022956 100644
--- a/docs/source/doxygen_layout.xml
+++ b/docs/source/doxygen_layout.xml
@@ -41,6 +41,8 @@
           title="Use in Python"/>
       <tab type="user" url="@ref flatbuffers_guide_use_dart"
           title="Use in Dart"/>
+      <tab type="user" url="@ref flatbuffers_guide_use_lua"
+          title="Use in Lua"/>
       <tab type="user" url="@ref flexbuffers"
           title="Schema-less version"/>
       <tab type="usergroup" url="" title="gRPC">