Control over update mode in JSON and CBOR and doc about itupstream/v0.4

6 files changed, 48 insertions, 25 deletions

diff --git a/docs/custom_serializers.md b/docs/custom_serializers.md
index 1b405f5e..3447189d 100644
--- a/docs/custom_serializers.md
+++ b/docs/custom_serializers.md
@@ -13,13 +13,13 @@ In this case, compiler plugin will generate for you:
 * `.serializer()` method on companion object to obtain serializer. If your class is
 a generic class, this method will have arguments `KSerializer<T1>, KSerializer<T2>`..., where `T1, T2` - your generic type parameters.
 * Special nested object in your class, which implements `KSerializer<MyData>`
-* Methods `save` and `load` of interfaces `KSerialSaver` and `KSerialLoader`
+* Methods `save`, `load` and `update` of interfaces `KSerialSaver` and `KSerialLoader`
 * Descriptor property `serialClassDesc` of `KSerializer`
 
 ## Customizing
 
 If you want to customize representation of the class, in most cases, you need to write your own `save`
-and `load` methods. Serial descriptor property typically used in generated version of those methods,
+and `load` methods. `update` method have default implementation of `throw UpdateNotSupportedException(serialClassDesc.name)`. Serial descriptor property typically used in generated version of those methods,
 so you likely don't need it. 
 
 You can write methods directly on companion object, annotate it with `@Serializer(forClass = ...)`, and serialization plugin will respect it as default serializer.
@@ -98,6 +98,10 @@ object MyDataSerializer: KSerializer<MyData> {
     override fun load(input: KInput): MyData {
         return MyData(stringFromUtf8Bytes(HexConverter.parseHexBinary(input.readStringValue())))
     }
+
+    override fun update(input: KInput, old: MyData): MyData {
+        throw UpdateNotSupportedException(serialClassDesc.name)
+    }
     
     override val serialClassDesc: KSerialClassDesc = SerialClassDescImpl("com.mypackage.MyData")
 }
diff --git a/docs/runtime_usage.md b/docs/runtime_usage.md
index 5754c568..8928e0fa 100644
--- a/docs/runtime_usage.md
+++ b/docs/runtime_usage.md
@@ -2,7 +2,9 @@
 
 ## Obtaining serializers
 
-Serializers are represented at runtime as `KSerializer<T>`, which in turn, implements interfaces `KSerialSaver<T>` and `KSerialLoader<T>`, where `T` is class you serialize. You don't need to call them by yourself; you just have to pass them properly to serialization format. You can write them on your own (see [custom serializers](custom_serializers.md)) or let the compiler plugin do the dirty work by marking class `@Serializable`. To retrieve the generated serializer, plugin emits special function on companion object called `.serializer()`.
+Serializers are represented at runtime as `KSerializer<T>`, which in turn, implements interfaces `KSerialSaver<T>` and `KSerialLoader<T>`, where `T` is class you serialize.
+You don't need to call them by yourself; you just have to pass them properly to serialization format. You can write them on your own (see [custom serializers](custom_serializers.md)) or let the compiler plugin do the dirty work by marking class `@Serializable`.
+To retrieve the generated serializer, plugin emits special function on companion object called `.serializer()`. (This synthetic function is relatively new and stable release of compiler may not able to see it in some cases; if you have some issues, try eap version of `1.2.20`).
 If your class has generic type arguments, this function will have arguments for specifying serializers on type parameters, because it's impossible to serialize generic class statically in general case:
 
 ```kotlin
@@ -36,7 +38,7 @@ In following special case:
 
 You can obtain serializer from KClass instance: `val d: KSerializer<MyData> = MyData::class.serializer()`. This approach is discouraged in general because of its implicitness, but maybe useful shorthand in some cases.
 
-All external serializers (defined by user) must be [registered](custom_serializers.md#registering-and-context) and instantiated in a user-specific way.
+All external serializers (defined by user) are instantiated in a user-specific way. To learn how to write them, see [docs](custom_serializers.md).
 
 ## Serialization formats
 
@@ -44,7 +46,7 @@ Runtime library provides three ready-to use formats: JSON, CBOR and ProtoBuf.
 
 ### JSON
 
-JSON format represented by `JSON` class from `kotlinx.serialization.json` package. It has constructor with four optional parameters:
+JSON format represented by `JSON` class from `kotlinx.serialization.json` package. It following parameters:
 
 * nonstrict - allow JSON parser skip fields which are not present in class. By default is false.
 * unquoted - means that all field names and other objects (where it's possible) would not be wrapped in quotes. Useful for debugging.
@@ -53,6 +55,8 @@ JSON format represented by `JSON` class from `kotlinx.serialization.json` packag
 
 You can also use one of predefined instances, like `JSON.plain`, `JSON.indented`, `JSON.nonstrict` or `JSON.unquoted`. API is duplicated in companion object, so `JSON.parse(...)` equals to `JSON.plain.parse(...)`
 
+You can also specify desired behaviour for duplicating keys. By default it is `UpdateMode.OVERWRITE`. You can use `UpdateMode.UPDATE`, and by doing that you'll be able to merge two lists or maps with same key into one; but be aware that serializers for non-collection types are throwing `UpdateNotSupportedException` by default. To prohibit duplicated keys, you can use `UpdateMode.BANNED`.
+
 JSON API:
 
 ```kotlin
@@ -73,7 +77,7 @@ so it wouldn't work with root-level collections or external serializers out of t
 
 ### CBOR
 
-`CBOR` object doesn't support any tweaking and provides following functions:
+`CBOR` class provides following functions:
 
 ```kotlin
 fun <T : Any> dump(saver: KSerialSaver<T>, obj: T): ByteArray // saves object to bytes
@@ -85,6 +89,8 @@ inline fun <reified T : Any> load(raw: ByteArray): T // save as above
 inline fun <reified T : Any> loads(hex: String): T // inverse operation for dumps
 ```
 
+It has `UpdateMode.BANNED` by default.
+
 **Note**: CBOR, unlike JSON, supports maps with non-trivial keys,
 and Kotlin maps are serialized as CBOR maps, but some parsers (like `jackson-dataformat-cbor`) don't support this.
 
@@ -112,12 +118,11 @@ Number format is set via `@ProtoType` annotation. `ProtoNumberType.DEFAULT` is d
 is signed ZigZag representation (`sintXX`), and `FIXED` is `fixedXX` type. `uintXX` and `sfixedXX` are not supported yet.
 
 Repeated fields represented as lists. Because format spec says that if the list is empty, there will be no elements in the stream with such tag,
-you must explicitly mark any field of list type with `@Optional` annotation with default ` = emptyList()`. Same for maps.
+you must explicitly mark any field of list type with `@Optional` annotation with default ` = emptyList()`. Same for maps. Update mode for Protobuf is set to `UPDATE` and can't be changed, thus allowing merging several scattered lists into one.
 
 Other known issues and limitations:
 
 * Packed repeated fields are not supported
-* If fields with list tag are going in the arbitrary order, they are not merged into one list, they get overwritten instead.
 
 More examples of mappings from proto definitions to Koltin classes can be found in test data:
 [here](../runtime/jvm/src/test/proto/test_data.proto) and [here](../runtime/jvm/src/test/kotlin/kotlinx/serialization/formats/RandomTests.kt#L47)
diff --git a/runtime/common/src/main/kotlin/kotlinx/serialization/cbor/CBOR.kt b/runtime/common/src/main/kotlin/kotlinx/serialization/cbor/CBOR.kt
index 7399cb95..32c95d8b 100644
--- a/runtime/common/src/main/kotlin/kotlinx/serialization/cbor/CBOR.kt
+++ b/runtime/common/src/main/kotlin/kotlinx/serialization/cbor/CBOR.kt
@@ -22,7 +22,7 @@ import kotlinx.serialization.internal.*
 import kotlin.experimental.or
 import kotlin.reflect.KClass
 
-class CBOR(val context: SerialContext? = null) {
+class CBOR(val context: SerialContext? = null, val updateMode: UpdateMode = UpdateMode.BANNED) {
     // Writes map entry as plain [key, value] pair, without bounds.
     private inner class CBOREntryWriter(encoder: CBOREncoder) : CBORWriter(encoder) {
         override fun writeBeginToken() {
@@ -201,6 +201,9 @@ class CBOR(val context: SerialContext? = null) {
             context = this@CBOR.context
         }
 
+        override val updateMode: UpdateMode
+            get() = this@CBOR.updateMode
+
         protected open fun skipBeginToken() = decoder.startMap()
 
         override fun readBegin(desc: KSerialClassDesc, vararg typeParams: KSerializer<*>): KInput {
diff --git a/runtime/common/src/main/kotlin/kotlinx/serialization/json/JSON.kt b/runtime/common/src/main/kotlin/kotlinx/serialization/json/JSON.kt
index 7d0ae3b6..9d0d8018 100644
--- a/runtime/common/src/main/kotlin/kotlinx/serialization/json/JSON.kt
+++ b/runtime/common/src/main/kotlin/kotlinx/serialization/json/JSON.kt
@@ -28,6 +28,7 @@ data class JSON(
         private val indented: Boolean = false,
         private val indent: String = "    ",
         internal val nonstrict: Boolean = false,
+        val updateMode: UpdateMode = UpdateMode.OVERWRITE,
         val context: SerialContext? = null
 ) {
 
@@ -339,6 +340,9 @@ data class JSON(
             context = this@JSON.context
         }
 
+        override val updateMode: UpdateMode
+            get() = this@JSON.updateMode
+
         override fun readBegin(desc: KSerialClassDesc, vararg typeParams: KSerializer<*>): KInput {
             val newMode = switchMode(mode, desc, typeParams)
             if (newMode.begin != INVALID) {
diff --git a/runtime/common/src/test/kotlin/kotlinx/serialization/UpdateTest.kt b/runtime/common/src/test/kotlin/kotlinx/serialization/UpdateTest.kt
index 6e42e288..65642651 100644
--- a/runtime/common/src/test/kotlin/kotlinx/serialization/UpdateTest.kt
+++ b/runtime/common/src/test/kotlin/kotlinx/serialization/UpdateTest.kt
@@ -42,40 +42,46 @@ class UpdateTest {
 
     @Test
     fun canUpdatePrimitiveList() {
-        val parsed = JSON(unquoted = true, nonstrict = true).parse<Updatable1>("""{l:[1,2],f:foo,l:[3,4]}""")
+        val parsed =
+                JSON(unquoted = true, nonstrict = true, updateMode = UpdateMode.UPDATE)
+                .parse<Updatable1>("""{l:[1,2],f:foo,l:[3,4]}""")
         assertEquals(Updatable1(listOf(1,2,3,4)), parsed)
     }
 
     @Test
     fun canUpdateObjectList() {
-        val parsed = JSON(unquoted = true, nonstrict = true).parse<Updatable2>("""{f:bar,l:[{a:42}],l:[{a:43}]}""")
+        val parsed =
+                JSON(unquoted = true, nonstrict = true, updateMode = UpdateMode.UPDATE)
+                .parse<Updatable2>("""{f:bar,l:[{a:42}],l:[{a:43}]}""")
         assertEquals(Updatable2(listOf(Data(42), Data(43))), parsed)
     }
 
     @Test
     fun cantUpdateNotUpdatable() {
         assertFailsWith<UpdateNotSupportedException> {
-            JSON(unquoted = true, nonstrict = true).parse<NotUpdatable>("""{d:{a:42},d:{a:43}}""")
+            JSON(unquoted = true, nonstrict = true, updateMode = UpdateMode.UPDATE).parse<NotUpdatable>("""{d:{a:42},d:{a:43}}""")
         }
     }
 
     @Test
     fun canUpdateNullableValuesInside() {
-        val a1 = JSON.parse<NullableInnerIntList>("""{data:[null],data:[1]}""")
+        val json = JSON(updateMode = UpdateMode.UPDATE)
+        val a1 = json.parse<NullableInnerIntList>("""{data:[null],data:[1]}""")
         assertEquals(NullableInnerIntList(listOf(null, 1)), a1)
-        val a2 = JSON.parse<NullableInnerIntList>("""{data:[42],data:[null]}""")
+        val a2 = json.parse<NullableInnerIntList>("""{data:[42],data:[null]}""")
         assertEquals(NullableInnerIntList(listOf(42, null)), a2)
-        val a3 = JSON.parse<NullableInnerIntList>("""{data:[31],data:[1]}""")
+        val a3 = json.parse<NullableInnerIntList>("""{data:[31],data:[1]}""")
         assertEquals(NullableInnerIntList(listOf(31, 1)), a3)
     }
 
     @Test
     fun canUpdateNullableValues() {
-        val a1 = JSON.parse<NullableUpdatable>("""{data:null,data:[{a:42}]}""")
+        val json = JSON(updateMode = UpdateMode.UPDATE)
+        val a1 = json.parse<NullableUpdatable>("""{data:null,data:[{a:42}]}""")
         assertEquals(NullableUpdatable(listOf(Data(42))), a1)
-        val a2 = JSON.parse<NullableUpdatable>("""{data:[{a:42}],data:null}""")
+        val a2 = json.parse<NullableUpdatable>("""{data:[{a:42}],data:null}""")
         assertEquals(NullableUpdatable(listOf(Data(42))), a2)
-        val a3 = JSON.parse<NullableUpdatable>("""{data:[{a:42}],data:[{a:43}]}""")
+        val a3 = json.parse<NullableUpdatable>("""{data:[{a:42}],data:[{a:43}]}""")
         assertEquals(NullableUpdatable(listOf(Data(42), Data(43))), a3)
     }
 }
\ No newline at end of file
diff --git a/runtime/jvm/src/test/kotlin/kotlinx/serialization/features/UpdateCustomTest.kt b/runtime/jvm/src/test/kotlin/kotlinx/serialization/features/UpdateCustomTest.kt
index b916746f..ba570fbb 100644
--- a/runtime/jvm/src/test/kotlin/kotlinx/serialization/features/UpdateCustomTest.kt
+++ b/runtime/jvm/src/test/kotlin/kotlinx/serialization/features/UpdateCustomTest.kt
@@ -16,12 +16,9 @@
 
 package kotlinx.serialization.features
 
-import kotlinx.serialization.KInput
-import kotlinx.serialization.Serializable
-import kotlinx.serialization.Serializer
+import kotlinx.serialization.*
 import kotlinx.serialization.internal.IntSerializer
 import kotlinx.serialization.json.JSON
-import kotlinx.serialization.list
 import org.junit.Test
 import kotlin.test.assertEquals
 
@@ -44,7 +41,9 @@ class UpdateTest {
 
     @Test
     fun canUpdateCustom() {
-        val parsed = JSON(unquoted = true, nonstrict = true).parse<Updatable>("""{d:{a:42},d:{a:43}}""")
+        val parsed =
+                JSON(unquoted = true, nonstrict = true, updateMode = UpdateMode.UPDATE)
+                        .parse<Updatable>("""{d:{a:42},d:{a:43}}""")
         assertEquals(Data(42 + 43), parsed.d)
     }
 
@@ -53,13 +52,15 @@ class UpdateTest {
 
     @Test
     fun canUpdateMap() {
-        val parsed = JSON.parse(WrappedMap.serializer(IntSerializer), """{"mp": { "x" : 23, "x" : 42, "y": 4 }}""")
+        val json = JSON(updateMode = UpdateMode.UPDATE)
+        val parsed = json.parse(WrappedMap.serializer(IntSerializer), """{"mp": { "x" : 23, "x" : 42, "y": 4 }}""")
         assertEquals(WrappedMap(mapOf("x" to 42, "y" to 4)), parsed)
     }
 
     @Test
     fun canUpdateValuesInMap() {
-        val parsed = JSON.parse(WrappedMap.serializer(IntSerializer.list), """{"mp": { "x" : [23], "x" : [42], "y": [4] }}""")
+        val json = JSON(updateMode = UpdateMode.UPDATE)
+        val parsed = json.parse(WrappedMap.serializer(IntSerializer.list), """{"mp": { "x" : [23], "x" : [42], "y": [4] }}""")
         assertEquals(WrappedMap(mapOf("x" to listOf(23, 42), "y" to listOf(4))), parsed)
     }
 }
\ No newline at end of file