| Age | Commit message (Collapse) | Author |
|---|
|
* stable:
Don't delete values from MySQL sequence tables
Conflicts:
pom.xml
Change-Id: I55e6e676fc339ea4c38dc0bafa4c2cbbb5baf43d
|
|
At server restart MySQL uses the maximum value in the table's
auto_increment column to restart the sequence. Deleting rows
causes this to always be 0, making sequences restart from 1 on
the next server start.
Change-Id: I50df069da1266161d21178359b45e6328f57e0c0
|
|
We changed the signature of the method, but didn't
update all of the callers that use queryList().
Change-Id: Id7d007e489d29f43da1eac645e49a118b46f4be8
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
When a secondary index row is scanned and it doesn't cache the main
object data, load all of the matching object rows in one call to
the database. This permits the implementation to batch the lookups
into a single call, reducing total latency.
Change-Id: I353329e65f0c75dcfe134a506292b999c97e4648
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
Performing multiple sequential row lookups can be inefficient,
so try to simplify the implementation of batching here by pushing
it down into the GenericSchema.
Change-Id: I770e8690d48ea1c22cb55f96dd636498b5a938bc
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
Implementing Map.Entry<byte[], byte[]> is painful, and its just not a
good interface given that it has a setValue method. Replace usages
of it in the NoSQL code with our own Row type that has a similar
API for the getters.
Change-Id: I9144d615e1fa9b5d0df9727c3d39b75db06f024a
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
Implementations of our NoSQL backend are likely to optimize
the GenericSchema.fetchRow method, but might not optimize their
GenericAccess.scanPrimaryKey routine. Redirect the single key
lookup through the fetchRow method instead.
Change-Id: If7b47c66077a2e24e6116f7f8035cf73ef260e6e
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
Just like with a SQL database, a NoSQL system might be able to return
results out of order more efficiently than in-order when there is
no explicit ORDER BY clause in the query. Support that by passing
through a hint indicating if the order needs to be preserved, or not.
Change-Id: Ic601fd3277936e599b590c508e2435bc5bf5f95b
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
This permits the NoSQL implementation to be informed when we abort
a scan early, because its scan routine can implement and receive the
close() method. It also gives us a way to stream the JDBC results,
so we aren't creating a full result set in memory.
Change-Id: Iaf53a1803fe985e49236e10bde5d838846e0c8f0
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
If the application is using upsert we can avoid performing a query
against the database for the old object and just blind-write the new
and old index rows. Since most NoSQL systems have higher round-trip
latency than they do per-row operation costs this should improve the
performance of most upsert operations.
During upsert we do try to avoid unnecessary index updates by
tracking the old copy of an object in a local cache.
Change-Id: I2f2aa7269f4049b86d5cb2c8d2079263335f37ec
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
Most NoSQL systems require one or more RPCs to remote servers, which
has a much higher latency than a write to a local MySQL server.
To support setAutoFlush(false) in these cases we want to allow the
schema handle to buffer any writes made by the access interfaces
until the current operation is over.
Change-Id: Ib642bbf021b25f054b0777d2d4302161300a3b4b
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
The flag controls whether or not writes are sent immediately to
the database. By delaying writes larger units can get sent in a
single request. Right now we only plan to make use of the flush
hint on NoSQL based systems.
Change-Id: I79b2a1aec822e1c4b6abf5c6788429ab7becce1d
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
This helps callers that want to know what the identity is for a
given relation when its wrapped up in a protobuf.x
Change-Id: I5c0730a832c44be59a55f01cf93c1ff5ee9f0832
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
This permits someone to write a tool that iterates through the
schema, such as to dump it to an external backup file.
Change-Id: I58049e5e66ff827c59959823df5dca787cc7db2e
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
Because we store all relations in effectively the same column, we
use a different protobuf field number wrapped around the individual
entity message. This allows scanning programs that just examine
every row to be able to tell different relation's types apart and
correctly inspect the fields of each entity.
Change-Id: I26161a69ba78e97f0145db3d781ae60e96f66e0e
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
I added support for generating a file containing all the Protobuf
type definitions for a given database schema. The JavaSchemaModel now
has a method to output a Protobuf definition to a given PrintWriter.
This allows tools using a NoSQL database to have a Protobuf file
describing the Protobuf data in the database.
Change-Id: I53f4902d2b335be2b20d966fd0eb58c050e8a53c
|
|
In order to support folding all database data into one protobuf type,
we need to tag each relation that needs to be stored with a unique
ID number. That will be used as the ID for the protobuf message
representing that relation.
Change-Id: I6f4a7a24d0c42f37399d82f37f434388aa63710b
|
|
When encoding to a stream we prefix the message with a varint
listing its length. When decoding we read that varint, then use an
input stream wrapper which limits the number of bytes we consume
to that length. This way the stream is left unaffected for the
next data segment, which might be something else.
Callers can use these to implement java.io.Serializable using their
own ProtobufCodec rather than relying on Java serialization.
Change-Id: I17fc5863b525fad22d14653831024eaf5b4ff9da
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
If a class to be encoded has no @Column annotations the CodecGen
should throw an Exception, as a class with no @Column annotations
cannot be correctly represented on protobuf. I added a check to
scanFields to check for this, and throw an appropriate exception.
Change-Id: Ib6962d1b2e05e5b494ca0cee2091caa1f5daa731
|
|
Encode enum values using their ordinal as a varint, similar to
what the proto compiler from Google would do. This requires that
application developers carefully assign their enum orders in order
to maintain values already encoded in protobufs.
Change-Id: I1f40170c583125f425f4ff4a3aa7045ee0a820ca
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
A typo had us using the wrong method name.
Change-Id: If67b526639628e5a6bcb90ffcace84428c28a92f
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
Some object types might not be annotated with @Column, but we may
still want to embed them into a protobuf format. Treat these as
opaque binary messages using the length delimited message format,
encoding them like any other nested message or arbitrary binary.
Change-Id: I8139e36838cbb2eabacbf3eab8c0156ddbdeb29b
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
Repeated fields are now supported for List<String>, Set<String>,
and List/Set of any protobuf encodable entity. These repeated
columns work only in the protobuf model, and fail horribly if
used in a schema running on SQL.
This is a small hack to try and implement some simple encoding of
cache values for Gerrit Code Review, to support storing the cached
data into byte string based systems like the open source memcached,
without relying on expensive Java serialization.
As a side effect of this change, nested message support was rewritten
to acquire the ProtobufCodec for the nested message and reference
that instance through a static field. This reduces the number of
times a common message's implementation exists in the JVM down to
only once, which may help the JIT to produce more optimal code.
It also can help to avoid costly field traversal on deeply nested
messages, as the message gets pushed onto the call stack rather
then being repeatedly fetched from the parent.
Invocation of the nested message encoders is made through
a type-specific overload of the generic Object based methods,
avoiding an unnecessary checkcast at runtime as the code enters
into the nested message.
Change-Id: I1a6f2ffc584adb8fda716edb7e42edd369e1f527
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
We can do better with our encode/decode passes by taking better
advantage of the various APIs expose through CodedInputStream
and CodedOutputStream to support fewer copies for a number of
different buffer types, including java.nio.ByteBuffer and the
protobuf ByteString wrapper.
During encode we no longer recursively encode a nested object into
a temporary ByteString and append it to our own buffer, but instead
compute its size via the same logic as sizeof(), and then do the
encoding of the data. Most nested messages will be fairly small
and thus quite easy to compute their nested size, so avoiding the
copy will improve encoding performance.
Also cleaned up the way sizeof generates, so decompiled code is
read correctly as "i += sz" rather than "i = sz + i".
Also fixed the decode function so that the input ByteBuffer is
correctly positioned at the end with no remaining input if we have
consumed everything in the stream.
Change-Id: Ie308c071c4302c55938a2e3cb014433d0c34e7aa
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
|
|
Some storage systems might prefer to use java.nio.ByteBuffer for
decoding, so provide a utility method for those.
Change-Id: Ib57caf90c5cbfe8dfda822abb42bf3787146f0c1
|
|
The documentation says a sequence starts at 1, but we were starting
from 0 when the starting value wasn't specified in the annotation.
Fix it by starting from 1 if the annotation has 0.
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
We can simplify the surface that an implementation has to deal with
by combining some actions through the atomicUpdate() method, and
just exposing more of the internal bits of GenericAccess in case the
implementor wants to roll more of their own logic for per-row edits,
but still wants our standard schema layout inside of the key space.
Single row lookup can be approximated by a scan over the primary
key, which we need to support anyway for certain types of queries.
We can drop the key IndexFunction and simplify the code by using
primaryKey(T) to obtain the primary key of an object, then encode
it using the encodePrimaryKey method.
We can avoid creating indexes for queries that fetch all records
in the relation by having those use the primary key index instead.
This should reduce the number of indexes needed for at least some
objects, but the reduction is still incomplete. We can use the
primary key index in a lot more situations than just an "all" query.
Change-Id: Ifc2896df6dc017c7e8a026dff90a0811f8fcda28
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
Change-Id: I6863f7b2241fc13ae169a7d19e013cdfb76334b0
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
Use the startValue from the schema when starting a new counter
pool. This way Gerrit Code Review's account_id field will be
starting from a different value range than its account_group_id.
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
This is a simple base that a specific NoSQL implementation can
build on top of as part of our public API.
Trivial implementations are supplied based upon a simple TreeMap in
the JVM, and persisting that map to disk. These can be useful for
development and testing of applications built on top of the library.
Change-Id: Ida7fdfa4aca5db5cfbc1ca7b7bbec5af70b551fc
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
The IndexFunction provides a way to construct a secondary row key
for an object instance, given some query that needs to scan along
that particular subset of fields. Filtering for constants and
any nulls is done in Java prior to constructing the key, such that
these are omitted from the index.
This may differ from how a SQL implementation would sort the same
information in an ORDER BY clause, but we shouldn't need to be
sorting by nullable columns.
Change-Id: I9cfecfa58420c8629aee8053beec075c34a139a6
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
In a NoSQL system we usually don't have a secondary index available
for columns and instead must fake that ourselves by creating a new
row that is a composite of the various columns. To ensure sorting
occurs properly in the NoSQL key range space, we need to encode the
column values to build the composite key.
Change-Id: Ieefe9757f0005bb55e30c3dc0786769bf2d402fb
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
Require that the query name be unique within the access
interface, so that the implementation can use it as the
index name within the data store.
Change-Id: I7e73878c8b12a7ae8a758d84025c5383343c100b
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
This is a minor refactoring to use the generic interface that
simplifies some other backend code I haven't yet committed.
Change-Id: I29cd3f451167b34080769916cd645d943d0b2ca4
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
These can be useful in other backends, as they are quite simple
implementations that are built on top of other primitives.
Change-Id: I847ce3b27c18ff4de65d7d5a69cd6690f99ff7bd
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
When we build a new Schema instance what we really wanted to
pass through was the SchemaFactory (e.g. jdbc.Database) and
not the Schema type itself (since that's redundant). Change
the API to pass through the database argument.
Change-Id: I2fa015070b00ddcbdc308c940281fb9d21296046
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
We forgot to push the field value onto the stack, so we couldn't
invoke getTime() on the timestamp object. This resulted in a class
verification error.
We also failed to denote the constructor as taking a primitive long,
causing a class verification error when a long was found where an
uninitialized object was expected.
Change-Id: I4c60fac33f2c1c8dea02d97ac6ad4522ffacf3b6
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
Some Java compilers won't compile this code correctly, so relax
the type annotations to be less complex.
Change-Id: Iea176366e81eac219af9417f106907aeaace42b8
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
AccessGen is only needed in our .jdbc package, so move it there
and make the API package-level only rather than public. This way
its not accidentally used by another backend, because its really
quite JDBC specific.
Move CodeGenSuppor to .server package alongside other common code
generation classes, as its used by not only the JDBC based AccessGen
but also by the protobuf CodecGen by overriding JDBC specific output
methods with protobuf based variants of the same function.
Change-Id: I4ef50add4dfbd47dcf2a4fd86a34888d19f9bb20
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
The entire schema implementation is just about constructing the
various methods that return the Access implementations for each
relation. This isn't unique to JDBC and can easily be reused
for a NoSQL backend as-is, if we properly abstract out the base
class it uses and the method to obtain each Access implementation
as its required by the schema code generator.
Change-Id: If9b1d63b1b83fafb1eab14b64861cd26dde73d2e
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
By changing the nextLong method to generate the SQL on the fly within
the increment rather than embedding it into the generated schema
implementation we can later refactor the schema code generator to be
more generic for the NoSQL type of backend where we don't necessarily
have a query statement. This also cleans up the corner case of
the MySQL dialect where we don't have a single query statement.
Change-Id: Id29923f742c847e16e48655b928ab40f7c827c0f
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
Using a WeakHashMap to cache transient objects which are created in
our constructor and quickly discarded is useless. Its very likely the
GC will purge these, and the map will lose the binding, so we just
generate the code again later.
Besides, applications aren't supposed to create Database objects
often. Instead they create them once and hold onto it, resuing
the object as a factory/connection pool.
Change-Id: Ia6dc43757443ca60820c066393a1746fc7a2f001
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
Rather than repeat this code for every backend, allow the backend to
reuse our existing code and just initialize any state within the
Schema implementation's constructor, and not in the SchemaFactory
open() method.
Change-Id: Ie4509c61d85cf38fdb1159647bbc2dbd5b85f5d5
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
Most NoSQL products cannot support the != operator, as this does
not provide a sufficient reduction in rows within the table space.
Remove the operator so that applications don't try to take advantage
of a feature that isn't fully portable.
Change-Id: I6aeb5fcc8e01428244d61d86c9466fbc2331cc28
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
Some NoSQL-type products cannot perform an ORDER BY x DESC in
a query. Remove support for it in gwtorm so that applications
don't try to take advantage of a feature that isn't fully portable.
Change-Id: I8fe011dbaa389ddbcfdb4a8ddee0102569213f7a
|
|
We use this ClassLoader to bring in any generated code, not just
the JDBC backend. Its also currently used by the protobuf code,
which isn't JDBC specific at all.
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
When we removed Transaction support we lost the only reason for the
indirection here. Drop that indirect method and implement the
interface directly in the JdbcAccess subclass.
Signed-off-by: Shawn O. Pearce <sop@google.com>
|
|
|
|
Removed transactions to support databases that do not have cross-row
transactions, e.g. MySQL on MyISAM tables or Apache Cassandra. OrmRunnable
was also removed.
Change-Id: I7ae90d73250d3059f71b56b03c257fbd71c2ea6f
|
