Reviewed and marked all security critical files in the
qtgrpc/src/protobuf directory.
More information: https://contribute.qt-project.org/quips/23
Fixes: QTBUG-135456
Pick-to: 6.10
Change-Id: I9cc05039ca5830d04816ada978ec193ece154732
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
Remove the duplicated preserveUnknownFields variable in
QProtobufSerializerImpl, use the parent pointer to acquire its
value instead.
Pick-to: 6.8 6.9
Task-number: QTBUG-128812
Change-Id: I8c81e5ea6565f056756703d5ca989b67d163dd19
Reviewed-by: Dennis Oberst <dennis.oberst@qt.io>
When looking for non-packed deserializer we may make the lookup only
once, since we may predict the non-packed field from the wire type.
If it differs from the Length Delimited type and the field is Repeated
according to the stored meta info, we may conclude that the received
data is non-packed and use the corresponding deserializer.
Pick-to: 6.8 6.9
Task-number: QTBUG-128812
Change-Id: I3ead7e8b64cda26179c7b5fc9917737b41e22882
Reviewed-by: Dennis Oberst <dennis.oberst@qt.io>
Move the scalar type serializers to a separated translation units. This
make code more readable. Adjust constraints to follow common project
coding standards and make constrains compilation faster. Make complex
functions out-of-line to avoid unwanted binary expansion.
Pick-to: 6.8 6.9
Task-number: QTBUG-128812
Change-Id: Ib5311f392e32e5f31416448d2f306ae91155f873
Reviewed-by: Dennis Oberst <dennis.oberst@qt.io>
Move the generic logic of serializer to a separate class and
use this class to implement the protobuf and JSON deserialization.
This also fixed one of the conformance usecases since related
to invalid enum handling.
Pick-to: 6.8 6.9
Task-number: QTBUG-128812
Fixes: QTBUG-112423
Fixes: QTBUG-112425
Fixes: QTBUG-112424
Change-Id: I400303a8666df90050a54bd7036daa0107adcce5
Reviewed-by: Dennis Oberst <dennis.oberst@qt.io>
Move the generic logic of serializer to a separate class and
use this class to implement the protobuf and JSON serialization.
This also fixed one of the conformance usecases since related
to invalid enum handling.
Pick-to: 6.8 6.9
Task-number: QTBUG-128812
Change-Id: Ie78da9103e68e7701b36f7d4e4c8e83f38d3c5c2
Reviewed-by: Dennis Oberst <dennis.oberst@qt.io>
The namespace contains the functionality that was a part of
QProtobufSerializerPrivate before. Move this functionality to a
separate file, since we want to unlink QProtobufJsonSerializer
implementation from the QProtobufSerializer one.
Pick-to: 6.8
Task-number: QTBUG-128812
Change-Id: I7b97ced9fbfa1076b2e898832575c1931923a2c3
Reviewed-by: Dennis Oberst <dennis.oberst@qt.io>
The actual design injects the weak vtable issue when we
implement the QtProtobufPrivate::AbstractRepeatedIterator interface
inline. Instead of using this interface simulate the limited vtable
we need in QProtobufRepeatedIterator. The class now holds the required
function pointers with the calls of the methods from the template
QtProtobufPrivate::ListIterator and QtProtobufPrivate::MapIterator
classes.
Drive-by change: Move fromList(QList<T> &list) and
fromHash(QHash<K, V> &hash) methods to QProtobufRepeatedIterator
they should've belong this class from the beginning.
Pick-to: 6.8
Task-number: QTBUG-123626
Change-Id: I5b488c69d761b31d4d9d08f2015e65cba19819e3
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
All new enums should be scoped.
Drive-by change: Remove the redundant suffixes of the enum elements.
It's obvious from the scope that those are errors.
Pick-to: 6.8
Task-number: QTBUG-123626
Change-Id: Ided4ab4a1ddb2d14713f876ffe4fe339af39ae6b
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
Rename the deserializationError(String) interfaces to
lastError(String). The new interface encourages to write the errors
not only during deserialization but also serialization.
TODO: Add serialization error handling to the existing
QAbstractProtobufSerializer implementations. See QTBUG-128331.
[ChangeLog][Protobuf] QAbstractProtobufSerializer::DeserializationError
is renamed to QAbstractProtobufSerializer::Error and now contains list
of both possible serialization and deserialization errors.
[ChangeLog][Protobuf][QAbstractProtobufSerializer] deserializationError
and deserializationErrorString renamed to lastError and lastErrorString
respecively. Interfaces return the error from the last operation,
either serialization or deserialization.
Pick-to: 6.8
Task-number: QTBUG-123626
Change-Id: I705eb69e07393aa6601cc79712845200873235a0
Reviewed-by: Dennis Oberst <dennis.oberst@qt.io>
- Unify the include style.
- Use forward declarations where applicable.
- Remove unused includes.
Pick-to: 6.8
Task-number: QTBUG-123626
Change-Id: I40b2d7428079145c5828c3579aa3f840c343390e
Reviewed-by: Dennis Oberst <dennis.oberst@qt.io>
Change the serializing mechanism that is used to serialize protobuf
repeated and map fields. We used the serializer registry to serialize
and deserialize repeated and map fields, which exposed the registry
as public API and was quite ugly. Meanwhile QMetaType allows to
register the mutable zero-copy converter and access meta-type the
special way we want, using QMetaType::registerMutableView.
This introduces QProtobufRepeatedIterator, special class that allows
accessing and adding new elements to the protobuf repeated and map
fields, that are represented as QList<T> and QHash<K, V> respectively.
Register the mutable view to this new class for all related classes.
QProtobufRepeatedIterator is a wrapper around
QtProtobufPrivate::AbstractRepeatedIterator interface and manages the
allocated AbstractRepeatedIterator implementation.
QtProtobufPrivate::AbstractRepeatedIterator got two implementations for
QList and QHash.
The serializer registry is moved to private API but stil can be and is
used to customize the serialization using the proxies.
[ChangeLog][Protobuf] QtProtobufPrivate::SerializationHandler,
QtProtobufPrivate::registerHandler and related functionality are now
private API.
[ChangeLog][Protobuf] Added QProtobufRepeatedIterator. QVariant that
contain repeated or map protobuf values can be converted to this class
using QVariant::view method.
Pick-to: 6.8
Task-number: QTBUG-123626
Change-Id: I282318e82596839ff436bb33a1bed5f3474488fd
Reviewed-by: Dennis Oberst <dennis.oberst@qt.io>
Replace the interface methods with the qxp::function_ref with the
respective arguments. This will break the dependency between the
protobuf message registry and the QAbstractProtobufSerializer.
Now the registry is standalone unit with own rules that can be used
or not used by the serializers. If serializers want to use the
registry, they now need to supply it with the respective
(de)serializers for the message fields of the protobuf message type.
QAbstractProtobufSerializer implementations meanwhile only need to
know how to (de)serialize the whole messages but not fields. This
was the last chunk of the interface cleaning.
Adjust the implementation of the existing serializers respectively.
Pick-to: 6.8
Task-number: QTBUG-123626
Change-Id: I772a330e8f9757d998f0e7ea138d12bbfce020b8
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Property-reading related API can be a separate functions that are used
by our serializer implementations. Move all such methods from
QProtobufMessage to a private header.
Pick-to: 6.8
Task-number: QTBUG-123626
Change-Id: Iffe4e32932d163bbd5008f00842c038387b3bd0f
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
The function is used by serializer implementations to find a custom
serializer for the specific field. We shouldn't expose it as public
API with no purpose. Move to the private header and export for
further use without binary compatibility promises.
Pick-to: 6.8
Task-number: QTBUG-123626
Change-Id: I755c0a9709519707051d4b070395b31b6284767b
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Dennis Oberst <dennis.oberst@qt.io>
Replace the QVariant argument property value with the void pointer one.
We should use the inner QVariant data anyway to avoid unnecessary
copies. This change should also address the hidden detaches of the
QVariant values.
Since the pool of (de)serialize handlers is restricted and always
is binded to the exact type, we allow use void pointer as the type
there.
Task-number: QTBUG-123626
Pick-to: 6.8
Change-Id: I47749b8f78b84e3d627efd270260d6545bce3240
Reviewed-by: Dennis Oberst <dennis.oberst@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
We can use QExplixitSharedDataPointer as the underlying class that
holds the QProtobufMessagePrivate instead of making the explicit
ref counting.
Make the QProtobufMessagePrivate::storeUnknownEntry static and make
the explicit detach there instead.
Task-number: QTBUG-123626
Pick-to: 6.8
Change-Id: I0e9b19af3b1a38635d11743db8a3a0fe8d97a922
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Move the docs to the actual function implementations and adjust
the text to the one is used in the QAbstractProtobufSerializer.
Pick-to: 6.8 6.7
Change-Id: Ie7aa366e4b9a77c6c45d5a96fe627f50cb3f9df6
Reviewed-by: Dennis Oberst <dennis.oberst@qt.io>
Covert the QtProtobufPrivate::FieldFlag enum to a enum class.
This will ensure that enum values will not conflict in the future.
Task-number: QTBUG-123626
Pick-to: 6.8
Change-Id: Ibd4d68ac4b27087e5e9e35ba6e4e28242b2cd893
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
C++ advices not to use the underscore prefixed identifiers. Rename
TransparentWrapper::_t accordingly.
Pick-to: 6.8
Task-number: QTBUG-123626
Change-Id: Iaee3260f7c253a631dab014e3ec523f8b02b49f9
Reviewed-by: Dennis Oberst <dennis.oberst@qt.io>
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
When deserializing protobuf messages, cache the property values in
the serializer. The cached value life-time is limited by field
number change. This improves the performance of the deserializer
when deserializing repeated fields. So before storing the new value
into the message we first collect all values of the repeated field
that we received from the wire and only when we recognize the change
of the field in the byte stream, update the property value.
Fixes: QTBUG-124594
Pick-to: 6.8 6.7
Change-Id: I5ce5d5c81bf0f7ebf071b5ad374a46787b2767e8
Reviewed-by: Dennis Oberst <dennis.oberst@qt.io>
The methods applied the conversion from QList<EnumType> to
QList<QtProtobuf::int64>. This can be done on the QMetaType level
using the covert function. Add the conversion from/to
QList<QtProtobuf::int64> for the QList<EnumType> and the conversion
from/to QStringList, which should be used in JSON serialization.
Pick-to: 6.8
Task-number: QTBUG-123626
Change-Id: Id0300536a8eda3503540089ba7bc29c1cd249752
Reviewed-by: Tatiana Borisova <tatiana.borisova@qt.io>
Use QProtobufPropertyOrderingBuilder to supply serializers with the
required information about map fields.
Pick-to: 6.8
Task-number: QTBUG-123626
Change-Id: I625b9003b87d010a8b563782f50dedc332f20293
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Dennis Oberst <dennis.oberst@qt.io>
The header should contain the useful for the implementation definitions
and constants.
This adds the ProtobufFieldNumMin and ProtobufFieldNumMax constants
which reflect the minimum and maximum field numbers for the protobuf
messages.
Use the newly introduced constants in the places where values were
duplicated.
Pick-to: 6.8
Change-Id: I27c253dc4102eea04379204f72c0d6d2376fdfbb
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Tatiana Borisova <tatiana.borisova@qt.io>
Reviewed-by: Dennis Oberst <dennis.oberst@qt.io>
If we were unable to find a deserializer for a given basic type we would
try to print out the wire type of the deserializer. But we didn't
necessarily have any.
Pick-to: 6.8 6.7
Change-Id: I8ccfc7828afb2d24c9a487ec9a5f88db82c71381
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
We were previously just restoring the iterator in one branch.
This made it really hard to debug unrelated issues that happened to
leave through another branch.
This changes what iterator we are checking for validity before leaving
the function, but the iterator shouldn't have consumed more data than
is available in the buffer, so it should be valid anyway.
Pick-to: 6.8 6.7
Change-Id: Iaa818a54961feacb50a20a1f9770a05308cb2485
Reviewed-by: Tatiana Borisova <tatiana.borisova@qt.io>
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
QFlags is a nice convenience type, and it makes it more consistent with
our generated code, which already uses it.
Change-Id: Ic588eec4c427c6dc148e3d068b770c8665f944a6
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
If we are confident that the property is message, we shouldn't attempt
to look for other serializers.
Pick-to: 6.7
Change-Id: I61c76d9dc2c43d5b8a91627d1e3a7aaebbb8d166
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
The struct represents the protobuf field information, so rename
it accordingly.
Task-number: QTBUG-123626
Change-Id: I79605a349f35e73c9fc808c76073d5c1c83c9731
Reviewed-by: Tatiana Borisova <tatiana.borisova@qt.io>
QProtobufMessage::propertyOrdering now provides information
about the message property ordering. This allow us to use
the pointers to QProtobufMessage but not to the exact class
when (de)serializing messages.
This commit makes all private getter of message fields mutable.
If the message field is not initialized it will be initialized
at first access, like it's done in public getters.
This mechanism allows to hold the field allocation inside message
but not in the type deserializer. All serializers no should expect
that the message pointer that is passed to the deserialization
is valid message pointer.
To avoid unwanted message field allocation when using property
getter, QProtobufMessage::property method now accepts an extra argument.
If allowInitializeOptional is set to true, it skips checking the
'has<MessageField>' property before accessing the message field it self.
Protobuf messages that are used in oneof fields now require explicit
definition of operator=, since it's used when copying the data in
QVariant. Otherwise the warning related to the implicit definition of
operator= is produced.
Task-number: QTBUG-120931
Change-Id: I1b816295297e7ef8cd6d3f0abdcbe1e670597fd0
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
QProtobufMessage::propertyOrdering now allows accessing the message
property ordering. So we can remove extra arguments from all interfaces
where the pointer or the reference to the related QProtobufMessage
is used as argument too.
[ChangeLog][Protobuf][QAbstractProtobufSerializer] The following
interfaces do not accept the QProtobufPropertyOrdering as an argument
anymore:
- serializeObject
- deserializeObject
- serializeListObject
- deserializeListObject
- serializeMessage
- deserializeMessage
Task-number: QTBUG-120931
Change-Id: I55cb8d3ceddd56751d542869e26aa0de73c22ebb
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Tatiana Borisova <tatiana.borisova@qt.io>
`isOneofOrOptionalField` is duplicated in different c++ files. with
unity builds, they are built into the same translation unit. moving the
function to the header fixes the name clash
Change-Id: If4b1a3d12029e5a575431496eb41cfa6f83082c3
Pick-to: 6.7
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
Move functionality related to the registration of the protobuf messages
to a separate header/source files.
Task-number: QTBUG-120931
Change-Id: I6076b41139d43982148e46f5f315808509c4db65
Reviewed-by: Tatiana Borisova <tatiana.borisova@qt.io>
Fix links to qtprotobufgen in QProtobufSerializer and
QAbstractProtobufSerializer docs.
Pick-to: 6.5 6.6 6.7
Change-Id: Ib17438308f3e5a27e4093a3c52bd3b20c6a442b5
Reviewed-by: Leena Miettinen <riitta-leena.miettinen@qt.io>
QProtobufSerializer was never "reentrant" since ported from Qt5 version.
Pick-to: 6.5 6.6 6.7
Change-Id: If559cee5171d7a034cdab4eba796d0215e0d51b3
Reviewed-by: Tatiana Borisova <tatiana.borisova@qt.io>
The map pair require explicit "optional" flag, since all map values
should be serialized independent of field presence. Use the
pre-generated ordering info of the following message:
message pair {
optional string key = 1;
optional string value = 2;
}
The field type is not considered when deserializing map pairs.
Also remove the previous map pair ordering functions from
QProtobufPropertyOrderingInfo.
[ChangeLog][QtProtobuf] Removed infoForMapKey and infoForMapValue
methods from the QtProtobufPrivate::QProtobufPropertyOrderingInfo
structure.
Pick-to: 6.7
Change-Id: I6c4411a35ac1d19b669b1acd8b56a41bf5a0dda8
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
We should clear the previously set error when start deserializing
messages.
Amends e3abfef0cd
Pick-to: 6.7
Change-Id: I5e58af831217539ab75b48558e696fb4f661b7c8
Reviewed-by: Tatiana Borisova <tatiana.borisova@qt.io>
Implement the QProtobufJsonSerializer::serializeEnum,
QProtobufJsonSerializer::serializeEnumList,
QProtobufJsonSerializer::deserializeEnum and
QProtobufJsonSerializer::deserializeEnumList functions.
[ChangeLog][QtProtobuf][QProtobufBaseSerializer] Changed the following
interfaces:
- serializeEnum
- serializeEnumList
- deserializeEnum
- deserializeEnumList
All interfaces now have the mandatory const QMetaEnum &metaEnum
argument. The argument is needed to convert the enum value to its
string representation.
Pick-to: 6.7
Task-number: QTBUG-113731
Change-Id: I72061fd303c217883cbbe053a7fdd68d089f4bf8
Reviewed-by: Tatiana Borisova <tatiana.borisova@qt.io>
QtProtobuf serializers now hold states that makes them mutable during
the serializing process.
QProtobufSerializer now iterate over the data received from wire
implicitly, without exposing QProtobufSelfcheckIterator to public API.
QProtobufJsonSerializer uses the single QJsonDocument to construct the
full JSON data and parse the data received from wire.
[ChangeLog][QtProtobuf] The QtProtobufBaseSerializer API is changed.
Remove the QProtobufSerializer specifics.
[ChangeLog][QtProtobuf] QProtobufSelfcheckIterator is not a part of
public API anymore.
[ChangeLog][QtProtobufWellKnownTypes][Any] Any type API is changed.
'as' and 'fromMessage' functions not accept the pointer to a
serializer.
Pick-to: 6.7
Task-number: QTBUG-113731
Change-Id: Iddf25bed690e2299c017b3894fc8a3d9306a22b9
Reviewed-by: Tatiana Borisova <tatiana.borisova@qt.io>
Also, move basic API for serializers into QProtobufBaseSerializer class
- add full basic types JSON serialization/deserialization functionality
- add list serialization/deserialization functionality
[ChangeLog][Protobuf] Edit existing QProtobufSerializer API,
also move serialization/deserialization methods from QProtobufSerializer
class into QProtobufBaseSerializer interface, since all
of them shall be re-usable for new Json-serializer.
Autotests cover: basic types and list serialization;
TBD: maps, enums, Any type, OneOf, enumLists
will be completed in subsequent tasks
Pick-to: 6.7
Task-number: QTBUG-113731
Change-Id: I7556308062e8e576d32a371f3cb1d4f694824498
Reviewed-by: 🌴 Alexey Edelev 🌴 <alexey.edelev@qt.io>
Add the QProtobufSerializer::shouldPreserveUnknownFields function.
The function controls whether the unknown fields received from the wire
should be stored in the resulting message or should be omitted.
Change-Id: I898e67ff4b162bd0e56970a704f7359adeb653e1
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
This fixes the broken order of non-packed repeated fields and
is prerequisite for the public QProtobufMessage::unknownFields API.
Fixes: QTBUG-119244
Task-number: QTBUG-118230
Pick-to: 6.6
Change-Id: I65ca899c0bb6b52cd0c7f533d7d4ccb4c3828df1
Reviewed-by: Dennis Oberst <dennis.oberst@qt.io>
Reviewed-by: Tatiana Borisova <tatiana.borisova@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Dennis Oberst
Alexey Edelev
Mårten Nordheim
Tim Blechmann
Kai Köhne
Tatiana Borisova