aboutsummaryrefslogtreecommitdiff
path: root/proto
AgeCommit message (Collapse)Author
2023-12-04Merge remote-tracking branch 'goog/androidx-platform-dev'Terry Wang
* goog/androidx-platform-dev: Update Icing from upstream. Update Icing from upstream. Update Icing from upstream. Descriptions: ======================================================================== Add @UsedByNative annotations to native* methods in IcingSearchEngineImpl. ======================================================================== [Icing][version 3] Bump kVersion to 3 ======================================================================== Make lite index magic dependent on `IcingSearchEngineOptions::build_property_existence_metadata_hits` ======================================================================== Add a flag in IcingSearchEngineOptions to control whether to build property existence metadata hits ======================================================================== Support `hasProperty(property_path)` in the advanced query language ======================================================================== Add PropertyExistenceIndexingHandler to index property existence metadata hit ======================================================================== [JoinIndex Improvement][11/x] Add IcingSearchEngine initialization unit test for switching join index ======================================================================== [JoinIndex Improvement][10/x] Change/Add IcingSearchEngine unit tests ======================================================================== [JoinIndex Improvement][9/x] Integrate QualifiedIdJoinIndexImplV2 with IcingSearchEngine ======================================================================== [JoinIndex Improvement][8/x] Integrate QualifiedIdJoinIndexImplV2 with JoinProcessor ======================================================================== [JoinIndex Improvement][8/x] Integrate QualifiedIdJoinIndexImplV2 with QualifiedIdJoinIndexingHandler ======================================================================== [JoinIndex Improvement][7/x] Create QualifiedIdJoinIndex interface ======================================================================== [JoinIndex Improvement][6.1/x] Unit test (Optimize) ======================================================================== [JoinIndex Improvement][6.0/x] Unit test (General, Put, GetIterator) ======================================================================== [JoinIndex Improvement][5.3/x] Implement Optimize ======================================================================== Remove accents from Greek letters in normalizer ======================================================================== Make arm emulator tests build-only. ======================================================================== [JoinIndex Improvement][5.2/x] Implement GetIterator ======================================================================== [JoinIndex Improvement][5.1/x] Implement Put ======================================================================== [JoinIndex Improvement][5.0/x] Branch QualifiedIdJoinIndex to QualifiedIdJoinIndexImplV2 ======================================================================== [JoinIndex Improvement][4/x] Implement PostingListJoinDataAccessor ======================================================================== [JoinIndex Improvement][3/x] Implement PostingListJoinDataSerializer and DocumentIdToJoinInfo data type ======================================================================== [JoinIndex Improvement][2/x] Create NamespaceFingerprintIdentifier ======================================================================== [JoinIndex Improvement][1/x] Implement namespace_id_old_to_new in Compaction ======================================================================== Update test to also handle ICU 74 segmentation rules. ======================================================================== [Icing][Expand QueryStats][3/x] Add new fields into QueryStats (1) ======================================================================== [Icing][Expand QueryStats][2/x] Refactor QueryStatsProto ======================================================================== [Icing][Expand QueryStats][1/x] Publish DocHitInfoIterator CallStats ======================================================================== Add additional property filter tests ======================================================================== Deprecate hit_intersect_section_ids_mask in DocHitInfoIterator ======================================================================== Change default requires_full_emulation to False for portable_cc_test (third_party/icing/testing) ======================================================================== Cleanup Set requires_full_emulation to True for selective tests ======================================================================== Fix monkey test failures ======================================================================== Complete monkey test logic to change schema during monkey test runtime ======================================================================== Refactor monkey test to prepare for schema update ======================================================================== Fix the schema bug found by monkey test with seed 2551429844 ======================================================================== Move set query stats to the very top of InternalSearch() ======================================================================== Apply section restriction only on leaf nodes ======================================================================== [6/n] Fix callsites in Icing that forgot to check libtextclassifier3::Status (Advanced query parser) ======================================================================== [5/n] Fix callsites in Icing that forgot to check libtextclassifier3::Status (PersistentHashMap) ======================================================================== [4/n] Fix callsites in Icing that forgot to check libtextclassifier3::Status (PostingListIntegerIndexSerializer) ======================================================================== [3/n] Fix callsites in Icing that forgot to check libtextclassifier3::Status (PostingListHitSerializer) ======================================================================== [2/n] Fix callsites in Icing that forgot to check libtextclassifier3::Status (Posting list storage) ======================================================================== [1/n] Fix callsites in Icing that forgot to check libtextclassifier3::Status (Non-functional changes) ======================================================================== Decouple section restriction data from iterators ======================================================================== Fix the crash when a schema type gets more indexable properties than allowed ======================================================================== Add a checker to verify the property data type matches the schema. ======================================================================== Change global std::string in i18n-utils to constexpr std::string_view. ======================================================================== Adjust LiteIndex sort at indexing check conditions. ======================================================================== Fix the build error in aosp/2856025 ======================================================================== Bug: 305098009 Bug: 307508735 Bug: 291130542 Bug: 275121148 Bug: 303239901 Bug: 301116242 Bug: 299321977 Bug: 300135897 Bug: 297549761 Bug: 309826655 Bug: 296349369 Bug: 302192690 Bug: 302609704 Bug: 301566713 Bug: 296938196 Change-Id: I957abd9da83e30106dac188677415bd68f64185b
2023-11-30Update Icing from upstream.Jiayu Hu
Descriptions: ======================================================================== [Icing][version 3] Bump kVersion to 3 ======================================================================== Make lite index magic dependent on `IcingSearchEngineOptions::build_property_existence_metadata_hits` ======================================================================== Add a flag in IcingSearchEngineOptions to control whether to build property existence metadata hits ======================================================================== Support `hasProperty(property_path)` in the advanced query language ======================================================================== Add PropertyExistenceIndexingHandler to index property existence metadata hit ======================================================================== [JoinIndex Improvement][11/x] Add IcingSearchEngine initialization unit test for switching join index ======================================================================== [JoinIndex Improvement][10/x] Change/Add IcingSearchEngine unit tests ======================================================================== [JoinIndex Improvement][9/x] Integrate QualifiedIdJoinIndexImplV2 with IcingSearchEngine ======================================================================== [JoinIndex Improvement][8/x] Integrate QualifiedIdJoinIndexImplV2 with JoinProcessor ======================================================================== [JoinIndex Improvement][8/x] Integrate QualifiedIdJoinIndexImplV2 with QualifiedIdJoinIndexingHandler ======================================================================== [JoinIndex Improvement][7/x] Create QualifiedIdJoinIndex interface ======================================================================== [JoinIndex Improvement][6.1/x] Unit test (Optimize) ======================================================================== [JoinIndex Improvement][6.0/x] Unit test (General, Put, GetIterator) ======================================================================== [JoinIndex Improvement][5.3/x] Implement Optimize ======================================================================== Remove accents from Greek letters in normalizer ======================================================================== Make arm emulator tests build-only. ======================================================================== [JoinIndex Improvement][5.2/x] Implement GetIterator ======================================================================== [JoinIndex Improvement][5.1/x] Implement Put ======================================================================== [JoinIndex Improvement][5.0/x] Branch QualifiedIdJoinIndex to QualifiedIdJoinIndexImplV2 ======================================================================== [JoinIndex Improvement][4/x] Implement PostingListJoinDataAccessor ======================================================================== [JoinIndex Improvement][3/x] Implement PostingListJoinDataSerializer and DocumentIdToJoinInfo data type ======================================================================== [JoinIndex Improvement][2/x] Create NamespaceFingerprintIdentifier ======================================================================== [JoinIndex Improvement][1/x] Implement namespace_id_old_to_new in Compaction ======================================================================== Update test to also handle ICU 74 segmentation rules. ======================================================================== [Icing][Expand QueryStats][3/x] Add new fields into QueryStats (1) ======================================================================== [Icing][Expand QueryStats][2/x] Refactor QueryStatsProto ======================================================================== [Icing][Expand QueryStats][1/x] Publish DocHitInfoIterator CallStats ======================================================================== Add additional property filter tests ======================================================================== Deprecate hit_intersect_section_ids_mask in DocHitInfoIterator ======================================================================== Change default requires_full_emulation to False for portable_cc_test (third_party/icing/testing) ======================================================================== Cleanup Set requires_full_emulation to True for selective tests ======================================================================== Fix monkey test failures ======================================================================== Complete monkey test logic to change schema during monkey test runtime ======================================================================== Refactor monkey test to prepare for schema update ======================================================================== Fix the schema bug found by monkey test with seed 2551429844 ======================================================================== Move set query stats to the very top of InternalSearch() ======================================================================== Apply section restriction only on leaf nodes ======================================================================== [6/n] Fix callsites in Icing that forgot to check libtextclassifier3::Status (Advanced query parser) ======================================================================== [5/n] Fix callsites in Icing that forgot to check libtextclassifier3::Status (PersistentHashMap) ======================================================================== [4/n] Fix callsites in Icing that forgot to check libtextclassifier3::Status (PostingListIntegerIndexSerializer) ======================================================================== [3/n] Fix callsites in Icing that forgot to check libtextclassifier3::Status (PostingListHitSerializer) ======================================================================== [2/n] Fix callsites in Icing that forgot to check libtextclassifier3::Status (Posting list storage) ======================================================================== [1/n] Fix callsites in Icing that forgot to check libtextclassifier3::Status (Non-functional changes) ======================================================================== Decouple section restriction data from iterators ======================================================================== Fix the crash when a schema type gets more indexable properties than allowed ======================================================================== Add a checker to verify the property data type matches the schema. ======================================================================== Change global std::string in i18n-utils to constexpr std::string_view. ======================================================================== Adjust LiteIndex sort at indexing check conditions. ======================================================================== Bug: 305098009 Bug: 307508735 Bug: 291130542 Bug: 275121148 Bug: 303239901 Bug: 301116242 Bug: 299321977 Bug: 300135897 Bug: 297549761 Bug: 309826655 Bug: 296349369 Bug: 302192690 Bug: 302609704 Bug: 301566713 NO_IFTTT="False Alarm: The path is only valid in G3. kVersion is changed to 3, and schema is compatible with version 1." Change-Id: I8c4c3cd9b93e5240bd774f0a3d6d812f7a9ec198
2023-09-05Merge remote-tracking branch 'goog/androidx-platform-dev' into ↵Terry Wang
udc-mainline-prod am: f0b8d56759 Original change: https://googleplex-android-review.googlesource.com/c/platform/external/icing/+/24645416 Change-Id: Ib17ddc0a006fb8a5fccee8cbd48deff203e4766b Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-09-03Merge remote-tracking branch 'goog/androidx-platform-dev' into udc-mainline-prodTerry Wang
* goog/androidx-platform-dev: Update Icing from upstream. Descriptions: ======================================================================== [fixit] Remove incorrect reserve usage for STL data structure ======================================================================== Move LiteIndex::SortHits() from the query path to the indexing path ======================================================================== Fix term frequency bug ======================================================================== Delete dead JNI functions. ======================================================================== Switch Icing JNI implementation to use RegisterNatives ======================================================================== Avoid unnecessary GetObjectClass and GetFieldID calls. ======================================================================== Add join test for SetSchemaNewIndexedDocumentPropertyTriggersIndexRestorationAndReturnsOk ======================================================================== Support different sets of properties to search for different schema types. ======================================================================== Do not cache the section mask for a schema type if no property filters apply while processing the type property filters. ======================================================================== Bug: 286418010 Bug: 297549761 Bug: 248295995 Bug: 296938196 Bug: 288310393 Bug: 291019114 Test: presubmit Change-Id: Iab66bcc3f92641122fae2ac36132f61d8e4b4f57
2023-08-30Update Icing from upstream.Grace Zhao
Descriptions: ======================================================================== [fixit] Remove incorrect reserve usage for STL data structure ======================================================================== Move LiteIndex::SortHits() from the query path to the indexing path ======================================================================== Bug: 286418010 Bug: 297549761 Bug: 248295995 Change-Id: I46168178e0227451b0f7dbf394bb6cc5db94e1e8
2023-08-07Update Icing from upstream.Yogesh Singh
Descriptions: ======================================================================== Add join test for SetSchemaNewIndexedDocumentPropertyTriggersIndexRestorationAndReturnsOk ======================================================================== Support different sets of properties to search for different schema types. ======================================================================== Do not cache the section mask for a schema type if no property filters apply while processing the type property filters. ======================================================================== Bug: 288310393 Bug: 291019114 Change-Id: Icb867b40a603c0d63e2d96a0e2ec641e424aad7e
2023-07-11Merge remote-tracking branch 'goog/androidx-platform-dev' into ↵Alexander Dorokhine
udc-mainline-prod am: 43d2dc4133 Original change: https://googleplex-android-review.googlesource.com/c/platform/external/icing/+/24027959 Change-Id: I790a7cde4e858162cd3de8ad0a14edccfa8abcae Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-07-11Merge remote-tracking branch 'goog/androidx-platform-dev' into ↵Alexander Dorokhine
udc-mainline-prod am: e7339eb9e9 Original change: https://googleplex-android-review.googlesource.com/c/platform/external/icing/+/23988624 Change-Id: Ib7865c3954846d99fd20cfd4e28c4f44527f0d0b Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-07-11Merge remote-tracking branch 'goog/androidx-platform-dev' into udc-mainline-prodAlexander Dorokhine
* goog/androidx-platform-dev: Update Icing from upstream. Update Icing from upstream. Change descriptions: ======================================================================== Fix MainIndex::TransferAndAddHits by adding a safety check ======================================================================== [Indexable_nested_properties_list 4.5/5] Modify sectionId assignment so that all property paths in the indexable_nested_properties_list consume sectionIds. ======================================================================== [Icing] Extend document store uri mapper ======================================================================== Implement dirty flag in PersistentStorage ======================================================================== [Icing][version 2][1/x] Implement ShouldRebuildDerivedFiles ======================================================================== [Icing][version 2][2/x] Bump kVersion to 2 ======================================================================== [NumericSearch][optimization] Change numeric index bucket splitting threshold ======================================================================== [NumericSearch][optimization] Flag guard integer index bucket splitting threshold ======================================================================== Bug: 288934779 Bug: 289150947 Bug: 193919210 Bug: 280094535 Bug: 288969109 Bug: 259743562 NO_IFTTT="Path is only valid in G3." Test: Presubmit Change-Id: I61b88153b6881d166cf61f3a3369dd6a8f8189ee Change-Id: I8ffd46a09d0ff1270e376e578b160747fe6c1bb0
2023-07-07Merge remote-tracking branch 'goog/androidx-platform-dev' into udc-mainline-prodAlexander Dorokhine
Descriptions: ======================================================================== [Icing] Extend document store uri mapper ======================================================================== Implement dirty flag in PersistentStorage ======================================================================== [Icing][version 2][1/x] Implement ShouldRebuildDerivedFiles ======================================================================== [Icing][version 2][2/x] Bump kVersion to 2 ======================================================================== [NumericSearch][optimization] Change numeric index bucket splitting threshold ======================================================================== [NumericSearch][optimization] Flag guard integer index bucket splitting threshold ======================================================================== Fix MainIndex::TransferAndAddHits by adding a safety check ======================================================================== [Indexable_nested_properties_list 4.5/5] Modify sectionId assignment so that all property paths in the indexable_nested_properties_list consume sectionIds. ======================================================================== Bug: 288934779 Bug: 289150947 Bug: 193919210 Bug: 280094535 Bug: 288969109 Bug: 259743562 Test: Presubmit Change-Id: I69eeff2a9a46b8c87a15a5558f3165b94e0229c0
2023-07-06Update Icing from upstream.Yen-Chieh Sung
Descriptions: ======================================================================== [Icing] Extend document store uri mapper ======================================================================== Implement dirty flag in PersistentStorage ======================================================================== [Icing][version 2][1/x] Implement ShouldRebuildDerivedFiles ======================================================================== [Icing][version 2][2/x] Bump kVersion to 2 ======================================================================== [NumericSearch][optimization] Change numeric index bucket splitting threshold ======================================================================== [NumericSearch][optimization] Flag guard integer index bucket splitting threshold ======================================================================== Bug: 193919210 Bug: 280094535 Bug: 288969109 Bug: 259743562 NO_IFTTT="Path is only valid in G3." Change-Id: Ie1e6f5969549f6c945ea32d03c724850ff5600a9
2023-06-28Update Icing from upstream.Grace Zhao
Descriptions: ======================================================================== [Join][cleanup][ez] Rename QualifiedIdTypeJoinableIndex as QualifiedIdJoinIndex ======================================================================== Better evaluation for persistent hash map size in join index ======================================================================== Add num_to_score into ResultSpecProto ======================================================================== [Indexable_nested_properties_list 1/5] Add indexable_nested_properties to DocumentIndexingConfig and check it during schema validation. ======================================================================== [Indexable_nested_properties_list 2/5] Add check for indexable_neseted_properties when computing SchemaDelta ======================================================================== [Indexable_nested_properties_list 3/5] Add indexable_nested_properties to schema-property-iterator ======================================================================== [Indexable_nested_properties_list 4/5] Integrate circular schema definitions with schema-property-iterator ======================================================================== Bug: 272145329 Bug: 284232550 Bug: 285636503 Bug: 265304217 Bug: 289150947 Change-Id: I0bc3282400e6c69f8551dbddf47585864bfdec06
2023-05-20Merge remote-tracking branch 'goog/androidx-platform-dev' into udc-dev am: ↵Terry Wang
5d49250d7a am: b412fe2c0a Original change: https://googleplex-android-review.googlesource.com/c/platform/external/icing/+/23356197 Change-Id: I58beaf71cf34c22d5f0ee994d2661d3b03849fb9 Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-05-19Merge remote-tracking branch 'goog/androidx-platform-dev' into udc-dev am: ↵Terry Wang
fcaadb9848 am: 04e9c60ecb Original change: https://googleplex-android-review.googlesource.com/c/platform/external/icing/+/23330232 Change-Id: Ia4675f972746cad8701d481e92955d1745918ad5 Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-05-19Merge remote-tracking branch 'goog/androidx-platform-dev' into udc-devTerry Wang
* goog/androidx-platform-dev: Update Icing from upstream. Descriptions: ======================================================================== Add flag to guard PersistentHashMapKeyMapper and DynamicTrieKeyMapper ======================================================================== Add flag to guard FileBackedVector premapping ======================================================================== Check tokenizer type when assigning section id ======================================================================== Bug: 193919210 Change-Id: Ifab41e03ab7dd38c803337a84259817e88f3764d
2023-05-19Update Icing from upstream.Tim Barron
Descriptions: ======================================================================== Add flag to guard PersistentHashMapKeyMapper and DynamicTrieKeyMapper ======================================================================== Add flag to guard FileBackedVector premapping ======================================================================== Check tokenizer type when assigning section id ======================================================================== Bug:193919210 Change-Id: I682e22d4ba647220b7222fcc63c41bd0e01bbeb9
2023-05-18Merge remote-tracking branch 'goog/androidx-platform-dev' into udc-devTerry Wang
* goog/androidx-platform-dev: AndroidXPlugin in icing Update Icing from upstream. Descriptions: ======================================================================== Minor fix for export to framework. ======================================================================== [Icing][version][3/x][ez] Call DataSync for version file ======================================================================== Add `use_read_only_search` flag for flag-guarding IcingSearchEngine::Search locking change in cl/506671441. ======================================================================== Bug: 146008613 Change-Id: Id432956b994c6ec1030959efa74e3cda55f26c99
2023-05-15Update Icing from upstream.Terry Wang
Descriptions: ======================================================================== Minor fix for export to framework. ======================================================================== [Icing][version][3/x][ez] Call DataSync for version file ======================================================================== Add `use_read_only_search` flag for flag-guarding IcingSearchEngine::Search locking change in cl/506671441. ======================================================================== Bug: 146008613 Test: presubmit Change-Id: Id863275cf5cc1ad7bcc69649527cc17981cc6cd2
2023-05-12Merge remote-tracking branch 'goog/androidx-platform-dev' into udc-dev am: ↵Terry Wang
50b5988361 am: c82a8ba55b Original change: https://googleplex-android-review.googlesource.com/c/platform/external/icing/+/23183335 Change-Id: I1a3f2b5a5dbbda36be94c217bde41d367aa2ca56 Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-05-12Merge remote-tracking branch 'goog/androidx-platform-dev' into udc-devTerry Wang
* goog/androidx-platform-dev: Update Icing from upstream. Update Icing from upstream. Descriptions: ======================================================================== Handle version changes in the schema store. ======================================================================== Modify the definition of propertyDefined: ======================================================================== Remove default args in SchemaStore::SetSchema and fix calls ======================================================================== Add allow_circular_schema_definitions flag ======================================================================== Onboard version detection to Icing ======================================================================== Add version util to help read/write version info ======================================================================== Add support for the overlay schema. ======================================================================== Allow cycles in schema-property-iterator ======================================================================== Add joinable properties into schema definition cycle restrictions. ======================================================================== Loosen circular references restriction for Schema Definitions. ======================================================================== Implement BackupSchemaProducer to generate a backup schema ======================================================================== Minor fix: remove a redundant log ======================================================================== Allow schema types to inherit from more than one parent ======================================================================== allow nested document properties to accept documents of subtype ======================================================================== Support polymorphism for Icing projection in Search and Get API ======================================================================== Add max_joined_child_per_parent into ResultSpec and change behavior ======================================================================== Support Icing schema type polymorphism for the search filter API ======================================================================== Verify that every child type's property set has included all compatible properties from parent types ======================================================================== Add individual type index latency ======================================================================== Build the iterator node for the propertyDefined() custom function ======================================================================== Advance all hits with same doc id from and merge sections once for the same bucket iter ======================================================================== Introduce DocHitInfoIteratorPropertyInSchema for property existence check ======================================================================== Add SchemaUtil::BuildTransitiveInheritanceGraph to build an inheritance map from schema ======================================================================== Introduce a lookup method for a property defined in a schema ======================================================================== Rollback of: Allow LanguageSegmenter::Iterators to declare AccessType. ======================================================================== Adds join info to QueryStatsProto ======================================================================== Bug: 280698419 Bug: 280698125 Bug: 280698121 Bug: 280697513 Bug: 276349029 Bug: 272145329 Bug: 270102295 Bug: 269295094 Bug: 268680462 Bug: 265304217 Bug: 259744228 Bug: 259743562 Bug: 256022027 Change-Id: I93b11525c3288667f3f434a6de6ec868048ee5fe
2023-05-11Update Icing from upstream.Tim Barron
Descriptions: ======================================================================== Modify the definition of propertyDefined: ======================================================================== Remove default args in SchemaStore::SetSchema and fix calls ======================================================================== Add allow_circular_schema_definitions flag ======================================================================== Onboard version detection to Icing ======================================================================== Add version util to help read/write version info ======================================================================== Add support for the overlay schema. ======================================================================== Allow cycles in schema-property-iterator ======================================================================== Add joinable properties into schema definition cycle restrictions. ======================================================================== Loosen circular references restriction for Schema Definitions. ======================================================================== Implement BackupSchemaProducer to generate a backup schema ======================================================================== Minor fix: remove a redundant log ======================================================================== Allow schema types to inherit from more than one parent ======================================================================== allow nested document properties to accept documents of subtype ======================================================================== Support polymorphism for Icing projection in Search and Get API ======================================================================== Add max_joined_child_per_parent into ResultSpec and change behavior ======================================================================== Support Icing schema type polymorphism for the search filter API ======================================================================== Verify that every child type's property set has included all compatible properties from parent types ======================================================================== Add individual type index latency ======================================================================== Build the iterator node for the propertyDefined() custom function ======================================================================== Advance all hits with same doc id from and merge sections once for the same bucket iter ======================================================================== Introduce DocHitInfoIteratorPropertyInSchema for property existence check ======================================================================== Add SchemaUtil::BuildTransitiveInheritanceGraph to build an inheritance map from schema ======================================================================== Introduce a lookup method for a property defined in a schema ======================================================================== Rollback of: Allow LanguageSegmenter::Iterators to declare AccessType. ======================================================================== Adds join info to QueryStatsProto ======================================================================== Bug:280698419 Bug:280698125 Bug:280698121 Bug:280697513 Bug:276349029 Bug:272145329 Bug:270102295 Bug:269295094 Bug:268680462 Bug:265304217 Bug:259744228 Bug:259743562 Bug:256022027 Change-Id: I54cd1d22121c314f8c238d2d49f0809165dc0ca3
2023-05-08Make apex_available of icing-java-proto-lite explicit in Android.bp am: ↵Spandan Das
6f3c4d12f6 am: c55f84504f am: a55430e3bd Original change: https://android-review.googlesource.com/c/platform/external/icing/+/2577142 Change-Id: Ifb56a20d1248ba926ec28a01563d216ee3db390f Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-05-05Make apex_available of icing-java-proto-lite explicit in Android.bpSpandan Das
This library is available to com.android.appsearch via the baseline apexAvailable map in build/soong/apex/apex.go. This CL makes this explicit in Android.bp Test: m nothing #passes Bug: 281077552 Change-Id: I1c96b8587341092ae09a80adea7ce8ddd47026e7
2023-04-29Merge remote-tracking branch 'goog/androidx-platform-dev' into udc-devTerry Wang
* goog/androidx-platform-dev: Remove custom loading of protobuf gradle plugin Update Icing from upstream. Use the new build tools version API Update Icing from upstream. Descriptions: ======================================================================== Make int64_t the CreateUsageReport timestamp type ======================================================================== Create ResultAdjustmentInfo and refactor ResultStateV2 ======================================================================== Wrap parent/child adjustment info to std::unique_ptr ======================================================================== Apply join child snippet ======================================================================== Apply join child projection ======================================================================== Add IntegerSectionIndexingHandlerTest ======================================================================== Delete Result Retriever this class is dead code. ======================================================================== Introduce a placeholder for the custom function hasPropertyDefined(member) ======================================================================== Fix libtextclassifier3::StatusOr ======================================================================== Performance improvements to SnippetRetriever. ======================================================================== Adds join info to QueryStatsProto ======================================================================== [Join][cleanup][1/x] Add Icing benchmark for qualified id join ======================================================================== [NumericSearch][benchmark][2/x] Modify run-benchmarks.py to onboard numeric search benchmark ======================================================================== [NumericSearch][benchmark][1/x] Add numeric search icing search engine benchmark ======================================================================== Switch Iterator access type used in SnippetRetriever back to kForwardIterator. ======================================================================== Fix `DocumentStore::OptimizeInto` by passing `options_.document_store_namespace_id_fingerprint()` ======================================================================== Add file size check when initializing mmap for existing metadata file. ======================================================================== Add compression level option to Icing and propagate down to PortableFileBackedProtoLog ======================================================================== Add portable_equals_proto namespace for EqualsProto in tokenized-document_test. ======================================================================== Refactor schema to check consistency of polymorphism by dependency map ======================================================================== Support the `this.propertyWeights()` function in advanced scoring ======================================================================== Randomly set optimize_rebuild_index_threshold in monkey test ======================================================================== Change benchmark batch query latency to average ======================================================================== Performance improvements to SnippetRetriever. ======================================================================== Bug: 193244409 Bug: 256022027 Bug: 259744228 Bug: 260625837 Bug: 268680462 Bug: 269295094 Bug: 269513147 Bug: 270102295 Bug: 271015984 Bug: 272145329 Bug: 274627497 Bug: 276345448 Bug: 276349029 Change-Id: I71827fa9914ae2549e068e04165ca261346dfcaf
2023-04-19Update Icing from upstream.Jiayu Hu
Descriptions: ======================================================================== Adds join info to QueryStatsProto ======================================================================== [Join][cleanup][1/x] Add Icing benchmark for qualified id join ======================================================================== [NumericSearch][benchmark][2/x] Modify run-benchmarks.py to onboard numeric search benchmark ======================================================================== [NumericSearch][benchmark][1/x] Add numeric search icing search engine benchmark ======================================================================== Switch Iterator access type used in SnippetRetriever back to kForwardIterator. ======================================================================== Fix `DocumentStore::OptimizeInto` by passing `options_.document_store_namespace_id_fingerprint()` ======================================================================== Add file size check when initializing mmap for existing metadata file. ======================================================================== Add compression level option to Icing and propagate down to PortableFileBackedProtoLog ======================================================================== Add portable_equals_proto namespace for EqualsProto in tokenized-document_test. ======================================================================== Refactor schema to check consistency of polymorphism by dependency map ======================================================================== Support the `this.propertyWeights()` function in advanced scoring ======================================================================== Randomly set optimize_rebuild_index_threshold in monkey test ======================================================================== Change benchmark batch query latency to average ======================================================================== Performance improvements to SnippetRetriever. ======================================================================== Bug: 270102295 Bug: 260625837 Bug: 269295094 Bug: 193244409 Bug: 269513147 Bug: 276345448 Bug: 259744228 Bug: 272145329 Bug: 276349029 Change-Id: Ief1cb2c21f67aac420d6b79d9b946bbecdc39a5c
2023-03-20Merge remote-tracking branch 'goog/androidx-platform-dev' into udc-devTerry Wang
Descriptions: ====================================================================== Add PropertyUtil for all property name/path related operations ====================================================================== [JoinableCache][2.0/x] Create SchemaPropertyIterator ====================================================================== [JoinableCache][2.1/x] Handle nested indexable flag ====================================================================== [JoinableCache][2.2/x] Add schema cycle dependency detection for SchemaPropertyIterator ====================================================================== [JoinableCache][3.0/x] Refactor SectionManager ====================================================================== [JoinableCache][3.1/x] Add unit tests for SectionManager::Builder and SchemaTypeManager ====================================================================== [NumericSearch][Storage][12/x] Implement Edit and GetIterator for IntegerIndex ====================================================================== [NumericSearch][Storage][13.0/x] Rename numeric-index_test as integer-index_test ====================================================================== [NumericSearch][Storage][13.1/x] Add IntegerIndexTest ====================================================================== Support the "len", "sum" and "avg" functions in advanced scoring. ====================================================================== Support the "this.childrenScores()" function to allow expressing children scores of joins in advanced scoring. ====================================================================== Create an integration test for Join with advanced scoring ====================================================================== Rename the word "children" to "args" for function related ScoreExpression ====================================================================== Improve IndexBlock by PRead/PWrite instead of repeating mmap/msync/unmap ====================================================================== Refactor QueryVisitor to prepare for support for function calls. ====================================================================== Add support for function calls. ====================================================================== Fix breakage in score-and-rank_benchmark. ====================================================================== [NumericSearch][Storage][adhoc][ez] Fix comment for IntegerIndex ====================================================================== [NumericSearch][Storage][14/x] Create first IntegerIndexStorage benchmark ====================================================================== Rename Icing schema related terminology to prepare for polymorphism support ====================================================================== [JoinableCache][4.0/x] Move common methods from SectionManager to PropertyUtil ====================================================================== [JoinableCache][4.1/x] Retire GetSectionContent ====================================================================== [JoinableCache][4.2/x] Polish SectionManagerTest ====================================================================== Modify QueryVisitor to do: ====================================================================== [NumericSearch][Storage][15/x] Implement TransferIndex for IntegerIndexStorage ====================================================================== [NumericSearch][Storage][16/x] Implement Optimize and last added document id for IntegerIndex ====================================================================== [NumericSearch][rollout][1/x] Include indexable int64 into SchemaDelta and backward compatibility ====================================================================== Add backwards compatibility test for Icing schema storage migration. ====================================================================== Implement trim the right-most node from the doc-hit-info-iterator. ====================================================================== Add TrimmedNode structure into doc-hit-info-iterator. ====================================================================== [JoinableCache][5/x] Implement JoinableProperty and JoinablePropertyManager ====================================================================== [JoinableCache][6/x] Add JoinablePropertyManager into SchemaTypeManager ====================================================================== [JoinableCache][7/x] Implement ExtractJoinableProperties ====================================================================== [JoinableCache][8/x] Create class QualifiedIdTypeJoinableCache ====================================================================== [JoinableCache][9/x] Implement factory method for QualifiedIdTypeJoinableCache ====================================================================== [JoinableCache][10/x] Implement Get and Put for QualifiedIdTypeJoinableCache ====================================================================== [JoinableCache][11/x] Add unit tests for QualifiedIdTypeJoinableCache ====================================================================== Modify DocHitInfoIteratorSectionRestrict to allow multi-property restricts ====================================================================== Fix the definition of LiteIndex::WantsMerge. ====================================================================== [NumericSearch][rollout][2.0/x] Rollout persistent IntegerIndex ====================================================================== [NumericSearch][rollout][2.1/x] Add more tests for integer index restoration and optimization ====================================================================== [JoinableCache][adhoc][ez] Remove qualified id type joinable cache size info from document storage info ====================================================================== Integrate trim right node into suggestion processor. ====================================================================== Modify the lexer to consider internal minus characters to be a part of TEXT segments (as hyphens). ====================================================================== Modify the query language to support an individual prefix operator '*'. ====================================================================== Fix the fingerprint issue for document key mapper in document store ====================================================================== Resolve issues around the handling of negative numbers. ====================================================================== Fix doc join info compiler warning/error ====================================================================== Codifies how colons are tokenized by the lexer ====================================================================== Modify the lexer to consider internal minus characters to be a part of TEXT segments (as hyphens). ====================================================================== Modify the query language to support an individual prefix operator '*'. ====================================================================== Fix the fingerprint issue for document key mapper in document store ====================================================================== Resolve issues around the handling of negative numbers. ====================================================================== Fix doc join info compiler warning/error ====================================================================== Codifies how colons are tokenized by the lexer ======================================================================== Cache an instance of UBreakIterator to reduce unnecessary creations. ======================================================================== Cap number of individual IntegerIndexStorages that IntegerIndex creates. ======================================================================== Change error in trimRightMostNode from Unimplemented to InvalidArgument. ======================================================================== Add detection for new language features of List Filters Query Language. ======================================================================== Add option to control threshold to rebuild index during optimize by flag ======================================================================== Add option to control use of namespace id to build urimapper by flag. ======================================================================== Enforce schema validation for joinable config. ======================================================================== Adopt bucket splitting for IntegerIndexStorage. ======================================================================== Implement bucket splitting function. ======================================================================== Add Icing initialization unit tests for QualifiedIdTypeJoinableIndex. ======================================================================== Add Icing schema change unit tests for QualifiedIdTypeJoinableIndex. ======================================================================== Add Icing optimization unit tests for QualifiedIdTypeJoinableIndex. ======================================================================== Integrate QualifiedIdTypeJoinableIndex into IcingSearchEngine. ======================================================================== Implement QualifiedIdJoinablePropertyIndexingHandler. ======================================================================== Change QualifiedIdTypeJoinableIndex to store raw qualified id string. ======================================================================== Pass info about unnormalized query terms through lexer/parser/visitor. ======================================================================== Integrate Advanced Query w/ Suggest, make ADVANCED_QUERY default parser. ======================================================================== Set overall index latency ======================================================================== Change GetUsageScores return type to optional. ======================================================================== Upstream fix to thread-safety annotation for cached_break_iterator_ ======================================================================== Bug: 208654892 Bug: 259743562 Bug: 272145329 Bug: 227356108 Bug: 208654892 Bug: 228240987 Bug: 249829533 Bug: 256081830 Bug: 259744228 Bug: 261474063 Bug: 263890397 Bug: 266103594 Bug: 268680462 Bug: 268738297 Bug: 269295094 Test:presubmit Change-Id: Ib31ee817e3faa1107483c7ee0ab6eced371bfed7
2023-03-14Update Icing from upstream.Tim Barron
Descriptions: ======================================================================== Cache an instance of UBreakIterator to reduce unnecessary creations. ======================================================================== Cap number of individual IntegerIndexStorages that IntegerIndex creates. ======================================================================== Change error in trimRightMostNode from Unimplemented to InvalidArgument. ======================================================================== Add detection for new language features of List Filters Query Language. ======================================================================== Add option to control threshold to rebuild index during optimize by flag ======================================================================== Add option to control use of namespace id to build urimapper by flag. ======================================================================== Enforce schema validation for joinable config. ======================================================================== Adopt bucket splitting for IntegerIndexStorage. ======================================================================== Implement bucket splitting function. ======================================================================== Add Icing initialization unit tests for QualifiedIdTypeJoinableIndex. ======================================================================== Add Icing schema change unit tests for QualifiedIdTypeJoinableIndex. ======================================================================== Add Icing optimization unit tests for QualifiedIdTypeJoinableIndex. ======================================================================== Integrate QualifiedIdTypeJoinableIndex into IcingSearchEngine. ======================================================================== Implement QualifiedIdJoinablePropertyIndexingHandler. ======================================================================== Change QualifiedIdTypeJoinableIndex to store raw qualified id string. ======================================================================== Pass info about unnormalized query terms through lexer/parser/visitor. ======================================================================== Bug: 208654892 Bug: 263890397 Bug: 259743562 Bug: 272145329 Bug: 227356108 Change-Id: I438a390ddda5673cf2b5781af502f2b7cfeaee74
2023-03-09Update Icing from upstream.Tim Barron
Descriptions: ====================================================================== Integrate Advanced Query w/ Suggest, make ADVANCED_QUERY default parser. ====================================================================== BUG: 208654892 Change-Id: I53889482e844648ec65565f9a7f7c8faa89baa7c
2023-03-01Update Icing from upstream.Terry Wang
Descriptions: ====================================================================== Add PropertyUtil for all property name/path related operations ====================================================================== [JoinableCache][2.0/x] Create SchemaPropertyIterator ====================================================================== [JoinableCache][2.1/x] Handle nested indexable flag ====================================================================== [JoinableCache][2.2/x] Add schema cycle dependency detection for SchemaPropertyIterator ====================================================================== [JoinableCache][3.0/x] Refactor SectionManager ====================================================================== [JoinableCache][3.1/x] Add unit tests for SectionManager::Builder and SchemaTypeManager ====================================================================== [NumericSearch][Storage][12/x] Implement Edit and GetIterator for IntegerIndex ====================================================================== [NumericSearch][Storage][13.0/x] Rename numeric-index_test as integer-index_test ====================================================================== [NumericSearch][Storage][13.1/x] Add IntegerIndexTest ====================================================================== Support the "len", "sum" and "avg" functions in advanced scoring. ====================================================================== Support the "this.childrenScores()" function to allow expressing children scores of joins in advanced scoring. ====================================================================== Create an integration test for Join with advanced scoring ====================================================================== Rename the word "children" to "args" for function related ScoreExpression ====================================================================== Improve IndexBlock by PRead/PWrite instead of repeating mmap/msync/unmap ====================================================================== Refactor QueryVisitor to prepare for support for function calls. ====================================================================== Add support for function calls. ====================================================================== Fix breakage in score-and-rank_benchmark. ====================================================================== [NumericSearch][Storage][adhoc][ez] Fix comment for IntegerIndex ====================================================================== [NumericSearch][Storage][14/x] Create first IntegerIndexStorage benchmark ====================================================================== Rename Icing schema related terminology to prepare for polymorphism support ====================================================================== [JoinableCache][4.0/x] Move common methods from SectionManager to PropertyUtil ====================================================================== [JoinableCache][4.1/x] Retire GetSectionContent ====================================================================== [JoinableCache][4.2/x] Polish SectionManagerTest ====================================================================== Modify QueryVisitor to do: ====================================================================== [NumericSearch][Storage][15/x] Implement TransferIndex for IntegerIndexStorage ====================================================================== [NumericSearch][Storage][16/x] Implement Optimize and last added document id for IntegerIndex ====================================================================== [NumericSearch][rollout][1/x] Include indexable int64 into SchemaDelta and backward compatibility ====================================================================== Add backwards compatibility test for Icing schema storage migration. ====================================================================== Implement trim the right-most node from the doc-hit-info-iterator. ====================================================================== Add TrimmedNode structure into doc-hit-info-iterator. ====================================================================== [JoinableCache][5/x] Implement JoinableProperty and JoinablePropertyManager ====================================================================== [JoinableCache][6/x] Add JoinablePropertyManager into SchemaTypeManager ====================================================================== [JoinableCache][7/x] Implement ExtractJoinableProperties ====================================================================== [JoinableCache][8/x] Create class QualifiedIdTypeJoinableCache ====================================================================== [JoinableCache][9/x] Implement factory method for QualifiedIdTypeJoinableCache ====================================================================== [JoinableCache][10/x] Implement Get and Put for QualifiedIdTypeJoinableCache ====================================================================== [JoinableCache][11/x] Add unit tests for QualifiedIdTypeJoinableCache ====================================================================== Modify DocHitInfoIteratorSectionRestrict to allow multi-property restricts ====================================================================== Fix the definition of LiteIndex::WantsMerge. ====================================================================== [NumericSearch][rollout][2.0/x] Rollout persistent IntegerIndex ====================================================================== [NumericSearch][rollout][2.1/x] Add more tests for integer index restoration and optimization ====================================================================== [JoinableCache][adhoc][ez] Remove qualified id type joinable cache size info from document storage info ====================================================================== Integrate trim right node into suggestion processor. Bug: 208654892 Bug: 228240987 Bug: 249829533 Bug: 256081830 Bug: 259744228 Bug: 261474063 Bug: 263890397 Bug: 266103594 Bug: 268738297 Bug: 269295094 Change-Id: I5f1b3f3ed0b5d6933dc8c2ab3279904f7706b23e
2023-02-10Merge remote-tracking branch 'goog/androidx-platform-dev'Alex Saveliev
* goog/androidx-platform-dev: Update icing from upstream Adding kotlinBom to icing Update Icing from upstream. ====================================================================== Adds a proto change for the delete propagation option ====================================================================== [ez] Change version to magic for PersistentHashMap ====================================================================== [iOS][testing][nitro] Disabling ICU language segmenter_test. ====================================================================== 1. Add support for segmentation in the QueryVisitor. ====================================================================== Support the new double list type in ScoreExpression ====================================================================== Pass JoinChildrenFetcher from IcingSearchEngine all the way down to ScoringVisitor ====================================================================== Refactor the logic of Icing Joins so that nested search and scoring will be performed before the parent ====================================================================== Add lite-index thread-safety tests. ====================================================================== Split IcingSearchEngineTest into separate tests to cover specific apis: ====================================================================== Fix index tests TearDown method. ====================================================================== Improve query concurrency by providing a finer-grained lock around the LiteIndex. ====================================================================== Fix Icing normalization bug ====================================================================== [ez] Fix integer overflow error for IntegerIndexStorage ====================================================================== [NumericSearch][Storage][11/x] Implement Reset and destructor for IntegerIndex ====================================================================== [NumericSearch][Storage][10/x] Add class IntegerIndex ====================================================================== Refactor NumericIndex based on PersistentStorage ====================================================================== Refactor IntegerIndexStorage based on PersistentStorage ====================================================================== Add "working_path" into PersistentStorage ====================================================================== Refactor PersistentHashMap based on PersistentStorage ====================================================================== Create virtual class PersistentStorage for refactoring ====================================================================== Avoids returning reference to local temporary object. ====================================================================== LSC: Clean up references to the legacy protobuf compat library ====================================================================== Fix time complexity regression for snippet retriever ====================================================================== Add support for QueryTerms SectionRestrict Map in Advanced Query. ====================================================================== [NumericSearch][Storage][9/x] Create unit tests for AddKeys and GetIterator ====================================================================== [NumericSearch][Storage][8/x] Add options ====================================================================== Fix the stack overflow issue for icing advanced scoring/query parsers/visitors by limiting the number of tokens allowed ====================================================================== Improves the has joinspec check ====================================================================== Fix the bug of DocHitInfoIteratorTermMain found by monkey test ====================================================================== Rename tests defined in DestructibleDirectoryTest to include "directory" instead of "file". ====================================================================== Support projection in icing monkey test ====================================================================== [NumericSearch][Storage][7/x] Implement iterator for query ====================================================================== [NumericSearch][Storage][6/x] Implement AddKeys ====================================================================== Add support for relevance scoring by enabling term_frequency retrieval in DocHitInfoIterators and by populating the QueryTermIteratorsMap. ====================================================================== [JoinableCache][1/x] Compute SchemaDelta for joinable properties ====================================================================== [NumericSearch][Storage][5/x] PersistToDisk ====================================================================== [NumericSearch][Storage][4.1/x] Create unit tests for initialize ====================================================================== [NumericSearch][Storage][4.0/x] Implement Create and Initialize functions ====================================================================== Rename PostingListUsedSerializer related classes Bug: 193244409 Bug: 246984163 Bug: 249829533 Bug: 265834832 Bug: 256022027 Bug: 193919210 Bug: 266132035 Bug: 208654892 Bug: 261474063 Bug: 266103594 Bug: 146008613 Bug: 253182853 Bug: 266204868 Bug: 249829533 Bug: 266665956 Bug: 265258364 Change-Id: I97f8c64b06a32b39930a0776f2d30d63d4ce564b
2023-02-07Update icing from upstreamAlex Saveliev
====================================================================== Adds a proto change for the delete propagation option ====================================================================== [ez] Change version to magic for PersistentHashMap ====================================================================== [iOS][testing][nitro] Disabling ICU language segmenter_test. ====================================================================== 1. Add support for segmentation in the QueryVisitor. ====================================================================== Support the new double list type in ScoreExpression ====================================================================== Pass JoinChildrenFetcher from IcingSearchEngine all the way down to ScoringVisitor ====================================================================== Refactor the logic of Icing Joins so that nested search and scoring will be performed before the parent ====================================================================== Add lite-index thread-safety tests. ====================================================================== Split IcingSearchEngineTest into separate tests to cover specific apis: ====================================================================== Fix index tests TearDown method. ====================================================================== Improve query concurrency by providing a finer-grained lock around the LiteIndex. ====================================================================== Fix Icing normalization bug ====================================================================== [ez] Fix integer overflow error for IntegerIndexStorage ====================================================================== [NumericSearch][Storage][11/x] Implement Reset and destructor for IntegerIndex ====================================================================== [NumericSearch][Storage][10/x] Add class IntegerIndex ====================================================================== Refactor NumericIndex based on PersistentStorage ====================================================================== Refactor IntegerIndexStorage based on PersistentStorage ====================================================================== Add "working_path" into PersistentStorage ====================================================================== Refactor PersistentHashMap based on PersistentStorage ====================================================================== Create virtual class PersistentStorage for refactoring ====================================================================== Avoids returning reference to local temporary object. ====================================================================== LSC: Clean up references to the legacy protobuf compat library ====================================================================== Fix time complexity regression for snippet retriever Bug: 256022027 Bug: 193919210 Bug: 266132035 Bug: 208654892 Bug: 261474063 Bug: 266103594 Bug: 146008613 Bug: 253182853 Bug: 266204868 Bug: 249829533 Bug: 266665956 Bug: 265258364 Change-Id: Ib2398c5097b6a2a57900e2ad4e3737502aa13820
2023-01-23Update Icing from upstream.Terry Wang
Descriptions: ====================================================================== Fix time complexity regression for snippet retriever ====================================================================== Add support for QueryTerms SectionRestrict Map in Advanced Query. ====================================================================== [NumericSearch][Storage][9/x] Create unit tests for AddKeys and GetIterator ====================================================================== [NumericSearch][Storage][8/x] Add options ====================================================================== Fix the stack overflow issue for icing advanced scoring/query parsers/visitors by limiting the number of tokens allowed ====================================================================== Improves the has joinspec check ====================================================================== Fix the bug of DocHitInfoIteratorTermMain found by monkey test ====================================================================== Rename tests defined in DestructibleDirectoryTest to include "directory" instead of "file". ====================================================================== Support projection in icing monkey test ====================================================================== [NumericSearch][Storage][7/x] Implement iterator for query ====================================================================== [NumericSearch][Storage][6/x] Implement AddKeys ====================================================================== Add support for relevance scoring by enabling term_frequency retrieval in DocHitInfoIterators and by populating the QueryTermIteratorsMap. ====================================================================== [JoinableCache][1/x] Compute SchemaDelta for joinable properties ====================================================================== [NumericSearch][Storage][5/x] PersistToDisk ====================================================================== [NumericSearch][Storage][4.1/x] Create unit tests for initialize ====================================================================== [NumericSearch][Storage][4.0/x] Implement Create and Initialize functions ====================================================================== Rename PostingListUsedSerializer related classes Bug: 193244409 Bug: 208654892 Bug: 246984163 Bug: 249829533 Bug: 256022027 Bug: 265258364 Bug: 265834832 Change-Id: Ibc7011a793ef5ed09eace0fb05d168adf5b4faca
2023-01-21Merge remote-tracking branch 'goog/androidx-platform-dev'Terry Wang
* goog/androidx-platform-dev: Update icing from upstream Sync from upstream. Descriptions: ====================================================================== Apply aggregation score to join result and create JoinProcessorTest ====================================================================== Create AggregationScorerTest ====================================================================== Add an EqualsTermMatchInfo matcher to simplify tests that interact with this struct. ====================================================================== Refactor the QueryVisitor to return QueryResults. ====================================================================== Simplify ScoreExpression to ConstantScoreExpression if its evaluation does not depend on a document ====================================================================== Use rvalue and std::move for DocumentStore methods ====================================================================== Downgrade log messages for NOT_FOUND errors in Delete calls. ====================================================================== [NumericSearch][Storage][3.1/x] Add metadata file declaration (Crcs, Info) for IntegerIndexStorage ====================================================================== [NumericSearch][Storage][3.0/x] Declare class IntegerIndexStorage ====================================================================== Cleanup Import from //tools/build_defs/objc ====================================================================== Cleanup LSC: Remove `portable_proto_library()` targets ====================================================================== Bug: 256022027 Bug: 208654892 Bug: 261474063 Bug: 246628980 Bug: 249829533 Bug: 263373214 Change-Id: I948f63c3a282c745c4d4db1febf8a3283561ca73
2023-01-06Update icing from upstreamAlex Saveliev
Descriptions: ====================================================================== Apply aggregation score to join result and create JoinProcessorTest ====================================================================== Create AggregationScorerTest ====================================================================== Add an EqualsTermMatchInfo matcher to simplify tests that interact with this struct. ====================================================================== Refactor the QueryVisitor to return QueryResults. ====================================================================== Simplify ScoreExpression to ConstantScoreExpression if its evaluation does not depend on a document ====================================================================== Use rvalue and std::move for DocumentStore methods ====================================================================== Downgrade log messages for NOT_FOUND errors in Delete calls. ====================================================================== [NumericSearch][Storage][3.1/x] Add metadata file declaration (Crcs, Info) for IntegerIndexStorage ====================================================================== [NumericSearch][Storage][3.0/x] Declare class IntegerIndexStorage ====================================================================== Cleanup Import from //tools/build_defs/objc ====================================================================== Cleanup LSC: Remove `portable_proto_library()` targets ====================================================================== Bug: 256022027 Bug: 208654892 Bug: 261474063 Bug: 246628980 Bug: 249829533 Bug: 263373214 Change-Id: Ib5ca7efffd575254e14a14d54a45f4efd1e8d6c9
2022-12-20Merge remote-tracking branch 'goog/androidx-platform-dev'Terry Wang
Descriptions: ========================================== Create class QualifiedId ========================================== Switch JoinProcessor to use new class FullyQualifiedId ========================================== Add JoinableConfig proto ========================================== Implement document-based functions for the Advanced Scoring Language ========================================== Support the RelevanceScore function for the Advanced Scoring Language ========================================== Enable the document-based member functions for Advanced Scoring Language ========================================== Fix naming of version_array variable, which should be snake_case. ========================================== Rename fullyQualifiedId to qualifiedId ========================================== Delete ResultState, which is dead code. ========================================== Implement support in the QueryVisitor for the property restrict operator (HAS) ========================================== Implement support in the QueryVisitor for implicit and, 'AND' and 'OR'. ========================================== Implement support in the QueryVisitor for NOT. ========================================== Implement support in the QueryVisitor for single term and single verbatim term queries. ========================================== Implement CorpusId as the key for ResultGrouping. ========================================== Implement simple feature checking for implicit new features in IcingLib. ========================================== Bug: 208654892 Bug: 256022027 Bug: 258715421 Bug: 261474063 Bug: 262512396 Change-Id: I671936fee37c91b5e5d391b524ab34bf9d5c1d5a
2022-12-16Sync from upstream.Tim Barron
Descriptions: ====================================================================== Implement CorpusId as the key for ResultGrouping. ====================================================================== Implement simple feature checking for implicit new features in IcingLib. ====================================================================== Bug: 258715421 Bug: 262512396 Change-Id: I815a68bc91d2750b2210f2f3e227044479a4de7a
2022-12-14Merge remote-tracking branch 'goog/androidx-platform-dev'Armaan Danewalia
* goog/androidx-platform-dev: Sync from upstream. Descriptions: ====================================================================== Add ScoringSpec into JoinSpec. Rename joined_document to child_document. ====================================================================== Create JoinedScoredDocumentHit class and refactor ScoredDocumentHitsRanker. ====================================================================== Implement initial Join workflow ====================================================================== Implement the Lexer for Icing Advanced Query Language ====================================================================== Create struct Options for PersistentHashMap ====================================================================== Premapping FileBackedVector ====================================================================== Create class PersistentHashMapKeyMapper ====================================================================== Add integer sections into TokenizedDocument and rename string sections ====================================================================== Create NumericIndex interface and DocHitInfoIteratorNumeric ====================================================================== Implement DummyNumericIndex and unit test ====================================================================== Change PostingListAccessor::Finalize to rvalue member function ====================================================================== Define the Abstract Syntax Tree for Icing's list_filter parser. ====================================================================== Refactor query processing and score ====================================================================== Refactor IcingSearchEngine for AppSearch Dynamite Module 0p APIs ====================================================================== Implement the Lexer for Icing Advanced Scoring Language ====================================================================== Add a common interface for IcingSearchEngine and dynamite client ====================================================================== Implement a subset of the query grammar. ====================================================================== Refactor index processor ====================================================================== Add integer index into IcingSearchEngine and IndexProcessor ====================================================================== Implement the parser for Icing Advanced Scoring Language ====================================================================== Implement IntegerIndexData and PostingListUsedIntegerIndexDataSerializer ====================================================================== Add PostingListAccessor abstract class for common components and methods ====================================================================== Implement PostingListIntegerIndexDataAccessor ====================================================================== Create PostingListIntegerIndexDataAccessorTest ====================================================================== Fix Icing Segmentation tests for word connectors that changed in ICU 72. ====================================================================== Modify the Advanced Query grammar to allow functions to accept expressions. ====================================================================== Implement QueryVisitor. ====================================================================== Enable the Advanced Query Parser to handle member functions ====================================================================== Refactor the Scorer class to support the Advanced Scoring Language ====================================================================== Integrate advanced query parser with the query processor. ====================================================================== Implement support for JoinSpec in Icing. ====================================================================== Implement the Advanced Scoring Language for basic functions and operators ====================================================================== Bug: 208654892 Bug: 249829533 Bug: 256022027 Bug: 261474063 Bug: 240333360 Bug: 193919210 Change-Id: Iffcbff7daa4a519e97e03c3c62dfa5e09524491e
2022-12-14Sync from upstream.Tim Barron
Descriptions: ====================================================================== Create class `QualifiedId` ====================================================================== Switch JoinProcessor to use new class FullyQualifiedId ====================================================================== Add `JoinableConfig` proto ====================================================================== Implement document-based functions for the Advanced Scoring Language ====================================================================== Support the RelevanceScore function for the Advanced Scoring Language ====================================================================== Enable the document-based member functions for Advanced Scoring Language Bug: 256022027 Bug: 261474063 Change-Id: I5f7e58dcc5e5cfc6089a856dd4f0617ccf485485
2022-12-12Sync from upstream.Tim Barron
Descriptions: ====================================================================== Add ScoringSpec into JoinSpec. Rename joined_document to child_document. ====================================================================== Create JoinedScoredDocumentHit class and refactor ScoredDocumentHitsRanker. ====================================================================== Implement initial Join workflow ====================================================================== Implement the Lexer for Icing Advanced Query Language ====================================================================== Create struct Options for PersistentHashMap ====================================================================== Premapping FileBackedVector ====================================================================== Create class PersistentHashMapKeyMapper ====================================================================== Add integer sections into TokenizedDocument and rename string sections ====================================================================== Create NumericIndex interface and DocHitInfoIteratorNumeric ====================================================================== Implement DummyNumericIndex and unit test ====================================================================== Change PostingListAccessor::Finalize to rvalue member function ====================================================================== Define the Abstract Syntax Tree for Icing's list_filter parser. ====================================================================== Refactor query processing and score ====================================================================== Refactor IcingSearchEngine for AppSearch Dynamite Module 0p APIs ====================================================================== Implement the Lexer for Icing Advanced Scoring Language ====================================================================== Add a common interface for IcingSearchEngine and dynamite client ====================================================================== Implement a subset of the query grammar. ====================================================================== Refactor index processor ====================================================================== Add integer index into IcingSearchEngine and IndexProcessor ====================================================================== Implement the parser for Icing Advanced Scoring Language ====================================================================== Implement IntegerIndexData and PostingListUsedIntegerIndexDataSerializer ====================================================================== Add PostingListAccessor abstract class for common components and methods ====================================================================== Implement PostingListIntegerIndexDataAccessor ====================================================================== Create PostingListIntegerIndexDataAccessorTest ====================================================================== Fix Icing Segmentation tests for word connectors that changed in ICU 72. ====================================================================== Modify the Advanced Query grammar to allow functions to accept expressions. ====================================================================== Implement QueryVisitor. ====================================================================== Enable the Advanced Query Parser to handle member functions ====================================================================== Refactor the Scorer class to support the Advanced Scoring Language ====================================================================== Integrate advanced query parser with the query processor. ====================================================================== Implement support for JoinSpec in Icing. ====================================================================== Implement the Advanced Scoring Language for basic functions and operators ====================================================================== Bug: 208654892 Bug: 249829533 Bug: 256022027 Bug: 261474063 Bug: 240333360 Bug: 193919210 Change-Id: I5f5bdc6249282ecc4b014b4fbdf8e2d1f8b20c19
2022-11-22Merge remote-tracking branch 'goog/androidx-platform-dev'Armaan Danewalia
* goog/androidx-platform-dev: Sync from upstream. Descriptions: ====================================================================== Include equals-proto and convert CodeToString methods to inline. ====================================================================== Add schema and document generators used by monkey test ====================================================================== Create in-memory icing for monkey testing ====================================================================== [NumericSearch][Storage][refactor_posting_list][1/x] Create PostingListUsedHitSerializer ====================================================================== [NumericSearch][Storage][refactor_posting_list][2/x] Create PostingListUsedHitSerializerTest ====================================================================== [NumericSearch][Storage][refactor_posting_list][3/x] Refactor all posting list related classes to use PostingListUsedSerializer ====================================================================== Adds a JoinSpecProto, and a new ranking strategy ====================================================================== Create Monkey Test Runner that randomly performs Icing API calls and check the results with the in-memory Icing ====================================================================== Support monkey testing the DeleteByQuery and Search APIs of Icing search engine ====================================================================== Directly include proto.h files from portable_proto_library() instead of wrapper pb.h files. ====================================================================== Directly include proto.h files from portable_proto_library() instead of wrapper pb.h files. ====================================================================== Directly include proto.h files from portable_proto_library() instead of wrapper pb.h files. ====================================================================== Removes nested_query from JoinSpec, as nested_search_spec includes it. ====================================================================== Directly include proto.h files from portable_proto_library() instead of wrapper pb.h files. ====================================================================== Swap the order when we build the doc-hit-info-iterator-and. ====================================================================== Support monkey testing section restrictions in the DeleteByQuery and Search APIs ====================================================================== Allow the in-memory icing to return the number of deleted documents, and check it with the delete stats of DeleteByNamespace, DeleteBySchemaType, and DeleteByQuery in the monkey test ====================================================================== [NumericSearch][Storage][refactor_posting_list][4/x] Move posting list common files into another directory ====================================================================== Add file-skipping for exports to Jetpack using @exportToAOSP:skipFile() tag ====================================================================== Change invalid type int32 to int32_t in icing-search-enging-jni.cc. ====================================================================== Minor fix, move generate term_iterator inside of the ranking_strategy checking ====================================================================== Address index out of bounds issue in third_party/icing/result/snippet-retriever.cc. This issue is causing test failures (see example failure: http://sponge2/cfcda71a-1312-455a-9a70-821c74c708e6). ====================================================================== Fix 1 DependencyCleaner findings: ====================================================================== Implement URL tokenization for Icing-lib [2/2]: ====================================================================== [NumericSearch][General][1/x] Create numeric.proto and add IntegerIndexingConfig ====================================================================== [NumericSearch][General][2/x] Refactor GetStringSectionContent and GetStringPropertyContent ====================================================================== [NumericSearch][General][3/x] Add DataType into SectionMetadata and change AssignSections ====================================================================== Move unit test constant definition into schema builder ====================================================================== [NumericSearch][General][4/x] Create templated Section and SectionGroup ====================================================================== [NumericSearch][General][5/x] Create BasicHit ====================================================================== Change URL tokenizer's url_parse dependency to use third_party/url_parse:url_parse_stripped_down ====================================================================== Exclude memory-mapped-file-leak_test.cc from export to AOSP. ====================================================================== Rollback of changelist 487633403. Reason: This cl breaks YouTube Music builds due to duplicate symbols. ====================================================================== [ez] Fix ScoredDocumentHit related comparators ====================================================================== Bug: 193244409 Bug: 246984163 Bug: 249829533 Bug: 256022027 Bug: 256679292 Bug: 246964044 Change-Id: Idd379d0b0561e4d229750bd7b04d80fa66b7ed49
2022-11-11Sync from upstream.Armaan Danewalia
Descriptions: ====================================================================== Include equals-proto and convert CodeToString methods to inline. ====================================================================== Add schema and document generators used by monkey test ====================================================================== Create in-memory icing for monkey testing ====================================================================== [NumericSearch][Storage][refactor_posting_list][1/x] Create PostingListUsedHitSerializer ====================================================================== [NumericSearch][Storage][refactor_posting_list][2/x] Create PostingListUsedHitSerializerTest ====================================================================== [NumericSearch][Storage][refactor_posting_list][3/x] Refactor all posting list related classes to use PostingListUsedSerializer ====================================================================== Adds a JoinSpecProto, and a new ranking strategy ====================================================================== Create Monkey Test Runner that randomly performs Icing API calls and check the results with the in-memory Icing ====================================================================== Support monkey testing the DeleteByQuery and Search APIs of Icing search engine ====================================================================== Directly include proto.h files from portable_proto_library() instead of wrapper pb.h files. ====================================================================== Directly include proto.h files from portable_proto_library() instead of wrapper pb.h files. ====================================================================== Directly include proto.h files from portable_proto_library() instead of wrapper pb.h files. ====================================================================== Removes nested_query from JoinSpec, as nested_search_spec includes it. ====================================================================== Directly include proto.h files from portable_proto_library() instead of wrapper pb.h files. ====================================================================== Swap the order when we build the doc-hit-info-iterator-and. ====================================================================== Support monkey testing section restrictions in the DeleteByQuery and Search APIs ====================================================================== Allow the in-memory icing to return the number of deleted documents, and check it with the delete stats of DeleteByNamespace, DeleteBySchemaType, and DeleteByQuery in the monkey test ====================================================================== [NumericSearch][Storage][refactor_posting_list][4/x] Move posting list common files into another directory ====================================================================== Add file-skipping for exports to Jetpack using @exportToAOSP:skipFile() tag ====================================================================== Change invalid type int32 to int32_t in icing-search-enging-jni.cc. ====================================================================== Minor fix, move generate term_iterator inside of the ranking_strategy checking ====================================================================== Address index out of bounds issue in third_party/icing/result/snippet-retriever.cc. This issue is causing test failures (see example failure: http://sponge2/cfcda71a-1312-455a-9a70-821c74c708e6). ====================================================================== Fix 1 DependencyCleaner findings: ====================================================================== Implement URL tokenization for Icing-lib [2/2]: ====================================================================== [NumericSearch][General][1/x] Create numeric.proto and add IntegerIndexingConfig ====================================================================== [NumericSearch][General][2/x] Refactor GetStringSectionContent and GetStringPropertyContent ====================================================================== [NumericSearch][General][3/x] Add DataType into SectionMetadata and change AssignSections ====================================================================== Move unit test constant definition into schema builder ====================================================================== [NumericSearch][General][4/x] Create templated Section and SectionGroup ====================================================================== [NumericSearch][General][5/x] Create BasicHit ====================================================================== Change URL tokenizer's url_parse dependency to use third_party/url_parse:url_parse_stripped_down ====================================================================== Exclude memory-mapped-file-leak_test.cc from export to AOSP. ====================================================================== Rollback of changelist 487633403. Reason: This cl breaks YouTube Music builds due to duplicate symbols. ====================================================================== [ez] Fix ScoredDocumentHit related comparators ====================================================================== Bug: 193244409 Bug: 246984163 Bug: 249829533 Bug: 256022027 Bug: 256679292 Bug: 246964044 Change-Id: I55c2ed417a31321c22de377e97ffe0096478d28c
2022-11-09Merge tracking branch 'androidx-platform-dev' into 'master'Alexander Dorokhine
* goog/androidx-platform-dev: Fix g3 sync for merge to androidx-main Sync from upstream. Remove url-tokenizer from upstream-master. Update Icing from upstream. Update Icing from upstream. Sync from upstream. Changes included: ====================================================================== [FileBackedVector Consolidation][4/x] Fix potential PWrite bug in GrowIfNecessary ====================================================================== [FileBackedVector Consolidation][5/x] Create benchmark for FileBackedVector ====================================================================== [FileBackedVector Consolidation][6/x] Avoid calling GetFileSize in GrowIfNecessary ====================================================================== [PersistentHashMap][3.3/x] Implement Delete ====================================================================== Fix the PopulateMatchedTermsStats bug ====================================================================== Add JNI latency for query latency stats breakdown. ====================================================================== [ResultStateManager] Thread safety test1 ====================================================================== [ResultStateManager][2/x] Thread safety test2 ====================================================================== Add native lock contention latency for measuring query latency ====================================================================== Fix implementation of HasMember operator in ANTLR-based list-filter prototype. ====================================================================== Fix improper uses of std::string_view ====================================================================== Extend the scale of Icing ====================================================================== Decouple the term frequency array from DocHitInfo ====================================================================== Disable hit_term_frequency for non-relevance queries ====================================================================== [ResultStateManager][3/x] Thread safety test3 ====================================================================== [PersistentHashMap][4/x] Implement iterator ======================================================================= Fix the lite index compaction bug ====================================================================== Integrate ANTLR-based advanced query prototype with query processor. ====================================================================== [PersistentHashMap][6.1/x] Wrap the return value of KeyMapper::ComputeChecksum by StatusOr ====================================================================== [PersistentHashMap][6.0/x] Replace GetValuesToKeys with iterator for KeyMapper ====================================================================== [PersistentHashMap][5.1/x] Allow client to specify initial num buckets ====================================================================== [PersistentHashMap][5/x] Implement rehashing ====================================================================== Add SchemaType filter and Document Id filter in Search Suggestion API. ====================================================================== Add Document Id filters in Search Suggestion API. ====================================================================== Cleanup LSC: Replace inclusion of *_proto_portable.pb.h files coming from portable_proto_library() with *(!_proto_portable).pb.h coming from cc_p roto_library(). ====================================================================== Cleanup LSC: Replace inclusion of *_proto_portable.pb.h files coming from portable_proto_library() with *(!_proto_portable).pb.h coming from cc_proto_library(). ====================================================================== Cleanup LSC: Replace inclusion of *_proto_portable.pb.h files coming from portable_proto_library() with *(!_proto_portable).pb.h coming from cc_proto_library(). ====================================================================== Cleanup Remove unused visibility specs (last referenced in codebase over 132 days ago). ====================================================================== Cleanup LSC: Replace inclusion of *_proto_portable.pb.h files coming from portable_proto_library() with *(!_proto_portable).pb.h coming from cc_proto_library(). ====================================================================== Cleanup LSC: Replace inclusion of *_proto_portable.pb.h files coming from portable_proto_library() with *(!_proto_portable).pb.h coming from cc_proto_library(). ====================================================================== Cleanup Move package level default_copts attribute to copts. ====================================================================== Refactor QueryProcessor and QueryProcessTest in preparation for adding ANTLR prototype to parse queries with search_type EXPERIMENTAL_ICING_ADVANCED_QUERY. ====================================================================== Implement URL tokenization for Icing-lib ====================================================================== Adds RFC822_HOST_ADDRESS ====================================================================== Support Suggestion API could be ordered by term's frequency. ====================================================================== Fix the bug in PostingListAccessor found by the Icing Monkey test ====================================================================== Add the logic to handle fatal errors from IcingDynamicTrie to avoid crashing ====================================================================== Clear out the dead code IcingDynamicTrie::Compact ====================================================================== [MemoryMappedFile][RemapV2][1/x] Add factory method for MemoryMappedFile ====================================================================== [MemoryMappedFile][RemapV2][2/x] Create GrowAndRemapIfNecessary and change factory method ====================================================================== [MemoryMappedFile][RemapV2][3/x] Migrate FileBackedVector to use GrowAndRemapIfNecessary ====================================================================== Add JNI latency for query latency stats breakdown ====================================================================== Bug: 146903474 Bug: 193919210 Bug: 241536882 Bug: 236412954 Bug: 218735359 Bug: 208654892 Bug: 241310816 Bug: 236412954 Bug: 242761389 Bug: 243889347 Bug: 208654892 Bug: 230553264 Bug: 237324702 Bug: 193919210 Bug: 246964044 Bug: 230553264 Bug: 247671531 Bug: 247929909 Bug: 253282365 Test: Presubmit Change-Id: I05234ae4c6487009f23b3c7b7c4b74f88153e74c
2022-10-11Update Icing from upstream.Terry Wang
Descriptions: ====================================================================== Implement URL tokenization for Icing-lib ====================================================================== Adds RFC822_HOST_ADDRESS ====================================================================== Support Suggestion API could be ordered by term's frequency. ====================================================================== Bug: 246964044 Bug: 230553264 Change-Id: Id7e7b1e080bb66ccf03452e75b72c5cceed2f7db
2022-09-30Update Icing from upstream.Terry Wang
Descriptions: ====================================================================== Integrate ANTLR-based advanced query prototype with query processor. ====================================================================== [PersistentHashMap][6.1/x] Wrap the return value of KeyMapper::ComputeChecksum by StatusOr ====================================================================== [PersistentHashMap][6.0/x] Replace GetValuesToKeys with iterator for KeyMapper ====================================================================== [PersistentHashMap][5.1/x] Allow client to specify initial num buckets ====================================================================== [PersistentHashMap][5/x] Implement rehashing ====================================================================== Add SchemaType filter and Document Id filter in Search Suggestion API. ====================================================================== Follow up to cl/463377778 ====================================================================== Add Document Id filters in Search Suggestion API. ====================================================================== Cleanup LSC: Replace inclusion of *_proto_portable.pb.h files coming from portable_proto_library() with *(!_proto_portable).pb.h coming from cc_proto_library(). ====================================================================== Cleanup LSC: Replace inclusion of *_proto_portable.pb.h files coming from portable_proto_library() with *(!_proto_portable).pb.h coming from cc_proto_library(). ====================================================================== Cloned from CL 464902284 by 'g4 patch'. ====================================================================== Cleanup LSC: Replace inclusion of *_proto_portable.pb.h files coming from portable_proto_library() with *(!_proto_portable).pb.h coming from cc_proto_library(). ====================================================================== Cleanup Remove unused visibility specs (last referenced in codebase over 132 days ago). ====================================================================== Cleanup LSC: Replace inclusion of *_proto_portable.pb.h files coming from portable_proto_library() with *(!_proto_portable).pb.h coming from cc_proto_library(). ====================================================================== Cleanup LSC: Replace inclusion of *_proto_portable.pb.h files coming from portable_proto_library() with *(!_proto_portable).pb.h coming from cc_proto_library(). ====================================================================== Remove dsaadati@ from third_party/icing OWNERS ====================================================================== Cleanup Move package level default_copts attribute to copts. ====================================================================== Refactor QueryProcessor and QueryProcessTest in preparation for adding ANTLR prototype to parse queries with search_type EXPERIMENTAL_ICING_ADVANCED_QUERY. ====================================================================== Bug: 208654892 Bug: 230553264 Bug: 237324702 Bug: 193919210 Change-Id: I2f0a612747ccb754502489a9b168406532cffaee
2022-09-08Sync from upstream.Grace Zhao
Descriptions: ====================================================================== [FileBackedVector Consolidation][4/x] Fix potential PWrite bug in GrowIfNecessary ====================================================================== [FileBackedVector Consolidation][5/x] Create benchmark for FileBackedVector ====================================================================== [FileBackedVector Consolidation][6/x] Avoid calling GetFileSize in GrowIfNecessary ====================================================================== [PersistentHashMap][3.3/x] Implement Delete ====================================================================== Fix the PopulateMatchedTermsStats bug ====================================================================== Add JNI latency for query latency stats breakdown. ====================================================================== [ResultStateManager] Thread safety test1 ====================================================================== [ResultStateManager][2/x] Thread safety test2 ====================================================================== Add native lock contention latency for measuring query latency ====================================================================== Fix implementation of HasMember operator in ANTLR-based list-filter prototype. ====================================================================== Fix improper uses of std::string_view ====================================================================== Extend the scale of Icing ====================================================================== Decouple the term frequency array from DocHitInfo ====================================================================== Disable hit_term_frequency for non-relevance queries ====================================================================== [ResultStateManager][3/x] Thread safety test3 ====================================================================== [PersistentHashMap][4/x] Implement iterator ======================================================================= Fix the lite index compaction bug ======================================================================= Change-Id: I0edad67affed97af107e2d7cd73770e0268c0903
2022-08-12Merge branch 'goog/androidx-platform-dev' into tm-mainline-prodAlexander Dorokhine
Descriptions: ====================================================================== Implement new version of ResultState and ResultStateManager to 1) enforce a page byte size limit and 2) improve handling of pagination when we encounter deleted documents. ====================================================================== Fix bugs in IcingDynamicTrie::Delete. ====================================================================== Implement IcingDynamicTrie::IsBranchingTerm. ====================================================================== Change Icing default logging level to INFO ====================================================================== Refactor KeyMapper class to be an interface. ====================================================================== Improve NamespaceChecker logic to improve Suggest latency. ====================================================================== Change icing native log tag to "AppSearchIcing" ====================================================================== Implement Index Compaction rather than rebuilding index during Compaction. ====================================================================== Implement reverse iterator for IcingDynamicTrie ====================================================================== Avoid adding unnecessary branch points during index compaction ====================================================================== Invalidate expired result states when adding to/retrieving from ResultStateManager. ====================================================================== Add new methods (MutableView, MutableArrayView, Append, Allocate) to FileBackedVector ====================================================================== Create and implement PersistentHashMap class. ====================================================================== Implement RFC822 Tokenizer ====================================================================== Remove uses of StringPrintf in ICING_LOG statements ====================================================================== Properly set query latency when an error is encountered or results are empty. ====================================================================== Bug: 146903474 Bug: 152934343 Bug: 193919210 Bug: 193453081 Bug: 231368517 Bug: 235395538 Bug: 236412165 Test: presubmit Change-Id: Ie045c6e70e0db1272971fcd60f147ab11889cbbe
2022-08-11Sync from upstream.Tim Barron
Descriptions: ====================================================================== Implement new version of ResultState and ResultStateManager to 1) enforce a page byte size limit and 2) improve handling of pagination when we encounter deleted documents. ====================================================================== Fix bugs in IcingDynamicTrie::Delete. ====================================================================== Implement IcingDynamicTrie::IsBranchingTerm. ====================================================================== Change Icing default logging level to INFO ====================================================================== Refactor KeyMapper class to be an interface. ====================================================================== Improve NamespaceChecker logic to improve Suggest latency. ====================================================================== Change icing native log tag to "AppSearchIcing" ====================================================================== Implement Index Compaction rather than rebuilding index during Compaction. ====================================================================== Implement reverse iterator for IcingDynamicTrie ====================================================================== Avoid adding unnecessary branch points during index compaction ====================================================================== Invalidate expired result states when adding to/retrieving from ResultStateManager. ====================================================================== Add new methods (MutableView, MutableArrayView, Append, Allocate) to FileBackedVector ====================================================================== Create and implement PersistentHashMap class. ====================================================================== Implement RFC822 Tokenizer ====================================================================== Remove uses of StringPrintf in ICING_LOG statements ====================================================================== Properly set query latency when an error is encountered or results are empty. ====================================================================== Bug: 146903474 Bug: 152934343 Bug: 193919210 Bug: 193453081 Bug: 231368517 Bug: 235395538 Bug: 236412165 Change-Id: I8aa278cebb12b25b39deb0ef584c0f198952659d
2022-06-22Merge remote-tracking branch 'goog/androidx-platform-dev' into tm-mainline-prodAlexander Dorokhine
Changes included: ====================================================================== Refactor DocumentStore::Initialize to improve readability of document store recovery. ====================================================================== Remove non-NDK API usages of ICU4C in libicing. ====================================================================== Move IcuDataFileHelper to the testing directory since it is a test-only util. ====================================================================== Support dump function for DocumentStore ====================================================================== Switch to use PRead rather than MMap in the proto log. ====================================================================== Support dump function for main/lite index and lexicon ====================================================================== Fix LiteIndex::AppendHits ====================================================================== Enable and fix DocumentStoreTest.LoadScoreCacheAndInitializeSuccessfully ====================================================================== Fix MainIndex::GetStorageInfo. ====================================================================== Fix icing-search-engine_fuzz_test by making IcuLanguageSegmenterIterator::Advance non-recursive. ====================================================================== Allow to return additional information for deleted documents in DeleteByQuery ====================================================================== Using enum class in Token::Type for better type safety. ====================================================================== Add some additional logging that will help diagnose b/218413237 ====================================================================== Mark VerbatimTokenizer::ResetToTokenStartingAfter as 'override'. ====================================================================== Support dump function for SchemaStore ====================================================================== Export Icing logging control to JNI ====================================================================== Prepare Icing logging class for JNI export ====================================================================== Export getDebugInfo to JNI ====================================================================== Expose the return_deleted_document_info parameter for deleteByQuery JNI ====================================================================== Enable runtime log control for Icing Library ====================================================================== Fix 1 ClangTidyBuild finding: ====================================================================== Update comments to run benchmarks. ====================================================================== Making icing's own logging class ====================================================================== Convert the string lexicon debug information to a protocol buffer ====================================================================== Fix issue with printing fingerprinted key in our error logs. ====================================================================== Support dump function for IcingSearchEngine ====================================================================== Fix bug in schema store where a failure during RegenerateDerivedFiles would lead to a dangling pointer. ====================================================================== Add RAII class that will create and destroy file directories. ====================================================================== Convert MainIndexDebugInfoProto and LiteIndexDebugInfoProto to string ====================================================================== Make SchemaStore move assignable. ====================================================================== Rollback of "convert the string lexicon debug information to a protocol buffer" ====================================================================== Fix NPE caused by a remap failure. ====================================================================== Unify the name "priority" and "severity" in Icing logging ====================================================================== Avoiding string formatting in Icing logging when we should not log ====================================================================== Switch to use an enum with BASIC/DETAILED to control the verbosity of getDebugInfo ====================================================================== Remove the behavior in the Language Segmenter to filter out non-ascii+non-alphanumeric characters. ====================================================================== Fix the SetSchema bug when we override a schema with nested incompatible types ====================================================================== Wrap __android_log_write with __android_is_loggable ====================================================================== Enable removing expired page tokens to free cache space ====================================================================== Export Icing logging tag to JNI ====================================================================== Update export_to_aosp.sh to change icing log tag to "AppSearchIcing" ====================================================================== Improve the logic of NamespaceChecker. ====================================================================== Step 4.1: Use ScoredDocumentHitsRanker in ResultStateV2 ====================================================================== Step 4.0: Create ScoredDocumentHitsRanker interface and PriorityQueueScoredDocumentHitsRanker ====================================================================== Refactor KeyMapper ====================================================================== Change Icing DEFAULT_LOGGING_LEVEL to INFO ====================================================================== Step 3.4: Create ResultRetrieverV2GroupResultLimiterTest (copied from ResultStateTest) ====================================================================== Add IcingDynamicTrie::IsBranchingTerm to check if a term is branching. ====================================================================== Fix IcingDynamicTrie::Delete bug ====================================================================== Step 3.3: Create ResultRetrieverV2ProjectionTest (copied from ResultRetrieverTest) ====================================================================== Step 3.2: Create ResultRetrieverV2SnippetTest (copied from ResultRetrieverTest) ====================================================================== Step 3.1: Create ResultRetrieverV2Test (copied from ResultRetrieverTest) ====================================================================== Enable legacy multidex ====================================================================== Fix NPE caused by improper handling of return value of GetFileSize. ====================================================================== Step 3.0: Create ResultRetrieverV2 (copied from ResultRetriever) ====================================================================== Step 2: Create PageResult (copied from PageResultState) ====================================================================== Change AppSearch hawkeye testing app multidex to legacy. ====================================================================== Fix icing-search-engine_benchmark bug ====================================================================== (Small fix for step 1) Fix unit test stack memory error ====================================================================== Step 1: Create ResultStateV2 and ResultStateV2Test (copied from ResultState, ResultStateTest) ====================================================================== Bug: 146903474 Bug: 152934343 Bug: 158089703 Bug: 185845269 Bug: 193453081 Bug: 193919210 Bug: 209071710 Bug: 211785521 Bug: 218413237 Bug: 222349894 Bug: 223549255 Bug: 225914361 Bug: 229770338 Bug: 229778472 Bug: 230879098 Bug: 231237897 Bug: 231368517 Bug: 231416401 Bug: 232273174 Bug: 233470404 Bug: 233657885 Test: Presubmit Change-Id: I61a1336455544d12c35ad29f1e79a9a1eb9af14a
2022-05-20Sync from upstream.Tim Barron
Descriptions: ====================================================================== Fix bug in schema store where a failure during RegenerateDerivedFiles would lead to a dangling pointer. ====================================================================== Add RAII class that will create and destroy file directories. ====================================================================== Convert MainIndexDebugInfoProto and LiteIndexDebugInfoProto to string ====================================================================== Make SchemaStore move assignable. ====================================================================== Rollback of "convert the string lexicon debug information to a protocol buffer" ====================================================================== Fix NPE caused by a remap failure. ====================================================================== Unify the name "priority" and "severity" in Icing logging ====================================================================== Avoiding string formatting in Icing logging when we should not log ====================================================================== Switch to use an enum with BASIC/DETAILED to control the verbosity of getDebugInfo ====================================================================== Remove the behavior in the Language Segmenter to filter out non-ascii+non-alphanumeric characters. ====================================================================== Fix the SetSchema bug when we override a schema with nested incompatible types ====================================================================== Wrap __android_log_write with __android_is_loggable ====================================================================== Enable removing expired page tokens to free cache space ====================================================================== Bug: 146903474 Bug: 193453081 Bug: 222349894 Bug: 229770338 Bug: 229778472 Bug: 230879098 Bug: 231416401 Bug: 231237897 Bug: 232273174 Change-Id: I22f050de16f56dce39e12a7033947519d598c840
2022-05-03Sync from upstream.Jiayu Hu
Descriptions: ====================================================================== Export Icing logging control to JNI ====================================================================== Prepare Icing logging class for JNI export ====================================================================== Export getDebugInfo to JNI ====================================================================== Expose the return_deleted_document_info parameter for deleteByQuery JNI ====================================================================== Enable runtime log control for Icing Library ====================================================================== Fix 1 ClangTidyBuild finding: ====================================================================== Update comments to run benchmarks. ====================================================================== Making icing's own logging class ====================================================================== Convert the string lexicon debug information to a protocol buffer ====================================================================== Fix issue with printing fingerprinted key in our error logs. ====================================================================== Support dump function for IcingSearchEngine ====================================================================== Bug: 146903474 Bug: 229778472 Bug: 209071710 Bug: 222349894 Bug: 225914361 Change-Id: I9750149d1ed0b59f345b8828ff312a62773667fe