aboutsummaryrefslogtreecommitdiff
path: root/icing
AgeCommit message (Collapse)Author
2023-09-15Fix the crash when a schema type gets more indexable properties than allowedJiayu Hu
This is a cherrypick for cl/565170429. Test: presubmits Change-Id: I42f619cfa02d33df9026c0638f9d202e0153ad37
2023-09-14Realign external/icing with upstream.Tim Barron
We ran into issues syncing external/icing because icing-search-engine-jni was trying to register functions that were being compiled out. With our updated proguard configuration in AppSearch, we can restore these registration calls and bring this code into sync with upstream. Test: presubmits Change-Id: I8f7e15475f7cf3ef4b1fe73983c6dea135548efe
2023-09-03Merge remote-tracking branch 'goog/androidx-platform-dev' into udc-mainline-prodTerry Wang
* goog/androidx-platform-dev: Update Icing from upstream. Descriptions: ======================================================================== [fixit] Remove incorrect reserve usage for STL data structure ======================================================================== Move LiteIndex::SortHits() from the query path to the indexing path ======================================================================== Fix term frequency bug ======================================================================== Delete dead JNI functions. ======================================================================== Switch Icing JNI implementation to use RegisterNatives ======================================================================== Avoid unnecessary GetObjectClass and GetFieldID calls. ======================================================================== Add join test for SetSchemaNewIndexedDocumentPropertyTriggersIndexRestorationAndReturnsOk ======================================================================== Support different sets of properties to search for different schema types. ======================================================================== Do not cache the section mask for a schema type if no property filters apply while processing the type property filters. ======================================================================== Bug: 286418010 Bug: 297549761 Bug: 248295995 Bug: 296938196 Bug: 288310393 Bug: 291019114 Test: presubmit Change-Id: Iab66bcc3f92641122fae2ac36132f61d8e4b4f57
2023-08-31Merge remote-tracking branch 'aosp/upstream-master' into androidx-mainandroidx-wear-wear-tooling-preview-releaseandroidx-core-performance-releaseGrace Zhao
* aosp/upstream-master: Update Icing from upstream. Descriptions: ======================================================================== [fixit] Remove incorrect reserve usage for STL data structure ======================================================================== Move LiteIndex::SortHits() from the query path to the indexing path ======================================================================== Bug: 286418010 Bug: 297549761 Bug: 248295995 Change-Id: I6fcfd150036dc47a42181dfa14b410220da93d31
2023-08-30Update Icing from upstream.Grace Zhao
Descriptions: ======================================================================== [fixit] Remove incorrect reserve usage for STL data structure ======================================================================== Move LiteIndex::SortHits() from the query path to the indexing path ======================================================================== Bug: 286418010 Bug: 297549761 Bug: 248295995 Change-Id: I46168178e0227451b0f7dbf394bb6cc5db94e1e8
2023-08-30Merge remote-tracking branch 'aosp/upstream-master' into androidx-mainTim Barron
* aosp/upstream-master: Update Icing from upstream. Descriptions: ======================================================================== Fix term frequency bug ======================================================================== Delete dead JNI functions. ======================================================================== Switch Icing JNI implementation to use RegisterNatives ======================================================================== Avoid unnecessary GetObjectClass and GetFieldID calls. ======================================================================== Bug: 296938196 Change-Id: Idaaf78068bcfa8d9a34efd9b892c20049646874f
2023-08-23Update Icing from upstream.Tim Barron
Descriptions: ======================================================================== Fix term frequency bug ======================================================================== Delete dead JNI functions. ======================================================================== Switch Icing JNI implementation to use RegisterNatives ======================================================================== Avoid unnecessary GetObjectClass and GetFieldID calls. ======================================================================== Bug: 296938196 Change-Id: I773bcc05e768a7ac80f1950985b618047d009641
2023-08-07Merge remote-tracking branch 'aosp/upstream-master' into androidx-mainYogesh Singh
* aosp/upstream-master: Update Icing from upstream. Descriptions: ======================================================================== Add join test for SetSchemaNewIndexedDocumentPropertyTriggersIndexRestorationAndReturnsOk ======================================================================== Support different sets of properties to search for different schema types. ======================================================================== Do not cache the section mask for a schema type if no property filters apply while processing the type property filters. ======================================================================== Bug: 288310393 Bug: 291019114 Change-Id: If510ff1ef81f95c2dc10027c1faef0bee6b49aa6
2023-08-07Update Icing from upstream.Yogesh Singh
Descriptions: ======================================================================== Add join test for SetSchemaNewIndexedDocumentPropertyTriggersIndexRestorationAndReturnsOk ======================================================================== Support different sets of properties to search for different schema types. ======================================================================== Do not cache the section mask for a schema type if no property filters apply while processing the type property filters. ======================================================================== Bug: 288310393 Bug: 291019114 Change-Id: Icb867b40a603c0d63e2d96a0e2ec641e424aad7e
2023-07-17Merge remote-tracking branch 'aosp/upstream-master' into androidx-mainGrace Zhao
* aosp/upstream-master: Update Icing from upstream. Descriptions: ======================================================================== [Icing][adhoc] Add backup schema producer unit tests for properties using indexable nested properties list ======================================================================== Fix SetSchema bug where adding an indexable nested document property does not trigger index-rebuild ======================================================================== Add join test for SetSchemaNewIndexedDocumentPropertyTriggersIndexRestorationAndReturnsOk ======================================================================== Bug: 291019114 NO_IFTTT="Path is only valid in G3." Change-Id: Ib0f38921783a9ea2a964523439f3547250853cb0
2023-07-14Update Icing from upstream.Grace Zhao
Descriptions: ======================================================================== [Icing][adhoc] Add backup schema producer unit tests for properties using indexable nested properties list ======================================================================== Fix SetSchema bug where adding an indexable nested document property does not trigger index-rebuild ======================================================================== Add join test for SetSchemaNewIndexedDocumentPropertyTriggersIndexRestorationAndReturnsOk ======================================================================== Bug: 291019114 NO_IFTTT="Path is only valid in G3." Change-Id: I443f0afe79b63c483ca01b96975229f0f926d80d
2023-07-07Merge remote-tracking branch 'aosp/upstream-master' into androidx-mainGrace Zhao
* aosp/upstream-master: Update Icing from upstream. Descriptions: ======================================================================== [Icing] Extend document store uri mapper ======================================================================== Implement dirty flag in PersistentStorage ======================================================================== [Icing][version 2][1/x] Implement ShouldRebuildDerivedFiles ======================================================================== [Icing][version 2][2/x] Bump kVersion to 2 ======================================================================== [NumericSearch][optimization] Change numeric index bucket splitting threshold ======================================================================== [NumericSearch][optimization] Flag guard integer index bucket splitting threshold ======================================================================== Fix MainIndex::TransferAndAddHits by adding a safety check ======================================================================== [Indexable_nested_properties_list 4.5/5] Modify sectionId assignment so that all property paths in the indexable_nested_properties_list consume sectionIds. ======================================================================== Bug: 288934779 Bug: 289150947 Bug: 193919210 Bug: 280094535 Bug: 288969109 Bug: 259743562 NO_IFTTT="Path is only valid in G3." Change-Id: I093500e50399dc90fc2e85cee8e5a48a3c50b1d5
2023-07-07Update Icing from upstream.Grace Zhao
Descriptions: ======================================================================== Fix MainIndex::TransferAndAddHits by adding a safety check ======================================================================== [Indexable_nested_properties_list 4.5/5] Modify sectionId assignment so that all property paths in the indexable_nested_properties_list consume sectionIds. ======================================================================== Bug: 288934779 Bug: 289150947 NO_IFTTT="Path is only valid in G3." Change-Id: Ibad4a09d2026ad038f6b64391497e4248fb891ed
2023-07-06Update Icing from upstream.Yen-Chieh Sung
Descriptions: ======================================================================== [Icing] Extend document store uri mapper ======================================================================== Implement dirty flag in PersistentStorage ======================================================================== [Icing][version 2][1/x] Implement ShouldRebuildDerivedFiles ======================================================================== [Icing][version 2][2/x] Bump kVersion to 2 ======================================================================== [NumericSearch][optimization] Change numeric index bucket splitting threshold ======================================================================== [NumericSearch][optimization] Flag guard integer index bucket splitting threshold ======================================================================== Bug: 193919210 Bug: 280094535 Bug: 288969109 Bug: 259743562 NO_IFTTT="Path is only valid in G3." Change-Id: Ie1e6f5969549f6c945ea32d03c724850ff5600a9
2023-06-28Merge remote-tracking branch 'aosp/upstream-master' into androidx-mainGrace Zhao
* aosp/upstream-master: Update Icing from upstream. Descriptions: ======================================================================== [Join][cleanup][ez] Rename QualifiedIdTypeJoinableIndex as QualifiedIdJoinIndex ======================================================================== Better evaluation for persistent hash map size in join index ======================================================================== Add num_to_score into ResultSpecProto ======================================================================== [Indexable_nested_properties_list 1/5] Add indexable_nested_properties to DocumentIndexingConfig and check it during schema validation. ======================================================================== [Indexable_nested_properties_list 2/5] Add check for indexable_neseted_properties when computing SchemaDelta ======================================================================== [Indexable_nested_properties_list 3/5] Add indexable_nested_properties to schema-property-iterator ======================================================================== [Indexable_nested_properties_list 4/5] Integrate circular schema definitions with schema-property-iterator ======================================================================== Bug: 272145329 Bug: 284232550 Bug: 285636503 Bug: 265304217 Bug: 289150947 Change-Id: Icfcd4901015b75f4af2d231f8a8613c6e33034a0
2023-06-28Update Icing from upstream.Grace Zhao
Descriptions: ======================================================================== [Join][cleanup][ez] Rename QualifiedIdTypeJoinableIndex as QualifiedIdJoinIndex ======================================================================== Better evaluation for persistent hash map size in join index ======================================================================== Add num_to_score into ResultSpecProto ======================================================================== [Indexable_nested_properties_list 1/5] Add indexable_nested_properties to DocumentIndexingConfig and check it during schema validation. ======================================================================== [Indexable_nested_properties_list 2/5] Add check for indexable_neseted_properties when computing SchemaDelta ======================================================================== [Indexable_nested_properties_list 3/5] Add indexable_nested_properties to schema-property-iterator ======================================================================== [Indexable_nested_properties_list 4/5] Integrate circular schema definitions with schema-property-iterator ======================================================================== Bug: 272145329 Bug: 284232550 Bug: 285636503 Bug: 265304217 Bug: 289150947 Change-Id: I0bc3282400e6c69f8551dbddf47585864bfdec06
2023-06-02Merge remote-tracking branch 'goog/upstream-master' into androidx-platform-devTim Barron
* goog/upstream-master: Update Icing from upstream. Update Icing from upstream. Descriptions: ======================================================================== Eliminate unnecessary GetCurrentTime syscalls in various hotspots. ======================================================================== Fix MSAN failure in SchemaStore::Header::Write. ======================================================================== Downgrade log priorities on filesystem errors. ======================================================================== Bug: 282308801 Bug: 282997030 Change-Id: I0c95300007aeab6460f6bb495dc38e548a52abbd
2023-06-02Update Icing from upstream.Tim Barron
Descriptions: ======================================================================== Eliminate unnecessary GetCurrentTime syscalls in various hotspots. ======================================================================== Fix MSAN failure in SchemaStore::Header::Write. ======================================================================== Bug: 282308801 Change-Id: Ifb11f890485329e6b0bc1a9cbf2bf046e39b7854
2023-05-23Update Icing from upstream.Tim Barron
Descriptions: ======================================================================== Downgrade log priorities on filesystem errors. ======================================================================== Bug: 282997030 Change-Id: Ia6dd9c9c1f3bdb68458cd1e9dba83e01a4b25123
2023-05-20Merge remote-tracking branch 'goog/upstream-master' into androidx-platform-devTim
* goog/upstream-master: Update Icing from upstream. Descriptions: ======================================================================== Rename `this.childrenScores` to `this.childrenRankingSignals`. ======================================================================== Change-Id: Ic4d6c549f7504b68caea87795ac291d88106ccd7
2023-05-19Update Icing from upstream.Tim Barron
Descriptions: ======================================================================== Rename `this.childrenScores` to `this.childrenRankingSignals`. ======================================================================== Change-Id: I6ba4087a6de76ea13d149f4af1b7a778e6623a5d
2023-05-19Merge remote-tracking branch 'goog/upstream-master' into androidx-platform-devTim Barron
* goog/upstream-master: Update Icing from upstream. Descriptions: ======================================================================== Add flag to guard PersistentHashMapKeyMapper and DynamicTrieKeyMapper ======================================================================== Add flag to guard FileBackedVector premapping ======================================================================== Check tokenizer type when assigning section id ======================================================================== Bug: 193919210 Change-Id: Ibb064fd62a9e5b867e72701e89448d67215c3c9f
2023-05-19Update Icing from upstream.Tim Barron
Descriptions: ======================================================================== Add flag to guard PersistentHashMapKeyMapper and DynamicTrieKeyMapper ======================================================================== Add flag to guard FileBackedVector premapping ======================================================================== Check tokenizer type when assigning section id ======================================================================== Bug:193919210 Change-Id: I682e22d4ba647220b7222fcc63c41bd0e01bbeb9
2023-05-15Merge remote-tracking branch 'goog/upstream-master' into androidx-platform-devTerry Wang
* goog/upstream-master: Update Icing from upstream. Descriptions: ======================================================================== Minor fix for export to framework. ======================================================================== [Icing][version][3/x][ez] Call DataSync for version file ======================================================================== Add `use_read_only_search` flag for flag-guarding IcingSearchEngine::Search locking change in cl/506671441. ======================================================================== Bug: 146008613 Change-Id: Ia22ef83345bdbdd73bf9896191cc91db2a3a0f4c
2023-05-15Update Icing from upstream.Terry Wang
Descriptions: ======================================================================== Minor fix for export to framework. ======================================================================== [Icing][version][3/x][ez] Call DataSync for version file ======================================================================== Add `use_read_only_search` flag for flag-guarding IcingSearchEngine::Search locking change in cl/506671441. ======================================================================== Bug: 146008613 Test: presubmit Change-Id: Id863275cf5cc1ad7bcc69649527cc17981cc6cd2
2023-05-11Merge remote-tracking branch 'goog/upstream-master' into androidx-platform-devTim
Update Icing from upstream. Descriptions: ======================================================================== Handle version changes in the schema store. ======================================================================== Modify the definition of propertyDefined: ======================================================================== Remove default args in SchemaStore::SetSchema and fix calls ======================================================================== Add allow_circular_schema_definitions flag ======================================================================== Onboard version detection to Icing ======================================================================== Add version util to help read/write version info ======================================================================== Add support for the overlay schema. ======================================================================== Allow cycles in schema-property-iterator ======================================================================== Add joinable properties into schema definition cycle restrictions. ======================================================================== Loosen circular references restriction for Schema Definitions. ======================================================================== Implement BackupSchemaProducer to generate a backup schema ======================================================================== Minor fix: remove a redundant log ======================================================================== Allow schema types to inherit from more than one parent ======================================================================== allow nested document properties to accept documents of subtype ======================================================================== Support polymorphism for Icing projection in Search and Get API ======================================================================== Add max_joined_child_per_parent into ResultSpec and change behavior ======================================================================== Support Icing schema type polymorphism for the search filter API ======================================================================== Verify that every child type's property set has included all compatible properties from parent types ======================================================================== Add individual type index latency ======================================================================== Build the iterator node for the propertyDefined() custom function ======================================================================== Advance all hits with same doc id from and merge sections once for the same bucket iter ======================================================================== Introduce DocHitInfoIteratorPropertyInSchema for property existence check ======================================================================== Add SchemaUtil::BuildTransitiveInheritanceGraph to build an inheritance map from schema ======================================================================== Introduce a lookup method for a property defined in a schema ======================================================================== Rollback of: Allow LanguageSegmenter::Iterators to declare AccessType. ======================================================================== Adds join info to QueryStatsProto ======================================================================== Bug:280698419 Bug:280698125 Bug:280698121 Bug:280697513 Bug:276349029 Bug:272145329 Bug:270102295 Bug:269295094 Bug:268680462 Bug:265304217 Bug:259744228 Bug:259743562 Bug:256022027 * goog/upstream-master: Update Icing from upstream. Update Icing from upstream. Change-Id: Ia9c5c88bf8e43122204acc8f5231fb8bf65019e3
2023-05-11Update Icing from upstream.Tim Barron
Descriptions: ======================================================================== Handle version changes in the schema store. ======================================================================== Bug: 280697513 Change-Id: I6da14f236cd8e4cf2af75c1fb2c115d1df512114
2023-05-11Update Icing from upstream.Tim Barron
Descriptions: ======================================================================== Modify the definition of propertyDefined: ======================================================================== Remove default args in SchemaStore::SetSchema and fix calls ======================================================================== Add allow_circular_schema_definitions flag ======================================================================== Onboard version detection to Icing ======================================================================== Add version util to help read/write version info ======================================================================== Add support for the overlay schema. ======================================================================== Allow cycles in schema-property-iterator ======================================================================== Add joinable properties into schema definition cycle restrictions. ======================================================================== Loosen circular references restriction for Schema Definitions. ======================================================================== Implement BackupSchemaProducer to generate a backup schema ======================================================================== Minor fix: remove a redundant log ======================================================================== Allow schema types to inherit from more than one parent ======================================================================== allow nested document properties to accept documents of subtype ======================================================================== Support polymorphism for Icing projection in Search and Get API ======================================================================== Add max_joined_child_per_parent into ResultSpec and change behavior ======================================================================== Support Icing schema type polymorphism for the search filter API ======================================================================== Verify that every child type's property set has included all compatible properties from parent types ======================================================================== Add individual type index latency ======================================================================== Build the iterator node for the propertyDefined() custom function ======================================================================== Advance all hits with same doc id from and merge sections once for the same bucket iter ======================================================================== Introduce DocHitInfoIteratorPropertyInSchema for property existence check ======================================================================== Add SchemaUtil::BuildTransitiveInheritanceGraph to build an inheritance map from schema ======================================================================== Introduce a lookup method for a property defined in a schema ======================================================================== Rollback of: Allow LanguageSegmenter::Iterators to declare AccessType. ======================================================================== Adds join info to QueryStatsProto ======================================================================== Bug:280698419 Bug:280698125 Bug:280698121 Bug:280697513 Bug:276349029 Bug:272145329 Bug:270102295 Bug:269295094 Bug:268680462 Bug:265304217 Bug:259744228 Bug:259743562 Bug:256022027 Change-Id: I54cd1d22121c314f8c238d2d49f0809165dc0ca3
2023-04-20Merge remote-tracking branch 'goog/upstream-master' into androidx-platform-devJiayu Hu
* goog/upstream-master: Update Icing from upstream. Descriptions: ======================================================================== Adds join info to QueryStatsProto ======================================================================== [Join][cleanup][1/x] Add Icing benchmark for qualified id join ======================================================================== [NumericSearch][benchmark][2/x] Modify run-benchmarks.py to onboard numeric search benchmark ======================================================================== [NumericSearch][benchmark][1/x] Add numeric search icing search engine benchmark ======================================================================== Switch Iterator access type used in SnippetRetriever back to kForwardIterator. ======================================================================== Fix `DocumentStore::OptimizeInto` by passing `options_.document_store_namespace_id_fingerprint()` ======================================================================== Add file size check when initializing mmap for existing metadata file. ======================================================================== Add compression level option to Icing and propagate down to PortableFileBackedProtoLog ======================================================================== Add portable_equals_proto namespace for EqualsProto in tokenized-document_test. ======================================================================== Refactor schema to check consistency of polymorphism by dependency map ======================================================================== Support the `this.propertyWeights()` function in advanced scoring ======================================================================== Randomly set optimize_rebuild_index_threshold in monkey test ======================================================================== Change benchmark batch query latency to average ======================================================================== Performance improvements to SnippetRetriever. ======================================================================== Bug: 270102295 Bug: 260625837 Bug: 269295094 Bug: 193244409 Bug: 269513147 Bug: 276345448 Bug: 259744228 Bug: 272145329 Bug: 276349029 Change-Id: Ibef24574826d6a2c7ea24caa13a073c10a490ceb
2023-04-19Update Icing from upstream.Jiayu Hu
Descriptions: ======================================================================== Adds join info to QueryStatsProto ======================================================================== [Join][cleanup][1/x] Add Icing benchmark for qualified id join ======================================================================== [NumericSearch][benchmark][2/x] Modify run-benchmarks.py to onboard numeric search benchmark ======================================================================== [NumericSearch][benchmark][1/x] Add numeric search icing search engine benchmark ======================================================================== Switch Iterator access type used in SnippetRetriever back to kForwardIterator. ======================================================================== Fix `DocumentStore::OptimizeInto` by passing `options_.document_store_namespace_id_fingerprint()` ======================================================================== Add file size check when initializing mmap for existing metadata file. ======================================================================== Add compression level option to Icing and propagate down to PortableFileBackedProtoLog ======================================================================== Add portable_equals_proto namespace for EqualsProto in tokenized-document_test. ======================================================================== Refactor schema to check consistency of polymorphism by dependency map ======================================================================== Support the `this.propertyWeights()` function in advanced scoring ======================================================================== Randomly set optimize_rebuild_index_threshold in monkey test ======================================================================== Change benchmark batch query latency to average ======================================================================== Performance improvements to SnippetRetriever. ======================================================================== Bug: 270102295 Bug: 260625837 Bug: 269295094 Bug: 193244409 Bug: 269513147 Bug: 276345448 Bug: 259744228 Bug: 272145329 Bug: 276349029 Change-Id: Ief1cb2c21f67aac420d6b79d9b946bbecdc39a5c
2023-03-31Merge remote-tracking branch 'goog/upstream-master' into androidx-platform-devTim Barron
* goog/upstream-master: Update Icing from upstream. Descriptions: ======================================================================== Make int64_t the CreateUsageReport timestamp type ======================================================================== Create ResultAdjustmentInfo and refactor ResultStateV2 ======================================================================== Wrap parent/child adjustment info to std::unique_ptr ======================================================================== Apply join child snippet ======================================================================== Apply join child projection ======================================================================== Add IntegerSectionIndexingHandlerTest ======================================================================== Delete Result Retriever this class is dead code. ======================================================================== Introduce a placeholder for the custom function hasPropertyDefined(member) ======================================================================== Fix libtextclassifier3::StatusOr ======================================================================== Performance improvements to SnippetRetriever. ======================================================================== Bug: 193244409 Bug: 256022027 Bug: 259744228 Bug: 268680462 Bug: 270102295 Bug: 271015984 Bug: 274627497 Change-Id: Ib0e26cb79fac7b5b6d1c26bd8b2910c517e9e2ab
2023-03-31Update Icing from upstream.Tim Barron
Descriptions: ======================================================================== Make int64_t the CreateUsageReport timestamp type ======================================================================== Create ResultAdjustmentInfo and refactor ResultStateV2 ======================================================================== Wrap parent/child adjustment info to std::unique_ptr ======================================================================== Apply join child snippet ======================================================================== Apply join child projection ======================================================================== Add IntegerSectionIndexingHandlerTest ======================================================================== Delete Result Retriever this class is dead code. ======================================================================== Introduce a placeholder for the custom function hasPropertyDefined(member) ======================================================================== Fix libtextclassifier3::StatusOr ======================================================================== Performance improvements to SnippetRetriever. ======================================================================== Bug: 193244409 Bug: 256022027 Bug: 259744228 Bug: 268680462 Bug: 270102295 Bug: 271015984 Bug: 274627497 Change-Id: I6bad316b28bb289fa8e3f5b0982d6aaa9e0d135f
2023-03-16Merge remote-tracking branch 'goog/upstream-master' into androidx-platform-devTim Barron
* goog/upstream-master: Update Icing from upstream. Descriptions: ======================================================================== Set overall index latency ======================================================================== Change GetUsageScores return type to optional. ======================================================================== Upstream fix to thread-safety annotation for cached_break_iterator_ ======================================================================== Bug: 259744228 Change-Id: I11212f22a3620289f28f5b12dbc6aab2d0b27fbc
2023-03-16Update Icing from upstream.Tim Barron
Descriptions: ======================================================================== Set overall index latency ======================================================================== Change GetUsageScores return type to optional. ======================================================================== Upstream fix to thread-safety annotation for cached_break_iterator_ ======================================================================== Bug: 259744228 Change-Id: Ia7a5032fd7655db773e311173f5735e4451b30c1
2023-03-14Merge remote-tracking branch 'goog/upstream-master' into androidx-platform-devTim Barron
* goog/upstream-master: Update Icing from upstream. Update Icing from upstream. Descriptions: ======================================================================== Cache an instance of UBreakIterator to reduce unnecessary creations. ======================================================================== Cap number of individual IntegerIndexStorages that IntegerIndex creates. ======================================================================== Change error in trimRightMostNode from Unimplemented to InvalidArgument. ======================================================================== Add detection for new language features of List Filters Query Language. ======================================================================== Add option to control threshold to rebuild index during optimize by flag ======================================================================== Add option to control use of namespace id to build urimapper by flag. ======================================================================== Enforce schema validation for joinable config. ======================================================================== Adopt bucket splitting for IntegerIndexStorage. ======================================================================== Implement bucket splitting function. ======================================================================== Add Icing initialization unit tests for QualifiedIdTypeJoinableIndex. ======================================================================== Add Icing schema change unit tests for QualifiedIdTypeJoinableIndex. ======================================================================== Add Icing optimization unit tests for QualifiedIdTypeJoinableIndex. ======================================================================== Integrate QualifiedIdTypeJoinableIndex into IcingSearchEngine. ======================================================================== Implement QualifiedIdJoinablePropertyIndexingHandler. ======================================================================== Change QualifiedIdTypeJoinableIndex to store raw qualified id string. ======================================================================== Pass info about unnormalized query terms through lexer/parser/visitor. ======================================================================== Integrate Advanced Query w/ Suggest, make ADVANCED_QUERY default parser. ====================================================================== Bug: 208654892 Bug: 263890397 Bug: 259743562 Bug: 272145329 Bug: 227356108 Change-Id: I44de5853bb6c55b42800ae34d8071016be6c87cd
2023-03-14Update Icing from upstream.Tim Barron
Descriptions: ======================================================================== Cache an instance of UBreakIterator to reduce unnecessary creations. ======================================================================== Cap number of individual IntegerIndexStorages that IntegerIndex creates. ======================================================================== Change error in trimRightMostNode from Unimplemented to InvalidArgument. ======================================================================== Add detection for new language features of List Filters Query Language. ======================================================================== Add option to control threshold to rebuild index during optimize by flag ======================================================================== Add option to control use of namespace id to build urimapper by flag. ======================================================================== Enforce schema validation for joinable config. ======================================================================== Adopt bucket splitting for IntegerIndexStorage. ======================================================================== Implement bucket splitting function. ======================================================================== Add Icing initialization unit tests for QualifiedIdTypeJoinableIndex. ======================================================================== Add Icing schema change unit tests for QualifiedIdTypeJoinableIndex. ======================================================================== Add Icing optimization unit tests for QualifiedIdTypeJoinableIndex. ======================================================================== Integrate QualifiedIdTypeJoinableIndex into IcingSearchEngine. ======================================================================== Implement QualifiedIdJoinablePropertyIndexingHandler. ======================================================================== Change QualifiedIdTypeJoinableIndex to store raw qualified id string. ======================================================================== Pass info about unnormalized query terms through lexer/parser/visitor. ======================================================================== Bug: 208654892 Bug: 263890397 Bug: 259743562 Bug: 272145329 Bug: 227356108 Change-Id: I438a390ddda5673cf2b5781af502f2b7cfeaee74
2023-03-09Update Icing from upstream.Tim Barron
Descriptions: ====================================================================== Integrate Advanced Query w/ Suggest, make ADVANCED_QUERY default parser. ====================================================================== BUG: 208654892 Change-Id: I53889482e844648ec65565f9a7f7c8faa89baa7c
2023-03-07Merge remote-tracking branch 'goog/upstream-master' into androidx-platform-devTim Barron
* goog/upstream-master: Update Icing from upstream. Descriptions: ====================================================================== Refactor IndexProcessor ====================================================================== Rename Joinable Cache as Joinable Index ====================================================================== Implement Optimize and Clear for QualifiedIdTypeJoinableCache ====================================================================== Add JoinablePropertyMetadata reverse lookup ====================================================================== Allow code creating LanguageSegmenter::Iterators to declare AccessType ====================================================================== Further codifies the escape behavior in the parser test ====================================================================== Bug: 263890397 Bug: 268680462 Bug: 270102295 Change-Id: I8e45ef63c202aa83c0a4cda9c59b31d46daa4056
2023-03-06Update Icing from upstream.Tim Barron
Descriptions: ====================================================================== Refactor IndexProcessor ====================================================================== Rename Joinable Cache as Joinable Index ====================================================================== Implement Optimize and Clear for QualifiedIdTypeJoinableCache ====================================================================== Add JoinablePropertyMetadata reverse lookup ====================================================================== Allow code creating LanguageSegmenter::Iterators to declare AccessType ====================================================================== Further codifies the escape behavior in the parser test ====================================================================== Bug: 263890397 Bug: 268680462 Bug: 270102295 Change-Id: I3233733b40e985e11c4a6d75c1528cd6a72c1173
2023-03-03Merge remote-tracking branch 'goog/upstream-master' into androidx-platform-devTerry Wang
* goog/upstream-master: Update Icing from upstream. Descriptions: ====================================================================== Modify the lexer to consider internal minus characters to be a part of TEXT segments (as hyphens). ====================================================================== Modify the query language to support an individual prefix operator '*'. ====================================================================== Fix the fingerprint issue for document key mapper in document store ====================================================================== Resolve issues around the handling of negative numbers. ====================================================================== Fix doc join info compiler warning/error ====================================================================== Codifies how colons are tokenized by the lexer Bug: 208654892 Bug: 268680462 Change-Id: I08d95f9058d5642e66ef4805a5599de41d2bbcfc
2023-03-03Update Icing from upstream.Terry Wang
Descriptions: ====================================================================== Modify the lexer to consider internal minus characters to be a part of TEXT segments (as hyphens). ====================================================================== Modify the query language to support an individual prefix operator '*'. ====================================================================== Fix the fingerprint issue for document key mapper in document store ====================================================================== Resolve issues around the handling of negative numbers. ====================================================================== Fix doc join info compiler warning/error ====================================================================== Codifies how colons are tokenized by the lexer Bug: 208654892 Bug: 268680462 Change-Id: I67169b2e656eeafb6500b5b03a6c46c3cc9c3d66
2023-03-01Merge remote-tracking branch 'goog/upstream-master' into androidx-platform-devTerry Wang
* goog/upstream-master: Update Icing from upstream. Descriptions: ====================================================================== Add PropertyUtil for all property name/path related operations ====================================================================== [JoinableCache][2.0/x] Create SchemaPropertyIterator ====================================================================== [JoinableCache][2.1/x] Handle nested indexable flag ====================================================================== [JoinableCache][2.2/x] Add schema cycle dependency detection for SchemaPropertyIterator ====================================================================== [JoinableCache][3.0/x] Refactor SectionManager ====================================================================== [JoinableCache][3.1/x] Add unit tests for SectionManager::Builder and SchemaTypeManager ====================================================================== [NumericSearch][Storage][12/x] Implement Edit and GetIterator for IntegerIndex ====================================================================== [NumericSearch][Storage][13.0/x] Rename numeric-index_test as integer-index_test ====================================================================== [NumericSearch][Storage][13.1/x] Add IntegerIndexTest ====================================================================== Support the "len", "sum" and "avg" functions in advanced scoring. ====================================================================== Support the "this.childrenScores()" function to allow expressing children scores of joins in advanced scoring. ====================================================================== Create an integration test for Join with advanced scoring ====================================================================== Rename the word "children" to "args" for function related ScoreExpression ====================================================================== Improve IndexBlock by PRead/PWrite instead of repeating mmap/msync/unmap ====================================================================== Refactor QueryVisitor to prepare for support for function calls. ====================================================================== Add support for function calls. ====================================================================== Fix breakage in score-and-rank_benchmark. ====================================================================== [NumericSearch][Storage][adhoc][ez] Fix comment for IntegerIndex ====================================================================== [NumericSearch][Storage][14/x] Create first IntegerIndexStorage benchmark ====================================================================== Rename Icing schema related terminology to prepare for polymorphism support ====================================================================== [JoinableCache][4.0/x] Move common methods from SectionManager to PropertyUtil ====================================================================== [JoinableCache][4.1/x] Retire GetSectionContent ====================================================================== [JoinableCache][4.2/x] Polish SectionManagerTest ====================================================================== Modify QueryVisitor to do: ====================================================================== [NumericSearch][Storage][15/x] Implement TransferIndex for IntegerIndexStorage ====================================================================== [NumericSearch][Storage][16/x] Implement Optimize and last added document id for IntegerIndex ====================================================================== [NumericSearch][rollout][1/x] Include indexable int64 into SchemaDelta and backward compatibility ====================================================================== Add backwards compatibility test for Icing schema storage migration. ====================================================================== Implement trim the right-most node from the doc-hit-info-iterator. ====================================================================== Add TrimmedNode structure into doc-hit-info-iterator. ====================================================================== [JoinableCache][5/x] Implement JoinableProperty and JoinablePropertyManager ====================================================================== [JoinableCache][6/x] Add JoinablePropertyManager into SchemaTypeManager ====================================================================== [JoinableCache][7/x] Implement ExtractJoinableProperties ====================================================================== [JoinableCache][8/x] Create class QualifiedIdTypeJoinableCache ====================================================================== [JoinableCache][9/x] Implement factory method for QualifiedIdTypeJoinableCache ====================================================================== [JoinableCache][10/x] Implement Get and Put for QualifiedIdTypeJoinableCache ====================================================================== [JoinableCache][11/x] Add unit tests for QualifiedIdTypeJoinableCache ====================================================================== Modify DocHitInfoIteratorSectionRestrict to allow multi-property restricts ====================================================================== Fix the definition of LiteIndex::WantsMerge. ====================================================================== [NumericSearch][rollout][2.0/x] Rollout persistent IntegerIndex ====================================================================== [NumericSearch][rollout][2.1/x] Add more tests for integer index restoration and optimization ====================================================================== [JoinableCache][adhoc][ez] Remove qualified id type joinable cache size info from document storage info ====================================================================== Integrate trim right node into suggestion processor. Bug: 208654892 Bug: 228240987 Bug: 249829533 Bug: 256081830 Bug: 259744228 Bug: 261474063 Bug: 263890397 Bug: 266103594 Bug: 268738297 Bug: 269295094 Change-Id: Id4b23bf397ca189c5050bfa6c5fd64e89138e321
2023-03-01Update Icing from upstream.Terry Wang
Descriptions: ====================================================================== Add PropertyUtil for all property name/path related operations ====================================================================== [JoinableCache][2.0/x] Create SchemaPropertyIterator ====================================================================== [JoinableCache][2.1/x] Handle nested indexable flag ====================================================================== [JoinableCache][2.2/x] Add schema cycle dependency detection for SchemaPropertyIterator ====================================================================== [JoinableCache][3.0/x] Refactor SectionManager ====================================================================== [JoinableCache][3.1/x] Add unit tests for SectionManager::Builder and SchemaTypeManager ====================================================================== [NumericSearch][Storage][12/x] Implement Edit and GetIterator for IntegerIndex ====================================================================== [NumericSearch][Storage][13.0/x] Rename numeric-index_test as integer-index_test ====================================================================== [NumericSearch][Storage][13.1/x] Add IntegerIndexTest ====================================================================== Support the "len", "sum" and "avg" functions in advanced scoring. ====================================================================== Support the "this.childrenScores()" function to allow expressing children scores of joins in advanced scoring. ====================================================================== Create an integration test for Join with advanced scoring ====================================================================== Rename the word "children" to "args" for function related ScoreExpression ====================================================================== Improve IndexBlock by PRead/PWrite instead of repeating mmap/msync/unmap ====================================================================== Refactor QueryVisitor to prepare for support for function calls. ====================================================================== Add support for function calls. ====================================================================== Fix breakage in score-and-rank_benchmark. ====================================================================== [NumericSearch][Storage][adhoc][ez] Fix comment for IntegerIndex ====================================================================== [NumericSearch][Storage][14/x] Create first IntegerIndexStorage benchmark ====================================================================== Rename Icing schema related terminology to prepare for polymorphism support ====================================================================== [JoinableCache][4.0/x] Move common methods from SectionManager to PropertyUtil ====================================================================== [JoinableCache][4.1/x] Retire GetSectionContent ====================================================================== [JoinableCache][4.2/x] Polish SectionManagerTest ====================================================================== Modify QueryVisitor to do: ====================================================================== [NumericSearch][Storage][15/x] Implement TransferIndex for IntegerIndexStorage ====================================================================== [NumericSearch][Storage][16/x] Implement Optimize and last added document id for IntegerIndex ====================================================================== [NumericSearch][rollout][1/x] Include indexable int64 into SchemaDelta and backward compatibility ====================================================================== Add backwards compatibility test for Icing schema storage migration. ====================================================================== Implement trim the right-most node from the doc-hit-info-iterator. ====================================================================== Add TrimmedNode structure into doc-hit-info-iterator. ====================================================================== [JoinableCache][5/x] Implement JoinableProperty and JoinablePropertyManager ====================================================================== [JoinableCache][6/x] Add JoinablePropertyManager into SchemaTypeManager ====================================================================== [JoinableCache][7/x] Implement ExtractJoinableProperties ====================================================================== [JoinableCache][8/x] Create class QualifiedIdTypeJoinableCache ====================================================================== [JoinableCache][9/x] Implement factory method for QualifiedIdTypeJoinableCache ====================================================================== [JoinableCache][10/x] Implement Get and Put for QualifiedIdTypeJoinableCache ====================================================================== [JoinableCache][11/x] Add unit tests for QualifiedIdTypeJoinableCache ====================================================================== Modify DocHitInfoIteratorSectionRestrict to allow multi-property restricts ====================================================================== Fix the definition of LiteIndex::WantsMerge. ====================================================================== [NumericSearch][rollout][2.0/x] Rollout persistent IntegerIndex ====================================================================== [NumericSearch][rollout][2.1/x] Add more tests for integer index restoration and optimization ====================================================================== [JoinableCache][adhoc][ez] Remove qualified id type joinable cache size info from document storage info ====================================================================== Integrate trim right node into suggestion processor. Bug: 208654892 Bug: 228240987 Bug: 249829533 Bug: 256081830 Bug: 259744228 Bug: 261474063 Bug: 263890397 Bug: 266103594 Bug: 268738297 Bug: 269295094 Change-Id: I5f1b3f3ed0b5d6933dc8c2ab3279904f7706b23e
2023-02-08Merge remote-tracking branch 'goog/upstream-master' into androidx-platform-devAlex Saveliev
* goog/upstream-master: Update icing from upstream ====================================================================== Adds a proto change for the delete propagation option ====================================================================== [ez] Change version to magic for PersistentHashMap ====================================================================== [iOS][testing][nitro] Disabling ICU language segmenter_test. ====================================================================== 1. Add support for segmentation in the QueryVisitor. ====================================================================== Support the new double list type in ScoreExpression ====================================================================== Pass JoinChildrenFetcher from IcingSearchEngine all the way down to ScoringVisitor ====================================================================== Refactor the logic of Icing Joins so that nested search and scoring will be performed before the parent ====================================================================== Add lite-index thread-safety tests. ====================================================================== Split IcingSearchEngineTest into separate tests to cover specific apis: ====================================================================== Fix index tests TearDown method. ====================================================================== Improve query concurrency by providing a finer-grained lock around the LiteIndex. ====================================================================== Fix Icing normalization bug ====================================================================== [ez] Fix integer overflow error for IntegerIndexStorage ====================================================================== [NumericSearch][Storage][11/x] Implement Reset and destructor for IntegerIndex ====================================================================== [NumericSearch][Storage][10/x] Add class IntegerIndex ====================================================================== Refactor NumericIndex based on PersistentStorage ====================================================================== Refactor IntegerIndexStorage based on PersistentStorage ====================================================================== Add "working_path" into PersistentStorage ====================================================================== Refactor PersistentHashMap based on PersistentStorage ====================================================================== Create virtual class PersistentStorage for refactoring ====================================================================== Avoids returning reference to local temporary object. ====================================================================== LSC: Clean up references to the legacy protobuf compat library ====================================================================== Fix time complexity regression for snippet retriever Bug: 256022027 Bug: 193919210 Bug: 266132035 Bug: 208654892 Bug: 261474063 Bug: 266103594 Bug: 146008613 Bug: 253182853 Bug: 266204868 Bug: 249829533 Bug: 266665956 Bug: 265258364 Change-Id: Icff89dee8ed04c09597c08c22e1d180b71c7d396
2023-02-07Update icing from upstreamAlex Saveliev
====================================================================== Adds a proto change for the delete propagation option ====================================================================== [ez] Change version to magic for PersistentHashMap ====================================================================== [iOS][testing][nitro] Disabling ICU language segmenter_test. ====================================================================== 1. Add support for segmentation in the QueryVisitor. ====================================================================== Support the new double list type in ScoreExpression ====================================================================== Pass JoinChildrenFetcher from IcingSearchEngine all the way down to ScoringVisitor ====================================================================== Refactor the logic of Icing Joins so that nested search and scoring will be performed before the parent ====================================================================== Add lite-index thread-safety tests. ====================================================================== Split IcingSearchEngineTest into separate tests to cover specific apis: ====================================================================== Fix index tests TearDown method. ====================================================================== Improve query concurrency by providing a finer-grained lock around the LiteIndex. ====================================================================== Fix Icing normalization bug ====================================================================== [ez] Fix integer overflow error for IntegerIndexStorage ====================================================================== [NumericSearch][Storage][11/x] Implement Reset and destructor for IntegerIndex ====================================================================== [NumericSearch][Storage][10/x] Add class IntegerIndex ====================================================================== Refactor NumericIndex based on PersistentStorage ====================================================================== Refactor IntegerIndexStorage based on PersistentStorage ====================================================================== Add "working_path" into PersistentStorage ====================================================================== Refactor PersistentHashMap based on PersistentStorage ====================================================================== Create virtual class PersistentStorage for refactoring ====================================================================== Avoids returning reference to local temporary object. ====================================================================== LSC: Clean up references to the legacy protobuf compat library ====================================================================== Fix time complexity regression for snippet retriever Bug: 256022027 Bug: 193919210 Bug: 266132035 Bug: 208654892 Bug: 261474063 Bug: 266103594 Bug: 146008613 Bug: 253182853 Bug: 266204868 Bug: 249829533 Bug: 266665956 Bug: 265258364 Change-Id: Ib2398c5097b6a2a57900e2ad4e3737502aa13820
2023-01-23Merge remote-tracking branch 'goog/upstream-master' into androidx-platform-devTerry Wang
* goog/upstream-master: Update Icing from upstream. Descriptions: ====================================================================== Fix time complexity regression for snippet retriever ====================================================================== Add support for QueryTerms SectionRestrict Map in Advanced Query. ====================================================================== [NumericSearch][Storage][9/x] Create unit tests for AddKeys and GetIterator ====================================================================== [NumericSearch][Storage][8/x] Add options ====================================================================== Fix the stack overflow issue for icing advanced scoring/query parsers/visitors by limiting the number of tokens allowed ====================================================================== Improves the has joinspec check ====================================================================== Fix the bug of DocHitInfoIteratorTermMain found by monkey test ====================================================================== Rename tests defined in DestructibleDirectoryTest to include "directory" instead of "file". ====================================================================== Support projection in icing monkey test ====================================================================== [NumericSearch][Storage][7/x] Implement iterator for query ====================================================================== [NumericSearch][Storage][6/x] Implement AddKeys ====================================================================== Add support for relevance scoring by enabling term_frequency retrieval in DocHitInfoIterators and by populating the QueryTermIteratorsMap. ====================================================================== [JoinableCache][1/x] Compute SchemaDelta for joinable properties ====================================================================== [NumericSearch][Storage][5/x] PersistToDisk ====================================================================== [NumericSearch][Storage][4.1/x] Create unit tests for initialize ====================================================================== [NumericSearch][Storage][4.0/x] Implement Create and Initialize functions ====================================================================== Rename PostingListUsedSerializer related classes Bug: 193244409 Bug: 208654892 Bug: 246984163 Bug: 249829533 Bug: 256022027 Bug: 265258364 Bug: 265834832 Change-Id: I0b74286176356fb88602f775a431b5a489ff4045
2023-01-23Update Icing from upstream.Terry Wang
Descriptions: ====================================================================== Fix time complexity regression for snippet retriever ====================================================================== Add support for QueryTerms SectionRestrict Map in Advanced Query. ====================================================================== [NumericSearch][Storage][9/x] Create unit tests for AddKeys and GetIterator ====================================================================== [NumericSearch][Storage][8/x] Add options ====================================================================== Fix the stack overflow issue for icing advanced scoring/query parsers/visitors by limiting the number of tokens allowed ====================================================================== Improves the has joinspec check ====================================================================== Fix the bug of DocHitInfoIteratorTermMain found by monkey test ====================================================================== Rename tests defined in DestructibleDirectoryTest to include "directory" instead of "file". ====================================================================== Support projection in icing monkey test ====================================================================== [NumericSearch][Storage][7/x] Implement iterator for query ====================================================================== [NumericSearch][Storage][6/x] Implement AddKeys ====================================================================== Add support for relevance scoring by enabling term_frequency retrieval in DocHitInfoIterators and by populating the QueryTermIteratorsMap. ====================================================================== [JoinableCache][1/x] Compute SchemaDelta for joinable properties ====================================================================== [NumericSearch][Storage][5/x] PersistToDisk ====================================================================== [NumericSearch][Storage][4.1/x] Create unit tests for initialize ====================================================================== [NumericSearch][Storage][4.0/x] Implement Create and Initialize functions ====================================================================== Rename PostingListUsedSerializer related classes Bug: 193244409 Bug: 208654892 Bug: 246984163 Bug: 249829533 Bug: 256022027 Bug: 265258364 Bug: 265834832 Change-Id: Ibc7011a793ef5ed09eace0fb05d168adf5b4faca
2023-01-09Merge remote-tracking branch 'aosp/upstream-master' into androidx-mainAlex Saveliev
Descriptions: ====================================================================== Apply aggregation score to join result and create JoinProcessorTest ====================================================================== Create AggregationScorerTest ====================================================================== Add an EqualsTermMatchInfo matcher to simplify tests that interact with this struct. ====================================================================== Refactor the QueryVisitor to return QueryResults. ====================================================================== Simplify ScoreExpression to ConstantScoreExpression if its evaluation does not depend on a document ====================================================================== Use rvalue and std::move for DocumentStore methods ====================================================================== Downgrade log messages for NOT_FOUND errors in Delete calls. ====================================================================== [NumericSearch][Storage][3.1/x] Add metadata file declaration (Crcs, Info) for IntegerIndexStorage ====================================================================== [NumericSearch][Storage][3.0/x] Declare class IntegerIndexStorage ====================================================================== Cleanup Import from //tools/build_defs/objc ====================================================================== Cleanup LSC: Remove `portable_proto_library()` targets ====================================================================== Bug: 256022027 Bug: 208654892 Bug: 261474063 Bug: 246628980 Bug: 249829533 Bug: 263373214 Change-Id: Ic3238f17efa72dd1c6dbc559d32c11f475cebf88
2023-01-06Update icing from upstreamAlex Saveliev
Descriptions: ====================================================================== Apply aggregation score to join result and create JoinProcessorTest ====================================================================== Create AggregationScorerTest ====================================================================== Add an EqualsTermMatchInfo matcher to simplify tests that interact with this struct. ====================================================================== Refactor the QueryVisitor to return QueryResults. ====================================================================== Simplify ScoreExpression to ConstantScoreExpression if its evaluation does not depend on a document ====================================================================== Use rvalue and std::move for DocumentStore methods ====================================================================== Downgrade log messages for NOT_FOUND errors in Delete calls. ====================================================================== [NumericSearch][Storage][3.1/x] Add metadata file declaration (Crcs, Info) for IntegerIndexStorage ====================================================================== [NumericSearch][Storage][3.0/x] Declare class IntegerIndexStorage ====================================================================== Cleanup Import from //tools/build_defs/objc ====================================================================== Cleanup LSC: Remove `portable_proto_library()` targets ====================================================================== Bug: 256022027 Bug: 208654892 Bug: 261474063 Bug: 246628980 Bug: 249829533 Bug: 263373214 Change-Id: Ib5ca7efffd575254e14a14d54a45f4efd1e8d6c9
2022-12-20Merge remote-tracking branch 'aosp/upstream-master' into androidx-mainTim Barron
* aosp/upstream-master: Sync from upstream. Descriptions: ====================================================================== Implement simple feature checking for implicit new features in IcingLib [2/2]. ====================================================================== Bug: 262512396 Change-Id: Idbe67cf51a67b0650d9c194e6a33dfc2dd1f440f
generated by cgit v1.2.3 (git 2.25.1) at 2024-07-18 04:40:09 +0000