aboutsummaryrefslogtreecommitdiff
path: root/icing/index/iterator/doc-hit-info-iterator-section-restrict.cc
AgeCommit message (Collapse)Author
2023-11-30Update Icing from upstream.Jiayu Hu
Descriptions: ======================================================================== [Icing][version 3] Bump kVersion to 3 ======================================================================== Make lite index magic dependent on `IcingSearchEngineOptions::build_property_existence_metadata_hits` ======================================================================== Add a flag in IcingSearchEngineOptions to control whether to build property existence metadata hits ======================================================================== Support `hasProperty(property_path)` in the advanced query language ======================================================================== Add PropertyExistenceIndexingHandler to index property existence metadata hit ======================================================================== [JoinIndex Improvement][11/x] Add IcingSearchEngine initialization unit test for switching join index ======================================================================== [JoinIndex Improvement][10/x] Change/Add IcingSearchEngine unit tests ======================================================================== [JoinIndex Improvement][9/x] Integrate QualifiedIdJoinIndexImplV2 with IcingSearchEngine ======================================================================== [JoinIndex Improvement][8/x] Integrate QualifiedIdJoinIndexImplV2 with JoinProcessor ======================================================================== [JoinIndex Improvement][8/x] Integrate QualifiedIdJoinIndexImplV2 with QualifiedIdJoinIndexingHandler ======================================================================== [JoinIndex Improvement][7/x] Create QualifiedIdJoinIndex interface ======================================================================== [JoinIndex Improvement][6.1/x] Unit test (Optimize) ======================================================================== [JoinIndex Improvement][6.0/x] Unit test (General, Put, GetIterator) ======================================================================== [JoinIndex Improvement][5.3/x] Implement Optimize ======================================================================== Remove accents from Greek letters in normalizer ======================================================================== Make arm emulator tests build-only. ======================================================================== [JoinIndex Improvement][5.2/x] Implement GetIterator ======================================================================== [JoinIndex Improvement][5.1/x] Implement Put ======================================================================== [JoinIndex Improvement][5.0/x] Branch QualifiedIdJoinIndex to QualifiedIdJoinIndexImplV2 ======================================================================== [JoinIndex Improvement][4/x] Implement PostingListJoinDataAccessor ======================================================================== [JoinIndex Improvement][3/x] Implement PostingListJoinDataSerializer and DocumentIdToJoinInfo data type ======================================================================== [JoinIndex Improvement][2/x] Create NamespaceFingerprintIdentifier ======================================================================== [JoinIndex Improvement][1/x] Implement namespace_id_old_to_new in Compaction ======================================================================== Update test to also handle ICU 74 segmentation rules. ======================================================================== [Icing][Expand QueryStats][3/x] Add new fields into QueryStats (1) ======================================================================== [Icing][Expand QueryStats][2/x] Refactor QueryStatsProto ======================================================================== [Icing][Expand QueryStats][1/x] Publish DocHitInfoIterator CallStats ======================================================================== Add additional property filter tests ======================================================================== Deprecate hit_intersect_section_ids_mask in DocHitInfoIterator ======================================================================== Change default requires_full_emulation to False for portable_cc_test (third_party/icing/testing) ======================================================================== Cleanup Set requires_full_emulation to True for selective tests ======================================================================== Fix monkey test failures ======================================================================== Complete monkey test logic to change schema during monkey test runtime ======================================================================== Refactor monkey test to prepare for schema update ======================================================================== Fix the schema bug found by monkey test with seed 2551429844 ======================================================================== Move set query stats to the very top of InternalSearch() ======================================================================== Apply section restriction only on leaf nodes ======================================================================== [6/n] Fix callsites in Icing that forgot to check libtextclassifier3::Status (Advanced query parser) ======================================================================== [5/n] Fix callsites in Icing that forgot to check libtextclassifier3::Status (PersistentHashMap) ======================================================================== [4/n] Fix callsites in Icing that forgot to check libtextclassifier3::Status (PostingListIntegerIndexSerializer) ======================================================================== [3/n] Fix callsites in Icing that forgot to check libtextclassifier3::Status (PostingListHitSerializer) ======================================================================== [2/n] Fix callsites in Icing that forgot to check libtextclassifier3::Status (Posting list storage) ======================================================================== [1/n] Fix callsites in Icing that forgot to check libtextclassifier3::Status (Non-functional changes) ======================================================================== Decouple section restriction data from iterators ======================================================================== Fix the crash when a schema type gets more indexable properties than allowed ======================================================================== Add a checker to verify the property data type matches the schema. ======================================================================== Change global std::string in i18n-utils to constexpr std::string_view. ======================================================================== Adjust LiteIndex sort at indexing check conditions. ======================================================================== Bug: 305098009 Bug: 307508735 Bug: 291130542 Bug: 275121148 Bug: 303239901 Bug: 301116242 Bug: 299321977 Bug: 300135897 Bug: 297549761 Bug: 309826655 Bug: 296349369 Bug: 302192690 Bug: 302609704 Bug: 301566713 NO_IFTTT="False Alarm: The path is only valid in G3. kVersion is changed to 3, and schema is compatible with version 1." Change-Id: I8c4c3cd9b93e5240bd774f0a3d6d812f7a9ec198
2023-08-07Update Icing from upstream.Yogesh Singh
Descriptions: ======================================================================== Add join test for SetSchemaNewIndexedDocumentPropertyTriggersIndexRestorationAndReturnsOk ======================================================================== Support different sets of properties to search for different schema types. ======================================================================== Do not cache the section mask for a schema type if no property filters apply while processing the type property filters. ======================================================================== Bug: 288310393 Bug: 291019114 Change-Id: Icb867b40a603c0d63e2d96a0e2ec641e424aad7e
2023-06-02Update Icing from upstream.Tim Barron
Descriptions: ======================================================================== Eliminate unnecessary GetCurrentTime syscalls in various hotspots. ======================================================================== Fix MSAN failure in SchemaStore::Header::Write. ======================================================================== Bug: 282308801 Change-Id: Ifb11f890485329e6b0bc1a9cbf2bf046e39b7854
2023-03-01Update Icing from upstream.Terry Wang
Descriptions: ====================================================================== Add PropertyUtil for all property name/path related operations ====================================================================== [JoinableCache][2.0/x] Create SchemaPropertyIterator ====================================================================== [JoinableCache][2.1/x] Handle nested indexable flag ====================================================================== [JoinableCache][2.2/x] Add schema cycle dependency detection for SchemaPropertyIterator ====================================================================== [JoinableCache][3.0/x] Refactor SectionManager ====================================================================== [JoinableCache][3.1/x] Add unit tests for SectionManager::Builder and SchemaTypeManager ====================================================================== [NumericSearch][Storage][12/x] Implement Edit and GetIterator for IntegerIndex ====================================================================== [NumericSearch][Storage][13.0/x] Rename numeric-index_test as integer-index_test ====================================================================== [NumericSearch][Storage][13.1/x] Add IntegerIndexTest ====================================================================== Support the "len", "sum" and "avg" functions in advanced scoring. ====================================================================== Support the "this.childrenScores()" function to allow expressing children scores of joins in advanced scoring. ====================================================================== Create an integration test for Join with advanced scoring ====================================================================== Rename the word "children" to "args" for function related ScoreExpression ====================================================================== Improve IndexBlock by PRead/PWrite instead of repeating mmap/msync/unmap ====================================================================== Refactor QueryVisitor to prepare for support for function calls. ====================================================================== Add support for function calls. ====================================================================== Fix breakage in score-and-rank_benchmark. ====================================================================== [NumericSearch][Storage][adhoc][ez] Fix comment for IntegerIndex ====================================================================== [NumericSearch][Storage][14/x] Create first IntegerIndexStorage benchmark ====================================================================== Rename Icing schema related terminology to prepare for polymorphism support ====================================================================== [JoinableCache][4.0/x] Move common methods from SectionManager to PropertyUtil ====================================================================== [JoinableCache][4.1/x] Retire GetSectionContent ====================================================================== [JoinableCache][4.2/x] Polish SectionManagerTest ====================================================================== Modify QueryVisitor to do: ====================================================================== [NumericSearch][Storage][15/x] Implement TransferIndex for IntegerIndexStorage ====================================================================== [NumericSearch][Storage][16/x] Implement Optimize and last added document id for IntegerIndex ====================================================================== [NumericSearch][rollout][1/x] Include indexable int64 into SchemaDelta and backward compatibility ====================================================================== Add backwards compatibility test for Icing schema storage migration. ====================================================================== Implement trim the right-most node from the doc-hit-info-iterator. ====================================================================== Add TrimmedNode structure into doc-hit-info-iterator. ====================================================================== [JoinableCache][5/x] Implement JoinableProperty and JoinablePropertyManager ====================================================================== [JoinableCache][6/x] Add JoinablePropertyManager into SchemaTypeManager ====================================================================== [JoinableCache][7/x] Implement ExtractJoinableProperties ====================================================================== [JoinableCache][8/x] Create class QualifiedIdTypeJoinableCache ====================================================================== [JoinableCache][9/x] Implement factory method for QualifiedIdTypeJoinableCache ====================================================================== [JoinableCache][10/x] Implement Get and Put for QualifiedIdTypeJoinableCache ====================================================================== [JoinableCache][11/x] Add unit tests for QualifiedIdTypeJoinableCache ====================================================================== Modify DocHitInfoIteratorSectionRestrict to allow multi-property restricts ====================================================================== Fix the definition of LiteIndex::WantsMerge. ====================================================================== [NumericSearch][rollout][2.0/x] Rollout persistent IntegerIndex ====================================================================== [NumericSearch][rollout][2.1/x] Add more tests for integer index restoration and optimization ====================================================================== [JoinableCache][adhoc][ez] Remove qualified id type joinable cache size info from document storage info ====================================================================== Integrate trim right node into suggestion processor. Bug: 208654892 Bug: 228240987 Bug: 249829533 Bug: 256081830 Bug: 259744228 Bug: 261474063 Bug: 263890397 Bug: 266103594 Bug: 268738297 Bug: 269295094 Change-Id: I5f1b3f3ed0b5d6933dc8c2ab3279904f7706b23e
2022-09-08Sync from upstream.Grace Zhao
Descriptions: ====================================================================== [FileBackedVector Consolidation][4/x] Fix potential PWrite bug in GrowIfNecessary ====================================================================== [FileBackedVector Consolidation][5/x] Create benchmark for FileBackedVector ====================================================================== [FileBackedVector Consolidation][6/x] Avoid calling GetFileSize in GrowIfNecessary ====================================================================== [PersistentHashMap][3.3/x] Implement Delete ====================================================================== Fix the PopulateMatchedTermsStats bug ====================================================================== Add JNI latency for query latency stats breakdown. ====================================================================== [ResultStateManager] Thread safety test1 ====================================================================== [ResultStateManager][2/x] Thread safety test2 ====================================================================== Add native lock contention latency for measuring query latency ====================================================================== Fix implementation of HasMember operator in ANTLR-based list-filter prototype. ====================================================================== Fix improper uses of std::string_view ====================================================================== Extend the scale of Icing ====================================================================== Decouple the term frequency array from DocHitInfo ====================================================================== Disable hit_term_frequency for non-relevance queries ====================================================================== [ResultStateManager][3/x] Thread safety test3 ====================================================================== [PersistentHashMap][4/x] Implement iterator ======================================================================= Fix the lite index compaction bug ======================================================================= Change-Id: I0edad67affed97af107e2d7cd73770e0268c0903
2022-06-15Sync from upstream.Jiayu Hu
Descriptions: ====================================================================== Export Icing logging tag to JNI ====================================================================== Update export_to_aosp.sh to change icing log tag to "AppSearchIcing" ====================================================================== Improve the logic of NamespaceChecker. ====================================================================== Step 4.1: Use ScoredDocumentHitsRanker in ResultStateV2 ====================================================================== Step 4.0: Create ScoredDocumentHitsRanker interface and PriorityQueueScoredDocumentHitsRanker ====================================================================== Refactor KeyMapper ====================================================================== Change Icing DEFAULT_LOGGING_LEVEL to INFO ====================================================================== Step 3.4: Create ResultRetrieverV2GroupResultLimiterTest (copied from ResultStateTest) ====================================================================== Add IcingDynamicTrie::IsBranchingTerm to check if a term is branching. ====================================================================== Fix IcingDynamicTrie::Delete bug ====================================================================== Step 3.3: Create ResultRetrieverV2ProjectionTest (copied from ResultRetrieverTest) ====================================================================== Step 3.2: Create ResultRetrieverV2SnippetTest (copied from ResultRetrieverTest) ====================================================================== Step 3.1: Create ResultRetrieverV2Test (copied from ResultRetrieverTest) ====================================================================== Enable legacy multidex ====================================================================== Fix NPE caused by improper handling of return value of GetFileSize. ====================================================================== Step 3.0: Create ResultRetrieverV2 (copied from ResultRetriever) ====================================================================== Step 2: Create PageResult (copied from PageResultState) ====================================================================== Change AppSearch hawkeye testing app multidex to legacy. ====================================================================== Fix icing-search-engine_benchmark bug ====================================================================== (Small fix for step 1) Fix unit test stack memory error ====================================================================== Step 1: Create ResultStateV2 and ResultStateV2Test (copied from ResultState, ResultStateTest) ====================================================================== Bug: 146903474 Bug: 152934343 Bug: 193919210 Bug: 231368517 Bug: 232273174 Bug: 233470404 Bug: 233657885 Change-Id: Iae5461f8650c2bd58f683128fdd243d90403bc06
2021-08-13Merge androidx-platform-dev/external/icing upstream-master into upstream-masterAlexander Dorokhine
Change-Id: Id39bea14b3a3d378d722dc552e4f3bd4249f3f94
2021-03-02Sync from upstream.Cassie Wang
Descriptions: ========== Add last optimized time to GetOptimizeInfo. ========== Update the implementation of snippeting to return property paths with value indices and remove the values_index field. ========== Create builders for SchemaProto, SchemaTypeConfigProto and PropertyConfigProto. ========== Rename some protos with the rules: - Remove "Native" prefix - Add "Proto" suffix for consistency with other protos ========== Upgrade your minimum iOS version to 11.4. ========== Fix PersistToDisk definitions to ensure that they properly call datasync. This change is meant to address the first part of ptd doc - that certain functions that claim to persist data don't actually explicitly flush. ========== Change function call from has_field() to field() ========== Add IcingStorageInfo. ========== Add IndexStorageStats. ========== Add SchemaStoreStorageStats. ========== Add DocumentStoreStorageStats. ========== Implement OptimizeStats. ========== Remove the max number of results per query limit (1000) and replace it with a more flexible way to limit memory use by the result-state-manager. ========== Add a test case to ensure we don't add UsageStore's checksum in DocumentStore's ComputeChecksum. ========== Account for UsageStore in GetDiskUsage. ========== Ensure that SchemaStore properly handles function calls when the schema isn't set. ========== Remove jlpl_strict_deps feature from package declarations. ========== Qualifies std::string in 3p directories ========== Section restricts should influence the relevance score. ========== Apply fixes upstream that were necessary to sync changes downstream. Also added a METADATA check to prevent any accidental adds of foo.proto.h includes. ========== Remove the 'com.google.protobuf' to 'com.google.android.icing.protobuf' translation in the export_to_aosp script. ========== Include usage store size in GetOptimizeInfo. This helps clients get a better idea of what savings they could get back if they called Optimize. Change-Id: Ia2339c7987267a73c49dadf1ced4a0a8ef001d4c
2020-01-09Pull upstream changes and copy libtextclassifier classes.Cassie Wang
Upstream synced @288789500. Copied text_classifier dependencies (hash, status, logging) into icing/. Bug: 146383629 Test: manual - ran 'm -j libicing' with a working Android.bp locally Change-Id: I187a98af95b362745a09d605ed8334a6ff6971bb
2019-12-20Port over Icing c++ code from upstreamCassie Wang
Change-Id: Ia3981fed7e0e70589efc027d4123f306cdfbe990