aboutsummaryrefslogtreecommitdiff
path: root/icing/index/main/main-index_test.cc
AgeCommit message (Collapse)Author
2023-08-30Update Icing from upstream.Grace Zhao
Descriptions: ======================================================================== [fixit] Remove incorrect reserve usage for STL data structure ======================================================================== Move LiteIndex::SortHits() from the query path to the indexing path ======================================================================== Bug: 286418010 Bug: 297549761 Bug: 248295995 Change-Id: I46168178e0227451b0f7dbf394bb6cc5db94e1e8
2023-08-23Update Icing from upstream.Tim Barron
Descriptions: ======================================================================== Fix term frequency bug ======================================================================== Delete dead JNI functions. ======================================================================== Switch Icing JNI implementation to use RegisterNatives ======================================================================== Avoid unnecessary GetObjectClass and GetFieldID calls. ======================================================================== Bug: 296938196 Change-Id: I773bcc05e768a7ac80f1950985b618047d009641
2023-03-01Update Icing from upstream.Terry Wang
Descriptions: ====================================================================== Add PropertyUtil for all property name/path related operations ====================================================================== [JoinableCache][2.0/x] Create SchemaPropertyIterator ====================================================================== [JoinableCache][2.1/x] Handle nested indexable flag ====================================================================== [JoinableCache][2.2/x] Add schema cycle dependency detection for SchemaPropertyIterator ====================================================================== [JoinableCache][3.0/x] Refactor SectionManager ====================================================================== [JoinableCache][3.1/x] Add unit tests for SectionManager::Builder and SchemaTypeManager ====================================================================== [NumericSearch][Storage][12/x] Implement Edit and GetIterator for IntegerIndex ====================================================================== [NumericSearch][Storage][13.0/x] Rename numeric-index_test as integer-index_test ====================================================================== [NumericSearch][Storage][13.1/x] Add IntegerIndexTest ====================================================================== Support the "len", "sum" and "avg" functions in advanced scoring. ====================================================================== Support the "this.childrenScores()" function to allow expressing children scores of joins in advanced scoring. ====================================================================== Create an integration test for Join with advanced scoring ====================================================================== Rename the word "children" to "args" for function related ScoreExpression ====================================================================== Improve IndexBlock by PRead/PWrite instead of repeating mmap/msync/unmap ====================================================================== Refactor QueryVisitor to prepare for support for function calls. ====================================================================== Add support for function calls. ====================================================================== Fix breakage in score-and-rank_benchmark. ====================================================================== [NumericSearch][Storage][adhoc][ez] Fix comment for IntegerIndex ====================================================================== [NumericSearch][Storage][14/x] Create first IntegerIndexStorage benchmark ====================================================================== Rename Icing schema related terminology to prepare for polymorphism support ====================================================================== [JoinableCache][4.0/x] Move common methods from SectionManager to PropertyUtil ====================================================================== [JoinableCache][4.1/x] Retire GetSectionContent ====================================================================== [JoinableCache][4.2/x] Polish SectionManagerTest ====================================================================== Modify QueryVisitor to do: ====================================================================== [NumericSearch][Storage][15/x] Implement TransferIndex for IntegerIndexStorage ====================================================================== [NumericSearch][Storage][16/x] Implement Optimize and last added document id for IntegerIndex ====================================================================== [NumericSearch][rollout][1/x] Include indexable int64 into SchemaDelta and backward compatibility ====================================================================== Add backwards compatibility test for Icing schema storage migration. ====================================================================== Implement trim the right-most node from the doc-hit-info-iterator. ====================================================================== Add TrimmedNode structure into doc-hit-info-iterator. ====================================================================== [JoinableCache][5/x] Implement JoinableProperty and JoinablePropertyManager ====================================================================== [JoinableCache][6/x] Add JoinablePropertyManager into SchemaTypeManager ====================================================================== [JoinableCache][7/x] Implement ExtractJoinableProperties ====================================================================== [JoinableCache][8/x] Create class QualifiedIdTypeJoinableCache ====================================================================== [JoinableCache][9/x] Implement factory method for QualifiedIdTypeJoinableCache ====================================================================== [JoinableCache][10/x] Implement Get and Put for QualifiedIdTypeJoinableCache ====================================================================== [JoinableCache][11/x] Add unit tests for QualifiedIdTypeJoinableCache ====================================================================== Modify DocHitInfoIteratorSectionRestrict to allow multi-property restricts ====================================================================== Fix the definition of LiteIndex::WantsMerge. ====================================================================== [NumericSearch][rollout][2.0/x] Rollout persistent IntegerIndex ====================================================================== [NumericSearch][rollout][2.1/x] Add more tests for integer index restoration and optimization ====================================================================== [JoinableCache][adhoc][ez] Remove qualified id type joinable cache size info from document storage info ====================================================================== Integrate trim right node into suggestion processor. Bug: 208654892 Bug: 228240987 Bug: 249829533 Bug: 256081830 Bug: 259744228 Bug: 261474063 Bug: 263890397 Bug: 266103594 Bug: 268738297 Bug: 269295094 Change-Id: I5f1b3f3ed0b5d6933dc8c2ab3279904f7706b23e
2023-02-07Update icing from upstreamAlex Saveliev
====================================================================== Adds a proto change for the delete propagation option ====================================================================== [ez] Change version to magic for PersistentHashMap ====================================================================== [iOS][testing][nitro] Disabling ICU language segmenter_test. ====================================================================== 1. Add support for segmentation in the QueryVisitor. ====================================================================== Support the new double list type in ScoreExpression ====================================================================== Pass JoinChildrenFetcher from IcingSearchEngine all the way down to ScoringVisitor ====================================================================== Refactor the logic of Icing Joins so that nested search and scoring will be performed before the parent ====================================================================== Add lite-index thread-safety tests. ====================================================================== Split IcingSearchEngineTest into separate tests to cover specific apis: ====================================================================== Fix index tests TearDown method. ====================================================================== Improve query concurrency by providing a finer-grained lock around the LiteIndex. ====================================================================== Fix Icing normalization bug ====================================================================== [ez] Fix integer overflow error for IntegerIndexStorage ====================================================================== [NumericSearch][Storage][11/x] Implement Reset and destructor for IntegerIndex ====================================================================== [NumericSearch][Storage][10/x] Add class IntegerIndex ====================================================================== Refactor NumericIndex based on PersistentStorage ====================================================================== Refactor IntegerIndexStorage based on PersistentStorage ====================================================================== Add "working_path" into PersistentStorage ====================================================================== Refactor PersistentHashMap based on PersistentStorage ====================================================================== Create virtual class PersistentStorage for refactoring ====================================================================== Avoids returning reference to local temporary object. ====================================================================== LSC: Clean up references to the legacy protobuf compat library ====================================================================== Fix time complexity regression for snippet retriever Bug: 256022027 Bug: 193919210 Bug: 266132035 Bug: 208654892 Bug: 261474063 Bug: 266103594 Bug: 146008613 Bug: 253182853 Bug: 266204868 Bug: 249829533 Bug: 266665956 Bug: 265258364 Change-Id: Ib2398c5097b6a2a57900e2ad4e3737502aa13820
2023-01-23Update Icing from upstream.Terry Wang
Descriptions: ====================================================================== Fix time complexity regression for snippet retriever ====================================================================== Add support for QueryTerms SectionRestrict Map in Advanced Query. ====================================================================== [NumericSearch][Storage][9/x] Create unit tests for AddKeys and GetIterator ====================================================================== [NumericSearch][Storage][8/x] Add options ====================================================================== Fix the stack overflow issue for icing advanced scoring/query parsers/visitors by limiting the number of tokens allowed ====================================================================== Improves the has joinspec check ====================================================================== Fix the bug of DocHitInfoIteratorTermMain found by monkey test ====================================================================== Rename tests defined in DestructibleDirectoryTest to include "directory" instead of "file". ====================================================================== Support projection in icing monkey test ====================================================================== [NumericSearch][Storage][7/x] Implement iterator for query ====================================================================== [NumericSearch][Storage][6/x] Implement AddKeys ====================================================================== Add support for relevance scoring by enabling term_frequency retrieval in DocHitInfoIterators and by populating the QueryTermIteratorsMap. ====================================================================== [JoinableCache][1/x] Compute SchemaDelta for joinable properties ====================================================================== [NumericSearch][Storage][5/x] PersistToDisk ====================================================================== [NumericSearch][Storage][4.1/x] Create unit tests for initialize ====================================================================== [NumericSearch][Storage][4.0/x] Implement Create and Initialize functions ====================================================================== Rename PostingListUsedSerializer related classes Bug: 193244409 Bug: 208654892 Bug: 246984163 Bug: 249829533 Bug: 256022027 Bug: 265258364 Bug: 265834832 Change-Id: Ibc7011a793ef5ed09eace0fb05d168adf5b4faca
2022-11-11Sync from upstream.Armaan Danewalia
Descriptions: ====================================================================== Include equals-proto and convert CodeToString methods to inline. ====================================================================== Add schema and document generators used by monkey test ====================================================================== Create in-memory icing for monkey testing ====================================================================== [NumericSearch][Storage][refactor_posting_list][1/x] Create PostingListUsedHitSerializer ====================================================================== [NumericSearch][Storage][refactor_posting_list][2/x] Create PostingListUsedHitSerializerTest ====================================================================== [NumericSearch][Storage][refactor_posting_list][3/x] Refactor all posting list related classes to use PostingListUsedSerializer ====================================================================== Adds a JoinSpecProto, and a new ranking strategy ====================================================================== Create Monkey Test Runner that randomly performs Icing API calls and check the results with the in-memory Icing ====================================================================== Support monkey testing the DeleteByQuery and Search APIs of Icing search engine ====================================================================== Directly include proto.h files from portable_proto_library() instead of wrapper pb.h files. ====================================================================== Directly include proto.h files from portable_proto_library() instead of wrapper pb.h files. ====================================================================== Directly include proto.h files from portable_proto_library() instead of wrapper pb.h files. ====================================================================== Removes nested_query from JoinSpec, as nested_search_spec includes it. ====================================================================== Directly include proto.h files from portable_proto_library() instead of wrapper pb.h files. ====================================================================== Swap the order when we build the doc-hit-info-iterator-and. ====================================================================== Support monkey testing section restrictions in the DeleteByQuery and Search APIs ====================================================================== Allow the in-memory icing to return the number of deleted documents, and check it with the delete stats of DeleteByNamespace, DeleteBySchemaType, and DeleteByQuery in the monkey test ====================================================================== [NumericSearch][Storage][refactor_posting_list][4/x] Move posting list common files into another directory ====================================================================== Add file-skipping for exports to Jetpack using @exportToAOSP:skipFile() tag ====================================================================== Change invalid type int32 to int32_t in icing-search-enging-jni.cc. ====================================================================== Minor fix, move generate term_iterator inside of the ranking_strategy checking ====================================================================== Address index out of bounds issue in third_party/icing/result/snippet-retriever.cc. This issue is causing test failures (see example failure: http://sponge2/cfcda71a-1312-455a-9a70-821c74c708e6). ====================================================================== Fix 1 DependencyCleaner findings: ====================================================================== Implement URL tokenization for Icing-lib [2/2]: ====================================================================== [NumericSearch][General][1/x] Create numeric.proto and add IntegerIndexingConfig ====================================================================== [NumericSearch][General][2/x] Refactor GetStringSectionContent and GetStringPropertyContent ====================================================================== [NumericSearch][General][3/x] Add DataType into SectionMetadata and change AssignSections ====================================================================== Move unit test constant definition into schema builder ====================================================================== [NumericSearch][General][4/x] Create templated Section and SectionGroup ====================================================================== [NumericSearch][General][5/x] Create BasicHit ====================================================================== Change URL tokenizer's url_parse dependency to use third_party/url_parse:url_parse_stripped_down ====================================================================== Exclude memory-mapped-file-leak_test.cc from export to AOSP. ====================================================================== Rollback of changelist 487633403. Reason: This cl breaks YouTube Music builds due to duplicate symbols. ====================================================================== [ez] Fix ScoredDocumentHit related comparators ====================================================================== Bug: 193244409 Bug: 246984163 Bug: 249829533 Bug: 256022027 Bug: 256679292 Bug: 246964044 Change-Id: I55c2ed417a31321c22de377e97ffe0096478d28c
2022-09-08Sync from upstream.Grace Zhao
Descriptions: ====================================================================== [FileBackedVector Consolidation][4/x] Fix potential PWrite bug in GrowIfNecessary ====================================================================== [FileBackedVector Consolidation][5/x] Create benchmark for FileBackedVector ====================================================================== [FileBackedVector Consolidation][6/x] Avoid calling GetFileSize in GrowIfNecessary ====================================================================== [PersistentHashMap][3.3/x] Implement Delete ====================================================================== Fix the PopulateMatchedTermsStats bug ====================================================================== Add JNI latency for query latency stats breakdown. ====================================================================== [ResultStateManager] Thread safety test1 ====================================================================== [ResultStateManager][2/x] Thread safety test2 ====================================================================== Add native lock contention latency for measuring query latency ====================================================================== Fix implementation of HasMember operator in ANTLR-based list-filter prototype. ====================================================================== Fix improper uses of std::string_view ====================================================================== Extend the scale of Icing ====================================================================== Decouple the term frequency array from DocHitInfo ====================================================================== Disable hit_term_frequency for non-relevance queries ====================================================================== [ResultStateManager][3/x] Thread safety test3 ====================================================================== [PersistentHashMap][4/x] Implement iterator ======================================================================= Fix the lite index compaction bug ======================================================================= Change-Id: I0edad67affed97af107e2d7cd73770e0268c0903
2021-12-28Sync from upstream.Dan Saadati
Descriptions: ================ Normalize Tokens by Token type when retrieving snippets ================ Rename max_window_bytes to max_window_utf32_length, Delete the max_tokens_per_doc field in IcingSearchEngineOptions. ================ Handle suggestion namespace ownership. ================ Fix OkStatus() is not a valid argument to StatusOr in Main_index.RetrieveMoreHits. ================ Allow advancing when current indices are negative in CharacterIterator ================ Adds support for verbatim tokenization and indexing in IcingLib ================ Renames TokenizerIterator Reset functions ================ Add term_match_type to SuggestionSpec proto ================ Unify the C++ proto enum style ================ Allow zero property weights in IcingLib ================ Bug: 204333391 Bug: 152934343 Bug: 205209589 Bug: 206147728 Bug: 209993976 Change-Id: Id94a377fd37c5eb7ebc3d7547cf8ff0ad4152620
2021-01-05Update Icing from upstream.Tim Barron
Change-Id: Iff50aebffb83529b0454e7c3a6dc6864e7a85f4a
2020-12-09Update Icing from upstream.Tim Barron
Change-Id: I662cd2f97794722eca9cc43b73549babed884be4
2020-10-28Pull upstream changes.Terry Wang
Change-Id: I73ea5f80ccf16a02519f6f7ccfc993e9b0f39f86
2020-10-01Pull upstream changes.Terry Wang
Change-Id: I794757716961569b5c02171cfc82785efb2cf106