aboutsummaryrefslogtreecommitdiff
path: root/synced_AOSP_CL_number.txt
AgeCommit message (Collapse)Author
2023-06-02Update Icing from upstream.Tim Barron
Descriptions: ======================================================================== Eliminate unnecessary GetCurrentTime syscalls in various hotspots. ======================================================================== Fix MSAN failure in SchemaStore::Header::Write. ======================================================================== Bug: 282308801 Change-Id: Ifb11f890485329e6b0bc1a9cbf2bf046e39b7854
2023-05-23Update Icing from upstream.Tim Barron
Descriptions: ======================================================================== Downgrade log priorities on filesystem errors. ======================================================================== Bug: 282997030 Change-Id: Ia6dd9c9c1f3bdb68458cd1e9dba83e01a4b25123
2023-05-19Update Icing from upstream.Tim Barron
Descriptions: ======================================================================== Rename `this.childrenScores` to `this.childrenRankingSignals`. ======================================================================== Change-Id: I6ba4087a6de76ea13d149f4af1b7a778e6623a5d
2023-05-19Update Icing from upstream.Tim Barron
Descriptions: ======================================================================== Add flag to guard PersistentHashMapKeyMapper and DynamicTrieKeyMapper ======================================================================== Add flag to guard FileBackedVector premapping ======================================================================== Check tokenizer type when assigning section id ======================================================================== Bug:193919210 Change-Id: I682e22d4ba647220b7222fcc63c41bd0e01bbeb9
2023-05-15Update Icing from upstream.Terry Wang
Descriptions: ======================================================================== Minor fix for export to framework. ======================================================================== [Icing][version][3/x][ez] Call DataSync for version file ======================================================================== Add `use_read_only_search` flag for flag-guarding IcingSearchEngine::Search locking change in cl/506671441. ======================================================================== Bug: 146008613 Test: presubmit Change-Id: Id863275cf5cc1ad7bcc69649527cc17981cc6cd2
2023-05-11Update Icing from upstream.Tim Barron
Descriptions: ======================================================================== Handle version changes in the schema store. ======================================================================== Bug: 280697513 Change-Id: I6da14f236cd8e4cf2af75c1fb2c115d1df512114
2023-05-11Update Icing from upstream.Tim Barron
Descriptions: ======================================================================== Modify the definition of propertyDefined: ======================================================================== Remove default args in SchemaStore::SetSchema and fix calls ======================================================================== Add allow_circular_schema_definitions flag ======================================================================== Onboard version detection to Icing ======================================================================== Add version util to help read/write version info ======================================================================== Add support for the overlay schema. ======================================================================== Allow cycles in schema-property-iterator ======================================================================== Add joinable properties into schema definition cycle restrictions. ======================================================================== Loosen circular references restriction for Schema Definitions. ======================================================================== Implement BackupSchemaProducer to generate a backup schema ======================================================================== Minor fix: remove a redundant log ======================================================================== Allow schema types to inherit from more than one parent ======================================================================== allow nested document properties to accept documents of subtype ======================================================================== Support polymorphism for Icing projection in Search and Get API ======================================================================== Add max_joined_child_per_parent into ResultSpec and change behavior ======================================================================== Support Icing schema type polymorphism for the search filter API ======================================================================== Verify that every child type's property set has included all compatible properties from parent types ======================================================================== Add individual type index latency ======================================================================== Build the iterator node for the propertyDefined() custom function ======================================================================== Advance all hits with same doc id from and merge sections once for the same bucket iter ======================================================================== Introduce DocHitInfoIteratorPropertyInSchema for property existence check ======================================================================== Add SchemaUtil::BuildTransitiveInheritanceGraph to build an inheritance map from schema ======================================================================== Introduce a lookup method for a property defined in a schema ======================================================================== Rollback of: Allow LanguageSegmenter::Iterators to declare AccessType. ======================================================================== Adds join info to QueryStatsProto ======================================================================== Bug:280698419 Bug:280698125 Bug:280698121 Bug:280697513 Bug:276349029 Bug:272145329 Bug:270102295 Bug:269295094 Bug:268680462 Bug:265304217 Bug:259744228 Bug:259743562 Bug:256022027 Change-Id: I54cd1d22121c314f8c238d2d49f0809165dc0ca3
2023-04-19Update Icing from upstream.Jiayu Hu
Descriptions: ======================================================================== Adds join info to QueryStatsProto ======================================================================== [Join][cleanup][1/x] Add Icing benchmark for qualified id join ======================================================================== [NumericSearch][benchmark][2/x] Modify run-benchmarks.py to onboard numeric search benchmark ======================================================================== [NumericSearch][benchmark][1/x] Add numeric search icing search engine benchmark ======================================================================== Switch Iterator access type used in SnippetRetriever back to kForwardIterator. ======================================================================== Fix `DocumentStore::OptimizeInto` by passing `options_.document_store_namespace_id_fingerprint()` ======================================================================== Add file size check when initializing mmap for existing metadata file. ======================================================================== Add compression level option to Icing and propagate down to PortableFileBackedProtoLog ======================================================================== Add portable_equals_proto namespace for EqualsProto in tokenized-document_test. ======================================================================== Refactor schema to check consistency of polymorphism by dependency map ======================================================================== Support the `this.propertyWeights()` function in advanced scoring ======================================================================== Randomly set optimize_rebuild_index_threshold in monkey test ======================================================================== Change benchmark batch query latency to average ======================================================================== Performance improvements to SnippetRetriever. ======================================================================== Bug: 270102295 Bug: 260625837 Bug: 269295094 Bug: 193244409 Bug: 269513147 Bug: 276345448 Bug: 259744228 Bug: 272145329 Bug: 276349029 Change-Id: Ief1cb2c21f67aac420d6b79d9b946bbecdc39a5c
2023-03-31Update Icing from upstream.Tim Barron
Descriptions: ======================================================================== Make int64_t the CreateUsageReport timestamp type ======================================================================== Create ResultAdjustmentInfo and refactor ResultStateV2 ======================================================================== Wrap parent/child adjustment info to std::unique_ptr ======================================================================== Apply join child snippet ======================================================================== Apply join child projection ======================================================================== Add IntegerSectionIndexingHandlerTest ======================================================================== Delete Result Retriever this class is dead code. ======================================================================== Introduce a placeholder for the custom function hasPropertyDefined(member) ======================================================================== Fix libtextclassifier3::StatusOr ======================================================================== Performance improvements to SnippetRetriever. ======================================================================== Bug: 193244409 Bug: 256022027 Bug: 259744228 Bug: 268680462 Bug: 270102295 Bug: 271015984 Bug: 274627497 Change-Id: I6bad316b28bb289fa8e3f5b0982d6aaa9e0d135f
2023-03-16Update Icing from upstream.Tim Barron
Descriptions: ======================================================================== Set overall index latency ======================================================================== Change GetUsageScores return type to optional. ======================================================================== Upstream fix to thread-safety annotation for cached_break_iterator_ ======================================================================== Bug: 259744228 Change-Id: Ia7a5032fd7655db773e311173f5735e4451b30c1
2023-03-14Update Icing from upstream.Tim Barron
Descriptions: ======================================================================== Cache an instance of UBreakIterator to reduce unnecessary creations. ======================================================================== Cap number of individual IntegerIndexStorages that IntegerIndex creates. ======================================================================== Change error in trimRightMostNode from Unimplemented to InvalidArgument. ======================================================================== Add detection for new language features of List Filters Query Language. ======================================================================== Add option to control threshold to rebuild index during optimize by flag ======================================================================== Add option to control use of namespace id to build urimapper by flag. ======================================================================== Enforce schema validation for joinable config. ======================================================================== Adopt bucket splitting for IntegerIndexStorage. ======================================================================== Implement bucket splitting function. ======================================================================== Add Icing initialization unit tests for QualifiedIdTypeJoinableIndex. ======================================================================== Add Icing schema change unit tests for QualifiedIdTypeJoinableIndex. ======================================================================== Add Icing optimization unit tests for QualifiedIdTypeJoinableIndex. ======================================================================== Integrate QualifiedIdTypeJoinableIndex into IcingSearchEngine. ======================================================================== Implement QualifiedIdJoinablePropertyIndexingHandler. ======================================================================== Change QualifiedIdTypeJoinableIndex to store raw qualified id string. ======================================================================== Pass info about unnormalized query terms through lexer/parser/visitor. ======================================================================== Bug: 208654892 Bug: 263890397 Bug: 259743562 Bug: 272145329 Bug: 227356108 Change-Id: I438a390ddda5673cf2b5781af502f2b7cfeaee74
2023-03-09Update Icing from upstream.Tim Barron
Descriptions: ====================================================================== Integrate Advanced Query w/ Suggest, make ADVANCED_QUERY default parser. ====================================================================== BUG: 208654892 Change-Id: I53889482e844648ec65565f9a7f7c8faa89baa7c
2023-03-06Update Icing from upstream.Tim Barron
Descriptions: ====================================================================== Refactor IndexProcessor ====================================================================== Rename Joinable Cache as Joinable Index ====================================================================== Implement Optimize and Clear for QualifiedIdTypeJoinableCache ====================================================================== Add JoinablePropertyMetadata reverse lookup ====================================================================== Allow code creating LanguageSegmenter::Iterators to declare AccessType ====================================================================== Further codifies the escape behavior in the parser test ====================================================================== Bug: 263890397 Bug: 268680462 Bug: 270102295 Change-Id: I3233733b40e985e11c4a6d75c1528cd6a72c1173
2023-03-03Update Icing from upstream.Terry Wang
Descriptions: ====================================================================== Modify the lexer to consider internal minus characters to be a part of TEXT segments (as hyphens). ====================================================================== Modify the query language to support an individual prefix operator '*'. ====================================================================== Fix the fingerprint issue for document key mapper in document store ====================================================================== Resolve issues around the handling of negative numbers. ====================================================================== Fix doc join info compiler warning/error ====================================================================== Codifies how colons are tokenized by the lexer Bug: 208654892 Bug: 268680462 Change-Id: I67169b2e656eeafb6500b5b03a6c46c3cc9c3d66
2023-03-01Update Icing from upstream.Terry Wang
Descriptions: ====================================================================== Add PropertyUtil for all property name/path related operations ====================================================================== [JoinableCache][2.0/x] Create SchemaPropertyIterator ====================================================================== [JoinableCache][2.1/x] Handle nested indexable flag ====================================================================== [JoinableCache][2.2/x] Add schema cycle dependency detection for SchemaPropertyIterator ====================================================================== [JoinableCache][3.0/x] Refactor SectionManager ====================================================================== [JoinableCache][3.1/x] Add unit tests for SectionManager::Builder and SchemaTypeManager ====================================================================== [NumericSearch][Storage][12/x] Implement Edit and GetIterator for IntegerIndex ====================================================================== [NumericSearch][Storage][13.0/x] Rename numeric-index_test as integer-index_test ====================================================================== [NumericSearch][Storage][13.1/x] Add IntegerIndexTest ====================================================================== Support the "len", "sum" and "avg" functions in advanced scoring. ====================================================================== Support the "this.childrenScores()" function to allow expressing children scores of joins in advanced scoring. ====================================================================== Create an integration test for Join with advanced scoring ====================================================================== Rename the word "children" to "args" for function related ScoreExpression ====================================================================== Improve IndexBlock by PRead/PWrite instead of repeating mmap/msync/unmap ====================================================================== Refactor QueryVisitor to prepare for support for function calls. ====================================================================== Add support for function calls. ====================================================================== Fix breakage in score-and-rank_benchmark. ====================================================================== [NumericSearch][Storage][adhoc][ez] Fix comment for IntegerIndex ====================================================================== [NumericSearch][Storage][14/x] Create first IntegerIndexStorage benchmark ====================================================================== Rename Icing schema related terminology to prepare for polymorphism support ====================================================================== [JoinableCache][4.0/x] Move common methods from SectionManager to PropertyUtil ====================================================================== [JoinableCache][4.1/x] Retire GetSectionContent ====================================================================== [JoinableCache][4.2/x] Polish SectionManagerTest ====================================================================== Modify QueryVisitor to do: ====================================================================== [NumericSearch][Storage][15/x] Implement TransferIndex for IntegerIndexStorage ====================================================================== [NumericSearch][Storage][16/x] Implement Optimize and last added document id for IntegerIndex ====================================================================== [NumericSearch][rollout][1/x] Include indexable int64 into SchemaDelta and backward compatibility ====================================================================== Add backwards compatibility test for Icing schema storage migration. ====================================================================== Implement trim the right-most node from the doc-hit-info-iterator. ====================================================================== Add TrimmedNode structure into doc-hit-info-iterator. ====================================================================== [JoinableCache][5/x] Implement JoinableProperty and JoinablePropertyManager ====================================================================== [JoinableCache][6/x] Add JoinablePropertyManager into SchemaTypeManager ====================================================================== [JoinableCache][7/x] Implement ExtractJoinableProperties ====================================================================== [JoinableCache][8/x] Create class QualifiedIdTypeJoinableCache ====================================================================== [JoinableCache][9/x] Implement factory method for QualifiedIdTypeJoinableCache ====================================================================== [JoinableCache][10/x] Implement Get and Put for QualifiedIdTypeJoinableCache ====================================================================== [JoinableCache][11/x] Add unit tests for QualifiedIdTypeJoinableCache ====================================================================== Modify DocHitInfoIteratorSectionRestrict to allow multi-property restricts ====================================================================== Fix the definition of LiteIndex::WantsMerge. ====================================================================== [NumericSearch][rollout][2.0/x] Rollout persistent IntegerIndex ====================================================================== [NumericSearch][rollout][2.1/x] Add more tests for integer index restoration and optimization ====================================================================== [JoinableCache][adhoc][ez] Remove qualified id type joinable cache size info from document storage info ====================================================================== Integrate trim right node into suggestion processor. Bug: 208654892 Bug: 228240987 Bug: 249829533 Bug: 256081830 Bug: 259744228 Bug: 261474063 Bug: 263890397 Bug: 266103594 Bug: 268738297 Bug: 269295094 Change-Id: I5f1b3f3ed0b5d6933dc8c2ab3279904f7706b23e
2023-02-07Update icing from upstreamAlex Saveliev
====================================================================== Adds a proto change for the delete propagation option ====================================================================== [ez] Change version to magic for PersistentHashMap ====================================================================== [iOS][testing][nitro] Disabling ICU language segmenter_test. ====================================================================== 1. Add support for segmentation in the QueryVisitor. ====================================================================== Support the new double list type in ScoreExpression ====================================================================== Pass JoinChildrenFetcher from IcingSearchEngine all the way down to ScoringVisitor ====================================================================== Refactor the logic of Icing Joins so that nested search and scoring will be performed before the parent ====================================================================== Add lite-index thread-safety tests. ====================================================================== Split IcingSearchEngineTest into separate tests to cover specific apis: ====================================================================== Fix index tests TearDown method. ====================================================================== Improve query concurrency by providing a finer-grained lock around the LiteIndex. ====================================================================== Fix Icing normalization bug ====================================================================== [ez] Fix integer overflow error for IntegerIndexStorage ====================================================================== [NumericSearch][Storage][11/x] Implement Reset and destructor for IntegerIndex ====================================================================== [NumericSearch][Storage][10/x] Add class IntegerIndex ====================================================================== Refactor NumericIndex based on PersistentStorage ====================================================================== Refactor IntegerIndexStorage based on PersistentStorage ====================================================================== Add "working_path" into PersistentStorage ====================================================================== Refactor PersistentHashMap based on PersistentStorage ====================================================================== Create virtual class PersistentStorage for refactoring ====================================================================== Avoids returning reference to local temporary object. ====================================================================== LSC: Clean up references to the legacy protobuf compat library ====================================================================== Fix time complexity regression for snippet retriever Bug: 256022027 Bug: 193919210 Bug: 266132035 Bug: 208654892 Bug: 261474063 Bug: 266103594 Bug: 146008613 Bug: 253182853 Bug: 266204868 Bug: 249829533 Bug: 266665956 Bug: 265258364 Change-Id: Ib2398c5097b6a2a57900e2ad4e3737502aa13820
2023-01-23Update Icing from upstream.Terry Wang
Descriptions: ====================================================================== Fix time complexity regression for snippet retriever ====================================================================== Add support for QueryTerms SectionRestrict Map in Advanced Query. ====================================================================== [NumericSearch][Storage][9/x] Create unit tests for AddKeys and GetIterator ====================================================================== [NumericSearch][Storage][8/x] Add options ====================================================================== Fix the stack overflow issue for icing advanced scoring/query parsers/visitors by limiting the number of tokens allowed ====================================================================== Improves the has joinspec check ====================================================================== Fix the bug of DocHitInfoIteratorTermMain found by monkey test ====================================================================== Rename tests defined in DestructibleDirectoryTest to include "directory" instead of "file". ====================================================================== Support projection in icing monkey test ====================================================================== [NumericSearch][Storage][7/x] Implement iterator for query ====================================================================== [NumericSearch][Storage][6/x] Implement AddKeys ====================================================================== Add support for relevance scoring by enabling term_frequency retrieval in DocHitInfoIterators and by populating the QueryTermIteratorsMap. ====================================================================== [JoinableCache][1/x] Compute SchemaDelta for joinable properties ====================================================================== [NumericSearch][Storage][5/x] PersistToDisk ====================================================================== [NumericSearch][Storage][4.1/x] Create unit tests for initialize ====================================================================== [NumericSearch][Storage][4.0/x] Implement Create and Initialize functions ====================================================================== Rename PostingListUsedSerializer related classes Bug: 193244409 Bug: 208654892 Bug: 246984163 Bug: 249829533 Bug: 256022027 Bug: 265258364 Bug: 265834832 Change-Id: Ibc7011a793ef5ed09eace0fb05d168adf5b4faca
2023-01-06Update icing from upstreamAlex Saveliev
Descriptions: ====================================================================== Apply aggregation score to join result and create JoinProcessorTest ====================================================================== Create AggregationScorerTest ====================================================================== Add an EqualsTermMatchInfo matcher to simplify tests that interact with this struct. ====================================================================== Refactor the QueryVisitor to return QueryResults. ====================================================================== Simplify ScoreExpression to ConstantScoreExpression if its evaluation does not depend on a document ====================================================================== Use rvalue and std::move for DocumentStore methods ====================================================================== Downgrade log messages for NOT_FOUND errors in Delete calls. ====================================================================== [NumericSearch][Storage][3.1/x] Add metadata file declaration (Crcs, Info) for IntegerIndexStorage ====================================================================== [NumericSearch][Storage][3.0/x] Declare class IntegerIndexStorage ====================================================================== Cleanup Import from //tools/build_defs/objc ====================================================================== Cleanup LSC: Remove `portable_proto_library()` targets ====================================================================== Bug: 256022027 Bug: 208654892 Bug: 261474063 Bug: 246628980 Bug: 249829533 Bug: 263373214 Change-Id: Ib5ca7efffd575254e14a14d54a45f4efd1e8d6c9
2022-12-20Sync from upstream.Tim Barron
Descriptions: ====================================================================== Implement simple feature checking for implicit new features in IcingLib [2/2]. ====================================================================== Bug: 262512396 Change-Id: Ia41f482a7cb442727e605407eb20c52f6021954f
2022-12-16Sync from upstream.Tim Barron
Descriptions: ====================================================================== Implement CorpusId as the key for ResultGrouping. ====================================================================== Implement simple feature checking for implicit new features in IcingLib. ====================================================================== Bug: 258715421 Bug: 262512396 Change-Id: I815a68bc91d2750b2210f2f3e227044479a4de7a
2022-12-15Sync from upstream.Tim Barron
Descriptions: ====================================================================== Rename fullyQualifiedId to qualifiedId ====================================================================== Delete ResultState, which is dead code. ====================================================================== Implement support in the QueryVisitor for the property restrict operator (HAS) ====================================================================== Implement support in the QueryVisitor for implicit and, 'AND' and 'OR'. ====================================================================== Implement support in the QueryVisitor for NOT. ====================================================================== Implement support in the QueryVisitor for single term and single verbatim term queries. ====================================================================== Bug: 208654892 Bug: 256022027 Change-Id: I6c689dd7070ac0a9818b7425f3beab7258181049
2022-12-14Sync from upstream.Tim Barron
Descriptions: ====================================================================== Fix naming of version_array variable, which should be snake_case. ====================================================================== Change-Id: Ia2a512b65e2aeeffdd4a51dfdaf3695c26fcb970
2022-12-14Sync from upstream.Tim Barron
Descriptions: ====================================================================== Create class `QualifiedId` ====================================================================== Switch JoinProcessor to use new class FullyQualifiedId ====================================================================== Add `JoinableConfig` proto ====================================================================== Implement document-based functions for the Advanced Scoring Language ====================================================================== Support the RelevanceScore function for the Advanced Scoring Language ====================================================================== Enable the document-based member functions for Advanced Scoring Language Bug: 256022027 Bug: 261474063 Change-Id: I5f7e58dcc5e5cfc6089a856dd4f0617ccf485485
2022-12-12Sync from upstream.Tim Barron
Descriptions: ====================================================================== Add ScoringSpec into JoinSpec. Rename joined_document to child_document. ====================================================================== Create JoinedScoredDocumentHit class and refactor ScoredDocumentHitsRanker. ====================================================================== Implement initial Join workflow ====================================================================== Implement the Lexer for Icing Advanced Query Language ====================================================================== Create struct Options for PersistentHashMap ====================================================================== Premapping FileBackedVector ====================================================================== Create class PersistentHashMapKeyMapper ====================================================================== Add integer sections into TokenizedDocument and rename string sections ====================================================================== Create NumericIndex interface and DocHitInfoIteratorNumeric ====================================================================== Implement DummyNumericIndex and unit test ====================================================================== Change PostingListAccessor::Finalize to rvalue member function ====================================================================== Define the Abstract Syntax Tree for Icing's list_filter parser. ====================================================================== Refactor query processing and score ====================================================================== Refactor IcingSearchEngine for AppSearch Dynamite Module 0p APIs ====================================================================== Implement the Lexer for Icing Advanced Scoring Language ====================================================================== Add a common interface for IcingSearchEngine and dynamite client ====================================================================== Implement a subset of the query grammar. ====================================================================== Refactor index processor ====================================================================== Add integer index into IcingSearchEngine and IndexProcessor ====================================================================== Implement the parser for Icing Advanced Scoring Language ====================================================================== Implement IntegerIndexData and PostingListUsedIntegerIndexDataSerializer ====================================================================== Add PostingListAccessor abstract class for common components and methods ====================================================================== Implement PostingListIntegerIndexDataAccessor ====================================================================== Create PostingListIntegerIndexDataAccessorTest ====================================================================== Fix Icing Segmentation tests for word connectors that changed in ICU 72. ====================================================================== Modify the Advanced Query grammar to allow functions to accept expressions. ====================================================================== Implement QueryVisitor. ====================================================================== Enable the Advanced Query Parser to handle member functions ====================================================================== Refactor the Scorer class to support the Advanced Scoring Language ====================================================================== Integrate advanced query parser with the query processor. ====================================================================== Implement support for JoinSpec in Icing. ====================================================================== Implement the Advanced Scoring Language for basic functions and operators ====================================================================== Bug: 208654892 Bug: 249829533 Bug: 256022027 Bug: 261474063 Bug: 240333360 Bug: 193919210 Change-Id: I5f5bdc6249282ecc4b014b4fbdf8e2d1f8b20c19
2022-11-11Sync from upstream.Armaan Danewalia
Descriptions: ====================================================================== Include equals-proto and convert CodeToString methods to inline. ====================================================================== Add schema and document generators used by monkey test ====================================================================== Create in-memory icing for monkey testing ====================================================================== [NumericSearch][Storage][refactor_posting_list][1/x] Create PostingListUsedHitSerializer ====================================================================== [NumericSearch][Storage][refactor_posting_list][2/x] Create PostingListUsedHitSerializerTest ====================================================================== [NumericSearch][Storage][refactor_posting_list][3/x] Refactor all posting list related classes to use PostingListUsedSerializer ====================================================================== Adds a JoinSpecProto, and a new ranking strategy ====================================================================== Create Monkey Test Runner that randomly performs Icing API calls and check the results with the in-memory Icing ====================================================================== Support monkey testing the DeleteByQuery and Search APIs of Icing search engine ====================================================================== Directly include proto.h files from portable_proto_library() instead of wrapper pb.h files. ====================================================================== Directly include proto.h files from portable_proto_library() instead of wrapper pb.h files. ====================================================================== Directly include proto.h files from portable_proto_library() instead of wrapper pb.h files. ====================================================================== Removes nested_query from JoinSpec, as nested_search_spec includes it. ====================================================================== Directly include proto.h files from portable_proto_library() instead of wrapper pb.h files. ====================================================================== Swap the order when we build the doc-hit-info-iterator-and. ====================================================================== Support monkey testing section restrictions in the DeleteByQuery and Search APIs ====================================================================== Allow the in-memory icing to return the number of deleted documents, and check it with the delete stats of DeleteByNamespace, DeleteBySchemaType, and DeleteByQuery in the monkey test ====================================================================== [NumericSearch][Storage][refactor_posting_list][4/x] Move posting list common files into another directory ====================================================================== Add file-skipping for exports to Jetpack using @exportToAOSP:skipFile() tag ====================================================================== Change invalid type int32 to int32_t in icing-search-enging-jni.cc. ====================================================================== Minor fix, move generate term_iterator inside of the ranking_strategy checking ====================================================================== Address index out of bounds issue in third_party/icing/result/snippet-retriever.cc. This issue is causing test failures (see example failure: http://sponge2/cfcda71a-1312-455a-9a70-821c74c708e6). ====================================================================== Fix 1 DependencyCleaner findings: ====================================================================== Implement URL tokenization for Icing-lib [2/2]: ====================================================================== [NumericSearch][General][1/x] Create numeric.proto and add IntegerIndexingConfig ====================================================================== [NumericSearch][General][2/x] Refactor GetStringSectionContent and GetStringPropertyContent ====================================================================== [NumericSearch][General][3/x] Add DataType into SectionMetadata and change AssignSections ====================================================================== Move unit test constant definition into schema builder ====================================================================== [NumericSearch][General][4/x] Create templated Section and SectionGroup ====================================================================== [NumericSearch][General][5/x] Create BasicHit ====================================================================== Change URL tokenizer's url_parse dependency to use third_party/url_parse:url_parse_stripped_down ====================================================================== Exclude memory-mapped-file-leak_test.cc from export to AOSP. ====================================================================== Rollback of changelist 487633403. Reason: This cl breaks YouTube Music builds due to duplicate symbols. ====================================================================== [ez] Fix ScoredDocumentHit related comparators ====================================================================== Bug: 193244409 Bug: 246984163 Bug: 249829533 Bug: 256022027 Bug: 256679292 Bug: 246964044 Change-Id: I55c2ed417a31321c22de377e97ffe0096478d28c
2022-10-27Sync from upstream.Grace Zhao
Descriptions: ====================================================================== Fix the bug in PostingListAccessor found by the Icing Monkey test ====================================================================== Add the logic to handle fatal errors from IcingDynamicTrie to avoid crashing ====================================================================== Clear out the dead code IcingDynamicTrie::Compact ====================================================================== [MemoryMappedFile][RemapV2][1/x] Add factory method for MemoryMappedFile ====================================================================== [MemoryMappedFile][RemapV2][2/x] Create GrowAndRemapIfNecessary and change factory method ====================================================================== [MemoryMappedFile][RemapV2][3/x] Migrate FileBackedVector to use GrowAndRemapIfNecessary ====================================================================== Add JNI latency for query latency stats breakdown ====================================================================== Bug: 247671531 Bug: 247929909 Bug: 253282365 Change-Id: Ic3b88d0f044edacfe2dfeb08fa381b2186c731cb
2022-10-11Update Icing from upstream.Terry Wang
Descriptions: ====================================================================== Implement URL tokenization for Icing-lib ====================================================================== Adds RFC822_HOST_ADDRESS ====================================================================== Support Suggestion API could be ordered by term's frequency. ====================================================================== Bug: 246964044 Bug: 230553264 Change-Id: Id7e7b1e080bb66ccf03452e75b72c5cceed2f7db
2022-09-30Update Icing from upstream.Terry Wang
Descriptions: ====================================================================== Integrate ANTLR-based advanced query prototype with query processor. ====================================================================== [PersistentHashMap][6.1/x] Wrap the return value of KeyMapper::ComputeChecksum by StatusOr ====================================================================== [PersistentHashMap][6.0/x] Replace GetValuesToKeys with iterator for KeyMapper ====================================================================== [PersistentHashMap][5.1/x] Allow client to specify initial num buckets ====================================================================== [PersistentHashMap][5/x] Implement rehashing ====================================================================== Add SchemaType filter and Document Id filter in Search Suggestion API. ====================================================================== Follow up to cl/463377778 ====================================================================== Add Document Id filters in Search Suggestion API. ====================================================================== Cleanup LSC: Replace inclusion of *_proto_portable.pb.h files coming from portable_proto_library() with *(!_proto_portable).pb.h coming from cc_proto_library(). ====================================================================== Cleanup LSC: Replace inclusion of *_proto_portable.pb.h files coming from portable_proto_library() with *(!_proto_portable).pb.h coming from cc_proto_library(). ====================================================================== Cloned from CL 464902284 by 'g4 patch'. ====================================================================== Cleanup LSC: Replace inclusion of *_proto_portable.pb.h files coming from portable_proto_library() with *(!_proto_portable).pb.h coming from cc_proto_library(). ====================================================================== Cleanup Remove unused visibility specs (last referenced in codebase over 132 days ago). ====================================================================== Cleanup LSC: Replace inclusion of *_proto_portable.pb.h files coming from portable_proto_library() with *(!_proto_portable).pb.h coming from cc_proto_library(). ====================================================================== Cleanup LSC: Replace inclusion of *_proto_portable.pb.h files coming from portable_proto_library() with *(!_proto_portable).pb.h coming from cc_proto_library(). ====================================================================== Remove dsaadati@ from third_party/icing OWNERS ====================================================================== Cleanup Move package level default_copts attribute to copts. ====================================================================== Refactor QueryProcessor and QueryProcessTest in preparation for adding ANTLR prototype to parse queries with search_type EXPERIMENTAL_ICING_ADVANCED_QUERY. ====================================================================== Bug: 208654892 Bug: 230553264 Bug: 237324702 Bug: 193919210 Change-Id: I2f0a612747ccb754502489a9b168406532cffaee
2022-09-08Sync from upstream.Grace Zhao
Descriptions: ====================================================================== [FileBackedVector Consolidation][4/x] Fix potential PWrite bug in GrowIfNecessary ====================================================================== [FileBackedVector Consolidation][5/x] Create benchmark for FileBackedVector ====================================================================== [FileBackedVector Consolidation][6/x] Avoid calling GetFileSize in GrowIfNecessary ====================================================================== [PersistentHashMap][3.3/x] Implement Delete ====================================================================== Fix the PopulateMatchedTermsStats bug ====================================================================== Add JNI latency for query latency stats breakdown. ====================================================================== [ResultStateManager] Thread safety test1 ====================================================================== [ResultStateManager][2/x] Thread safety test2 ====================================================================== Add native lock contention latency for measuring query latency ====================================================================== Fix implementation of HasMember operator in ANTLR-based list-filter prototype. ====================================================================== Fix improper uses of std::string_view ====================================================================== Extend the scale of Icing ====================================================================== Decouple the term frequency array from DocHitInfo ====================================================================== Disable hit_term_frequency for non-relevance queries ====================================================================== [ResultStateManager][3/x] Thread safety test3 ====================================================================== [PersistentHashMap][4/x] Implement iterator ======================================================================= Fix the lite index compaction bug ======================================================================= Change-Id: I0edad67affed97af107e2d7cd73770e0268c0903
2022-08-11Sync from upstream.Tim Barron
Descriptions: ====================================================================== Implement new version of ResultState and ResultStateManager to 1) enforce a page byte size limit and 2) improve handling of pagination when we encounter deleted documents. ====================================================================== Fix bugs in IcingDynamicTrie::Delete. ====================================================================== Implement IcingDynamicTrie::IsBranchingTerm. ====================================================================== Change Icing default logging level to INFO ====================================================================== Refactor KeyMapper class to be an interface. ====================================================================== Improve NamespaceChecker logic to improve Suggest latency. ====================================================================== Change icing native log tag to "AppSearchIcing" ====================================================================== Implement Index Compaction rather than rebuilding index during Compaction. ====================================================================== Implement reverse iterator for IcingDynamicTrie ====================================================================== Avoid adding unnecessary branch points during index compaction ====================================================================== Invalidate expired result states when adding to/retrieving from ResultStateManager. ====================================================================== Add new methods (MutableView, MutableArrayView, Append, Allocate) to FileBackedVector ====================================================================== Create and implement PersistentHashMap class. ====================================================================== Implement RFC822 Tokenizer ====================================================================== Remove uses of StringPrintf in ICING_LOG statements ====================================================================== Properly set query latency when an error is encountered or results are empty. ====================================================================== Bug: 146903474 Bug: 152934343 Bug: 193919210 Bug: 193453081 Bug: 231368517 Bug: 235395538 Bug: 236412165 Change-Id: I8aa278cebb12b25b39deb0ef584c0f198952659d
2022-06-15Sync from upstream.Jiayu Hu
Descriptions: ====================================================================== Export Icing logging tag to JNI ====================================================================== Update export_to_aosp.sh to change icing log tag to "AppSearchIcing" ====================================================================== Improve the logic of NamespaceChecker. ====================================================================== Step 4.1: Use ScoredDocumentHitsRanker in ResultStateV2 ====================================================================== Step 4.0: Create ScoredDocumentHitsRanker interface and PriorityQueueScoredDocumentHitsRanker ====================================================================== Refactor KeyMapper ====================================================================== Change Icing DEFAULT_LOGGING_LEVEL to INFO ====================================================================== Step 3.4: Create ResultRetrieverV2GroupResultLimiterTest (copied from ResultStateTest) ====================================================================== Add IcingDynamicTrie::IsBranchingTerm to check if a term is branching. ====================================================================== Fix IcingDynamicTrie::Delete bug ====================================================================== Step 3.3: Create ResultRetrieverV2ProjectionTest (copied from ResultRetrieverTest) ====================================================================== Step 3.2: Create ResultRetrieverV2SnippetTest (copied from ResultRetrieverTest) ====================================================================== Step 3.1: Create ResultRetrieverV2Test (copied from ResultRetrieverTest) ====================================================================== Enable legacy multidex ====================================================================== Fix NPE caused by improper handling of return value of GetFileSize. ====================================================================== Step 3.0: Create ResultRetrieverV2 (copied from ResultRetriever) ====================================================================== Step 2: Create PageResult (copied from PageResultState) ====================================================================== Change AppSearch hawkeye testing app multidex to legacy. ====================================================================== Fix icing-search-engine_benchmark bug ====================================================================== (Small fix for step 1) Fix unit test stack memory error ====================================================================== Step 1: Create ResultStateV2 and ResultStateV2Test (copied from ResultState, ResultStateTest) ====================================================================== Bug: 146903474 Bug: 152934343 Bug: 193919210 Bug: 231368517 Bug: 232273174 Bug: 233470404 Bug: 233657885 Change-Id: Iae5461f8650c2bd58f683128fdd243d90403bc06
2022-05-20Sync from upstream.Tim Barron
Descriptions: ====================================================================== Fix bug in schema store where a failure during RegenerateDerivedFiles would lead to a dangling pointer. ====================================================================== Add RAII class that will create and destroy file directories. ====================================================================== Convert MainIndexDebugInfoProto and LiteIndexDebugInfoProto to string ====================================================================== Make SchemaStore move assignable. ====================================================================== Rollback of "convert the string lexicon debug information to a protocol buffer" ====================================================================== Fix NPE caused by a remap failure. ====================================================================== Unify the name "priority" and "severity" in Icing logging ====================================================================== Avoiding string formatting in Icing logging when we should not log ====================================================================== Switch to use an enum with BASIC/DETAILED to control the verbosity of getDebugInfo ====================================================================== Remove the behavior in the Language Segmenter to filter out non-ascii+non-alphanumeric characters. ====================================================================== Fix the SetSchema bug when we override a schema with nested incompatible types ====================================================================== Wrap __android_log_write with __android_is_loggable ====================================================================== Enable removing expired page tokens to free cache space ====================================================================== Bug: 146903474 Bug: 193453081 Bug: 222349894 Bug: 229770338 Bug: 229778472 Bug: 230879098 Bug: 231416401 Bug: 231237897 Bug: 232273174 Change-Id: I22f050de16f56dce39e12a7033947519d598c840
2022-05-03Sync from upstream.Jiayu Hu
Descriptions: ====================================================================== Export Icing logging control to JNI ====================================================================== Prepare Icing logging class for JNI export ====================================================================== Export getDebugInfo to JNI ====================================================================== Expose the return_deleted_document_info parameter for deleteByQuery JNI ====================================================================== Enable runtime log control for Icing Library ====================================================================== Fix 1 ClangTidyBuild finding: ====================================================================== Update comments to run benchmarks. ====================================================================== Making icing's own logging class ====================================================================== Convert the string lexicon debug information to a protocol buffer ====================================================================== Fix issue with printing fingerprinted key in our error logs. ====================================================================== Support dump function for IcingSearchEngine ====================================================================== Bug: 146903474 Bug: 229778472 Bug: 209071710 Bug: 222349894 Bug: 225914361 Change-Id: I9750149d1ed0b59f345b8828ff312a62773667fe
2022-04-12Sync from upstream.Tim Barron
Descriptions: ====================================================================== Add some additional logging that will help diagnose b/218413237 ====================================================================== Mark VerbatimTokenizer::ResetToTokenStartingAfter as 'override'. ====================================================================== Support dump function for SchemaStore ====================================================================== Bug: 218413237 Change-Id: I9efd1dd388cd510df15989c84a8577d4ba56ab3c
2022-04-12Sync from upstream.Tim Barron
====================================================================== Refactor DocumentStore::Initialize to improve readability of document store recovery. ====================================================================== Remove non-NDK API usages of ICU4C in libicing. ====================================================================== Move IcuDataFileHelper to the testing directory since it is a test-only util. ====================================================================== Support dump function for DocumentStore ====================================================================== Switch to use PRead rather than MMap in the proto log. ====================================================================== Support dump function for main/lite index and lexicon ====================================================================== Fix LiteIndex::AppendHits ====================================================================== Enable and fix DocumentStoreTest.LoadScoreCacheAndInitializeSuccessfully ====================================================================== Fix MainIndex::GetStorageInfo. ====================================================================== Fix icing-search-engine_fuzz_test by making IcuLanguageSegmenterIterator::Advance non-recursive. ====================================================================== Allow to return additional information for deleted documents in DeleteByQuery ====================================================================== Using enum class in Token::Type for better type safety. ====================================================================== Bug: 158089703 Bug: 185845269 Bug: 209071710 Bug: 211785521 Bug: 218413237 Bug: 223549255 Change-Id: Id2786047ab279734bdd2aee883e82607b6a0e403
2021-12-28Sync from upstream.Dan Saadati
Descriptions: ================ Normalize Tokens by Token type when retrieving snippets ================ Rename max_window_bytes to max_window_utf32_length, Delete the max_tokens_per_doc field in IcingSearchEngineOptions. ================ Handle suggestion namespace ownership. ================ Fix OkStatus() is not a valid argument to StatusOr in Main_index.RetrieveMoreHits. ================ Allow advancing when current indices are negative in CharacterIterator ================ Adds support for verbatim tokenization and indexing in IcingLib ================ Renames TokenizerIterator Reset functions ================ Add term_match_type to SuggestionSpec proto ================ Unify the C++ proto enum style ================ Allow zero property weights in IcingLib ================ Bug: 204333391 Bug: 152934343 Bug: 205209589 Bug: 206147728 Bug: 209993976 Change-Id: Id94a377fd37c5eb7ebc3d7547cf8ff0ad4152620
2021-10-21Sync from upstream.Tim Barron
Descriptions: ================ Replace refs to c lib headers w/ c++ stdlib equivalents. ================ Update IDF component of BM25F Calculator in IcingLib ================ Expose QuerySuggestions API. ================ Change the tokenizer used in QuerySuggest. ================ Add SectionWeights API to Icing. ================ Apply SectionWeights to BM25F Scoring. ================ Replaces uses of u_strTo/FromUTF32 w/ u_strTo/FromUTF8. Bug: 152934343 Bug: 202308641 Bug: 203700301 Change-Id: Ic884a84e5ff4c9c04b2cd6dd1fce90765aa4446e
2021-09-13Sync from upstream.Tim Barron
Descriptions: ================ Implement ranking in FindTermByPrefix. ================ Fork proto's GzipStream into Icing Lib. ================ Remove token limit behavior from index-processor. ================ Bug: 152934343 Bug: 147509515 Bug: 196771754 Change-Id: I7f95392afc97001df0ae3f60720743c74d64ba84
2021-09-08Sync from upstream.My Name
Descriptions: ================ Remove no-longer-used write paths for file-backed-proto-log. ================ Modify segmentation rules to consider any segment that begins with a non-Ascii alphanumeric character as valid ================= Implement CalculateNormalizedMatchLength for IcuNormalizer. ================ Add additional benchmark cases that were useful in developing submatching and CalculateNormalizedMatchLength for IcuNormalizer ================= Switch NormalizationMap from static const std::unordered_map<char16_t, char16_t>& to static const std::unordered_map<char16_t, char16_t> *const. ================== Bug: 147509515 Bug: 149610413 Bug: 195720764 Bug: 196257995 Change-Id: Iabdb34a983b5d47daca808888a46c241767d93bf
2021-08-13Sync from upstream.Tim Barron
Descriptions: ================== Add Initialization Count marker file to break out of crash loops. ================== Delete the Simple Normalizer. ================== Add submatch information to identify a submatch within a document. ================== Bug: 195720764 Change-Id: I27bcc66c400da625d595609049d281b809dd6028
2021-08-13Merge androidx-platform-dev/external/icing upstream-master into upstream-masterAlexander Dorokhine
Change-Id: Id39bea14b3a3d378d722dc552e4f3bd4249f3f94
2021-03-03Sync from upstream.Cassie Wang
Descriptions: ========== New java package name for internal/optimize.proto ========== Migrate IcingSearchEngineWithIcuFileTest and IcingSearchEngineFuzzTest to use SchemaBuilder Change-Id: I4113bf2d7cff536538e45c3afea2c4ced2dd29ee
2021-03-02Sync from upstream.Cassie Wang
Descriptions: ========== Add last optimized time to GetOptimizeInfo. ========== Update the implementation of snippeting to return property paths with value indices and remove the values_index field. ========== Create builders for SchemaProto, SchemaTypeConfigProto and PropertyConfigProto. ========== Rename some protos with the rules: - Remove "Native" prefix - Add "Proto" suffix for consistency with other protos ========== Upgrade your minimum iOS version to 11.4. ========== Fix PersistToDisk definitions to ensure that they properly call datasync. This change is meant to address the first part of ptd doc - that certain functions that claim to persist data don't actually explicitly flush. ========== Change function call from has_field() to field() ========== Add IcingStorageInfo. ========== Add IndexStorageStats. ========== Add SchemaStoreStorageStats. ========== Add DocumentStoreStorageStats. ========== Implement OptimizeStats. ========== Remove the max number of results per query limit (1000) and replace it with a more flexible way to limit memory use by the result-state-manager. ========== Add a test case to ensure we don't add UsageStore's checksum in DocumentStore's ComputeChecksum. ========== Account for UsageStore in GetDiskUsage. ========== Ensure that SchemaStore properly handles function calls when the schema isn't set. ========== Remove jlpl_strict_deps feature from package declarations. ========== Qualifies std::string in 3p directories ========== Section restricts should influence the relevance score. ========== Apply fixes upstream that were necessary to sync changes downstream. Also added a METADATA check to prevent any accidental adds of foo.proto.h includes. ========== Remove the 'com.google.protobuf' to 'com.google.android.icing.protobuf' translation in the export_to_aosp script. ========== Include usage store size in GetOptimizeInfo. This helps clients get a better idea of what savings they could get back if they called Optimize. Change-Id: Ia2339c7987267a73c49dadf1ced4a0a8ef001d4c
2021-01-14Update Icing from upstream.Tim Barron
Change-Id: I43038a59e7170fb8ecbaf6098a37221b3682ce09
2021-01-05Update Icing from upstream.Tim Barron
Change-Id: Iff50aebffb83529b0454e7c3a6dc6864e7a85f4a
generated by cgit v1.2.3 (git 2.25.1) at 2024-06-30 06:59:14 +0000