Update Icing from upstream.

Descriptions:
======================================================================
Add PropertyUtil for all property name/path related operations
======================================================================
[JoinableCache][2.0/x] Create SchemaPropertyIterator
======================================================================
[JoinableCache][2.1/x] Handle nested indexable flag
======================================================================
[JoinableCache][2.2/x] Add schema cycle dependency detection for SchemaPropertyIterator
======================================================================
[JoinableCache][3.0/x] Refactor SectionManager
======================================================================
[JoinableCache][3.1/x] Add unit tests for SectionManager::Builder and SchemaTypeManager
======================================================================
[NumericSearch][Storage][12/x] Implement Edit and GetIterator for IntegerIndex
======================================================================
[NumericSearch][Storage][13.0/x] Rename numeric-index_test as integer-index_test
======================================================================
[NumericSearch][Storage][13.1/x] Add IntegerIndexTest
======================================================================
Support the "len", "sum" and "avg" functions in advanced scoring.
======================================================================
Support the "this.childrenScores()" function to allow expressing children scores of joins in advanced scoring.
======================================================================
Create an integration test for Join with advanced scoring
======================================================================
Rename the word "children" to "args" for function related ScoreExpression
======================================================================
Improve IndexBlock by PRead/PWrite instead of repeating mmap/msync/unmap
======================================================================
Refactor QueryVisitor to prepare for support for function calls.
======================================================================
Add support for function calls.
======================================================================
Fix breakage in score-and-rank_benchmark.
======================================================================
[NumericSearch][Storage][adhoc][ez] Fix comment for IntegerIndex
======================================================================
[NumericSearch][Storage][14/x] Create first IntegerIndexStorage benchmark
======================================================================
Rename Icing schema related terminology to prepare for polymorphism support
======================================================================
[JoinableCache][4.0/x] Move common methods from SectionManager to PropertyUtil
======================================================================
[JoinableCache][4.1/x] Retire GetSectionContent
======================================================================
[JoinableCache][4.2/x] Polish SectionManagerTest
======================================================================
Modify QueryVisitor to do:
======================================================================
[NumericSearch][Storage][15/x] Implement TransferIndex for IntegerIndexStorage
======================================================================
[NumericSearch][Storage][16/x] Implement Optimize and last added document id for IntegerIndex
======================================================================
[NumericSearch][rollout][1/x] Include indexable int64 into SchemaDelta and backward compatibility
======================================================================
Add backwards compatibility test for Icing schema storage migration.
======================================================================
Implement trim the right-most node from the doc-hit-info-iterator.
======================================================================
Add TrimmedNode structure into doc-hit-info-iterator.
======================================================================
[JoinableCache][5/x] Implement JoinableProperty and JoinablePropertyManager
======================================================================
[JoinableCache][6/x] Add JoinablePropertyManager into SchemaTypeManager
======================================================================
[JoinableCache][7/x] Implement ExtractJoinableProperties
======================================================================
[JoinableCache][8/x] Create class QualifiedIdTypeJoinableCache
======================================================================
[JoinableCache][9/x] Implement factory method for QualifiedIdTypeJoinableCache
======================================================================
[JoinableCache][10/x] Implement Get and Put for QualifiedIdTypeJoinableCache
======================================================================
[JoinableCache][11/x] Add unit tests for QualifiedIdTypeJoinableCache
======================================================================
Modify DocHitInfoIteratorSectionRestrict to allow multi-property restricts
======================================================================
Fix the definition of LiteIndex::WantsMerge.
======================================================================
[NumericSearch][rollout][2.0/x] Rollout persistent IntegerIndex
======================================================================
[NumericSearch][rollout][2.1/x] Add more tests for integer index restoration and optimization
======================================================================
[JoinableCache][adhoc][ez] Remove qualified id type joinable cache size info from document storage info
======================================================================
Integrate trim right node into suggestion processor.



Bug: 208654892
Bug: 228240987
Bug: 249829533
Bug: 256081830
Bug: 259744228
Bug: 261474063
Bug: 263890397
Bug: 266103594
Bug: 268738297
Bug: 269295094
Change-Id: I5f1b3f3ed0b5d6933dc8c2ab3279904f7706b23e

1 files changed, 39 insertions, 18 deletions

diff --git a/icing/index/main/main-index_test.cc b/icing/index/main/main-index_test.cc
index 816d46e..ac724b0 100644
--- a/icing/index/main/main-index_test.cc
+++ b/icing/index/main/main-index_test.cc
@@ -52,18 +52,20 @@ std::vector<DocHitInfo> GetHits(std::unique_ptr<DocHitInfoIterator> iterator) {
 }
 
 std::vector<DocHitInfo> GetExactHits(
-    MainIndex* main_index, const std::string& term,
-    SectionIdMask section_mask = kSectionIdMaskAll) {
+    MainIndex* main_index, int term_start_index, int unnormalized_term_length,
+    const std::string& term, SectionIdMask section_mask = kSectionIdMaskAll) {
   auto iterator = std::make_unique<DocHitInfoIteratorTermMainExact>(
-      main_index, term, section_mask, /*need_hit_term_frequency=*/true);
+      main_index, term, term_start_index, unnormalized_term_length,
+      section_mask, /*need_hit_term_frequency=*/true);
   return GetHits(std::move(iterator));
 }
 
 std::vector<DocHitInfo> GetPrefixHits(
-    MainIndex* main_index, const std::string& term,
-    SectionIdMask section_mask = kSectionIdMaskAll) {
+    MainIndex* main_index, int term_start_index, int unnormalized_term_length,
+    const std::string& term, SectionIdMask section_mask = kSectionIdMaskAll) {
   auto iterator = std::make_unique<DocHitInfoIteratorTermMainPrefix>(
-      main_index, term, section_mask, /*need_hit_term_frequency=*/true);
+      main_index, term, term_start_index, unnormalized_term_length,
+      section_mask, /*need_hit_term_frequency=*/true);
   return GetHits(std::move(iterator));
 }
 
@@ -271,9 +273,12 @@ TEST_F(MainIndexTest, MergeIndexToEmpty) {
       MainIndex::Create(main_index_file_name, &filesystem_,
                         &icing_filesystem_));
 
-  std::vector<DocHitInfo> hits = GetExactHits(main_index.get(), "foot");
+  std::vector<DocHitInfo> hits =
+      GetExactHits(main_index.get(), /*term_start_index=*/0,
+                   /*unnormalized_term_length=*/0, "foot");
   EXPECT_THAT(hits, IsEmpty());
-  hits = GetPrefixHits(main_index.get(), "fo");
+  hits = GetPrefixHits(main_index.get(), /*term_start_index=*/0,
+                       /*unnormalized_term_length=*/0, "fo");
   EXPECT_THAT(hits, IsEmpty());
 
   // 3. Merge the index. The main index should contain "fool", "foot"
@@ -281,7 +286,8 @@ TEST_F(MainIndexTest, MergeIndexToEmpty) {
   // should not be present because it is not a branch point.
   ICING_ASSERT_OK(Merge(*lite_index_, *term_id_codec_, main_index.get()));
   // Get hits from an exact posting list.
-  hits = GetExactHits(main_index.get(), "foot");
+  hits = GetExactHits(main_index.get(), /*term_start_index=*/0,
+                      /*unnormalized_term_length=*/0, "foot");
   // We should get hits for "foot" in doc1 and doc0
   EXPECT_THAT(
       hits,
@@ -292,7 +298,8 @@ TEST_F(MainIndexTest, MergeIndexToEmpty) {
                            std::vector<SectionId>{doc0_hit.section_id()})));
 
   // Get hits from a branching point posting list. "fo" should redirect to "foo"
-  hits = GetPrefixHits(main_index.get(), "fo");
+  hits = GetPrefixHits(main_index.get(), /*term_start_index=*/0,
+                       /*unnormalized_term_length=*/0, "fo");
   // We should get hits for "foot" in doc1 and "fool" in doc1. We shouldn't get
   // the hits for "foot" in doc0 and "fool" in doc0 and doc2 because they
   // weren't hits in prefix sections.
@@ -390,7 +397,9 @@ TEST_F(MainIndexTest, MergeIndexToPreexisting) {
   // and "fall", a branch points for "fou" and backfill points for "fo".
   ICING_ASSERT_OK(Merge(*lite_index_, *term_id_codec_, main_index.get()));
   // Get hits from an exact posting list the existed before the merge.
-  std::vector<DocHitInfo> hits = GetExactHits(main_index.get(), "foot");
+  std::vector<DocHitInfo> hits =
+      GetExactHits(main_index.get(), /*term_start_index=*/0,
+                   /*unnormalized_term_length=*/0, "foot");
 
   // We should get hits for "foot" in doc3, doc1 and doc0
   EXPECT_THAT(
@@ -403,7 +412,8 @@ TEST_F(MainIndexTest, MergeIndexToPreexisting) {
           EqualsDocHitInfo(doc0_hit.document_id(),
                            std::vector<SectionId>{doc0_hit.section_id()})));
   // Get hits from backfill posting list.
-  hits = GetPrefixHits(main_index.get(), "fo");
+  hits = GetPrefixHits(main_index.get(), /*term_start_index=*/0,
+                       /*unnormalized_term_length=*/0, "fo");
   // We should get hits for "four" and "foul" in doc4 and hits for "foot" and
   // "fool" in doc1. We shouldn't get the hits for "foot" in doc0 and doc3,
   // "fool" in doc0 and doc2 or the hits for "four" and "foul" in doc4 because
@@ -455,7 +465,9 @@ TEST_F(MainIndexTest, ExactRetrievedInPrefixSearch) {
   // 3. Merge the lite lexicon. The main lexicon should contain "foot" and
   // "foo".
   ICING_ASSERT_OK(Merge(*lite_index_, *term_id_codec_, main_index.get()));
-  std::vector<DocHitInfo> hits = GetPrefixHits(main_index.get(), "foo");
+  std::vector<DocHitInfo> hits =
+      GetPrefixHits(main_index.get(), /*term_start_index=*/0,
+                    /*unnormalized_term_length=*/0, "foo");
   // We should get hits for "foo" in doc1 and doc0, but not in doc2 because it
   // is not a prefix hit.
   EXPECT_THAT(
@@ -504,7 +516,9 @@ TEST_F(MainIndexTest, PrefixNotRetrievedInExactSearch) {
   // 3. Merge the lite lexicon. The main lexicon should contain "foot" and
   // "foo".
   ICING_ASSERT_OK(Merge(*lite_index_, *term_id_codec_, main_index.get()));
-  std::vector<DocHitInfo> hits = GetExactHits(main_index.get(), "foo");
+  std::vector<DocHitInfo> hits =
+      GetExactHits(main_index.get(), /*term_start_index=*/0,
+                   /*unnormalized_term_length=*/0, "foo");
 
   // We should get hits for "foo" in doc2 and doc1, but not in doc0 because it
   // is not an exact hit.
@@ -554,7 +568,9 @@ TEST_F(MainIndexTest, SearchChainedPostingLists) {
   // 3. Merge the lite index.
   ICING_ASSERT_OK(Merge(*lite_index_, *term_id_codec_, main_index.get()));
   // Get hits for all documents containing "foot" - which should be all of them.
-  std::vector<DocHitInfo> hits = GetExactHits(main_index.get(), "foot");
+  std::vector<DocHitInfo> hits =
+      GetExactHits(main_index.get(), /*term_start_index=*/0,
+                   /*unnormalized_term_length=*/0, "foot");
 
   EXPECT_THAT(hits, SizeIs(2048));
   EXPECT_THAT(hits.front(),
@@ -607,11 +623,14 @@ TEST_F(MainIndexTest, MergeIndexBackfilling) {
   // and a backfill point for "foo".
   ICING_ASSERT_OK(Merge(*lite_index_, *term_id_codec_, main_index.get()));
   // Get hits from an exact posting list the existed before the merge.
-  std::vector<DocHitInfo> hits = GetExactHits(main_index.get(), "foo");
+  std::vector<DocHitInfo> hits =
+      GetExactHits(main_index.get(), /*term_start_index=*/0,
+                   /*unnormalized_term_length=*/0, "foo");
   EXPECT_THAT(hits, IsEmpty());
 
   // Get hits from backfill posting list.
-  hits = GetPrefixHits(main_index.get(), "foo");
+  hits = GetPrefixHits(main_index.get(), /*term_start_index=*/0,
+                       /*unnormalized_term_length=*/0, "foo");
   // We should get a hit for "fool" in doc0.
   EXPECT_THAT(hits, ElementsAre(EqualsDocHitInfo(
                         doc0_hit.document_id(),
@@ -642,7 +661,9 @@ TEST_F(MainIndexTest, OneHitInTheFirstPageForTwoPagesMainIndex) {
                         &icing_filesystem_));
 
   ICING_ASSERT_OK(Merge(*lite_index_, *term_id_codec_, main_index.get()));
-  std::vector<DocHitInfo> hits = GetExactHits(main_index.get(), "foo");
+  std::vector<DocHitInfo> hits =
+      GetExactHits(main_index.get(), /*term_start_index=*/0,
+                   /*unnormalized_term_length=*/0, "foo");
   ASSERT_THAT(hits, SizeIs(num_docs));
   for (DocumentId document_id = num_docs - 1; document_id >= 0; --document_id) {
     ASSERT_THAT(