licensing: support per package build time licensing generation.

Removed gather-licenses.sh process-pkg.py (unfinished prototypes) now
that they are replaced by this script.

- updated documentation in comments

- cleaned up yaml import as recommended by David.

- ebuild_dir and description were unused in PackageInfo. Removed them.

- build_source_tree is used to create a PackageInfo from an unpackage
  package (as opposed to the other way that queries an ebuild).

- _BuildInfo was added to retrieve package information from the
  build-info fork present at package build time when called as a hook.

- Licensing object can now be initialized without visibility on the
  template files (when run as a hook), which is fine since they are not
  used anyway.

- Per package license bits are now saved/loaded in
  /var/db/pkg/cat/name/license.yaml to match the location we get from
  portage when we install a prebuilt.

- HookPackageProcess is a new simpler way to enter/generate a license
  object and output a per package license bit in its build-info
  directory.

- If a licensing run realizes that per package licensing bits are
  missing, it creates them on the fly, but only if it is run under sudo
  (of course that also slows down image generation from <20sec to
  minutes).

BUG=chromium:197970 chromium:271832 chromium:207004 chromium:356539

TEST=(cr) (license-generation) merlin@polgara ~/trunk/chromite/licensing $ FEATURES='noclean' emerge-x86-alex libc-bench
Calculating dependencies... done!

>>> Emerging (1 of 1) dev-util/libc-bench-0.0.1-r8 from chromiumos for /build/x86-alex/
(...)
File not built with -Wl,-z,now: /build/x86-alex/tmp/portage/dev-util/libc-bench-0.0.1-r8/image/usr/local/libc-bench/libc-bench
Generating license for dev-util/libc-bench-0.0.1-r8 in /build/x86-alex/tmp/portage/dev-util/libc-bench-0.0.1-r8
13:34:39: INFO: Read licenses for dev-util/libc-bench-0.0.1-r8: MIT
13:34:39: INFO: dev-util/libc-bench-0.0.1-r8: can't use MIT, will scan source code for copyright
13:34:39: INFO: License(s) for dev-util/libc-bench-0.0.1-r8: COPYRIGHT
13:34:39: INFO: Adding License /build/x86-alex/tmp/portage/dev-util/libc-bench-0.0.1-r8/work/libc-bench/COPYRIGHT (UTF-8)
strip: i686-pc-linux-gnu-strip --strip-unneeded -R .comment -R .GCC.command.line
(...)
>>> Auto-cleaning packages...

>>> Using system located in ROOT tree /build/x86-alex/

>>> No outdated packages were found on your system.
(cr) (license-generation) merlin@polgara ~/trunk/chromite/licensing $ qtbz2 -x -O  /build/x86-alex/packages/dev-util/libc-bench-0.0.1-r8.tbz2 | qxpak -x -O - license.yaml
- [category, !!python/unicode 'dev-util']
- [licensing_failed, false]
- [name, !!python/unicode 'libc-bench']
- - ebuild_license_names
  - [!!python/unicode 'MIT']
- - homepages
  - [!!python/unicode 'http://www.etalabs.net/libc-bench.html', !!python/unicode 'http://git.musl-libc.org/cgit/libc-bench/']
- [ebuild_path, null]
- [need_copyright_attribution, true]
- - license_names
  - !!set {}
- [version, !!python/unicode '0.0.1']
- [board, /build/x86-alex]
- [skip, false]
- [scan_source_for_licenses, true]
- - license_text_scanned
  - ["Scanned Source License COPYRIGHT:\n\nlibc-bench, a performance and memory usage\
      \ benchmark for comparing\nlibc implementations\n\nCopyright \xA9 2011 Rich\
      \ Felker\n\nPermission is hereby granted, free of charge, to any person obtaining\n\
      a copy of this software and associated documentation files (the\n\"Software\"\
      ), to deal in the Software without restriction, including\nwithout limitation\
      \ the rights to use, copy, modify, merge, publish,\ndistribute, sublicense,\
      \ and/or sell copies of the Software, and to\npermit persons to whom the Software\
      \ is furnished to do so, subject to\nthe following conditions:\n\nThe above\
      \ copyright notice and this permission notice shall be\nincluded in all copies\
      \ or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS\
      \ IS\", WITHOUT WARRANTY OF ANY KIND,\nEXPRESS OR IMPLIED, INCLUDING BUT NOT\
      \ LIMITED TO THE WARRANTIES OF\nMERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE\
      \ AND NONINFRINGEMENT.\nIN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE\
      \ LIABLE FOR ANY\nCLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF\
      \ CONTRACT,\nTORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE\n\
      SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n"]
- [build_source_tree, /build/x86-alex/tmp/portage/dev-util/libc-bench-0.0.1-r8]
- [revision, '8']
(cr) (license-generation) merlin@polgara ~/trunk/chromite/licensing $ ls -l /build/x86-alex/var/db/pkg/dev-util/libc-bench-0.0.1-r8/license.yaml
-rw-r--r-- 1 root root 1983 Apr 10 13:34 /build/x86-alex/var/db/pkg/dev-util/libc-bench-0.0.1-r8/license.yaml

Change-Id: I787edaaf63c848e0e2dd76dcd687c8358c92152a
Reviewed-on: https://chromium-review.googlesource.com/194178
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Marc MERLIN <merlin@chromium.org>
Tested-by: Marc MERLIN <merlin@chromium.org>

3 files changed, 261 insertions, 487 deletions

diff --git a/licensing/gather-licenses.sh b/licensing/gather-licenses.sh
deleted file mode 100755
index 0384e4012..000000000
--- a/licensing/gather-licenses.sh
+++ /dev/null
@@ -1,19 +0,0 @@
-#!/bin/bash
-#
-# Copyright (c) 2012 The Chromium OS Authors. All rights reserved.
-# Use of this source code is governed by a BSD-style license that can be
-# found in the LICENSE file.
-
-source "${T}/environment" || { echo "failed sourcing ${T}/environment"; exit 1;}
-if [[ "${QA_LICENSE_IGNORE}" == "yes" ]]; then
-    echo "license/copyright gathering suppressed for ${CATEGORY}/${PN}-${PVR}"
-    exit 0
-fi
-
-# This will be enabled when it's finished and tested.
-#$(dirname $(readlink -f $0))/licenses/process-pkg.py \
-#    "${CATEGORY}/${PN}-${PVR}" \
-#    "${CHROME_FORCE_LICENSE-${LICENSE}}" \
-#    "${HOMEPAGE}" "${D}" "${WORKDIR}" "${PORTDIR}/licenses"
-#exit $(( $? ))
-exit 0
diff --git a/licensing/licenses.py b/licensing/licenses.py
index af8631049..6275dd89b 100644
--- a/licensing/licenses.py
+++ b/licensing/licenses.py
@@ -9,6 +9,9 @@
 Documentation on this script is also available here:
 http://www.chromium.org/chromium-os/licensing-for-chromiumos-developers
 
+End user (i.e. package owners) documentation is here:
+http://www.chromium.org/chromium-os/licensing-for-chromiumos-package-owners
+
 Usage:
 For this script to work, you must have built the architecture
 this is being run against, _after_ you've last run repo sync.
@@ -17,9 +20,8 @@ that are out of date in your build.
 
 Recommended build:
   cros_sdk
-  export board=x86-alex
-  sudo rm -rf /build/$board
-  sudo install -o $(whoami) -d /build/x86-alex//var/lib/licenses/
+  export BOARD=x86-alex
+  sudo rm -rf /build/$BOARD
   cd ~/trunk/src/scripts
   # If you wonder why we need to build Chromium OS just to run
   # `emerge -p -v virtual/target-os` on it, we don't.
@@ -27,25 +29,49 @@ Recommended build:
   # configure. Configure will fail due to aclocal macros missing in
   # /build/x86-alex/usr/share/aclocal (those are generated during build).
   # This will take about 10mn on a Z620.
-  ./build_packages --board=$board --nowithautotest --nowithtest --nowithdev \
+  ./build_packages --board=$BOARD --nowithautotest --nowithtest --nowithdev
                    --nowithfactory
   cd ~/trunk/chromite/licensing
   # This removes left over packages from an earlier build that could cause
   # conflicts.
-  eclean-$board packages
-  %(prog)s [--debug] [--all-packages] --board $board -o out.html 2>&1 | tee out
+  eclean-$BOARD packages
+  %(prog)s [--debug] [--all-packages] --board $BOARD [-o o.html] 2>&1 | tee out
+
+The workflow above is what you would do to generate a licensing file by hand
+given a chromeos tree.
+Note that building packages now creates a license.yaml fork in the package
+which you can see with
+qtbz2 -x -O  /build/x86-alex/packages/dev-util/libc-bench-0.0.1-r8.tbz2 |
+     qxpak -x -O - license.yaml
+This gets automatically installed in
+/build/x86-alex/var/db/pkg/dev-util/libc-bench-0.0.1-r8/license.yaml
+
+Unless you run with --generate, the script will now gather those license
+bits and generate a license file from there.
+License bits for each package are generated by default from
+src/scripts/hooks/install/gen-package-licenses.sh which gets run automatically
+by emerge as part of a package build (by running this script with
+--hook /path/to/tmp/portage/build/tree/for/that/package
+
+If license bits are missing, they are generated on the fly if you were running
+with sudo. If you didn't use sudo, this on the fly late generation will fail
+and act as a warning that your prebuilts were missing package build time
+licenses.
 
 You can check the licenses and/or generate a HTML file for a list of
 packages using --package or -p:
   %(prog)s --package "dev-libs/libatomic_ops-7.2d" --package
-  "net-misc/wget-1.14" --board $board -o out.html
+  "net-misc/wget-1.14" --board $BOARD -o out.html
+
+Note that you'll want to use --generate to force regeneration of the licensing
+bits from a package source you may have just modified but not rebuilt.
 
 If you want to check licensing against all ChromeOS packages, you should
-run ./build_packages --board=$board to build everything and then run
+run ./build_packages --board=$BOARD to build everything and then run
 this script with --all-packages.
 
 By default, when no package is specified, this script processes all
-packages for the board. The output HTML file is meant to update
+packages for $BOARD. The output HTML file is meant to update
 http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/resources/ +
   chromeos/about_os_credits.html?view=log
 (gclient config svn://svn.chromium.org/chrome/trunk/src)
@@ -88,8 +114,10 @@ Once it's been updated to "Merge-Approved" by a TPM, please merge into the
 required release branch. You can ask karen@ for merge approve help.
 Example: http://crbug.com/221281
 
-Usage: %(prog)s [opts] <board>
-
+Note however that this is only during the transition period.
+build-image will be modified to generate the license for each board and save
+the file in /opt/google/chrome/resources/about_os_credits.html or as defined
+in http://crbug.com/271832 .
 """
 
 import cgi
@@ -105,11 +133,18 @@ from chromite.lib import commandline
 from chromite.lib import cros_build_lib
 from chromite.lib import osutils
 
+# We are imported by src/repohooks/pre-upload.py in a non chroot environment
+# where yaml may not be there, so we don't error on that since it's not needed
+# in that case.
+try:
+  import yaml
+except ImportError:
+  yaml = None
 
 debug = False
 
 # See http://crbug.com/207004 for discussion.
-PER_PKG_LICENSE_DIR = '/var/lib/licenses'
+PER_PKG_LICENSE_DIR = '/var/db/pkg'
 
 STOCK_LICENSE_DIRS = [
     os.path.join(constants.SOURCE_ROOT,
@@ -237,6 +272,7 @@ ENTRY_TMPL = 'about_credits_entry.tmpl'
 SHARED_LICENSE_TMPL = 'about_credits_shared_license_entry.tmpl'
 
 
+# This is called directly by src/repohooks/pre-upload.py
 def GetLicenseTypesFromEbuild(ebuild_path):
   """Returns a list of license types from the ebuild file.
 
@@ -326,12 +362,9 @@ class PackageInfo(object):
     # /mnt/host/source/src/
     #           third_party/portage-stable/net-misc/rsync/rsync-3.0.8.ebuild
     self.ebuild_path = None
-    # dirname of ebuild_path.
-    self.ebuild_dir = None
 
     # Array of license names retrieved from ebuild or override in this code.
     self.ebuild_license_names = []
-    self.description = None
     self.homepages = []
     # This contains licenses names we can read from Gentoo or custom licenses.
     # These are supposed to be shared licenses (i.e. licenses referenced by
@@ -357,6 +390,12 @@ class PackageInfo(object):
     # it can be flagged when the full license file is being generated.
     self.licensing_failed = False
 
+    # If we are called from a hook, we grab package info from the soure tree.
+    # This is also used as a flag to know whether we should do package work
+    # based on an installed package, or one that is being built and we got
+    # called from the hook.
+    self.build_source_tree = None
+
   @property
   def fullnamerev(self):
     s = '%s-%s' % (self.fullname, self.version)
@@ -368,6 +407,22 @@ class PackageInfo(object):
   def fullname(self):
     return '%s/%s' % (self.category, self.name)
 
+  @property
+  def license_dump_path(self):
+    """e.g. /build/x86-alex//var/db/pkg/sys-apps/dtc-1.4.0/license.yaml."""
+    return "%s/%s/%s/license.yaml" % (cros_build_lib.GetSysroot(self.board),
+                                      PER_PKG_LICENSE_DIR, self.fullnamerev)
+
+  def _BuildInfo(self, filename):
+    filename = '%s/build-info/%s' % (self.build_source_tree, filename)
+    # Buildinfo properties we read are in US-ASCII, not Unicode.
+    try:
+      bi = open(filename).read().rstrip()
+    # Some properties like HOMEPAGE may be absent.
+    except IOError:
+      bi = ""
+    return bi
+
   def _RunEbuildPhases(self, phases):
     """Run a list of ebuild phases on an ebuild.
 
@@ -447,36 +502,39 @@ class PackageInfo(object):
       self.license_text_scanned = [license_override]
       return
 
-    self._RunEbuildPhases(['clean', 'fetch'])
-    output = self._RunEbuildPhases(['unpack']).output.splitlines()
-    # Output is spammy, it looks like this:
-    #  * gc-7.2d.tar.gz RMD160 SHA1 SHA256 size ;-) ...                  [ ok ]
-    #  * checking gc-7.2d.tar.gz ;-) ...                                 [ ok ]
-    #  * Running stacked hooks for pre_pkg_setup
-    #  *    sysroot_build_bin_dir ...
-    #  [ ok ]
-    #  * Running stacked hooks for pre_src_unpack
-    #  *    python_multilib_setup ...
-    #  [ ok ]
-    # >>> Unpacking source...
-    # >>> Unpacking gc-7.2d.tar.gz to /build/x86-alex/tmp/po/[...]tops-7.2d/work
-    # >>> Source unpacked in /build/x86-alex/tmp/portage/[...]ops-7.2d/work
-    # So we only keep the last 2 lines, the others we don't care about.
-    output = [line for line in output if line[0:3] == ">>>" and
-              line != ">>> Unpacking source..."]
-    for line in output:
-      logging.info(line)
-
-    args = ['portageq-%s' % self.board, 'envvar', 'PORTAGE_TMPDIR']
-    result = cros_build_lib.RunCommand(args, print_cmd=debug,
-                                       redirect_stdout=True)
-    tmpdir = result.output.splitlines()[0]
-    # tmpdir gets something like /build/daisy/tmp/
-    workdir = os.path.join(tmpdir, 'portage', self.fullnamerev, 'work')
-
-    if not os.path.exists(workdir):
-      raise AssertionError("Unpack of %s didn't create %s. Version mismatch?" %
-                           (self.fullnamerev, workdir))
+    if self.build_source_tree:
+      workdir = "%s/work" % self.build_source_tree
+    else:
+      self._RunEbuildPhases(['clean', 'fetch'])
+      output = self._RunEbuildPhases(['unpack']).output.splitlines()
+      # Output is spammy, it looks like this:
+      #  * gc-7.2d.tar.gz RMD160 SHA1 SHA256 size ;-) ...                 [ ok ]
+      #  * checking gc-7.2d.tar.gz ;-) ...                                [ ok ]
+      #  * Running stacked hooks for pre_pkg_setup
+      #  *    sysroot_build_bin_dir ...
+      #  [ ok ]
+      #  * Running stacked hooks for pre_src_unpack
+      #  *    python_multilib_setup ...
+      #  [ ok ]
+      # >>> Unpacking source...
+      # >>> Unpacking gc-7.2d.tar.gz to /build/x86-alex/tmp/po/[...]ps-7.2d/work
+      # >>> Source unpacked in /build/x86-alex/tmp/portage/[...]ops-7.2d/work
+      # So we only keep the last 2 lines, the others we don't care about.
+      output = [line for line in output if line[0:3] == ">>>" and
+                line != ">>> Unpacking source..."]
+      for line in output:
+        logging.info(line)
+
+      args = ['portageq-%s' % self.board, 'envvar', 'PORTAGE_TMPDIR']
+      result = cros_build_lib.RunCommand(args, print_cmd=debug,
+                                         redirect_stdout=True)
+      tmpdir = result.output.splitlines()[0]
+      # tmpdir gets something like /build/daisy/tmp/
+      workdir = os.path.join(tmpdir, 'portage', self.fullnamerev, 'work')
+
+      if not os.path.exists(workdir):
+        raise AssertionError("Unpack of %s didn't create %s. Version mismatch" %
+                             (self.fullnamerev, workdir))
 
     # You may wonder how deep should we go?
     # In case of packages with sub-packages, it could be deep.
@@ -491,15 +549,20 @@ class PackageInfo(object):
     files = [x[len(workdir):].lstrip('/') for x in result]
     license_files = []
     for name in files:
+      # When we scan a source tree managed by git, this can contain license
+      # files that are not part of the source. Exclude those.
+      # (e.g. .git/refs/heads/licensing)
+      if ".git/" in name:
+        continue
       basename = os.path.basename(name)
+      # Looking for license.* brings up things like license.gpl, and we
+      # never want a GPL license when looking for copyright attribution,
+      # so we skip them here. We also skip regexes that can return
+      # license.py (seen in some code).
+      if re.search(r".*GPL.*", basename) or re.search(r"\.py$", basename):
+        continue
       for regex in LICENSE_NAMES_REGEX:
-        if (re.search(regex, basename, re.IGNORECASE) and
-            # Looking for license.* brings up things like license.gpl, and we
-            # never want a GPL license when looking for copyright attribution,
-            # so we skip them here. We also skip regexes that can return
-            # license.py (seen in some code).
-            not re.search(r".*GPL.*", basename) and
-            not re.search(r"\.py$", basename)):
+        if re.search(regex, basename, re.IGNORECASE):
           license_files.append(name)
           break
 
@@ -557,7 +620,7 @@ being scraped currently).""",
     # self._RunEbuildPhases(['clean'])
 
   def GetPackageInfo(self, fullnamewithrev):
-    """Populate PackageInfo with package license, homepage and description.
+    """Populate PackageInfo with package license, and homepage.
 
     self.ebuild_license_names will not be filled if the package is skipped
     or if there was an issue getting data from the ebuild.
@@ -574,6 +637,14 @@ being scraped currently).""",
     Raises:
       AssertionError: on runtime errors
     """
+    if not fullnamewithrev:
+      if not self.build_source_tree:
+        raise AssertionError("Cannot continue without full name or source tree")
+      fullnamewithrev = "%s/%s" % (self._BuildInfo("CATEGORY"),
+                                   self._BuildInfo("PF"))
+      logging.debug("Computed package name %s from %s", fullnamewithrev,
+                    self.build_source_tree)
+
     try:
       cpv = portage_utilities.SplitCPV(fullnamewithrev)
       # A bad package can either raise a TypeError exception or return None,
@@ -602,19 +673,8 @@ being scraped currently).""",
       self.skip = True
       return
 
-  def GetLicenses(self):
-    """Get licenses from the ebuild field and the unpacked source code.
-
-    Some packages have static license mappings applied to them that get
-    retrieved from the ebuild.
-
-    For others, we figure out whether the package source should be scanned to
-    add licenses found there.
-
-    Raises:
-      AssertionError: on runtime errors
-      PackageLicenseError: couldn't find license in ebuild and source.
-    """
+  def _ReadEbuildInfo(self):
+    """Populate package info from an ebuild retrieved via equery."""
     # By default, equery returns the latest version of the package. A
     # build may have used an older version than what is currently
     # available in the source tree (a build dependency can be pinned
@@ -638,23 +698,41 @@ being scraped currently).""",
     if not os.access(path, os.F_OK):
       raise AssertionError("Can't access %s", path)
 
-    self.ebuild_dir = os.path.dirname(path)
     self.ebuild_path = path
 
     args = ['portageq-%s' % self.board, 'metadata',
             cros_build_lib.GetSysroot(board=self.board), 'ebuild',
-            self.fullnamerev, 'HOMEPAGE', 'LICENSE', 'DESCRIPTION']
-    lines = cros_build_lib.RunCommand(args, print_cmd=debug,
-                                      redirect_stdout=True).output.splitlines()
+            self.fullnamerev, 'HOMEPAGE', 'LICENSE']
+    tmp = cros_build_lib.RunCommand(args, print_cmd=debug,
+                                    redirect_stdout=True)
+    lines = tmp.output.splitlines()
     # Runs:
     # portageq metadata /build/x86-alex ebuild net-misc/wget-1.12-r2 \
-    #                                               HOMEPAGE LICENSE DESCRIPTION
+    #                                             HOMEPAGE LICENSE
     # Returns:
     # http://www.gnu.org/software/wget/
     # GPL-3
-    # Network utility to retrieve files from the WWW
-    (self.homepages, self.ebuild_license_names, self.description) = (
-        lines[0].split(), lines[1].split(), lines[2:])
+    (self.homepages, self.ebuild_license_names) = (
+        lines[0].split(), lines[1].split())
+
+  def GetLicenses(self):
+    """Get licenses from the ebuild field and the unpacked source code.
+
+    Some packages have static license mappings applied to them that get
+    retrieved from the ebuild.
+
+    For others, we figure out whether the package source should be scanned to
+    add licenses found there.
+
+    Raises:
+      AssertionError: on runtime errors
+      PackageLicenseError: couldn't find license in ebuild and source.
+    """
+    if self.build_source_tree:
+      self.homepages = self._BuildInfo("HOMEPAGE").split()
+      self.ebuild_license_names = self._BuildInfo("LICENSE").split()
+    else:
+      self._ReadEbuildInfo()
 
     if self.fullname in PACKAGE_HOMEPAGES:
       self.homepages = PACKAGE_HOMEPAGES[self.fullname]
@@ -666,7 +744,7 @@ being scraped currently).""",
       logging.info("Static license mapping for %s: %s", self.fullnamerev,
                    ",".join(self.ebuild_license_names))
     else:
-      logging.info("Read licenses from ebuild for %s: %s", self.fullnamerev,
+      logging.info("Read licenses for %s: %s", self.fullnamerev,
                    ",".join(self.ebuild_license_names))
 
     # Lots of packages in chromeos-base have their license set to BSD instead
@@ -787,9 +865,7 @@ being scraped currently).""",
 class Licensing(object):
   """Do the actual work of extracting licensing info and outputting html."""
 
-  def __init__(self, board, package_fullnames, gen_licenses,
-               entry_template_file=ENTRY_TMPL):
-
+  def __init__(self, board, package_fullnames, gen_licenses):
     # eg x86-alex
     self.board = board
     # List of stock and custom licenses referenced in ebuilds. Used to
@@ -807,7 +883,7 @@ class Licensing(object):
     self.incomplete_packages = []
 
     self.package_text = {}
-    self.entry_template = ReadUnknownEncodedFile(entry_template_file)
+    self.entry_template = None
 
     # We need to have a dict for the list of packages objects, index by package
     # fullnamerev, so that when we scan our licenses at the end, and find out
@@ -821,22 +897,24 @@ class Licensing(object):
     return sorted(self.licenses.keys(), key=str.lower)
 
   def _SaveLicenseDump(self, pkg):
-    save_dir = "%s/%s/%s/%s" % (cros_build_lib.GetSysroot(self.board),
-                                PER_PKG_LICENSE_DIR, pkg.category, pkg.name)
-    logging.debug("Saving license to %s", save_dir)
+    if pkg.build_source_tree:
+      save_file = "%s/build-info/license.yaml" % pkg.build_source_tree
+    else:
+      save_file = pkg.license_dump_path
+    logging.debug("Saving license to %s", save_file)
+    save_dir = os.path.dirname(save_file)
     if not os.path.isdir(save_dir):
       os.makedirs(save_dir, 0755)
-    with open("%s/license.yaml" % save_dir, "w") as f:
+    with open(save_file, "w") as f:
       yaml_dump = []
       for key, value in pkg.__dict__.items():
         yaml_dump.append([key, value])
       f.write(yaml.dump(yaml_dump))
 
   def _LoadLicenseDump(self, pkg):
-    save_dir = "%s/%s/%s/%s" % (cros_build_lib.GetSysroot(self.board),
-                                PER_PKG_LICENSE_DIR, pkg.category, pkg.name)
-    logging.debug("Getting license from %s for %s", save_dir, pkg.name)
-    with open("%s/license.yaml" % save_dir, "r") as f:
+    save_file = pkg.license_dump_path
+    logging.debug("Getting license from %s for %s", save_file, pkg.name)
+    with open(save_file, "r") as f:
       # yaml.safe_load barfs on unicode it output, but we don't really need it.
       yaml_dump = yaml.load(f)
       for key, value in yaml_dump:
@@ -854,6 +932,21 @@ class Licensing(object):
       pkg.GetPackageInfo(package_name)
       self.packages[package_name] = pkg
 
+  def HookPackageProcess(self, pkg_build_path):
+    """Different entry point to populate a packageinfo.
+
+    This is called instead of LoadPackageInfo when called by a package build.
+
+    Args:
+      pkg_build_path: unpacked being built by emerge.
+    """
+    pkg = PackageInfo()
+    pkg.build_source_tree = pkg_build_path
+    pkg.GetPackageInfo(None)
+    if not pkg.skip:
+      pkg.GetLicenses()
+    self._SaveLicenseDump(pkg)
+
   def ProcessPackageLicenses(self):
     """Iterate through all packages provided and gather their licenses.
 
@@ -862,18 +955,34 @@ class Licensing(object):
 
     Do not call this after adding virtual packages with AddExtraPkg.
     """
-    if self.gen_licenses:
-      for package_name in self.packages:
-        pkg = self.packages[package_name]
-        if pkg.skip:
+    for package_name in self.packages:
+      pkg = self.packages[package_name]
+      if pkg.skip:
+        if self.gen_licenses:
           logging.info("Package %s is in skip list", package_name)
         else:
+          # If we do a licensing run expecting to get licensing objects from
+          # an image build, virtual packages will be missing such objects
+          # because virtual packages do not get the install hook run at build
+          # time. Because this script may not have permissions to write in the
+          # /var/db/ directory, we don't want it to generate useless license
+          # bits for virtual packages. As a result, ignore virtual packages
+          # here.
+          if pkg.category == "virtual":
+            logging.debug("Ignoring %s virtual package", package_name)
+            continue
+
+      # Other skipped packages get dumped with incomplete info and the skip flag
+      if not os.path.exists(pkg.license_dump_path) and not self.gen_licenses:
+        logging.warning(">>> License for %s is missing, creating now <<<",
+                        package_name)
+      if not os.path.exists(pkg.license_dump_path) or self.gen_licenses:
+        if not pkg.skip:
           try:
             pkg.GetLicenses()
           except PackageLicenseError:
             pkg.licensing_failed = True
-        # Skipped packages get dumped with incomplete info and the skip flag.
-        # Similarly, we dump packages where licensing failed too.
+        # We dump packages where licensing failed too.
         self._SaveLicenseDump(pkg)
 
     # To debug the code, we force the data to be re-read from the dumps
@@ -907,6 +1016,7 @@ class Licensing(object):
     pkg.ebuild_license_names = pkg_data[4]
     self.packages[pkg.fullnamerev] = pkg
 
+  # Called directly by src/repohooks/pre-upload.py
   @staticmethod
   def FindLicenseType(license_name):
     """Says if a license is stock Gentoo, custom, or doesn't exist."""
@@ -1007,14 +1117,17 @@ after fixing the license.""" %
 
   def GenerateHTMLLicenseOutput(self, output_file,
                                 output_template=TMPL,
+                                entry_template=ENTRY_TMPL,
                                 license_template=SHARED_LICENSE_TMPL):
     """Generate the combined html license file used in ChromeOS.
 
     Args:
       output_file: resulting HTML license output.
       output_template: template for the entire HTML file.
+      entry_template: template for per package entries.
       license_template: template for shared license entries.
     """
+    self.entry_template = ReadUnknownEncodedFile(entry_template)
     sorted_license_txt = []
 
     # Keep track of which licenses are used by which packages.
@@ -1191,44 +1304,16 @@ def ReadUnknownEncodedFile(file_path, logging_text=None):
   return file_txt
 
 
-def main(args):
-  # We are imported by src/repohooks/pre-upload.py in a non chroot environment
-  # where yaml may not be there, so we only import yaml if we're being run
-  # through the normal workflow.
-  # pylint: disable=W0603
-  global yaml
-  import yaml
-
-  global debug
-  # pylint: enable=W0603
+def NonHookMain(opts):
+  """Do the work when we're not called as a hook."""
 
-  parser = commandline.ArgumentParser(usage=__doc__)
-  parser.add_argument("-b", "--board", default=cros_build_lib.GetDefaultBoard(),
-                      help="which board to run for, like x86-alex")
-  parser.add_argument("-p", "--package", action="append", default=[],
-                      help="check the license of the package, e.g.,"
-                      "dev-libs/libatomic_ops-7.2d")
-  parser.add_argument("-a", "--all-packages", action="store_true",
-                      dest="all_packages",
-                      help="Run licensing against all packages in the "
-                      "build tree")
-  parser.add_argument("-g", "--generate-licenses", action="store_true",
-                      dest="gen_licenses",
-                      help="Generate licensing bits for each package before "
-                      "making license file\n(default is to use build time "
-                      "license bits)")
-  parser.add_argument("-o", "--output", type="path",
-                      help="which html file to create with output")
-  opts = parser.parse_args(args)
-  debug = opts.debug
   board, all_packages, gen_licenses, output_file = (
       opts.board, opts.all_packages, opts.gen_licenses, opts.output)
-  logging.info("Using board %s.", board)
-
   packages_mode = bool(opts.package)
 
-  if not output_file and not packages_mode:
-    logging.warning('No output file is specified.')
+  if not board:
+    raise AssertionError("No board given (--board)")
+  logging.info("Using board %s.", board)
 
   builddir = os.path.join(cros_build_lib.GetSysroot(board=board),
                           'tmp', 'portage')
@@ -1236,13 +1321,16 @@ def main(args):
     raise AssertionError(
         "FATAL: %s missing.\n"
         "Did you give the right board and build that tree?" % builddir)
+  if not output_file and not gen_licenses:
+    logging.warning("You are not generating licenses and you didn't ask for "
+                    "output. As a result this script will do nothing useful.")
+    license_dir = "%s/%s/" % (cros_build_lib.GetSysroot(board),
+                              PER_PKG_LICENSE_DIR)
+    if not os.path.exists(license_dir):
+      raise AssertionError("FATAL: %s missing.\n" % license_dir)
 
-  license_dir = "%s/%s/" % (cros_build_lib.GetSysroot(board),
-                            PER_PKG_LICENSE_DIR)
-  if not os.path.exists(license_dir):
-    raise AssertionError(
-        "FATAL: %s missing. Fix with:\n"
-        "sudo install -o $(whoami) -d %s" % (license_dir, license_dir))
+  if gen_licenses and os.geteuid() != 0:
+    raise AssertionError("Run with sudo if you use --generate-licenses.")
 
   if packages_mode:
     packages = opts.package
@@ -1281,8 +1369,47 @@ def main(args):
   if licensing.incomplete_packages:
     raise AssertionError("""
 DO NOT USE OUTPUT!!!
-Some packages are missing due to errors, please look at errors generated during
-this run.
+Some packages are missing due to errors, please look at errors generated
+during this run.
 List of packages with errors:
 %s
   """ % '\n'.join(licensing.incomplete_packages))
+
+
+def main(args):
+  # pylint: disable=W0603
+  global debug
+  # pylint: enable=W0603
+
+  parser = commandline.ArgumentParser(usage=__doc__)
+  parser.add_argument("-b", "--board",
+                      help="which board to run for, like x86-alex")
+  parser.add_argument("-p", "--package", action="append", default=[],
+                      help="check the license of the package, e.g.,"
+                      "dev-libs/libatomic_ops-7.2d")
+  parser.add_argument("-a", "--all-packages", action="store_true",
+                      dest="all_packages",
+                      help="Run licensing against all packages in the "
+                      "build tree")
+  parser.add_argument("-g", "--generate-licenses", action="store_true",
+                      dest="gen_licenses",
+                      help="Generate licensing bits for each package before "
+                      "making license file\n(default is to use build time "
+                      "license bits)")
+  parser.add_argument("-k", "--hook", type="path", dest="hook",
+                      help="Hook mode takes a single package and outputs its "
+                      "license on stdout. Give $PORTAGE_BUILDDIR as argument.")
+  parser.add_argument("-o", "--output", type="path",
+                      help="which html file to create with output")
+  opts = parser.parse_args(args)
+  debug = opts.debug
+  debug = True
+  hook_path = opts.hook
+
+
+  # This get called from src/scripts/hooks/install/gen-package-licenses.sh
+  if hook_path:
+    licensing = Licensing(None, None, True)
+    licensing.HookPackageProcess(hook_path)
+  else:
+    NonHookMain(opts)
diff --git a/licensing/process-pkg.py b/licensing/process-pkg.py
deleted file mode 100755
index 0ad845f22..000000000
--- a/licensing/process-pkg.py
+++ /dev/null
@@ -1,334 +0,0 @@
-#!/usr/bin/python
-#
-# Copyright (c) 2012 The Chromium OS Authors. All rights reserved.
-# Use of this source code is governed by a BSD-style license that can be
-# found in the LICENSE file.
-#
-# Original Author: Brian Harring <ferringb@chromium.org>
-# Maintainer: Marc MERLIN <merlin@chromium.org>
-
-"""Cheesy script that attempts to generate an HTML file containing license
-information and homepage links for all installed packages.
-
-This script is NOT ready for production, not style compliant, and is only
-checked in as a placeholder until new logic from licenses.py can be moved
-into it.
-"""
-
-import cgi
-import os
-import subprocess
-import sys
-import shutil
-
-PKG_DATA_TARGET_DIR = '/usr/share/licenses'
-
-STOCK_LICENSE_DIRS = [
-  os.path.expanduser('~/trunk/src/third_party/portage/licenses'),
-  'licenses',
-]
-
-SKIPPED_CATEGORIES = [
-  'chromeos-base',
-  'virtual',
-]
-
-SKIPPED_PACKAGES = [
-  # These are Chrome-OS-specific packages.
-  'media-libs/libresample',
-  'sys-apps/rootdev',
-  'sys-kernel/chromeos-kernel',  # already manually credit Linux
-
-  'www-plugins/adobe-flash',
-
-  # These have been split across several packages, so we skip listing the
-  # individual components (and just list the main package instead).
-  'app-editors/vim-core',
-  'x11-apps/mesa-progs',
-
-  # Portage metapackage.
-  'x11-base/xorg-drivers',
-
-  # These are covered by app-i18n/ibus-mozc (BSD, copyright Google).
-  'app-i18n/ibus-mozc-chewing',
-  'app-i18n/ibus-mozc-hangul',
-
-  # These are all X.org sub-packages; shouldn't be any need to list them
-  # individually.
-  'media-fonts/encodings',
-  'x11-apps/iceauth',
-  'x11-apps/intel-gpu-tools',
-  'x11-apps/mkfontdir',
-  'x11-apps/rgb',
-  'x11-apps/setxkbmap',
-  'x11-apps/xauth',
-  'x11-apps/xcursorgen',
-  'x11-apps/xdpyinfo',
-  'x11-apps/xdriinfo',
-  'x11-apps/xev',
-  'x11-apps/xgamma',
-  'x11-apps/xhost',
-  'x11-apps/xinit',
-  'x11-apps/xinput',
-  'x11-apps/xkbcomp',
-  'x11-apps/xlsatoms',
-  'x11-apps/xlsclients',
-  'x11-apps/xmodmap',
-  'x11-apps/xprop',
-  'x11-apps/xrandr',
-  'x11-apps/xrdb',
-  'x11-apps/xset',
-  'x11-apps/xwininfo',
-  'x11-base/xorg-server',
-  'x11-drivers/xf86-input-evdev',
-  'x11-drivers/xf86-input-keyboard',
-  'x11-drivers/xf86-input-mouse',
-  'x11-drivers/xf86-input-synaptics',
-  'x11-drivers/xf86-video-intel',
-  'x11-drivers/xf86-video-vesa',
-  'x11-drivers/xf86-video-vmware',
-  'x11-libs/libICE',
-  'x11-libs/libSM',
-  'x11-libs/libX11',
-  'x11-libs/libXScrnSaver',
-  'x11-libs/libXau',
-  'x11-libs/libXcomposite',
-  'x11-libs/libXcursor',
-  'x11-libs/libXdamage',
-  'x11-libs/libXdmcp',
-  'x11-libs/libXext',
-  'x11-libs/libXfixes',
-  'x11-libs/libXfont',
-  'x11-libs/libXfontcache',
-  'x11-libs/libXft',
-  'x11-libs/libXi',
-  'x11-libs/libXinerama',
-  'x11-libs/libXmu',
-  'x11-libs/libXp',
-  'x11-libs/libXrandr',
-  'x11-libs/libXrender',
-  'x11-libs/libXres',
-  'x11-libs/libXt',
-  'x11-libs/libXtst',
-  'x11-libs/libXv',
-  'x11-libs/libXvMC',
-  'x11-libs/libXxf86vm',
-  'x11-libs/libdrm',
-  'x11-libs/libfontenc',
-  'x11-libs/libpciaccess',
-  'x11-libs/libxkbfile',
-  'x11-libs/libxkbui',
-  'x11-libs/pixman',
-  'x11-libs/xtrans',
-  'x11-misc/util-macros',
-  'x11-misc/xbitmaps',
-  'x11-proto/bigreqsproto',
-  'x11-proto/compositeproto',
-  'x11-proto/damageproto',
-  'x11-proto/dri2proto',
-  'x11-proto/fixesproto',
-  'x11-proto/fontcacheproto',
-  'x11-proto/fontsproto',
-  'x11-proto/inputproto',
-  'x11-proto/kbproto',
-  'x11-proto/printproto',
-  'x11-proto/randrproto',
-  'x11-proto/recordproto',
-  'x11-proto/renderproto',
-  'x11-proto/resourceproto',
-  'x11-proto/scrnsaverproto',
-  'x11-proto/trapproto',
-  'x11-proto/videoproto',
-  'x11-proto/xcmiscproto',
-  'x11-proto/xextproto',
-  'x11-proto/xf86bigfontproto',
-  'x11-proto/xf86dgaproto',
-  'x11-proto/xf86driproto',
-  'x11-proto/xf86rushproto',
-  'x11-proto/xf86vidmodeproto',
-  'x11-proto/xineramaproto',
-  'x11-proto/xproto',
-]
-
-LICENSE_FILENAMES = [
-  'COPYING',
-  'COPYRIGHT',  # used by strace
-  'LICENCE',    # used by openssh
-  'LICENSE',
-  'LICENSE.txt',  # used by NumPy, glew
-  'LICENSE.TXT',  # used by hdparm
-  'License',    # used by libcap
-  'IPA_Font_License_Agreement_v1.0.txt',  # used by ja-ipafonts
-]
-LICENSE_FILENAMES = frozenset(x.lower() for x in LICENSE_FILENAMES)
-
-SKIPPED_LICENSE_DIRECTORIES = frozenset([
-  'third_party'
-])
-
-PACKAGE_LICENSES = {
-  'app-crypt/nss': ['MPL-1.1'],
-  'app-editors/gentoo-editor': ['MIT-gentoo-editor'],
-  'app-editors/vim': ['vim'],
-  'app-i18n/ibus-mozc': ['BSD-Google'],
-  'dev-db/sqlite': ['sqlite'],
-  'dev-libs/libevent': ['BSD-libevent'],
-  'dev-libs/nspr': ['GPL-2'],
-  'dev-libs/nss': ['GPL-2'],
-  'dev-libs/protobuf': ['BSD-Google'],
-  'dev-util/bsdiff': ['BSD-bsdiff'],
-  'media-fonts/font-util': ['font-util'],  # COPYING file from git repo
-  'media-libs/freeimage': ['GPL-2'],
-  'media-libs/jpeg': ['jpeg'],
-  'media-plugins/o3d': ['BSD-Google'],
-  'net-dialup/ppp': ['ppp-2.4.4'],
-  'net-dns/c-ares': ['MIT-MIT'],
-  'net-misc/dhcpcd': ['BSD-dhcpcd'],
-  'net-misc/iputils': ['BSD-iputils'],
-  'net-wireless/wpa_supplicant': ['GPL-2'],
-  'net-wireless/iwl1000-ucode': ['Intel-iwl1000'],
-  'net-wireless/marvell_sd8787': ['Marvell'],
-  'sys-apps/less': ['BSD-less'],
-  'sys-libs/ncurses': ['ncurses'],
-  'sys-libs/talloc': ['LGPL-3'],  # ebuild incorrectly says GPL-3
-  'sys-libs/timezone-data': ['public-domain'],
-  'sys-process/vixie-cron': ['vixie-cron'],
-  'x11-drivers/xf86-input-cmt': ['BSD-Google'],
-}
-
-INVALID_STOCK_LICENSES = frozenset([
-  'BSD',   # requires distribution of copyright notice
-  'BSD-2', # same
-  'BSD-4', # same
-  'MIT',   # same
-])
-
-TEMPLATE_FILE = 'about_credits.tmpl'
-ENTRY_TEMPLATE_FILE = os.path.dirname(os.path.abspath(__file__))
-ENTRY_TEMPLATE_FILE = os.path.join(ENTRY_TEMPLATE_FILE,
-                                      'about_credits_entry.tmpl')
-
-def FilterUsableLicenses(raw_license_names):
-  for license_name in raw_license_names:
-    if license_name in INVALID_STOCK_LICENSES:
-      print 'skipping invalid stock license %s' % license_name
-      continue
-    yield license_name
-
-def _FindLicenseInSource(workdir):
-  names = []
-  for name in LICENSE_FILENAMES:
-    if names:
-      names.append('-o')
-    names.extend(('-iname', name))
-  if names:
-    names = ['('] + names + [')']
-
-  # ensure the directory has a trailing slash, else if it's a symlink
-  # find will just stop at the sym.
-  args = ['find', workdir +'/', '-maxdepth', '3', '-mindepth', '1',
-    '-type', 'f']
-  args += names
-  names = []
-  for dirname in SKIPPED_LICENSE_DIRECTORIES:
-    if names:
-      names.append('-o')
-    names.extend(('-ipath', '*/%s/*' % dirname))
-  if names:
-    args += ['-a', '!', '('] + names + [')']
-  p = subprocess.Popen(args, stdin=None, stderr=None,
-    stdout=subprocess.PIPE, shell=False)
-  stdout, _ = p.communicate()
-  files = filter(None, stdout.split("\n"))
-  if files:
-    return open(files[0], 'r').read()
-  return None
-
-def _GetStockLicense(licenses, stock_licenses):
-  if not licenses:
-    return None
-
-  license_texts = []
-  for license_name in licenses:
-    license_path = stock_licenses.get(license_name)
-    if license_path is None:
-      # TODO: We should probably report failure if we're unable to
-      # find one of the licenses from a dual-licensed package.
-      continue
-    license_texts.append(open(license_path).read())
-
-  return '\n'.join(license_texts)
-
-def identifyLicenseText(workdir, metadata_licenses, stock_licenses):
-  license_text = _FindLicenseInSource(workdir)
-  if not license_text:
-    license_text = _GetStockLicense(metadata_licenses, stock_licenses)
-  if not license_text:
-    raise Exception("failed finding a license in both the source, and a "
-                    "usable stock license")
-  return license_text
-
-def EvaluateTemplate(template, env, escape=True):
-  """Expand |template| with content like {{foo}} using a dict of expansions."""
-  for key, val in env.items():
-    if escape:
-      val = cgi.escape(val)
-    template = template.replace('{{%s}}' % key, val)
-  return template
-
-def WritePkgData(image_dir, pkgname, license_text, homepage):
-  data = EvaluateTemplate(open(ENTRY_TEMPLATE_FILE).read(),
-    dict(name=pkgname, license=license_text, url=homepage))
-  base = os.path.join(image_dir, PKG_DATA_TARGET_DIR.lstrip('/'), pkgname)
-  shutil.rmtree(base, ignore_errors=True)
-  os.makedirs(base, 0o755)
-  with open(os.path.join(base, 'license_text'), 'w') as f:
-    f.write(data)
-
-def main(fullname, license_data, homepage_data, image_dir, workdir,
-  stock_license_dir, forced_license_files=None):
-  metadata_licenses = list(FilterUsableLicenses(license_data.split()))
-
-  stock_licenses = {}
-  # ordering matters; via this, our license texts override the gentoo
-  # provided ones where desired.
-  our_license_dir = os.path.dirname(os.path.abspath(__file__))
-  our_license_dir = os.path.join(our_license_dir, 'licenses')
-  for license_dir in [stock_license_dir, our_license_dir]:
-    stock_licenses.update((x, os.path.join(license_dir, x))
-      for x in os.listdir(license_dir)
-        if x not in INVALID_STOCK_LICENSES)
-
-  if forced_license_files:
-    licenses_text = "\n".join(open(os.path.join(workdir, x)).read()
-      for x in forced_license_files.split())
-  else:
-    licenses_text = identifyLicenseText(workdir, metadata_licenses,
-      stock_licenses)
-
-  homepages = homepage_data.split()
-  if not homepages:
-    homepage = ''
-  else:
-    homepage = homepages[0]
-
-  if not licenses_text:
-    raise Exception("failed identifying license text for %s" % fullname)
-
-  WritePkgData(image_dir, fullname, licenses_text, homepage)
-
-if __name__ == '__main__':
-  # ./process-pkg.py "${CATEGORY}/${PN}-${PVR}" "${LICENSE}" "${HOMEPAGE}"
-  # "${D}" "${WORKDIR}"
-  # "${LICENSES_DIR}"
-  if len(sys.argv) < 7:
-    sys.stderr.write("not enough args; expected ${CATEGORY}/${PN}-${PVR} "
-      "${LICENSE} ${HOMEPAGE} ${D} ${WORKDIR} ${PORTDIR}/licenses"
-      "[FORCED_LICENSE_FILE1 .. FORCED_LICENSE_FILE9]\n"
-      "got %i args: %r\n"
-      % (len(sys.argv) -1, sys.argv))
-    sys.exit(1)
-  args = sys.argv[1:7]
-  forced_license_files = sys.argv[7:]
-  main(forced_license_files=forced_license_files, *sys.argv[1:])
-  sys.exit(0)