Clean up ESP and ISAKMP decryption.

At least as I read RFC 5996 section 3.14 and RFC 4303 section 2.4, if
the cipher has a block size of which the ciphertext's size must be a
multiple, the payload must be padded to make that happen, so the
ciphertext length must be a multiple of the block size.  Instead of
allocating a buffer, copying the ciphertext to it, and padding it to the
block size, fail if its size isn't a multiple of the block size.

(Note also that the old padding code added a block's worth of padding to
the end of a ciphertext block that *was* a multiple of the cipher block
size; this might have caused problems.)

Don't use the undocumented EVP_Cipher(); the lack of documentation means
a lack of information about whatever requirements it might impose.  Use
EVP_DecryptUpdate() instead.

Before calling it, use EVP_CIPHER_CTX_set_padding() to say "don't do
your own padding, this block is a multiple of the cipher block size".

Instead of using EVP_CipherInit() or EVP_CipherInit_ex(), use
EVP_DecryptInit() or EVP_DecryptInit_ex().  as we're always doing
decryption and never doing encryption - the extra parameter to
EVP_CipherInit() and EVP_CipherInit_ex() is always 0.

This may address GitHub issue #814.

It may also make it a bit easier to have the code use Common Crypto on
macOS (rather than requiring that OpenSSL be installed - macOS ships
with an OpenSSL shared library for binary compatibility with older
releases, but doesn't ship with the headers, because Apple wants you
using their crypto code) and use Cryptography API: Next Generation on
Windows (Vista/Server 2008 and later) (rather than requiring a Windows
build of OpenSSL).

(Hopefully this will all work with LibreSSL.)

1 files changed, 4 insertions, 4 deletions

diff --git a/configure.ac b/configure.ac
index 26c1b3a8..d81ff2b0 100644
--- a/configure.ac
+++ b/configure.ac
@@ -924,19 +924,19 @@ if test "$want_libcrypto" != "no"; then
 			# EVP_CIPHER_CTX, as EVP_CIPHER_CTX may be
 			# opaque; otherwise, we allocate it ourselves.
 			#
-			# 2) do we have EVP_CipherInit_ex()?
+			# 2) do we have EVP_DecryptInit_ex()?
 			# If so, we use it, because we need to be
 			# able to make two "initialize the cipher"
 			# calls, one with the cipher and key, and
 			# one with the IV, and, as of OpenSSL 1.1,
-			# You Can't Do That with EVP_CipherInit(),
-			# because a call to EVP_CipherInit() will
+			# You Can't Do That with EVP_DecryptInit(),
+			# because a call to EVP_DecryptInit() will
 			# unconditionally clear the context, and
 			# if you don't supply a cipher, it'll
 			# clear the cipher, rendering the context
 			# unusable and causing a crash.
 			#
-			AC_CHECK_FUNCS(EVP_CIPHER_CTX_new EVP_CipherInit_ex)
+			AC_CHECK_FUNCS(EVP_CIPHER_CTX_new EVP_DecryptInit_ex)
 		fi
 	])
 fi