| Age | Commit message (Collapse) | Author |
|---|
|
Replace all hand crafted BAD_ERROR() messages with MEM_ERROR()
which does the same thing.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Replace all hand crafted BAD_ERROR() messages with MEM_ERROR()
which does the same thing.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Replace all hand crafted BAD_ERROR() messages with MEM_ERROR()
which does the same thing.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Add a new MEM_ERROR() macro that prints out of space with
the function it occurred in, and then aborts mksquashfs,
restoring the original filesystem if appending.
This macro allows "Out of memory" handling to be regularised
accross Mksquashfs, rather than at present where every
out of memory error has a different and hand crafted
error message.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Write_bytes() already prints an error message if write() fails,
so this error message is largely redundant.
So replace with a more user friendly error message that says
we're aborting due to failure to write the filesystem image.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Write_bytes() already prints an error message if write() fails,
so this error message is largely redundant.
So replace with a more user friendly error message that says
we're aborting due to failure to write the filesystem image.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
If the main thread hits "lseek error" when writing the metadata
in write_destination() it is probably because the
destination file is out of space, this is especially true if the
destination is a block device, but could conceivably happen on some
(non Linux?) filesystems too if they don't implement holes. This
is because lseeking to the write position could be beyond the end
of the available destination file end, and hence fail before
attempting to write.
Recent correspondence with users shows that "Lseek error" is
meaningless, and so augment the error message with an "Out of
Space" error which is presumbly something they will understand.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
If the writer thread hits "lseek error" it is probably because the
destination file is out of space, this is especially true if the
destination is a block device, but could conceivably happen on some
(non Linux?) filesystems too if they don't implement holes. This
is because lseeking to the write position could be beyond the end
of the available destination file end, and hence fail before
attempting to write.
Recent correspondence with users shows that "Lseek error" is
meaningless, and so augment the error message with an "Out of
Space" error which is presumbly something they will understand.
Note we can't use BAD_ERROR here because it will cause mksquashfs to
immediately exit, and we instead want to signal the the main thread to
exit.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
There has been a long term problem with out of space handling
in mksquashfs, dating back to the parallelisation of mksquashfs
in 2006. The writer thread discovers out of space when writing
the filesystem, but, it cannot tell the main thread to abort, as
there is no "back channel" to the main thread. Adding a back
channel wasn't an attractive option because checking of this
by the main thread would limit parallelisation, and I was keen on
maxmising parallelism. In the end I implemented a workaround where
the writer() thread flaged the error, ignored further write requests,
and at the end of the filesystem creation when the main thread had to
synchronise with the writer thread, it passed an error flag back,
which would cause the main thread to abort then.
This workaround was determined to be "good enough" for the first
release of "parallel mksquashfs" when as usual due to work commitments
I needed to finish the difficult and complex task of parallelising
Mksquashfs by a fixed deadline. This was encouraged by the view that
nobody knowingly creates Squashfs filesystems on a device insufficiently
large, and if it does prove too small, it is likely to be towards the
end of filesystem creation, and so no-one would notice Mksquashfs running
on until the end of the filesystem creation.... So it proved to be,
no-one raised issues about the out of space handling, and although
the out of space handling was never entirely forgotten about, it
was never re-visited as there was always something more important to
deal with, and the years went by ...
Fast-forward to 2013, and I get some emails about "Lseek failure" on
the destination device. Although it's not been conclusively proved
because the reporter has not got back to me, it is almost certainly
a subtle case of the "out of space" handling. The reporter is confused
as there's been no "out of space" error reported, and I was confused for
a while until I realised the "out of space" error has likely scrolled off the
screen - the source filesystem is huge, the destination filesystem is far
too small, and by the time the "Lseek error" is reported the out of space
error is long gone. The "Lseek error" is a symptom of the earlier
"out of space" error where the writer thread has ceased to write
data to the output filesystem - when the main thread finds a "potential"
duplicate file alread stored in the filesystem, it reads the original
file out of the filesystem for a byte-by-byte compare, but the file
doesn't exist as it would have been written after the "out of space"
error, and on a block device this will result in an "Lseek error".
So, the out of space handling finally causes an issue seven years
afterwards.
Revisiting the problem the solution is obvious, have the writer thread
send a signal to the main process, the main process on receipt of the
signal aborts and restores the filesystem if it is appending. In fact
the infrastructure already exists to deal with the user hitting ^C twice.
This commit fixes that. The next commit will fix a subtle edge condition
with the writer thread where it may experience Lseek error before
hitting Out of Space on block devices. This is already handled, but it is
not flagged as an Out of Space error. This could be the Lseek error reported
but it is unlikely, as the Lseek error is reported as immediately
terminating Mksquashfs, if the writer thread experienced Lseek Mksquashfs
would in fact continue.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Add description to determine which function the lseek error
occurred in, and add the offset the lseek was trying to seek to.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Add support to display the compression options used to
compress the filesystem in -stat.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Forgot to update error message when function was renamed.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
sBlk.s.fragment_table_start isn't the end of the
directory table, it instead points to the
fragment_index_table, in which the first index of that
points to the end of the directory table. The only exception
to that is where fragments == 0, where fragment_table_start
does represent the end of the directory table.
The upshot of this is that for years Unsquashfs has been
reading and decompressing the directory_table and the
table after it. Not a major bug, but a waste of memory
all the same.
It has shown up now because of the additional sanity checks
when reading metadata blocks that have been added recently,
where all metadata blocks in the directory table bar the last one
should be SQUASHFS_METADATA_SIZE in size.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
with the other filesystem table reading functions.
In particular, rename to read_directory_table() and return
errors as FALSE rather than exiting inside the function.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
with the other filesystem table reading functions.
In particular, rename to read_inode_table() and return
errors as FALSE rather than exiting inside the function.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Harden inode and directory table reading, checking against
unexpected metadata block length, ensuring no buffer
overflow/underflow occurs.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Harden uid/gid and fragment table reading, checking against
unexpected metadata block length, ensuring no buffer
overflow/underflow occurs.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Harden fragment table reading, checking against unexpected metadata
block length, ensuring no buffer overflow/underflow occurs.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Harden fragment table reading, checking against unexpected metadata
block length, ensuring no buffer overflow/underflow occurs.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Harden xattr data reading, checking against unexpected metadata
block length, ensuring no buffer overflow/underflow occurs.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
filesystems
Prevent buffer overflow and underflow in read_block() with corrupted
filesystems.
Overflow is easy to understand, read_block() is called to read the next
metadata block pointed to by <start>... Often the buffer passed in is
large enough to hold the expected return bytes, which can be less than
SQUASHFS_METADATA_SIZE. For example filesystem tables are compressed
in SQUASHFS_METADATA_SIZEd chunks, the last compressed chunk will normally
be smaller than SQUASHFS_METADATA_SIZE, so when read_block() is called,
the passed buffer is only large enough to hold the expected size.
Underflow is rather more subtle, when read_block() is called, it is
expected that the returned block will fill the expected amount of
bytes in the filesystem table (stored as an array). If the returned
block is smaller than expected, then there will be uninitialised
data in the filesystem table which will cause unexpected behaviour later.
Fix both cases by passing in an additional parameter <expected>
which contains the expected number of bytes in the metadata block.
Refuse to read blocks which are larger than expected to avoid
buffer overflow and also return error if the block proves to be
smaller than expected, to avoid using unitialised data.
For the callers where the expected number of bytes is unknown support
<expected> containing 0, in which case the metadata block is checked to
ensure it doesn't overflow a SQUASHFS_METADATA_SIZEd buffer. Callers of
read_block() with <expected> == 0 are expected to pass in a
SQUASHFS_METADATA_SIZEd buffer. For instance with compressor specific
options data, the correct length is only known by the compressor specific
code, and this is later called to check the length.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Add a lot more sanity checks to the data read from the
filesystem.
Also ensure we do not try to access beyond the malloced buffer
holding the data.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Rename block_size parameter in uncompress() call to outsize, as it
now holds the size of the output buffer, which can be less than
block_size.
Also in the wrappers where the uncompressed length is available
in the header, don't bother calling the codec if outsize is too
small.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Further hardening of read_fs. If the filesystem is corrupted we may
get the situation where in the inode_table metatadata block scan we
fail to find the metadata block containing the root inode. In this
case *root_inode_block will be used unitialised.
Add code to check against the small possibility this happens. In the
majority of cases if the filesystem is corrupted we will fail to read
a metadata block in the block scan, and so we will never get to this
place in the code, however, it could happen and so it needs to be
guarded against.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Note we can't use expected in read_block() here because at the time
of calling read_block() we don't know what the expected size is.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
read_block()
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
filesystems
Prevent buffer overflow and underflow in read_block() with corrupted
filesystems.
Overflow is easy to understand, read_block() is called to read the next
metadata block pointed to by <start>... Often the buffer passed in is
large enough to hold the expected return bytes, which can be less than
SQUASHFS_METADATA_SIZE. For example filesystem tables are compressed
in SQUASHFS_METADATA_SIZEd chunks, the last compressed chunk will normally
be smaller than SQUASHFS_METADATA_SIZE, so when read_block() is called,
the passed buffer is only large enough to hold the expected size.
Underflow is rather more subtle, when read_block() is called, it is
expected that the returned block will fill the expected amount of
bytes in the filesystem table (stored as an array). If the returned
block is smaller than expected, then there will be uninitialised
data in the filesystem table which will cause unexpected behaviour later.
Fix both cases by passing in an additional parameter <expected>
which contains the expected number of bytes in the metadata block.
Refuse to read blocks which are larger than expected to avoid
buffer overflow and also return error if the block proves to be
smaller than expected, to avoid using unitialised data.
For the callers where the expected number of bytes is unknown support
<expected> containing 0, in which case the metadata block is checked to
ensure it doesn't overflow a SQUASHFS_METADATA_SIZEd buffer. Callers of
read_block() with <expected> == 0 are expected to pass in a
SQUASHFS_METADATA_SIZEd buffer. For instance with compressor specific
options data, the correct length is only known by the compressor specific
code, and this is later called to check the length.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Small optimisation - subpath is only needed if there are
exclude actions specified, or if the dir_ent is a directory.
In other cases don't calculate subpath as it is not needed.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
This simplifies the logic, eliminating the need to strdup on the
recursive call to dir_scan1(), and eliminating the need to free
them on recursive dir_scan1() failure.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Non anchored excludes (specified with the "... name" syntax to exclude
files matching name in any directory) are implemented with the
"stickypath" path variable. This is an exclude specification which
"sticks" as it is present in every directory.
This was previously implemented by adding the stickypath to each
new set of paths returned from the excluded() function, as obviously,
we want stickypath to be present in the next sub-directory.
However, this has the undesirable effect that the new paths variable
"new" is always returned holding at least one entry (the stickypath).
This partially breaks the original design intention of excluded() where
by putting the exclude specifications only in the directories where
they are applicable, for the majority of the time dir_scan1() will have
an empty set of paths to check for exclude matches, thus eliminating
the overhead of exclude matching.
Fix this by not implementing stickypaths as an entry added to the
new set of paths returned from excluded(), instead explictly match
against stickypath if it exists. This slightly complicates
excluded() requiring a new helper function excluded_match() split out
of excluded(), but the benefit is the new variable in the majority
of cases now gets returned empty (NULL) for stickypaths which was the
original intention.
Also get rid of init_subdir() and instead initialise new in
add_subdir() when first called. This not only eliminates code (so the
overall changes are 15 lines shorter than before despite the
addition of the new excluded_match() helper function) but it also
means "new" is never allocated in the event we don't get
any matches.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Fix leak in dir_scan1() where new returned by excluded() was
not freed. Normally new will be null unless exclude files and
wildcards have been used.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Add the ability to parse strings quoted with ", and also add the
ability to backslash "special characters" that would
normally be treated as end of string.
Quoted and non quoted strings can be specified multiply and the
strings will be concatenated together.
String examples
"Hello World"
Hello\ World
This" is a "string
"So"\ is\ "this"
"This \" is too"
etc.
Also switch to using an internal buffer (increased as necessary) to
return strings from the lexical analyser, rather than dynamically
allocating each string returned. This reduces unnecessary overhead
(mostly strings returned from the lexical analyser are discarded
without being stored, the ones that are stored can be explicitly
copied when necessary), and it fixes a memory leak in the parser
as returned strings are not freed.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
ERROR() since commit 738570df60f65c10cd7fde4134f3af1e005b671a calls
progressbar_error() to synchronise with the progress bar output,
however, this function uses pthread mutexes which are meaningless/
useless in a child process.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
ERROR() since commit 738570df60f65c10cd7fde4134f3af1e005b671a calls
progressbar_error() to synchronise with the progress bar output,
however, this function uses pthread mutexes which are meaningless/
useless in a child process.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Exec() adds the ability to run a command, with the
result of the test being the exit status of the command;
if the command returns a valid exit status (0), exec() returns
TRUE, otherwise on a bad exit status exec() returns FALSE.
Three environment variables hold:
- NAME: name of file
- PATHNAME: pathname of file relative to squashfs root
- SOURCE_PATHNAME: the pathname of the file in the source
directory
Currently only a single argument (without spaces) can be entered
as a command, this is because the actions parser currently lacks the
ability to have arguments with spaces in them (either argument quoted
with '"', or spaces blackslashed with '\')!
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
XZ is the only compression algorithm using the "new"
compression option framework. Add some comments on the
functions that describe how the functions work, and how
they interact with the overall compression options framework.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
|
Plus add checking for trailing characters
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
|
