summaryrefslogtreecommitdiff
path: root/smalloc.c
AgeCommit message (Collapse)Author
2017-05-19Update to fio-2.20.Elliott Hughes
Bug: N/A Test: builds, seems to work Change-Id: I455dcc898f00dc50ae5cb65d1b381982df5a51e0
2015-03-06smalloc: bump initial size to 8 poolsJens Axboe
Commit 23bd40f944b7 shrank our pool size from 128 pools at 8MB max, to one static 16MB pool. Christian reports: "For our tests with about 250k files we found the smalloc pool being depleted. Now for us values of 3-4 would be enough, but since it is a compile time switch I'd like to make it safe for everybody and set 8." Bump the pool alloc to 8 pools again, retaining the 16MB size. That's still substantially less than before, but should be enough for most cases. Allocate the pools at init time, to avoid the issue that the original commit fixed. Also fix a bug where we failed to iterate some pools when restarting the allocation scan, resulting in less-than-optimal exhaustion of all pools before smalloc() gave up and returned NULL. Signed-off-by: Jens Axboe <axboe@fb.com>
2015-03-06fio: fix smalloc strdop allocation failureChristian Ehrhardt
smalloc_strdup didn't check for allocation success and thereby ran into segfaults if the single pool went out of memory. Now with this patch applied it is still failing, but in a more consistent way than segfaulting. You still get a bad allocation, but it looks like this now: fio: smalloc OOM fio: filesetup.c:1495: dup_files: Assertion `0' failed. Aborted In fact the upper layers expected smalloc_strdup to retrun NULL on failure. Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com> Signed-off-by: Jens Axboe <axboe@fb.com>
2015-03-06smalloc: limit to 1 pool, and bump size to 16MBJens Axboe
We can easily add new pools, but if we do this after jobs have been forked off, then it wont really be shared storage as the new mmap mappings will only apply to the process that instantiated them. Since we moved the randommap mappings to private storage, we should not need a lot of room for shared allocations. Hence I hope the 16MB will suffice. Signed-off-by: Jens Axboe <axboe@fb.com>
2015-03-06smalloc: add zeroing scalloc() variantJens Axboe
Signed-off-by: Jens Axboe <axboe@fb.com>
2015-03-06smalloc: use log_err() for errorsJens Axboe
If this happens at runtime, stderr/stdout may have been closed and used for server/client connections. Signed-off-by: Jens Axboe <axboe@fb.com>
2015-03-06Add support for compiling for ESXJens Axboe
With contributions from Ryan Haynes <rhaynes@fusionio.com> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-01Cleanup symbols that should be staticJens Axboe
Run analysis on symbols not used outside of their current file, turn them into statics. Signed-off-by: Jens Axboe <axboe@fb.com>
2013-03-20Rework lockfile= file lock handlingJens Axboe
Get rid of the hand rolled rw semaphores, just use pthread rwlocks instead. Kill the batching too, it was broken by default, so nobody could have been using it. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2012-11-02smalloc: move to size_t for allocationsJens Axboe
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2012-11-02smalloc: fix int truncation issueJens Axboe
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2012-08-02mutex: make 0/1 FIO_MUTEX_LOCKED and FIO_MUTEX_UNLOCKEDJens Axboe
Makes the API look cleaner. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2012-02-20Fix assumption that pointers fits in a 'long'Bruce Cran
Windows uses LLP64 model so [u]intptr_t is more correct. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2011-08-15Fix bug in smalloc size calculationJiri Horky
Jiri reports: I think I found a bug. If the device (file) used in fio has "wrong" size, and is used with "wrong" block size, the fio fails to allocate memory for randommap. Following conditions must be true, considering just one thread for simplicity: 1) number of blocks on the device are too big to fill in the default's pool 2) number of blocks on the device must be "right", so the size of newly allocated pool by smalloc.c:add_pool is exactly as big as requested size (there are some roundings in place, so not every size triggers this error) When this is true, function smalloc.c:__smalloc_pool fails to use the newly created (exactly big enough) pool, because of this line: idx = find_next_zero(pool->bitmap[i], last_idx); which always returns non-zero value (the minimum returned value is last_idx +1), and so in our case the idx is 1 even though, the block number 0 is free. As the direct consequence, the function doesn't find enough free space in the given pool and returns NULL pointer. This leads to another try to allocate enough memory in smalloc.c:smalloc_real function, and so on... The behavior could be reprocuded using both stable 1.57 and git version of the tool. The values that triggers the error are: init_random_map - real_file_size: 21999995584512, o.rw_min_bs: 4096, blocks: 5371092672, num_maps: 83923323, alloc size: 671386584 whereas this block device is fine: init_random_map - real_file_size: 19999995985920, o.rw_min_bs: 4096, blocks: 4882811520, num_maps: 76293930, alloc size: 610351440 the configuration file used: [global] description=CESNET_test [cesnet] filename=/dev/sdd rw=randread size=10000g ioengine=libaio iodepth=4 bs=4k runtime=10m time_based numjobs=1 group_reporting Get rid of the +1, it looks like a remnant from when the pools had a header associated with them. Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
2011-06-29Make smalloc use anonymous memory mmapsJens Axboe
Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
2011-01-08Various fixes/updatesBruce Cran
- Update HOWTO to note that directio and fallocate don't work with ZFS on Solaris. Refactor the Makefile's to add CPPFLAGS and LIBS. - Add -fno-omit-frame-pointer from Linux Makefile to every platform - Change undefined $(ALL_CFLAGS) to $(CFLAGS) - Pass -std=gnu99, without which OS X fails to build. - Add -D__EXTENSIONS__ on Solaris since some functions we need are behind it. - Pull in <limits.h> in fio.c to get PTHREAD_STACK_MIN. - NetBSD doesn't define PTHREAD_STACK_MIN so set it to 4k in os-netbsd.h - If we have posix_fallocate don't error out if it fails during mutex and malloc operations since it will fail on Solaris with a ZFS filesystem. As I understand it these aren't performance-critical operations so do they need to be considered critical? - Remove fio_unused from os-* files since it's defined in fio.h and we don't really need it. - FreeBSD has an idprio command but not the API so don't claim it does. - OS X doesn't have the timer_* API so emulate it using setitimer/sigaction. - NetBSD and Solaris don't support CLOCK_MONOTONIC in timer_create so remove FIO_HAVE_CLOCK_MONOTONIC from their os-* files. I've noticed that a change I made a while ago to use fmin/fmax could cause issues on older OSes - I had a CD with NetBSD 5.0.2 and found they had only been implemented in 5.1 so I'm not sure if I should revert it. Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
2010-10-14Fix compile warning on platforms with posix_fallocateJens Axboe
Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
2010-06-28gracefully handle full /tmp file systemGreg Edwards
If /tmp happens to be full, fio gives you a very unhelpful error: # ./fio rand-write-256k-q256.job Bus error Use posix_fallocate() to gracefully handle this condition. Signed-off-by: Greg Edwards <greg.edwards@hp.com> Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
2009-09-11Silence the style/whitespace policeJens Axboe
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2009-07-31smalloc: enlarge pool sizeJens Axboe
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2009-07-07smalloc: just round alloc_size to int aligned for post redzone pointerJens Axboe
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2009-07-07Make smalloc redzone pointer properly alignedJens Axboe
It would SIGBUS on archs that require proper alignment, and work suboptimally on others. Real fix is pending, making sure that we always return naturally aligned pointers. But this at least makes it work on sparc64. Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2009-01-06smalloc: get rid of MP_SAFE define and lock checksJens Axboe
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2008-12-04smalloc: pool->file is only used in add_pool(), so make it localJens Axboe
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2008-12-04smalloc: unlink pool file in add_pool()Jens Axboe
Don't defer until cleanup time, since we may not clean up if fio crashes or is killed in other ways. Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2008-09-10smalloc: Remove read/write lock race condition which will result in ↵Shaozhi Shawn Ye
segmentation fault Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2008-07-31smalloc: default to 128 pools instead of 4Jens Axboe
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2008-06-13Revert "smalloc: convert to spinlocks"Jens Axboe
This reverts commit 3f1959101ed6a935ff3e2aec0fe7f5828f23fbcb.
2008-06-12smalloc: convert to spinlocksJens Axboe
Makes the allocator 10-20% faster. Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2008-06-02smalloc: use optimized ffz()Jens Axboe
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2008-05-31Add sparc and sparc64 supportJens Axboe
Also get rid of the arch private ffz functions, the generic one is just fine. Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2008-05-27smalloc: final cleanupsJens Axboe
Passes all regressions and tests, must be Bug Free. Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2008-05-27smalloc: fix map overflowJens Axboe
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2008-05-27smalloc: fix off-by-one in ptr_valid()Jens Axboe
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2008-05-27smalloc: kill another debug printf()Jens Axboe
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2008-05-27smalloc: fix compile errorJens Axboe
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2008-05-27smalloc: cleanup and remove debug printf()Jens Axboe
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2008-05-27smalloc: turn on the thread safe flagJens Axboe
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2008-05-27smalloc: change to a bitmap allocatorJens Axboe
It's a lot cleaner and easier to understand than the previous one, even if it does waste some more memory. Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2008-05-26smalloc: add pre and post redzone checks to pointersJens Axboe
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2008-05-23smalloc: debugJens Axboe
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2008-05-23smalloc: increase alloc sizeJens Axboe
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2008-04-17smalloc: remove pool resize supportJens Axboe
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2008-04-16smalloc: cleanups and allow sfree(NULL)Jens Axboe
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2008-04-16smalloc: remember to account for sizeof block headerJens Axboe
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2008-04-16smalloc: when adding a new pool, make it big enough to hold the failing allocJens Axboe
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2008-03-26Add softrandommap and --alloc-size optionsJens Axboe
The softrandommap job option allows fio to continue if it cannot allocate from the smalloc backing for the random map. Or the user can specify --alloc-size with a size larger than the default of 1024k to increase the size of the shared allocation pool. Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2008-03-26smalloc: increase initial pool size from 64k to 1mJens Axboe
We allocate randommaps from that pool as well and they can grow quite large. Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2008-03-06Style fixupsJens Axboe
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2008-03-04Revamp file lockingJens Axboe
Get rid of the semaphore implementation, no need to carry both. Add different locking modes (exclusive and readwrite) to enable a wider range of testing. Also combine lockfile and lockfile_batch, the latter is now a postfix option to the former. So to enable readers-excluding-writers locking mode with a lock batch count of 4, you would write: lockfile=readwrite:4 instead. Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
generated by cgit v1.2.3 (git 2.25.1) at 2024-06-30 20:48:09 +0000