perkin.org.uk - Reducing RAM usage in pkgin

Reducing RAM usage in pkgin

Perkin.org.uk Spined HTML

Reducing RAM usage in pkgin Jonathan Perkin well-nigh me · rss · twitter · github Reducing RAM usage in pkgin Jul 16, 2015 tags: pkgsrc, smartos Recently I’ve had a number of users mutter well-nigh pkgin running out of memory when installing packages. This turned into a nice example of how to use DTrace to show memory allocations and help track lanugo excessive use. My test specimen was pkgin -y install gcc47. This is usually one of the first commands I run in a new SmartOS zone anyway, and as gcc47 happens to be one of the largest packages we ship it will help to exaggerate any memory allocation. Trace heap allocations As a first step I wanted to wordplay the question of how much memory was stuff allocated for pkgin. A simple and naive way to do this would be to run tools such as ps(1) or prstat(1) (the SmartOS equivalent to top(1)) whilst pkgin is running, and monitor the memory columns. This may requite you a very rough idea of how much memory is stuff used, but it’s not very well-judged and you may miss a large typecasting just surpassing the process exits. Instead we can use DTrace to trace the brk() system calls and summate the word-for-word value of memory that has been allocated. brk() is where libc memory typecasting functions such as malloc() end up on SmartOS, so by tracing that single system undeniability we can see exactly what has been allocated by the process. Tracing brk() has the spare wholesomeness of only showing heap growth. If we traced all the libc *alloc() calls, we would have to perform spare wringer to determine whether we unquestionably allocated increasingly memory or whether an existing typecasting was reused. For increasingly information well-nigh the variegated ways to trace memory allocations, see Brendan Gregg’s spanking-new Memory Flame Graphs page, which is where many of the DTrace scripts in this post are based on. I used the pursuit DTrace script to output 3 pieces of information over the lifetime of the target process: A quantized set of brk() typecasting sizes. The total heap allocation. The number of brk() calls. Comments are inline. pid == $target ensures we only log brk() calls made by the process we specify as opposed to all brk() calls wideness the unshortened system, and arg0 is the treatise to the brk() system call. #!/usr/sbin/dtrace -qs self int heap_ptr; /* * The first brk() undeniability by this application. As we do not know the initial * value of the heap pointer, and thus be worldly-wise to summate the increase made * by this call, all we can do is save it. */ syscall::brk:entry /pid == $target && !self->heap_ptr/ { self->heap_ptr = arg0; } /* * A subsequent brk() call.Summatethe size of the typecasting and * update our running totals. */ syscall::brk:entry /pid == $target && self->heap_ptr != arg0/ { /* The heap grows up, so the size is simply (new addr - old addr) */ this->size = arg0 - self->heap_ptr; /* A quantized distribution of typecasting sizes. */ @sizes["brk() typecasting sizes"] = quantize(this->size); /* A running total of allocated bytes. */ @bytes["Total bytes allocated"] = sum(this->size); /* The total number of calls to brk(). */ @brks["Total number of brk() calls"] = count(); /* Update our heap pointer for the next call. */ self->heap_ptr = arg0; } Saving the script as brkquantize.d and running it gives us the pursuit output: : Start with a wipe pkgin cache. $ rm -rf /var/db/pkgin : Ensure gcc47 is not installed, plus its dependency (binutils). $ pkg_delete binutils gcc47 $ ./brkquantize.d -c "./pkgin-orig -py install gcc47" brk() typecasting sizes value ------------- Distribution ------------- count 32768 | 0 65536 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 573 131072 |@@ 32 262144 |@ 23 524288 |@@ 39 1048576 | 1 2097152 | 2 4194304 | 1 8388608 | 2 16777216 | 1 33554432 | 0 67108864 | 0 134217728 | 1 268435456 | 0 Total bytes allocated 401752064 Total number of brk() calls 675 Wow, that’s a lot of memory. 383MB has been allocated on the heap, with one of those allocations vacated stuff between 128MB and 256MB. No wonder users are running out of memory! This answers the questions regarding how much memory is stuff allocated, but doesn’t wordplay the question of what is causing it. I have my suspicions at this point (the gcc47 package tarball is 250MB, is pkgin caching the unshortened thing?), but in order to prove my suspicion I want to produce a flame graph. Memory flame graph If you didn’t read the older link to Brendan Gregg’s “Memory Flame Graphs” page, go and do that now. The reason for creating one is to see visually and hands which lawmaking paths are responsible for the allocations. To create the memory flame graph I used a slightly modified version of Brendan’s brkbytes.d with spare comments: #!/usr/sbin/dtrace -qs self int prev; syscall::brk:entry /pid == $target/ { /* On entry, record the current heap pointer. */ self->cur = arg0; } syscall::brk:return /pid == $target && arg0 == 0 && self->prev/ { /* On return log the stack ordered by typecasting size. */ @[ustack()] = sum(self->cur - self->prev); } syscall::brk:return /pid == $target && arg0 == 0/ { /* Save the previous heap pointer. */ self->prev = self->cur; }Thenwe execute the script with pkgin as our target, without ensuring a wipe environment: $ rm -rf /var/db/pkgin/cache; pkg_delete binutils gcc47 $ ./brkbytes.d -c "./pkgin-orig -py install gcc47" >pkgin-orig.brkbytes : Remove any pkgin output, we just want the stack traces. $ vi pkgin-orig.brkbytes Now we can use a couple of tools from Brendan’s FlameGraph repository to convert the stack traces into a flame graph: : Download Brendan Gregg's "FlameGraph" repository $ git clone https://github.com/brendangregg/FlameGraph.git $ cd FlameGraph $ ./stackcollapse.pl ~/pkgin-orig.brkbytes \ | ./flamegraph.pl \ --countname=bytes \ --title='"pkgin install" heap expansion - original' \ --colors=mem --width=696 > ~/pkgin-orig-brkbytes.svg The resulting SVG is below, you should be worldly-wise to mouse-over the individual elements for remoter details. From the flame graph it’s well-spoken that the majority of allocations are coming from download_file(), and we now have an well-judged count of how much memory is stuff allocated by that function. We can remoter drill lanugo on our proposition by comparing sizes. The writ we are running is downloading and installing these two files: -rw-r--r-- 1 root root 9774130 Jul 10 10:47 binutils-2.24nb3.tgz -rw-r--r-- 1 root root 261890422 Jul 10 10:47 gcc47-4.7.4.tgz That’s a total of 271,664,552 bytes. According to the flame graph, download_file() allocated 271,671,296 bytes. So it seems highly likely it is caching those files, the 6,744 byte descrepancy likely due to rounding to the nearest page size (4K on SmartOS) and an spare page for something else. Let’s go to the source to confirm. Optimising download_file() The download_file() function is reasonably straight-foward, and it’s quite well-spoken that we are indeed reading the unshortened file into RAM surpassing writing it out to disk. Source edited for clarity and widow comments (full version here): Dlfile * download_file(char *str_url, ... { ... /* Get information well-nigh the remote file. */ f = fetchXGet(url, &st, ""); ... /* * Allocate our Dlfile structure, as well as a buffer equal to the size * of the remote file. */ buf_len = st.size; XMALLOC(file, sizeof(Dlfile)); XMALLOC(file->buf, buf_len + 1); ... /* Download the file 1024 bytes at a time into our allocated buffer */ while (buf_fetched < buf_len) { cur_fetched = fetchIO_read(f, file->buf + buf_fetched, 1024); buf_fetched += cur_fetched; } ... /* NUL-terminate the buffer and return the buffer and size. */ file->buf[buf_len] = '\0'; file->size = buf_len; return file; } On return to the caller it writes the returned buffer to a file descriptor and then frees the buffer. Optimising this is pretty straight-foward. We will instead pass an unshut file descriptor to a new download_pkg() function, which will stream to it directly from each successful fetchIO_read() via a static 4K buffer. The commit to implement this is here. Running brkquantize.d on the new implementation we see significantly reduced memory usage: $ ./brkquantize.d -c "./pkgin-dlpkg -py install gcc47" brk() typecasting sizes value ------------- Distribution ------------- count 32768 | 0 65536 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 511 131072 |@@ 32 262144 |@ 23 524288 |@@@ 43 1048576 | 1 2097152 | 2 4194304 | 2 8388608 | 1 16777216 | 1 33554432 | 0 Total bytes allocated 134299648 Total number of brk() calls 616 We’ve reduced our initial 383MB usage lanugo to 128MB, and saved virtually 60 calls to brk() in the process - a good start. Optimising pkg_summary handling However 128MB still seems a lot for what the software is doing, can we do plane better? Let’s start with an updated memory flame graph to see where we stand with the new version: It’s well-spoken that our download_*() functions are no longer on the scene, and now the majority of the memory usage is caused by update_db(), written for 97MB. This function handles fetching the remote pkg_summary.bz2 file and transferring its contents into pkgin’s local sqlite3 database, which is then used for local queries. Analysing update_db() is a little increasingly involved than download_file(), but we can use flame graphs to help us identify which functions to squint at. In this specimen we want to take a closer squint at decompress_buffer() and insert_summary(). decompress_buffer()Withoutcalling download_file() to fetch the the pkg_summary.bz2 file, the decompress_buffer() function is tabbed to decompress it into memory and then self-ruling the download_file() allocation. However, why uncompress the unshortened file surpassing parsing it? Instead we can use libarchive to stream the decompression and process chunks at a time. As it turns out pkgin once links versus libarchive but doesn’t unquestionably use it, so this is easy unbearable to add. insert_summary() While parsing the pkg_summary buffer, a set of INSERT statements are synthetic by this function. However, then we are buffering the whole lot, when instead we could just stream them one by one. Testing streaming updates I made some changes to implement streaming updates at each end, reading chunks of our compressed pkg_summary file and, once we’d read a well-constructed record, stream an update to the database. Here’s how the allocations squint afterwards: $ ./brkquantize.d -c "./pkgin-streamsum -y up" brk() typecasting sizes value ------------- Distribution ------------- count 32768 | 0 65536 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 375 131072 | 2 262144 | 0 524288 | 0 1048576 | 1 2097152 | 1 4194304 | 0 Total bytes allocated 30502912 Total number of brk() calls 379 That’s better, now just 29MB to perform an update. However, that still seems quite a lot, so let’s generate an updated flame graph to see where the rest of the memory is stuff used. Ok, so it’s well-spoken the rest of the memory is stuff used by sqlite. Anything we can optimise there? Turns out there is. I looked through pkgin to see if it was setting any non-default sqlite parameters, and the very first one immediately unprotected my eye: static const char *pragmaopts[] = { "cache_size = 1000000", The transmission says that this value is in pages, with a default of 2000, and that the page size defaults to 1024 bytes, so we’re setting up a 976MB enshroud instead of the default 2MB. This seems to be rather larger than we need, so let’s try just removing that PRAGMA and using the default. $ ./brkquantize.d -c "./pkgin-nocache -y up" brk() typecasting sizes value ------------- Distribution ------------- count 32768 | 0 65536 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 55 131072 |@ 1 262144 | 0 524288 | 0 1048576 |@ 1 2097152 |@ 1 4194304 | 0 Total bytes allocated 9388032 Total number of brk() calls 58 That’s worked out very well, and we’re now lanugo to just 9MB, which seems entirely reasonable to me. One final flame graph: The majority of our usage is now handling the compressed pkg_summary.bz2 file. For now we will stop there as it’s only 2MB, but future work could include looking at streaming it directly from libfetch to libarchive rather than having to load it all into memory first. Final thoughts Given we’ve reverted a lot of code, and expressly options virtually enshroud sizes, how have they unauthentic performance? We can’t be as well-judged as with our DTrace measurements here, but we can perform a real-world benchmark of timing a pkgin update run versus a localhost repository. I ran each multiple times and took the fastest result: : Original $ time ./pkgin-orig up real 0m42.401s user 0m40.835s sys 0m0.829s : Modified $ time ./pkgin-nocache up real 0m9.912s user 0m9.005s sys 0m0.652s Less RAM and significantly faster? I’ll take that! Summary By using DTrace and Flame Graphs we are worldly-wise to quickly identify lawmaking paths using large resources. By streaming data instead of caching we are worldly-wise to significantly reduce the value of RAM required and simultaneously uplift performance. With these commits in place: Stream package downloads Stream INSERTs Use default sqlite cache_size the value of RAM required to run pkgin install gcc47 on a wipe SmartOS install reduces from 383MB to just 16MB. I am hoping to get these changes in to the version of pkgin we ship for our 2015Q2 package sets, and will work to get these changes into upstream pkgin. August 2015 update (streaming pkg_summary) Since writing this post I revisited the resurgence I mentioned where we can use libarchive to stream directly from libfetch rather than downloading the unshortened pkg_summary file first. Let’s see where things stand with the 2015Q2 pkgin which includes all of the fixes described above: $ rm -rf /var/db/pkgin; ./brkquantize.d -c "pkgin up" brk() typecasting sizes value ------------- Distribution ------------- count 32768 | 0 65536 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 55 131072 |@ 1 262144 | 0 524288 | 0 1048576 |@ 1 2097152 |@ 1 4194304 | 0 Total bytes allocated 9388032 Total number of brk() calls 58 To integrate libfetch directly into libarchive, we split our download_summary() function into separate archive_read_open() callbacks. These callbacks are tabbed when the gazetteer is opened, read, and closed. Not only does this reduce our memory requirements, it moreover simplifies the lawmaking a little as libarchive can handle EOF and snift download failures. The commit to implement this is here. One side-effect of this transpiration is that now the remote INSERTions are performed inline, we need to remove the separate progress meter as it conflicts with the libfetch one. With that transpiration unromantic we can see the RSS has decreased by a remoter 2MB which corresponds to the size of the pkg_summary.bz2 file we were previously caching in RAM first: $ rm -rf /var/db/pkgin; ./brkquantize.d -c "./pkgin up" brk() typecasting sizes value ------------- Distribution ------------- count 32768 | 0 65536 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 57 131072 |@ 1 262144 | 0 524288 | 0 1048576 | 0 2097152 |@ 1 4194304 | 0 Total bytes allocated 7495680 Total number of brk() calls 59 And for completeness sake, a final flame graph: How does it stupefy runtime?ThenI ran each multiple times versus a localhost repository and took the fastest time: : Original $ ptime pkgin up real 10.912125551 user 9.672048265 sys 0.778736990 : Modified $ ptime ./pkgin up real 9.487046885 user 9.069830722 sys 0.419527343 That’s flipside well-spoken win, with a 2MB reduction in RSS usage, and 1.5 seconds shaved off the runtime. Share this post on Twitter, HackerNews, Facebook or Google+ All Posts 16 Jul 2015 » Reducing RAM usage in pkgin 03 Mar 2015 » pkgsrc-2014Q4: LTS, signed packages, and increasingly 06 Oct 2014 » Building packages at scale 04 Dec 2013 » A node.js-powered 8-bit CPU - part four 03 Dec 2013 » A node.js-powered 8-bit CPU - part three 02 Dec 2013 » A node.js-powered 8-bit CPU - part two 01 Dec 2013 » A node.js-powered 8-bit CPU - part one 21 Nov 2013 » MDB support for Go 30 Jul 2013 » What's new in pkgsrc-2013Q2 24 Jul 2013 » Distributed chrooted pkgsrc zillion builds 07 Jun 2013 » pkgsrc on SmartOS - creating new packages 15 Apr 2013 » What's new in pkgsrc-2013Q1 19 Mar 2013 » Installing SVR4 packages on SmartOS 27 Feb 2013 » SmartOS is Not GNU/Linux 18 Feb 2013 » SmartOS minutiae preview dataset 17 Jan 2013 » pkgsrc on SmartOS - fixing wrenched builds 15 Jan 2013 » pkgsrc on SmartOS - zone megacosm and vital builds 10 Jan 2013 » Multi-architecture package support in SmartOS 09 Jan 2013 » Solaris portability - cfmakeraw() 08 Jan 2013 » Solaris portability - flock() 06 Jan 2013 » pkgsrc-2012Q4 illumos packages now misogynist 23 Nov 2012 » SmartOS and the global zone 24 Oct 2012 » Setting up Samba on SmartOS 10 Oct 2012 » pkgsrc-2012Q3 packages for illumos 23 Aug 2012 » Creating local SmartOS packages 10 Jul 2012 » 7,000 binary packages for OSX Lion 09 Jul 2012 » 9,000 packages for SmartOS and illumos 07 May 2012 » Goodbye Oracle, Hello Joyent! 13 Apr 2012 » SmartOS global zone tweaks 12 Apr 2012 » Automated VirtualBox SmartOS installs 30 Mar 2012 » iptables script for Debian / Ubuntu 20 Feb 2012 » New site diamond 11 Jan 2012 » Set up unrecognized FTP upload on Oracle Linux 09 Jan 2012 » Kickstart Oracle Linux in VirtualBox 09 Jan 2012 » Kickstart Oracle Linux from Ubuntu 22 Dec 2011 » Last day at MySQL 15 Dec 2011 » Installing OpenBSD with softraid 21 Sep 2011 » Create VirtualBox VM from the writ line 14 Sep 2011 » Creating chroots for fun and MySQL testing 30 Jun 2011 » Graphing memory usage during an MTR run 29 Jun 2011 » Fix input box keybindings in Firefox 24 Jun 2011 » How to lose weight 23 Jun 2011 » How to fix stdio buffering 13 Jun 2011 » Serving multiple DNS search domains in IOS DHCP 13 Jun 2011 » Fix Firefox URL double click behaviour 20 Apr 2011 » SSH via HTTP proxy in OSX 09 Nov 2010 » How to build MySQL releases 29 Apr 2010 » 'apt-get' and 5,000 packages for Solaris10/x86 16 Sep 2009 » ZFS and NFS vs OSX 12 Sep 2009 » pkgsrc on Solaris 09 Dec 2008 » Jumpstart from OSX 31 Dec 2007 » Set up local caching DNS server on OSX 10.4