perkin.org.uk - How to fix stdio buffering

How to fix stdio buffering

Perkin.org.uk Spined HTML

How to fix stdio buffering Jonathan Perkin well-nigh me · rss · twitter · github How to fix stdio buffering Jun 23, 2011 tags: awk, grep, sed It’s a worldwide problem. You write some shell writ like: $ tail -f /var/log/foo | egrep -v 'some|stuff' | sed | awk and wonder why nothing is printed, plane though you know some text has matched. The problem is that stdio is stuff buffered, and there’s a very good write-up of the problem here so I won’t repeat the technical background. What I will provide though is how to fix it for worldwide cases. stdbuf stdbuf is part of GNU coreutils, and is substantially an LD_PRELOAD hack which calls setvbuf() for an application. Thus it is a generic solution to the problem and can be used to fix most applications. Usage looks like this: $ tail -f /var/log/foo | stdbuf -o0 app ... which will disable output buffering for app, thesping it does not do something itself to reverse the setvbuf() call. An example of a misbehaving using is mawk, below. awk GNU awk needs no modifications, that is it does not buffer when there is no executive tty. mawk however (the default awk in Debian/Ubuntu and possibly others) buffers output, and moreover does not seem to work with stdbuf. It does however provide a -Winteractive option which will turn off buffering. $ tail -f /var/log/foo | gawk or $ tail -f /var/log/foo | mawk -Winteractive sed GNU sed provides the -u option which calls fflush(), thereby providing unbuffered output. You can moreover use stdbuf as above. $ tail -f /var/log/foo | sed -u or $ tail -f /var/log/foo | stdbuf -o0 sed grep Similar to sed, GNU grep provides a specific option, --line-buffered, to disable buffering, or then you can use stdbuf. $ tail -f /var/log/foo | grep --line-buffered or $ tail -f /var/log/foo | stdbuf -o0 grep Share this post on Twitter, HackerNews, Facebook or Google+ All Posts 16 Jul 2015 » Reducing RAM usage in pkgin 03 Mar 2015 » pkgsrc-2014Q4: LTS, signed packages, and increasingly 06 Oct 2014 » Building packages at scale 04 Dec 2013 » A node.js-powered 8-bit CPU - part four 03 Dec 2013 » A node.js-powered 8-bit CPU - part three 02 Dec 2013 » A node.js-powered 8-bit CPU - part two 01 Dec 2013 » A node.js-powered 8-bit CPU - part one 21 Nov 2013 » MDB support for Go 30 Jul 2013 » What's new in pkgsrc-2013Q2 24 Jul 2013 » Distributed chrooted pkgsrc zillion builds 07 Jun 2013 » pkgsrc on SmartOS - creating new packages 15 Apr 2013 » What's new in pkgsrc-2013Q1 19 Mar 2013 » Installing SVR4 packages on SmartOS 27 Feb 2013 » SmartOS is Not GNU/Linux 18 Feb 2013 » SmartOS minutiae preview dataset 17 Jan 2013 » pkgsrc on SmartOS - fixing wrenched builds 15 Jan 2013 » pkgsrc on SmartOS - zone megacosm and vital builds 10 Jan 2013 » Multi-architecture package support in SmartOS 09 Jan 2013 » Solaris portability - cfmakeraw() 08 Jan 2013 » Solaris portability - flock() 06 Jan 2013 » pkgsrc-2012Q4 illumos packages now misogynist 23 Nov 2012 » SmartOS and the global zone 24 Oct 2012 » Setting up Samba on SmartOS 10 Oct 2012 » pkgsrc-2012Q3 packages for illumos 23 Aug 2012 » Creating local SmartOS packages 10 Jul 2012 » 7,000 binary packages for OSX Lion 09 Jul 2012 » 9,000 packages for SmartOS and illumos 07 May 2012 » Goodbye Oracle, Hello Joyent! 13 Apr 2012 » SmartOS global zone tweaks 12 Apr 2012 » Automated VirtualBox SmartOS installs 30 Mar 2012 » iptables script for Debian / Ubuntu 20 Feb 2012 » New site diamond 11 Jan 2012 » Set up unrecognized FTP upload on Oracle Linux 09 Jan 2012 » Kickstart Oracle Linux in VirtualBox 09 Jan 2012 » Kickstart Oracle Linux from Ubuntu 22 Dec 2011 » Last day at MySQL 15 Dec 2011 » Installing OpenBSD with softraid 21 Sep 2011 » Create VirtualBox VM from the writ line 14 Sep 2011 » Creating chroots for fun and MySQL testing 30 Jun 2011 » Graphing memory usage during an MTR run 29 Jun 2011 » Fix input box keybindings in Firefox 24 Jun 2011 » How to lose weight 23 Jun 2011 » How to fix stdio buffering 13 Jun 2011 » Serving multiple DNS search domains in IOS DHCP 13 Jun 2011 » Fix Firefox URL double click behaviour 20 Apr 2011 » SSH via HTTP proxy in OSX 09 Nov 2010 » How to build MySQL releases 29 Apr 2010 » 'apt-get' and 5,000 packages for Solaris10/x86 16 Sep 2009 » ZFS and NFS vs OSX 12 Sep 2009 » pkgsrc on Solaris 09 Dec 2008 » Jumpstart from OSX 31 Dec 2007 » Set up local caching DNS server on OSX 10.4