perkin.org.uk - pkgsrc on SmartOS - fixing broken builds

pkgsrc on SmartOS - fixing broken builds

Perkin.org.uk Spined HTML

pkgsrc on SmartOS - fixing wrenched builds Jonathan Perkin well-nigh me · rss · twitter · github pkgsrc on SmartOS - fixing wrenched builds Jan 17, 2013 tags: pkgsrc, smartos This is the second in a series of posts looking at pkgsrc on SmartOS. In the previous post I got us up and running with towers packages. This post will focus on what to do when the build fails. Currently pkgsrc is worldly-wise to build virtually 9,500 packages on SmartOS, however pkgsrc carries over 12,000 packages in total, so there is a reasonable endangerment that you will come wideness a package which will not produce a binary package. Let’s have a squint at some of the most worldwide failure modes and the facilities pkgsrc provides for them to be fixed. Adjusting the environment Probably the most worldwide failures on Solaris are those caused by incorrect compiler or linker flags, where the tragedian of the software has not taken into worth differences wideness Unix platforms. Examples include: the usage of u_int* types (e.g. u_int32_t) instead of the portable uint* C99 types (e.g. uint32_t). missing -lsocket -lnsl when using the socket interface. pkgsrc provides an easy way to pass CFLAGS, LDFLAGS and other worldwide environment variables to the build, and these are often unbearable to resolve such issues. Taking net/nsd as an example: # cd net/nsd # bmake package ... gcc -I/opt/local/include -I/opt/local/include -I. -I. -O -I/opt/local/include -c ./util.c ./util.c:745:1: error: unknown type name 'u_int32_t' *** [util.o] Error lawmaking 1 If we edit the Makefile and add the pursuit line unelevated CONFIGURE_ARGS: CFLAGS.SunOS+= -Du_int32_t=uint32_t then a rebuild resolves the problem and results in a binary package: # bmake wipe # bmake package ... => Creating binary package /content/packages/All/nsd-3.2.14.tgz By using CFLAGS.SunOS rather than the global CFLAGS this is only performed on systems where uname is SunOS. We can resolve missing libraries in a similar way, taking net/rootprobe as an example: # cd net/rootprobe # bmake package ... gcc -Wl,-R/opt/local/lib rootprobe.o -o rootprobe Undefined first referenced symbol in file recv rootprobe.o send rootprobe.o getsockname rootprobe.o socket rootprobe.o getdomainname rootprobe.o connect rootprobe.o recvfrom rootprobe.o inet_aton rootprobe.o inet_ntoa rootprobe.o shutdown rootprobe.o ld: fatal: symbol referencing errors. No output written to rootprobe collect2: error: ld returned 1 exit status *** [rootprobe] Error lawmaking 1 Add the pursuit to the Makefile: LDFLAGS.SunOS+= -lsocket -lnsl and hey presto, out comes a binary package: # bmake wipe # bmake package ... gcc -lsocket -lnsl -Wl,-R/opt/local/lib rootprobe.o -o rootprobe /bin/rm -f cctldprobe; ln rootprobe cctldprobe ... => Creating binary package /content/packages/All/rootprobe-200301.tgz Not all packages can be stock-still directly like this, only those which obey the normal ${CC} and ${LD} environment variables, as pkgsrc creates wrappers for those commands where it can insert these alterations. For packages which directly undeniability e.g. gcc, some spare digging will be required to see how the arguments can be passed. Taking net/3proxy as an example, the build fails with missing socket libraries: # cd net/3proxy # bmake package ... gcc -opop3p -Wall -O2 -pthread sockmap.o pop3p.o sockgetchar.o myalloc.o common.o Undefined first referenced symbol in file tighten pop3p.o send sockgetchar.o getsockname common.o winnow pop3p.o listen pop3p.o gethostbyname common.o sendto sockmap.o socket pop3p.o setsockopt pop3p.o connect common.o recvfrom sockmap.o shutdown sockmap.o ld: fatal: symbol referencing errors. No output written to pop3p collect2: error: ld returned 1 exit status *** [pop3p] Error lawmaking 1 but subtracting LDFLAGS.SunOS+= -lsocket -lnsl to the Makefile as surpassing does not resolve the problem.Proberemoter into the Makefile we can see that the build is driven from a custom Makefile rather than through autoconf/automake: MAKE_FILE= Makefile.unix and it is there we will need to perform the changes. First, let’s find the work area, which we can get from the WRKSRC variable: # bmake show-var VARNAME=WRKSRC /var/tmp/pkgsrc-build/net/3proxy/work # cd /var/tmp/pkgsrc-build/net/3proxy/work # ls -l Makefile.unix -rw-r--r-- 1 11001 10512 675 Apr 30 2005 Makefile.unix Picking out the relevant shit from Makefile.unix: CC = gcc ... CFLAGS = -Wall -g -O2 -c -pthread -D_THREAD_SAFE -D_REENTRANT -DNOODBC -DWITH_STD_MALLOC -DFD_SETSIZE=4096 -DWITH_POLL ... LDFLAGS = -Wall -O2 -pthread ... Here, the tragedian has provided no functionality for subtracting to the environment, and has instead chosen to hardcode a specific compiler and build flags, significantly reducing the portability of their software - good luck to Clang/LLVM or SunStudio users! If the tragedian had provided a way in to add to these flags, for example: LDFLAGS = -Wall -O2 -pthread ${USER_LDFLAGS} then we could have stock-still that in the main pkgsrc Makefile with MAKE_ENV+= USER_LDFLAGS="-lsocket -lnsl" or similar, however we now have no nomination but to directly edit Makefile.unix. Thankfully, pkgsrc provides a couple of ways to hands do this, which we will squint at over the next few sections. The substitution framework The subst framework allows vital editing of files within the work area. I will show the solution for the whilom problem, and then discuss it: .include "../../mk/bsd.prefs.mk" .if ${OPSYS} == "SunOS" SUBST_CLASSES+= libs SUBST_STAGE.libs= pre-build SUBST_MESSAGE.libs=SubtractingSunOS socket libraries SUBST_FILES.libs= Makefile.unix SUBST_SED.libs= -e '/^LDFLAGS/s/$$/ -lsocket -lnsl/' .endif Firstly, we need to limit this to SunOS systems, else we would unravel platforms which do not have libsocket or libnsl. Including ../../mk/bsd.prefs.mk gives us wangle to the OPSYS variable so we can test the platform we are running on. Next we set up the substitution framework: SUBST_CLASSES creates a new class, which I have named libs. This matriculation name is then appended to the remaining SUBST_* variables to assign them to that class. SUBST_STAGE defines the make stage when the substitution will be run, and should scrutinizingly unchangingly be pre-build, to ensure it is washed-up without any configuration stage which could rewrite files itself. SUBST_MESSAGE is optional, and is simply a line which will be printed to the user when the substitution takes places. SUBST_FILES is a list of files the substitution operates on. SUBST_SED is the very substitution, in the form of a sed(1) operation. Here we suspend -lsocket -lnsl to any line whence with LDFLAGS. Note $$ is required to get make to escape a $. # bmake wipe # bmake package ... ===>Towersfor 3proxy-0.5.3.11nb1 =>SubtractingSunOS socket libraries ... => Creating binary package /content/packages/All/3proxy-0.5.3.11nb1.tgz Two other subst features you may want are: SUBST_VARS.foo= VARNAME is a shortcut for the -e 's,@VARNAME@,${VARNAME},g' operation, worldwide with autoconf-based packages. SUBST_FILTER_CMD.foo= <cmd> allows you to run an wrong-headed command, rather than the default of sed. For increasingly information, see the implementation in pkgsrc/mk/subst.mk.  Patches While CFLAGS, LDFLAGS and the substitution framework indulge for simple one-liner fixes, often increasingly significant changes are required, and in those cases the only sensible option is to create patches. Thankfully, there are some tools provided in pkgsrc to make this a relatively easy process. First, let’s take a wrenched package, devel/bglibs: # cd devel/bglibs # bmake package ... ./ltcompile net/cork.c net/cork.c: In function 'socket_cork': net/cork.c:39:27: error: 'SOL_TCP' undeclared (first use in this function) net/cork.c:39:27: note: each undeclared identifier is reported only once for each function it appears in *** [net/cork.lo] Error lawmaking 1 which comes from the pursuit function: /* * ..It is known to work on Linux (with the TCP_CORK option) and to at least * compile on BSD (with the TCP_NOPUSH option). On OS's which lack either of * these two options, this function is substantially a no-op. */ int socket_cork(int sock) { #if defined(TCP_CORK) int flag = 1; return setsockopt(sock, SOL_TCP, TCP_CORK, &flag, sizeof flag) == 0; #elif defined(TCP_NOPUSH) int flag = 1; return setsockopt(sock, SOL_SOCKET, TCP_NOPUSH, &flag, sizeof flag) == 0; #else return 1; #endif } Unfortunately, while SmartOS provides the TCP_CORK flag, it does not understand SOL_TCP, and so we need to modify the test and add an spare one.Surpassingdoing this, we should install the pkgtools/pkgdiff package, which contains a pkgvi wrapper. This allows you to edit a file, and if you make changes, it will save the resulting unequal in the pkgsrc patches directory ready for use. Much easier than probe into the work zone and creating patches yourself. : Install via pkgin.. # pkgin in pkgdiff : ..or through pkgsrc # (cd ../../pkgtools/pkgdiff && bmake install) Now edit the offending file: # pkgvi $(bmake show-var VARNAME=WRKSRC)/net/cork.c This is what I reverted the function to: int socket_cork(int sock) { #if defined(TCP_CORK) && defined(SOL_TCP) int flag = 1; return setsockopt(sock, SOL_TCP, TCP_CORK, &flag, sizeof flag) == 0; #elif defined(TCP_CORK) && defined(SOL_SOCKET) int flag = 1; return setsockopt(sock, SOL_SOCKET, TCP_CORK, &flag, sizeof flag) == 0; #elif defined(TCP_NOPUSH) int flag = 1; return setsockopt(sock, SOL_SOCKET, TCP_NOPUSH, &flag, sizeof flag) == 0; #else return 1; #endif } This still isn’t ideal, and would be largest converted to a proper autoconf test where we can test functionality instead of definitions, but it will do for example purposes.Withoutwriting, you should get output such as: pkgvi: File was modified. For a diff, type: pkgdiff "/var/tmp/pkgsrc-build/devel/bglibs/work/bglibs-1.106/net/cork.c" and running the pkgdiff writ will show you the diff. The final step is storing the unequal into a patch file, and that is workaday with: # mkpatches # bmake mps The mkpatches writ creates a new file in the patches directory tabbed patch-net_cork.c, and the bmake mps (short for the makepatchsum target) regenerates the distinfo file with the correct checksum for that patch. Finally, you can rebuild # bmake wipe # bmake package and, hey presto .. flipside failure! --- net/uncork.lo --- net/uncork.c: In function 'socket_uncork': net/uncork.c:30:27: error: 'SOL_TCP' undeclared (first use in this function) net/uncork.c:30:27: note: each undeclared identifier is reported only once for each function it appears in Now that you know how to fix this, I’ll leave it to you to come up with a patch ;) One final word on this, you will perhaps notice during the build some warnings: => Applying pkgsrc patches for bglibs-1.106nb1 => Ignoring patchfile /content/pkgsrc/devel/bglibs/patches/patch-ab.orig => Ignoring patchfile /content/pkgsrc/devel/bglibs/patches/patch-ac.orig => Ignoring patchfile /content/pkgsrc/devel/bglibs/patches/patch-net_cork.c.orig These files are left by mkpatches, and to remove them you can run # mkpatches -c Also note that both patch-ab and patch-ac changed, plane though we only modified the net/cork.c file. This is due to differences in diff(1) output, and is ultimately harmless. Summary pkgsrc provides a number of ways to fix up wrenched software: CFLAGS, LDFLAGS, and other environment variables. The subst.mk framework for simple file changes. Patch files for increasingly substantial changes. The important thing, without using any of these features, is to feed when your changes so that we can integrate them into pkgsrc and everyone can benefit. Probably the easiest way to do that is simply raise an issue versus our GitHub pkgsrc fork, and either myself or Filip can commit the patch upstream. Share this post on Twitter, HackerNews, Facebook or Google+ All Posts 16 Jul 2015 » Reducing RAM usage in pkgin 03 Mar 2015 » pkgsrc-2014Q4: LTS, signed packages, and increasingly 06 Oct 2014 »Towerspackages at scale 04 Dec 2013 » A node.js-powered 8-bit CPU - part four 03 Dec 2013 » A node.js-powered 8-bit CPU - part three 02 Dec 2013 » A node.js-powered 8-bit CPU - part two 01 Dec 2013 » A node.js-powered 8-bit CPU - part one 21 Nov 2013 » MDB support for Go 30 Jul 2013 » What's new in pkgsrc-2013Q2 24 Jul 2013 » Distributed chrooted pkgsrc zillion builds 07 Jun 2013 » pkgsrc on SmartOS - creating new packages 15 Apr 2013 » What's new in pkgsrc-2013Q1 19 Mar 2013 » Installing SVR4 packages on SmartOS 27 Feb 2013 » SmartOS is Not GNU/Linux 18 Feb 2013 » SmartOS minutiae preview dataset 17 Jan 2013 » pkgsrc on SmartOS - fixing wrenched builds 15 Jan 2013 » pkgsrc on SmartOS - zone megacosm and vital builds 10 Jan 2013 » Multi-architecture package support in SmartOS 09 Jan 2013 » Solaris portability - cfmakeraw() 08 Jan 2013 » Solaris portability - flock() 06 Jan 2013 » pkgsrc-2012Q4 illumos packages now misogynist 23 Nov 2012 » SmartOS and the global zone 24 Oct 2012 » Setting up Samba on SmartOS 10 Oct 2012 » pkgsrc-2012Q3 packages for illumos 23 Aug 2012 » Creating local SmartOS packages 10 Jul 2012 » 7,000 binary packages for OSX Lion 09 Jul 2012 » 9,000 packages for SmartOS and illumos 07 May 2012 » Goodbye Oracle, Hello Joyent! 13 Apr 2012 » SmartOS global zone tweaks 12 Apr 2012 » Automated VirtualBox SmartOS installs 30 Mar 2012 » iptables script for Debian / Ubuntu 20 Feb 2012 » New site diamond 11 Jan 2012 » Set up unrecognized FTP upload on Oracle Linux 09 Jan 2012 » Kickstart Oracle Linux in VirtualBox 09 Jan 2012 » Kickstart Oracle Linux from Ubuntu 22 Dec 2011 » Last day at MySQL 15 Dec 2011 » Installing OpenBSD with softraid 21 Sep 2011 » Create VirtualBox VM from the writ line 14 Sep 2011 » Creating chroots for fun and MySQL testing 30 Jun 2011 » Graphing memory usage during an MTR run 29 Jun 2011 » Fix input box keybindings in Firefox 24 Jun 2011 » How to lose weight 23 Jun 2011 » How to fix stdio buffering 13 Jun 2011 » Serving multiple DNS search domains in IOS DHCP 13 Jun 2011 » Fix Firefox URL double click behaviour 20 Apr 2011 » SSH via HTTP proxy in OSX 09 Nov 2010 » How to build MySQL releases 29 Apr 2010 » 'apt-get' and 5,000 packages for Solaris10/x86 16 Sep 2009 » ZFS and NFS vs OSX 12 Sep 2009 » pkgsrc on Solaris 09 Dec 2008 » Jumpstart from OSX 31 Dec 2007 » Set up local caching DNS server on OSX 10.4