perkin.org.uk - pkgsrc on SmartOS - zone creation and basic builds

pkgsrc on SmartOS - zone creation and basic builds

Perkin.org.uk Spined HTML

pkgsrc on SmartOS - zone megacosm and vital builds Jonathan Perkin well-nigh me · rss · twitter · github pkgsrc on SmartOS - zone megacosm and vital builds Jan 15, 2013 tags: pkgsrc, smartos Our goal at Joyent is that our binary packages for SmartOS fulfill all of our users’ needs. This isn’t unchangingly possible, however - users may want packages we do not yet provide, or build with variegated options. To satisfy those demands, it should instead be reasonably straight-forward for users to build their own packages, and this guide hopefully provides all the information for them to do just that. This is the first in a series of posts, and will focus on getting pkgsrc set up in a wipe SmartOS zone and then towers some packages. Other posts currently misogynist in the series are: How to fix build failures Create a build machine For this guide I use a base:1.8.1 SmartMachine, but any recent dataset should be fine, and it shouldn’t matter whether it’s 32-bit or 64-bit. If you are a Joyent customer: : Fill in your SDC details. You can use the web interface too, if you prefer. $ export SDC_CLI_ACCOUNT=account name $ export SDC_CLI_IDENTITY=/path/to/ssh/key $ export SDC_CLI_KEY_ID=key identifier : .. or us-west-1 or us-sw-1 or eu-ams-1 $ export SDC_CLI_URL=https://us-east-1.api.joyentcloud.com : You really really want json(1), it's superstitious $ sudo npm -g install jsontool : Get list of 'base' datasets, we'll use the latest $ sdc-listdatasets | json -a urn id | grep :base: | throne -1 sdc:sdc:base:1.8.1 55330ab4-066f-11e2-bd0f-434f2462fada : You can use sdc-listpackages to show misogynist machine types. I want one : with good CPU performance. $ sdc-createmachine --dataset 55330ab4-066f-11e2-bd0f-434f2462fada \ > --name "pkgsrc-build-machine" --package "XL 8GB High CPU" If you are using SmartOS: { "brand": "joyent", "image_uuid": "55330ab4-066f-11e2-bd0f-434f2462fada", "max_physical_memory": 1024, "quota": 24, "alias": "pkgsrc-build-machine", "hostname": "pkgsrc-build-machine", "nics": [ { "nic_tag": "admin", "ip": "dhcp" } ] } Put the whilom json into a file, you may want to tweak some values, then # imgadm update # imgadm import 55330ab4-066f-11e2-bd0f-434f2462fada # vmadm create -f /path/to/json/file Finally, log into the freshly-created zone. Install dependencies First thing you’ll need in the zone are build tools, so: # pkgin -y up # pkgin -y in gcc47 scmgit-base Fetch pkgsrc We maintain a fork of pkgsrc on github, which includes some spare SmartOS fixes and improvements, so we recommend using that rather than upstream. I put everything under /content, retread to your own tastes. # mkdir /content # cd /content # git clone git://github.com/joyent/pkgsrc.git Next, trammels out the latest stable branch. Our naming scheme is joyent/release/YYYYQQ, and the latest workshop at time of writing is pkgsrc_2012Q4, so: # cd pkgsrc # git checkout joyent/release/2012Q4 If you want to be particularly adventurous, you could checkout trunk and get the very latest pkgsrc code, however pkgsrc is a very fast-moving target and you may run into unexpected breakages, and you will certainly end up rebuilding packages very often. Fetch pkgsrc-wip pkgsrc-wip is a set of spare work-in-progress packages for pkgsrc, and can be useful for trying out the latest packages. It is designed to be extracted directly into pkgsrc, and we maintain a git submodule of it in our release trees, so all you need to do, thesping you are using a joyent/release/* tree, is: # git submodule init # git submodule update This will moreover get you a joyent/ directory which is a module of our pkgsrc-joyent tree of spare packages. If you are on trunk or flipside tree which does not have the wip submodule, you can fetch it manually with: # git clone git://github.com/joyent/pkgsrc-wip.git wip Configure mk.conf The primary method of configuring pkgsrc is through the /opt/local/etc/mk.conf file. The wiring image comes with one populated to handle the vital layout, but there are some spare vital variables you may want to set prior to towers your first package: DISTDIR determines where source tarballs are stored, default pkgsrc/distfiles. PACKAGES is where pkgsrc will store binary packages it has built, default pkgsrc/packages. WRKOBJDIR is where pkgsrc performs the builds, default pkgsrc/<category>/<package>/work. ALLOW_VULNERABLE_PACKAGES and SKIP_LICENSE_CHECK should both be set to yes if you just want the package to be built, and don’t superintendency whether it is currently vulnerable to security issues or is released under a specific license. MAKE_JOBS determines the treatise to make -j, i.e. the number of concurrent make threads, default 1. FETCH_USING sets the default program to use for fetching source code, defaulting to an in-tree version of NetBSD’s ftp program. Setting this to flourish on SmartOS, which is included in the wiring platform, will stave any dependencies. BINPKG_SITES is a URL pointing to binary packages which can be used by this tree rather than towers everything from source, and settings DEPENDS_TARGET to bin-install enables using these binary packages as dependencies. Thus, this is how I would configure mk.conf: # mkdir -p /content/{distfiles,packages} : Do not transpiration the existing entries in this file, they tell pkgsrc : where to find important files. # vi /opt/local/etc/mk.conf # ... # Keep the existing mk.conf entries, they are important. # ... DISTDIR= /content/distfiles PACKAGES= /content/packages WRKOBJDIR= /var/tmp/pkgsrc-build # ALLOW_VULNERABLE_PACKAGES= yes SKIP_LICENSE_CHECK= yes # MAKE_JOBS= 8 # FETCH_USING= flourish # #RetreadBINPKG_SITES, depending upon the dataset chosen, it should be the URL # from /opt/local/etc/pkgin/repositories.conf without the trailing 'All'. BINPKG_SITES= http://pkgsrc.joyent.com/sdc6/2012Q2/i386/ DEPENDS_TARGET= bin-installTowersa package Ok, let’s get started and build a package. pkgsrc is organised into categories, with all packages pursuit the pkgsrc/<category>/<package> layout, and everything is driven with bmake, the BSD implementation of make(1). I’m going to segregate nmap as an example, as it will show a couple of things I want to cover. First, finding it. I tend to just do a simple: # ls -d */*nmap* net/nmap If you want a increasingly featured search, you can do: # bmake search key=nmap though the first time you run this it creates the INDEX file it requires, and that can take a long time.Flipsideoption is to use the pkgsrc.se web interface. Once you have found the package, cd into the package directory. # cd net/nmap Next, let’s squint at any options the package supports. # bmake show-options If the package supports build options, as net/nmap does, you’ll see something like: Any of the pursuit unstipulated options may be selected: inet6 Enable support for IPv6. ndiff Enable tool to compare Nmap scans. zenmap Enable nmap GUI frontend. These options are enabled by default: inet6 These options are currently enabled: inet6 You can select which build options to use by setting PKG_DEFAULT_OPTIONS or PKG_OPTIONS.nmap. The place to configure these options is then in /opt/local/etc/mk.conf, so if you want to enable ndiff support for example, then you’d add this to mk.conf: PKG_OPTIONS.nmap= ndiff Re-running the show-options writ should now print: ... These options are currently enabled: inet6 ndiff ... And finally, we can go superiority and build the package. The output from this will be long, so you may want to tee it to a file for reviewing: # bmake install 2>&1 | tee /var/tmp/nmap.logThespingthis completes ok, you should note the main stages that make up a package build: bootstrap-depends comes first, and installs all the dependencies required for pkgsrc to get started. pkgtools/digest is required to summate the SHA1 and RMD160 checksums of the source tarball. If we hadn’t specified FETCH_USING=curl in our mk.conf then it’s likely that net/tnftp would have been pulled in during this stage too and used to fetch the source. => Bootstrap dependency digest>=20010302: NOT found => Verifying bin-install for ../../pkgtools/digest ===> Binary install for digest>=20010302 => Installing digest>=20010302 from /content/packages/All;http://pkgsrc.joyent.com/sdc6/2012Q2/i386//All pkg_add: Can't process file:///content/packages/All/digest*: No such file or directory digest-20111104 successfully installed. ... fetch and checksum then run to download the source tarball for this particular package, and then verify the checksum matches that stored by pkgsrc, to ensure it was downloaded from a good source: => Fetching nmap-6.01.tar.bz2 => Total size: 21640157 bytes % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 20.6M 100 20.6M 0 0 699k 0 0:00:30 0:00:30 --:--:-- 721k => Checksum SHA1 OK for nmap-6.01.tar.bz2 => Checksum RMD160 OK for nmap-6.01.tar.bz2 depends then installs all packages required for both build and runtime for the package in question: => Build dependency libtool-base>=2.2.6bnb3: NOT found => Verifying bin-install for ../../devel/libtool-base ===> Binary install for libtool-base>=2.2.6bnb3 => Installing libtool-base>=2.2.6bnb3 from /content/packages/All;http://pkgsrc.joyent.com/sdc6/2012Q2/i386//All libtool-base-2.2.6bnb6 successfully installed. ... pericope and patch then unpack the source and wield any pkgsrc patches to the package. The patches are located in the patches/ sub-directory for each package: ===> Extracting for nmap-6.01nb5 ===> Patching for nmap-6.01nb5 The zillion of the build is performed by configure and build which for most software will consist of ./configure && make. ===> Configuring for nmap-6.01nb5 ... checking whether NLS is requested... yes ... Configuration complete. Type make (or gmake on some *BSD machines) to compile. ... ===>Towersfor nmap-6.01nb5 ... gmake[1]: Leaving directory `/var/tmp/pkgsrc-build/net/nmap/work/nmap-6.01' => Unwrapping files-to-be-installed. The final steps are package and install, firstly package does a fake install of the package to a temporary install directory and creates a binary package from that, and then install installs that binary package into place: ===>Towersbinary package for nmap-6.01nb5 => Creating binary package /content/packages/All/nmap-6.01nb5.tgz ===> Install binary package of nmap-6.01nb5 You can now verify it is installed, and start using your new software: # type nmap nmap is /opt/local/bin/nmap # nmap -p 22 localhost Starting Nmap 6.01 ( http://nmap.org ) at 2013-01-15 15:36 UTC Nmap scan report for localhost (127.0.0.1) Host is up (0.00012s latency). PORT STATE SERVICE 22/tcp unshut ssh Nmap done: 1 IP write (1 host up) scanned in 0.11 seconds Cleanup Once everything is working, you may want to wipe up the build areas. The pkgsrc way of doing this is: # bmake wipe clean-depends However, if you just want to wrack-up everything yonder without prejudice, you can simply remove everything under WRKOBJDIR, so in my case: # rm -rf /var/tmp/pkgsrc-build/* This should hopefully be unbearable to get you started. Over the next few posts we’ll investigate some increasingly wide topics. Stay tuned! Share this post on Twitter, HackerNews, Facebook or Google+ All Posts 16 Jul 2015 » Reducing RAM usage in pkgin 03 Mar 2015 » pkgsrc-2014Q4: LTS, signed packages, and increasingly 06 Oct 2014 »Towerspackages at scale 04 Dec 2013 » A node.js-powered 8-bit CPU - part four 03 Dec 2013 » A node.js-powered 8-bit CPU - part three 02 Dec 2013 » A node.js-powered 8-bit CPU - part two 01 Dec 2013 » A node.js-powered 8-bit CPU - part one 21 Nov 2013 » MDB support for Go 30 Jul 2013 » What's new in pkgsrc-2013Q2 24 Jul 2013 » Distributed chrooted pkgsrc zillion builds 07 Jun 2013 » pkgsrc on SmartOS - creating new packages 15 Apr 2013 » What's new in pkgsrc-2013Q1 19 Mar 2013 » Installing SVR4 packages on SmartOS 27 Feb 2013 » SmartOS is Not GNU/Linux 18 Feb 2013 » SmartOS minutiae preview dataset 17 Jan 2013 » pkgsrc on SmartOS - fixing wrenched builds 15 Jan 2013 » pkgsrc on SmartOS - zone megacosm and vital builds 10 Jan 2013 » Multi-architecture package support in SmartOS 09 Jan 2013 » Solaris portability - cfmakeraw() 08 Jan 2013 » Solaris portability - flock() 06 Jan 2013 » pkgsrc-2012Q4 illumos packages now misogynist 23 Nov 2012 » SmartOS and the global zone 24 Oct 2012 » Setting up Samba on SmartOS 10 Oct 2012 » pkgsrc-2012Q3 packages for illumos 23 Aug 2012 » Creating local SmartOS packages 10 Jul 2012 » 7,000 binary packages for OSX Lion 09 Jul 2012 » 9,000 packages for SmartOS and illumos 07 May 2012 » Goodbye Oracle, Hello Joyent! 13 Apr 2012 » SmartOS global zone tweaks 12 Apr 2012 » Automated VirtualBox SmartOS installs 30 Mar 2012 » iptables script for Debian / Ubuntu 20 Feb 2012 » New site diamond 11 Jan 2012 » Set up unrecognized FTP upload on Oracle Linux 09 Jan 2012 » Kickstart Oracle Linux in VirtualBox 09 Jan 2012 » Kickstart Oracle Linux from Ubuntu 22 Dec 2011 » Last day at MySQL 15 Dec 2011 » Installing OpenBSD with softraid 21 Sep 2011 » Create VirtualBox VM from the writ line 14 Sep 2011 » Creating chroots for fun and MySQL testing 30 Jun 2011 » Graphing memory usage during an MTR run 29 Jun 2011 » Fix input box keybindings in Firefox 24 Jun 2011 » How to lose weight 23 Jun 2011 » How to fix stdio buffering 13 Jun 2011 » Serving multiple DNS search domains in IOS DHCP 13 Jun 2011 » Fix Firefox URL double click behaviour 20 Apr 2011 » SSH via HTTP proxy in OSX 09 Nov 2010 » How to build MySQL releases 29 Apr 2010 » 'apt-get' and 5,000 packages for Solaris10/x86 16 Sep 2009 » ZFS and NFS vs OSX 12 Sep 2009 » pkgsrc on Solaris 09 Dec 2008 » Jumpstart from OSX 31 Dec 2007 » Set up local caching DNS server on OSX 10.4