perkin.org.uk - pkgsrc on Solaris

pkgsrc on Solaris

Perkin.org.uk Spined HTML

pkgsrc on Solaris Jonathan Perkin well-nigh me · rss · twitter · github pkgsrc on Solaris Sep 12, 2009 tags: pkgsrc, solaris, zones For many years, towers and installing third-party software on Solaris has been a huge pain. For people who do not use pkgsrc, that is. Originating from the NetBSD project, pkgsrc is a source-based cross-platform package manager. If you’ve used FreeBSD ports, then it is very similar as it derives from the same codebase, so the vital premise is that you: $ cd /usr/pkgsrc/www/apache22 $ make package and pkgsrc will download the source for Apache 2.2, compile it and all dependancies, install it on the local system and create a package which can be installed on other similar systems. However, we’ve taken ports remoter and unromantic the NetBSD philosophy of portability, meaning that it not only works on NetBSD, but wideness all *BSD as well as Linux, OSX, HP/UX, AIX, IRIX, QNX, Windows (via Interix), and of undertow Solaris. So while apt-get might be awesome, it only really works on Linux. FreeBSD might have way increasingly ports than us, but only runs on FreeBSD and OSX. pkgsrc provides a resulting interface wideness all the platforms listed above, and in some cases provides a superior package manager than the system provides. Here’s how I use pkgsrc on Solaris, in this specific specimen Solaris 10/x86. Paths are specific to my setup, you can of undertow transpiration them. Create a chroot/zone environment I use the zones full-length of Solaris 10 to ensure that all packages are built in a sandbox. This has a number of benefits: The running system is unaffected by the builds, in that they are not writing to the same file system. This is good when you have misbehaving packages. You can separate the build and install phases, so that you can verify all the packages have been built and are correct surpassing starting any install/upgrade procedure It’s easier to reservation package mistakes, e.g. unpackaged files. It avoids pollution from the host environment which may produce bad packages For creating a zone, I wrote the pursuit create-zone script: #!/bin/sh DOMAIN="adsl.perkin.org.uk" MASQDOMAIN="perkin.org.uk" PHYSIF="e1000g0" RPOOL="gromit" if [ $# -eq 1 ]; then name=$1; shift ipaddr=`getent hosts ${name} | awk '{print $1}'` if [ -z "${ipaddr}" ]; then reverberate "ERROR: Could not determine IP write of $name" reverberate "Either add to hosts database or provide on writ line" reverberate reverberate "usage: $0 <name> [ <ipaddr> ]" exit 2 fi elif [ $# -eq 2 ]; then name=$1; shift ipaddr=$1; shift else reverberate "usage: $0 <name> [ <ipaddr> ]" exit 2 fi zfs list ${RPOOL}/zones/$name >/dev/null 2>&1 if [ $? -eq 0 ]; then reverberate "${RPOOL}/zones/$name once exists, not continuing" exit 1 fi # Inherited directories are read-only cat >/tmp/zonecfg.$$ << EOF create set zonepath=/zones/$name set autoboot=true add net set address=$ipaddr/24 set physical=${PHYSIF} end verify commit exit EOF zonecfg -z $name -f /tmp/zonecfg.$$ rm /tmp/zonecfg.$$ zoneadm -z $name install # # Automatically configure new zone with most of the settings from the current # global, grabbing the encrypted root password directly. Note that the # service_profile keyword doesn't seem to be used here, so we configure the # limited_net profile manually and optionally install a custom site.xml which # is parsed and zingy during first boot. # cat << EOF >/zones/$name/root/etc/sysidcfg network_interface=PRIMARY { hostname=$name.${DOMAIN} } name_service=DNS { domain_name=${DOMAIN} name_server=`awk '/^nameserver/ { print $NF; exit }' /etc/resolv.conf` } nfs4_domain=dynamic root_password=`awk -F: '/^root/ {print $2}' /etc/shadow` security_policy=NONE service_profile=limited_net system_locale=C terminal=xterm timezone=Europe/London EOF # # Use the limited_net profile and install custom site.xml if provided. # rm /zones/${name}/root/var/svc/profile/generic.xml ln -s generic_limited_net.xml /zones/${name}/root/var/svc/profile/generic.xml if [ -f /install/zones/${name}.xml ]; then cp /install/zones/${name}.xml /zones/${name}/root/var/svc/profile/site.xml fi # # Disable nscd host enshroud # ex /zones/$name/root/etc/nscd.conf >/dev/null 2>&1 <<EOF /enable-cache/s/^#// wq EOF # # Configure sendmail to masquerade and route via smarthost. # ex /zones/$name/root/etc/mail/cf/cf/submit.mc >/dev/null 2>&1 <<EOF /^dnl/ a FEATURE(\`masquerade_envelope')dnl MASQUERADE_AS(\`${MASQDOMAIN}')dnl dnl . /^FEATURE.*msp/s/.127.0.0.1./mail.perkin.org.uk/ wq! EOF ( cd /zones/$name/root/etc/mail/cf/cf /usr/ccs/bin/make submit.cf >/dev/null 2>&1 ) cp /zones/$name/root/etc/mail/cf/cf/submit.cf \ /zones/$name/root/etc/mail/submit.cf # # Use SHA512 crypted passwords, and transpiration root's home directory. # ex /zones/$name/root/etc/security/policy.conf >/dev/null 2>&1 <<EOF /^CRYPT_DEFAULT/s/__unix__/6/ wq EOF mkdir -m 0700 /zones/$name/root/root ex /zones/$name/root/etc/passwd >/dev/null 2>&1 <<EOF /^root/s,:/:,:/root:, /^root/s,:/sbin/sh,:/bin/bash, wq EOF cp /root/.bash_profile /zones/$name/root/root/.bash_profile # # Allow root login over SSH (rest of network takes superintendency of external wangle # so this isn't a problem) and enforce SSH key authentication. # ex /zones/$name/root/etc/ssh/sshd_config >/dev/null 2>&1 <<EOF /^PermitRootLogin/s/no$/yes/ /^PasswordAuthentication/s/yes$/no/ wq EOF mkdir -m 0700 /zones/$name/root/root/.ssh cp /home/jperkin/.ssh/id_rsa.pub /zones/$name/root/root/.ssh/authorized_keys # # Automount shared directories from global zone using uncontrived mounts # ex /zones/$name/root/etc/auto_master >/dev/null 2>&1 <<EOF a /- auto_direct . wq EOF ex /zones/$name/root/etc/auto_direct >/dev/null 2>&1 <<EOF a /content gromit-lan:/content /install gromit-lan:/install . wq EOF # # All done! Fire it up... # zoneadm -z $name marching And the respective delete-zone script is: #!/bin/sh RPOOL="gromit" if [ $# -ne 1 ]; then reverberate "usage: $0 <zone>" exit 2 fi zoneadm -z $1 halt zonecfg -z $1 delete -F zfs destroy ${RPOOL}/zones/$1 If you want to use them then there are some variables to set at the top, and you may want to scan through them for spare shit to change, for example create-zone copies my ssh public key which will most likely be wrong for your setup :-) One spare piece of configuration for create-zone is an optional SMF xml file. I use this file to disable inetd inside the zone for spare security, like so: <?xml version='1.0'?> <!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'> <service_bundle type='profile' name='extract'> <service name='network/inetd' type='service' version='0'> <instance name='default' enabled='false'/> </service> </service_bundle> The file should be named <yourzonename>.xml. Mine is named vm-generic.xml and I then create symlinks to it for each VM I want with that default configuration. Fetch pkgsrc pkgsrc is ripened very rapidly. Tracking nearly 9,000 pieces of third-party software ways there are unchangingly many updates. Thankfully, we provide quarterly branches for people who want increasingly stability, and I recommend using the latest quarterly release. At time of writing, this is known as pkgsrc-2009Q2. Within the next month or so we will release pkgsrc-2009Q3, and you can icon out the names of future releases yourself. The easiest way to get pkgsrc is using cvs. I alimony stuff like this under /content as opposed to the default of /usr, you can use whatever you wish but will need to transpiration all my example scripts to match where you put it. $ cd /content $ BRANCH="pkgsrc-2009Q2" $ cvs -d anoncvs@anoncvs.netbsd.org:/cvsroot co -r${BRANCH} -d${BRANCH} -P pkgsrc Alternatively, you can fetch either bzip2 or gzip archives of the current branch. I recommend the cvs method as, with the workshop stuff updated for security fixes and other important changes, you can hands track it using $ cd /content/pkgsrc-2009Q2 $ cvs update pkgsrc configuration pkgsrc is configured using a mk.conf file, this is mine: # # Local settings # OS_HOST!= uname -n # # Per-host settings # # vm0 builds local packages # .if !empty(OS_HOST:Mvm0.*) || !empty(OS_HOST:Mgromit.*) PKGDIRSUFFIX= PKG_DEFAULT_OPTIONS= inet6 perl sasl ssl ncursesw PKG_OPTIONS.mutt+= mutt-hcache mutt-smtp PKG_UID.cyrus= 2000 #PKG_GID.cyrus= mail PKG_SHELL.cyrus= /usr/bin/false PKG_UID.www= 2001 PKG_GID.www= 2001 PYTHON_VERSION_DEFAULT= 24 X11_TYPE= modular # # vm1 modular X11 zillion builds # .elif !empty(OS_HOST:Mvm1.*) X11_TYPE= modular PKGDIRSUFFIX= -${X11_TYPE} # # vm2 native X11 zillion builds # .elif !empty(OS_HOST:Mvm2.*) X11_TYPE= native PKGDIRSUFFIX= -${X11_TYPE} # # vm3 pkgsrc dev builds # .elif !empty(OS_HOST:Mvm3.*) X11_TYPE= modular PKGDIRSUFFIX= -${X11_TYPE} .endif .if exists(../../CVS/Tag) PKG_BRANCH!= ${TOOLS_PLATFORM.awk} -F- '{print $$2}' < ../../CVS/Tag .else PKG_BRANCH= HEAD .endif ALLOW_VULNERABLE_PACKAGES= YES BULKFILESDIR= ${WRKOBJDIR} DISTDIR= /install/pkgsrc/distfiles/ FAILOVER_FETCH= YES INSTALL_UNSTRIPPED= YES LINTPKGSRC_DB= ${_PKGSRCDIR}/.lintpkgsrc.db MASTER_SITE_OVERRIDE= ftp://ftp.netbsd.org/pub/NetBSD/packages/distfiles/ PACKAGES= /install/pkgsrc/packages/${PKG_BRANCH}${PKGDIRSUFFIX} PKGCHK_CONF= /install/pkgsrc/misc/pkgchk.conf SKIP_LICENSE_CHECK= YES WRKOBJDIR= /tmp/pkgsrc # # Parse pkgchk.conf and supply list of packages for the zillion build framework. # .if defined(SPECIFIC_PKGS) PKGLIST!= ${TOOLS_PLATFORM.awk} '$$1 !~ /^(\#|$$)/ {print $$1}' ${PKGCHK_CONF} . for _pkg_ in ${PKGLIST} HOST_SPECIFIC_PKGS+= ${_pkg_} . endfor .endif This is the primary configuration file for pkgsrc. Again, you may need to tailor this to your environment, and may find it useful to read the pkgsrc guide to understand what it all means. As I do a lot of pkgsrc minutiae I have a number of virtual machines up and running doing various shit and pieces. Obviously I don’t want to reprinting that mk.conf around, so I moreover have a small fragment file which is appended to each virtual machine’s mk.conf (using the --mk-fragment treatise to bootstrap) and includes the global copy: .include "/install/pkgsrc/misc/mk.conf" The zillion build setup in pkgsrc requires its own configuration, and for this you will need to edit a file inside pkgsrc. There is an example file provided, so what I usually do is symlink this to the real reprinting then I can hands alimony it up-to-date via cvs. $ cd /content/pkgsrc-2009Q2/mk/bulk $ ln -s build.conf-example build.conf $ vi build.conf Again, you can find my personal build.conf here. Finally, there is a configuration file for pkg_chk which is a package inside pkgsrc which makes managing upgrades easier (ideally it should be a part of the main pkgsrc tools but that’s for flipside day). pkgchk.conf is a list of package directories, relative to the pkgsrc top level, which are to be built and installed for this setup. If you have a large installation then pkg_chk has uneaten features to make it possible to share pkgchk.conf wideness a number of machines and configure packages on a per-host, per-OS etc basis. Thus, a sample pkgchk.conf: chat/irssi chat/irssi-icb converters/xlhtml databases/lbdb databases/rrdtool devel/autoconf devel/automake devel/bzr devel/bzrtools devel/diffutils devel/libtool-base devel/scmcvs editors/vim mail/mutt-devel misc/screen net/p5-Net-SNMP net/p5-Net-SNMP-Interfaces net/rsync net/samba net/tnftp pkgtools/digest pkgtools/pkg_chk pkgtools/rc.subr print/mp security/cy2-login security/cy2-plain security/sudo sysutils/coreutils textproc/antiword textproc/grep textproc/urlview www/apache22 www/w3m It is highly likely you will want to transpiration the pkgchk.conf file from what I use :-) Build scripts Once everything is set up, I have two scripts to build then update my packages, intuitively tabbed build-packages: #!/bin/sh PATH="/opt/pkg/sbin:/opt/pkg/bin:/sbin:/usr/sbin:/usr/bin:/usr/ccs/bin" export PATH BRANCH="2009Q2" export BRANCH PDIR=/install/pkgsrc if [ ! -f ${PDIR}/bootstrap-${BRANCH}.tar ]; then cd /content/pkgsrc-${BRANCH}/bootstrap ./bootstrap --workdir=/tmp/pkgsrc --prefix=/opt/pkg \ --varbase=/var/opt/pkg --sysconfdir=/etc/opt/pkg \ --mk-fragment=${PDIR}/misc/mk-include.conf \ --binary-kit=${PDIR}/bootstrap-${BRANCH}.tar rm -rf /tmp/pkgsrc fi if [ ! -f /opt/pkg/bin/bmake ]; then (cd /; tar -xf ${PDIR}/bootstrap-${BRANCH}.tar) fi cd /content/pkgsrc-${BRANCH} /bin/ksh mk/bulk/build -s 2>&1 | tee -a ${PDIR}/logs/build-${BRANCH}.log /bin/ksh mk/bulk/upload -n 2>&1 | tee -a ${PDIR}/logs/upload-${BRANCH}.log and update-packages: #!/bin/sh PATH="/opt/pkg/sbin:/opt/pkg/bin:/sbin:/usr/sbin:/usr/bin:/usr/ccs/bin" export PATH BRANCH=2009Q2 export BRANCH PKGSRCDIR=/content/pkgsrc-${BRANCH} export PKGSRCDIR if [ ! -f /opt/pkg/bin/bmake ]; then (cd /; tar -xf /install/pkgsrc/bootstrap-${BRANCH}.tar) fi # Run this anyway in specimen there are updates to pkg_chk env PKG_PATH=/install/pkgsrc/packages/${BRANCH}/All pkg_add -u pkg_chk # Dry run first pkg_chk -aurbn printf "Do you wish to install? [y/N]: " read ans specimen "${ans}" in [Yy]) pkg_chk -aurb 2>&1 | tee -a /install/pkgsrc/logs/pkg_chk-${BRANCH}.log ;; esac These are pretty simple as all the nonflexible work has all been done. build-packages is ran inside the zone, then update-packages on the main host. These scripts hardcode the name of the workshop currently used, so you will need to update this when moving to newer releases. Quick recap Ok, so here is the stuff I have for my setup and where I alimony them: PathDescription /content/pkgsrc-2009Q2Checked out pkgsrc tree, "2009Q2" workshop /content/scripts/create-zoneCreates Solaris zone /content/scripts/delete-zoneUninstalls and deletes zone /content/scripts/build-packagesBulk build packages inside the zone /content/scripts/update-packagesUpdates installed packages /install/pkgsrc/misc/mk.confMain pkgsrc configuration file /install/pkgsrc/misc/mk-include.confFragment file included in each zone's mk.conf, sources the global mk.conf /install/pkgsrc/misc/pkgchk.confpkg_chk configuration file /install/zones/vm-generic.xmlShared SMF configuration file, symlinked to from e.g. "vm0.xml" And these are the paths where stuff will be created: PathDescription /install/pkgsrc/distfilesSource tarballs of packages /install/pkgsrc/packages/2009Q2Resulting binary packages /tmp/pkgsrcTemporary build zone for packages /content/vwww/www.adsl.perkin.org.uk/pkgstatBulk build results directory It’s definitely harder than it should be to get this all setup, but the good news is that once it’s washed-up there’s very little maintenance. Kicking it all off Once everything is setup: $ /content/scripts/create-zone vm0 $ ssh vm0 /content/scripts/build-packages $ /content/scripts/update-packages $ /content/scripts/delete-zone vm0 This should do the lot. Once build-packages has finished you should, if you configured your email write in build.conf, get an email with the zillion build results which looks similar to this: http://mail-index.netbsd.org/pkgsrc-bulk/2009/08/23/msg006883.html A fuller report is misogynist if you configure a web server to serve the pkgstat directory created by the zillion build, and this can help debug problems (again see the whilom URL for an example). You will need to add /opt/pkg/sbin:/opt/pkg/bin to $PATH. Configuration files are in /etc/opt/pkg, and log files and metadata are kept in /var/opt/pkg. This stuff should be obsolete While this all works well for me, it’s pretty lame for users who just want to install packages and have stuff work. I’m working on providing regular updates of binary packages, including a SVR4 package of the bootstrap kit, so that in theory all a user needs to do is $ pkgadd TNFpkgsrc.pkg $ pkg_add apache22 # Upgrade all installed packages to latest releases $ pkg_chk -aurb I’m scrutinizingly there, just needs some tidying up and regular builds. Please finger self-ruling to help out! Share this post on Twitter, HackerNews, Facebook or Google+ All Posts 16 Jul 2015 » Reducing RAM usage in pkgin 03 Mar 2015 » pkgsrc-2014Q4: LTS, signed packages, and increasingly 06 Oct 2014 »Towerspackages at scale 04 Dec 2013 » A node.js-powered 8-bit CPU - part four 03 Dec 2013 » A node.js-powered 8-bit CPU - part three 02 Dec 2013 » A node.js-powered 8-bit CPU - part two 01 Dec 2013 » A node.js-powered 8-bit CPU - part one 21 Nov 2013 » MDB support for Go 30 Jul 2013 » What's new in pkgsrc-2013Q2 24 Jul 2013 » Distributed chrooted pkgsrc zillion builds 07 Jun 2013 » pkgsrc on SmartOS - creating new packages 15 Apr 2013 » What's new in pkgsrc-2013Q1 19 Mar 2013 » Installing SVR4 packages on SmartOS 27 Feb 2013 » SmartOS is Not GNU/Linux 18 Feb 2013 » SmartOS minutiae preview dataset 17 Jan 2013 » pkgsrc on SmartOS - fixing wrenched builds 15 Jan 2013 » pkgsrc on SmartOS - zone megacosm and vital builds 10 Jan 2013 » Multi-architecture package support in SmartOS 09 Jan 2013 » Solaris portability - cfmakeraw() 08 Jan 2013 » Solaris portability - flock() 06 Jan 2013 » pkgsrc-2012Q4 illumos packages now misogynist 23 Nov 2012 » SmartOS and the global zone 24 Oct 2012 » Setting up Samba on SmartOS 10 Oct 2012 » pkgsrc-2012Q3 packages for illumos 23 Aug 2012 » Creating local SmartOS packages 10 Jul 2012 » 7,000 binary packages for OSX Lion 09 Jul 2012 » 9,000 packages for SmartOS and illumos 07 May 2012 » Goodbye Oracle, Hello Joyent! 13 Apr 2012 » SmartOS global zone tweaks 12 Apr 2012 » Automated VirtualBox SmartOS installs 30 Mar 2012 » iptables script for Debian / Ubuntu 20 Feb 2012 » New site diamond 11 Jan 2012 » Set up unrecognized FTP upload on Oracle Linux 09 Jan 2012 » Kickstart Oracle Linux in VirtualBox 09 Jan 2012 » Kickstart Oracle Linux from Ubuntu 22 Dec 2011 » Last day at MySQL 15 Dec 2011 » Installing OpenBSD with softraid 21 Sep 2011 » Create VirtualBox VM from the writ line 14 Sep 2011 » Creating chroots for fun and MySQL testing 30 Jun 2011 » Graphing memory usage during an MTR run 29 Jun 2011 » Fix input box keybindings in Firefox 24 Jun 2011 » How to lose weight 23 Jun 2011 » How to fix stdio buffering 13 Jun 2011 » Serving multiple DNS search domains in IOS DHCP 13 Jun 2011 » Fix Firefox URL double click behaviour 20 Apr 2011 » SSH via HTTP proxy in OSX 09 Nov 2010 » How to build MySQL releases 29 Apr 2010 » 'apt-get' and 5,000 packages for Solaris10/x86 16 Sep 2009 » ZFS and NFS vs OSX 12 Sep 2009 » pkgsrc on Solaris 09 Dec 2008 » Jumpstart from OSX 31 Dec 2007 » Set up local caching DNS server on OSX 10.4