Difference between revisions of "DEB package notes - dpkg, apt, aptitude, and friends"

From Noah.org
Jump to navigationJump to search
(12 intermediate revisions by the same user not shown)
Line 4: Line 4:
  
 
= Notes on APT, apt-get, aptittude, dpkg, and friends =
 
= Notes on APT, apt-get, aptittude, dpkg, and friends =
 +
 +
== Apt-get or aptitude gets stuck trying to use IPv6 addresses ==
 +
 +
This seems stupid to me, but I have a thing against IPv6... At any rate, if apt-get seems to get stuck trying to connect to a repository and it is showing an IPv6 address for the address of the remote repository then you can add the following option to your command to make it stop doing this. This will force IPv4:
 +
<pre>
 +
apt-get -o Acquire::ForceIPv4=true update
 +
aptitude -o Acquire::ForceIPv4=true
 +
</pre>
 +
 +
== Apt installs stuck in queue ==
 +
 +
Having packages left in the queue can cause problems for the next admin to use the system. They might not notice the actions already in the queue and may inadvertently trigger them. This can have devastating consequences to the system. See [[Aptitude_safety_precautions]] for more information.
 +
 +
The first command will show a list of any packages in the queue waiting to be installed. The second command will clear everything from the queue.
 +
<pre>
 +
aptitude -ysfD install
 +
aptitude keep-all
 +
</pre>
 +
 +
== Get a list of all packages last installed ==
 +
 +
<pre>
 +
awk '!/^Commandline:|^Start-Date:|^End-Date:|^Upgrade:|^Error:/ { gsub( /\([^()]*\)/ ,"" ); \
 +
gsub(/ ,/," ");sub(/^Install:/,""); print}' /var/log/apt/history.log
 +
</pre>
 +
Need to also check the gziped rotated historical logs (history.1.log.gz,
 +
 +
== open and extract files from a DEB package without using Debian tools==
 +
 +
You can use standard UNIX tools to work with a DEB file. Use the '''ar''' command to extract the data files from the DEB. These are stored in a tar.gz file. For this example, assume there is a file stored in /usr/bin/foo in the data file.
 +
<pre>
 +
ar p foo.deb data.tar.xz > data.tar.xz
 +
tar JxfO data.tar.xz ./usr/bin/foo >foo
 +
</pre>
  
 
== open and extract files from a DEB package ==
 
== open and extract files from a DEB package ==
Line 33: Line 67:
 
== download a specific DEB file from a repository ==
 
== download a specific DEB file from a repository ==
  
This will force a download of the given PACKAGE_NAME. The file will be stored in '''/var/cache/apt/archives''' and the exact name might be different than the package name. It will at least have a version number appended to it.
+
This will download the given PACKAGE_NAME. The file will be stored in '''/var/cache/apt/archives'''. Note that the exact name will be different than the package name. It will at least have a version number appended to it. This action may also download dependency DEB packages.
 
<pre>
 
<pre>
 
apt-get --download-only --force-yes -yy install --reinstall [PACKAGE_NAME]
 
apt-get --download-only --force-yes -yy install --reinstall [PACKAGE_NAME]
Line 62: Line 96:
 
<pre>
 
<pre>
 
--print-uris
 
--print-uris
 +
</pre>
 +
 +
= Caching apt requests vs. maintaining a local mirror =
 +
 +
Maintaining a local mirror takes more effort, storage, bandwidth, and maintenance. The main advantage of running a local mirror is that it isolates you from upstream updates to a repository. This is particularly important when working with Debian Sid. You can update the mirror on a staging server, point some test hosts at the stage repository, do a full update and upgrade on the test hosts, test and confirm that the hosts work as required; finally, push stage repository to the primary local mirror. A secondary advantage of a local mirror is much faster upgrade and updates of hosts.
 +
 +
Setting up a proxy cache is easy and quick. Hosts do need to be reconfigured to use the proxy, but this a one time setup. The main advantage of running a proxy cache is that subsequent upgrades and updates by other hosts using the cache will be much faster. The downside is that a proxy cache does not provide isolation from upstream changes.
 +
 +
This script will install the '''apt-cacher-ng''' package and then configure each host in the '''HOSTS''' list to use the '''MIRROR_HOST''' as a proxy.
 +
<pre>
 +
#!/bin/sh
 +
export MIRROR_HOST=10.188.121.121
 +
export HOSTS="solute-1 solute-2 solute-3 solute-4 solute-5"
 +
apt-get install -y apt-cacher-ng
 +
# lynx http://localhost:3142/
 +
for HOST in ${HOSTS}; do
 +
        ssh root@${HOST} 'echo "Acquire::http{Proxy \"http://'${MIRROR_HOST}':3142\";};" > /etc/apt/apt.conf.d/01proxy'
 +
done
 +
</pre>
 +
 +
You can also use the proxy cache selectively by providing configuration through environment variables rather than updating /etc/apt/apt.conf.
 +
<pre>
 +
apt-get install apt-cacher-ng
 +
/etc/init.d/apt-cacher-ng start
 +
export http_proxy=http://localhost:3142/
 +
</pre>
 +
 +
= Running an Apt repository =
 +
 +
Architecture
 +
 +
#.Ubuntu Mirror of 14.04 LTS (trusty)
 +
#.Local homemade packages
 +
#.Staging repository on all packages, including Ubuntu Mirror
 +
#.Production repository
 +
#. Components:  MirrorTrusty/main MirrorTrust/universe Staging/main Staging/universe Staging/local Production/main Production/universe Production/local
 +
 +
== Install reprepro ==
 +
 +
Install the '''reprepro''' package.
 +
<pre>
 +
aptitude install reprepro
 +
</pre>
 +
 +
Create a repository for it to manage.
 +
<pre>
 +
mkdir -p /var/repo/conf
 +
mkdir /var/repo/gpg
 +
</pre>
 +
 +
Create the following file, '''/var/repo/conf/distributions'''. If you do not want to use key signing with your repository then remove the last option line '''SignWith:"
 +
<pre>
 +
Origin: custom
 +
Label: Local Repository
 +
Description: Local repository for 14.04 LTS (trusty) packages.
 +
Codename: trusty
 +
Components: main
 +
Architectures: i386 amd64
 +
SignWith: FFFFFFFF
 +
</pre>
 +
Create the following file, '''/var/repo/conf/options'''.
 +
<pre>
 +
outdir /var/repo
 +
gnuphhome +b/gpg
 +
ask-passphrase
 +
</pre>
 +
 +
Create a key options file for batch key creation, '''key.options''':
 +
<pre>
 +
Key-Type: default
 +
Key-Length: 4096
 +
Subkey-Length: 4096
 +
Expire-Date: 0
 +
Name-Real: Repo Robot
 +
Name-Comment: repoman
 +
Name-Email: admin@example.co,
 +
%pubring example.pub
 +
%secring example.sec
 +
%commit
 +
</pre>
 +
 +
Generate the GPG keys from the options batch file, or see next example to do it interactively.
 +
<pre>
 +
GNUPGHOME=/var/repo/gpg gpg2 --batch --gen-key key.options
 +
</pre>
 +
 +
Generate GPG keys interactively. Press enter when asked for a password to disable encrypted keys.
 +
<pre>
 +
GNUPGHOME=/var/repo/gpg gpg2 --gen-key
 +
</pre>
 +
 +
Set repository ownership.
 +
<pre>
 +
chown -R www-data:www-data /var/repo
 +
</pre>
 +
 +
Configure Apache2 with something like the following in '''/etc/apache2/sites-enabled/000-default.conf''':
 +
<pre>
 +
ServerName localhost
 +
<VirtualHost *:80>
 +
        ServerAdmin root@example.com
 +
        DocumentRoot /var/repo
 +
        <Directory /var/repo/>
 +
            Options Indexes FollowSymLinks
 +
            AllowOverride None
 +
            Require all granted
 +
        </Directory>
 +
        # Possible values: debug, info, notice, warn, error, crit, alert, emerg
 +
        LogLevel warn
 +
        CustomLog ${APACHE_LOG_DIR}/access.log combined
 +
        ErrorLog ${APACHE_LOG_DIR}/error.log
 +
#        # Redirect http requests to https.
 +
#        RewriteEngine On
 +
#        RewriteCond %{HTTP:X-Forwarded-Proto} !https
 +
#        RewriteRule !/server.html https://%{SERVER_NAME}%{REQUEST_URI} [L,R]
 +
</VirtualHost>
 +
</pre>
 +
 +
Download a few example debs from [https://github.com/Silvenga/examples Silvenga Examples].
 +
<pre>
 +
wget https://github.com/Silvenga/examples/raw/master/example-helloworld_1.0.0.0_amd64.deb
 +
wget https://github.com/Silvenga/examples/raw/master/example-helloworld_1.0.0.0_i386.deb
 +
</pre>
 +
 +
Use '''reprepro''' to add some example deb packages to our repository. This copies the debs to their proper locations under '''/var/repositories''' and rebuilds repository metadata files. Unfortunately, this does not update the directory structure to the ownership required by Apache, so we have to manually fix this.
 +
<pre>
 +
reprepro --basedir /var/repo includedeb trusty example-helloworld_1.0.0.0_*
 +
chown -R www-data:www-data /var/repo/
 +
</pre>
 +
 +
If you decide you don't want these files in your repository anymore use this command to remove them:
 +
<pre>
 +
reprepro --basedir /var/repo remove trusty example-helloworld
 
</pre>
 
</pre>
  

Revision as of 07:36, 25 May 2016


Notes on APT, apt-get, aptittude, dpkg, and friends

Apt-get or aptitude gets stuck trying to use IPv6 addresses

This seems stupid to me, but I have a thing against IPv6... At any rate, if apt-get seems to get stuck trying to connect to a repository and it is showing an IPv6 address for the address of the remote repository then you can add the following option to your command to make it stop doing this. This will force IPv4:

apt-get -o Acquire::ForceIPv4=true update
aptitude -o Acquire::ForceIPv4=true

Apt installs stuck in queue

Having packages left in the queue can cause problems for the next admin to use the system. They might not notice the actions already in the queue and may inadvertently trigger them. This can have devastating consequences to the system. See Aptitude_safety_precautions for more information.

The first command will show a list of any packages in the queue waiting to be installed. The second command will clear everything from the queue.

aptitude -ysfD install
aptitude keep-all

Get a list of all packages last installed

awk '!/^Commandline:|^Start-Date:|^End-Date:|^Upgrade:|^Error:/ { gsub( /\([^()]*\)/ ,"" ); \
gsub(/ ,/," ");sub(/^Install:/,""); print}' /var/log/apt/history.log

Need to also check the gziped rotated historical logs (history.1.log.gz,

open and extract files from a DEB package without using Debian tools

You can use standard UNIX tools to work with a DEB file. Use the ar command to extract the data files from the DEB. These are stored in a tar.gz file. For this example, assume there is a file stored in /usr/bin/foo in the data file.

ar p foo.deb data.tar.xz > data.tar.xz
tar JxfO data.tar.xz ./usr/bin/foo >foo

open and extract files from a DEB package

This will create the destination directory that you give as the last argument. The filesystem contents of the deb package will be created. Note that this does not extract the control scripts and all the logic actually responsible for setting up the package. See the next example for that.

dpkg-deb --extract example-package-name.deb /tmp/example-package-name

Extract only the control files.

dpkg-deb --control example-package-name.deb /tmp/example-package-name

Extract both the filesystem tree and the control files.

dpkg-deb --raw-extract example-package-name.deb /tmp/example-package-name

download source package and extract the package source directory tree

This will download a source package and unpack its source tree.

apt-get source python-pexpect

This will download a source package but leave it unpacked.

apt-get --download-only source python-pexpect

download a specific DEB file from a repository

This will download the given PACKAGE_NAME. The file will be stored in /var/cache/apt/archives. Note that the exact name will be different than the package name. It will at least have a version number appended to it. This action may also download dependency DEB packages.

apt-get --download-only --force-yes -yy install --reinstall [PACKAGE_NAME]

This script will download into the current directory:

#!/bin/sh
# deb-download

PACKAGE_NAME=$1
DESTINATION_DIR=$(readlink -f .)
PARTIAL_DIR=${DESTINATION_DIR}/partial
[ -d ./partial/ ] && PARTIAL_EXISTS=1

if [ -z "${PARTIAL_EXISTS}" ]; then
    mkdir ${DESTINATION_DIR}/partial
fi

apt-get -d --force-yes -y install --reinstall -o Dir::Cache::archives=${DESTINATION_DIR} ${PACKAGE_NAME}

if [ -z "${PARTIAL_EXISTS}" ]; then
    rmdir ${DESTINATION_DIR}/partial
fi

You can also see the URL of where the package would be downloaded from:

--print-uris

Caching apt requests vs. maintaining a local mirror

Maintaining a local mirror takes more effort, storage, bandwidth, and maintenance. The main advantage of running a local mirror is that it isolates you from upstream updates to a repository. This is particularly important when working with Debian Sid. You can update the mirror on a staging server, point some test hosts at the stage repository, do a full update and upgrade on the test hosts, test and confirm that the hosts work as required; finally, push stage repository to the primary local mirror. A secondary advantage of a local mirror is much faster upgrade and updates of hosts.

Setting up a proxy cache is easy and quick. Hosts do need to be reconfigured to use the proxy, but this a one time setup. The main advantage of running a proxy cache is that subsequent upgrades and updates by other hosts using the cache will be much faster. The downside is that a proxy cache does not provide isolation from upstream changes.

This script will install the apt-cacher-ng package and then configure each host in the HOSTS list to use the MIRROR_HOST as a proxy.

#!/bin/sh
export MIRROR_HOST=10.188.121.121
export HOSTS="solute-1 solute-2 solute-3 solute-4 solute-5"
apt-get install -y apt-cacher-ng
# lynx http://localhost:3142/
for HOST in ${HOSTS}; do 
        ssh root@${HOST} 'echo "Acquire::http{Proxy \"http://'${MIRROR_HOST}':3142\";};" > /etc/apt/apt.conf.d/01proxy'
done

You can also use the proxy cache selectively by providing configuration through environment variables rather than updating /etc/apt/apt.conf.

apt-get install apt-cacher-ng
/etc/init.d/apt-cacher-ng start
export http_proxy=http://localhost:3142/

Running an Apt repository

Architecture

  1. .Ubuntu Mirror of 14.04 LTS (trusty)
  2. .Local homemade packages
  3. .Staging repository on all packages, including Ubuntu Mirror
  4. .Production repository
  5. . Components: MirrorTrusty/main MirrorTrust/universe Staging/main Staging/universe Staging/local Production/main Production/universe Production/local

Install reprepro

Install the reprepro package.

aptitude install reprepro

Create a repository for it to manage.

mkdir -p /var/repo/conf
mkdir /var/repo/gpg

Create the following file, /var/repo/conf/distributions. If you do not want to use key signing with your repository then remove the last option line SignWith:"

Origin: custom
Label: Local Repository
Description: Local repository for 14.04 LTS (trusty) packages.
Codename: trusty
Components: main
Architectures: i386 amd64
SignWith: FFFFFFFF

Create the following file, /var/repo/conf/options.

outdir /var/repo
gnuphhome +b/gpg
ask-passphrase

Create a key options file for batch key creation, key.options:

Key-Type: default
Key-Length: 4096
Subkey-Length: 4096
Expire-Date: 0
Name-Real: Repo Robot
Name-Comment: repoman
Name-Email: admin@example.co,
%pubring example.pub
%secring example.sec
%commit

Generate the GPG keys from the options batch file, or see next example to do it interactively.

GNUPGHOME=/var/repo/gpg gpg2 --batch --gen-key key.options

Generate GPG keys interactively. Press enter when asked for a password to disable encrypted keys.

GNUPGHOME=/var/repo/gpg gpg2 --gen-key

Set repository ownership.

chown -R www-data:www-data /var/repo

Configure Apache2 with something like the following in /etc/apache2/sites-enabled/000-default.conf:

ServerName localhost
<VirtualHost *:80>
        ServerAdmin root@example.com
        DocumentRoot /var/repo
        <Directory /var/repo/>
            Options Indexes FollowSymLinks
            AllowOverride None
            Require all granted
        </Directory>
        # Possible values: debug, info, notice, warn, error, crit, alert, emerg
        LogLevel warn
        CustomLog ${APACHE_LOG_DIR}/access.log combined
        ErrorLog ${APACHE_LOG_DIR}/error.log
#        # Redirect http requests to https.
#        RewriteEngine On
#        RewriteCond %{HTTP:X-Forwarded-Proto} !https
#        RewriteRule !/server.html https://%{SERVER_NAME}%{REQUEST_URI} [L,R]
</VirtualHost>

Download a few example debs from Silvenga Examples.

wget https://github.com/Silvenga/examples/raw/master/example-helloworld_1.0.0.0_amd64.deb
wget https://github.com/Silvenga/examples/raw/master/example-helloworld_1.0.0.0_i386.deb

Use reprepro to add some example deb packages to our repository. This copies the debs to their proper locations under /var/repositories and rebuilds repository metadata files. Unfortunately, this does not update the directory structure to the ownership required by Apache, so we have to manually fix this.

reprepro --basedir /var/repo includedeb trusty example-helloworld_1.0.0.0_*
chown -R www-data:www-data /var/repo/

If you decide you don't want these files in your repository anymore use this command to remove them:

reprepro --basedir /var/repo remove trusty example-helloworld

Creating DEB packages

debian control directory

debian/
root name of a package control directory.
dh-make
Convert an existing source package to your own package for modification.
aptitude -q -y install dh-make fakeroot pbuilder cdebootstrap

How To

  1. Put all source into a directory named package-version.
  2. Modify source directory as specified in http://www.debian.org/doc/maint-guide/ch-modify.en.html
  3. When you are done cd above this directory: cd ..
  4. create package-version.tar.gz source tarball: `tar czf package-version.tar.gz package-version`.
  5. Create initial debian package environment: `cd package-version/;dh_make -e user@example.com -f ../package-version.tar.gz`. This will create a new "debian" directory.
  6. Edit these files ./debian/control ./debian/copyright ./debian/license
  7. Run `dpkg-buildpackage -rfakeroot`

clean environment and workspace

This depends on `cdebootstrap`. This will take a while as it create an entire Debian chroot directory that looks like a fresh install.

sudo pbuilder create

Errors

Package is in a very bad inconsistent state - you should reinstall it before attempting a removal.

If an install or remove gets interrupted the package database might get left in an inconsistent state. This can cause errors like the one below:

dpkg: error processing xserver-xorg-video-s3virge (--remove):
Package is in a very bad inconsistent state - you should
reinstall it before attempting a removal.


The following may resolve the issue. This forces dpkg to remove the package despite the errors. This may break stuff!

dpkg --force-remove-reinstreq --remove xserver-xorg-video-s3virge