Difference between revisions of "DEB package notes - dpkg, apt, aptitude, and friends"
(23 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
[[Category:Engineering]] | [[Category:Engineering]] | ||
+ | [[Category:deb]] | ||
+ | [[Category:Ubuntu]] | ||
= Notes on APT, apt-get, aptittude, dpkg, and friends = | = Notes on APT, apt-get, aptittude, dpkg, and friends = | ||
− | == | + | == Apt-get or aptitude gets stuck trying to use IPv6 addresses == |
+ | This seems stupid to me, but I have a thing against IPv6... At any rate, if apt-get seems to get stuck trying to connect to a repository and it is showing an IPv6 address for the address of the remote repository then you can add the following option to your command to make it stop doing this. This will force IPv4: | ||
+ | <pre> | ||
+ | apt-get -o Acquire::ForceIPv4=true update | ||
+ | aptitude -o Acquire::ForceIPv4=true | ||
+ | </pre> | ||
+ | |||
+ | == Apt packages left stuck in queue -- clear the pending package actions == | ||
+ | |||
+ | Having packages left in the queue can cause problems for the next admin to use the system. They might not notice the actions already in the queue and may inadvertently trigger them. This can have devastating consequences to the system. See [[Aptitude_safety_precautions]] for more information. | ||
+ | |||
+ | The first command will show a list of any packages in the queue waiting to be installed. The second command will clear everything from the queue. | ||
+ | <pre> | ||
+ | aptitude -ysfD install | ||
+ | aptitude keep-all | ||
+ | </pre> | ||
+ | |||
+ | == Get a list of all packages last installed == | ||
+ | |||
+ | <pre> | ||
+ | awk '!/^Commandline:|^Start-Date:|^End-Date:|^Upgrade:|^Error:/ { gsub( /\([^()]*\)/ ,"" ); \ | ||
+ | gsub(/ ,/," ");sub(/^Install:/,""); print}' /var/log/apt/history.log | ||
+ | </pre> | ||
+ | Need to also check the gziped rotated historical logs (history.1.log.gz, | ||
+ | |||
+ | == open and extract files from a DEB package without using Debian tools== | ||
+ | |||
+ | You can use standard UNIX tools to work with a DEB file. Use the '''ar''' command to extract the data files from the DEB. These are stored in a tar.gz file. For this example, assume there is a file stored in /usr/bin/foo in the data file. | ||
+ | <pre> | ||
+ | ar p foo.deb data.tar.xz > data.tar.xz | ||
+ | tar JxfO data.tar.xz ./usr/bin/foo >foo | ||
+ | </pre> | ||
+ | |||
+ | == open and extract files from a DEB package == | ||
+ | |||
+ | This will create the destination directory that you give as the last argument. The filesystem contents of the '''deb''' package will be created. Note that this '''does not''' extract the '''control''' scripts and all the logic actually responsible for setting up the package. See the next example for that. | ||
+ | <pre> | ||
+ | dpkg-deb --extract example-package-name.deb /tmp/example-package-name | ||
+ | </pre> | ||
+ | Extract only the control files. | ||
+ | <pre> | ||
+ | dpkg-deb --control example-package-name.deb /tmp/example-package-name | ||
+ | </pre> | ||
+ | Extract both the filesystem tree and the control files. | ||
+ | <pre> | ||
+ | dpkg-deb --raw-extract example-package-name.deb /tmp/example-package-name | ||
+ | </pre> | ||
+ | |||
+ | == download source package and extract the package source directory tree == | ||
− | This will | + | This will download a source package and unpack its source tree. |
+ | <pre> | ||
+ | apt-get source python-pexpect | ||
+ | </pre> | ||
+ | This will download a source package but leave it unpacked. | ||
+ | <pre> | ||
+ | apt-get --download-only source python-pexpect | ||
+ | </pre> | ||
+ | |||
+ | == download a specific DEB file from a repository == | ||
+ | |||
+ | This will download the given PACKAGE_NAME. The file will be stored in '''/var/cache/apt/archives'''. Note that the exact name will be different than the package name. It will at least have a version number appended to it. This action may also download dependency DEB packages. | ||
<pre> | <pre> | ||
apt-get --download-only --force-yes -yy install --reinstall [PACKAGE_NAME] | apt-get --download-only --force-yes -yy install --reinstall [PACKAGE_NAME] | ||
Line 35: | Line 96: | ||
<pre> | <pre> | ||
--print-uris | --print-uris | ||
+ | </pre> | ||
+ | |||
+ | = Caching apt requests vs. maintaining a local mirror = | ||
+ | |||
+ | Maintaining a local mirror takes more effort, storage, bandwidth, and maintenance. The main advantage of running a local mirror is that it isolates you from upstream updates to a repository. This is particularly important when working with Debian Sid. You can update the mirror on a staging server, point some test hosts at the stage repository, do a full update and upgrade on the test hosts, test and confirm that the hosts work as required; finally, push stage repository to the primary local mirror. A secondary advantage of a local mirror is much faster upgrade and updates of hosts. | ||
+ | |||
+ | Setting up a proxy cache is easy and quick. Hosts do need to be reconfigured to use the proxy, but this a one time setup. The main advantage of running a proxy cache is that subsequent upgrades and updates by other hosts using the cache will be much faster. The downside is that a proxy cache does not provide isolation from upstream changes. | ||
+ | |||
+ | This script will install the '''apt-cacher-ng''' package and then configure each host in the '''HOSTS''' list to use the '''MIRROR_HOST''' as a proxy. | ||
+ | <pre> | ||
+ | #!/bin/sh | ||
+ | export MIRROR_HOST=10.188.121.121 | ||
+ | export HOSTS="solute-1 solute-2 solute-3 solute-4 solute-5" | ||
+ | apt-get install -y apt-cacher-ng | ||
+ | # lynx http://localhost:3142/ | ||
+ | for HOST in ${HOSTS}; do | ||
+ | ssh root@${HOST} 'echo "Acquire::http{Proxy \"http://'${MIRROR_HOST}':3142\";};" > /etc/apt/apt.conf.d/01proxy' | ||
+ | done | ||
+ | </pre> | ||
+ | |||
+ | You can also use the proxy cache selectively by providing configuration through environment variables rather than updating /etc/apt/apt.conf. | ||
+ | <pre> | ||
+ | apt-get install apt-cacher-ng | ||
+ | /etc/init.d/apt-cacher-ng start | ||
+ | export http_proxy=http://localhost:3142/ | ||
+ | </pre> | ||
+ | |||
+ | = Running an Apt repository = | ||
+ | |||
+ | Architecture | ||
+ | |||
+ | #.Ubuntu Mirror of 14.04 LTS (trusty) | ||
+ | #.Local homemade packages | ||
+ | #.Staging repository on all packages, including Ubuntu Mirror | ||
+ | #.Production repository | ||
+ | #. Components: MirrorTrusty/main MirrorTrust/universe Staging/main Staging/universe Staging/local Production/main Production/universe Production/local | ||
+ | |||
+ | == Install reprepro == | ||
+ | |||
+ | Install the '''reprepro''' package. | ||
+ | <pre> | ||
+ | aptitude install reprepro | ||
+ | </pre> | ||
+ | |||
+ | Create a repository for it to manage. | ||
+ | <pre> | ||
+ | mkdir -p /var/repo/conf | ||
+ | mkdir /var/repo/gpg | ||
+ | </pre> | ||
+ | |||
+ | Create the following file, '''/var/repo/conf/distributions'''. If you do not want to use key signing with your repository then remove the last option line '''SignWith:" | ||
+ | <pre> | ||
+ | Origin: custom | ||
+ | Label: Local Repository | ||
+ | Description: Local repository for 14.04 LTS (trusty) packages. | ||
+ | Codename: trusty | ||
+ | Components: main | ||
+ | Architectures: i386 amd64 | ||
+ | SignWith: FFFFFFFF | ||
+ | </pre> | ||
+ | Create the following file, '''/var/repo/conf/options'''. | ||
+ | <pre> | ||
+ | outdir /var/repo | ||
+ | gnuphhome +b/gpg | ||
+ | ask-passphrase | ||
+ | </pre> | ||
+ | |||
+ | Create a key options file for batch key creation, '''key.options''': | ||
+ | <pre> | ||
+ | Key-Type: default | ||
+ | Key-Length: 4096 | ||
+ | Subkey-Length: 4096 | ||
+ | Expire-Date: 0 | ||
+ | Name-Real: Repo Robot | ||
+ | Name-Comment: repoman | ||
+ | Name-Email: admin@example.co, | ||
+ | %pubring example.pub | ||
+ | %secring example.sec | ||
+ | %commit | ||
+ | </pre> | ||
+ | |||
+ | Generate the GPG keys from the options batch file, or see next example to do it interactively. | ||
+ | <pre> | ||
+ | GNUPGHOME=/var/repo/gpg gpg2 --batch --gen-key key.options | ||
+ | </pre> | ||
+ | |||
+ | Generate GPG keys interactively. Press enter when asked for a password to disable encrypted keys. | ||
+ | <pre> | ||
+ | GNUPGHOME=/var/repo/gpg gpg2 --gen-key | ||
+ | </pre> | ||
+ | |||
+ | Set repository ownership. | ||
+ | <pre> | ||
+ | chown -R www-data:www-data /var/repo | ||
+ | </pre> | ||
+ | |||
+ | Configure Apache2 with something like the following in '''/etc/apache2/sites-enabled/000-default.conf''': | ||
+ | <pre> | ||
+ | ServerName localhost | ||
+ | <VirtualHost *:80> | ||
+ | ServerAdmin root@example.com | ||
+ | DocumentRoot /var/repo | ||
+ | <Directory /var/repo/> | ||
+ | Options Indexes FollowSymLinks | ||
+ | AllowOverride None | ||
+ | Require all granted | ||
+ | </Directory> | ||
+ | # Possible values: debug, info, notice, warn, error, crit, alert, emerg | ||
+ | LogLevel warn | ||
+ | CustomLog ${APACHE_LOG_DIR}/access.log combined | ||
+ | ErrorLog ${APACHE_LOG_DIR}/error.log | ||
+ | # # Redirect http requests to https. | ||
+ | # RewriteEngine On | ||
+ | # RewriteCond %{HTTP:X-Forwarded-Proto} !https | ||
+ | # RewriteRule !/server.html https://%{SERVER_NAME}%{REQUEST_URI} [L,R] | ||
+ | </VirtualHost> | ||
+ | </pre> | ||
+ | |||
+ | Download a few example debs from [https://github.com/Silvenga/examples Silvenga Examples]. | ||
+ | <pre> | ||
+ | wget https://github.com/Silvenga/examples/raw/master/example-helloworld_1.0.0.0_amd64.deb | ||
+ | wget https://github.com/Silvenga/examples/raw/master/example-helloworld_1.0.0.0_i386.deb | ||
+ | </pre> | ||
+ | |||
+ | Use '''reprepro''' to add some example deb packages to our repository. This copies the debs to their proper locations under '''/var/repositories''' and rebuilds repository metadata files. Unfortunately, this does not update the directory structure to the ownership required by Apache, so we have to manually fix this. | ||
+ | <pre> | ||
+ | reprepro --basedir /var/repo includedeb trusty example-helloworld_1.0.0.0_* | ||
+ | chown -R www-data:www-data /var/repo/ | ||
+ | </pre> | ||
+ | |||
+ | If you decide you don't want these files in your repository anymore use this command to remove them: | ||
+ | <pre> | ||
+ | reprepro --basedir /var/repo remove trusty example-helloworld | ||
</pre> | </pre> | ||
Line 65: | Line 259: | ||
<pre> | <pre> | ||
sudo pbuilder create | sudo pbuilder create | ||
+ | </pre> | ||
+ | |||
+ | = Errors = | ||
+ | |||
+ | == Package is in a very bad inconsistent state - you should reinstall it before attempting a removal.== | ||
+ | |||
+ | If an install or remove gets interrupted the package database might get left in an inconsistent state. This can cause errors like the one below: | ||
+ | |||
+ | <pre> | ||
+ | dpkg: error processing xserver-xorg-video-s3virge (--remove): | ||
+ | Package is in a very bad inconsistent state - you should | ||
+ | reinstall it before attempting a removal. | ||
+ | </pre> | ||
+ | |||
+ | |||
+ | The following may resolve the issue. This forces dpkg to remove the package despite the errors. This may break stuff! | ||
+ | |||
+ | <pre> | ||
+ | dpkg --force-remove-reinstreq --remove xserver-xorg-video-s3virge | ||
</pre> | </pre> |
Latest revision as of 00:33, 10 January 2019
Contents
- 1 Notes on APT, apt-get, aptittude, dpkg, and friends
- 1.1 Apt-get or aptitude gets stuck trying to use IPv6 addresses
- 1.2 Apt packages left stuck in queue -- clear the pending package actions
- 1.3 Get a list of all packages last installed
- 1.4 open and extract files from a DEB package without using Debian tools
- 1.5 open and extract files from a DEB package
- 1.6 download source package and extract the package source directory tree
- 1.7 download a specific DEB file from a repository
- 2 Caching apt requests vs. maintaining a local mirror
- 3 Running an Apt repository
- 4 Creating DEB packages
- 5 Errors
Notes on APT, apt-get, aptittude, dpkg, and friends
Apt-get or aptitude gets stuck trying to use IPv6 addresses
This seems stupid to me, but I have a thing against IPv6... At any rate, if apt-get seems to get stuck trying to connect to a repository and it is showing an IPv6 address for the address of the remote repository then you can add the following option to your command to make it stop doing this. This will force IPv4:
apt-get -o Acquire::ForceIPv4=true update aptitude -o Acquire::ForceIPv4=true
Apt packages left stuck in queue -- clear the pending package actions
Having packages left in the queue can cause problems for the next admin to use the system. They might not notice the actions already in the queue and may inadvertently trigger them. This can have devastating consequences to the system. See Aptitude_safety_precautions for more information.
The first command will show a list of any packages in the queue waiting to be installed. The second command will clear everything from the queue.
aptitude -ysfD install aptitude keep-all
Get a list of all packages last installed
awk '!/^Commandline:|^Start-Date:|^End-Date:|^Upgrade:|^Error:/ { gsub( /\([^()]*\)/ ,"" ); \ gsub(/ ,/," ");sub(/^Install:/,""); print}' /var/log/apt/history.log
Need to also check the gziped rotated historical logs (history.1.log.gz,
open and extract files from a DEB package without using Debian tools
You can use standard UNIX tools to work with a DEB file. Use the ar command to extract the data files from the DEB. These are stored in a tar.gz file. For this example, assume there is a file stored in /usr/bin/foo in the data file.
ar p foo.deb data.tar.xz > data.tar.xz tar JxfO data.tar.xz ./usr/bin/foo >foo
open and extract files from a DEB package
This will create the destination directory that you give as the last argument. The filesystem contents of the deb package will be created. Note that this does not extract the control scripts and all the logic actually responsible for setting up the package. See the next example for that.
dpkg-deb --extract example-package-name.deb /tmp/example-package-name
Extract only the control files.
dpkg-deb --control example-package-name.deb /tmp/example-package-name
Extract both the filesystem tree and the control files.
dpkg-deb --raw-extract example-package-name.deb /tmp/example-package-name
download source package and extract the package source directory tree
This will download a source package and unpack its source tree.
apt-get source python-pexpect
This will download a source package but leave it unpacked.
apt-get --download-only source python-pexpect
download a specific DEB file from a repository
This will download the given PACKAGE_NAME. The file will be stored in /var/cache/apt/archives. Note that the exact name will be different than the package name. It will at least have a version number appended to it. This action may also download dependency DEB packages.
apt-get --download-only --force-yes -yy install --reinstall [PACKAGE_NAME]
This script will download into the current directory:
#!/bin/sh # deb-download PACKAGE_NAME=$1 DESTINATION_DIR=$(readlink -f .) PARTIAL_DIR=${DESTINATION_DIR}/partial [ -d ./partial/ ] && PARTIAL_EXISTS=1 if [ -z "${PARTIAL_EXISTS}" ]; then mkdir ${DESTINATION_DIR}/partial fi apt-get -d --force-yes -y install --reinstall -o Dir::Cache::archives=${DESTINATION_DIR} ${PACKAGE_NAME} if [ -z "${PARTIAL_EXISTS}" ]; then rmdir ${DESTINATION_DIR}/partial fi
You can also see the URL of where the package would be downloaded from:
--print-uris
Caching apt requests vs. maintaining a local mirror
Maintaining a local mirror takes more effort, storage, bandwidth, and maintenance. The main advantage of running a local mirror is that it isolates you from upstream updates to a repository. This is particularly important when working with Debian Sid. You can update the mirror on a staging server, point some test hosts at the stage repository, do a full update and upgrade on the test hosts, test and confirm that the hosts work as required; finally, push stage repository to the primary local mirror. A secondary advantage of a local mirror is much faster upgrade and updates of hosts.
Setting up a proxy cache is easy and quick. Hosts do need to be reconfigured to use the proxy, but this a one time setup. The main advantage of running a proxy cache is that subsequent upgrades and updates by other hosts using the cache will be much faster. The downside is that a proxy cache does not provide isolation from upstream changes.
This script will install the apt-cacher-ng package and then configure each host in the HOSTS list to use the MIRROR_HOST as a proxy.
#!/bin/sh export MIRROR_HOST=10.188.121.121 export HOSTS="solute-1 solute-2 solute-3 solute-4 solute-5" apt-get install -y apt-cacher-ng # lynx http://localhost:3142/ for HOST in ${HOSTS}; do ssh root@${HOST} 'echo "Acquire::http{Proxy \"http://'${MIRROR_HOST}':3142\";};" > /etc/apt/apt.conf.d/01proxy' done
You can also use the proxy cache selectively by providing configuration through environment variables rather than updating /etc/apt/apt.conf.
apt-get install apt-cacher-ng /etc/init.d/apt-cacher-ng start export http_proxy=http://localhost:3142/
Running an Apt repository
Architecture
- .Ubuntu Mirror of 14.04 LTS (trusty)
- .Local homemade packages
- .Staging repository on all packages, including Ubuntu Mirror
- .Production repository
- . Components: MirrorTrusty/main MirrorTrust/universe Staging/main Staging/universe Staging/local Production/main Production/universe Production/local
Install reprepro
Install the reprepro package.
aptitude install reprepro
Create a repository for it to manage.
mkdir -p /var/repo/conf mkdir /var/repo/gpg
Create the following file, /var/repo/conf/distributions. If you do not want to use key signing with your repository then remove the last option line SignWith:"
Origin: custom Label: Local Repository Description: Local repository for 14.04 LTS (trusty) packages. Codename: trusty Components: main Architectures: i386 amd64 SignWith: FFFFFFFF
Create the following file, /var/repo/conf/options.
outdir /var/repo gnuphhome +b/gpg ask-passphrase
Create a key options file for batch key creation, key.options:
Key-Type: default Key-Length: 4096 Subkey-Length: 4096 Expire-Date: 0 Name-Real: Repo Robot Name-Comment: repoman Name-Email: admin@example.co, %pubring example.pub %secring example.sec %commit
Generate the GPG keys from the options batch file, or see next example to do it interactively.
GNUPGHOME=/var/repo/gpg gpg2 --batch --gen-key key.options
Generate GPG keys interactively. Press enter when asked for a password to disable encrypted keys.
GNUPGHOME=/var/repo/gpg gpg2 --gen-key
Set repository ownership.
chown -R www-data:www-data /var/repo
Configure Apache2 with something like the following in /etc/apache2/sites-enabled/000-default.conf:
ServerName localhost <VirtualHost *:80> ServerAdmin root@example.com DocumentRoot /var/repo <Directory /var/repo/> Options Indexes FollowSymLinks AllowOverride None Require all granted </Directory> # Possible values: debug, info, notice, warn, error, crit, alert, emerg LogLevel warn CustomLog ${APACHE_LOG_DIR}/access.log combined ErrorLog ${APACHE_LOG_DIR}/error.log # # Redirect http requests to https. # RewriteEngine On # RewriteCond %{HTTP:X-Forwarded-Proto} !https # RewriteRule !/server.html https://%{SERVER_NAME}%{REQUEST_URI} [L,R] </VirtualHost>
Download a few example debs from Silvenga Examples.
wget https://github.com/Silvenga/examples/raw/master/example-helloworld_1.0.0.0_amd64.deb wget https://github.com/Silvenga/examples/raw/master/example-helloworld_1.0.0.0_i386.deb
Use reprepro to add some example deb packages to our repository. This copies the debs to their proper locations under /var/repositories and rebuilds repository metadata files. Unfortunately, this does not update the directory structure to the ownership required by Apache, so we have to manually fix this.
reprepro --basedir /var/repo includedeb trusty example-helloworld_1.0.0.0_* chown -R www-data:www-data /var/repo/
If you decide you don't want these files in your repository anymore use this command to remove them:
reprepro --basedir /var/repo remove trusty example-helloworld
Creating DEB packages
debian control directory
- debian/
- root name of a package control directory.
- dh-make
- Convert an existing source package to your own package for modification.
aptitude -q -y install dh-make fakeroot pbuilder cdebootstrap
How To
- Put all source into a directory named package-version.
- Modify source directory as specified in http://www.debian.org/doc/maint-guide/ch-modify.en.html
- When you are done cd above this directory: cd ..
- create package-version.tar.gz source tarball: `tar czf package-version.tar.gz package-version`.
- Create initial debian package environment: `cd package-version/;dh_make -e user@example.com -f ../package-version.tar.gz`. This will create a new "debian" directory.
- Edit these files ./debian/control ./debian/copyright ./debian/license
- Run `dpkg-buildpackage -rfakeroot`
clean environment and workspace
This depends on `cdebootstrap`. This will take a while as it create an entire Debian chroot directory that looks like a fresh install.
sudo pbuilder create
Errors
Package is in a very bad inconsistent state - you should reinstall it before attempting a removal.
If an install or remove gets interrupted the package database might get left in an inconsistent state. This can cause errors like the one below:
dpkg: error processing xserver-xorg-video-s3virge (--remove): Package is in a very bad inconsistent state - you should reinstall it before attempting a removal.
The following may resolve the issue. This forces dpkg to remove the package despite the errors. This may break stuff!
dpkg --force-remove-reinstreq --remove xserver-xorg-video-s3virge