t2cd-logo
+
puppy-logo
=
puppy-from-scratch

Puppy From Scratch

Page updated December 23, 2007

Introduction

The T2 project is a set of Bash scripts to compile a complete Linux distro from source packages.
Puppy Linux is the tiniest and cutest and easiest and the best little Linux distro there is.
Put the two together, and we have Puppy From Scratch!

If you want to create your own custom version of Puppy, you can do it in three different ways:
  1. Puppy Remaster-CD script
  2. Puppy Unleashed
  3. Puppy-from-scratch
You will find the Remaster-CD script in the "Setup" menu when you are running Puppy. This enables you to install the selection of packages you want, then burn it to CD/DVD. So, you have a Puppy live-CD with a different mix of packages -- though, a limitation of the current script is you can only add packages to those already on the CD.

The Puppy distro is created from a pool of binary packages, along with a build script, collectively called Puppy Unleashed. I use Unleashed to build each new official release of Puppy, as does Nathan for his Grafpup, John for his MeanPup, and so on. Basically, you can choose exactly what packages you want in Puppy, then create the live-CD iso file. There is a web page that explains what Unleashed is and how to use it:
http://www.puppyos.com/puppy-unleashed.htm

Number 3 is the lowest level. This is rebuilding Puppy totally from the original source packages. This is strictly only for "core developers". Say for example, we want to upgrade Puppy from the GTK 2.8.17 library to 2.12.1 -- by recompiling everything, we will not only recompile the GTK library packages but also all the application packages that use those libraries. Thus, the applications work properly and reliably with the new libraries. There is a lot more to this -- for example, we could compile for a CPU other than x86 (Puppy is currently compiled for a 486 CPU, so he will run on any 486, 586, 686, Pentium and so on).

I must warn you that Number 3 is for a very limited audience. If in doubt, stay away! The T2 build system does not create Puppy from source!!! It simply creates a whole heap of raw binary packages, and a lot of manual work is required to convert those to Unleashed packages and the "devx" module. Further automation of these conversions is planned, but not yet implemented.

Some history

T2 was first used for Puppy 2.10, and that build was used up until version 2.17.1. After that, Puppy jumped to version 3.00 which was based on Slackware binary packages.

I returned to T2 for Puppy version 4.00, that we have codenamed "Dingo". All versions of Puppy prior to Dingo were not quite adequate as a host for compiling T2, and to run T2 for the creation of Puppy 2.10, Vector Linux 5.1STD was used.

To use T2, you download a core source package, and later on download all the required application source packages. You can obtain this core source package from the T2 website, however I have a special version modified for creating packages suitable for use in Puppy, named t2-puppy-4.00.tar.gz.

NOTICE: I currently have a tarball, t2-dingo.tar.gz, with all the sources included. It's big though, getting on to 1GB. You will find it at ibiblio.org.

Getting started with T2 and Puppy

Recompiling everything from source is not easy. Considerable Linux experience is required. A special build environment is required, especially if we want to upgrade the gcc compiler and glibc C library, or to compile for another CPU. Also, source packages may need to have patches applied, or have some kind of pre- or post- script executed. It's incredibly complicated, and for this reason we have teamed up with the T2 project, lead by Rene Rebe. T2 was a fork from ROCK Linux, but since then has developed its own special flavour and is very actively developed and maintained with recent packages.

Learn more about T2 here: http://www.t2-project.org

When I first started to learn how to use T2, I found the online manual to be incomplete in places, but I posted questions to the mail-list and Rene is extremely helpful. For those who come after me it will be easier, as I have a preconfigured T2-build-package, ready to go to compile packages for Puppy.

Step 1: Getting everything ready

Puppy has a devx_xxx.sfs file, that turns Puppy into a compiler environment, where 'xxx' is the version number for example '400' for Puppy version 4.00. You must be using "Dingo" Puppy v4.00 or later.

You download the 'devx' file to /mnt/home (the same place as the 'pup_save' personal storage file, then reboot. It's as simple as that. If you have a full hard drive installation, there are more steps to install the 'devx' file, see the "Hard Puppy" web page.

Step 2: Install T2 core-build-package

I have a T2 core-build-pkg, taken from SVN on the 15th of October 2007. This core package has filename t2-puppy-4.00.tar.gz, and is is preconfigured for Puppy. Therefore, do not download it from the T2 site. Instead, get it from my ibiblio.org file hosting site. This page has download links: http://www.puppylinux.com/download/downpage.htm

NOTICE: I currently have a tarball, t2-dingo.tar.gz, with all the sources included. It's big though, getting on to 1GB. You will find it at ibiblio.org.

You need to download it to a Linux partition (ext2, ext3 or reiserfs) with at least 10GB free space, preferably 15-20GB if you want to compile more than the basic packages needed for Puppy.

When the T2-pkg is expanded, you will see these directories:
t2-trunk
architecture
build
config
download
misc
package
scripts
source
target


piglet001





piglet

The 'target' directory is where you will find the package-selection and special configurations for Puppy. I have created a directory named 'piglet' which is used for building Puppy. If you poke around in here, the main file of interest is 'pkgsel' which is a list of all the packages that will be compiled.

If we have specified that we want 'abiword' in Puppy, what version, where do we download it from, and so on? Look in the 'package' directory and you will find detailed specs for each package. Basically, you will find files with extensions .desc, .conf and .cache. You should get the basic idea of what these files do by viewing them in a text editor. For example, if I look at package/gnome2/abiword/abiword.desc, I see that the version is 2.4.6, and the original download URL is also given. However, T2 will first look at its own online repositories before downloading from the original URL.

The download directory is where source packages get downloaded to. If you are online, T2 will do this automatically, but beware, even if you only download the selection of packages required by Puppy, it is going to be about 1.5GB. You can also download them manually if you want, but T2 does a checksum so is the preferred way of downloading them.

The config directory is where you can create one or more build "profiles". I have already created one called 'piglet001', and you will see a directory of that name. I created this profile by running this script:
# ./scripts/Config -cfg piglet001
When I ran the Config script, I chose the 'piglet' target, plus various other customisations.

If you want to make any changes to the piglet001 profile, you can run this yourself:
# ./scripts/Config -cfg piglet001
You get a text-mode "kconfig" menu system that offers all kinds of choices, like what CPU to build for and what packages to compile with dietlibc. You can run the above just to see what the choices are, and exit without making any changes.

Step 3: Download source packages

To get going and compile the packages for Puppy, it is very simple. You have to be online, so that T2 can download the required packages. Type this:
# ./scripts/Download -cfg piglet001 -required NOT REQUIRED FOR FULL TARBALL
Then wait while about 1.5GB of packages will download. Note, '-required' means that only the packages required for the 'piglet001' profile will be downloaded (not the whole lot, which is over 2400 packages!).

A warning: I changed some of the .desc files inside the 'package' directory. For example, I changed package/www/seamonkey/seamonkey.desc to a later version. These are the changed lines:
[V] 1.1.5
[D] 1889686604 seamonkey-1.1.5.source.tar.bz2 http://ftp.mozilla.org/pub/mozilla.org/seamonkey/releases/1.0.4/
I then downloaded seamonkey-1.1.5.source.tar.bz2 directly (manually) into download/mirror/s/. Notice that number '1889686604' -- T2 uses the 'cksum' program for doing checksums, but it is done on the uncompressed file. So, to do it yourself: '# bunzip2 seamonkey-1.1.5.source.tar.bz2' then '# cksum seamonkey-1.1.5.source.tar' and use the first number.
Notice the URL -- this is the author's fallback download site, if the package is not available on the T2 repositories.

However, correcting the cksum is optional. If you manually download a later package directly into the 'download' folder, T2 will not check its cksum. All you really need to update in the '.desc' file is the '[V]' entry.

If T2 is not able to download the package, or you didn't do it manually, the Download script will hiccup at that point and output some bold red text. You might like to run this a second time just to get a list of all the failures:

# ./scripts/Download -cfg piglet001 -required

T2 does fall back to trying to obtain the packages from the original author's download sites, but they can change, meaning that you should look in the .desc files and correct any wrong URLs -- and let us know!
I have a list of the source repository URLs here:
http://www.puppylinux.com/download/downpage.htm

My download page also has links to other URLs where we host more source packages, so you will be able to locate any that are missing.

Step 4: Start compiling!

The 'build' directory is where all output is created. A complete root filesystem is built in here, an emulated environment, complete with its own kernel and CPU. The binary packages will also be created in here as .tar.bz2 files, and these are what we want for our Puppy.

After downloading all required source packages, the entire process of building the binary (compiled) packages is done with just this one line:
# ./scripts/Build-Target -cfg piglet001
After about one day it should be done!

I'm not kidding, it takes ages, and needs heaps of space.  A 2GHz CPU and at least 256M of RAM is required if you want it to complete in "only" 1 day.
You can still simultaneously use the PC for Internet access, but response is going to be sluggish, and don't expect to do anything else heavy-duty.

Step 5: Compile failures

When T2 compiles a package, the source tarball is expanded and T2 creates a symlink in the 't2-trunk' directory to it. Thus, click on the link and you will be taken into the build environment for the package. Normally T2 will remove the package build environment, but if the build fails it is left there. So, later on you can go in and try to find out what went wrong.

Also, T2 maintains a build log in build/piglet001-8.0-trunk-piglet-x86-i486/var/adm/logs/. For example, if Seamonkey has failed to compile, there is this file: build/piglet001-8.0-trunk-piglet-x86-i486/var/adm/logs/5-seamonkey.err.
This file is the most helpful for finding the reason for failure.

Lots of failures result in very large package build-environments being left behind, quickly reducing free space in the partition. Plus there are all sorts of other temporary files. You can clean everything up (which does not delete the successfully compiled binaries) by doing this:
# ./scripts/Cleanup

If you do want to delete all the compiled packages and start from scratch, do this:

# ./scripts/Cleanup -build -cache

Step 6: Fixing failures

A build failure may be due to a missing prerequisite, or even a prerequisite compiling after the failed package. Or, configure options may be wrong.

Each package has its own .conf file, where you can control the build process. For example, Seamonkey has package/www/seamonkey/seamonkey.conf. This can be edited, though Rene mentioned that this is not desirable as it affects upgradability of the T2-build-pkg -- by customising these files we are making it difficult to sync with the main T2 build package when they have new releases. To get around this problem, an "over-arch" .conf file can be created in the target/piglet/ directory -- see the online T2 Handbook for details on how to do this.

When you think that you have fixed the failure, maybe cleanup then run the Build-Target script again:
# ./scripts/Cleanup
# ./scripts/Build-Target -cfg piglet001

Step 6B: Fixing Dingo 4.00

There are some hacks that you have to do manually to get Dingo Puppy 4.00 packages to compile. This is the T2-core-build package named t2-puppy-4.00.tar.gz. Later versions may have fixed these problems.
  1. Perl compile, stage 1, fails. To fix this, go into directory package/database/bdb/ and rename the file bdb.desc-SECOND to bdb.desc. Then do this:
    # ./scripts/Cleanup
    # ./scripts/Config -cfg piglet001  (and just exit)
    # ./scripts/Build-Target -cfg piglet001
    After Perl has successfully compiled, restore the original bdb.desc file.
  2. librsvg installation, stage 5, fails. To fix this, open the file build/..../var/adm/flists/gtk+ in a text editor and rename the text string "gdk-pixbuf.loaders" to anything else, for example "gdk-pixbuf.loaders-HIDE". Then do this:
    # ./scripts/Cleanup -cache
    # ./scripts/Build-Target -cfg piglet001
    After librsvg has succesfully installed, edit the gtk+ file and change that text string back to what it was.
  3. mplayer compile, stage 5, fails. To fix this, copy the file libavutil/intreadwrite.h out of the mplayer source package into build/..../usr/include/ffmpeg/
    # ./scripts/Cleanup -cache
    # ./scripts/Build-Target -cfg piglet001
    This is a problem with Mplayer using the external ffmpeg package.

Step 7: Adding more packages

Open the file target/piglet/pkgsel and you will see some packages commented out. You can uncomment these, then run the Built-Target script again. You can also add any more packages that you want. That is, do this:
# ./scripts/Build-Target -cfg piglet001

Puppy will just continue on compiling and adding these new packages to the build system.

Step 8: Exporting to Unleashed

The complete new filesystem has been built in build/piglet001-8.0-trunk-piglet-x86-i486/. Also, all successfully compiled packages have been made into .tar.bz2 tarballs, to be found in:
build/piglet001-8.0-trunk-piglet-x86-i486/TOOLCHAIN/pkgs/

You can copy these packages out, and use them in Puppy.


(c) Copyright 2007 Barry Kauler www.puppylinux.com