Munckfish.net

Following on from my previous posts I’ll now make some notes on how to synchronise with the upstream kernel Git HEAD.

The KernelTeam’s KernelMaintenance page does mention a little about this but provides no examples or detail. I found out what was necessary by sending questions to the kernel team mailing list.

In my repository the “origin” repository is an Ubuntu one (ubuntu-intrepid-ports) as this is where I cloned from. But originally this will have been based on a repository from git.kernel.org. So for convenience, we can add a git “remote” definition for our target upstream kernel.org repository so that we can refer to it by a simple name. In this case I’ll just use the name of the upstream tree, linux-2.6.25.y. linux-2.6.25.y is the stable tree for maintenance releases of the 2.6.25 kernel which the ubuntu-intrepid-ports kernel is currently based on.

git remote add linux-2.6.25.y git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.25.y.git

Now we need to fetch all the new upstream changes into our repository. This doesn’t make any changes to our current branch it just downloads upstream revisions into a remote-tracking branch in the background. You can view remote-tracking branches by running git branch -r, you should see linux-2.6.25.y/master listed.

git fetch linux-2.6.25.y

Next we can see what new revisions are available upstream by listing logs from our HEAD through to the HEAD of the upstream remote-tracking branch.

git log HEAD..linux-2.6.25.y/master

We can also see which of our local changes will be rebased by asking git log to show us commits which are in the HEAD but not in the remote. For ubuntu-intrepid-ports this will be all commits which add to or modify the packaging scripts in the “debian/” directory, and any other special patches which have been applied. Most of the commit logs for these revisions should start with “UBUNTU:”.

git log linux-2.6.25.y/master..HEAD

Now we’re ready to actually rebase. Run the following command. You should see lots of output from git as it saves away local changes, resets back to the last commit pulled from the remote, pulls in all the new revisions from the remote, then replays all our commits on top.

git rebase linux-2.6.25.y/master

That’s it. We’ve resynced with upstream. Now we’re ready to build and test.

If you’re happy the tree is in a good state the next job is to make your updates public. If you have been allocated a git tree on kernel.ubuntu.com you can push the changes there. Otherwise you’ll need to investigate other ways of hosting a remote git repository.

Add a “remote” definition for your public repo. In this example I’m accessing the remote repo using SSH as the transport.

git remote add mypublic myusername@myhost:/path/to/git/repos/dmunckton/ubuntu-intrepid-ports.git

You have to force-push the changes to your remote public tree because where we’ve rebased our history has completely changed. Any history already in your public tree will completely different. The -f option to git push forces it to make the remote repository reflect your local changes. If you don’t do this the push will fail.

git push -f mypublic master

Be careful to warn anyone following your public tree as this type of push will prevent easy tracking using git pull. They will need to follow the instructions in the KernelGitGuide wiki page to track your changes.

As usual if you find any innaccuracies or mistakes please do leave a comment.

The easiest way to build the kernel for PS3 is to do so on the PS3 itself. However even with great tools like ccache to speed up repeated builds it can take a long time on PS3 most probably because of its relative small amount of RAM (256MB). So I made a point of working out a way to cross build the Ubuntu packaged version of the kernel on my x86 machine. Here are my notes on how to do that.

Prerequisites

I’m going to presume the reader already has a git checkout of the ubuntu ports source tree and knows how to build the Ubuntu kernel packages on the target architecture. Notes on this can be found in the “Performing Builds” section of the KernelMaintenance wiki page.

I recommend the reader either runs Hardy or Intrepid on their x86 system so that toolchain packages from the PS3 Dev Team PPA will install easily.

The Problem

Cross compiling the Linux Kernel directly is well supported. You need only set 2 environment variables so the build scripts know the target architecture and where to find the cross build tool chain. E.g. if you’re targeting PowerPC and your build toolchain (binutils, gcc etc) binaries are named as powerpc-linux-gnu-gcc, powerpc-linux-gnu-ld etc then you’d run something like this to build the kernel:

make ARCH=powerpc CROSS_COMPILE=powerpc-linux-gnu-

But what I wanted was a valid installable Ubuntu package at the end. Usually to avoid any chance of missing build platform related build errors Ubuntu/Debian packages are always built on the target architecture. However there’s quite a bit of work happening in Debian to help support cross building for low resource embedded platforms (e.g. mobile, set top boxes etc) so with a little hunting I was able to find out a very simple way of achieving my goal.

The Toolchain

First problem is we need to get a toolchain built for x86 but compiled to build for our target architecture which is powerpc64 with specific knowledge of the Cell CPU.

Luckily, the old Cell-SDK had already been packaged for Ubuntu by Matthias Klose, and is built for both powerpc and x86 architectures. However there was a bug which prevented kernel compilation. So I updated these packages to the latest from the Cell-SDK site. To get my updated packages add the PS3 Dev Team’s PPA to your APT sources list (basic instructions on that last page, make sure you choose ‘hardy’ or ‘intrepid’ ) and run:

sudo apt-get install ppu-gcc

Now your cross toolchain should be available as ppu-gcc and ppu-ld etc.

While the toolchain patches for the Cell were being developed they were created against GCC 4.1 and Binutils 2.17 and sources and binaries were (and still are) hosted at Barcelona Supercomputing Centre and known as the Cell-SDK. These patches have now been absorbed into the current versions of GCC and Binutils (GCC 4.3 and Binutils 2.18 I believe) so in theory it’s no longer necessary to use the Cell-SDK.

Intrepid’s toolchain is based on these newer versions and despite there being notes on a way to create Debian packaged cross toolchains using them I wasn’t able to get them both to compile. See debian/README.cross in the binutils and gcc package sources for more info. I’m sure the problem can be easily resolved but for now the Cell-SDK has got me up and running quickly.

Setting The Target Architecture For The Package Scripts

Now we have a toolchain the next problem was how to force the Ubuntu packaging scripts to use it. After much scanning of source code this turned out to be very simple indeed. There are whole a series of environment variables which need to be set. Luckily the utility dpkg-architecture makes it very simple to set these. You just need to pass it your desired architecture and evaluate it’s output in your shell, thus:

eval $(dpkg-architecture -apowerpc -s)

I’ll leave it to the reader to look at the man page for dpkg-architecture to understand more about this utility and the variables which get set.

Putting It All Together

So putting all these things together, this is roughly the script I use to cross build the kernel.

#!/bin/sh

# Decide where we're going to log to this time
LOG=$(pwd)/../$(basename $(pwd))-$(date +%F-%H%M).log

# Make sure ccache is used for HOST_CC
export PATH=/usr/lib/ccache:$PATH

# Setup the deb arch variables for cross building to target powerpc
eval $(dpkg-architecture -apowerpc -s)

# Run the build
CROSS_COMPILE="ccache ppu-" fakeroot debian/rules binary-powerpc64-smp 2>&1 | tee "$LOG"

# Let you know it's finished
aplay /usr/share/sounds/phone.wav

This script should be run from the top of the Ubuntu kernel source. It logs all the build output to a file in the directory above but also outputs everything to the shell.

Note CROSS_COMPILE is set before running the build command. Although this variable is not touched directly by debian/rules it is propogated to the kernel build scripts.

ccache can be used for cross building. For me it halved the build time to 25 mins on second run so it’s well worth using it. Caching for the host architecture is supported on Ubuntu by pre-pending the /usr/lib/ccache directory to $PATH. I’ll leave it to the reader to find out how this works. In order to support caching for the foreign architecture we can simply modify the CROSS_COMPILE variable to prepend a call to ccache.

I also like to use the screen utility so I can walk away/logout etc and return later. See my previous post on how to use that. Finally the script runs a WAV file so I know when it’s finished.

UPDATES:

30 Aug 08: Updated notes on using ccache for cross compilation. Updated the build script to use ccache for the target architecture.

I want to make a series of posts to record some of the things I’ve found out about maintaining the Ports Kernel for PS3 so far. I’m still to complete the full cycle to getting an updated kernel build into the repos but I’d like to note what I’ve found out so far. As I know more I’ll endeavor to update these notes so they remain fairly accurate.

Background

The kernel used for PS3 in Ubuntu is the generic powerpc64-smp flavour. Unfortunately the PowerPC architecture is no longer an officially supported architecture in Ubuntu and therefore it falls to the community to keep it alive. The kernel source for this and the other unsupported architectures have been moved out into a separate git source tree to reduce the complexity and fragility of kernel maintenance. E.g. a bad change in the ports tree won’t break the build for the supported flavours. For the upcoming Intrepid release this is the ubuntu-intrepid-ports tree.

The important thing to note here is the PS3 kernel source is still shared with other architectures such as HPPA, ia64, SPARC etc, as well as for the standard PowerPC platform so any changes we make for PS3 must not break the kernel for any of these platforms. To verify this I will need to find interested parties who have relevant equipment to test on as I will not have access to any of these platforms myself. More on this in later posts.

Stuff To Do

First thing to do is read the KernelTeam/KnowledgeBase articles on the Ubuntu Wiki. In particular the KernelGitGuide and the KernelMaintenance pages.

Presuming you’re already subscribed to and known on the ubuntu-cell list, next subscribe to the Ubuntu Kernel Team mailing list and introduce yourself and your intentions. You will also need to monitor or subscribe to the actual kernel mailing list, and also specifically to the PS3/Cell Kernel development list.

Geoff Levand is (one of the folks) employed by Sony to work on the Linux Kernel for Cell and PS3. He keeps releases of the PS3 “Distro Kit” and lots of interesting documentation in his directory on kernel.org: http://www.kernel.org/pub/linux/kernel/people/geoff/cell/. You’ll need to be familiar with what’s there and hoover up the information provided.

Having done all this you should now have

• A local clone of the ubuntu-<codename>-ports tree (replace <codename> with ‘intrepid’ or whichever is the current codename as you read this) to work in.
• A good (but theoretical) idea of the maintenance process to follow.
• And you should know where ‘upstream’ is and where relevant development discussion is happening.

Getting More Official

The way development is structured is each of the Ubuntu Kernel Devs have their own Git source trees on http://kernel.ubuntu.com/git. They work locally then push there changes to their tree hosted there. Changes from their hosted tree can then be pulled into the intended mainline source trees as necessary, e.g. I may push my changes to dmunckton/ubuntu-intrepid-ports and when ready the changes will be moved to ubuntu/ubuntu-intrepid-ports. In theory this is how it works, still to get my changes approved and moved across.

Ben Collins is the guardian of kernel.ubuntu.com (a.k.a. zinc). To get your own tree there you need to convince him you’re serious and he can then get you a login setup. Expect this to take a while - until I had a login I made my own local builds and uploaded them to my website for review.

Once you have a login follow the instructions under the section titled “Developers with access to kernel.ubuntu.com” on the KernelGitGuide page to get your hosted tree setup.

What Next?

In the next few posts I will hopefully deal with building, cross-compiling, rebasing against new official upstream kernel releases, looking for PS3 specific fixes, testing, and (fingers crossed) hopefully record the process of getting an updated build into the repos.

Netbeans is not currently available in the Ubuntu/Debian repositories. I believe this is because it uses the Sun CDDL licence which is considered incompatible with the GPL.

However, as an end-user you may wish to use Netbeans despite that. Although it comes with a neat installer wizard I personally prefer to try to get native .deb packages where possible in order to get native package management and menu/launcher icons. I found a debian packaged version by Daniel Baumann. It appears he put it on the Debian new queue but is wasn’t accepted. He’s made the prepacked binaries available here:

http://archive.daniel-baumann.ch/debian/packages/netbeans/5.5-1/

However, I wanted to learn something so I decided to try building a package using the current 5.5 source dist from Netbeans archive and Daniel Baumann’s debian package source. Here are the steps I took.

Building and Packaging Netbeans 5.5 for Ubuntu

1) Grab daniel-baumann’s patch here

netbeans_5.5-1.diff.gz

2) Get the Netbeans 5.5 source from:

http://www.netbeans.info/downloads/all.php?b_id=2180&src=1

I picked up the ’source’ tgz file from the table of links at the bottom of the page.

3) Untar the sources

4) Now apply the patch to the sources

The patch simply installs the debian packaging kit into the Netbeans 5.5 src directory. After applying you will find the new files under the newly created debian directory. For more info on what these files do see The Ubuntu Packaging Guide or the Debian New Maintainers Guide.

6) Next build the packages and install

When debuild completes there will be 2 .deb files left in the directory above:

netbeans-ide_5.5-1_all.deb
netbeans-platform_5.5-1_all.deb


debi installs both files by reading the netbeans_5.5-1_i386.changes file which will have been generated in the directory above.

Update: As of Fri 19th Jan 07 Java 6 has now been officially included into the Dapper Backports repository. See it’s package page.

Today I found out how to package Java for Ubuntu/Debian. I wanted to see if a .deb package file of Java 6 was available for Ubuntu I found details on creating one in the following forum post: Re: JDK 6 Available…now when is a deb coming?. The instructions there are great except they don’t work perfectly for Dapper so here are the modified steps for Dapper:

1) Patch ‘java-package’

The Debian Java Maintainers have created a tool make-jpkg which is a neat script to take the Sun Java native linux .bin installer and turn it into a .deb package. This script comes in the ‘java-package’ package which is in the multiverse repo in Ubuntu.

Dapper stocks the 0.27 version which does not support conversion of Java 6 yet. However, mlind attached a patch to his forum post (link above) to add support for Java 6 to java-package version 0.28. Luckily it’s not difficult to tweak the patch for 0.27. Here’s the modified version of the patch:

java-package_sun-java6.dapper.patch

Here’s the instructions on how apply the patch and build the updated package:

2) Package sun-jdk6

Now that you’ve updated java-package it’s time to package Java 6. First download Java 6 from Sun. You need to download the Linux self extracting file (jdk-6-linux-i586.bin). Here are the instructions on building it (taken from the forum post listed above).

Screen is a window manager for unix shells. It’s useful when logged in to a remote server because it allows you to have more than one shell running through the same terminal window. So you don’t have to log in multiple times through SSH to, for example, run a long running command in one shell and continue working in another.

This is just a quick note to help remember the most common commands. For more information see man screen.

Getting in and out

Start screen

Type screen in the terminal

Detach from a screen session (session remains open)

Ctrl+a d

Reattach to an open screen session

screen -D -R

Quit screen completely (session closed)

Ctrl+a Ctrl+\

Controlling windows

Create a new window with a shell and switch to it:

Ctrl+a c

List currently open windows for selection:

Ctrl+a "

Close current window:

Ctrl+a k

Controlling Regions

Using regions allows you to display more than one shell on your terminal.

Split the current region into 2 new ones (note: new region will be blank, you need to switch to it then select a window to display in it):

Ctrl+a S

Switch focus to the next region:

Ctrl+a TAB

Remove the current region:

Ctrl+a X

That should be enough to get started

This is a rough quick start guide. It was written to help a relative of mine who was starting to learn Java.

1. Get the Java Development kit here: Java Platform, Standard Edition (Java SE). The current version at the time of writing is Java 5
2. Once installed set the following environment variable in order to ease use of the development tools on the command line. This environment variable is used by various tools including Ant and Tomcat also.

   JAVA_HOME=[set this to the path you installed Java in]

   The default install location for Java 5 on Windows is C:\Program Files\Java\jdk1.5.0_06.

   To set permanent environment variables on Windows XP/2000:

   1. Right-click on My Computer icon
   2. Select Properties from the context menu
   3. In the dialog that appears click on the ‘Advanced’ tab
   4. At the bottom of this tab there is a button labelled ‘Environment Variables’ click it
   5. In the dialog that appears create the new environment variable in the bottom box labelled ‘System variables’ - that will set the variable for all users of the system.

   This process is completely different on Unix/Linux systems, I can describe the setup process for Redhat and Ubuntu systems in a comment if someone needs it.

3. Next make sure the Java executable binaries are on your system path, this is so that you can use the Java tools to compile your code. On Windows modify the definition of the Path environment variable to include this at the very end (semi-colon at the beginning is required):

   ;%JAVA_HOME%\bin

   When the system loads %JAVA_HOME% will expand to the value you defined in point 2.

4. Test the setup from above by opening a command console (Start > Programs > Accessories > Command Prompt) and type:

   java -version

   You should see the following output:

   
java version "1.5.0_06"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
Java HotSpot(TM) Client VM (build 1.5.0_06-b05, mixed mode)


   If you don’t get that type set to list the environment variables and check that the two variables you set in steps 2 and 3 are ok.

5. Now you are all set to start learning. I recommend starting with the Java Tutorial.
6. For alternate/advanced coverage see Bruce Eckel’s free book “Thinking In Java”.

Advice

• Don’t use an IDE to start coding Java. It may get you started quickly but IDEs hide a lot of the real details of how a technology works. Eventually all good developers need to understand these details; it’s quicker and easier to do this from the start. Get yourself a decent text editor and learn to use the command line tools. Once you understand how Java works an IDE such as Eclipse can be very empowering.
• Decent editors are (IMHO on Windows): JEdit, Notepad2, Vim, of course there are many others
• When you get fed-up typing javac to compile your code learn to use Ant
• Learn as much as you can about Object Orientated Programming

Enjoy.

This was written quickly so if you notice any mistakes or things which could be improved please add a comment and I’ll fix it.