For more accurate orientation output, the magnetometer needs to be adjusted to compensate for offset errors (meaning a shift in either direction on the X/Y/Z axis), as well as something called 'soft iron error', which causes what should be spherical output of the magnetometer to actually be an elongated 'pill' shape or something similar.

In order to compensate for these two problems and generate accurate magnetometer data, which is critical to getting high quality orientation results, you will need to calibrate your magnetometer in the final enclosure and setup in which the device will be used.

PJRC MotionCal

There are a variety of calibration tools out there, but MotionCal from always awesome PJRC (Teensy, etc.) is probably the easiest to use, and they provide pre-compiled binaries for all major operating systems.

Download the Motion Sensor Calibration tool for your OS from the following link and open it up on your system: http://www.pjrc.com/store/prop_shield.html

By default, you should see something like this:

Next open up the ahrs_calibration example from Adafruit_AHRS:

Be default this is setup for the sensors found on the Adafruit 9DOF and 10DOF breakouts, but can be adjusted to work with any accelerometer, magnetometer and gyroscope that is based on the Adafruit Unified Sensor system.

Generating Calibration Data

Next run the sketch, and if it isn't already open, open the MotionCal app.

Select the appropriate serial port for your board, and once select start rotating the board around gently in figure eight type movements.

As you move the device, points should start to appear in a point cloud.

You need to keep moving the device in as many orientations as possible until the sphere is mostly complete and the Gaps percentage on the bottom of the app is reasonably low.

A good example of what you should aim for is shown below:

Note Offsets and Magnetic Mapping Values

Now that you have determined the values to compensate for magnetometer offset and soft-iron error, make a note of the values in the top-right hand corner of the app, which in this case is:

Magnetic Offset

-2.20, -5.53, -26.34

Magnetic Mapping

0.934, 0.005, 0.013
0.005, 0.948, 0.012
0.013, 0.012, 1.129

Magnetic Field

48.41

That's It!

Congratulations, you have everything you need to generate reasonably accurate ouput on the magnetometer taking into account any local interference in your environment or caused by the board itself.

You'll need these values in the next step, which is feeding your compensated sensor output into a sensor fusion algorithm, but you can close the MotionCal app for now until you need to calibrate the device again, such as if you move it to a different setup or environment.