There are three general types of Arduino Libraries:
The Arduino IDE comes with a set of standard libraries for commonly used functionality. These libraries support all the examples included with the IDE. Standard library functionality includes basic communication functions and support for some of the most common types of hardware like servo motors and character LCD displays.
Standard Libraries are pre-installed in the "Libraries" folder of the Arduino install. If you have multiple versions of the IDE installed, each version will have its own set of libraries. For the most part, it is not a good idea to change the Standard Libraries or install your libraries in the same folder.
The newer Arduino IDE has a Library Manager which facilitates installing third-party libraries submitted to Arduino for use. Adafruit has most of its libraries and drivers in the Library Manager for easy use.
There are many other libraries with useful functionality and device drivers for all sorts of hardware. These drivers are available from places like Arduino Playground, Github and Google Code.
User installed libraries should be installed in your sketchbook libraries folder so they can be used with all versions of the IDE. This way, if a new version of the IDE is released, you don't have to re-install all your favorite libraries!