GATT comes into play once a dedicated connection is established between two devices, meaning that you have already gone through the advertising process governed by GAP.
The most important thing to keep in mind with GATT and connections is that connections are exclusive. What is meant by that is that a BLE peripheral can only be connected to one central device (a mobile phone, etc.) at a time! As soon as a peripheral connects to a central device, it will stop advertising itself and other devices will no longer be able to see it or connect to it until the existing connection is broken.
Establishing a connection is also the only way to allow two way communication, where the central device can send meaningful data to the peripheral and vice versa.
The following diagram should explain the way that Bluetooth Low Energy devices work in a connected environment. A peripheral can only be connected to one central device (such as a mobile phone) at a time, but the central device can be connected to multiple peripherals.
If data needs to be exchanged between two peripherals, a custom mailbox system will need to be implemented where all messages pass through the central device.
Once a connection is established between a peripherals and central device, however, communication can take place in both directions, which is different than the one-way broadcasting approach using only advertising data and GAP.
An important concept to understand with GATT is the server/client relationship.
The peripheral is known as the GATT Server, which holds the ATT lookup data and service and characteristic definitions, and the GATT Client (the phone/tablet), which sends requests to this server.
All transactions are started by the main device, the GATT Client, which receives response from the secondary device, the GATT Server.
When establishing a connection, the peripheral will suggest a 'Connection Interval' to the central device, and the central device will try to reconnect every connection interval to see if any new data is available, etc. It's important to keep in mind that this connection interval is really just a suggestion, though! Your central device may not be able to honour the request because it's busy talking to another peripheral or the required system resources just aren't available.
The following diagram should illustrate to data exchange process between a peripheral (the GATT Server) and a central device (the GATT Client), with the main device initiating every transaction:
GATT transactions in BLE are based on high-level, nested objects called Profiles, Services and Characteristics, which can be seen in the illustration below:
A Profile doesn't actually exist on the BLE peripheral itself, it's simple a pre-defined collection of Services that has been compiled by either the Bluetooth SIG or by the peripheral designers. The Heart Rate Profile, for example, combines the Heart Rate Service and the Device Information Service. The complete list of officially adopted GATT-based profiles can be seen here: Profiles Overview.
Services are used to break data up into logic entities, and contain specific chunks of data called characteristics. A service can have one or more characteristics, and each service distinguishes itself from other services by means of a unique numeric ID called a UUID, which can be either 16-bit (for officially adopted BLE Services) or 128-bit (for custom services).
A full list of officially adopted BLE services can be seen on the Services page of the Bluetooth Developer Portal. If you look at the Heart Rate Service, for example, we can see that this officially adopted service has a 16-bit UUID of 0x180D, and contains up to 3 characteristic, though only the first one is mandatory: Heart Rate Measurement, Body Sensor Location and Heart Rate Control Point.
The lowest level concept in GATT transactions is the Characteristic, which encapsulates a single data point (though it may contain an array of related data, such as X/Y/Z values from a 3-axis accelerometer, etc.).
Similarly to Services, each Characteristic distinguishes itself via a pre-defined 16-bit or 128-bit UUID, and you're free to use the standard characteristics defined by the Bluetooth SIG (which ensures interoperability across and BLE-enabled HW/SW) or define your own custom characteristics which only your peripheral and SW understands.
As an example, the Heart Rate Measurement characteristic is mandatory for the Heart Rate Service, and uses a UUID of 0x2A37. It starts with a single 8-bit value describing the HRM data format (whether the data is UINT8 or UINT16, etc.), and the goes on to include the heart rate measurement data that matches this config byte.
Characteristics are the main point that you will interact with your BLE peripheral, so it's important to understand the concept. They are also used to send data back to the BLE peripheral, since you are also able to write to characteristic. You could implement a simple UART-type interface with a custom 'UART Service' and two characteristics, one for the TX channel and one for the RX channel, where one characteristic might be configured as read only and the other would have write privileges.