GATT comes into play once a dedicated connection is established between two devices, meaning that you have already gone through the advertising process governed by GAP.
The most important thing to keep in mind with GATT and connections is that connections are exclusive. What is meant by that is that a BLE peripheral can only be connected to one central device (a mobile phone, etc.) at a time! As soon as a peripheral connects to a central device, it will stop advertising itself and other devices will no longer be able to see it or connect to it until the existing connection is broken.
Establishing a connection is also the only way to allow two way communication, where the central device can send meaningful data to the peripheral and vice versa.
If data needs to be exchanged between two peripherals, a custom mailbox system will need to be implemented where all messages pass through the central device.
Once a connection is established between a peripherals and central device, however, communication can take place in both directions, which is different than the one-way broadcasting approach using only advertising data and GAP.
The peripheral is known as the GATT Server, which holds the ATT lookup data and service and characteristic definitions, and the GATT Client (the phone/tablet), which sends requests to this server.
All transactions are started by the master device, the GATT Client, which receives response from the slave device, the GATT Server.
When establishing a connection, the peripheral will suggest a 'Connection Interval' to the central device, and the central device will try to reconnect every connection interval to see if any new data is available, etc. It's important to keep in mind that this connection interval is really just a suggestion, though! Your central device may not be able to honour the request because it's busy talking to another peripheral or the required system resources just aren't available.
The following diagram should illustrate to data exchange process between a peripheral (the GATT Server) and a central device (the GATT Client), with the master device initiating every transaction:
A full list of officially adopted BLE services can be seen on the Services page of the Bluetooth Developer Portal. If you look at the Heart Rate Service, for example, we can see that this officially adopted service has a 16-bit UUID of 0x180D, and contains up to 3 characteristic, though only the first one is mandatory: Heart Rate Measurement, Body Sensor Location and Heart Rate Control Point.
Similarly to Services, each Characteristic distinguishes itself via a pre-defined 16-bit or 128-bit UUID, and you're free to use the standard characteristics defined by the Bluetooth SIG (which ensures interoperability across and BLE-enabled HW/SW) or define your own custom characteristics which only your peripheral and SW understands.
As an example, the Heart Rate Measurement characteristic is mandatory for the Heart Rate Service, and uses a UUID of 0x2A37. It starts with a single 8-bit value describing the HRM data format (whether the data is UINT8 or UINT16, etc.), and the goes on to include the heart rate measurement data that matches this config byte.
Characteristics are the main point that you will interact with your BLE peripheral, so it's important to understand the concept. They are also used to send data back to the BLE peripheral, since you are also able to write to characteristic. You could implement a simple UART-type interface with a custom 'UART Service' and two characteristics, one for the TX channel and one for the RX channel, where one characteristic might be configured as read only and the other would have write privileges.