The PCA9865 chip has an "All Call" address of 0x70. This is in addition to the configured address. Set the backpacks to address 0x71 or anything other than the default 0x70 to make the issue go away.
It can be used for LEDs as well as any other PWM-able device!
If you want to turn the LEDs totally off use setPWM(pin, 4096, 0); not setPWM(pin, 4095, 0);