The main losses in perovskite solar cells (PSCs) are often related to their charge-transport layers (CTLs). While it has been shown how losses related to charge extraction and recombination at the CTL interfaces can be minimized individually, a comprehensive picture is still lacking. In this work, we describe how recombination at interfaces is related to the distribution of electric potential within the device based on drift-diffusion simulations. Our results show that interface recombination in PSCs can only be fully understood when the whole device is considered. We find that devices with poor conductivity in the CTLs or small built-in potential will suffer from fast recombination at interfaces and that the interface recombination can be avoided by improving these. To illustrate our results, we present a framework that provides an intuitive way to understand electrical potential and compare losses in PSCs. We show that the simulation data can be well understood within the framework for the whole range of parameters studied.