Volumetric brain reconstructions provide an unprecedented opportunity to gain insights into the complex connectivity patterns of neurons in an increasing number of organisms. Here, we model and quantify the complexity of the resulting neural connectomes in the fruit fly, mouse, and human and unveil a simple set of shared organizing principles across these organisms. To put the connectomes in a physical context, we also construct contactomes, the network of neurons in physical contact in each organism. With these, we establish that physical constraints—either given by pairwise distances or the contactome—play a crucial role in shaping the network structure. For example, neuron positions are highly optimal in terms of distance from their neighbors. Yet, spatial constraints alone cannot capture the network topology, including the broad degree distribution. Conversely, the degree sequence alone is insufficient to recover the spatial structure. We resolve this apparent mismatch by formulating scalable maximum entropy models, incorporating both types of constraints. The resulting generative models have predictive power beyond the input data, as they capture several additional biological and network characteristics, like synaptic weights and graphlet statistics. Author Summary We investigate the interplay of the spatial and topological structure of millimeter-scale neural connectomes in fly, mouse, and human. As a spatial observation, we demonstrate that the probability of synaptic connection decays exponentially with distance. Additionally, we show that the wiring length in neural connectomes is highly optimal. To quantify the physical constraints on synapse formation, we construct the physical contact network for each organism and demonstrate that contact edge probability follows the same exponential functional form as the connectome. At the same time, we show that spatial constraints are necessary but not sufficient to reconstruct the connectome topology. We present maximum-entropy models capturing key spatial and topological aspects of the connectomes and demonstrate their predictive power beyond the input data.