Promiscuity among gene regulatory networks (GRNs) has been widely observed, however no in silico studies have characterized their contribution to evolution. However, recently developed biophysically interpretable models which approximate sequence-to-expression mapping of yeast via datasets of 100 million promoters may offer a solution. Here, we develop a naturalistic framework using such models to simulate the evolution of populations of GRNs. By investigating these simulations, we find that evolutionary adaptation is driven primarily by transcription factors (TFs) with nonspecific binding motifs. Systems in which TFs lack a sufficient variation in binding affinity consistently fail to adapt and improve population mean fitness. These data suggest that TF-promoter promiscuity is a necessary condition of evolutionary adaptation.