Novelty search was proposed as a means of circumventing deception and providing selective pressure towards novel behaviours to provide a path towards open-ended evolution. Initial implementations relied on neuro-evolution approaches which increased network complexity over time. However, although many studies have reported impressive results, it is still not clear whether the benefits of evolving topologies are outweighed by the overall complexity of the approach. Given that novelty search can also be combined with evolutionary methods that utilise fixed topologies, we undertake a systematic comparison of evolving topologies, using two types of fixed topology networks in conjunction with novelty search on two test-beds. We show that evolving topologies do not systematically help, and discuss the practical consequences of these results and the research perspectives opened up.