No, yeah, that's how the LoGN works.
For best results, you want to have relatively large Mana Producers, such that they have relatively large total culture values, so that they will have higher priority for receiving the NH from Neighbors who decide to buff your Culture buildings. When I first put my own LoGN in, I had a bunch of Lava Eggs for Mana (pretty good sized Culture), but also was using up a bunch of Ferris Wheel Galores (even betterer Culture), and so didn't start seeing the benefits on my Lava Eggs until the Wheels timed out.
For double benefit, if you don't already have it, also consider the Dragon Abbey, which gives Mana when using Enchantments, so putting EEs on your Mana producers gives Mana for casting, and then Mana for collecting after buffing after spelling.