You don´t need Prestige at all if you have the PP and Portal. I for example never built prestige or settlement at all.
But if your question is what happens after you build the settlement, but then sell the prestige, I can´t actually tell you. I suppose selling the smallest prestige building you have will answer the question...
Well luckily I have sold prestige after the fact, I replaced Temples with the smaller 3x3 Palm Gardens in order to fit in more production buildings, and then upgraded them. It must have cost a fortune in diamonds to level the portal without grave goods and traps, I would rather just play the chapter and get into the spirit of hosting guests.
It doesn't cost any diamonds. Just portal profit spells. I haven't built a full settlement since Halflings. I put some of the settlement up for the first part of Elementals, but since then I've only ever built the portals.