How can RF grounding performance be improved on a compact PCB with limited ground area?

Use a 4+ layer PCB design, allocating an entire inner layer as a solid ground plane. Implement dense stitch vias (via shielding) along high-frequency trace margins to minimize ground loop impedance.