Intro to Guild Leveling
Cataclysm is introducing a guild leveling system, where guilds gain experience as their members complete things (such as winning rated BGs, killing bosses, completing quests and unlocking achievements).
It had been planned that guilds would get talent points to spend in talents as they leveled, so they could choose between getting boosts to small boosts in economic, PvE or PvP related areas. The talent points have now been removed and replaced with perks that every guild will automatically get as they level.
Other Guild Bonuses
Players will gain guild reputation as they do things in-game. Gaining reputation for the player gives access to things like tabards, heirlooms and fancy mounts as long as the guild is a high enough level to unlocked them. Changing guilds will reset reputation but leaving and rejoining the same guild should have a limited impact (assuming it is done relatively quickly)