Ok this has nothing to do with DKP... but the brains are here so i shall ask it here.

We are having some conflict as far as how to set up an instance team, of 20 - 40 person team. Both Ventrilo as well as the team its self. Some say set it up warrior with the healer and the dps. others say set it up with kepping alike classes together. I guess what i am after is what is the best way to organize an end game type run, both player set up and "Chat" set up.

Any help on this would be great.

I'm not sure I get your last part about player setup and chat setup but I'll give it a shot.

There is absolutely no sense in setting up the raid with like classes being together in a party. There are several reasons for this but I'll list the 2 most obvious. Paladins Auras are meant to provide specific bonuses their party (devotion = greater armor for tanks, concentration = less interrupt chance for casters). Shamans totems were meant to give specific bonus to their party (there are specific totems a Resto shaman would drop to affect healing as opposed to an elemental shaman for caster dps as opposed to an enhancement shaman for melee dps). Now that we have 2 clear examples of why I think you never group like classes lets talk about group makeup and what we try do.

Grouping by Role
Healer/Caster/Ranged DPS Group
This group will consist of mostly healers (depending on raid size) and DPS casters with 1 of them being a shadow priest. The reason for this is that boss fights are typically long multistage fights. Having a shadow priest with you Caster group and most importantly your Healers ensures that they are less likely to run out of mana.

Melee Group
Typically this group will consist of varying components based on benefit. Depending on Raid size you have 1 or more of these groups. Your tank/s are usually in this group and are placed with those classes that can increase their bonuses to things such as Armor, HP, and Dodge. Those class specs to place them with would be a Paladin, Warlock (imp bloodpact), and enhancement Shaman if these are available. If you have all of these then you fair melee/tank and can add a rouge, ret paladin, feral druid, or arms/fury warrior to round it out.

From a pure melee dps perspective we look at things for group to amplify damage out put.
Feral Druid = Leader of the Pack - if your druid is a main or off tank then you satisfy a couple of things already by just having them in the group. (they don't amplify damage but a Paladin Tank also frees up that slot of having a paladin in that group that might not be a melee dps spec to have devotion on themselves)
Enhancement Shaman - The various totems that an enhancement shaman can drop and maintain during combat give a pretty good bonus to all the melee dps in their group in the area.

What about the warlock in this group. They aren't melee dps and might not be close enough for blood pact. Remember that a warlock and put their pet on stay anywhere you want them to and we often have positions in a fight where you have melee versus ranged and healing groups standing. But if we have a shadow priest available then why not stick them in this group. While that might be a better choice if you only like is destruction when it comes to a demonology warlock or an affliction warlock there are talents that allow them to DPS and maintain mana on their own.

Now for chat groupings. I think I might understand but people are probably confused about how mages and rogues communicate with each other in raid if they are spread across many groups. For this we have always designated Class Chat Channels for use of the class. We usually make the name a 4-5 character guild abbreviation followed by the class name (examples for us InconWarlock, InconMage, InconRogue). All the members of each class need to do is /join InconWarlock for example. Blizzard is kind enough to index your general and named channels (general, trade, local defense, world defense) and each character can be a member of 9 channels so if you wanted you could even extend this to class channels like InconMelee InconRanged, InconHealer so that each raid role could talk amongst themselves. To access these once they are created typing out /InconWarlock and hitting the space will translate to the channel and people will just need to look at the channel number so that in the future they can just hit /7 or whatever to access the channel. These are also password protectable so that you can moderate them within each class and role and keep outsiders from randomly stumbling across your channels. If you set this up now during a raid your people have these varying chat channels to have their conversations with each other to ensure they are doing the right thing or working together most efficiently. It frees up Raid chat for Raid matters. Keeps party specific, Class specific, and Role specific chat in condensed places.

Sorry for the long response but I hope that helps and I hope I covered and understood both you questions.