Large Level Gaps Are Bad

Here's the problem. You have a level 9 party. They're probably fighting stuff somewhere around their power level. If you throw a level 1 character into that, they are both highly ineffective (anything with a save will be made, few spells, limited ability to contribute), and absurdly fragile (very low HP, lower saves, probably limited money for gear unless you give them level 9 equivalent treasure).

That combination just doesn't work in 3.5. An experienced player can maybe make it work by knowing all the tricks to get the most out of their tiny allotment of level 1 spells. A newbie will not, and one mistake will get them killed. They do not have a margin for error, and as a new player they need a margin for error more than anyone else.

As the DM, you'd have to avoid trying to kill that character. You can't even do things like throw fireballs around with your NPCs, because a single one anywhere near that character and it's dead.

You could tone things down instead, but then the level 9 characters will steamroll over everything and not be challenged. That's also bad.

Given that, your best bet is to start the new character at level 9.

Mentorship

You and the other players will have to help mentor your new player. That goes with a level 1 character, and it goes with a level 9 character. The main difference is that the level 9 character is going to have more abilities, and some margin for error (a single melee attack won't kill a level 9 character very often).

Your new player will need help building a character, picking gear, learning what their stats mean, when to use skills, and so on. You and the other players can help with that in the first couple of sessions, mostly by helping with character creation, answering questions, encouraging questions, and offering hints and suggestions.

Gradual Spellbook Additions

Since your new player wants to play a spellcaster, the real overwhelming thing is trying to master the spell system and the large spell list. That's where you can help.

Shrink the list. A lot. Start the player off with a pared down spell list, where you pick out a handful of essential spells. If the player wants to, let them look through the book and pick out a few more that sound interesting. Don't give them 100 spells to start. Use a much smaller number (under 30).

As the player starts to grasp things in future sessions, start adding spells. For a Wizard this is easy, as you can say what's in the spellbook initially, and you can introduce spells to add to it later. For a Divine Caster, you may just have to explain why you want him to focus on a smaller set of spells initially to get the hang of it, and that the other ones are available if he wants to use them in the future.

I know when a friend of mine played a caster for the first time (a Cleric, which is a great class), we sat down together and built a "typical daily memorization list". Those were the spells he'd prepare each day, normally. He could make additions or changes if events warranted it, but I built the first list with him and he didn't have to try to figure out what spells were must have on his own. It made trying to figure out what to cast at first a lot easier for him, because that list is much shorter.

For a spontaneous caster, you can handle it differently depending on the class. Something with a fixed list like Beguiler you can treat like a Divine Caster: shorten the available list up a bit initially, and then add spells back as the player starts to get the hang of it. For Sorcerer where you have to pick the spells you will use forever... well, I'd pick an initial list of spells, and then give the new player the option to swap them out later if he decides he wants something else. Sorcerers already have this option every even numbered level, but he will have missed some chances to do it by starting at level 9 and by not knowing what to pick initially. You can be a bit more flexible with the swapping rules by allowing one time swapping of any spell that he came into the game with, so he can customize his character once he understands how it works.

So Which Spellcaster Should He Be?

You mentioned spells, but not which class he's going to play. If he's already picked a class to play, great! If he hasn't picked a class yet, I have a couple of recommendations.

1. Beguiler - Spontaneous caster, so no preparation of spells at the start of the day is required. Preparation can often be a real stumbling block for new players, so this takes it out of the way. The fixed spell list is a bit limiting, but he won't have to choose which spells he knows, as he just gets all of the ones the class has access to. Also gets d6 HD and access to light armor, which makes them tougher than the Sorcerer (handy for a new player). A lot of skill points and a good skill list is handy too. This is a pretty solid "pick up and play" spellcaster that doesn't need much book keeping.
2. Cleric - Has to prepare spells, which may be an issue. But other than that, there's a lot to recommend here. It's easy to make a Cleric that's tough as nails (full plate, two good saves, and d8 HD go a long way!), their spell list is strong, they can be strong melee combatants, easy access to healing whenever you want it, and there's no spell book to manage.

Druid and Wizard are both great classes, but they're also harder to play and may not be an ideal choice for a new player. That's not to say he can't play them, because he can. He'll just need more help:

• Druids have to also manage an animal companion, and wild shape. The animal companion can be a plus, because at level 9 you can get something that's pretty tough (like a Brown Bear or Dire Wolf), but now there's more stats to know and more things to manage. Wild Shape exacerbates that problem even more by changing the player's stats themselves.
• Wizards not only have to prepare spells each day, but also have to keep track of which ones they're allowed to prepare due to what's in their spellbook. They're also very, very fragile if you don't know how to mitigate that, and the familiar is an extra thing to manage if you don't trade it out for something else.

My Rule of Thumb

There isn't an official rule of thumb. Mine (and it seems to be pretty common amongst players I know) is that the party should be the same level unless there is a good reason why they're not (someone spending XP on item creation feats for example). If a new player joins, I put him at the same XP as everyone else, and try to keep treasure from being too different (though the new person may start with a bit less, as I typically give new players standard wealth from the DMG).

Level is one of those things that can make someone horribly ineffective if you set them back too far. They can make that up by being more optimized than everyone else is, or know how to use their abilities more successfully, but the game in my experience just works better when party members don't have level gaps between them.

A small gap is one thing, but a level 1 new player in a level 9 party is just unworkable.

Yes, it is

By level 30 there are all sorts of epic tier powers available to players to regen health, have massive heals, avoid taking damage at all, or basically avoid death once per day. All of these powers are in there to compensate for the deadliness of the creatures that players will face at that point.

Contrariwise Different classes (generally by role) have different HP gains per level to reflect the nature of both their class and role. Defenders get a lot of HP per level (and often have a higher con as well) because they need it, its part of their roles. Likewise, the rest of the roles get less (striker down to controller) because the nature of the roles makes them less and less apt to be the focus of damage. There are of course some classes that tend to have more HP for a given role, such as the Warlord, because their class is more likely to be found on the front lines.

The Fix

Simply recalculate everyone's HP to where it should be by the rules. 4e is a very, very balanced system with a few exceptions (some dragon magazine content that never got erratta'd or a few classes that never quite worked out) and it's in your favor to keep things RAW vs. introducing house-rules that can cause unintended consequences rippling through the system.