I say thee NAY! There shall not be such thing as XP but onley the progression to thee skills and arts thou hast achieved!
To increase characteristics there shall be MAGICK! An the true magick I bespake of here be they magick of Creo!
clears throat
Ok, there is more than enough to gain Characteristics as well as virtues (and flaws) so there is absolutely no need to crack with the HR whip.
Virtues to power.
Some virtues can be gained - like some variant of the gift - if you do the right things. Like gaining an infernal contract can instill you with the Infernal Gift. You can also be introduced to a house mystery, like the Bjornaers Heartbeast. And then there are even more obscure ways, for example through Original Research1A or via Ancient Magic1B or other means as decided by the GM, especially in respect to mundane or physical virtues.
You might deem some seasons dedicated to positioning yourself better in the courts (and gaining experience in this or that skill) fitting to grant some virtues, such as title, or an Adventure just brings it with it. Other virtues, like a mages rank, do have specific requests what needs to be fulfilled and come automatic.
Creo Corpus/Mentem & Muto Corpus
Creo can be used to alter physical (corpus) and mental (mentem) statistics with Magic, but doing so in a semi-permanent way demands a pretty hefty spell. Reference the Tables in the core book!
Level 35: Increase one of an animal’s Characteristics by one point, to no more than one higher than the average score for that kind of animal.2
Level 30: Increase one of a person’s physical Characteristics by one point, to no more than 0.3a
Level 35: Level 35: Increase one of a person’s mental Characteristics by one point, to no more than +1. 4
There is all the guideline you need! No XP involved. There is actually no XP involved in the whole Ars Magica 5E, just experience in certain things, and you can't have experience in being beautiful or strong, you can only warp the body or mind to befit the picture. And keep in mind, that permanencya increases the magnitude of spells: Those levels are for spells that befit the "Personal, Momentary, and Individual"5 spells.
For a longer duration you normally would need to push the magnitude by 4 for Year, netting you at least a Level 55 ritualistic spell (due to beiong a level>=50 spell) to gain a personal +1 on one physical or mental stat for one year - which clearly is not a task easily done and is a full 5 points into the "must be a ritual" area. But boosting the power like this is not even necessary really:
There are explicitly spells under Creo Mentem, that raise a characteristic permanently while being ritualistic at a lower level. The Gift of Reason (CrMe35) explicitly says:
"R: Touch, D: Mom, T: Ind, Ritual
Permanently increases the target’s
Intelligence by 1 point, to no higher than 0.
(Base 30, +1 Touch)3b
It is permanent, and it has to be done as a Rite, and thus also demands 1 point of appropriate Vis per Magnitude6, so a sum of 7 of Creo and Mentem Vis in this case. This adds 7 botch dice - dangerous! - unless you manage to achieve calm conditions. Acquiring and preserving these is not the easiest task, but clearly "easier" than doing a rite that has a 1-year duration.
On the other hand, Muto Corpus can give minor abilities pretty easily:
Level 2: Change someone to give them a minor ability.7
For one year, adding 4 magnitudes makes this a spell of level 10 (3 magnitudes for 5, then 1 for 10), allowing it to be cast easily and spontaneously. However, what counts as a minor ability? The standard example is "Eyes of the cat", granting some sort of low light vision.
Using the level 10 base ("turn a human into a land animal") would allow adding drastic changes: virtues like Large, Small frame are well within the doable here, and 1-year duration would bring it to 30, so not even demanding a ritual spell.
Conclusion
What XP does he want to spend there at all? There are no unbound Experience Points in the system! And how to relate "XP" spent on increasing characteristics or virtues to experience in skills? No, that is not intended in the system, it would break the game. You can gain all those nifty characteristics by casting them upon yourself, while many virtues (and flaws!) are to be gained with roleplaying, research and yet again research!
You might tell him about some of the fundamental ideas behind Ars Magica's progression system:
You don't lift weight to become stronger and inflict more damage in the middle ages, you hone your skills in swordsmanship and fighting in armor to increase the damage you inflict. You might become stronger on the way, you might not, but that's a GM decision, as you also age while training. If you start as a child, then you do mayhaps get stronger, but at the same time, children characters do get different characteristics: a flat out -4 to 0 modifier on all of them8.
You don't become smarter because you study, you know better, thus it is raising your skills! Magic though allows you to alter your Mind and Body, so train your arts if you want to boost yourself.
Notes
a - the longest valid duration as of the basic rules unless some parts reference permanency is year5. This is a House Rule suggestion:
The GM might increase the chart by two+ steps. If this is done, I suggest to do this somewhat analog to the Arcane Connections9 table, adding Decade and Permanent/Infinite for +6 magnitudes from momentary.
The GM might at his own discretion add other steps between Decade and Permanent for balancing the rather large steps. Another idea to enforce the absolute rarity of such potent spells, would be to have the two+ extra steps each being worth 2+ magnitudes to step up, turning permanency into a +8+ magnitude adjustment (for a minimum of level 75 to alter Characteristics to +1).
Literature
1A - Houses of Hermes: True Lineages, p. 26-37, "Original Research"
1B - Ancient Magic (Preview), p6, "integrating ~"
2 - Core Rules 5th Ed, p 116, "Creo Animal Guidelines"
3a - Core Rules* 5th Ed, p 130, "Creo Corpus Guidelines"
3b - Core Rules* 5th Ed, p 130, "Creo Corpus Spells"
4 - Core Rules 5th Ed, p 148, "Creo Mentem Guidelines"
5 - Core Rules 5th Ed, p 111-115, "Chapter Nine - Spells"; here: p 112
6 - Core Rules 5th Ed, p 81: "Ritual Magic" & p83: "Vis must be used when casting Ritual spells, as noted above. [...] For every pawn of vis used, [...] roll an extra botch die if the casting roll is stress
and comes up a zero.[...] includes [...] Ritual spell."
7 - Core Rules 5th Ed, p 132, "Muto Corpus Guidelines"
8 - Core Rules 5th Ed, p 29, "Starting Character Age"
9 - Core Rules 5th Ed, p 84, "Arcane Connections"
I think that if your players say that they learn by doing then it is a good idea to listen to them. Otherwise you will just end up banging your head against a brick wall. The dynamic of every table playing a campaign is different and, as their DM, it is your role to set things up to make the game enjoyable - without neglecting your own sanity of course!
Here are some articles that might be helpful in finding some further guidance with this.
Keep it simple
It is important to give new players the support they need, e.g help them with their character creation so they don't feel overwhelmed. There are a lot of boxes to fill in a character sheet. As a DM it is important we do not forget this or assume others know what to us may seem "obvious".
There are more suggestions here:
Keeping it simple for new players
Learning rules, wait, what...
There are some great suggestions about how to manage this in the article below, but one thing I would suggest is: patience, and more patience. It can take some players much longer to learn rules than others. The 5e of D&D has some rule economy in it which is a glob-send! Still, for someone who has never played a tabletop RPG, it might feel quite daunting. So, your being patient is essential. As is suggested in the article, if you have other experienced members in your team, ask them to buddy up and help those who are newer. Another tip is to provide them with a cheat sheet/summary of the main mechanics and terminology.
Plenty more here:
How to play with friends who don't want to read the rules?
The homework
Yuck, even the word is enough to send crowds running for the hills. I personally enjoy it - you might too as a DM. But, not everyone is the same. In my experience as a DM, the only reasonable homework I expected was for spell-casters to read up on the next tier of spells, when they were about to go up a level and acquire a higher spell-slot. This saves a lot of time. They did not memorise them, but had a good idea what the the spell did. As DM, you can decide this with your players. Again, making these decisions collaboratively is important to have them on board.
There is more food for thought here:
How can I get my players to do extra “homework”?
Tic toc
If you find that after a fair few sessions you have still got one or two stragglers who for neither love or money want to put in the effort to learn the basics, then I would suggest using a timer. In the lower levels it may not make that much of a difference if someone forgets how to roll for initiative, or where their skills are on their sheet, or what a spell does, or how to roll a saving throw, or how to calculate crit damage, but later on it will make a big difference. They might have several attacks and a bonus attack, plus companions and minions, etc. If every player takes a long time to make decisions on their turn in combat at this stage, it can bring the game to a grinding, and boring, halt. Zzz.. so learning the essentials in the first few levels is key.
One of my favourite strategies is using a simple time limit, as a gentle nudge. I love visual aids so I tend to use a 1m or 2m sand timer which the players can see on the table. This encourages them to think ahead and be responsive. If the timer runs out, so does their turn. Next... It may feel mean the first time, but they soon become quick-thinkers after missing a turn.
There are some other excellent suggestions in this article:
How can I speed up combat?
Last, and not least: Session Zero
I would definitely recommend setting up a Session Zero to help the players learn some of the basic mechanics of the game. If you are DM for 5e for the first time, it will take some getting used to also. So this will be a win-win for all of you. I particularly suggest making it a collaborative experience.
See: What is a session 0?
Bring plenty of snacks!
Note: thanks to MikeQ for some of these suggestions.
Best Answer
There are two ways to deal with this that I have experience with. Which is better depends on your situation, but both are workable.
Ask for a list of houserules.
For me personally, I have a really big problem playing in a system where I can't know all of the rules up front. If I'm playing in a game where there are significant house rules, then I'd ask that they be written down before play, so I know what I'm getting into. I'd prefer that such a list was handed out before character creation, so I can know if I'm going to need to change around my character based on the houserules.
I can understand why asking for a list of rules might seem pushy, but it's important to know the rules of a game when you want to play it. If you don't want to ask for a written list, maybe just ask for an explanation of the houserules, rather than a written list.
Let it ride.
Sometimes, the kind of house rule that the GM is using isn't the kind of thing that you really need to worry about. For example, you say that your GM is going to use the 4th ed skill challenge rules. Practically speaking, a player doesn't have to know how those work. All the player needs to know is that they are making a bunch of skill checks to achieve their goal, and that their normal combat powers aren't going to be of much help. That kind of conflict resolution rule can be handled entirely behind the screen, and doesn't really affect how a player will play the game, or build their character. If you can be confident that the GM isn't going to suddenly change how your character works in the middle of the game, then it might not end up being a big deal that you don't know all of the rules that they're using.
The downside of this approach is that you can't really get attached to the mechanics of your character. This approach assumes that you can play your character narratively without being terribly worried if one of your abilities stops working the way you thought it would. If the mechanics of your character are important to you as a player, then this approach won't work well for you.