I love rules which are both useful game rules (they provide a system for players to interact with that has risks and rewards) and explanations for why the game world is the way it is. That's why I really liked a recent post at Necropraxis on how a magic user goes about learning spells. It's really excellent - go read it.
My own spin on this system follows. It's really a tweaking, so I make no claim to great originality here. This is just how I would do it. My main design concern here is that I want players to take risks, or at least want them to want to take risks, and to sympathize with the NPC wizards who take those risks and get a bad result. I'm also not going to re-state everything that Benjamin made in his post, so you need to read them together to appreciate the full effect. Here it is again.
Default rules are given in D&D 5th Ed. terms. Alternative rules compatible with B/X games also given.
Means of Learning Spells
There are only two ways for a magic-user (or wizard, if your game has other magic-user types like sorcerers) to learn spells. You automatically learn two spells when you level up. You can pick one of them and the other is rolled randomly. (Roll first, then pick) The only other means to learn spells is experimentation and research.
Magic is called "the Art", but there's a logic and system to it as well. A magic-user must learn and master Arcanum, the language of magic, before casting his first cantrip. But learning to speak arcanum means that even the most puissant and rare spells can be understood by the competent wizard (at least in theory), for all magic is based on the vocabulary and grammar of power. For this reason, any magic-user may attempt to learn any spell of a level he can cast simply by taking a week to work it out. No special materials or tools are required. However, the odds of success when taking such an informal route are not good.
The Roll: Roll a d6 and apply the modifiers below. The total of your roll plus any modifiers must equal 6 or greater to succeed.
Mishap: On a result of -1 or less (after modifiers), a Risk is taken (see below) even if none was intended. 50% chance of a bad result. Roll d6 to determine which Risk is threatened.
One Chance Only: You may only attempt to learn a spell once. Each wizard player should keep a list of all possible spells in the game, and upon failing to learn a spell, that spell is crossed off forever and may never be learned without the intercession of a god or Wish.
Each of the below actions modifies the odds of success by the specified amount. There is no need to improve the odds of success beyond +5. It is possible to guarantee success on the Learn Spell roll - if you're willing to pay the price. Each modifier can only be applied once, except Risk, which can equal any number the player chooses - but see below.
Base Modifier [-3 to +5]: Your base modifier for learning a spell is your Proficiency Bonus - the Spell's Level.
Extended Study [+1]. Instead of taking 1 week, take 1 month per spell level. The magic-user cannot be interrupted more than one day out of the week during this time to get the benefit, so adventuring is out of the question. The magic-user must have comfortable accommodations (at least a private room, but a private tower in a remote forest is better to reduce interruptions) and at least one servant (to prepare meals and such) for the full length of time.
Rushed Study [-1]. Instead of taking one week, roll after one day.
Lab and Materials [+1]. Access to a wizard lab with at least 2,000 gp in equipment (this cannot be assembled quickly; players should assume a six month lead time to commission the necessary tools) and the use of rare and exotic materials in experiments, costing d4+spell level * 100 gp.
Magic Library [+1]. Have access to a library of magical lore. The library must have at least 1,000 gp in books per spell level of the spell you seek to learn. The spellbooks of magic-users other than yourself counts towards this total and are assumed to have a value of 500 gp each.
Written Example of Spell at Hand [+1]. You have a scroll or spellbook at hand that has a copy of the spell you wish to learn.
Assistance of Familiar [+1]. You have a Familiar, and it does you a favor.
Assistance of an Outsider [+1]. You invoke the true name of a demon, djinn, undead spirit, or similar and ask for his guidance in your research. He provides it, for a favor. See Benjamin's post for full details.
Magical Tutoring [+1]. Pay an experienced (at least 3 levels higher than you) wizard to collaborate on your research. The price will probably be steep and not denominated in cash.
Apprentice [-1 or +1]. An Apprentice is a Negative penalty during their first six months with you, after which they become a positive modifier. See Benjamin's post for more details on their personality. After 1d6+2 years an Apprentice becomes a Level 1 magic-user and leaves his master's service. [Note that an Apprentice and Master cannot assist each other at the same time, as assisting someone else's research counts as an interruption to your own research. This is why most apprentice magic-users leave their master's service as soon as they have mastered first level spells.]
Risk [+X]. By being reckless in your experimentation you increase the odds of finding a solution but cut corners on controlling the magic at hand. You can take as much risk as you like when researching as spell, but for each +1 value of the Risk you roll once on the below set of Risks. You cannot control which Risk you take. Each risk outlined below has a 50% chance of occurring. Roll separately for each risk taken.
1. Mark of the Beast. Your flesh is permanently altered. Permanent loss of -1 Chr and roll on this Table.
2. Addiction. You used mind-expanding drugs, and now you're addicted. Roll 4d6-Con. This is how many gold pieces you must spend each day (Minimum 1) to get your fix. -4 to all rolls if you go into withdrawal and casting spells has a 25% failure rate. Curing the addiction requires a Restoration spell and the permanent Level Drain of 1d2 class levels. It's possible more powerful magical healing can cure you without the loss of levels, but there is no firm information on that.
3. Explosion. As the Fireball spell, in your face. Damage type is Magic Flame. No Resistance can apply. Make a Saving Throw as normal. Damage is a Xd6, where X is the level of the spell you were trying to learn. Your Lab is a write-off, if applicable. So is your Library if you were dumb enough to keep it nearby.
4. Alignment Shift. Your alignment slips one step towards Chaotic Evil. If you are already Chaotic or Evil, the other half of your alignment slips a step. If you're neither Chaotic nor Evil, flip a coin to determine which half slips a step. If you're already Chaotic Evil, you go insane and your PC is forfeit. Hand your character sheet over to the DM and pick up 3d6.
5. Defilement. The very land around you becomes defiled and blighted. The exact range, power and nature of the defilement will vary by the type and level of spell being researched. If this is done in a populated area, 1d4 bystanders will be transformed into monsters. If performed near many corpses (such a graveyard or catacomb) the dead will rise. If done in a remote wilderness, the land becomes blighted and attracts fell beasts and magical creatures. Players should expect social and legal repercussions if they are connected with the harm done.
6. Create Monster. Perhaps you summoned a demon with the intent of making aid you, or a djinn who was supposed to give you answers. Or maybe you lost control of your magic and it turned a normal earthworm near your tower into a ravening beast 100 meters long. Or the corpse you'd been dissecting got up and jumped out the window. Regardless, you made something, and now it's wandering the world. Taking the time to stop it while it gained consciousness would have interrupted the spell research and caused the spell to be lost forever - by now it could be miles away. Normally the monster has a Challenge level equal to the level of the spell you sought to learn, but roll d20 - on a 2-4 its CR is doubled, and on a 1 its CR is tripled. Roll on the random encounter chart at the appropriate CR. The DM may specify to keep rolling until you get a result that is thematically appropriate for the character of the spell being researched.
[Note that the Created Monster is assumed to flee at its maximum movement rate for several days after its creation, until it finds a lair and can make sense of its circumstances, however that may not always be possible. Intelligent monsters are fully aware of who created them and under what circumstances. Whether intelligent or not, all reaction roles towards their creator are Hostile.]
The Odds of Success
To give a sense of what the odds of success of researching spells are, I have created the below tables. The first one spells out your Base Modifier for each character and spell level.
The second table shows your minimum guaranteed roll assuming you have access to a Lab, a Library, a Scroll or Spellbook with the spell, a Familiar, and the assistance of either a Tutor (if you're low level) or Apprentice (if you're higher). As you can see, if you assemble your resources and make friends, you can usually have pretty good odds of researching a spell.
The third table shows your minimum guarantees roll if you add Extended Research and Demon Summoning to the list of Modifiers. This is probably difficult for PCs to achieve often (as Extended Research takes you out of campaign play, and Demon Summoning is pretty dicey). But again, it shows that Risk isn't always required.
So why have these rules?
If it's so easy to guarantee success, or at least pretty good odds, why have these rules at all? Three reasons.
One, I like explaining why my campaign world is the way it is. Many of the monsters that roam its dungeons or sewers are the result of magical experiments gone wrong. These rules explain why so many wizards go mad. And so forth.
Two, I want to introduce some risk into learning spells. The primary risk I mean is the risk of failure and never being able to learn that spell. It's a defining moment in a character's development, and will shape their character for the rest of the campaign. Important rolls like that are memorable.
And three, I want players to be tempted. When they're rushed, or don't have access to a library and lab, and really, really need a spell - I want that option to always be there, tempting them. "Here you go, you can have exactly what you need, just roll the dice ..." Muahahahahahaha!!
1: I don't like allowing wizards to learn spells from scrolls and captured spell books. In practice, since I like to use wizards as NPC bad-guys a lot, I find this leads to PC wizards quickly learning every spell in the game.
2: What I've done here is make magic-users (both PCs and NPCs) nervous. If there's a spell that a magic-user desperately wants to learn, they will be tempted to do anything necessary to guarantee the odds of success. This sort of temptation is what leads to interesting role-playing, and great NPC villains.
3: I am proposing a change from Arnold K's system. Rather than the Familiar teaching you a spell, his favor only improves your odds here. The cost of the favor is the same though.