Even as someone who loves fantasy novels, I've come to recognize that one of the genre's greatest appeals - magic - can quickly become its most crippling weakness when put in the hands of the wrong author. Sometimes the system used is so generic that any sense of wonder one might experience at uncovering a new world is gone; other times, the characters' actions seem meaningless or nonsensical because virtually any problem they come across could theoretically have a simple, magic-based solution.
Brandon Sanderson, the author of books and series such as Mistborn and The Stormlight Archive, has long been one of my favorite authors in regards to incorporating fantastical elements that take his readers to an entirely new setting while still allowing them to suspend disbelief and feel grounded in his works. So when I heard that he had developed a set of three guidelines for creating believable magic systems, I knew I wanted to start constructing the physical laws of my novel's world based on his suggestions.
Sanderson's First Law is that "An author's ability to solve conflict with magic is DIRECTLY PROPORTIONAL to how well the reader understands said magic." He posits that magic systems exist on a spectrum between soft and hard. Worlds based on soft magic systems don't define their rules as clearly as those in hard systems, and readers are not given as clear an understanding of how or what makes the fantastical occurrences that happen in the book possible. Though this can certainly be fun, it also limits the degree to which it is feasible for characters to overcome their struggles through magical means without it feeling too convenient or forced. He suggests that writers choose their place on the spectrum based on the mood they want for their intended work.
In contemplating what I want for my novel, I realized that one of the things that have been holding me back in the drafting is that even though I have a general idea of I want my rules to be, I find it difficult to write without knowing what my setting is or isn't capable of. As such, I decided that I would try to tend towards the hard magic end of the spectrum, keeping some soft magic attributes to add the element of mystery that Sanderson implies it can lend to a potential narrative. My story (at least as I currently envision it) takes place in a society on the cusp of major technological and social change, but I want to place my characters in conflict with the fear of the unknown, working with a magic system that has rules but not necessarily ones they understand particularly well.
With this established, I moved on to Sanderson's Second Law: "Limitations > Powers", by which he means that what characters can do with magic is often much less engaging than what they can't - that having things come too easily to a protagonist makes for a far less interesting story than one in which they have to genuinely work and think to solve a problem. He proposes looking at three different aspects of this dynamic - "limitations" (what magic is incapable of), "weaknesses" (what magic is at the mercy of), and "costs" (what the character has to give up to use the magic).
This bit of advice definitely proved more challenging to contemplate. I realized that I hadn't done nearly enough when it came to thinking about magic in terms of where it falls short and letting the needs of the plot overrule the needs of the system. My initial plan had been to work with a fairly typical fantasy "cost" dynamic and have the use of magic involve some kind of exertion that left the individual drained afterward, but I realized that I could build a much more compelling story if I came up with a more disturbing twist that fits in better with the ominous atmosphere I hope to create.
Finally, Sanderson's third law warns aspiring authors to, "Expand on what you have before building something new," which I interpret as the writer's equivalent of "Keep it simple, stupid!" What he means by this is that it's better to take a simple premise for your system and explore all the ways it can shape the setting and interactions between characters than by continually adding new components to the system that don't fit in well with each other.
Fortunately, I feel I have this one down pat in attempting to lay down at least a rough set of guidelines for myself before I sit down to start "officially" writing so that I don't end up haphazardly adding details in a way that makes the story less cohesive. (Though if I had been truly working in the spirit of the third law, I would have written this article before the one about mapmaking.)
I may not have fleshed out the system in enough detail that I feel comfortable moving on just yet, but reading through Sanderson's explanations helped me re-conceptualize the direction I should take in designing a compelling world.