Just a little thing: who says it's invisible to your character?

You don't know that you're under a Great Curse and are precisely this many Madness Units from going mad, but surely you have noticed that you're doing things that take a toll on you. You get Limit primarily from:

Things that tweak your Limit Trigger

Violating a Defining or Major Imtimacy

Spending WP to shake off influences that otherwise would've worked.

It's absolutely in-character to think "y'know, maybe I *should* start accepting the points my debate partners...