This topic is old, but I'm having the problem as well. I've established the resolutions in my mugen.cfg and system.def are the same, and everything is spelled correctly. It may be related to my attempt to use an Open Type font, but I don't know for sure...
Here's what my font/options.def says:
[FNT v2]
fntversion = 2,00
name = "Makinas Scrap 5"
author = "unknown"
[Def]
Type = opentype
Size = 6,11
Spacing = 1,0
Offset = 0,0
File = Makinas-Scrap-5.otf
Blend = 1
The font is font/Makinas-Scrap-5.otf
I'll be worrying about size, spacing, and offset after the font is working (unless therein lies the problem)