The issue here is primarily not Windows Installer at all, it's that Visual Studio setups don't offer IDE support for a lot of its functionality. In this case Visual Studio doesn't let you run custom actions in the UI sequence to validate serial numbers, which is why there is this:
http://support.microsoft.com/kb/253683/en-us
The masked edit template is a sanity check that the keystrokes are of the correct type.
If you want extra features in your setups, Stefan maintains an excellent comparison of tools here:
http://installsite.org/pages/en/msi/authoring.htm
and all I can say is that you're supposed to choose the tool that fits your requirements, not choose a tool and then find out that it doesn't do what you want it to.