But under the VLC’s GPL, we need to publish the whole source codes, including the "copy protection/content security" mechanism part, right?
If you want to be 100% legit with everything you do, yes, you must publish the source code. But believe me, nobody is going to pursue litigation against you if you don't. Even companies like SonyBMG have included pieces of GPL code in their software, and it didn't hurt them.
So, if I was in your shoes, I would just modify VLC to suit my purposes (remove record button, add the code for the decryption etc) and then put it my DVDs without publishing the source code.
The real problem is that standalone DVD players will not be able to play the disc. So when Average Joe puts the disc in his standalone player and nothing happens, he will be disapointed.
And the really big problem is that when the next version of Windows rolls out, your software may not be compatible with that version. The same way some old games/edutainment titles for Windows XP don't play on Windows 7 pcs. Same for Linux and MacOS.
So, everytime a new version of Windows, Linux of MacOS rolls out, you will have to ship a new disc to your customers which will include the latest (modified) version of VLC.
PS: For the record, this has been done in the past with the so called "
Movie CDs". These CDs contained video files in a proprietary format, and a special software player inside the disc automatically launched under Windows 95 and played the files. Sounds really similar with what you are intending to do. Needless to say, the software doesn't work in Windows XP/Vista/Seven pcs, so the discs are pretty useless now.
This is why nowadays content is always shipped in some standarized format (ex VideoCD, DVD-Video) and companies try to prevent copying by inserting errors in the non-playable parts of the disc (in segments of the video that are never accesed by the menus). See sony arccos and macrovision ripguard.