I also encountered this problem, I found that libvlc 3.0.11 version does not have this problem, but the 3.0.12 and newer versions have this problem.
3.0.11 version play so fast with buffered 6 seconde,but 3.0.11 and newer versions buffer 30 seconds with the same play options.