I am having the same issue in VLC for Android 3.4.4. It has always happened in all earlier versions that I tried. Seeking around with the bar causes it to jump forward, and it does seem worse with larger files and worse the further in the file.
Seeking to near the end causes the file to stop playing and also fail to play the next time it's played, I guess it seeks to an invalid position after the end of the file, and then stores that position for next time? It sometimes works to reset at the beginning if I try to start playing it twice, or play another file and then go back to the first file.
I didn't find a "fast seek" setting. I have tried toggling hardware acceleration, Audio Output, and time stretching settings and nothing gets better (may get worse).
It is strange that some shorter ogg files (say, ~3 minutes long) do seem to skip about ~2s ahead of where I seek, but others work fine with no incorrect seeking.
Here's a publically available ogg that has this issue:
https://ftp.openbsd.org/pub/OpenBSD/songs/song35.ogg
via
https://www.openbsd.org/lyrics.html#35