Topfield *.rec Files

[The DJ]

Thank you for the upload.

This turned out to be a very simple change. Basically it's just MPEG-TS with a GIGANTIC (up to 2000 Bytes) private custom header in front of it. Normally VLC only evaluates the first 600 bytes to make sure it has a MPEG-TS file, which is of course too few in this case.

We now take the ID of the Topfield recording as enough insurance that this actually IS a mpeg-ts file, and it just works.

I also tried to support some of the fields in the header, but it doesn't seem very well designed (rather i call it a 14 years old kids hackjob). The documentation that they themselves made available is incorrect, incomplete and model specific (not even portable). I did my best, but after a couple of hours I decided that the little info that was there, didn't warrant anymore wasting of my time.

[Penguin]

It is true that header in beginning of Topfield/Procaster recordings is not portable. How ever it is not very complicated to parse for reading. My Topfield 5x00PVRt compatible Procaster VF PVR5102T saves 3760 bytes of proprietary header in front of transport stream. It can also play *.rec files saved with MPEG Streamclip having only 1316 bytes long Topfield header.

I have made some headers available in http://www.saunalahti.fi/~edelman/topfield/

You may read the information from headers with a short perl script I have written (originally for my personal use) to read and modify the Topfield header. That script (http://www.saunalahti.fi/edelman/recheader.pl) may help you understand Topfield headers. I am sorry the script does not have a proper documentation.

I hope these will help you develop better support for Topfield *.rec files.

[nwhitfield]

That's great news; thanks very much.

Now I can do some more work on my cunning plan, automatically archiving recordings from the Topfield onto the network.

Then all I have to do is set something up so they can be made available as VOD streams and I can view them wherever I want.

Nigel.

[Guest]

Thats great now I hope to be able to offload my .rec files to a main server and the rest of the family can watch them without me running them through Projectx.
I've just had a look at the 0.8.5-test3 beta release notes and I didn't see anything about .rec file support.
Any idea when this will be available?

Regards Gary

[Garyvx]

Sorry I forgot to login, the "Guest" post above was from me.

Thats great now I hope to be able to offload my .rec files to a main server and the rest of the family can watch them without me running them through Projectx.
I've just had a look at the 0.8.5-test3 beta release notes and I didn't see anything about .rec file support.
Any idea when this will be available?

Regards Gary

[louisjj]

I wish to buy a TF5500PVR. Can someone upload a Topfield registration (.rec file) please?
I have tried invain to download Topfield.rec.
I wish to see how conversion from .rec to mpeg works.
Thanks.

[nwhitfield]

The best way to find out is probably to download a copy of a tool like MPEG Streamclip from http://www.squared5.com

This can open MPEG transport and program streams, and one of the menu options is to save the file with a .rec header, which is compatible with the Topfield PVRs.

It won't be as complete a header as something recorded on the boxes, obviously, since it won't have the details of the programme, channel, and so on, but it will still play and should give you some idea of the format and how easy the conversion is.

Streamclip works on Windows and Mac; for other systems, Project X will do the trick.

Nigel.

[theJ0ker]

Hi, I have recently purchased a Topfield TF5800 which creates these .rec files, and I have found a problem with transcoding them to an iPod compatible format. I'm using VLC 0.8.6 which plays the files with no trouble, but when I try to transcode the stream to mp4 or h.264 video at 320 x 240 resolution, 128 bit 2 channel mp4 audio, it reliably fails on large files. It starts off OK, but about 1/2 an hour into it, the audio continues OK, but the video is about 20 minutes out of sync (it skips a large chunk of video and shows video from later in the stream). I can upload a sample file if required. The transcoded file plays incorrectly in VLC and on the iPod, and the original .rec file plays OK in VLC. Here are the settings I used:
%VLC_PATH% %FILE_TO_CONVERT% :sout="#transcode{vcodec=mp4v, vb=512, acodec=mp4a, ab=128, channels=2, audio-sync}:std{access=file, mux=mp4,url=%IPOD_COMPATIBLE_FILE%}" --sout-transcode-width=320 --sout-transcode-height=240 --aspect-ratio=width:height vlc:quit

(I also tried vcodec=h264 but with the same results)

[VLC_help]

Have you demuxed the file with projectX (and mux it together) and tried again?

[ragtime2]

We now take the ID of the Topfield recording as enough insurance that this actually IS a mpeg-ts file, and it just works.

I guess that's the reason why VLC does not play/recognize transport streams recorded with the Topfield TF4000PVR series!

At least the first bytes of the header are different... not starting with "TFrc5000" as for the 50xx models, but with some strange (but fixed?) numbers. Anyhow, maybe it's a better idea to extend the scan buffer to more than the mentioned 600bytes, since after the header part (3 packets?) a normal MPEG-TS is starting, AFAIK...?

Would be nice if this small issue could be fixed somehow in future releases... I uploaded a small sample recording to: ftp://streams.videolan.org/incoming/Top ... ertest.tts

[KingTutt]

We now take the ID of the Topfield recording as enough insurance that this actually IS a mpeg-ts file, and it just works.

I guess that's the reason why VLC does not play/recognize transport streams recorded with the Topfield TF4000PVR series!

I've got a Topfield 7700 HCCI and the .rec files cannot be played with VLC 1.0.3 as well (mplayer works fine). I always get the following error:
main error: option sub-original-fps does not exist
main error: no suitable demux module for `/subtitle://C:\Video.idx'

In another thread viewtopic.php?f=7&t=56114&p=185963&hili ... ld#p185963 I've seen that the problem seems to be a"different header in each receiver model". Unfortunately I haven't got any technical specifications

Is there anything else I can do that TF 7700 HCCI or other models are supported by VLC on day (perhaps providing some recorded examples)?

[thannoy]

Using command_line to read your file, you can try this (not tested):

Code: Select all

vlc file/ts://myrecord.rec

If I am not wrong, it will force TS demuxer to be used (Topfield TV records are MPEG-TS files with a header)
You can also try to edit the file using a hex editor and remove the header :-). You have to find regular 0x47 bytes every 188 bytes. The header to remove is rougly 1000 to 1300 bytes long iirc and the result file should start with 0x47.

[KingTutt]

Using command_line to read your file, you can try this (not tested):

Code: Select all

vlc file/ts://myrecord.rec

Unfortunately the command_line doesn't work for me, too.

For removing the the header using an hex editor would imply, that the headers of TF-Recorders always have got a fixed length. I found the regular 0x47 bytes every 188 bytes, but I did not get a vlc working file when I cut the 188byte pices away until a length between 1000 to 1300 bytes.

What I does not understand: The FAQ on Topfield Homepage says:

To play a rec file on your PC take vlc player Version 0.8.5 or above

I think that Topfield should know, that this does only work with Series 4000 Players.


I'm no expert concerning MPEG-TS files. If the proprietary header of TF-Recorders has always got the same length wouldn't it be a possible solution to just ignore the amount of bytes when playing an rec-file?
Another question would be how mplayer solves that problem. Does mplayer simply ignore all headers?

[VLC_help]

If the proprietary header of TF-Recorders has always got the same length wouldn't it be a possible solution to just ignore the amount of bytes when playing an rec-file?

Yes. But you would also have to know when the file has problematic header and when not.

[KingTutt]

I've contacted the Topfield support in order to get some more informations. The first problem is, that I'm no expert concerning video streams and codecs and the second one the 1st level support ...
According to my request I got the answer,

that I would have to install a proper codec to get vlc playing the stream.

As I mentioned, that vlc already provides all necessary codecs the support said,

that I will need an H264 Codec in order to view HD-Recordings with vlc and that the header is in the index file (.idx). I should use vlc only with the .rec file, themselves were able to play recordings using vlc.

First of all if I remember correctly vlc can play H264 without using an extra codec, furthermore my recordings were simple SD-recordings.
Second, if the header is only in the .idx file, why do I have such problems playing the .rec files?

But you would also have to know when the file has problematic header and when not.

What do you think when saying "problematic header". Is the Topfield-header repeated continuously during the TS-Stream or available only once at the beginning of a recording? Perhaps anyone can clarify the problem a little bit for me

Supposing the proprietary Topfield header is in the idx-file as mentioned by the TF support. What are the problems when trying to play rec-files using vlc, the mplayer seems not to have that problems? Are there proprietary informations in the rec file, too?
Perhaps anyone of the programmes expert could precise what informations would be helpful under the viewpoint that Topfield will not give all detailed specs to public. Is there an abstract layer which TF could provide the vlc developers in order to safe their "know how" AND to help fixing the reported problems playing .rec files?

At the moment I've got the problem that I would like to help the topfield community (rec-file support in vlc) and the vlc community (improving vlc) but I don't know where to start with my point of knowledge.

[VLC_help]

Well the issue only bother certain Topfield models, for example .rec files from my TF 5100 PVRc HDMI work in VLC.

What do you think when saying "problematic header".

Topfield boxes insert extra header to file, where some info is stored, so boxes can display it after recording. But it seems this header isn't TS compliant and because of that, VLC doesn't identify the file correctly. I think Mplayer is doing longer lookahead and finds the actual starting point of original TS section.

[thannoy]

VLC relies on first four bytes of the media to detect TopField files. They must be "TFrc" for demux/ts.c to properly detect it, in which case it outputs "this is a topfield file" in debugs.

If it do not find this magic code, it will try to find 0x47 bytes in the beginning of the file. The first 0x47 must be in the first 204 bytes followed by 2 other 0x47 separated by 188 bytes each (or one of the other TS allowed packet size). If TS demuxer is forced and 0x47 sync is not found, then TS accept to open the file using 188 bytes packets.

To sum up, if your file do not start with TFrc, I suggest you to force the demuxer. I think using "file/ts://filename.rec" but I am not sure.
If the first 0x47 found if not a sync byte (not followed by others 0x47 after 188 bytes), TS demuxer need to be forced to accept the file.

To help more, two informations are required:
1- What are the first bytes of your .rec file? (first 10 bytes in hexadecimal for example)
2- does the first 0x47 of you file belongs to sync bytes?
3- What does the verbose VLC output says about this file? (please use "Code" button to paste them)
3.1 opening normal way, opening with file/ts://, opening with "--demux=ts"

[KingTutt]

Thank you for the clarifications

The file does not start with TFrc. The required informations (using vlc 1.0.3) are:

1- What are the first bytes of your .rec file? (first 10 bytes in hexadecimal for example)

Code: Select all

47 05 79 2d b7 10 1b 87 2a b4

2- does the first 0x47 of you file belongs to sync bytes?

How can I find that out? The first bytes are posted above. The next 6 blocks (always 188 bytes later) starts with

Code: Select all

47 05 79 1e 8c ... 47 45 79 3f 07 ... 47 05 79 10 ba ... 47 05 79 11 80 ... 47 05 7c 1f 02 ... 47 05 79 12 00

3.1 verbose VLC output (opening normal way)

Code: Select all

C:\Programme\VLC>vlc --intf=logger --verbose=2 file:///"test.rec" -- logger module started -- main debug: using interface module "logger" main debug: TIMER module_need() : 3.000 ms - Total 3.000 ms / 1 intvls (Avg 3.000 ms) main debug: thread (interface) created at priority 0 (../.././src/interface/interface.c:151) main debug: processing request item null node Wiedergabeliste skip 0 main debug: thread started main debug: rebuilding array of current - root Wiedergabeliste main debug: rebuild done - 1 items, index -1 main debug: starting new item main debug: creating new input thread main debug: Creating an input for 'test.rec' main debug: thread ended main debug: thread (input) created at priority 1 (../.././src/input/input.c:230) main debug: thread started main debug: using timeshift granularity of 50 MBytes main debug: using timeshift path 'C:\DOKUME~1\user\LOKALE~1\Temp' main debug: `file:///test.rec' gives access `file' demux `' path `test.rec' main debug: creating demux: access='file' demux='' path='test.rec' main debug: looking for access_demux module: 1 candidate main warning: no access_demux module matching "file" could be loaded main debug: TIMER module_need() : 0.000 ms - Total 0.000 ms / 1 intvls (Avg 0.000 ms) main debug: creating access 'file' path='test.rec' main debug: looking for access module: 2 candidates access_file debug: opening file `test.rec' main debug: using access module "access_file" main debug: TIMER module_need() : 1.000 ms - Total 1.000 ms / 1 intvls (Avg 1.000 ms) main debug: Using AStream*Stream main debug: pre buffering main debug: received first data after 0 ms main debug: pre-buffering done 1024 bytes in 0s - 1000000 kbytes/s main debug: looking for stream_filter module: 2 candidates main debug: TIMER module_need() : 0.000 ms - Total 0.000 ms / 1 intvls (Avg 0.000 ms) main debug: looking for stream_filter module: 1 candidate main debug: using stream_filter module "stream_filter_record" main debug: TIMER module_need() : 0.000 ms - Total 0.000 ms / 1 intvls (Avg 0.000 ms) main debug: creating demux: access='file' demux='' path='test.rec' main debug: looking for demux module: 50 candidates main debug: using demux module "ts" main debug: TIMER module_need() : 11.000 ms - Total 11.000 ms / 1 intvls (Avg 11.000 ms) main debug: looking for a subtitle file in C:\\ ts debug: DEMUX_SET_GROUP 0 0 main debug: `file:///test.rec' successfully opened ts debug: pid[1401] unknown ts debug: pid[1404] unknown ts debug: pid[1402] unknown ts debug: pid[5960] unknown ts debug: pid[1403] unknown ts debug: pid[5961] unknown ts debug: pid[1409] unknown ts debug: pid[101] unknown ts debug: eof ? main debug: EOF reached ts debug: pid list: main debug: finished input ts debug: - pid[101] seen ts debug: - pid[1401] seen ts debug: - pid[1402] seen ts debug: - pid[1403] seen ts debug: - pid[1404] seen ts debug: - pid[1409] seen ts debug: - pid[5960] seen ts debug: - pid[5961] seen ts debug: - pid[8191] seen main debug: removing module "ts" main debug: removing module "stream_filter_record" main debug: removing module "access_file" main debug: thread ended main debug: dead input main debug: thread times: real 0m0.171875s, kernel 0m0.062500s, user 0m0.093750s main debug: TIMER input launching for 'test.rec' : 21.000 ms - Total 21.000 ms / 1 intvls (Avg 21.000 ms) main debug: changing item without a request (current 0/1) main debug: nothing to play

3.2 verbose VLC output (opening with file/ts://)

Code: Select all

C:\Programme\VLC>vlc --intf=logger --verbose=2 file/ts:///"test.rec" -- logger module started -- main debug: using interface module "logger" main debug: TIMER module_need() : 8.000 ms - Total 8.000 ms / 1 intvls (Avg 8.000 ms) main debug: thread (interface) created at priority 0 (../.././src/interface/interface.c:151) main debug: processing request item null node Wiedergabeliste skip 0 main debug: thread started main debug: rebuilding array of current - root Wiedergabeliste main debug: rebuild done - 1 items, index -1 main debug: starting new item main debug: creating new input thread main debug: Creating an input for 'test.rec' main debug: thread ended main debug: thread started main debug: thread (input) created at priority 1 (../.././src/input/input.c:230) main debug: using timeshift granularity of 50 MBytes main debug: using timeshift path 'C:\DOKUME~1\user\LOKALE~1\Temp' main debug: `file/ts:///test.rec' gives access `file' demux `ts' path `test.rec' main debug: creating demux: access='file' demux='ts' path='test.rec' main debug: looking for access_demux module: 1 candidate main warning: no access_demux module matching "file" could be loaded main debug: TIMER module_need() : 0.000 ms - Total 0.000 ms / 1 intvls (Avg 0.000 ms) main debug: creating access 'file' path='test.rec' main debug: looking for access module: 2 candidates access_file debug: opening file `test.rec' main debug: using access module "access_file" main debug: TIMER module_need() : 1.000 ms - Total 1.000 ms / 1 intvls (Avg 1.000 ms) main debug: Using AStream*Stream main debug: pre buffering main debug: received first data after 0 ms main debug: pre-buffering done 1024 bytes in 0s - 1000000 kbytes/s main debug: looking for stream_filter module: 2 candidates main debug: TIMER module_need() : 1.000 ms - Total 1.000 ms / 1 intvls (Avg 1.000 ms) main debug: looking for stream_filter module: 1 candidate main debug: using stream_filter module "stream_filter_record" main debug: TIMER module_need() : 0.000 ms - Total 0.000 ms / 1 intvls (Avg 0.000 ms) main debug: creating demux: access='file' demux='ts' path='test.rec' main debug: looking for demux module: 1 candidate main debug: using demux module "ts" main debug: TIMER module_need() : 1.000 ms - Total 1.000 ms / 1 intvls (Avg 1.000 ms) main debug: looking for a subtitle file in C:\\ ts debug: DEMUX_SET_GROUP 0 0 main debug: `file/ts:///test.rec' successfully opened ts debug: pid[1401] unknown ts debug: pid[1404] unknown ts debug: pid[1402] unknown ts debug: pid[5960] unknown ts debug: pid[1403] unknown ts debug: pid[5961] unknown ts debug: pid[1409] unknown ts debug: pid[101] unknown ts debug: eof ? main debug: EOF reached main debug: finished input ts debug: pid list: ts debug: - pid[101] seen ts debug: - pid[1401] seen ts debug: - pid[1402] seen ts debug: - pid[1403] seen ts debug: - pid[1404] seen ts debug: - pid[1409] seen ts debug: - pid[5960] seen ts debug: - pid[5961] seen ts debug: - pid[8191] seen main debug: removing module "ts" main debug: removing module "stream_filter_record" main debug: removing module "access_file" main debug: thread ended main debug: dead input main debug: thread times: real 0m0.250000s, kernel 0m0.140625s, user 0m0.109375s main debug: TIMER input launching for 'test.rec' : 11.000 ms - Total 11.000 ms / 1 intvls (Avg 11.000 ms) main debug: changing item without a request (current 0/1) main debug: nothing to play

3.3 verbose VLC output (opening with "--demux=ts")

Code: Select all

C:\Programme\VLC>vlc --intf=logger --verbose=2 --demux=ts file:///"test.rec" -- logger module started -- main debug: using interface module "logger" main debug: TIMER module_need() : 6.000 ms - Total 6.000 ms / 1 intvls (Avg 6.000 ms) main debug: thread (interface) created at priority 0 (../.././src/interface/interface.c:151) main debug: processing request item null node Wiedergabeliste skip 0 main debug: thread started main debug: thread ended main debug: rebuilding array of current - root Wiedergabeliste main debug: rebuild done - 1 items, index -1 main debug: starting new item main debug: creating new input thread main debug: Creating an input for 'test.rec' main debug: thread (input) created at priority 1 (../.././src/input/input.c:230) main debug: thread started main debug: using timeshift granularity of 50 MBytes main debug: using timeshift path 'C:\DOKUME~1\user\LOKALE~1\Temp' main debug: `file:///test.rec' gives access `file' demux `' path `test.rec' main debug: enforced demux ` ts' main debug: creating demux: access='file' demux='ts' path='test.rec' main debug: looking for access_demux module: 1 candidate main warning: no access_demux module matching "file" could be loaded main debug: TIMER module_need() : 1.000 ms - Total 1.000 ms / 1 intvls (Avg 1.000 ms) main debug: creating access 'file' path='test.rec' main debug: looking for access module: 2 candidates access_file debug: opening file `test.rec' main debug: using access module "access_file" main debug: TIMER module_need() : 0.000 ms - Total 0.000 ms / 1 intvls (Avg 0.000 ms) main debug: Using AStream*Stream main debug: pre buffering main debug: received first data after 0 ms main debug: pre-buffering done 1024 bytes in 0s - 1000000 kbytes/s main debug: looking for stream_filter module: 2 candidates main debug: TIMER module_need() : 0.000 ms - Total 0.000 ms / 1 intvls (Avg 0.000 ms) main debug: looking for stream_filter module: 1 candidate main debug: using stream_filter module "stream_filter_record" main debug: TIMER module_need() : 0.000 ms - Total 0.000 ms / 1 intvls (Avg 0.000 ms) main debug: creating demux: access='file' demux='ts' path='test.rec' main debug: looking for demux module: 1 candidate main debug: using demux module "ts" main debug: TIMER module_need() : 2.000 ms - Total 2.000 ms / 1 intvls (Avg 2.000 ms) main debug: looking for a subtitle file in C:\\ ts debug: DEMUX_SET_GROUP 0 0 main debug: `file:///test.rec' successfully opened ts debug: pid[1401] unknown ts debug: pid[1404] unknown ts debug: pid[1402] unknown ts debug: pid[5960] unknown ts debug: pid[1403] unknown ts debug: pid[5961] unknown ts debug: pid[1409] unknown ts debug: pid[101] unknown ts debug: eof ? main debug: EOF reached ts debug: pid list: ts debug: - pid[101] seen main debug: finished input ts debug: - pid[1401] seen ts debug: - pid[1402] seen ts debug: - pid[1403] seen ts debug: - pid[1404] seen ts debug: - pid[1409] seen ts debug: - pid[5960] seen ts debug: - pid[5961] seen ts debug: - pid[8191] seen main debug: removing module "ts" main debug: removing module "stream_filter_record" main debug: removing module "access_file" main debug: thread ended main debug: dead input main debug: thread times: real 0m0.250000s, kernel 0m0.125000s, user 0m0.125000s main debug: TIMER input launching for 'test.rec' : 10.000 ms - Total 10.000 ms / 1 intvls (Avg 10.000 ms) main debug: changing item without a request (current 0/1) main debug: nothing to play

If it is helpful I can also provide the one minute recording example (43MB). If I remember correctly there is somewhere a possibility to upload it to an FTP-Server, right?

What has struck me:
The recordings made from RTL have got a regular beginning with

Code: Select all

47 05 79

each 188 bytes. The record I made from ZDF has got

Code: Select all

47 00 6e

and from ARD I got

Code: Select all

47 00 65

The 4th byte is counting one up each block in all recordings from

Code: Select all

10, 11, 12.... 1f, 10,11

Maybe the second and third byte is a Program specific number while the fourth is a kind of counter?

Thanks in advance for your help!

[thannoy]

Maybe the second and third byte is a Program specific number while the fourth is a kind of counter?

A kind of, yes. From MPEG TS standard.

I have not the time right now to decode hex bytes but informations you gave are precious, thanks. I will ask you one more test as a fast check of my (bad) hypothesis. (My guess is that your MPEG file do not contains PAT tables (kind of entry points, identified by 0x00), so VLC can't guess by itself that stream 101 is a PMT)
- Does your file contains hexa sequence "47 40 00"? often? My guess is no :(

@see http://en.wikipedia.org/wiki/MPEG_transport_stream#PAT to unserdtand why PAT packet is the entrypoint in MPEG decoding. For your information, I think VLC prior to 0.9 was able to play such out-of-standard files out of the box.

Using command-line, you can cope with it (using the "--ts-extra-pmt" parameter to build a PMT by hand, not so easy to use). I am surprised there is not yet a simpler parameter to simply tag a PID (101 here) as PMT.. An idea for a feature to add.

[KingTutt]

Does your file contains hexa sequence "47 40 00"? often? My guess is no

Yes, it does! In the one minute recording (43MB) about 50times [in another recording (141MB) about 151times]. The first time at

Code: Select all

8768D

but it is not regular, the second and third time is at

Code: Select all

16B687 ... 247D25

Using command-line, you can cope with it (using the "--ts-extra-pmt" parameter to build a PMT by hand, not so easy to use)

I tried to use this parameter, but in wiki MPEG-PS I did not understand the syntax (pmt_pid=pid:stream_type) correctly.
I tried

Code: Select all

--ts-extra-pmt 2=101:1 and --ts-extra-pmt 2=101:2

A random pmt_pid (2), pid 101 as mentioned by you, but I didn' know the "stream_type", so I tried just 1 and 2 without success.
Are there any further options for me to help?

Today I got from the Topfield support the following statement:

The files of 7700 HSCI and 7700 HCCI are nearly identical TS-Streams. The TF5000PVR has got the header in the REC-File, the TF7700-Series has got the header in the IDX-file.