Page 1 of 1

Setting custom HTTP headers inside SD scripts

Posted: 05 Jul 2018 09:45
by ale5000
Previously I was appending the custom headers to the User-Agent but it was blocked in VLC.
My script strictly depend from custom HTTP headers, is there a new way to set custom headers?

Re: Setting custom HTTP headers inside SD scripts

Posted: 05 Jul 2018 09:52
by Jean-Baptiste Kempf
Previously I was appending the custom headers to the User-Agent but it was blocked in VLC.
My script strictly depend from custom HTTP headers, is there a new way to set custom headers?
Ale5000, you're back???

Useragent, yes; custom headers, no.

Re: Setting custom HTTP headers inside SD scripts

Posted: 05 Jul 2018 10:43
by ale5000
Useragent, yes; custom headers, no.
The Rai streaming server don't work at all without setting authentication in the custom HTTP header.
Is there any alternative to fix my script?

A lot of work was needed for creating the Rai TV script and it is frustrating that it cannot work for a minimal thing.

Edit: It would be nice to be able to set custom headers in a way like this: http://w3.impa.br/~diego/software/luasocket/http.html

Re: Setting custom HTTP headers inside SD scripts

Posted: 05 Jul 2018 17:48
by ale5000
Ale5000, you're back???
I haven't too much free time but yes :)
How are you?

Re: Setting custom HTTP headers inside SD scripts

Posted: 05 Jul 2018 19:19
by Jean-Baptiste Kempf
Ale5000, you're back???
I haven't too much free time but yes :)
How are you?
Great! We've made sooo much progress on subtitles :)

Re: Setting custom HTTP headers inside SD scripts

Posted: 06 Jul 2018 14:07
by mederi
Edit: It would be nice to be able to set custom headers in a way like this: http://w3.impa.br/~diego/software/luasocket/http.html
Check following topic: https://forum.videolan.org/viewtopic.php?f=29&t=109033

Re: Setting custom HTTP headers inside SD scripts

Posted: 06 Jul 2018 20:49
by Rémi Denis-Courmont
Previously I was appending the custom headers to the User-Agent but it was blocked in VLC.
My script strictly depend from custom HTTP headers, is there a new way to set custom headers?
Yes, send your custom HTTP request directly over sockets. The low-level HTTP request API has been not exposed to Lua.

Re: Setting custom HTTP headers inside SD scripts

Posted: 09 Jul 2018 08:25
by ale5000
Previously I was appending the custom headers to the User-Agent but it was blocked in VLC.
My script strictly depend from custom HTTP headers, is there a new way to set custom headers?
Yes, send your custom HTTP request directly over sockets. The low-level HTTP request API has been not exposed to Lua.
Well, with some efforts maybe I can do this but then I will receive the streaming data directly from the live streaming; how can I make VLC play data directly?

Is there a way to pipe them directly to VLC?
I cannot use a temporary file since it is a live streaming and grow indefinitely and also it is ugly to use it.

Re: Setting custom HTTP headers inside SD scripts

Posted: 09 Jul 2018 18:52
by Rémi Denis-Courmont
Lua scripts can only generate playlist items, not streams. You may need to write a custom native plugin for that specific use case - but on the bright side, a native plugin might be able to use the low-level HTTP request API.

Re: Setting custom HTTP headers inside SD scripts

Posted: 09 Jul 2018 20:33
by ale5000
Lua scripts can only generate playlist items, not streams. You may need to write a custom native plugin for that specific use case - but on the bright side, a native plugin might be able to use the low-level HTTP request API.
Isn't there any plan to extend the functionality of LUA Addons?
They are nice but there are a lot of limitations.

Workaround this limitation will require a lot of hacks :(

Re: Setting custom HTTP headers inside SD scripts

Posted: 10 Jul 2018 17:54
by Rémi Denis-Courmont
I don't think Lua should be allowed to generate actual streams. Performance would most certainly suck.

Re: Setting custom HTTP headers inside SD scripts

Posted: 10 Jul 2018 20:48
by ale5000
I don't think Lua should be allowed to generate actual streams. Performance would most certainly suck.
Well, this is just the last resort until someone add the feature to pass custom headers (that shouldn't be hard, I don't need to set cookies or things where security can be a problem but just custom non-standard headers "viaurl" and "ttAuth", maybe also others to simulate a real browser but they aren't really important).

Obviously I wouldn't pass data directly if it was possible, but this option is useful for edge cases, one can also implement a full demuxer in Lua in a short time (until a proper one in C is implemented).
In my "IIS Smooth Streaming parser" Lua script that I have written a lot of time ago I actually create a full container data from scratch in a temp file then appended the video data from the streaming (endlessly) and passed the link of the temp file to another VLC instance; it worked fine but the downside is that the temp file grow indefinitely until you kill the first VLC instance since the live stream never end.

For now I'm thinking of maybe having 2 instances of VLC; one with a Lua SD that pass a link like http://127.0.0.1 that point to a local proxy in Lua in another instance of VLC (where its only function is to create the request with custom headers and pass the data as is).

Re: Setting custom HTTP headers inside SD scripts

Posted: 11 Jul 2018 17:19
by Rémi Denis-Courmont
"Custom headers" will never happen. The high-level HTTP API cannot know how to handle unknown headers, by definition. You have to use the low-level HTTP API if you want to insert custom headers.

Re: Setting custom HTTP headers inside SD scripts

Posted: 11 Jul 2018 17:28
by ale5000
Custom headers shouldn't be handled at all (they should just be passed), it should simply reject them if they contain special characters like ' " : \r \n \0 (or to be safer allow only alphanumeric characters) under header field name and under header field value (name and value handled separately) and nothing bad can happen.
And also it could skip the ones already handled like: Connection, Cookie, User-Agent, etc.

What can happen in the worst case?
I don't see how it could cause problems.

Re: Setting custom HTTP headers inside SD scripts

Posted: 11 Jul 2018 20:05
by Rémi Denis-Courmont
Custom headers shouldn't be handled at all (they should just be passed), it should simply reject them if they contain special characters like ' " : \r \n \0 (or to be safer allow only alphanumeric characters) under header field name and under header field value (name and value handled separately) and nothing bad can happen.
And also it could skip the ones already handled like: Connection, Cookie, User-Agent, etc.

What can happen in the worst case?
1) A new version VLC supports the header line, and suddenly everything breaks.
2) The header line changes the server behaviour in a way that the high-level state machine does not handle.
3) It is unknown whether the header line is a connection or an end-to-end header line.
...
You cannot simply bypass a state machine and expects things to keep working.

You have to remove the state machine - use the low-level HTTP message API.

Re: Setting custom HTTP headers inside SD scripts

Posted: 12 Jul 2018 01:11
by ale5000
1) They are custom headers so they will never be supported;
2) The problem is that the missing of the header now cause changes in the server behaviour; it is always possible to have changes in the server behavior also without custom headers;
3) I don't really understand this point, I know only request headers and response headers; beside that it can be added in the C code an array (or another variable type) that tag the origin of every header, the ones set by VLC and the ones set by Lua so that in every step is known where the header originate.

So the worst thing that can happen is that the Lua script stop working which is what it do now, so it isn't really worse.

Re: Setting custom HTTP headers inside SD scripts

Posted: 12 Jul 2018 21:22
by Rémi Denis-Courmont
1) Begging the question.
2) Red herring.
3) If you don't even know, then how will code know?

Re: Setting custom HTTP headers inside SD scripts

Posted: 13 Jul 2018 09:47
by ale5000
Ale5000, you're back???
I haven't too much free time but yes :)
How are you?
Great! We've made sooo much progress on subtitles :)
I haven't yet had time to look at them directly but from the VLC changelog I see many new subtitles format supported and many interesting changes, great work :)

Re: Setting custom HTTP headers inside SD scripts

Posted: 13 Jul 2018 09:53
by ale5000
3) If you don't even know, then how will code know?

Quote from: http://freesoft.org/CIE/RFC/2068/143.htm
13.5.1 End-to-end and Hop-by-hop Headers
For the purpose of defining the behavior of caches and non-caching proxies, we divide HTTP headers into two categories:

End-to-end headers, which must be transmitted to the ultimate recipient of a request or response. End-to-end headers in responses must be stored as part of a cache entry and transmitted in any response formed from a cache entry.
Hop-by-hop headers, which are meaningful only for a single transport-level connection, and are not stored by caches or forwarded by proxies.
The following HTTP/1.1 headers are hop-by-hop headers:

Connection
Keep-Alive
Public
Proxy-Authenticate
Transfer-Encoding
Upgrade
All other headers defined by HTTP/1.1 are end-to-end headers.

Hop-by-hop headers introduced in future versions of HTTP MUST be listed in a Connection header, as described in section 14.10.

So, all custom headers are end-to-end headers, from the quote it doesn't seems hard to detect it.

Edit:
A possible implementation idea:

Code: Select all

my_item.set_custom_request_header(string name, string val, bool end_to_end_header=true)
Example:

Code: Select all

local element = { title=current_title, path=current_path } local my_item = vlc.sd.add_item(element) my_item.set_custom_request_header("MyCustomHeader1", "123", true) my_item.set_custom_request_header("MyCustomHeader2", "345", true)

Re: Setting custom HTTP headers inside SD scripts

Posted: 13 Jul 2018 09:58
by ale5000
Edit: It would be nice to be able to set custom headers in a way like this: http://w3.impa.br/~diego/software/luasocket/http.html
Check following topic: https://forum.videolan.org/viewtopic.php?f=29&t=109033
It may be a solution but it is too much problematic to load compiled code since it is OS dependent and the script will never be universal.

Re: Setting custom HTTP headers inside SD scripts

Posted: 13 Jul 2018 18:16
by Rémi Denis-Courmont
So, all custom headers are end-to-end headers, from the quote it doesn't seems hard to detect it.
Wrong. The specification only says that Connection, Keep-Alive, Public, Proxy-Authenticate, Transfer-Encoding, Upgrade are hop-by-hop, and other header lines from the specification are end-to-end.

That says nothing of custom headers.