Welcome Guest ( Log In | Register )

21 Pages  1 2 3 > » Bottom
Bump TopicReply to this topicRSS feed Start new topic Start Poll

Outline · [ Standard ] · Linear+

> YouTube Deep Packet Inspection, All HTTP connections being MITMed

rizvanrp
post May 1 2013, 04:00 AM, updated 5y ago

Getting Started
Group Icon
Networks & Broadband
Group: Elite
Posts: 166

Joined: Sep 2006



Hi all,

I'm experiencing some anomalies while streaming videos off YouTube on Unifi. For certain 'political' videos -- I've observed that the HTTP connection for the videoplayback stream to YT's local CDNs are being disrupted as follows :

1. Client video player makes a connection to the YT CDN
2. HTTP GET request is sent

There's a few different behaviors after this .. :

3a. HTTP 200 OK is received however it arrives 90 seconds later (should be instant) :
user posted image

3b. HTTP 200 OK is received instantly, first 1-4KB of video stream traffic is sent (allowing the YT player to show the first frame of the video with a timestamp of 0:00).. then no traffic is received for 90 seconds once again :
user posted image

There's a duplicate TCP ACK when the stream returns, did my ACK at packet #271 ever reach the CDN in the first place??

Further testing :

1. Using an unencrypted SOCKS proxy on a remote server + non standard TCP port results in the same behavior with packet loss between the client and SOCKS proxy server

2. Using an encrypted SSH tunnel to the same remote server results in absolutely no issues with viewing the videos

Sample videos :
http://www.youtube.com/watch?v=hHTz22bTBRw
http://www.youtube.com/watch?v=uVWxB4AWOxc

UPDATE :

I performed a simultaneous packet capture on both my client + remote server while encapsulating the HTTP connection via plaintext SOCKS. All the video payload packets were dropped en route back to my SOCKS client :

user posted image

Dafuq?

UPDATE 2 :

Confirming all plaintext HTTP connections on Unifi (and maybe Celcom + Maxis) are being man-in-the-middle'd and dropped if they contain blacklisted data.

UPDATE 3 :

Other sources confirming this .. (thanks wkkay):

https://plus.google.com/1013966581485225280...sts/ak6opfbDxwa

UPDATE 4 :
What we know :

i. The DPI isn't only being used to selectively block YouTube videos, however unencrypted Facebook pages belonging to certain parties are also being blocked. You can get around this by appending 'https://' to the Facebook URLs rather than trying to use 'http://'.

ii. The DPI is based on TCP segment analysis. Basically, every single TCP packet has its payload analyzed for certain request URI strings that have been blacklisted. Obfuscation attacks such as packet fragmentation (splitting a large TCP payload containing a single HTTP request into smaller TCP segments) as well as packet padding (appending large amount of junk data to the HTTP request URI in order to force the 'HTTP/1.1\r\n' trailer into a separate TCP segment) will also work however you need specialized HTTP proxy software or iptables rules (on Linux) to do this.

iii. Once a blacklisted payload is detected within a packet, the header information for the TCP stream (SRC/DST port + SRC/DST IP address) is added to some kind of blacklist for 90 seconds. This causes all traffic for that particular TCP stream to be dropped for 90 seconds (hence the 90 second gaps in my packet capture samples above). This is also why some of you have noticed that if you wait long enough (well, 90 seconds in my tests).. the videos/sites that are blocked will eventually continue to load. Due to the persistent nature of TCP, once the 90 second blacklist window passes.. your TCP stream will continue and the payload data for whatever you're requesting will reach your computer.


Mitigation techniques :

i. Use 'https://' wherever possible (especially on Facebook). Users in the thread have recommended HTTPS Everywhere which is a Firefox/Chrome addon to do this automatically for most major websites.

* While YouTube supports HTTPS for their main website, their player does not support it so even if you were to use HTTPS on YT.. the videos won't load.

ii. For accessing blocked YouTube videos, you can use some of the various YouTube proxy sites such as ProxFree.

iii. Get a VPN/SSH tunnel service if you're worried about having your HTTP requests intercepted.


UPDATE 5 :
Response from MCMC
QUOTE
GE13: ISPs not restricting access, says MCMC
By PATRICK LEE

SUNGAI PETANI: Internet Service Providers (ISP) have not been restricting access to local online portals, according to the Malaysian Communications and Multimedia Commission (MCMC). "Preliminary investigations indicate no such restrictions by ISPs as alleged by certain quarters," it said in a statement.

It said network congestion could have caused users to experience difficulties in accessing the sites, adding there was an increase in traffic for GE13-related articles.
Hey, here's a simple test you can do with less than 2 commands on a Linux box + Wireshark :

CODE
wget http://www.facebook.com/DAPMalaysia

user posted image

So a HTTP GET request for /DAPMalaysia results in the query taking 109 seconds to respond along with 8 TCP retransmissions (I'm basically getting 0 TCP responses from the server for 109 seconds). Let's see what happens when we request for the exact same URL however we append 1500 bytes of junk URI padding to the end :

CODE
#!/bin/bash
for i in {1..1500}
do
PADDING=$PADDING"A"
done
wget "http://www.facebook.com/DAPMalaysia?test="$PADDING

.. which results in ..
CODE
wget "http://www.facebook.com/DAPMalaysia?test=AAAAA... (1500 times)"

user posted image

Oh? What do you know, no issues at all. Apparently appending an extra 1500 bytes of junk data to every HTTP request in a 'congested' network results in less network congestion. Who would have guessed /s

---

My final comments on this issue ..

I'm pretty apolitical when it comes to the Internet and networking. The only reason I have to keep testing what some may call PR-friendly URLs is because it seems that the only time we have 'congestion' is when accessing such content.. and the 'congestion' goes away the moment you obfuscate the requests enough. With the resources that the MCMC has available to debug these kind of issues, I'm honestly surprised they haven't figured this out already.

The tests we've done here show at the very least there is some kind of HTTP request inspection happening and traffic is being dropped once certain strings have been identified. As Internet users and/or caretakers, we should be against any form of Internet censorship. I leave you with these two articles hosted on the MCMC/SKMM website :

http://www.skmm.gov.my/Media/Press-Clippin...sur-fitnah.aspx
QUOTE
SKMM pantau, sekat blog ada unsur fitnah
03/03/2013, Berita Harian

Butterworth: Suruhanjaya Komunikasi dan Multimedia Malaysia (SKMM) akan memantau dan menyekat mana-mana blog yang didapati memuatkan kenyataan berunsur fitnah menjelang Pilihan Raya Umum Ke-13 (PRU-13).

Timbalan Menteri Penerangan, Komunikasi dan Kebudayaan, Senator Datuk Maglin Dennis D'Cruz, berkata kebanyakan blog yang disiasat SKMM kebanyakannya mempunyai agenda tersendiri dengan menulis kenyataan yang tidak betul dan cuba berbohong untuk menjatuhkan maruah seseorang.
“Oleh itu, sempena PRU-13, SKMM diminta memantau dan menyekat mana-mana blog yang cuba menjatuhkan seseorang dengan menulis perkara tidak benar dan mempunyai unsur fitnah, sama ada pada pihak pembangkang atau Barisan Nasional (BN). Kita mahu PRU-13 berjalan aman tanpa sebarang isu,” katanya.


Beliau berkata demikian selepas merasmikan Program Kenali dan Mesra Jiran Peringkat Negeri Pulau Pinang di Rumah Pangsa Taman Bagan, di sini semalam. Hadir sama, Ketua UMNO Bagan, Datuk Abdul Latiff Mirasa; Penyelaras BN Bagan Dalam, M Karuppanan dan Penyelaras BN Parlimen Bagan, David Chua Teik Siang.


http://www.skmm.gov.my/Media/Press-Clippin...edia-In-Ge.aspx
QUOTE
MCMC To Monitor, Control Use Of Social Media In General Election 13 To Prevent Abuse
02/03/2013, Bernama

BUTTERWORTH, March 2  (Bernama) -- The Malaysian Communications and Multimedia Commission (MCMC) is looking into suitable methods to monitor and control the use of social media in the 13th general election (GE13).

Deputy Information Communications and Culture Minister Datuk Maglin Dennis D'Cruz said this was to ensure that the social media would not be abused by irresponsible quarters to achieve their own political agenda.

Last week, Prime Minister Datuk Seri Najib Tun Razak said the GE13 would be the first 'social media election' in the country where internet would be widely used as a campaign tool.

The MCMC will monitor all users of social media, regardless of their political beliefs, to ensure peace and smooth running of the GE13, Maglin said after opening the 'Know Your Neighbours' programme organised by Penang Information Department at Taman Bagan flats here Saturday.

The deputy minister said the monitoring of the social media was vital as certain quarters were only good at making baseless allegations and spreading lies to gain political mileage, adding that he himself had once fallen victim to such lies and accusations.

Maglin said the culture of making baseless allegations and distorting facts among politicians was indeed unhealthy and would only confuse the public, especially the young generation.

"They should not be so selfish and lie just for the sake of gaining political mileage because what matters most in politics was to ensure that the people will live in peace and harmony.

"Therefore, the public, especially the young voters should be wise enough to do their parts in selecting the right government with vast experience in managing the country, so that their future will be secured.

"Don't believe the lies and accusations made by those whose aim is only to create disharmony among the people," he added.

user posted image

This post has been edited by rizvanrp: May 3 2013, 07:04 AM
Volvagia356
post May 1 2013, 10:45 AM

New Member
*
Group: Newbie
Posts: 3

Joined: Oct 2010


Confirmed on my side. Doesn't work on UniFi, works perfectly via SSH tunnel. Also, I've noticed that if I WHOIS the IP address of the CDN server, it's a TMNet address.
JinXXX
post May 1 2013, 10:49 AM

Look at all my stars!!
*******
Group: Senior Member
Posts: 2,464

Joined: Feb 2007
From: Uarla Umpur



maybe someone should report to youtube regarding this.. if "someone" is really messing with youtube's CDN

can we open a streaming session using https ?

im guessing they are detecting the video id .com/?v=xxxxxx and using that as a key block ?

best if we use other methods to download the same video .. reupload it .. and streaming it back...

im guessing there won't be any problem.. with that

This post has been edited by JinXXX: May 1 2013, 10:52 AM
MNet
post May 1 2013, 11:10 AM

Look at all my stars!!
*******
Group: Senior Member
Posts: 8,856

Joined: May 2007



i can see the video no problem
rizvanrp
post May 1 2013, 11:14 AM

Getting Started
Group Icon
Networks & Broadband
Group: Elite
Posts: 166

Joined: Sep 2006



@volvagia356
The local CDNs are hosted by TM

@jinxxx
Can't use HTTPS for streaming, I tried but Google's YT CDNs aren't configured for it. They return an invalid cert and a HTTP 5xx error. The DPI is inspecting the '/videoplayback?' request. I was able to bypass the block without using an encrypted tunnel by appending 1000 bytes of junk padding to the end of the '/videoplayback?' request.. I guess this causes their protocol classifier to screw up.

Based on the behavior, it seems the 90 second delay is because after they detect a blacklisted video stream, it starts dropping traffic from the source for 90 seconds. There's a small gap where 1-4KBs of data can slip through because I guess it takes some CPU cycles to update their source IP + TCP port blacklist. If you can still receive the retransmitted TCP ACK from YT's servers (after the 90 second block), your video will play normally.

This is still pretty scary IMO.. it's looking into all unencrypted protocols on all ports. Running 24/7 via a VPN now just to be safe.
JinXXX
post May 1 2013, 11:22 AM

Look at all my stars!!
*******
Group: Senior Member
Posts: 2,464

Joined: Feb 2007
From: Uarla Umpur



QUOTE(rizvanrp @ May 1 2013, 11:14 AM)
This is still pretty scary IMO.. it's looking into all unencrypted protocols on all ports. Running 24/7 via a VPN now just to be safe.
*
yeah its in deed scary, possible to cross check the "tech" that they use to block ? or are they using sandvine to block it as its a DPI used on streamyx for bt traffic...

maybe we should monitor it a few more days.. you did the test located in which area ? maybe they starting to online/production the DPI systems one by one according to area...
Volvagia356
post May 1 2013, 12:11 PM

New Member
*
Group: Newbie
Posts: 3

Joined: Oct 2010


I've tried blocking TM's CDN
CODE
iptables -A OUTPUT -d 58.27.108.142 -j DROP

and making it directly go to Google (173.194.38.132), but looks like it's still blocked.
lsding
post May 1 2013, 02:09 PM

New Member
*
Group: Junior Member
Posts: 12

Joined: Mar 2013
I am using Celcom Broadband now and also cannot load the above two videos link. where is our freedom of information?
zaqplm
post May 1 2013, 04:13 PM

New Member
*
Group: Junior Member
Posts: 5

Joined: Sep 2005


I can confirm using Maxis broadband, I can't open the above videos. After the advertisement, youtube just shows "an error has occurred". Not even 1 sec of the video is shown. This is very similar to "HTTP-Video" block we applied on our Fortigate device in our office, but I've tested this without any firewall device. So TM is really applying Deep Packet Inspection to certain youtube videos.

You can also test the links below:

http://www.facebook.com/DAPMalaysia <-- Fails to open
https://www.facebook.com/DAPMalaysia <-- Opened OK
rizvanrp
post May 1 2013, 05:28 PM

Getting Started
Group Icon
Networks & Broadband
Group: Elite
Posts: 166

Joined: Sep 2006



@zaqplm

Holy shit you're right :

CODE
echo -e 'GET /DAPMalaysia HTTP/1.1\r\nHost: www.facebook.com\r\n\r\n' | nc.traditional any.ip.address 80

.. fails after first ACK with TCP stream being dropped :
user posted image

Anything else :

CODE
echo -e 'GET /DAPMereisia HTTP/1.1\r\nHost: www.facebook.com\r\n\r\n' | nc.traditional any.ip.address 80

.. works without any issues :
user posted image

They're inspecting all HTTP request and blocking anything with 'Host: *facebook.com' with a GET request method for '/DAPMalaysia'..! Btw are you on Maxis fiber over TM infra or directly in a Maxis broadband area?

What kind of third world country is this

This post has been edited by rizvanrp: May 1 2013, 05:32 PM
zaqplm
post May 1 2013, 06:07 PM

New Member
*
Group: Junior Member
Posts: 5

Joined: Sep 2005


QUOTE(rizvanrp @ May 1 2013, 05:28 PM)
@zaqplm

Holy shit you're right :

CODE
echo -e 'GET /DAPMalaysia HTTP/1.1\r\nHost: www.facebook.com\r\n\r\n' | nc.traditional any.ip.address 80

.. fails after first ACK with TCP stream being dropped :
user posted image

Anything else :

CODE
echo -e 'GET /DAPMereisia HTTP/1.1\r\nHost: www.facebook.com\r\n\r\n' | nc.traditional any.ip.address 80

.. works without any issues :
user posted image

They're inspecting all HTTP request and blocking anything with 'Host: *facebook.com' with a GET request method for '/DAPMalaysia'..! Btw are you on Maxis fiber over TM infra or directly in a Maxis broadband area?

What kind of third world country is this
*
I'm using normal 3G HSDPA broadband dongle with a Maxis sim card.
JinXXX
post May 1 2013, 08:55 PM

Look at all my stars!!
*******
Group: Senior Member
Posts: 2,464

Joined: Feb 2007
From: Uarla Umpur



QUOTE(zaqplm @ May 1 2013, 04:13 PM)
umobile data plan
confirm can load... cause can also torrent smile.gif

This post has been edited by JinXXX: May 1 2013, 08:56 PM
andrew9292
post May 1 2013, 09:28 PM

-/Livin' On A Prayer/-
*****
Group: Senior Member
Posts: 939

Joined: Sep 2008
From: Petaling Jaya


Had the same symptomps with u guys, now i know why everything else in 1080p is rocket fast. But 'rocket' videos are all 'crashing'
runsing
post May 1 2013, 09:44 PM

Casual
***
Group: Junior Member
Posts: 381

Joined: Jan 2011
From: Alor Setar, Kedah


QUOTE(zaqplm @ May 1 2013, 04:13 PM)
I can confirm using Maxis broadband, I can't open the above videos. After the advertisement, youtube just shows "an error has occurred". Not even 1 sec of the video is shown. This is very similar to "HTTP-Video" block we applied on our Fortigate device in our office, but I've tested this without any firewall device. So TM is really applying Deep Packet Inspection to certain youtube videos.

You can also test the links below:

http://www.facebook.com/DAPMalaysia  <-- Fails to open
https://www.facebook.com/DAPMalaysia  <-- Opened OK
*
i'm on tmnet's SDSL. fixed ip.
unable to open the 1st link. able to open the 2nd. sweat.gif

getting more and more desperate eh, good. savor the feeling. maybe you'll be good government in the far future, but coming may 5th, try play opposition for some time, BN

rizvanrp
post May 1 2013, 09:58 PM

Getting Started
Group Icon
Networks & Broadband
Group: Elite
Posts: 166

Joined: Sep 2006



QUOTE(Volvagia356 @ May 1 2013, 12:11 PM)
I've tried blocking TM's CDN
CODE
iptables -A OUTPUT -d 58.27.108.142 -j DROP

and making it directly go to Google (173.194.38.132), but looks like it's still blocked.
*
This works for me :

CODE
iptables -t mangle -A PREROUTING -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 10

Fragmenting the blacklisted HTTP requests into smaller TCP segments bypasses the blocks. It seems they are analyzing traffic on a per TCP segment basis.
Maxieos
post May 1 2013, 10:10 PM

Look at all my stars!!
*******
Group: Senior Member
Posts: 2,780

Joined: May 2008
Temp solution
http://www.proxfree.com/youtube-proxy.php

This post has been edited by Maxieos: May 1 2013, 10:16 PM
wKkaY
post May 1 2013, 10:14 PM

misutā supākoru
Group Icon
Group: VIP
Posts: 5,883

Joined: Jan 2003
I tried the accessing the DAP facebook page and confirm that something is blocking it.

I also found a way to defeat their DPI. They are matching within a single packet instead of a stream. So if you make your HTTP request in two segments, one containing the URL and another containing the Host header, your request will not be filtered.

This (python code) will get stuck:
CODE
from socket import socket, IPPROTO_TCP, TCP_NODELAY
s = socket()
s.setsockopt(IPPROTO_TCP, TCP_NODELAY, 1)
s.connect(("www.facebook.com", 80))
s.send("GET /DAPMalaysia HTTP/1.1\r\nHost: www.facebook.com\r\n\r\n")
print s.recv(65536)


While this one succeeds:
CODE
from socket import socket, IPPROTO_TCP, TCP_NODELAY
s = socket()
s.setsockopt(IPPROTO_TCP, TCP_NODELAY, 1)
s.connect(("www.facebook.com", 80))
s.send("GET /DAPMalaysia HTTP/1.1\r\n")
s.send("Host: www.afacebook.com\r\n\r\n")
print s.recv(65536)

wKkaY
post May 1 2013, 10:15 PM

misutā supākoru
Group Icon
Group: VIP
Posts: 5,883

Joined: Jan 2003
QUOTE(rizvanrp @ May 1 2013, 09:58 PM)
This works for me :

CODE
iptables -t mangle -A PREROUTING -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 10

Fragmenting the blacklisted HTTP requests into smaller TCP segments bypasses the blocks. It seems they are analyzing traffic on a per TCP segment basis.
*
Damn, you posted this 15 minutes before I did! tongue.gif
mechanicalKB
post May 1 2013, 10:15 PM

On my way
****
Group: Senior Member
Posts: 577

Joined: Apr 2012
Could be where they insert the elections video? You know the ones that you have to view ads before you can play your intended video?
cyberloner
post May 1 2013, 10:18 PM

Regular
******
Group: Senior Member
Posts: 1,536

Joined: May 2005
From: Penang


shit nia block now............ lol too late..... we change goverment!

21 Pages  1 2 3 > » Top
Bump TopicReply to this topicTopic OptionsStart new topic
 

Switch to:
| Lo-Fi Version
0.0982sec    3.95    5 queries    GZIP Disabled
Time is now: 19th October 2017 - 09:55 AM