Welcome Guest ( Log In | Register )

Outline · [ Standard ] · Linear+

> YouTube Deep Packet Inspection, All HTTP connections being MITMed

views
     
TSrizvanrp
post May 1 2013, 04:00 AM, updated 10y ago

Getting Started
Group Icon
Elite
190 posts

Joined: Sep 2006



Hi all,

I'm experiencing some anomalies while streaming videos off YouTube on Unifi. For certain 'political' videos -- I've observed that the HTTP connection for the videoplayback stream to YT's local CDNs are being disrupted as follows :

1. Client video player makes a connection to the YT CDN
2. HTTP GET request is sent

There's a few different behaviors after this .. :

3a. HTTP 200 OK is received however it arrives 90 seconds later (should be instant) :
user posted image

3b. HTTP 200 OK is received instantly, first 1-4KB of video stream traffic is sent (allowing the YT player to show the first frame of the video with a timestamp of 0:00).. then no traffic is received for 90 seconds once again :
user posted image

There's a duplicate TCP ACK when the stream returns, did my ACK at packet #271 ever reach the CDN in the first place??

Further testing :

1. Using an unencrypted SOCKS proxy on a remote server + non standard TCP port results in the same behavior with packet loss between the client and SOCKS proxy server

2. Using an encrypted SSH tunnel to the same remote server results in absolutely no issues with viewing the videos

Sample videos :
http://www.youtube.com/watch?v=hHTz22bTBRw
http://www.youtube.com/watch?v=uVWxB4AWOxc

UPDATE :

I performed a simultaneous packet capture on both my client + remote server while encapsulating the HTTP connection via plaintext SOCKS. All the video payload packets were dropped en route back to my SOCKS client :

user posted image

Dafuq?

UPDATE 2 :

Confirming all plaintext HTTP connections on Unifi (and maybe Celcom + Maxis) are being man-in-the-middle'd and dropped if they contain blacklisted data.

UPDATE 3 :

Other sources confirming this .. (thanks wkkay):

https://plus.google.com/1013966581485225280...sts/ak6opfbDxwa

UPDATE 4 :
What we know :

i. The DPI isn't only being used to selectively block YouTube videos, however unencrypted Facebook pages belonging to certain parties are also being blocked. You can get around this by appending 'https://' to the Facebook URLs rather than trying to use 'http://'.

ii. The DPI is based on TCP segment analysis. Basically, every single TCP packet has its payload analyzed for certain request URI strings that have been blacklisted. Obfuscation attacks such as packet fragmentation (splitting a large TCP payload containing a single HTTP request into smaller TCP segments) as well as packet padding (appending large amount of junk data to the HTTP request URI in order to force the 'HTTP/1.1\r\n' trailer into a separate TCP segment) will also work however you need specialized HTTP proxy software or iptables rules (on Linux) to do this.

iii. Once a blacklisted payload is detected within a packet, the header information for the TCP stream (SRC/DST port + SRC/DST IP address) is added to some kind of blacklist for 90 seconds. This causes all traffic for that particular TCP stream to be dropped for 90 seconds (hence the 90 second gaps in my packet capture samples above). This is also why some of you have noticed that if you wait long enough (well, 90 seconds in my tests).. the videos/sites that are blocked will eventually continue to load. Due to the persistent nature of TCP, once the 90 second blacklist window passes.. your TCP stream will continue and the payload data for whatever you're requesting will reach your computer.


Mitigation techniques :

i. Use 'https://' wherever possible (especially on Facebook). Users in the thread have recommended HTTPS Everywhere which is a Firefox/Chrome addon to do this automatically for most major websites.

* While YouTube supports HTTPS for their main website, their player does not support it so even if you were to use HTTPS on YT.. the videos won't load.

ii. For accessing blocked YouTube videos, you can use some of the various YouTube proxy sites such as ProxFree.

iii. Get a VPN/SSH tunnel service if you're worried about having your HTTP requests intercepted.


UPDATE 5 :
Response from MCMC
QUOTE
GE13: ISPs not restricting access, says MCMC
By PATRICK LEE

SUNGAI PETANI: Internet Service Providers (ISP) have not been restricting access to local online portals, according to the Malaysian Communications and Multimedia Commission (MCMC). "Preliminary investigations indicate no such restrictions by ISPs as alleged by certain quarters," it said in a statement.

It said network congestion could have caused users to experience difficulties in accessing the sites, adding there was an increase in traffic for GE13-related articles.
Hey, here's a simple test you can do with less than 2 commands on a Linux box + Wireshark :

CODE
wget http://www.facebook.com/DAPMalaysia

user posted image

So a HTTP GET request for /DAPMalaysia results in the query taking 109 seconds to respond along with 8 TCP retransmissions (I'm basically getting 0 TCP responses from the server for 109 seconds). Let's see what happens when we request for the exact same URL however we append 1500 bytes of junk URI padding to the end :

CODE
#!/bin/bash
for i in {1..1500}
do
PADDING=$PADDING"A"
done
wget "http://www.facebook.com/DAPMalaysia?test="$PADDING

.. which results in ..
CODE
wget "http://www.facebook.com/DAPMalaysia?test=AAAAA... (1500 times)"

user posted image

Oh? What do you know, no issues at all. Apparently appending an extra 1500 bytes of junk data to every HTTP request in a 'congested' network results in less network congestion. Who would have guessed /s

---

My final comments on this issue ..

I'm pretty apolitical when it comes to the Internet and networking. The only reason I have to keep testing what some may call PR-friendly URLs is because it seems that the only time we have 'congestion' is when accessing such content.. and the 'congestion' goes away the moment you obfuscate the requests enough. With the resources that the MCMC has available to debug these kind of issues, I'm honestly surprised they haven't figured this out already.

The tests we've done here show at the very least there is some kind of HTTP request inspection happening and traffic is being dropped once certain strings have been identified. As Internet users and/or caretakers, we should be against any form of Internet censorship. I leave you with these two articles hosted on the MCMC/SKMM website :

http://www.skmm.gov.my/Media/Press-Clippin...sur-fitnah.aspx
QUOTE
SKMM pantau, sekat blog ada unsur fitnah
03/03/2013, Berita Harian

Butterworth: Suruhanjaya Komunikasi dan Multimedia Malaysia (SKMM) akan memantau dan menyekat mana-mana blog yang didapati memuatkan kenyataan berunsur fitnah menjelang Pilihan Raya Umum Ke-13 (PRU-13).

Timbalan Menteri Penerangan, Komunikasi dan Kebudayaan, Senator Datuk Maglin Dennis D'Cruz, berkata kebanyakan blog yang disiasat SKMM kebanyakannya mempunyai agenda tersendiri dengan menulis kenyataan yang tidak betul dan cuba berbohong untuk menjatuhkan maruah seseorang.
“Oleh itu, sempena PRU-13, SKMM diminta memantau dan menyekat mana-mana blog yang cuba menjatuhkan seseorang dengan menulis perkara tidak benar dan mempunyai unsur fitnah, sama ada pada pihak pembangkang atau Barisan Nasional (BN). Kita mahu PRU-13 berjalan aman tanpa sebarang isu,” katanya.


Beliau berkata demikian selepas merasmikan Program Kenali dan Mesra Jiran Peringkat Negeri Pulau Pinang di Rumah Pangsa Taman Bagan, di sini semalam. Hadir sama, Ketua UMNO Bagan, Datuk Abdul Latiff Mirasa; Penyelaras BN Bagan Dalam, M Karuppanan dan Penyelaras BN Parlimen Bagan, David Chua Teik Siang.


http://www.skmm.gov.my/Media/Press-Clippin...edia-In-Ge.aspx
QUOTE
MCMC To Monitor, Control Use Of Social Media In General Election 13 To Prevent Abuse
02/03/2013, Bernama

BUTTERWORTH, March 2  (Bernama) -- The Malaysian Communications and Multimedia Commission (MCMC) is looking into suitable methods to monitor and control the use of social media in the 13th general election (GE13).

Deputy Information Communications and Culture Minister Datuk Maglin Dennis D'Cruz said this was to ensure that the social media would not be abused by irresponsible quarters to achieve their own political agenda.

Last week, Prime Minister Datuk Seri Najib Tun Razak said the GE13 would be the first 'social media election' in the country where internet would be widely used as a campaign tool.

The MCMC will monitor all users of social media, regardless of their political beliefs, to ensure peace and smooth running of the GE13, Maglin said after opening the 'Know Your Neighbours' programme organised by Penang Information Department at Taman Bagan flats here Saturday.

The deputy minister said the monitoring of the social media was vital as certain quarters were only good at making baseless allegations and spreading lies to gain political mileage, adding that he himself had once fallen victim to such lies and accusations.

Maglin said the culture of making baseless allegations and distorting facts among politicians was indeed unhealthy and would only confuse the public, especially the young generation.

"They should not be so selfish and lie just for the sake of gaining political mileage because what matters most in politics was to ensure that the people will live in peace and harmony.

"Therefore, the public, especially the young voters should be wise enough to do their parts in selecting the right government with vast experience in managing the country, so that their future will be secured.

"Don't believe the lies and accusations made by those whose aim is only to create disharmony among the people," he added.

user posted image

This post has been edited by rizvanrp: May 3 2013, 07:04 AM
TSrizvanrp
post May 1 2013, 11:14 AM

Getting Started
Group Icon
Elite
190 posts

Joined: Sep 2006



@volvagia356
The local CDNs are hosted by TM

@jinxxx
Can't use HTTPS for streaming, I tried but Google's YT CDNs aren't configured for it. They return an invalid cert and a HTTP 5xx error. The DPI is inspecting the '/videoplayback?' request. I was able to bypass the block without using an encrypted tunnel by appending 1000 bytes of junk padding to the end of the '/videoplayback?' request.. I guess this causes their protocol classifier to screw up.

Based on the behavior, it seems the 90 second delay is because after they detect a blacklisted video stream, it starts dropping traffic from the source for 90 seconds. There's a small gap where 1-4KBs of data can slip through because I guess it takes some CPU cycles to update their source IP + TCP port blacklist. If you can still receive the retransmitted TCP ACK from YT's servers (after the 90 second block), your video will play normally.

This is still pretty scary IMO.. it's looking into all unencrypted protocols on all ports. Running 24/7 via a VPN now just to be safe.
TSrizvanrp
post May 1 2013, 05:28 PM

Getting Started
Group Icon
Elite
190 posts

Joined: Sep 2006



@zaqplm

Holy shit you're right :

CODE
echo -e 'GET /DAPMalaysia HTTP/1.1\r\nHost: www.facebook.com\r\n\r\n' | nc.traditional any.ip.address 80

.. fails after first ACK with TCP stream being dropped :
user posted image

Anything else :

CODE
echo -e 'GET /DAPMereisia HTTP/1.1\r\nHost: www.facebook.com\r\n\r\n' | nc.traditional any.ip.address 80

.. works without any issues :
user posted image

They're inspecting all HTTP request and blocking anything with 'Host: *facebook.com' with a GET request method for '/DAPMalaysia'..! Btw are you on Maxis fiber over TM infra or directly in a Maxis broadband area?

What kind of third world country is this

This post has been edited by rizvanrp: May 1 2013, 05:32 PM
TSrizvanrp
post May 1 2013, 09:58 PM

Getting Started
Group Icon
Elite
190 posts

Joined: Sep 2006



QUOTE(Volvagia356 @ May 1 2013, 12:11 PM)
I've tried blocking TM's CDN
CODE
iptables -A OUTPUT -d 58.27.108.142 -j DROP

and making it directly go to Google (173.194.38.132), but looks like it's still blocked.
*
This works for me :

CODE
iptables -t mangle -A PREROUTING -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 10

Fragmenting the blacklisted HTTP requests into smaller TCP segments bypasses the blocks. It seems they are analyzing traffic on a per TCP segment basis.
TSrizvanrp
post May 1 2013, 11:20 PM

Getting Started
Group Icon
Elite
190 posts

Joined: Sep 2006



QUOTE(wKkaY @ May 1 2013, 10:15 PM)
Damn, you posted this 15 minutes before I did! tongue.gif
*
Ha, I modified tinyproxy to test padding attacks and those worked as well (as long as you push the HTTP/1.x/r/n protocol trailer into the next segment).

QUOTE(Enigmatic @ May 1 2013, 11:14 PM)
For HTTPS channels, would there be anything which the ISPs can do to prevent access? Perhaps wKkay/prasys/rizvanrp may shed some light on this?
*
They could completely block TCP port 443 on the networks or have a local CA issue some forged HTTPS certs for popular websites. I don't think their DPI tech is at a level that can intercept HTTPS yet though (since fragmentation and padding attacks are getting through).. but they could block HTTPS easily if they wanted to.

Another possibility would be to MITM DNS requests so you wouldn't be able to get around DNS blocks by switching to OpenDNS/Google DNS.

This post has been edited by rizvanrp: May 1 2013, 11:21 PM
TSrizvanrp
post May 1 2013, 11:33 PM

Getting Started
Group Icon
Elite
190 posts

Joined: Sep 2006



QUOTE(blacktubi @ May 1 2013, 11:29 PM)
rizvanrp How to solve this on OpenWRT based router

Using this command through SSH the router don't seem to work

CODE
iptables -t mangle -A PREROUTING -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 10

*
I'm not so sure, I was also having issues implementing it on my Mikrotik. Apparently you can't do TCP MSS clamping on the prerouting chain.. but it works fine on my Linux box running kernel 3.2.0-37. Other option would be to do it at layer 7 with a fragmenting/padding HTTP proxy.. but at this point it would be safer to just switch to a VPN or SSH tunnel rather than trying to obfuscate the requests.
TSrizvanrp
post May 1 2013, 11:52 PM

Getting Started
Group Icon
Elite
190 posts

Joined: Sep 2006



QUOTE(nyemah_mulya @ May 1 2013, 11:49 PM)
I can't open the page "Kami Boikot Buletin Utama" too. Can anyone else check if it is just me?

www.facebook.com/boikotutusan
*
Confirming a block for facebook GET requests to /boikotutusan as well (on Unifi). I also notice that if I append any data to /boikotutusan such as /boikotutusan1 or whatever, the plaintext HTTP request is blocked as well as they're filtering for the 'boikotutusan' string.
TSrizvanrp
post May 2 2013, 09:43 AM

Getting Started
Group Icon
Elite
190 posts

Joined: Sep 2006



QUOTE(Asali @ May 2 2013, 09:35 AM)
It will be good if your guys could upload the Pcaps. I believe it's can be dissected in more clearance. Also, don't forget, in order to look in to more rigorous, it is requires to have one pcap from client and another one pcap from exact OCS.
*
I don't feel like uploading my PCAPs as it would reveal too much information on my internal network structure as well as on my remote servers. I could of course isolate the TCP streams + scrub layer 1/2 but I feel it would be better if other people could independently verify these findings as well. I have posted my full methodology + findings which anyone can replicate themselves.

Just in case it wasn't clear in the first post.. Since I do not have access to run PCAPs on YT's CDN servers, it was just luck that I realized the DPI was also affecting SOCKS traffic allowing me to perform a PCAP on the client as well my SOCKS servers. The TCP streams should have been identical (with the exception of the destination addresses being different due to the internal NAT on my home network and the server talking to my router's public WAN IP) .. but it's fairly obvious the return SOCKS traffic was dropped after my initial YT CDN request for those videos.

With the discovery of the GET request URI + Host header blocks for certain Facebook sites, I'm sure any network admin or security researcher worth their salt will be able to replicate those findings as well (wkkay already has.. and we came to the same workaround as well) laugh.gif
TSrizvanrp
post May 3 2013, 06:59 AM

Getting Started
Group Icon
Elite
190 posts

Joined: Sep 2006



*from update 5 on my first post*
...

Hey, here's a simple test you can do with less than 2 commands on a Linux box + Wireshark :

CODE
wget http://www.facebook.com/DAPMalaysia

user posted image

So a HTTP GET request for /DAPMalaysia results in the query taking 109 seconds to respond along with 8 TCP retransmissions (I'm basically getting 0 TCP responses from the server for 109 seconds). Let's see what happens when we request for the exact same URL however we append 1500 bytes of junk URI padding to the end :

CODE
#!/bin/bash
for i in {1..1500}
do
PADDING=$PADDING"A"
done
wget "http://www.facebook.com/DAPMalaysia?test="$PADDING

.. which results in ..
CODE
wget "http://www.facebook.com/DAPMalaysia?test=AAAAA... (1500 times)"

user posted image
Oh? What do you know, no issues at all. Apparently appending an extra 1500 bytes of junk data to every HTTP request in a 'congested' network results in less network congestion. Who would have guessed /s

---

My final comments on this issue ..

I'm pretty apolitical when it comes to the Internet and networking. The only reason I have to keep testing what some may call PR-friendly URLs is because it seems that the only time we have 'congestion' is when accessing such content.. and the 'congestion' goes away the moment you obfuscate the requests enough. With the resources that the MCMC has available to debug these kind of issues, I'm honestly surprised they haven't figured this out already.

The tests we've done here show at the very least there is some kind of HTTP request inspection happening and traffic is being dropped once certain strings have been identified. As Internet users and/or caretakers, we should be against any form of Internet censorship. I leave you with these two articles hosted on the MCMC/SKMM website :

http://www.skmm.gov.my/Media/Press-Clippin...sur-fitnah.aspx
QUOTE
SKMM pantau, sekat blog ada unsur fitnah
03/03/2013, Berita Harian

Butterworth: Suruhanjaya Komunikasi dan Multimedia Malaysia (SKMM) akan memantau dan menyekat mana-mana blog yang didapati memuatkan kenyataan berunsur fitnah menjelang Pilihan Raya Umum Ke-13 (PRU-13).

Timbalan Menteri Penerangan, Komunikasi dan Kebudayaan, Senator Datuk Maglin Dennis D'Cruz, berkata kebanyakan blog yang disiasat SKMM kebanyakannya mempunyai agenda tersendiri dengan menulis kenyataan yang tidak betul dan cuba berbohong untuk menjatuhkan maruah seseorang.
“Oleh itu, sempena PRU-13, SKMM diminta memantau dan menyekat mana-mana blog yang cuba menjatuhkan seseorang dengan menulis perkara tidak benar dan mempunyai unsur fitnah, sama ada pada pihak pembangkang atau Barisan Nasional (BN). Kita mahu PRU-13 berjalan aman tanpa sebarang isu,” katanya.


Beliau berkata demikian selepas merasmikan Program Kenali dan Mesra Jiran Peringkat Negeri Pulau Pinang di Rumah Pangsa Taman Bagan, di sini semalam. Hadir sama, Ketua UMNO Bagan, Datuk Abdul Latiff Mirasa; Penyelaras BN Bagan Dalam, M Karuppanan dan Penyelaras BN Parlimen Bagan, David Chua Teik Siang.


http://www.skmm.gov.my/Media/Press-Clippin...edia-In-Ge.aspx
QUOTE
MCMC To Monitor, Control Use Of Social Media In General Election 13 To Prevent Abuse
02/03/2013, Bernama

BUTTERWORTH, March 2  (Bernama) -- The Malaysian Communications and Multimedia Commission (MCMC) is looking into suitable methods to monitor and control the use of social media in the 13th general election (GE13).

Deputy Information Communications and Culture Minister Datuk Maglin Dennis D'Cruz said this was to ensure that the social media would not be abused by irresponsible quarters to achieve their own political agenda.

Last week, Prime Minister Datuk Seri Najib Tun Razak said the GE13 would be the first 'social media election' in the country where internet would be widely used as a campaign tool.

The MCMC will monitor all users of social media, regardless of their political beliefs, to ensure peace and smooth running of the GE13, Maglin said after opening the 'Know Your Neighbours' programme organised by Penang Information Department at Taman Bagan flats here Saturday.

The deputy minister said the monitoring of the social media was vital as certain quarters were only good at making baseless allegations and spreading lies to gain political mileage, adding that he himself had once fallen victim to such lies and accusations.

Maglin said the culture of making baseless allegations and distorting facts among politicians was indeed unhealthy and would only confuse the public, especially the young generation.

"They should not be so selfish and lie just for the sake of gaining political mileage because what matters most in politics was to ensure that the people will live in peace and harmony.

"Therefore, the public, especially the young voters should be wise enough to do their parts in selecting the right government with vast experience in managing the country, so that their future will be secured.

"Don't believe the lies and accusations made by those whose aim is only to create disharmony among the people," he added.

user posted image

This post has been edited by rizvanrp: May 3 2013, 07:04 AM
TSrizvanrp
post May 7 2013, 07:47 PM

Getting Started
Group Icon
Elite
190 posts

Joined: Sep 2006



QUOTE(flying spaghetti monster @ May 7 2013, 04:01 PM)
This kena blocked too http://www.youtube.com/watch?v=9jQuCRcdbVM

Now the racist Malay papagomo's website kena attack back by Pakatan Rakyat IT elite? lol this is internet war man. http://www.papagomo.com/
*
Requests containing 'Host: www.papagomo.com' started to get blocked today morning based on my tests

 

Change to:
| Lo-Fi Version
0.0502sec    0.84    6 queries    GZIP Disabled
Time is now: 27th May 2022 - 11:49 AM