doing it my way

Almost 3 years ago pandora.com decided to limit the service only to U.S residents, making non US users to find some similar services like last.fm or Jango.

Although there are many free proxies available, pandora is keeping a black list of proxies IP and blocks connection made using these IPs.

Tor IPs are blocked as well, so the only option is using a non-public proxy.

I’ve got a web hosting service hosted somewhere in Seattle I use for hosting some sites I have like this site and I’ve decided to try to use that web hosting account as a tunnel for pandora.com, or hulu.com or even to hide my ip.

However this account does not have a SSH access. or more accurate, it does have an access to SSH protocol, but not for executing a shell/command on that connection….

it appears that shell access is not necessary at all and such tunnel can be made without the starting up the shell.

Eventually I’ve managed to connect to pandora (I bumped into more issues, but overcome all of them) , Here is a quick cookbook of how to do it (I’m using Putty) :

1. Setup a tunnel to an ssh account, you can even use a non-shell ssh account like I have  (as long as you use ssh2 protocol) using a flag to do it. (-N or  “Don’t start a shell or command at all” when using Putty)

2. Next issue is a dynamic tunnel, ssh allows using the -D option (probebly stands for dynamic) which setup a Socks proxy on a specific local port which will redirect all traffic using the SSH connection (ssh -D <port> user@host). If you use putty specify a dynamic tunnel with no destination.

3. To make sure tunnel will not be closed due to inactivity, you will need to instract putty to send “keep alive” packets on a constant interval, you can find it at the connection tab in putty (5 seconds will be enough).

4.By now the tunnel created will be available for every program that uses socks proxy (browsers for instance), in order to use it in flash components (such as the one used in hulu), you will need a program named proxifier which redirects all traffic (in our case flash RTMP) threw a predefined proxy (the one we set up before)

and that’s it, pandora is working, hulu is workin….

Enjoy.

If you ever wanted to make an SSH tunnel , you probably know it’s an easy task.
(If you not familier with the way to do it, here is a nice article)

Setting up SSH it’s not a hard task, but it comes for quite a challenge when you need to do it from within a restrictive network that blocks most of the ports (SSH port for example) , and allows connection only from within a proxy server (and only for a restricted list of ports).
such restrictive networks can be found for example, in your work places, or in your universities.

But before i’ll talk about how it can be done, I want to talk a little about security.
After all, if your network administrator added a restriction, there is probably a good reason for that:
Work effeciancy, bandwidth consumption, and mostly reasonable, security.
SSH tunnels can be easily used for bypassing proxy black list sites restriction.
However, exposing yout inner network to some blacklisted site is not a wise thing to do, even if it seems not to be a harming site, it might put your network in risk.
you probebly hear about security exploits in browsers once a while. there was probebly a reason for restricting that site !

The main goal for the information here, is to bypass the restriction in order to connect your home computer. (for case you want to extract some files from there, or want to read your email, etc)
I don’t encourage you to do it for other tasks like connecting peer2peer networks, or bypassing proxy blacklist.

Ok, enough talking….this is how it is done.
(I assume you want to connect from some restrictive network into your home computer)

Most of the restricting networks will block all the ports except 80 & 443.
In order to bypass such restirction you need to use these ports, (80/443) to connect your home computer :

you need to setup your ssh server, to either listen on ports 80/443, or in case you have a personal router,
setup up a redirection from one port to your ssh port, for example (from port 443 to 22)

The Idea behind this trick, that although the port is used for HTTP/HTTPS, it does not necessarily must transfer HTTP packets…..

Here are the cookbook steps:

1. setup dynamic DNS for your home computer, (for example by dyndns.com or by no-ip.com), basically every dynamic dns will be good.
2. If you’re home computer is connected to the internet using only a ADSL/Cable modem you will need to make sure the desired ports (80/443) are opened in your personal firewall.
3. if you’re home computer is connected to the internet behind a router, set up your home computer with a static IP (make sure it is not getting a dynamic IP from your router), and make a redirection from the desired ports in the router into your static IP home computer.
4. for case you need to bypass HTTP proxy server, you can not use port 80, and will have to work only with port 443 (the reason for that is that the proxy server process the request when it is unencrypted and transfer it as as when using HTTPS connection)

let’s take for example the following scenario :

your work computer is using HTTP proxy server “my_internal_proxy_server” on port 8080.
your home computer is using the LAN IP 192.168.1.2 and behing a router with the LAN IP 192.168.1.1
your ssh server is using port 22 on your home computer (IP 192.168.1.2 )
In addition the home network can be accessed using the domane name “my-dyn-dns-host”

you will have to set up redirection in the router from port 443 to 192.168.1.2, port 22, and using the following putty screens to connect your home computer.

(Note - in order to use HTTP proxy you will need to setup keep alive packes so the connection will not end.)

The session configuration

The connection configuration

The proxy configuratoin

Good luck.

Couple of weeks ago I helped some friend of mine to implement a “Contact me” form on her site.

During that time, I came up with a nice script to make a generic email form that can help create “contact me” form in seconds on a site that does not execute php scripts.

The following script with “Generate” a unique URL, that whenever a form submit its data to it, will send an email to some predefined email address, with all the submitted data.

The steps are as follows :

• Open the script
• Supply the target email (i.e. your email), the source email (just for your convinence ), the subject, and the ‘redirect page’ (since you want the post action to return to some “thank you for submitting” page)
• a URL will be generate (along with sample form html) that upon submitting to that url, an email will be sent to the target email (with the form fields) and the page will be redirected back to the ‘redirected page’.
• put the URL in your form action property.

I hope this script will help someone.

The script

Let’s face it, the architecture behind Facebook is quite impressing.
I mean, besides the fancy UI, and the AJAX, Facebook’s developers supply a nice interface for integrating almost anything to Facebook.

I’m talking of course, about the Facebook application interface.
It’s not hard to develop a Facebook application, you can find a good documentation, along with code examples on how to do the common tasks: set the profile box, add info about your recent activity to the feed box, query for user data using FQL, render your page with Facebook Markup language, etc.

however when it comes for setting up an environment for developing the application, it seems that the work flow becomes less comfortable.

setting up an environment for developing a web application is pretty easy, you need to set up a local web server (along with your favorite interpreter), surf to your page on localhost, edit your files with your favorite editor, and refresh once a change was made to the source files.

for PHP I’m using XAMPP with notepad++. Setting it up takes couple of minutes, the work flow is easy, and changes are reflected immediately.

when I started developing a Facebook application I’ve discovered that since the testing is done on Facebook site, your files must be hosted on some static location, which should be available and visible for Facebook.

So in order to to develop a Facebook Application, I’ve considered the following work flows:

1. work on remote site, using FTP/SSH/HTTPS to edit files on the server.
2. work on locally, and make my application online and visible to Facebook.

each of this option has its drawbacks.
- working on remote server, is a lot slower than locally, your possible editors are limited, and you must deal with online traffic (unless your application is not public yet).

- working locally has also its drawbacks, which mostly the fact that you must set up some dynamic name hosting (no-ip.com for example), which means you’re less mobile (can’t develop the application at a cafe.), and your localhost is open to the wild all the time.

The best solution for me was to somehow work locally without the need to open the port at the router, and defining a dynamic host name. I like to call it :
“Facebook emulation”, and it looks like that :

Although Facebook does not have many documents on the way it interact with applications, the flow can be understood from the Facebook client:
when your application calls require_login/require_add, and expects the user variable to be returned, the client checks whether it got some variables posted to the script. it mostly looks for the user id and some session id.

if the script does not have these variables it redirect the page to the login/add pages.
which means that if I’ll transfer these variables to the page it will behave as if it was hosted by Facebook, and by having a session id, it can do FQLs, and call Facebook APIs.

giving some serious thoughts about how to achieve a full Facebook application hosting emulation, led me to write fbemu, using the following steps:

- Fetch The application page
- Convert the result FBML into HTML
- Proxify The links inside the created HTML
- Display the HTML.

Another two things I kept in mind to handle were:
- Take care of posted variables
- Take care or Content other than HTML.

Fetching The page

SO for the start, all I have to do is to POST some variables to my locally facebook application, and that should do the trick…..well…it do…really….the question was how to get the posted variables values……
well…actually, it’s also easy, I’ve added a “if ($user == MY_ID) echo (serialize($_POST));” statement at the top of my application (the one that on the static domain), and use the result of it as the POST arguments for the my application page.
The result of the fetch is the created FBML, which leads us to the next question, how to transform it to HTML?

Convert the result FBML into HTML

my first idea (actually a friend suggested it to me) was to replace all the FBML tags with the hard coded generated HTML code. This might be straightforward for <fb:error> , but requires FQL actions when dealiing with <fb:user>.
Then I realized that I can use the FBML converter tool at Facebook developer site.
The FB Developer site has a nice tool to convert FBML into HTML, you just need to provide the FBML, the user, and the API_key of your facebook application.
Pretty simple, All I had to do is to post these variables to the page an I got formatted page with the original FBML, The Transformed HTML, and the HTML as text (using htmlentities()). I’ve decided to extract the result from the “htmlentities” <textarea> tag, since it is was pretty unique tag. Next step was to do a little transformation, (since the resulte was transformed to be shown as HTML text, I had to transform it back to HTML.)

So For now, I’ve got a way to transform the FBML into HTML, which led me to the next issue, I had to parse the links in the HTML result to point the testing page.

Proxify The links inside the created HTML

when I fetch a page, I must process all the links in that page to ensure the user will not navigate outside the emulator, so statment lik <a href=”http://www.domain.com/page.php”> should be changed into <a href=”fbemu.php?url=http://www.domain.com/page.php”> and the simulator should handle that argument, another thing to take care of is to handle partial or relative links. here I used a function that merges between the current page and the relative page, and can handle partial URL.

Displaing the HTML

as simple as it sounde, just take the result from the proxify function, and display it to the user.
To make it appear like FB, I’ve grabbed FB HTML, rip off the ads, replace all the user links and make it display the application content. I’ve also added a small “emu” near the Facebook logo so the user can distinguish whether is the real site or emulation.

If you look for more clean display, I’ve created another template with less images and border, just with a box at the same size FB is using to supply the correct border for the application (in this way you can see if your content is exceeding facebook width).

As mentioned above, besides the big issues I had to take care of two additional things :
- Take care of posted variables
- Take care or Content other than HTML.

Take care of posted variables
meaning that if the application post some data, I need to pass it to the application.
as simple as it sounds, take the posted variables, add it to the one needed by Facebook client, and that’s it.

Take care or Content other than HTML
I had to check the type of the content received from the application, before passing it to the FBML conversion, and to the proxify function to ensure it is HTML content, and to process it only in that case.
In other cases, image for example, I need to present it as is.
In order to do it, I’m processing the returned header result and look for the Content-Type header.

Conclusion
I’ve decided to share my script to the world to ease the development of facebook .so whoever needs to test /develop its facebook application locally, can use this script.

In order to get the correct values for the session key, I’m using serialized variables (just add “if ($user == MY_ID) echo (serialize($_POST));” at your application and use the output for the local script.

As far as I know there is only one thing it does not handle which is the Invitation page (I’ve added a box to html just to notify about it), besides that I’m now aware of any issues/bugs.

here is a screen shot of the application :

fb emu image

and fbemu can be download from here:

fb emu v1.0

Enjoy.

If you run a site on a low budget, and need small amount of SMS notifications from your site (SERVER is down, DB is not responding, etc…) Instead of buying a bulk SMS bundle, you can use the following script.
It uses the ICQ service to send the SMS (so you need a valid ID & password), and it is limited to some amount of messages a day (20 I think),
but…..it’s free, and it’s working….

here is the script, it is based on PHP2ICQ. (I’ve only added the SMS send function)

Send SMS using php2ICQ

Enjoy.

as promised, i’ve ported the HOT EPG Grabber into unix, actually, to any OS supporting PHP.
I wanted to post the results here for quick download, but, according to the Site policy the site content is for personal use only.

well…enjoy the HotEPG in it’s PHP version.

(note that you can not run it in a browser , due to a long file run)

while developing a web site these days, I was thinking about preventing from automatic registration into site.

The first though was to write / use some sort of captcha (”Completely Automated Public Turing test to tell Computers and Humans Apart”) filter.
however the latest publishing in this area, tells that yahoo captcha has been broken.

I’ve decided to try to implment one of the captcha that has not been yet broken :
Google’s

and came up with this :

you can check it yourself, here’s the source

Enjoy.

yesterday I got an email from a reader, saying it is not allowed to post a comment, I tried to do it myself and everything went well.
when I investigate it some more, I’ve found that for some comments, I’m getting the following message :

Forbidden
You don’t have permission to access /wp-comments-post.php on this server.

when I Google it I found that there is a filtering engine in Apache’s mod_security module which blocks the post request based on it’s contents.
Since I’m not worried of spammers right now, and every comment need approval i’ve decided to turn it off, by adding the following line to the .htaccess :

SecFilterEngine Off

I guess comments are now more welcomed than before.

I’ve encountered a weird behavour in VS2005, VS2003 that sometimes the output DLLs (in case you’re working in a solution with multiple libraries) are locked by devenv.exe (VS)

I didn’t discovered which portion of the IDE locks those DLLs, but my guess it is related to the intelisense machanism ( since when deleting the intelisense file, it is solved for couple of days/hours)

The problem with the wasted (leaked) lock (and this is also the reason I call it “wasted”) , is that it happens prior/during the compilation, causing the compilation to fail (since the output file is locked for write).

Moreover, the DLL is loaded as a module inside devenv.exe, so the locking is done using LoadLibrary (my guess is that there’s a typical call to loadAssembly .NET API, which loads the DLL as a module) , and cannot be remove by tools like ProcessExplorer (from sysinternals.com).

now, this can be solved by either closing the solution (works for couple of minutes/hours), or deleting the intelisense file and reopening the solution.

in both solutions there is a need to reload the solution, which on large solutions might take a lot of time.

This state led me to write ForceUnload, which is my solution for the problem.

ForceUnload is a program that remove the DLL from the loaded modules by calling UnloadLibrary API (KERNEL32) in the process contex, for the DLL needed to be removed.

(the method used in this application is CreateRemoteThread which is one of the methods to inject code in a process, I’ll discuss Code Injection in one of my next posts).

Well…before I continue, I must state the forcing the unload of a DLL or closing a file descriptor in a remote process should not be common task at all….this WILL PROBABLY CAUSE a crash whenever the application will access this DLL or file descriptor.

This should only be done in case it seems the DLL/File descriptor is a leaked resource, and in that case the application will not access it.

Also, I don’t take any responsibility of this application. from my knowledge, it might even cause system halt in case of improper use (try unloading KERNEL32.DLL from WINLOGON.EXE :-)). It did caused some Exceptions in devenv sometimes (which eventually forced me to reload the solution), but most of the times, it gave me couple of hours of coding without the need to reload the solution every 5 mintues…..

Anyway, here is the source of ForceUnload.

enjoy, and take care….

a year ago, I’ve decided to build my own PVR (Personal Video Recorder).

Setting it up was pretty easy, I installed a 800MHz PC with 120GB of storage and 256 MB of RAM. for the capturing, I used Hauppauge WinTV-PVR-150. this card come with IR Blaster ( a small device that can transmit IR signals).

I’ve used GBPVR as the main application, which suites all my needs, the only left question was where to get the EPG from.

According to Israeli communication law, all TV content providers (Cable & Satalite) need to provide their program guide in their site. in practice they don’t. instead, there is a company named TV2day that provide the content to the major portals in israel, and the portals provide the EPG in there pages (HTML format).

Extracting the Program guide from the HTML is simple (using XMLTV), however, the program guide provided is missing lot of channels and there are time mismatch in their data.

So I looked for some other way to get the data. a possible option was to combine multiple sources, however, I found a better way, to extract the guide from the cable provider site.

The cable provider in Israel supply its guide in flash control : (http://www.hot.net.il/EPG/Templates/Homepage/Homepage.aspx?lang=he)

and since the flash control runs on the client, and gets the data somehow, it must be accessable in some other way.

Sniffing the request was pretty easy, and the output of that was HOTEPG, my application to extract the Cable provider (HOT) guide,

here is an output of the guide in GBPVR :

and the sources of HOTEPG can be downloaded from here.

Next thing to do is to schedule it automatically, and to provide it online. (currently it is schudled to on my XP using ‘at’ command)

That’s it for now, enjoy.