Sergio Veiga

I have done this path more than 1 thousand times, just like you have done some path that many times… and today i got this feeling that i needed to record this path, so one day in the future, i can remember why i did it in the first place… and why i stopped.

• • Comments
  • Categories [Uncategorized]

I’m really bad organizing my music so i end up always with a lot of songs duplicated in my itunes library… and since googling for discovering how itunes can do that for you is less funny that do it yourself, it seemed a nice opportunity to programme some more erlang code for humanity great good!

So you can find on github http://github.com/darkua/Itunes-KISS the code and the how to use it.

i have tested on Mac Osx but it should work in any platform.

4 hours of coding for removing 2GB out of 15GB of duplicated music in less then 1 minute looks good

Special thanks to @citizen428 for the id3tagV1 parser ( http://citizen428.net/archives/993 )

• • Comments
  • Categories [Uncategorized]

“Measuring programming progress by lines of code is like measuring
aircraft building progress by weight.“Bill Gates

“Nine people can’t make a baby in a month.” Fred Brooks

“Weeks of coding can save you hours of planning.” unknown

You can find a lot more here

• • Comments
  • Categories [Uncategorized]

Websocket get update and handshacke get broken… this is not for sure and end version of the protocol, but since newer versions of chrome are using it and its not compatible with previous version its a good option to implement both.

In order to understand what changed, i recommend you to read the great resume done in Axod’s Hack .

Mainly you need to get 3 keys, 2 of them are send in the headers and the 3rd is sent after the headers. Then you need to do some “weird” transformation with the 3 and send them back after the header.

After having the 3 keys, here is the code in erlang to build the challenge to send back to client:

Now that you have the challenge just build the response like this :

Challenge = build_challenge(Key1,Key2,Key3),

HandshakeServer = [
        "HTTP/1.1 101 Web Socket Protocol Handshake\r\n",
	"Upgrade: WebSocket\r\n",
	"Connection: Upgrade\r\n",
	"Sec-WebSocket-Origin: ", Origin , "\r\n",
	"Sec-WebSocket-Location: ws://", Location , "\r\n\r\n",
	Challenge
],

As soon i can i will create a new branch on misultin web server so you can see the code working.

• • Comments
  • Categories [Uncategorized]

Other Day @fdmananatold me that he discover that lists key find was much more fast then proplists. Since i use a lot of proplists on my code, i did some tests to check how much fast it is and i was shocked!

(af83@dakua.local)34> timer:tc(proplists,get_value,[99999,List]).
{8685,99999}
(af83@dakua.local)35> timer:tc(lists,keyfind,[99999,1,List]).
{829,{99999,99999}}

WOW! for a list of 100.000 elements proplists:get_value can be 10x slower then lists:key_find!

So now i’m using this encapsulation in order to maintain proplists syntax and avoid extra typing

proplist_get_value(Key,List)->
  case lists:keyfind(Key,1,List) of
    {_K,V}->
      V;
    false->
      undefined
  end.

Now the question is Why? why lists is much more faste then proplists? Our guess is that since lists functions are BIF’s, they must be build in C and proplists no, but this is just a guess, if someone has a better explanation please share.

• • Comments
  • Categories [Uncategorized]

• • Comments
  • Categories [Uncategorized]

A simple erlang gen_server to return original url from short ones.

It accepts any kind of of url, retrieving zero or one redirect location.

Just copy the code and give it a test :

darkua:~ sergioveiga$ erl
Erlang R13B03 (erts-5.7.4) [source] [smp:2:2] [rq:2] [async-threads:0] [kernel-poll:false]

Eshell V5.7.4 (abort with ^G)
1> c(unshort_srv).
{ok,unshort_srv}
2> unshort_srv:start_link().
{ok,<0.39.0>}
3> unshort_srv:unshort(”http://tarpipe.com/kE”).
{ok,”http://sergioveiga.com/index.php/2010/02/21/erlang-code-to-unshort-urls/”}
4> unshort_srv:unshort(”http://google.com”).
{ok,”http://www.google.com/”}
5>

• • Comments
  • Categories [Uncategorized]

So now that you have already know nitrogen basics in part 1 of this tutorial, lets see what he can real do with it. So lets see a real simple web application that will track a keyword and wait for twitter to push “tweets” that contains this keyword, and stream them in a never ending stream. So lets grab our code :

Clone this git repositorium :

darkua:~ sergioveiga$ git clone git://github.com/darkua/ErlanOnNitro_TwitterStream.git
Initialized empty Git repository in /Users/sergioveiga/ErlanOnNitro_TwitterStream/.git/
remote: Counting objects: 72, done.
remote: Compressing objects: 100% (68/68), done.
remote: Total 72 (delta 4), reused 0 (delta 0)
Receiving objects: 100% (72/72), 144.83 KiB | 182 KiB/s, done.
Resolving deltas: 100% (4/4), done.
darkua:~ sergioveiga$ cd ErlanOnNitro_TwitterStream
darkua:ErlanOnNitro_TwitterStream sergioveiga$ ./start.sh

NOTE : in this project we will be use a lib packed with mochiweb project, so please check if already you have mochiweb intalled. (say thanks to @bpedro, for pointing this out )

twitter_stream@localhost)4> mochijson2:encode(<<"hello">>).
[34,<<"hello">>,34]

if you don’t see this, you need to install mochiweb first. Its very easy:

darkua:twitterstream sergioveiga$ echo $ERL_LIBS
/opt/local/lib/erlang/lib/:/usr/lib/erlang/lib/:/Users/sergioveiga/erlangProjects/tedi
darkua:twitterstream sergioveiga$ cd /opt/local/lib/erlang/lib/
darkua:lib sergioveiga$ sudo svn checkout http://mochiweb.googlecode.com/svn/trunk/ mochiweb-read-only
darkua:lib sergioveiga$ cd mochiweb-read-only/
darkua:mochiweb-read-only sergioveiga$ make

Now you probably need to kill erlang shell, just hit ctr-c twice. Start it again and now mochijson should already be available.

Before you go and check your browser you need to enter your twitter credentials, needed for twitter stream. So please go to include/include.hrl and add your own credentials, and compile the project.

Make sure you dont any other server running on port 8000, and go to your browser http://localhost:8000/. Here you will find some descriptions and links to both examples we will follow in this part.

Now choose you favorite IDE, we are using textmate, and open the project dir.

So this is ver similar with your previous nitrogen project, with some little changes. Open your project app file inside ebin dir. Here we have added templateroot so we can put all templates inside a different directory.

We have created an include dir, where we will put all our include files, and on src, we divide the pages, from the libs, that will hold “hard-core” code.

Example 1 – Twitter Stream

A screenshot of how twitter stream looks like :

Now open /src/pages/web_index.erl. You can see that we almost don’t do nothing here, because this is a static file, so we are only are adding the page title, and saying witch template will be used to render the page. All the code you can see its on index.html template file.

Now open web_stream page.

Lets check body function

body() ->

 %%builds the stream
 Stream = start_stream(),

 %% the html to render
 Body= [
 #panel{class="header", body=[
 #h1{text="Erlang on Nitro - Twitter Stream"},
 #panel{id=feedback,class="feedback", body=[
 "Calling Twitter...pls wait..."
 ]},
 #panel{id=control,body=[#button{text="Stop", postback={stop,Stream}}]}
 ]},
 #panel{id=stream,body=[]}
 ],

 wf:render(Body).

Here we are calling stream_start() to open stream connections and building the page html structure. Now check the start_stream() to see where is the magic.

start_stream()->
 Pid = wf:comet(fun()-> ?MODULE:loop() end),

%%check for track
 case wf:get_path_info() of
 []->
 twitter_stream:sample(Pid);
 Track->
 twitter_stream:track(Track,Pid)
 end.

When calling wf:comet, we are building a comet connection to this page, and passing what function will be used to “comunicate” with the browser. Then we use wf:get_path_info() to get extra info on path, that we will use to call twitter, for example the twitter or hello keyword from the examples. All the connection to twitter stream api happens inside twitter_stream, and for now you can look at it as a black box that streams tweets for you, later you can check it out and try to understand what is happening.

Now before checking loop function, its time to better undersand one of erlang top features, creating and communicating with processes. When we did wf:comet, what is happening in resume, is that we have spawned loop function into a new process, and now we can comunicate with it by sending to the Process Id messages. And to do that we just need to do this :

Pid ! message

In order to put a process waiting for messages is also very simple, just do this :

receive
 Message->
 ...
 end.

When creating a receive block, you are telling erlang to wait there until he receives a new message, that he will try to match to execute the respective code.

Now lets check a very simple example in order to better understand this. Open file spawn_example inside libs. You can see that we just have 2 functions, start and loop. On start we are spawning function loop, and registering the process with a name to be easier to call them after, like your fathers did with you

Now go to erlang shell and call this:

(twitter_stream@localhost)2> spawn_example:start().
true
(twitter_stream@localhost)3> bot ! hello.
Hello. How are you?
hello
(twitter_stream@localhost)4>

Very simple and cool, but now try to call it again

(twitter_stream@localhost)15> bot ! hello.
** exception error: bad argument
 in operator  !/2
 called as bot ! hello
(twitter_stream@localhost)16>

Humm… the reason it failed its very simple. After the process received the first message he executed the code matched to hello and it ended. So when you try to send a new message to it, it no longer exists, and it fails. So if you want to create process that after processing a message continues waiting for other you need to recall the function. So in the end of the loop add a call to loop.

loop()->
 receive
 hello ->
 io:format("Hello. How are you?~n",[]);
 _->
 void
 end,
 loop().

Now lets try again

(twitter_stream@localhost)16> sync:go().
Recompile: ./src/libs/spawn_example
ok
(twitter_stream@localhost)17> spawn_example:start().
true
(twitter_stream@localhost)18> bot ! hello.
Hello. How are you?
hello
(twitter_stream@localhost)19> bot ! hello.
Hello. How are you?
hello

As you can see now the process no longer dies. Now lets see another great feature of erlang: hot code swapping, the ability to change the code without stopping the system. Try to change the text on the message, for example to:

io:format("Hot swapping rocks!~n",[])

sync:go and lets see what happens:

(twitter_stream@localhost)24> sync:go().
Recompile: ./src/libs/spawn_example
ok
(twitter_stream@localhost)25> bot ! hello.
Hello. How are you?
hello

Humm… the old message is still there. The reason this happens is very simple. If we dont call “externally” by  prefixing the module name we are saying to erlang to “use this version of this method” and not the current version. So all we need to do is add the ?MODULE to the loop call, in the start function and also in the end of the loop. Now sync your code, and start your bot again.

(twitter_stream@localhost)38> sync:go().
Recompile: ./src/libs/spawn_example
ok
(twitter_stream@localhost)39> spawn_example:start().
true
(twitter_stream@localhost)40> bot ! hello.
Hot swapping rocks!
hello

Now lets change the message again, to “Hot swapping really rocks”. Sync your code again and send him a message.

(twitter_stream@localhost)41> sync:go().
Recompile: ./src/libs/spawn_example
ok
(twitter_stream@localhost)42> bot ! hello.
Hot swapping rocks!
hello
(twitter_stream@localhost)43> bot ! hello.
Hot swapping really rocks!
hello

As you can see the first message the bot receives the old code will be executed, because the process was already blocked waiting for a message, but when it run again it will already be using the current version of the code. You can see final version of spawn_example here.

Now back to our example, you can see exactly this happening with the loop function inside web_stream module. As you can see the loop function will be waiting for a subset of messages that twitter stream will send to him, and according to each we will be doing something different. Now what is important to understand is how nitrogen communicates with the client browser.

We are using 2 nitrogen functions  wf:update(id, content) and wf:insert_top(id, content) that will update and insert on the top of a DOM element. Nitrogen will translate this to the corresponding javascript, using jquery lib, and send it to the browser when we call wf:comet_flush(). So only when call this function inside a comet loop, we will be sending all previous javascript code built with wf:update/wf:insert_top” as a response to the comet connection.

loop()->
 receive
 {unauthorized,Message}->
 io:format("~p~n",[Message]),
 wf:update(feedback,"Your twitter credentials are wrong, check include.htr file!");
 {start,_Message}->
 wf:update(feedback,"Stream Running...");
 {stream,Message}->
 get_tweet_info(Message);
 {stream_end,_Message}->
 restart_button(),
 wf:update(feedback,"Stream Ended...");
 {stop,_Reason}->
 restart_button(),
 wf:update(feedback,"Stream Stoped...");
 {error,Reason}->
 error_logger:error_msg("Error : ~p~n",[Reason]),
 restart_button(),
 wf:update(feedback,#panel{class="error",body=wf:f("Error : ~s !",[Reason])});
 Any->
 error_logger:info_msg("ANY : ~p~n",[Any])
 after 55000 ->
 wf:update(feedback,"to quiet for me...")
 end,
 wf:comet_flush(),
 ?MODULE:loop().

With this simple code you have learn the basic blocks to create parallel computing, and the possibility to push content to users without the need to be constantly polling a service. And this is really cool because is all you need to create cool real-time / multi-user applications.

Example 2 – Twitter Cloud

Twitter Cloud Screenshot :

Nitrogen does already lot of work for us on erlang/javascript interaction, making easier  the creation of appealing  and rich web applications. Check web_cloud example where we have created another visualization for twitter stream, but now its a cloud of twitter users, where the size of the pictures is related to the ratio following/followers, where you can click to see the status update.

if you open web_cloud.erl, you can see that its very closed to web_stream, but now instead of building the html on the server we are passing the info to the the browser and will will build the dom with javascript. This can be achieved very easily with nitrogen using wf:wire constructor. As you can check on line 126 of web_cloud

wf:wire(wf:f("buildTweet({id:'~s',screen_name:'~s',picture:'~s',text:\"~s\",size:'~p'})",
[Tweet#tweet.id,
Tweet#tweet.screen_name,
Tweet#tweet.picture,
wf_utils:js_escape(Tweet#tweet.text),
Tweet#tweet.size])
).

Mainly we are calling the js function buildTweet that you can find on /js/main.js a js object with all the tweet function. We also use another great nitrogen function wf:f, that helps you build complex strings.

Another thing really useful is giving to an event different actions, like you can see in line 27.

#panel{id=control,body=[#button{text="Stop", actions=#event{actions="$.growl('','Stream is Stoping...');"}, postback={stop,Stream}}]}

Inside each event you can define your postback, and you can also use actions parameters to execute javascript. So if you click on the stop button, we will be calling event({stop,Stream}) and also calling jquery-growl plugin that will show a message to the user.  We can even define more than one event type to the same element. For example change control panel this on body function

#panel{id=control,body=[#button{text="Stop", actions=[

#event{actions="$.growl('','Stream is Stoping...');", postback={stop,Stream}},

#event{type=mouseover,actions="$.growl('','Click if you are a mouse!...');"}]
}]}

sync:go, and check your example. You should be seeing a message appear when you put your mouse over stop button and other when you click it.
As you can imagine this is a very simple example of all you can do with nitrogen . So please check nitrogen project where will have a lot of cool examples ,  nitrogen api with all functions available, and you can check some more documentation on on github wiki.

So now you know already enough to start coding your own ideas and projects, but before read last part of this tutorial, where we will resume some of the tools erlang has to help on your adventure.

• • Comments
  • Categories [Uncategorized]