reederzhttp://reederz.com/2018-09-23T11:00:00+02:00Blacklist Referer Spam Bots with NGINX2018-09-23T11:00:00+02:002018-09-23T11:00:00+02:00Justas Ažnatag:reederz.com,2018-09-23:/nginx-referer-spam-blacklist.html<p>Seeing weird things in your analytics dashboard? Here’s an easy approach for blacklisting Referer Spam Bots with <span class="caps">NGINX</span>.</p><p><strong>Originally posted on 2015-04-22 at fadeit.dk/blog which is no longer available</strong></p>
<h2>Some Background</h2>
<p>Recently, we submitted fadeit.dk to <a href="http://www.awwwards.com/best-websites/fadeit-1/"><span class="caps">AWWWARDS</span></a> - the awards for design, creativity and innovation on the Internet. It gave us a nice little boost of website visitors. However, not all of that attention was positive…</p>
<h2>Referer Spam Bots</h2>
<p>While the majority of our traffic came from genuine sources, we started noticing a pattern in our referral traffic.</p>
<p><img alt="Google Analytics Dashboard - Acquisition / Referrals" src="http://reederz.com/images/nginx-referer-spam-blacklist/ga_dashboard.png"></p>
<p>What’s up with <code>social-buttons.com</code>? We didn’t sign up for that… Apparently, this is something called <a href="http://en.wikipedia.org/wiki/Referer_spam">Referer Spam</a>*:</p>
<blockquote>
<p>Referrer spam (also known as log spam or referrer bombing) is a kind of spamdexing (spamming aimed at search engines). The technique involves making repeated web site requests using a fake referer <span class="caps">URL</span> to the site the spammer wishes to advertise. Sites that publish their access logs, including referer statistics, will then inadvertently link back to the spammer’s site. These links will be indexed by search engines as they crawl the access logs. - Wikipedia</p>
</blockquote>
<p>This is not cool, Mr. social-buttons.com. P**s off!</p>
<p><em>* No, I didn’t mispell. The mispelling actually made it into the <a href="http://tools.ietf.org/html/rfc1945"><span class="caps">HTTP</span>/1.0 standard</a> and now it’s there forever :)</em></p>
<h2><span class="caps">NGINX</span> Solution</h2>
<p>Since we’re using <span class="caps">NGINX</span> to serve our site, the solution is going to be described for <span class="caps">NGINX</span>. Apache people can take a look at this excellent <a href="https://www.addedbytes.com/blog/block-referrer-spam/">article</a>.</p>
<p>The official <span class="caps">NGINX</span> wiki does mention <a href="http://wiki.nginx.org/Referrer_Spam_Blocking">a solution</a> to this problem. Basically, you just use <a href="http://nginx.org/en/docs/http/ngx_http_referer_module.html">ngx_http_referer_module</a>* and add something like this to your location or server block:</p>
<div class="highlight"><pre><span></span><code>valid_referers none blocked server_names *.social-buttons.com social-buttons.com badreferer2.com<span class="p">;</span>
<span class="k">if</span> <span class="o">(</span><span class="nv">$invalid_referer</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="m">444</span><span class="p">;</span>
<span class="o">}</span>
</code></pre></div>
<p>This works, but what if we want to maintain a larger blacklist of referers? Our <code>valid_referers</code> directive would get crazy long. If that’s fine with you, you can stop reading here. It sure isn’t fine with me :).</p>
<p>In order to make our blacklist more maintainable, we can use <a href="http://nginx.org/en/docs/http/ngx_http_map_module.html">ngx_http_map_module</a>. Let’s save <code>/etc/nginx/conf.d/blacklist.conf</code> file with the following content:</p>
<div class="highlight"><pre><span></span><code><span class="c1"># /etc/nginx/conf.d/blacklist.conf</span>
map <span class="nv">$http_referer</span> <span class="nv">$bad_referer</span> <span class="o">{</span>
hostnames<span class="p">;</span>
default <span class="m">0</span><span class="p">;</span>
<span class="c1"># Put regexes for undesired referers here</span>
<span class="s2">"~social-buttons.com"</span> <span class="m">1</span><span class="p">;</span>
<span class="s2">"~semalt.com"</span> <span class="m">1</span><span class="p">;</span>
<span class="s2">"~kambasoft.com"</span> <span class="m">1</span><span class="p">;</span>
<span class="s2">"~savetubevideo.com"</span> <span class="m">1</span><span class="p">;</span>
<span class="s2">"~descargar-musica-gratis.net"</span> <span class="m">1</span><span class="p">;</span>
<span class="s2">"~7makemoneyonline.com"</span> <span class="m">1</span><span class="p">;</span>
<span class="s2">"~baixar-musicas-gratis.com"</span> <span class="m">1</span><span class="p">;</span>
<span class="s2">"~iloveitaly.com"</span> <span class="m">1</span><span class="p">;</span>
<span class="s2">"~ilovevitaly.ru"</span> <span class="m">1</span><span class="p">;</span>
<span class="s2">"~fbdownloader.com"</span> <span class="m">1</span><span class="p">;</span>
<span class="s2">"~econom.co"</span> <span class="m">1</span><span class="p">;</span>
<span class="s2">"~buttons-for-website.com"</span> <span class="m">1</span><span class="p">;</span>
<span class="s2">"~buttons-for-your-website.com"</span> <span class="m">1</span><span class="p">;</span>
<span class="s2">"~srecorder.co"</span> <span class="m">1</span><span class="p">;</span>
<span class="s2">"~darodar.com"</span> <span class="m">1</span><span class="p">;</span>
<span class="s2">"~priceg.com"</span> <span class="m">1</span><span class="p">;</span>
<span class="s2">"~blackhatworth.com"</span> <span class="m">1</span><span class="p">;</span>
<span class="s2">"~adviceforum.info"</span> <span class="m">1</span><span class="p">;</span>
<span class="s2">"~hulfingtonpost.com"</span> <span class="m">1</span><span class="p">;</span>
<span class="s2">"~best-seo-solution.com"</span> <span class="m">1</span><span class="p">;</span>
<span class="s2">"~googlsucks.com"</span> <span class="m">1</span><span class="p">;</span>
<span class="s2">"~theguardlan.com"</span> <span class="m">1</span><span class="p">;</span>
<span class="s2">"~i-x.wiki"</span> <span class="m">1</span><span class="p">;</span>
<span class="s2">"~buy-cheap-online.info"</span> <span class="m">1</span><span class="p">;</span>
<span class="s2">"~Get-Free-Traffic-Now.com"</span> <span class="m">1</span><span class="p">;</span>
<span class="o">}</span>
</code></pre></div>
<p>Now add conditions to the sites, for which you want to block referer spam bots:</p>
<div class="highlight"><pre><span></span><code><span class="c1"># /etc/nginx/sites-enabled/mysite.conf</span>
server <span class="o">{</span>
<span class="c1"># ...</span>
<span class="k">if</span> <span class="o">(</span><span class="nv">$bad_referer</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="m">444</span><span class="p">;</span>
<span class="o">}</span>
<span class="c1"># ...</span>
<span class="o">}</span>
</code></pre></div>
<p><span class="caps">OK</span>, now let’s test if this thing works:</p>
<div class="highlight"><pre><span></span><code><span class="c1"># with subdomain</span>
$ curl --referer http://www.social-buttons.com https://fadeit.dk/en
curl: <span class="o">(</span><span class="m">52</span><span class="o">)</span> Empty reply from server
<span class="c1"># without subdomain</span>
$ curl --referer http://social-buttons.com https://fadeit.dk/en
curl: <span class="o">(</span><span class="m">52</span><span class="o">)</span> Empty reply from server
</code></pre></div>
<p>Sweet! It worked.</p>
<p><em>* Both <a href="http://nginx.org/en/docs/http/ngx_http_referer_module.html">ngx_http_referer_module</a> and <a href="http://nginx.org/en/docs/http/ngx_http_map_module.html">ngx_http_map_module</a> are included in the standard <span class="caps">NGINX</span> distribution and you don’t need to recompile your server.</em></p>
<h2>That’s it!</h2>
<p>What’s your experience with Referer Spam? Don’t hesitate to use the comment section :)</p>
<h2>Additional Resources</h2>
<ul>
<li><a href="https://github.com/oohnoitz/nginx-blacklist">Blacklist by oohnoitz</a> which blocks bad bots, pentest tools, surveillance bots, etc. It’s an excellent addition to the referer spam blacklist described in this post.</li>
<li><a href="http://perishablepress.com/4g-ultimate-referrer-blacklist/">Ultimate referer blacklist</a></li>
</ul>Testing Express.js + CouchDB Applications with mock-couch2018-09-23T10:00:00+02:002018-09-23T10:00:00+02:00Justas Ažnatag:reederz.com,2018-09-23:/testing-expressjs-couchdb-applications-with-mock-couch.html<p>There’s a way to test Express.js + CouchDB web apps with confidence of end-to-end tests and execution speed of unit-tests. Meet mock-couch- a node.js module which pretends to be a CouchDB server for testing purposes.</p><h2>Prerequisites</h2>
<p><strong>Originally posted on 2015-07-05 at fadeit.dk/blog which is no longer available</strong></p>
<p>The code in this post is based upon <a href="http://coffeescript.org/">CoffeeScript</a>, <a href="http://mochajs.org/">Mocha</a> and <a href="http://gulpjs.com/">Gulp</a> but it should be easily adaptable to other stacks (e.g. <a href="https://www.javascript.com/">JavaScript</a>, <a href="https://jasmine.github.io/">Jasmine</a> and <a href="http://gruntjs.com/">Grunt</a>).</p>
<h2>Intro</h2>
<p>Recently, I worked on a <a href="http://expressjs.com/">Express.js</a> based RESTful <span class="caps">API</span> with <a href="https://couchdb.apache.org/">CouchDB</a> storage. For various reasons, I mostly had end-to-end tests (testing <span class="caps">HTTP</span> endpoints) and very few unit tests. In the tests, I would rebuild the database before every <code>describe</code> block. This worked quite well but at one point I noticed that my feedback loops were beginning to get tediously slow (~30 seconds to run the full test suite). Thankfully, I discovered <a href="https://github.com/chris-l/mock-couch">mock-couch</a>: in-memory storage with CouchDB <span class="caps">API</span>. The switch was rather easy and, as a result, my test suite would finish in less than 10 seconds (even when reseting test data before each test).</p>
<h2>Setup</h2>
<p>Our sample <span class="caps">API</span> will expose functionality to access a database of sofa’s. Let’s set it up.</p>
<p>Structure and dependencies</p>
<div class="highlight"><pre><span></span><code>mkdir -p sofa-api/specs
<span class="nb">cd</span> sofa-api
npm install mock-couch coffee-script gulp gulp-mocha should supertest express nano q
</code></pre></div>
<p>gulpfile.coffee</p>
<div class="highlight"><pre><span></span><code><span class="nx">require</span> <span class="s">'coffee-script/register'</span>
<span class="nv">gulp = </span><span class="nx">require</span> <span class="s">'gulp'</span>
<span class="nv">mocha = </span><span class="nx">require</span> <span class="s">'gulp-mocha'</span>
<span class="nv">sources =</span>
<span class="nv">tests: </span><span class="s">'./specs/**/*.spec.coffee'</span>
<span class="c1"># Task to run tests with mock-couch backend</span>
<span class="nx">gulp</span><span class="p">.</span><span class="nx">task</span> <span class="s">'test'</span><span class="p">,</span> <span class="nf">-></span>
<span class="nv">process.env.NODE_ENV = </span><span class="s">'test'</span>
<span class="nv">process.env.PORT = </span><span class="mi">3010</span>
<span class="nv">process.env.COUCH_PORT = </span><span class="mi">5987</span>
<span class="nv">process.env.DATABASE = </span><span class="s">'sofas_test'</span>
<span class="nx">gulp</span><span class="p">.</span><span class="nx">src</span><span class="p">(</span><span class="nx">sources</span><span class="p">.</span><span class="nx">tests</span><span class="p">)</span>
<span class="p">.</span><span class="nx">pipe</span><span class="p">(</span><span class="nx">mocha</span><span class="p">())</span>
<span class="c1"># Task to run integration tests (real CouchDB backend)</span>
<span class="nx">gulp</span><span class="p">.</span><span class="nx">task</span> <span class="s">'test-int'</span><span class="p">,</span> <span class="nf">-></span>
<span class="nv">process.env.NODE_ENV = </span><span class="s">'test-int'</span>
<span class="nv">process.env.PORT = </span><span class="mi">3010</span>
<span class="nv">process.env.COUCH_PORT = </span><span class="mi">5984</span>
<span class="nv">process.env.DATABASE = </span><span class="s">'sofas_test'</span>
<span class="nx">gulp</span><span class="p">.</span><span class="nx">src</span><span class="p">(</span><span class="nx">sources</span><span class="p">.</span><span class="nx">tests</span><span class="p">)</span>
<span class="p">.</span><span class="nx">pipe</span><span class="p">(</span><span class="nx">mocha</span><span class="p">())</span>
<span class="c1"># Task to run our server</span>
<span class="nx">gulp</span><span class="p">.</span><span class="nx">task</span> <span class="s">'default'</span><span class="p">,</span> <span class="nf">-></span>
<span class="nv">process.env.NODE_ENV = </span><span class="s">'development'</span>
<span class="nv">process.env.PORT = </span><span class="mi">3000</span>
<span class="nv">process.env.COUCH_PORT = </span><span class="mi">5984</span>
<span class="nv">process.env.DATABASE = </span><span class="s">'sofas'</span>
<span class="nv">server = </span><span class="nx">require</span> <span class="s">'./server'</span>
<span class="nx">server</span><span class="p">.</span><span class="nx">listen</span><span class="p">()</span>
</code></pre></div>
<p>server.coffee</p>
<div class="highlight"><pre><span></span><code><span class="nv">express = </span><span class="nx">require</span> <span class="s">'express'</span>
<span class="nv">app = </span><span class="nx">express</span><span class="p">()</span>
<span class="nx">app</span><span class="p">.</span><span class="nx">get</span> <span class="s">'/sofas'</span><span class="p">,</span> <span class="nf">(req, res) -></span>
<span class="k">throw</span> <span class="k">new</span> <span class="nb">Error</span> <span class="s">'not implemented'</span>
<span class="nx">app</span><span class="p">.</span><span class="nx">get</span> <span class="s">'/sofas/:sofa'</span><span class="p">,</span> <span class="nf">(req, res) -></span>
<span class="k">throw</span> <span class="k">new</span> <span class="nb">Error</span> <span class="s">'not implemented'</span>
<span class="nv">exports.listen = </span><span class="nf">(callback) -></span>
<span class="nv">port = </span><span class="nx">process</span><span class="p">.</span><span class="nx">env</span><span class="p">.</span><span class="nx">PORT</span>
<span class="nx">app</span><span class="p">.</span><span class="nx">listen</span> <span class="nx">port</span><span class="p">,</span> <span class="nx">callback</span>
</code></pre></div>
<p>Let’s create file <em>specs/sofas.spec.coffee</em> which is going to contain our tests.</p>
<div class="highlight"><pre><span></span><code><span class="nv">should = </span><span class="nx">require</span> <span class="s">'should'</span>
<span class="nx">describe</span> <span class="s">'Sofas API'</span><span class="p">,</span> <span class="nf">-></span>
<span class="nx">describe</span> <span class="s">'/sofas'</span><span class="p">,</span> <span class="nf">-></span>
<span class="nx">it</span> <span class="s">'should return all sofas with GET'</span><span class="p">,</span> <span class="nf">-></span>
<span class="c1"># TODO: write test</span>
<span class="p">(</span><span class="kc">false</span><span class="p">).</span><span class="nx">should</span><span class="p">.</span><span class="nx">be</span><span class="p">.</span><span class="nx">true</span><span class="p">()</span>
<span class="nx">describe</span> <span class="s">'/sofas/:sofa'</span><span class="p">,</span> <span class="nf">-></span>
<span class="nx">it</span> <span class="s">'should return a particular sofa with GET'</span><span class="p">,</span> <span class="nf">-></span>
<span class="c1"># TODO: write test</span>
<span class="p">(</span><span class="kc">false</span><span class="p">).</span><span class="nx">should</span><span class="p">.</span><span class="nx">be</span><span class="p">.</span><span class="nx">true</span><span class="p">()</span>
</code></pre></div>
<p>If you run the tests right now, you should see similar output:</p>
<div class="highlight"><pre><span></span><code>gulp <span class="nb">test</span>
Sofas API
/sofas
<span class="m">1</span><span class="o">)</span> should <span class="k">return</span> all sofas with GET
/sofas/:sofa
<span class="m">2</span><span class="o">)</span> should <span class="k">return</span> a particular sofa with GET
<span class="m">0</span> passing <span class="o">(</span>30ms<span class="o">)</span>
<span class="m">2</span> failing
<span class="m">1</span><span class="o">)</span> Sofas API /sofas should <span class="k">return</span> all sofas with GET:
AssertionError: expected <span class="nb">false</span> to be <span class="nb">true</span>
at Context.<anonymous> <span class="o">(</span>specs/sofas.spec.coffee:8:7<span class="o">)</span>
<span class="m">2</span><span class="o">)</span> Sofas API /sofas/:sofa should <span class="k">return</span> a particular sofa with GET:
AssertionError: expected <span class="nb">false</span> to be <span class="nb">true</span>
at Context.<anonymous> <span class="o">(</span>specs/sofas.spec.coffee:13:7<span class="o">)</span>
Message:
<span class="m">2</span> tests failed.
</code></pre></div>
<h2>Implementation</h2>
<p>Sofas will be stored in CouchDB with the following structure:</p>
<div class="highlight"><pre><span></span><code><span class="p">{</span>
<span class="nt">"_id"</span><span class="p">:</span> <span class="s2">"CouchDB assigned id"</span><span class="p">,</span>
<span class="nt">"_rev"</span><span class="p">:</span> <span class="s2">"CouchDB assigned revision"</span><span class="p">,</span>
<span class="nt">"name"</span><span class="p">:</span> <span class="s2">"Name of the sofa"</span><span class="p">,</span>
<span class="nt">"price"</span><span class="p">:</span> <span class="s2">"Price of the sofa"</span>
<span class="p">}</span>
</code></pre></div>
<p>With this in mind, let’s build a module for accessing the database.</p>
<div class="highlight"><pre><span></span><code><span class="nv">nano = </span><span class="nx">require</span> <span class="s">'nano'</span>
<span class="nv">Q = </span><span class="nx">require</span> <span class="s">'q'</span>
<span class="k">class</span> <span class="nx">DAL</span>
<span class="nv">constructor: </span><span class="nf">(port, @database) -></span>
<span class="vi">@conn = </span><span class="nx">nano</span><span class="p">(</span><span class="s">"http://localhost:</span><span class="si">#{</span><span class="nx">port</span><span class="si">}</span><span class="s">"</span><span class="p">)</span>
<span class="c1"># Returns a promise reseting database</span>
<span class="nv">resetDatabase: </span><span class="nf">(withSamples=false) -></span>
<span class="nv">docs = </span><span class="p">[</span><span class="nx">@_sofaViews</span><span class="p">]</span>
<span class="k">if</span> <span class="nx">withSamples</span>
<span class="nv">docs = </span><span class="nx">docs</span><span class="p">.</span><span class="nx">concat</span> <span class="nx">@_sofaSamples</span>
<span class="nx">@_deleteDb</span><span class="p">()</span>
<span class="p">.</span><span class="nx">then</span> <span class="nf">=></span> <span class="nx">@_createDb</span><span class="p">()</span>
<span class="p">.</span><span class="nx">then</span> <span class="nf">=></span> <span class="nx">Q</span><span class="p">.</span><span class="nx">all</span><span class="p">(</span><span class="nx">docs</span><span class="p">.</span><span class="nx">map</span> <span class="nf">(d) =></span> <span class="nx">@_insertDoc</span><span class="p">(</span><span class="nx">d</span><span class="p">))</span>
<span class="c1"># Returns a promise of sofas</span>
<span class="c1"># If you provide the id- you'll get an array with a particular sofa in it</span>
<span class="nv">getSofas: </span><span class="nf">(id=null) -></span>
<span class="nv">params = </span><span class="p">{}</span>
<span class="nv">params.key = </span><span class="nx">id</span> <span class="k">if</span> <span class="nx">id</span><span class="o">?</span>
<span class="nx">@_view</span><span class="p">(</span><span class="s">'sofas'</span><span class="p">,</span> <span class="s">'all'</span><span class="p">,</span> <span class="nx">params</span><span class="p">)</span>
<span class="p">.</span><span class="nx">then</span> <span class="nf">(res) -></span>
<span class="nv">body = </span><span class="nx">res</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="nv">sofas = </span><span class="p">(</span><span class="nx">row</span><span class="p">.</span><span class="nx">value</span> <span class="k">for</span> <span class="nx">row</span> <span class="k">in</span> <span class="nx">body</span><span class="p">.</span><span class="nx">rows</span><span class="p">)</span>
<span class="k">return</span> <span class="nx">sofas</span>
<span class="nv">_sofaViews:</span>
<span class="nv">_id: </span><span class="s">'_design/sofas'</span>
<span class="nv">views:</span>
<span class="nv">all:</span>
<span class="nv">map: </span><span class="nf">(doc) -></span>
<span class="k">if</span> <span class="nx">doc</span><span class="p">.</span><span class="nx">type</span> <span class="o">is</span> <span class="s">'Sofa'</span>
<span class="nx">emit</span> <span class="nx">doc</span><span class="p">.</span><span class="nx">_id</span><span class="p">,</span> <span class="nx">doc</span>
<span class="nv">_sofaSamples: </span><span class="p">[</span>
<span class="p">{</span>
<span class="nv">_id: </span><span class="s">'sofa1'</span>
<span class="nv">type: </span><span class="s">'Sofa'</span>
<span class="nv">name: </span><span class="s">'Sofa1'</span>
<span class="nv">price: </span><span class="mi">400</span>
<span class="p">}</span>
<span class="p">{</span>
<span class="nv">_id: </span><span class="s">'sofa2'</span>
<span class="nv">type: </span><span class="s">'Sofa'</span>
<span class="nv">name: </span><span class="s">'Sofa2'</span>
<span class="nv">price: </span><span class="mi">300</span>
<span class="p">}</span>
<span class="p">{</span>
<span class="nv">_id: </span><span class="s">'sofa3'</span>
<span class="nv">type: </span><span class="s">'Sofa'</span>
<span class="nv">name: </span><span class="s">'Sofa3'</span>
<span class="nv">price: </span><span class="mi">500</span>
<span class="p">}</span>
<span class="p">]</span>
<span class="c1"># Returns promise of creating database</span>
<span class="nv">_createDb: </span><span class="nf">-></span>
<span class="nv">d = </span><span class="nx">Q</span><span class="p">.</span><span class="nx">defer</span><span class="p">()</span>
<span class="nx">@conn</span><span class="p">.</span><span class="nx">db</span><span class="p">.</span><span class="nx">create</span> <span class="nx">@database</span><span class="p">,</span> <span class="nx">d</span><span class="p">.</span><span class="nx">makeNodeResolver</span><span class="p">()</span>
<span class="nx">d</span><span class="p">.</span><span class="nx">promise</span>
<span class="c1"># Returns promise of deleting database</span>
<span class="nv">_deleteDb: </span><span class="nf">(ignoreMissing=true)-></span>
<span class="nv">d = </span><span class="nx">Q</span><span class="p">.</span><span class="nx">defer</span><span class="p">()</span>
<span class="nx">@conn</span><span class="p">.</span><span class="nx">db</span><span class="p">.</span><span class="nx">destroy</span> <span class="nx">@database</span><span class="p">,</span> <span class="nf">(err, res) -></span>
<span class="nv">err = </span><span class="kc">null</span> <span class="k">if</span> <span class="nx">err</span><span class="o">?</span> <span class="o">and</span> <span class="nx">err</span><span class="p">.</span><span class="nx">statusCode</span> <span class="o">is</span> <span class="mi">404</span> <span class="o">and</span> <span class="nx">ignoreMissing</span>
<span class="nx">d</span><span class="p">.</span><span class="nx">makeNodeResolver</span><span class="p">()(</span><span class="nx">err</span><span class="p">,</span> <span class="nx">res</span><span class="p">)</span>
<span class="nx">d</span><span class="p">.</span><span class="nx">promise</span>
<span class="c1"># Returns promise of inserting a doc</span>
<span class="nv">_insertDoc: </span><span class="nf">(doc) -></span>
<span class="nv">d = </span><span class="nx">Q</span><span class="p">.</span><span class="nx">defer</span><span class="p">()</span>
<span class="nx">@conn</span><span class="p">.</span><span class="nx">use</span><span class="p">(</span><span class="nx">@database</span><span class="p">).</span><span class="nx">insert</span> <span class="nx">doc</span><span class="p">,</span> <span class="nx">d</span><span class="p">.</span><span class="nx">makeNodeResolver</span><span class="p">()</span>
<span class="nx">d</span><span class="p">.</span><span class="nx">promise</span>
<span class="c1"># Returns promise of CouchDB view</span>
<span class="nv">_view: </span><span class="nf">(designDoc, view, params={}) -></span>
<span class="nv">d = </span><span class="nx">Q</span><span class="p">.</span><span class="nx">defer</span><span class="p">()</span>
<span class="nx">@conn</span><span class="p">.</span><span class="nx">use</span><span class="p">(</span><span class="nx">@database</span><span class="p">).</span><span class="nx">view</span> <span class="nx">designDoc</span><span class="p">,</span> <span class="nx">view</span><span class="p">,</span> <span class="nx">params</span><span class="p">,</span> <span class="nx">d</span><span class="p">.</span><span class="nx">makeNodeResolver</span><span class="p">()</span>
<span class="nx">d</span><span class="p">.</span><span class="nx">promise</span>
<span class="nv">module.exports = </span><span class="nx">DAL</span>
</code></pre></div>
<p>What’s missing is the endpoint and test implementation. Let’s start with the tests.</p>
<div class="highlight"><pre><span></span><code><span class="gh">diff --git a/specs/sofas.spec.coffee b/specs/sofas.spec.coffee</span>
<span class="gh">index 8c1eb84..efbff58 100644</span>
<span class="gd">--- a/specs/sofas.spec.coffee</span>
<span class="gi">+++ b/specs/sofas.spec.coffee</span>
<span class="gu">@@ -1,13 +1,54 @@</span>
<span class="gd">-should = require 'should'</span>
<span class="gi">+should = require 'should'</span>
<span class="gi">+DAL = require '../dal'</span>
<span class="gi">+request = require 'supertest'</span>
<span class="gi">+request = request "http://localhost:#{process.env.PORT}"</span>
<span class="gi">+server = require '../server'</span>
<span class="gi">+</span>
<span class="gi">+app = null</span>
describe 'Sofas API', ->
<span class="gi">+ beforeEach (done) -></span>
<span class="gi">+ # Reset database and start express server before each test</span>
<span class="gi">+ dal = new DAL(process.env.COUCH_PORT, process.env.DATABASE)</span>
<span class="gi">+ dal.resetDatabase(true)</span>
<span class="gi">+ .then -></span>
<span class="gi">+ app = server.listen done</span>
<span class="gi">+ .fail (err) -> done err</span>
<span class="gi">+</span>
<span class="gi">+ # Close express server after each test</span>
<span class="gi">+ afterEach (done) -></span>
<span class="gi">+ app.close done</span>
<span class="gi">+</span>
<span class="gi">+</span>
describe '/sofas', ->
<span class="gd">- it 'should return all sofas with GET', -></span>
<span class="gd">- # TODO: write test</span>
<span class="gd">- (false).should.be.true()</span>
<span class="gi">+ it 'should return all sofas with GET', (done) -></span>
<span class="gi">+ request</span>
<span class="gi">+ .get '/sofas'</span>
<span class="gi">+ .expect 200</span>
<span class="gi">+ .expect (res) -></span>
<span class="gi">+ sofas = res.body</span>
<span class="gi">+ sofas.should.have.length(3)</span>
<span class="gi">+ for s in sofas</span>
<span class="gi">+ s.should.have.property('_id')</span>
<span class="gi">+ s.should.have.property('_rev')</span>
<span class="gi">+ s.should.have.property('name')</span>
<span class="gi">+ s.should.have.property('price')</span>
<span class="gi">+</span>
<span class="gi">+ .end done</span>
<span class="gi">+ </span>
<span class="gi">+</span>
describe '/sofas/:sofa', ->
<span class="gd">- it 'should return a particular sofa with GET', -></span>
<span class="gd">- # TODO: write test</span>
<span class="gd">- (false).should.be.true()</span>
<span class="gi">+ it 'should return a particular sofa with GET', (done) -></span>
<span class="gi">+ request</span>
<span class="gi">+ .get '/sofas/sofa1'</span>
<span class="gi">+ .expect 200</span>
<span class="gi">+ .expect (res) -></span>
<span class="gi">+ sofa = res.body</span>
<span class="gi">+ sofa.should.have.property('_id')</span>
<span class="gi">+ sofa.should.have.property('_rev')</span>
<span class="gi">+ sofa.should.have.property('name')</span>
<span class="gi">+ sofa.should.have.property('price')</span>
<span class="gi">+</span>
<span class="gi">+ .end done</span>
</code></pre></div>
<p>Now the endpoints:</p>
<div class="highlight"><pre><span></span><code><span class="gh">diff --git a/server.coffee b/server.coffee</span>
<span class="gh">index 98f1918..a0d30ce 100644</span>
<span class="gd">--- a/server.coffee</span>
<span class="gi">+++ b/server.coffee</span>
<span class="gu">@@ -1,16 +1,27 @@</span>
express = require 'express'
<span class="gi">+DAL = require './dal'</span>
<span class="gi">+dal = new DAL(process.env.COUCH_PORT, process.env.DATABASE)</span>
app = express()
app.get '/sofas', (req, res) ->
<span class="gd">- throw new Error 'not implemented'</span>
<span class="gi">+ dal.getSofas()</span>
<span class="gi">+ .then (sofas) -></span>
<span class="gi">+ res.json(sofas)</span>
<span class="gi">+ .fail (err) -> res.status(err.statusCode).send()</span>
app.get '/sofas/:sofa', (req, res) ->
<span class="gd">- throw new Error 'not implemented'</span>
<span class="gi">+ dal.getSofas(req.params.sofa)</span>
<span class="gi">+ .then (sofas) -></span>
<span class="gi">+ if sofas.length isnt 1</span>
<span class="gi">+ res.status(404).send()</span>
<span class="gi">+ else</span>
<span class="gi">+ res.json(sofas[0])</span>
<span class="gi">+ .fail (err) -> res.status(err.statusCode).send()</span>
exports.listen = (callback) ->
</code></pre></div>
<p>At this point we can execute <em>gulp test-int</em> command to run tests on real CouchDB:</p>
<div class="highlight"><pre><span></span><code>gulp test-int
Sofas API
/sofas
✓ should <span class="k">return</span> all sofas with GET
/sofas/:sofa
✓ should <span class="k">return</span> a particular sofa with GET
<span class="m">2</span> passing <span class="o">(</span>204ms<span class="o">)</span>
</code></pre></div>
<h2>mock-couch tests</h2>
<p>So how do we use mock-couch for this? Mock-couch is an <span class="caps">HTTP</span> server based on <a href="http://mcavage.me/node-restify/">Restify</a>. Note that <a href="https://nodejs.org/">Node.js</a> runtime is single-threaded therefore we won’t be able to run both express and mock-couch at the same time. However, it is possible to run mock-couch on another process.</p>
<p>Node provides a couple of ways to create processes out of the box. We’ll use one called <a href="https://nodejs.org/api/child_process.html#child_process_child_process_fork_modulepath_args_options">forking</a>: not only we’ll be able to control the lifecycle of the forked child process but we’ll also have the ability to send and receive messages between parent and the child.</p>
<p><span class="caps">OK</span>, less talk and more forking. We’ll do the forking in <em>gulp test</em> task:</p>
<div class="highlight"><pre><span></span><code><span class="gh">diff --git a/gulpfile.coffee b/gulpfile.coffee</span>
<span class="gh">index cbd61b5..1200834 100644</span>
<span class="gd">--- a/gulpfile.coffee</span>
<span class="gi">+++ b/gulpfile.coffee</span>
<span class="gu">@@ -6,14 +6,48 @@ mocha = require 'gulp-mocha'</span>
sources =
tests: './specs/**/*.spec.coffee'
<span class="gi">+</span>
# Task to run tests with mock-couch backend
gulp.task 'test', ->
process.env.NODE_ENV = 'test'
process.env.PORT = 3010
process.env.COUCH_PORT = 5987
process.env.DATABASE = 'sofas_test'
<span class="gd">- gulp.src(sources.tests)</span>
<span class="gd">- .pipe(mocha())</span>
<span class="gi">+</span>
<span class="gi">+ fork = require('child_process').fork</span>
<span class="gi">+ couchProcess = fork('node_modules/gulp/bin/gulp.js', ['mock-couch'], {cwd: __dirname})</span>
<span class="gi">+ couchProcess.on 'message', (msg) =></span>
<span class="gi">+ if msg is 'listening'</span>
<span class="gi">+ gulp.src(sources.tests)</span>
<span class="gi">+ .pipe(mocha())</span>
<span class="gi">+ .once('error', (err) -></span>
<span class="gi">+ process.exit(1)</span>
<span class="gi">+ )</span>
<span class="gi">+ .once('end', -></span>
<span class="gi">+ process.exit()</span>
<span class="gi">+ )</span>
<span class="gi">+</span>
<span class="gi">+ process.on 'exit', -></span>
<span class="gi">+ couchProcess.kill()</span>
<span class="gi">+</span>
<span class="gi">+gulp.task 'mock-couch', -></span>
<span class="gi">+ process.env.NODE_ENV = 'test'</span>
<span class="gi">+ process.env.PORT = 3010</span>
<span class="gi">+ process.env.COUCH_PORT = 5987</span>
<span class="gi">+ process.env.DATABASE = 'sofas_test'</span>
<span class="gi">+</span>
<span class="gi">+ mockCouch = require 'mock-couch'</span>
<span class="gi">+ couchdb = mockCouch.createServer()</span>
<span class="gi">+ DAL = require './dal'</span>
<span class="gi">+ dal = new DAL(process.env.COUCH_PORT, process.env.DATABASE)</span>
<span class="gi">+ docs = [dal._sofaViews]</span>
<span class="gi">+ docs = docs.concat dal._sofaSamples</span>
<span class="gi">+</span>
<span class="gi">+ couchdb.addDB process.env.DATABASE, docs</span>
<span class="gi">+</span>
<span class="gi">+ couchdb.listen process.env.COUCH_PORT, -></span>
<span class="gi">+ process.send 'listening'</span>
<span class="gi">+</span>
# Task to run integration tests (real CouchDB backend)
gulp.task 'test-int', ->
</code></pre></div>
<p>Also, we need a slight change to the test setup:</p>
<div class="highlight"><pre><span></span><code><span class="gh">diff --git a/specs/sofas.spec.coffee b/specs/sofas.spec.coffee</span>
<span class="gh">index efbff58..93e7bd2 100644</span>
<span class="gd">--- a/specs/sofas.spec.coffee</span>
<span class="gi">+++ b/specs/sofas.spec.coffee</span>
<span class="gu">@@ -9,12 +9,17 @@ app = null</span>
describe 'Sofas API', ->
beforeEach (done) ->
<span class="gd">- # Reset database and start express server before each test</span>
<span class="gd">- dal = new DAL(process.env.COUCH_PORT, process.env.DATABASE)</span>
<span class="gd">- dal.resetDatabase(true)</span>
<span class="gd">- .then -></span>
<span class="gd">- app = server.listen done</span>
<span class="gd">- .fail (err) -> done err</span>
<span class="gi">+</span>
<span class="gi">+ if process.env.NODE_ENV is 'test-int'</span>
<span class="gi">+ # Reset database and start express server before each test</span>
<span class="gi">+ dal = new DAL(process.env.COUCH_PORT, process.env.DATABASE)</span>
<span class="gi">+ dal.resetDatabase(true)</span>
<span class="gi">+ .then -></span>
<span class="gi">+ app = server.listen done</span>
<span class="gi">+ .fail (err) -> done err</span>
<span class="gi">+ else</span>
<span class="gi">+ app = server.listen done</span>
<span class="gi">+</span>
# Close express server after each test
afterEach (done) ->
</code></pre></div>
<h2>Moment of Truth</h2>
<p>If you try running <code>gulp test</code> right now, you should see similar output:</p>
<div class="highlight"><pre><span></span><code>gulp <span class="nb">test</span>
Sofas API
/sofas
✓ should <span class="k">return</span> all sofas with GET <span class="o">(</span>125ms<span class="o">)</span>
/sofas/:sofa
✓ should <span class="k">return</span> a particular sofa with GET <span class="o">(</span>41ms<span class="o">)</span>
<span class="m">2</span> passing <span class="o">(</span>178ms<span class="o">)</span>
</code></pre></div>
<p>As you can see the tests completed slightly faster than with real CouchDB. In a small code base like this, it’s hardly worth switching to mock-couch, if execution speed is your only motivation. However, with bigger codebases, the speed improvement can become very apparent.</p>
<p>Besides speed, there’s another benefit to this approach: you don’t actually need CouchDB to test your project. The simplified build process is especially useful with Continuous Integration platforms like <a href="http://travis-ci.org/">Travis <span class="caps">CI</span></a> and <a href="https://codeship.com">Codeship</a>.</p>
<h2>Resources</h2>
<ul>
<li><a href="https://github.com/reederz/express-mock-couch-showcase">Code on GitHub</a></li>
<li><a href="https://github.com/chris-l/mock-couch">mock-couch</a></li>
</ul>I got hired by Toptal2016-09-13T13:30:00+02:002016-09-13T13:30:00+02:00Justas Ažnatag:reederz.com,2016-09-13:/got-hired-by-toptal.html<p class="first last">I became a freelance software engineer at toptal.com</p>
<p>A few weeks ago, I passed my last screening stage for <a class="reference external" href="https://www.toptal.com/#select-just-spectacular-programmers-now">toptal.com</a> software freelancing platform. Unlike other similar platforms, such as <a class="reference external" href="https://upwork.com">upwork</a> and <a class="reference external" href="https://freelancer.com">freelancer.com</a>, this one is pretty hard to get in. They hire top 3% of freelance talent and they have a tough interviewing process to ensure that.</p>
<p>First of all, why would you want to be a part of toptal? I’ve tried a couple other freelancing platforms and I didn’t really like them that much. Since there is little to no skill assesment of the freelancers and a huge competition, you have to invest a lot of effort into justifying your skills to the clients every single time. Toptal is different. Because of the hard screening process, merely by passing it you have proven that you are good at what you do (and the clients know that).</p>
<p>The screening process itself has <a class="reference external" href="https://www.toptal.com/top-3-percent#select-just-spectacular-programmers-now">4 stages</a>:</p>
<ol class="arabic simple">
<li>Skype interview with an <span class="caps">HR</span> person to assess your English skills and personality.</li>
<li>Timed algorithm test: 3 coding exercises in 90 minutes on <a class="reference external" href="https://codility.com">codility</a> platform to assess your problem solving and algorithm skills.</li>
<li>Live technical screening with a Sr. Engineer where you are asked technical questions and have to do live programming.</li>
<li>A test project to prove that you are capable of working on a real life project. The project takes around 2 weeks to complete after which you have to present it to “the client” (one of toptal engineers).</li>
</ol>
<p>The stages 2-4 were the most challenging and I’ve initially failed the 3rd stage because I ran out of time when doing the live coding. Fortunately, they let you retry it after 1 month and I was luckier with my second attempt.</p>
<p>Once I got accepted and setup <a class="reference external" href="https://www.toptal.com/resume/justas-azna#select-just-spectacular-programmers-now">my profile</a> I could apply for jobs (yeah, you still have to do that). But since I was busy at the moment, I put it on hold. Even so, a few weeks later one of the recruiters found me an interview and after talking to the client I got hired as a Senior Software Developer/Architect.</p>
<p>All in all, my short experience with Toptal is very good and I’m happy to have joined their network. Go ahead and check them out, if you are <a class="reference external" href="https://www.toptal.com/talent/apply#select-just-spectacular-programmers-now">looking for challening and exciting software projects</a> or <a class="reference external" href="https://www.toptal.com/companies/apply#select-just-spectacular-programmers-now">are in need of help with one</a>.</p>
Coursera - Cryptography I2015-07-07T11:30:00+02:002015-07-07T11:30:00+02:00Justas Ažnatag:reederz.com,2015-07-07:/coursera-crypto1.html<p class="first last">I have recently finished Cryptography I - an online course by Prof. Dan Boneh on Coursera.org. Here’s what I have learnt.</p>
<p>I have recently finished <a class="reference external" href="https://www.coursera.org/course/crypto">Cryptography I</a> - an introductory online course on Cryptography by Prof. Dan Boneh of Stanford and today I received my fancy <a class="reference external" href="/docs/coursera/crypto1.pdf">Statement of Accomplishment</a>. It was an interesting and challenging experience.</p>
<p>The course took 6 weeks and covered Stream and Block Ciphers, Message Integrity, Authenticated Encryption, Basic Key Exchange and Public Key Encryption. We have discussed several formal security definitions such as Semantic Security, Chosen Plaintext Security, Chosen Ciphertext Security, Collision Resistance; each of which were used to describe security of different cryptographic primitives.</p>
<p>Each week we would have a problem set which usually consisted of multiple-choice questions and an optional programming assignment. The programming assignments were especially great: we got to break poorly used cryptography and practice using various crypto algorithms I.e. we have broken many-time-pad, exploited <span class="caps">CBC</span> padding oracle vulnerabilities, factored <span class="caps">RSA</span> modulus with poorly generated prime factors and more.</p>
<p>At the end of 6 weeks we had the final exam where we reviewed what we have done throughout the course.</p>
<p>The main takeaways for me are:</p>
<ul class="simple">
<li>never roll your own crypto</li>
<li>don’t even try to implement crypto standards- stick to standard, peer-reviewed and battle-tested implementations</li>
<li>paranoia level up</li>
</ul>
<p>While this course is really fun, it’s not something that you can do casually. It would take me 6-12 quality hours every week to watch the lectures and finish the course work. It’s actually my second attempt- the first time I gave up because my schedule was too hectic and I didn’t want to invest half ass effort into this (all or nothing :).</p>
<p>Before crypto1, I had a course on computer security in college, read books like <a class="reference external" href="https://www.goodreads.com/book/show/29608.The_Codebreakers">The Codebreakers by David Khan</a> and <a class="reference external" href="https://www.goodreads.com/book/show/9319468-kingpin">Kingpin by Kevin Poulsen</a>, dabbled with hacking challenges on <a class="reference external" href="https://www.hackthissite.org/">HackThisSite</a> and had some experience from my sysadmin/webdev work. Shortly: I knew a bit more than average Joe. Well, this course made me feel like I know nothing about crypto (in a good way) and that was a really sobering experience. It’s definitely going to affect my decisions both as webdev and as a web citizen.</p>
<p>As a final note, I want to thank Prof. Dan Boneh- you really know your stuff and I admire your work. I look forward to <a class="reference external" href="https://www.coursera.org/course/crypto2">Cryptography <span class="caps">II</span></a> in October.</p>