Jekyll2021-10-31T00:18:03+00:00https://deaddesk.top/feed.xmlDeadDesk.topJust Another IT BlogPiotr Szymuraundernotic@gmail.comServerless for dummies - pragmatic encounter on azure2020-07-16T00:00:00+00:002020-07-16T00:00:00+00:00https://deaddesk.top/serverless-for-dummies-pragmatic-encounter<h2 id="here-i-am-without-the-server">Here I am without the server</h2>
<p>Serverless in on the rise. Gaining a lot of traction to push new abilities, features and there is more and more hosting/deployment options.</p>
<p>It is not only bound to public cloud providers but it’s also getting more popular on on-premise environments where you can tune it to your needs.
This is backed up by frameworks like OpenFAAS, Kubeless, which are often build in open source world.</p>
<blockquote>
<p>Main idea of serverless is to not worry about infrastructure technicalities (no server) and instead focus on solving real problems/writing code.</p>
</blockquote>
<p>Serverless code generally is short-living and needs to be stateless, meaning state should be put into/handled by external system, database, etc.
Thus it is harder/trickier to do small stateful things like connection pooling (i.e. connections pooling to sql databases).</p>
<figure class="align-center" style="width: 75%">
<img src="https://deaddesk.top/assets/images/serverless-for-dummies/pros.jpeg" alt="" />
<figcaption>Best place for servers is in the trash.</figcaption>
</figure>
<hr />
<p>When thinking about it in context of cloud platforms (azure, aws), payment model is very interesting:</p>
<ul>
<li>pay as you go</li>
<li>scale as you go</li>
</ul>
<p>On other hand there are some limitations:</p>
<ul>
<li>limited message size</li>
<li>limited active connection time(only request reply)</li>
<li>limited number of concurrent connections</li>
</ul>
<h2 id="but-what-can-i-build-">But What can I build ?</h2>
<blockquote>
<p>Typical question is what I can build it with it and answer to that is typical <code class="language-plaintext highlighter-rouge">it depends</code> or <code class="language-plaintext highlighter-rouge">everything</code>.</p>
</blockquote>
<p>Most of the time serverless is used as a additional layer on top of typical server-full apps.
It fullfils role to solve little problems, threat serverless function as single purpose endpoint that solves one problem.
Then compose those little problem-solvers (functions) together to create something <code class="language-plaintext highlighter-rouge">bigger</code>.</p>
<h2 id="not-another-hello-world-example">Not another hello world example</h2>
<p>Let’s create something that might be useful.</p>
<blockquote>
<p>Function that will accept as a parameter <code class="language-plaintext highlighter-rouge">id of spotify playlist</code> and return out of it list of songs together with youtube link.</p>
</blockquote>
<p>Here is the final result:</p>
<figure class="align-center" style="width: 80%">
<img src="https://deaddesk.top/assets/images/serverless-for-dummies/app.jpeg" alt="" />
<figcaption>Final look of the app in the browser.</figcaption>
</figure>
<p>You can experience that by executing, although it might return 500 if youtube api limits were exceeded:</p>
<p><a href="https://spotifytranslatorfunctionapp.azurewebsites.net/api/map?playlist=2GK6j1Wh1NfmIPcFVXC97t" target="_blank">https://spotifytranslatorfunctionapp.azurewebsites.net/api/map?playlist=2GK6j1Wh1NfmIPcFVXC97t</a></p>
<p>Where:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">playlist=</code> is a id of spotify playlist</li>
</ul>
<p class="notice--warning">Please note i will focus on local development, thus i won’t go through <code class="language-plaintext highlighter-rouge">azure portal</code> and creating/managing/deploying stuff there.<br />
Never the less this of course can be deployed to cloud without a problem.</p>
<h2 id="what-we-need-">What we need ?</h2>
<p>We will use azure, below are prerequisites:</p>
<ul>
<li>Setup <code class="language-plaintext highlighter-rouge">.net core</code> (version 3.1 >= required).</li>
<li>Install <code class="language-plaintext highlighter-rouge">Azure Functions Core Tools</code> from <a href="https://docs.microsoft.com/en-us/azure/azure-functions/functions-run-local" target="_blank">here.</a></li>
<li>We will need to connect to api of spotify and youtube. Since only limited usage of these api’s is free, we need to create <code class="language-plaintext highlighter-rouge">youtube app</code> and <code class="language-plaintext highlighter-rouge">spotify app</code> so our usage can be tracked.</li>
</ul>
<h2 id="scaffolding">Scaffolding</h2>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nb">mkdir </span>SpotifyToYoutubeTranslator <span class="o">&&</span> <span class="nb">cd</span> <span class="nv">$_</span>
func init <span class="nt">--worker-runtime</span> dotnet
func new <span class="nt">--language</span> C# <span class="nt">--name</span> SpotifyToYoutubeTranslator
</code></pre></div></div>
<p><img src="/assets/images/serverless-for-dummies/serverless-setup.gif" alt="full" class="full" /></p>
<h2 id="code-walkthrough">Code walkthrough</h2>
<p>Entire code is available <a href="https://github.com/UnderNotic/spotify-to-youtube-translator" target="_blank">here.</a>
It’s splitted into 3 major sections:</p>
<figure>
<a href="/assets/images/serverless-for-dummies/diagram.jpeg"><img src="/assets/images/serverless-for-dummies/diagram.jpeg" /></a>
</figure>
<h2 id="before-we-begin">Before we begin</h2>
<p>Youtube/spotify application keys need to be stored somewhere safe. Those are secrets that authenticate our app, that can not be compromised.
We will use simple approach, which is <code class="language-plaintext highlighter-rouge">local.settings.json</code>.</p>
<p class="notice--warning">Cloud version is using <code class="language-plaintext highlighter-rouge">azure application settings</code>, as secrets from <code class="language-plaintext highlighter-rouge">local.settings.json</code> are of course not commited.</p>
<p>Then in code these will be read as follow.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">private</span> <span class="k">static</span> <span class="k">readonly</span> <span class="kt">string</span> <span class="n">SPOTIFY_APP_KEY</span> <span class="p">=</span>
<span class="n">System</span><span class="p">.</span><span class="n">Environment</span><span class="p">.</span><span class="nf">GetEnvironmentVariable</span><span class="p">(</span><span class="s">"SPOTIFY_APP_KEY"</span><span class="p">);</span>
<span class="k">private</span> <span class="k">static</span> <span class="k">readonly</span> <span class="kt">string</span> <span class="n">YOUTUBE_APP_KEY</span> <span class="p">=</span>
<span class="n">System</span><span class="p">.</span><span class="n">Environment</span><span class="p">.</span><span class="nf">GetEnvironmentVariable</span><span class="p">(</span><span class="s">"YOUTUBE_APP_KEY"</span><span class="p">);</span>
</code></pre></div></div>
<p class="notice--info">If something more advanced is needed, have a look at <code class="language-plaintext highlighter-rouge">azure key-vault</code>.
It is ment to store secrets but with more features on top of it :)</p>
<h2 id="section-1"><b style="color: #ffce2f">Section #1</b></h2>
<p>First we need to know what playlist we want to take from spotify.
This is easy as getting <code class="language-plaintext highlighter-rouge">playlist id</code> from incoming request query parameters.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">var</span> <span class="n">playlist</span> <span class="p">=</span> <span class="n">req</span><span class="p">.</span><span class="n">Query</span><span class="p">[</span><span class="s">"playlist"</span><span class="p">];</span>
</code></pre></div></div>
<p>If there is no cached spotify token or it expired, a new one is obtained and cached.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">private</span> <span class="k">static</span> <span class="k">async</span> <span class="n">Task</span><span class="p"><</span><span class="n">SpotifyItems</span><span class="p">></span> <span class="nf">GetSpotifyItems</span><span class="p">(</span><span class="n">HttpRequest</span> <span class="n">req</span><span class="p">,</span> <span class="kt">string</span> <span class="n">playlist</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">_spotifyToken</span> <span class="p">==</span> <span class="k">null</span> <span class="p">||</span> <span class="n">_spotifyToken</span><span class="p">.</span><span class="n">ExpiryDate</span> <span class="p"><</span> <span class="n">DateTime</span><span class="p">.</span><span class="n">UtcNow</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">fullToken</span> <span class="p">=</span> <span class="k">await</span> <span class="nf">GetSpotifyAccessToken</span><span class="p">();</span>
<span class="n">_spotifyToken</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">SpotifyToken</span><span class="p">(</span><span class="n">fullToken</span><span class="p">[</span><span class="s">"access_token"</span><span class="p">].</span><span class="nf">ToString</span><span class="p">(),</span> <span class="n">Int32</span><span class="p">.</span><span class="nf">Parse</span><span class="p">(</span><span class="n">fullToken</span><span class="p">[</span><span class="s">"expires_in"</span><span class="p">].</span><span class="nf">ToString</span><span class="p">()));</span>
<span class="p">}</span>
<span class="kt">var</span> <span class="n">token</span> <span class="p">=</span> <span class="n">_spotifyToken</span><span class="p">.</span><span class="n">Token</span><span class="p">;</span>
</code></pre></div></div>
<p>To obtain fresh token, a <code class="language-plaintext highlighter-rouge">POST</code> request with spotify app key has to be send to spotify api.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">private</span> <span class="k">static</span> <span class="k">async</span> <span class="n">Task</span><span class="p"><</span><span class="kt">dynamic</span><span class="p">></span> <span class="nf">GetSpotifyAccessToken</span><span class="p">()</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">tokenHeaders</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">FormUrlEncodedContent</span><span class="p">(</span><span class="k">new</span><span class="p">[]{</span>
<span class="k">new</span> <span class="n">KeyValuePair</span><span class="p"><</span><span class="kt">string</span><span class="p">,</span> <span class="kt">string</span><span class="p">>(</span><span class="s">"grant_type"</span><span class="p">,</span> <span class="s">"client_credentials"</span><span class="p">)</span>
<span class="p">});</span>
<span class="kt">var</span> <span class="n">tokenRequest</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">HttpRequestMessage</span><span class="p">(</span><span class="n">HttpMethod</span><span class="p">.</span><span class="n">Post</span><span class="p">,</span> <span class="s">"https://accounts.spotify.com/api/token"</span><span class="p">);</span>
<span class="n">tokenRequest</span><span class="p">.</span><span class="n">Content</span> <span class="p">=</span> <span class="n">tokenHeaders</span><span class="p">;</span>
<span class="n">tokenRequest</span><span class="p">.</span><span class="n">Headers</span><span class="p">.</span><span class="nf">Add</span><span class="p">(</span><span class="s">"Authorization"</span><span class="p">,</span> <span class="n">SPOTIFY_APP_KEY</span><span class="p">);</span>
<span class="kt">var</span> <span class="n">tokenRequestResponse</span> <span class="p">=</span> <span class="k">await</span> <span class="n">_httpClient</span><span class="p">.</span><span class="nf">SendAsync</span><span class="p">(</span><span class="n">tokenRequest</span><span class="p">);</span>
<span class="k">return</span> <span class="k">await</span> <span class="n">tokenRequestResponse</span><span class="p">.</span><span class="n">Content</span><span class="p">.</span><span class="n">ReadAsAsync</span><span class="p"><</span><span class="kt">dynamic</span><span class="p">>();</span>
<span class="p">}</span>
</code></pre></div></div>
<h2 id="section-2"><b style="color: #0036e9">Section #2</b></h2>
<p>Secondly, We need actual playlist of songs, given above <code class="language-plaintext highlighter-rouge">playlist id</code>.
For that, <code class="language-plaintext highlighter-rouge">GET</code> request is executed. In the header We pass spotify app token from previous section.</p>
<p>Query parameters contain items, that are going to be returned back by spotify api.
In this case we would have:</p>
<ul>
<li>name of a track</li>
<li>artist/s</li>
<li>name of an album</li>
</ul>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">private</span> <span class="k">static</span> <span class="k">async</span> <span class="n">Task</span><span class="p"><</span><span class="n">SpotifyItems</span><span class="p">></span> <span class="nf">GetSpotifyItems</span><span class="p">(</span><span class="n">HttpRequest</span> <span class="n">req</span><span class="p">,</span> <span class="kt">string</span> <span class="n">playlist</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">_spotifyToken</span> <span class="p">==</span> <span class="k">null</span> <span class="p">||</span> <span class="n">_spotifyToken</span><span class="p">.</span><span class="n">ExpiryDate</span> <span class="p"><</span> <span class="n">DateTime</span><span class="p">.</span><span class="n">UtcNow</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">fullToken</span> <span class="p">=</span> <span class="k">await</span> <span class="nf">GetSpotifyAccessToken</span><span class="p">();</span>
<span class="n">_spotifyToken</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">SpotifyToken</span><span class="p">(</span><span class="n">fullToken</span><span class="p">[</span><span class="s">"access_token"</span><span class="p">].</span><span class="nf">ToString</span><span class="p">(),</span> <span class="n">Int32</span><span class="p">.</span><span class="nf">Parse</span><span class="p">(</span><span class="n">fullToken</span><span class="p">[</span><span class="s">"expires_in"</span><span class="p">].</span><span class="nf">ToString</span><span class="p">()));</span>
<span class="p">}</span>
<span class="kt">var</span> <span class="n">token</span> <span class="p">=</span> <span class="n">_spotifyToken</span><span class="p">.</span><span class="n">Token</span><span class="p">;</span>
<span class="kt">var</span> <span class="n">playlistQuery</span> <span class="p">=</span> <span class="n">HttpUtility</span><span class="p">.</span><span class="nf">ParseQueryString</span><span class="p">(</span><span class="kt">string</span><span class="p">.</span><span class="n">Empty</span><span class="p">);</span>
<span class="n">playlistQuery</span><span class="p">[</span><span class="s">"fields"</span><span class="p">]</span> <span class="p">=</span> <span class="s">"items(track(name,artists, album(name))), next"</span><span class="p">;</span>
<span class="kt">string</span> <span class="n">playlistQueryString</span> <span class="p">=</span> <span class="n">playlistQuery</span><span class="p">.</span><span class="nf">ToString</span><span class="p">();</span>
<span class="kt">var</span> <span class="n">playlistRequest</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">HttpRequestMessage</span><span class="p">(</span><span class="n">HttpMethod</span><span class="p">.</span><span class="n">Get</span><span class="p">,</span> <span class="s">$"https://api.spotify.com/v1/playlists/</span><span class="p">{</span><span class="n">playlist</span><span class="p">}</span><span class="s">/tracks?</span><span class="p">{</span><span class="n">playlistQueryString</span><span class="p">}</span><span class="s">"</span><span class="p">);</span>
<span class="n">playlistRequest</span><span class="p">.</span><span class="n">Headers</span><span class="p">.</span><span class="nf">Add</span><span class="p">(</span><span class="s">"Authorization"</span><span class="p">,</span> <span class="s">$"Bearer </span><span class="p">{</span><span class="n">token</span><span class="p">}</span><span class="s">"</span><span class="p">);</span>
<span class="kt">var</span> <span class="n">playlistRequestResponse</span> <span class="p">=</span> <span class="k">await</span> <span class="n">_httpClient</span><span class="p">.</span><span class="nf">SendAsync</span><span class="p">(</span><span class="n">playlistRequest</span><span class="p">);</span>
<span class="n">playlistRequestResponse</span><span class="p">.</span><span class="nf">EnsureSuccessStatusCode</span><span class="p">();</span>
<span class="kt">var</span> <span class="n">spotifyItems</span> <span class="p">=</span> <span class="k">await</span> <span class="n">playlistRequestResponse</span><span class="p">.</span><span class="n">Content</span><span class="p">.</span><span class="n">ReadAsAsync</span><span class="p"><</span><span class="n">SpotifyItems</span><span class="p">>();</span>
<span class="k">return</span> <span class="n">spotifyItems</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<h2 id="section-3"><b style="color: #ff1b1b">Section #3</b></h2>
<p>Finally, for each song from the playlist, we need to have link to youtube video and return it as a final response.
To do so we call youtube with <code class="language-plaintext highlighter-rouge">GET</code> request.
For this request no token is needed, we just set the <code class="language-plaintext highlighter-rouge">app key</code> in query parameters.</p>
<p>Also in query params we provide:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">q</code>, search phrase <code class="language-plaintext highlighter-rouge">{artist name - track name}</code></li>
<li><code class="language-plaintext highlighter-rouge">type</code> of a resource, hardcoded to <code class="language-plaintext highlighter-rouge">video</code></li>
<li><code class="language-plaintext highlighter-rouge">part</code>, we want to get as a result, hardcoded to <code class="language-plaintext highlighter-rouge">id</code></li>
<li><code class="language-plaintext highlighter-rouge">maxResults</code>, we are interested only in the first video matching</li>
</ul>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">var</span> <span class="n">resultItems</span> <span class="p">=</span> <span class="n">spotifyItems</span><span class="p">.</span><span class="n">Items</span><span class="p">.</span><span class="nf">Select</span><span class="p">(</span><span class="k">async</span> <span class="n">track</span> <span class="p">=></span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">youtubeItems</span> <span class="p">=</span> <span class="k">await</span> <span class="nf">GetYoutubeItems</span><span class="p">(</span><span class="n">track</span><span class="p">);</span>
<span class="k">return</span> <span class="k">new</span> <span class="nf">ResultItem</span><span class="p">(</span>
<span class="n">track</span><span class="p">.</span><span class="n">Track</span><span class="p">.</span><span class="n">Name</span><span class="p">,</span>
<span class="n">track</span><span class="p">.</span><span class="n">Track</span><span class="p">.</span><span class="n">Artists</span><span class="p">[</span><span class="m">0</span><span class="p">].</span><span class="n">Name</span><span class="p">,</span>
<span class="n">track</span><span class="p">.</span><span class="n">Track</span><span class="p">.</span><span class="n">Album</span><span class="p">.</span><span class="n">Name</span><span class="p">,</span>
<span class="n">youtubeItems</span><span class="p">.</span><span class="n">Items</span><span class="p">[</span><span class="m">0</span><span class="p">].</span><span class="n">Id</span><span class="p">.</span><span class="n">VideoId</span>
<span class="p">);</span>
<span class="p">});</span>
<span class="k">return</span> <span class="p">(</span><span class="n">ActionResult</span><span class="p">)</span><span class="k">new</span> <span class="nf">OkObjectResult</span><span class="p">(</span><span class="k">await</span> <span class="n">Task</span><span class="p">.</span><span class="nf">WhenAll</span><span class="p">(</span><span class="n">resultItems</span><span class="p">));</span>
</code></pre></div></div>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">private</span> <span class="k">static</span> <span class="k">async</span> <span class="n">Task</span><span class="p"><</span><span class="n">YoutubeItems</span><span class="p">></span> <span class="nf">GetYoutubeItems</span><span class="p">(</span><span class="n">SpotifyItem</span> <span class="n">track</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">youtubeRequest</span> <span class="p">=</span>
<span class="k">new</span> <span class="nf">HttpRequestMessage</span><span class="p">(</span><span class="n">HttpMethod</span><span class="p">.</span><span class="n">Get</span><span class="p">,</span> <span class="s">$"https://www.googleapis.com/youtube/v3/search?q=</span><span class="p">{</span><span class="n">track</span><span class="p">.</span><span class="n">Track</span><span class="p">.</span><span class="n">Artists</span><span class="p">[</span><span class="m">0</span><span class="p">].</span><span class="n">Name</span><span class="p">}</span><span class="s">-</span><span class="p">{</span><span class="n">track</span><span class="p">.</span><span class="n">Track</span><span class="p">.</span><span class="n">Name</span><span class="p">}</span><span class="s">&type=video&part=id&maxResults=1&key=</span><span class="p">{</span><span class="n">YOUTUBE_APP_KEY</span><span class="p">}</span><span class="s">"</span><span class="p">);</span>
<span class="kt">var</span> <span class="n">youtubeRequestResponse</span> <span class="p">=</span> <span class="k">await</span> <span class="n">_httpClient</span><span class="p">.</span><span class="nf">SendAsync</span><span class="p">(</span><span class="n">youtubeRequest</span><span class="p">);</span>
<span class="kt">var</span> <span class="n">youtubeItems</span> <span class="p">=</span> <span class="k">await</span> <span class="n">youtubeRequestResponse</span><span class="p">.</span><span class="n">Content</span><span class="p">.</span><span class="n">ReadAsAsync</span><span class="p"><</span><span class="n">YoutubeItems</span><span class="p">>();</span>
<span class="k">return</span> <span class="n">youtubeItems</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Final result is composed, by taking <code class="language-plaintext highlighter-rouge">ids</code> returned by api and putting them as <code class="language-plaintext highlighter-rouge">v</code> query param into youtube video link.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="nf">ResultItem</span><span class="p">(</span><span class="kt">string</span> <span class="n">name</span><span class="p">,</span> <span class="kt">string</span> <span class="n">artist</span><span class="p">,</span> <span class="kt">string</span> <span class="n">album</span><span class="p">,</span> <span class="kt">string</span> <span class="n">ytId</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">Name</span> <span class="p">=</span> <span class="n">name</span><span class="p">;</span>
<span class="n">Artist</span> <span class="p">=</span> <span class="n">artist</span><span class="p">;</span>
<span class="n">Album</span> <span class="p">=</span> <span class="n">album</span><span class="p">;</span>
<span class="n">Url</span> <span class="p">=</span> <span class="s">$"https://www.youtube.com/watch?v=</span><span class="p">{</span><span class="n">ytId</span><span class="p">}</span><span class="s">"</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<h2 id="important-note---httpclient-is-in-a-static-field">Important note - HttpClient is in a static field</h2>
<p>We don’t want to create <code class="language-plaintext highlighter-rouge">HttpClient</code> for each request. Rather it should be long-lived resource, this will lower memory footprint and it will benefit from reusing tcp connections.</p>
<p>Important thing to remember is that, there won’t be one and only instance of <code class="language-plaintext highlighter-rouge">HttpClient</code>, instead each instance of a function runtime will have single <code class="language-plaintext highlighter-rouge">HttpClient</code> instance.</p>
<p class="notice--info">For more info about <code class="language-plaintext highlighter-rouge">HttpClient</code> check <a href="/http-client-explained-with-netstat/" target="_blank">this blog post.</a></p>
<h2 id="running-locally">Running locally</h2>
<p>You can run it locally by cloning <a href="https://github.com/UnderNotic/spotify-to-youtube-translator" target="_blank">repo</a> and creating config file <code class="language-plaintext highlighter-rouge">local.settings.json</code>.</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="w"> </span><span class="nl">"Values"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"AzureWebJobsStorage"</span><span class="p">:</span><span class="w"> </span><span class="s2">""</span><span class="p">,</span><span class="w">
</span><span class="nl">"FUNCTIONS_WORKER_RUNTIME"</span><span class="p">:</span><span class="w"> </span><span class="s2">"dotnet"</span><span class="p">,</span><span class="w">
</span><span class="nl">"SPOTIFY_APP_KEY"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Basic YourSpotifyAppKey"</span><span class="p">,</span><span class="w">
</span><span class="nl">"YOUTUBE_APP_KEY"</span><span class="p">:</span><span class="w"> </span><span class="s2">"YourYoutubeAppKey"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>Then just run <code class="language-plaintext highlighter-rouge">func start</code> and call <a href="http://localhost:7071/api/map?playlist=YourSpotifyPlaylistId" target="_blank">http://localhost:7071/api/map?playlist=YourSpotifyPlaylistId</a></p>
<p>Have fun :)</p>Piotr Szymuraundernotic@gmail.comCreating music app using azure functionsHttpClient explained using netstat2019-11-26T00:00:00+00:002019-11-26T00:00:00+00:00https://deaddesk.top/http-client-explained-with-netstat<h2 id="learn-http-client-once-again-this-time-via-netstat">Learn http client once again this time via netstat</h2>
<p>This is not a guide to:</p>
<blockquote>
<p>How to use HttpClient in your app ?</p>
</blockquote>
<p>But rather how to use netstat and how TCP connections are behaving in different code scenarios.</p>
<p>Below examples will showcase dotnet <code class="language-plaintext highlighter-rouge">HttpClient</code> running next to nestat being refreshed every 1sec.<br />
<code class="language-plaintext highlighter-rouge">Keep-alive</code> is set to true on default so connections will be reused. Output from netstat is filtered to show only relevant data.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>watch <span class="nt">-n</span> 1 <span class="s2">"netstat --TCP -pnW | grep 104.27"</span>
</code></pre></div></div>
<h2 id="netstat-explained">Netstat explained</h2>
<p>Every TCP connection is a pair of local and foreign address.</p>
<p>Address is combination of IP/port, where:</p>
<ul>
<li>IP is used to locate machine</li>
<li>Port is used to determine application on that machine.</li>
</ul>
<p>If machine is talking to the server, port for local address will be picked from pool of available ports. Port for foreign address will be of course port on what remote server is listening (in this case 443).</p>
<p><img src="/assets/images/http-client-explained-using-netstat/tcp-conn.png" alt="image-center" class="align-center" style="width: 75%" /></p>
<h3 id="netstat-columns">Netstat columns:</h3>
<ul>
<li><code class="language-plaintext highlighter-rouge">Local address</code> IP/port of local machine</li>
<li><code class="language-plaintext highlighter-rouge">Foreign address</code> IP/port of remote machine</li>
<li>Every connection is in some state, here are the most important:
<ul>
<li><code class="language-plaintext highlighter-rouge">ESTABLISHED</code> connection is opened after 3-way handshake and ready to transmit the data</li>
<li><code class="language-plaintext highlighter-rouge">TIME_WAIT</code> client side (<code class="language-plaintext highlighter-rouge">HttpClient</code>) closed the connection, but there can be still something flowing from server, because of that connection is still open and will be closed after some time</li>
<li><code class="language-plaintext highlighter-rouge">CLOSE_WAIT</code> same as above, but from server side</li>
<li><code class="language-plaintext highlighter-rouge">FIN_WAIT2</code> remote server has sent acknowledgement of receiving connection termination request (from httpclient)</li>
<li><code class="language-plaintext highlighter-rouge">LISTENING</code> local server is waiting for incoming remote connections</li>
</ul>
</li>
</ul>
<figure class="align-center">
<img src="https://deaddesk.top/assets/images/http-client-explained-using-netstat/ns.png" alt="" />
<figcaption>Output of `netstat --TCP -pnW`</figcaption>
</figure>
<h2 id="example-1">Example 1</h2>
<p>In code:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">HttpClient</code> is instantiated per single request</li>
<li><code class="language-plaintext highlighter-rouge">HttpClient</code> is not disposed</li>
</ul>
<p>In netstat:</p>
<ul>
<li>Each request gets its own TCP connection</li>
<li>TCP connections are not closed after completion of requests</li>
<li>TCP connections are closed after program is terminated</li>
</ul>
<p>If program is continuously running:</p>
<ul>
<li>After some time of idleing, foreign address sends connections termination requests and local connections go to <code class="language-plaintext highlighter-rouge">CLOSE_WAIT</code> state</li>
<li>System is waiting for dotnet program to close the connections if not after some time connections will be closed by operating system.</li>
</ul>
<p class="notice--danger">We created overhead by not-reusing existing TCP connection to (104.31.66.95:443).
Make this loop iterate over 65,535 time and you will reach maximum number of ports available, making this machine completely unresponsive.</p>
<video controls="" autoplay="" loop="" muted="" width="100%">
<source src="/assets/images/http-client-explained-using-netstat/example_1.mp4" type="video/mp4" />
</video>
<h2 id="example-2">Example 2</h2>
<p>In code:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">HttpClient</code> is instantiated per single request</li>
<li><code class="language-plaintext highlighter-rouge">HttpClient</code> is disposed after each request
In netstat:
<ul>
<li>Each call gets its own TCP connection (see port changing on client side (192.168.1.229))</li>
<li>Connection immediately goes to <code class="language-plaintext highlighter-rouge">TIME_WAIT</code> state after disposing <code class="language-plaintext highlighter-rouge">HttpClient</code></li>
</ul>
</li>
</ul>
<p class="notice--warning">We still have overhead by not reusing existing connection, on the other hand port deplation is much less likely to happen. Although make this loop parallel <code class="language-plaintext highlighter-rouge">Parallel.ForEach</code> and see machine becoming unresponsive.</p>
<video controls="" autoplay="" loop="" muted="" width="100%">
<source src="/assets/images/http-client-explained-using-netstat/example_2.mp4" type="video/mp4" />
</video>
<h2 id="example-3">Example 3</h2>
<p>In code:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">HttpClient</code> is instantiated only once</li>
<li><code class="language-plaintext highlighter-rouge">HttpClient</code> is disposed once it executed all requests
In netstat:
<ul>
<li>Each http call goes through single TCP connection</li>
<li>All connections immediately go to <code class="language-plaintext highlighter-rouge">TIME_WAIT</code> state after disposing <code class="language-plaintext highlighter-rouge">HttpClient</code></li>
</ul>
</li>
</ul>
<p>TCP connections are finally reused but remember this is only example.<br />
In real-life scenarios I recommend using <code class="language-plaintext highlighter-rouge">HttpClient Factories</code>, which takes care of <code class="language-plaintext highlighter-rouge">HttpClients</code> life-cycle, <a href="https://dotnetcoretutorials.com/2018/05/03/httpclient-factories-in-net-core-2-1/" target="_blank">more over here.</a></p>
<video controls="" autoplay="" loop="" muted="" width="100%">
<source src="/assets/images/http-client-explained-using-netstat/example_3.mp4" type="video/mp4" />
</video>
<p class="notice">Keep in my mind everything above was conducted on Linux Mint, on Windows situation in netstat might be slightly different.</p>Piotr Szymuraundernotic@gmail.comAnimated examples showcasing HttpClient and netstatKeeping node applications up to date by syncing with github repo2019-03-20T00:00:00+00:002019-03-20T00:00:00+00:00https://deaddesk.top/keeping-node-applications-up-to-date-by-syncing-with-github-repo<h2 id="the-need">The need</h2>
<blockquote>
<p>Ever wanted to have your nodejs app synced with github repository at runtime?</p>
</blockquote>
<p>Let say you have raspberry pi that is running your node app.
Probably You don’t want to pull the changes manually every time just to see updates in your app? It could be great to have some kind of automated deployment, so your app is running up to date source code from github repository.</p>
<p>I faced this and tried solving it with teamcity and jenkins but those were to heavy and complicated for raspberry so I decided to write something more lightweight and today I will share it with You :).</p>
<h2 id="one-thing-that-is-required">One thing that is required</h2>
<p>To set it up, github has to know what to call, meaning server with node application needs to be exposed to public internet.</p>
<p>If that’s already the case, great, if not then it has to be exposed.
There are two options:</p>
<ul>
<li>fully manual that requires public IP and enabled forwarding of port 5000 on router, you can read about it <a href="https://deaddesk.top/exposing-local-server-to-the-public-internet/" target="_blank">here.</a></li>
<li>tools like ngrok, which do all the heavy lifting for you</li>
</ul>
<p>Eventually this public url will be used by github to send post request with information, that given repo has been updated.</p>
<h2 id="github-repo-setup">Github repo setup</h2>
<p>To make it work:</p>
<ul>
<li>Clone <a href="https://github.com/UnderNotic/auto-deploy-raspberrypi" class="btn btn--primary" target="_blank">the repo</a> to raspberry pi or whatever device on which, nodejs app runs</li>
<li>Setup password needed for github hooks and put it to secret.txt file placed in the root of cloned repo</li>
<li>Create github hook in your app repo so on push it will send a signal to koa webserver listening for it
<img src="/assets/images/keeping-node-applications-up-to-date-by-syncing-with-github-repo/webhook.png" alt="image-center" class="align-center" style="width: 100%" /></li>
</ul>
<h2 id="running-git-sync">Running git-sync</h2>
<p>Cloned repo can be run like this:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>node index <span class="nt">-p</span> 3000 <span class="nt">-r</span> repo1,repo2 <span class="nt">-w</span> ~/mine_workspace/
</code></pre></div></div>
<p>Meaning that repos in following directories will be synced:</p>
<ul>
<li>~/mine_workspace/repo1</li>
<li>~/mine_workspace/repo2</li>
</ul>
<p>Paramerts are (in order):</p>
<ul>
<li>p = port (default: 5000)</li>
<li>r = comma seperated repositories names</li>
<li>repository directory (default: ~/projects/)</li>
</ul>
<p>To ensure git-sync is running enter localhost on specified port:
<img src="/assets/images/keeping-node-applications-up-to-date-by-syncing-with-github-repo/package.png" alt="image-center" class="align-center" style="width: 70%" /></p>
<h2 id="note-on-the-code">Note on the code</h2>
<p>If You’r interested in details, this is what exactly happens on every code change:</p>
<ul>
<li>github sends post request with metadata
<ul>
<li>to which repo new code was just pushed</li>
<li>who made the push</li>
</ul>
</li>
<li>repo name is used to check if repo should be managed by <code class="language-plaintext highlighter-rouge">git-sync</code></li>
<li>tokens from request are checked using local secret to ensure request was done by github</li>
<li>repo code is reset - every dirty change is reseted</li>
<li>repo code is cleaned - every added local file is removed</li>
<li>new code is pulled</li>
<li>npm install all dependencies for production environment</li>
</ul>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">router</span><span class="p">.</span><span class="nx">post</span><span class="p">(</span><span class="dl">'</span><span class="s1">/payload</span><span class="dl">'</span><span class="p">,</span> <span class="k">async</span> <span class="nx">ctx</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">let</span> <span class="nx">repo</span> <span class="o">=</span> <span class="nx">ctx</span><span class="p">.</span><span class="nx">request</span><span class="p">.</span><span class="nx">body</span><span class="p">.</span><span class="nx">repository</span><span class="p">.</span><span class="nx">name</span><span class="p">;</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s2">`</span><span class="p">${</span><span class="nx">ctx</span><span class="p">.</span><span class="nx">request</span><span class="p">.</span><span class="nx">body</span><span class="p">.</span><span class="nx">sender</span><span class="p">.</span><span class="nx">login</span><span class="p">}</span><span class="s2"> just pushed to </span><span class="p">${</span><span class="nx">repo</span><span class="p">}</span><span class="s2">`</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">allowedRepos</span><span class="p">.</span><span class="nx">indexOf</span><span class="p">(</span><span class="nx">repo</span><span class="p">)</span> <span class="o">===</span> <span class="o">-</span><span class="mi">1</span> <span class="o">&&</span> <span class="k">await</span> <span class="nx">verifySignature</span><span class="p">(</span><span class="nx">ctx</span><span class="p">.</span><span class="nx">request</span><span class="p">,</span> <span class="nx">secretTokenPromise</span><span class="p">))</span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">error</span><span class="p">(</span><span class="s2">`</span><span class="p">${</span><span class="nx">repo</span><span class="p">}</span><span class="s2"> is not allowed`</span><span class="p">);</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="k">await</span> <span class="nx">execAsync</span><span class="p">(</span><span class="s2">`git -C ~/projects/</span><span class="p">${</span><span class="nx">repo</span><span class="p">}</span><span class="s2"> reset --hard`</span><span class="p">);</span>
<span class="k">await</span> <span class="nx">execAsync</span><span class="p">(</span><span class="s2">`git -C ~/projects/</span><span class="p">${</span><span class="nx">repo</span><span class="p">}</span><span class="s2"> clean -df`</span><span class="p">);</span>
<span class="k">await</span> <span class="nx">execAsync</span><span class="p">(</span><span class="s2">`git -C ~/projects/</span><span class="p">${</span><span class="nx">repo</span><span class="p">}</span><span class="s2"> pull -f`</span><span class="p">);</span>
<span class="k">await</span> <span class="nx">execAsync</span><span class="p">(</span><span class="s2">`npm -C ~/projects/</span><span class="p">${</span><span class="nx">repo</span><span class="p">}</span><span class="s2"> install --production`</span><span class="p">);</span>
<span class="p">}</span>
<span class="nx">ctx</span><span class="p">.</span><span class="nx">status</span> <span class="o">=</span> <span class="mi">200</span><span class="p">;</span>
<span class="p">});</span>
</code></pre></div></div>
<h2 id="last-but-not-least---automatic-server-restarts---pm2">Last but not least - automatic server restarts - PM2</h2>
<p>By adding pm2 into the equation, You can have automatic server restarts every time code for app changes. In other words pushing code to github will automatically refresh server to use the newest code and this is done gracefully without any hiccups.</p>
<p>PM2 service will start on system startup and by default will watch directory for any code changes.
If pm2 is too heavy, use nodemon as a replacement.</p>
<p>Try it out and save your time :)</p>Piotr Szymuraundernotic@gmail.comLightweight automated CI pipe aka Poor's man CIExposing local server to the public internet2019-02-28T00:00:00+00:002019-02-28T00:00:00+00:00https://deaddesk.top/exposing-local-server-to-the-public-internet<h2 id="intro">Intro</h2>
<p>In days when we have cloud operator, exposing local server to the public internet is not a common thing to do.
On other hand with IOT beign a thing, sometimes we want to have some device sitting in our private home network behind router nat accessible from internet.</p>
<h2 id="router-port-forwarding">Router port forwarding</h2>
<p>In general typical home network consist of router, which is our gate to internet. Everything behind router sits in private network, to expose a server in this private network, router has to be configured to forward all trafic comming to it.</p>
<p>Keep in mind that this approach requires public ip assigned to our router from ISP.</p>
<p>In tp-link routers it looks like this:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">Forwarding</code> -> <code class="language-plaintext highlighter-rouge">Virtual Servers</code> -> <code class="language-plaintext highlighter-rouge">Add virtual server entry</code>
<ul>
<li><code class="language-plaintext highlighter-rouge">service port</code> - port that will be called from outside</li>
<li><code class="language-plaintext highlighter-rouge">internal port</code> - port of our local server</li>
<li><code class="language-plaintext highlighter-rouge">ip address</code> - local ip address to the server</li>
<li><code class="language-plaintext highlighter-rouge">protocol</code> - tcp, udp or both can be chosen</li>
<li><code class="language-plaintext highlighter-rouge">status</code> - turn off forwarding without deleting entry</li>
<li><code class="language-plaintext highlighter-rouge">common service port</code> - configuration templates</li>
</ul>
</li>
</ul>
<p><img src="/assets/images/exposing-local-server-to-the-public-internet/new_forwarding.png" alt="image-center" class="align-center" style="width: 100%" /></p>
<p><img src="/assets/images/exposing-local-server-to-the-public-internet/virtual-servers.png" alt="image-center" class="align-center" style="width: 100%" /></p>
<p>Now check your public ip <a href="https://www.whatismyip.com/what-is-my-public-ip-address/" target="_blank">here</a> and use it to access the server.</p>
<h2 id="aternatives">Aternatives</h2>
<p>Alternatively there are tools that do all this trickery and more.They can even help you if you don’t have public ip. If interested check <a href="https://ngrok.com/" target="_blank">ngrok here.</a></p>Piotr Szymuraundernotic@gmail.comYour server visible everywhereTop 10 javascript tricks that can save You on job interview2019-01-17T00:00:00+00:002019-01-17T00:00:00+00:00https://deaddesk.top/Top-10-javascript-tricks-you-should-know<h2 id="1-array-range">#1 Array range</h2>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[...</span><span class="nb">Array</span><span class="p">(</span><span class="mi">10</span><span class="p">).</span><span class="nx">keys</span><span class="p">()];</span>
<span class="c1">// [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]</span>
</code></pre></div></div>
<h2 id="2-array-distinct">#2 Array distinct</h2>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">let</span> <span class="nx">arr</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">];</span>
<span class="p">[...</span><span class="k">new</span> <span class="nb">Set</span><span class="p">(</span><span class="nx">arr</span><span class="p">)];</span>
<span class="c1">// [1, 2, 3]</span>
</code></pre></div></div>
<h2 id="3-difference-of-two-arrays">#3 Difference of two arrays</h2>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">let</span> <span class="nx">arr1</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">];</span>
<span class="kd">let</span> <span class="nx">arr2</span> <span class="o">=</span> <span class="p">[</span><span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">];</span>
<span class="nx">arr1</span><span class="p">.</span><span class="nx">filter</span><span class="p">(</span><span class="nx">i</span> <span class="o">=></span> <span class="o">!</span><span class="nx">arr2</span><span class="p">.</span><span class="nx">includes</span><span class="p">(</span><span class="nx">i</span><span class="p">))</span>
<span class="c1">// [1]</span>
</code></pre></div></div>
<h2 id="4-intersection-of-two-arrays">#4 Intersection of two arrays</h2>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">let</span> <span class="nx">arr1</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">];</span>
<span class="kd">let</span> <span class="nx">arr2</span> <span class="o">=</span> <span class="p">[</span><span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">];</span>
<span class="nx">arr1</span><span class="p">.</span><span class="nx">filter</span><span class="p">(</span><span class="nx">i</span> <span class="o">=></span> <span class="nx">arr2</span><span class="p">.</span><span class="nx">includes</span><span class="p">(</span><span class="nx">i</span><span class="p">))</span>
<span class="c1">// [2, 3]</span>
</code></pre></div></div>
<h2 id="5-array-group-by">#5 Array Group by</h2>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">let</span> <span class="nx">arr</span> <span class="o">=</span> <span class="p">[</span>
<span class="p">{</span> <span class="na">name</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Peter</span><span class="dl">"</span><span class="p">,</span> <span class="na">city</span><span class="p">:</span> <span class="dl">"</span><span class="s2">New York</span><span class="dl">"</span> <span class="p">},</span>
<span class="p">{</span> <span class="na">name</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Jane</span><span class="dl">"</span><span class="p">,</span> <span class="na">city</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Berlin</span><span class="dl">"</span> <span class="p">},</span>
<span class="p">{</span> <span class="na">name</span><span class="p">:</span> <span class="dl">"</span><span class="s2">John</span><span class="dl">"</span><span class="p">,</span> <span class="na">city</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Berlin</span><span class="dl">"</span> <span class="p">}</span>
<span class="p">];</span>
<span class="nx">arr</span><span class="p">.</span><span class="nx">reduce</span><span class="p">((</span><span class="nx">res</span><span class="p">,</span> <span class="nx">obj</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">res</span><span class="p">[</span><span class="nx">obj</span><span class="p">.</span><span class="nx">city</span><span class="p">]</span> <span class="o">=</span> <span class="nx">res</span><span class="p">[</span><span class="nx">obj</span><span class="p">.</span><span class="nx">city</span><span class="p">]</span> <span class="o">||</span> <span class="p">[];</span>
<span class="nx">res</span><span class="p">[</span><span class="nx">obj</span><span class="p">.</span><span class="nx">city</span><span class="p">].</span><span class="nx">push</span><span class="p">(</span><span class="nx">obj</span><span class="p">);</span>
<span class="k">return</span> <span class="nx">res</span><span class="p">;</span>
<span class="p">},</span> <span class="p">{})</span>
<span class="c1">// {</span>
<span class="c1">// "New York": [{ name: "Peter", city: "New York" }],</span>
<span class="c1">// "Berlin": [{ name: "Jane", city: "Berlin" }, { name: "John", city: "Berlin" }]</span>
<span class="c1">// }</span>
</code></pre></div></div>
<h2 id="6-object-into-keyvalue-pair">#6 Object into keyvalue pair</h2>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">let</span> <span class="nx">obj</span> <span class="o">=</span> <span class="p">{</span>
<span class="na">key1</span><span class="p">:</span> <span class="dl">"</span><span class="s2">values1</span><span class="dl">"</span><span class="p">,</span>
<span class="na">key2</span><span class="p">:</span> <span class="dl">"</span><span class="s2">values2</span><span class="dl">"</span>
<span class="p">};</span>
<span class="nb">Object</span><span class="p">.</span><span class="nx">entries</span><span class="p">(</span><span class="nx">obj</span><span class="p">);</span>
<span class="c1">//[["key1", "value1"], ["key2", "value2"]]</span>
</code></pre></div></div>
<h2 id="7-max--min-by">#7 Max \ Min by</h2>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">let</span> <span class="nx">arr</span> <span class="o">=</span> <span class="p">[{</span> <span class="na">prop</span><span class="p">:</span> <span class="mi">1</span><span class="p">},</span> <span class="p">{</span> <span class="na">prop</span><span class="p">:</span> <span class="mi">3</span><span class="p">},</span> <span class="p">{</span> <span class="na">prop</span><span class="p">:</span> <span class="mi">2</span><span class="p">}]</span>
<span class="nx">arr</span><span class="p">.</span><span class="nx">reduce</span><span class="p">((</span><span class="nx">res</span><span class="p">,</span> <span class="nx">obj</span><span class="p">)</span> <span class="o">=></span> <span class="nx">res</span><span class="p">.</span><span class="nx">prop</span> <span class="o">></span> <span class="nx">obj</span><span class="p">.</span><span class="nx">prop</span> <span class="p">?</span> <span class="nx">res</span> <span class="p">:</span> <span class="nx">obj</span><span class="p">)</span>
<span class="c1">// 3</span>
<span class="nx">arr</span><span class="p">.</span><span class="nx">reduce</span><span class="p">((</span><span class="nx">res</span><span class="p">,</span> <span class="nx">obj</span><span class="p">)</span> <span class="o">=></span> <span class="nx">res</span><span class="p">.</span><span class="nx">prop</span> <span class="o"><</span> <span class="nx">obj</span><span class="p">.</span><span class="nx">prop</span> <span class="p">?</span> <span class="nx">res</span> <span class="p">:</span> <span class="nx">obj</span><span class="p">)</span>
<span class="c1">// 1</span>
</code></pre></div></div>
<h2 id="8-merge-objects">#8 Merge objects</h2>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">var</span> <span class="nx">obj1</span> <span class="o">=</span> <span class="p">{</span> <span class="na">prop1</span><span class="p">:</span> <span class="dl">"</span><span class="s2">prop1</span><span class="dl">"</span> <span class="p">};</span>
<span class="kd">var</span> <span class="nx">obj2</span> <span class="o">=</span> <span class="p">{</span> <span class="na">prop2</span><span class="p">:</span> <span class="dl">"</span><span class="s2">prop2</span><span class="dl">"</span> <span class="p">};</span>
<span class="kd">var</span> <span class="nx">merged1</span> <span class="o">=</span> <span class="p">{</span> <span class="p">...</span><span class="nx">obj1</span><span class="p">,</span> <span class="p">...</span><span class="nx">obj2</span><span class="p">};</span>
<span class="c1">// or</span>
<span class="kd">var</span> <span class="nx">merged2</span> <span class="o">=</span> <span class="nb">Object</span><span class="p">.</span><span class="nx">assign</span><span class="p">({},</span> <span class="nx">obj1</span><span class="p">,</span> <span class="nx">obj2</span><span class="p">);</span>
<span class="c1">// { prop1: "prop1", prop2: "prop2" }</span>
</code></pre></div></div>
<h2 id="9-array-functions-on-string">#9 Array functions on string</h2>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="dl">"</span><span class="s2">hello</span><span class="dl">"</span><span class="p">.</span><span class="nx">split</span><span class="p">(</span><span class="dl">""</span><span class="p">).</span><span class="nx">map</span><span class="p">(</span><span class="nx">_</span> <span class="o">=></span> <span class="dl">"</span><span class="s2">x</span><span class="dl">"</span><span class="p">).</span><span class="nx">join</span><span class="p">(</span><span class="dl">""</span><span class="p">);</span>
<span class="c1">// "xxxxx"</span>
</code></pre></div></div>
<h2 id="10-iterate-over-object-properties-in-declarative-way">#10 Iterate over object properties in declarative way</h2>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">let</span> <span class="nx">obj</span> <span class="o">=</span> <span class="p">{</span> <span class="na">prop</span><span class="p">:</span> <span class="dl">"</span><span class="s2">prop</span><span class="dl">"</span> <span class="p">};</span>
<span class="nb">Object</span><span class="p">.</span><span class="nx">keys</span><span class="p">(</span><span class="nx">obj</span><span class="p">).</span><span class="nx">map</span><span class="p">((</span><span class="nx">key</span><span class="p">,</span> <span class="nx">i</span><span class="p">)</span> <span class="o">=></span> <span class="nx">obj</span><span class="p">[</span><span class="nx">key</span><span class="p">]</span> <span class="o">+</span> <span class="nx">i</span><span class="p">);</span>
</code></pre></div></div>
<h2>…</h2>
<p>Be aware that if you’r looking for top notch performance all above and much more is covered with optimization in mind in lodash library.</p>Piotr Szymuraundernotic@gmail.comGood to know javascript snippetsReal life testing with TestCafe2018-03-12T00:00:00+00:002018-03-12T00:00:00+00:00https://deaddesk.top/real-life-testing-with-TestCafe<h2 id="testcafe">TestCafe</h2>
<p>TestCafe has lately become one of the most popular E2E testing tool in js/browser world.<br />
End to end testing gives a confidence booster, especially when we doing something tricky and still we want to support shenanigans like IE11.</p>
<p>Let’s jump into real-world example and make our hands dirty with TestCafe.</p>
<h2 id="lets-grab-something-that-can-be-tested">Let’s grab something that can be tested</h2>
<p>Recently I created small library than can help with loading big files, by reading chunk by chunk.<br />
It’s called <code class="language-plaintext highlighter-rouge">readery</code> and it’s available on <a href="https://github.com/UnderNotic/readery" target="_blank">github</a>. Since testing file-reading in unit tests is not ideal scenario,
let’s check how this piece of code works for real.</p>
<p>The plan is to:</p>
<ul>
<li>create html page that will make us of testing library</li>
<li>write test file</li>
<li>use TestCafe to run tests</li>
</ul>
<p>Let’s have html page setup out of the way:</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><!DOCTYPE html></span>
<span class="nt"><html></span>
<span class="nt"><head></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"readery-iife.min.js"</span><span class="nt">></script></span>
<span class="nt"></head></span>
<span class="nt"><body></span>
<span class="nt"><input</span> <span class="na">type=</span><span class="s">"file"</span> <span class="na">id=</span><span class="s">"file-input"</span> <span class="na">name=</span><span class="s">"file"</span> <span class="nt">/></span>
<span class="nt"><output</span> <span class="na">id=</span><span class="s">"file_output"</span><span class="nt">></output></span>
<span class="nt"><script></span>
<span class="kd">function</span> <span class="nx">handleFileSelect</span><span class="p">(</span><span class="nx">evt</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">file</span> <span class="o">=</span> <span class="nx">evt</span><span class="p">.</span><span class="nx">target</span><span class="p">.</span><span class="nx">files</span><span class="p">[</span><span class="mi">0</span><span class="p">];</span>
<span class="kd">const</span> <span class="nx">fileOutput</span> <span class="o">=</span> <span class="nb">document</span><span class="p">.</span><span class="nx">getElementById</span><span class="p">(</span><span class="dl">'</span><span class="s1">file_output</span><span class="dl">'</span><span class="p">);</span>
<span class="nx">readery</span><span class="p">.</span><span class="nx">readFromFile</span><span class="p">(</span><span class="nx">file</span><span class="p">,</span> <span class="nx">d</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">div</span> <span class="o">=</span> <span class="nb">document</span><span class="p">.</span><span class="nx">createElement</span><span class="p">(</span><span class="dl">"</span><span class="s2">div</span><span class="dl">"</span><span class="p">);</span>
<span class="nx">div</span><span class="p">.</span><span class="nx">appendChild</span><span class="p">(</span><span class="nb">document</span><span class="p">.</span><span class="nx">createTextNode</span><span class="p">(</span><span class="nx">d</span><span class="p">));</span>
<span class="nx">fileOutput</span><span class="p">.</span><span class="nx">appendChild</span><span class="p">(</span><span class="nx">div</span><span class="p">);</span>
<span class="p">});</span>
<span class="p">}</span>
<span class="nb">document</span><span class="p">.</span><span class="nx">getElementById</span><span class="p">(</span><span class="dl">'</span><span class="s1">file-input</span><span class="dl">'</span><span class="p">).</span><span class="nx">addEventListener</span><span class="p">(</span><span class="dl">'</span><span class="s1">change</span><span class="dl">'</span><span class="p">,</span> <span class="nx">handleFileSelect</span><span class="p">,</span> <span class="kc">false</span><span class="p">);</span>
<span class="nt"></script></span>
<span class="nt"></body></span>
<span class="nt"></html></span>
</code></pre></div></div>
<p>Should be self explanatory if not this is just a page with a button, which can be used to load up a file with values seperated by newline and append them as a div to dom.</p>
<h2 id="testcafe-test-file">TestCafe test file</h2>
<p>First step is to define name for test fixture and set url which will be tested. Test cafe is testing tool to run it, tested website must be already hosted, here it will be available locally on port 8080.</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">fixture</span><span class="s2">`Readery tests`</span><span class="p">.</span><span class="nx">page</span><span class="s2">`http://localhost:8080`</span><span class="p">;</span>
</code></pre></div></div>
<p>Next we will finally start implementing test case.</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">test</span><span class="p">(</span><span class="dl">"</span><span class="s2">Should split correctly with default new line split</span><span class="dl">"</span><span class="p">,</span> <span class="k">async</span> <span class="nx">t</span> <span class="o">=></span> <span class="p">{</span>
<span class="p">})</span>
</code></pre></div></div>
<p>It’s worth to mention testcafe is in nature asynchronous.
Every method on <code class="language-plaintext highlighter-rouge">t</code> is async and can be awaited. By the way test file can be ES2017 code.</p>
<p>Next TestCafe needs to be instructed to upload a <a href="https://github.com/UnderNotic/readery/blob/master/src/tests/test_file.txt" target="_blank">test file</a> and then check if it was read correctly by checking content of <code class="language-plaintext highlighter-rouge">#file_output</code> element.</p>
<p>Uploading file is super easy:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">await</span> <span class="nx">t</span><span class="p">.</span><span class="nx">setFilesToUpload</span><span class="p">(</span><span class="dl">"</span><span class="s2">#file-input</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">./test_file.txt</span><span class="dl">"</span><span class="p">);</span>
</code></pre></div></div>
<p>Getting element content even easier:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="kd">var</span> <span class="nx">children</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">Selector</span><span class="p">(</span><span class="dl">"</span><span class="s2">#file_output</span><span class="dl">"</span><span class="p">).</span><span class="nx">child</span><span class="p">();</span>
</code></pre></div></div>
<p>Now we can cover assertions:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="kd">var</span> <span class="nx">count</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">Selector</span><span class="p">(</span><span class="dl">"</span><span class="s2">#file_output</span><span class="dl">"</span><span class="p">).</span><span class="nx">childElementCount</span><span class="p">;</span>
<span class="k">await</span> <span class="nx">t</span><span class="p">.</span><span class="nx">expect</span><span class="p">(</span><span class="dl">"</span><span class="s2">1</span><span class="dl">"</span><span class="p">).</span><span class="nx">eql</span><span class="p">(</span><span class="k">await</span> <span class="nx">children</span><span class="p">.</span><span class="nx">nth</span><span class="p">(</span><span class="mi">0</span><span class="p">).</span><span class="nx">innerText</span><span class="p">);</span>
<span class="k">await</span> <span class="nx">t</span><span class="p">.</span><span class="nx">expect</span><span class="p">(</span><span class="dl">"</span><span class="s2">2</span><span class="dl">"</span><span class="p">).</span><span class="nx">eql</span><span class="p">(</span><span class="k">await</span> <span class="nx">children</span><span class="p">.</span><span class="nx">nth</span><span class="p">(</span><span class="mi">1</span><span class="p">).</span><span class="nx">innerText</span><span class="p">);</span>
<span class="k">await</span> <span class="nx">t</span><span class="p">.</span><span class="nx">expect</span><span class="p">(</span><span class="dl">"</span><span class="s2">3</span><span class="dl">"</span><span class="p">).</span><span class="nx">eql</span><span class="p">(</span><span class="k">await</span> <span class="nx">children</span><span class="p">.</span><span class="nx">nth</span><span class="p">(</span><span class="mi">2</span><span class="p">).</span><span class="nx">innerText</span><span class="p">);</span>
<span class="k">await</span> <span class="nx">t</span><span class="p">.</span><span class="nx">expect</span><span class="p">(</span><span class="dl">""</span><span class="p">).</span><span class="nx">eql</span><span class="p">(</span><span class="k">await</span> <span class="nx">children</span><span class="p">.</span><span class="nx">nth</span><span class="p">(</span><span class="mi">3</span><span class="p">).</span><span class="nx">innerText</span><span class="p">);</span>
<span class="k">await</span> <span class="nx">t</span><span class="p">.</span><span class="nx">expect</span><span class="p">(</span><span class="dl">"</span><span class="s2">8</span><span class="dl">"</span><span class="p">).</span><span class="nx">eql</span><span class="p">(</span><span class="k">await</span> <span class="nx">children</span><span class="p">.</span><span class="nx">nth</span><span class="p">(</span><span class="mi">4</span><span class="p">).</span><span class="nx">innerText</span><span class="p">);</span>
<span class="k">await</span> <span class="nx">t</span><span class="p">.</span><span class="nx">expect</span><span class="p">(</span><span class="dl">"</span><span class="s2">a</span><span class="dl">"</span><span class="p">).</span><span class="nx">eql</span><span class="p">(</span><span class="k">await</span> <span class="nx">children</span><span class="p">.</span><span class="nx">nth</span><span class="p">(</span><span class="mi">5</span><span class="p">).</span><span class="nx">innerText</span><span class="p">);</span>
<span class="k">await</span> <span class="nx">t</span><span class="p">.</span><span class="nx">expect</span><span class="p">(</span><span class="dl">""</span><span class="p">).</span><span class="nx">eql</span><span class="p">(</span><span class="k">await</span> <span class="nx">children</span><span class="p">.</span><span class="nx">nth</span><span class="p">(</span><span class="mi">6</span><span class="p">).</span><span class="nx">innerText</span><span class="p">);</span>
<span class="k">await</span> <span class="nx">t</span><span class="p">.</span><span class="nx">expect</span><span class="p">(</span><span class="dl">""</span><span class="p">).</span><span class="nx">eql</span><span class="p">(</span><span class="k">await</span> <span class="nx">children</span><span class="p">.</span><span class="nx">nth</span><span class="p">(</span><span class="mi">7</span><span class="p">).</span><span class="nx">innerText</span><span class="p">);</span>
<span class="k">await</span> <span class="nx">t</span><span class="p">.</span><span class="nx">expect</span><span class="p">(</span><span class="dl">""</span><span class="p">).</span><span class="nx">eql</span><span class="p">(</span><span class="k">await</span> <span class="nx">children</span><span class="p">.</span><span class="nx">nth</span><span class="p">(</span><span class="mi">8</span><span class="p">).</span><span class="nx">innerText</span><span class="p">);</span>
<span class="k">await</span> <span class="nx">t</span><span class="p">.</span><span class="nx">expect</span><span class="p">(</span><span class="dl">""</span><span class="p">).</span><span class="nx">eql</span><span class="p">(</span><span class="k">await</span> <span class="nx">children</span><span class="p">.</span><span class="nx">nth</span><span class="p">(</span><span class="mi">9</span><span class="p">).</span><span class="nx">innerText</span><span class="p">);</span>
<span class="k">await</span> <span class="nx">t</span><span class="p">.</span><span class="nx">expect</span><span class="p">(</span><span class="dl">"</span><span class="s2">!</span><span class="dl">"</span><span class="p">).</span><span class="nx">eql</span><span class="p">(</span><span class="k">await</span> <span class="nx">children</span><span class="p">.</span><span class="nx">nth</span><span class="p">(</span><span class="mi">10</span><span class="p">).</span><span class="nx">innerText</span><span class="p">);</span>
<span class="k">await</span> <span class="nx">t</span><span class="p">.</span><span class="nx">expect</span><span class="p">(</span><span class="dl">"</span><span class="s2">9</span><span class="dl">"</span><span class="p">).</span><span class="nx">eql</span><span class="p">(</span><span class="k">await</span> <span class="nx">children</span><span class="p">.</span><span class="nx">nth</span><span class="p">(</span><span class="mi">11</span><span class="p">).</span><span class="nx">innerText</span><span class="p">);</span>
<span class="k">await</span> <span class="nx">t</span><span class="p">.</span><span class="nx">expect</span><span class="p">(</span><span class="dl">"</span><span class="s2">@</span><span class="dl">"</span><span class="p">).</span><span class="nx">eql</span><span class="p">(</span><span class="k">await</span> <span class="nx">children</span><span class="p">.</span><span class="nx">nth</span><span class="p">(</span><span class="mi">12</span><span class="p">).</span><span class="nx">innerText</span><span class="p">);</span>
<span class="k">await</span> <span class="nx">t</span><span class="p">.</span><span class="nx">expect</span><span class="p">(</span><span class="dl">"</span><span class="s2">123</span><span class="dl">"</span><span class="p">).</span><span class="nx">eql</span><span class="p">(</span><span class="k">await</span> <span class="nx">children</span><span class="p">.</span><span class="nx">nth</span><span class="p">(</span><span class="mi">13</span><span class="p">).</span><span class="nx">innerText</span><span class="p">);</span>
<span class="k">await</span> <span class="nx">t</span><span class="p">.</span><span class="nx">expect</span><span class="p">(</span><span class="dl">"</span><span class="s2">abc123zxy</span><span class="dl">"</span><span class="p">).</span><span class="nx">eql</span><span class="p">(</span><span class="k">await</span> <span class="nx">children</span><span class="p">.</span><span class="nx">nth</span><span class="p">(</span><span class="mi">14</span><span class="p">).</span><span class="nx">innerText</span><span class="p">);</span>
<span class="k">await</span> <span class="nx">t</span><span class="p">.</span><span class="nx">expect</span><span class="p">(</span><span class="dl">"</span><span class="s2">wwwwwwaaaaaaaazxczxcasddfgrtyyuiopoiklj[]kjhnbv!iklj[]kjhnbv!iklj[]kjhnbv!sdsdsd</span><span class="dl">"</span><span class="p">).</span><span class="nx">eql</span><span class="p">(</span><span class="k">await</span> <span class="nx">children</span><span class="p">.</span><span class="nx">nth</span><span class="p">(</span><span class="mi">15</span><span class="p">).</span><span class="nx">innerText</span><span class="p">);</span>
<span class="k">await</span> <span class="nx">t</span><span class="p">.</span><span class="nx">expect</span><span class="p">(</span><span class="nx">count</span><span class="p">).</span><span class="nx">eql</span><span class="p">(</span><span class="mi">16</span><span class="p">);</span>
<span class="p">});</span>
</code></pre></div></div>
<h2 id="test-run">Test run</h2>
<p>To run tests TestCafe CLI can be used:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nx">testcafe</span> <span class="nx">chrome</span> <span class="o">**</span><span class="cm">/*.test.js
</span></code></pre></div></div>
<p>Second parameter specifies browser, IE is also available, of course only on windows :)</p>
<p>However to actually run the test, website has to be hosted somewhere, most probably on localhost if you’r testing local code.
For that You can use good old <a href="https://github.com/indexzero/http-server" target="_blank">http-server</a> npm package.</p>
<p>Now to run those two commands together, there is <code class="language-plaintext highlighter-rouge">|</code> operator but the problem with it is, that using http-server will cause process to never exit, which results in infinite continous integration test runs.</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="dl">"</span><span class="s2">test</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">http-server ./my/website/dir/ | testcafe chrome **/*.test.js</span><span class="dl">"</span>
</code></pre></div></div>
<p>To the rescue comes test cafe with <code class="language-plaintext highlighter-rouge">--app</code> parameter:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="dl">"</span><span class="s2">test</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">testcafe chrome **/*.test.js --app 'http-server /my/website/dir/'</span><span class="dl">"</span>
</code></pre></div></div>
<h2 id="technology-is-here">Technology is here</h2>
<p>If You’r using travici for integration testing You can setup it easily to run testcafe with browser of your choice provided by saucelabs.
Pretty neat tutorial exaplaining it in details <a href="http://devexpress.github.io/testcafe/documentation/recipes/running-tests-using-travis-ci-and-sauce-labs.html" target="_blank">here.</a></p>
<p>Cheers, for fully working example You can check mentioned readery <a href="https://github.com/UnderNotic/readery">github repo</a>.</p>Piotr Szymuraundernotic@gmail.comCheck if your stuff runs on IE11 as expectedDon’t fall for TaskCompletionSource traps2018-02-09T00:00:00+00:002018-02-09T00:00:00+00:00https://deaddesk.top/don't-fall-for-TaskCompletionSource-traps<h2 id="wrapping-callback-hell-with-taskcompletionsource">Wrapping callback hell with TaskCompletionSource</h2>
<p>Ever wanted to turn callback style async code to awaitable form?
You might use <code class="language-plaintext highlighter-rouge">TaskCompletionSource</code> for it.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">class</span> <span class="nc">Program</span>
<span class="p">{</span>
<span class="k">static</span> <span class="k">void</span> <span class="nf">Main</span><span class="p">(</span><span class="kt">string</span><span class="p">[]</span> <span class="n">args</span><span class="p">)</span>
<span class="p">{</span>
<span class="nf">Run</span><span class="p">();</span>
<span class="n">Console</span><span class="p">.</span><span class="nf">ReadLine</span><span class="p">();</span>
<span class="p">}</span>
<span class="k">static</span> <span class="k">async</span> <span class="n">Task</span> <span class="nf">Run</span><span class="p">()</span>
<span class="p">{</span>
<span class="nf">CallbackStyleAsyncMethod</span><span class="p">((</span><span class="n">result</span><span class="p">)</span> <span class="p">=></span> <span class="n">Console</span><span class="p">.</span><span class="nf">WriteLine</span><span class="p">(</span><span class="n">result</span><span class="p">));</span> <span class="c1">// callback style</span>
<span class="kt">var</span> <span class="n">asyncResult</span> <span class="p">=</span> <span class="k">await</span> <span class="nf">CallbackStyleAsyncMethodWrappedAsync</span><span class="p">();</span> <span class="c1">// async/await style</span>
<span class="n">Console</span><span class="p">.</span><span class="nf">WriteLine</span><span class="p">(</span><span class="n">asyncResult</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">static</span> <span class="n">Task</span><span class="p"><</span><span class="kt">string</span><span class="p">></span> <span class="nf">CallbackStyleAsyncMethodWrappedAsync</span><span class="p">()</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">tsc</span> <span class="p">=</span> <span class="k">new</span> <span class="n">TaskCompletionSource</span><span class="p"><</span><span class="kt">string</span><span class="p">>();</span>
<span class="nf">CallbackStyleAsyncMethod</span><span class="p">((</span><span class="n">res</span><span class="p">)</span> <span class="p">=></span> <span class="n">tsc</span><span class="p">.</span><span class="nf">SetResult</span><span class="p">(</span><span class="n">res</span><span class="p">));</span>
<span class="k">return</span> <span class="n">tsc</span><span class="p">.</span><span class="n">Task</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">static</span> <span class="k">void</span> <span class="nf">CallbackStyleAsyncMethod</span><span class="p">(</span><span class="n">Action</span><span class="p"><</span><span class="kt">string</span><span class="p">></span> <span class="n">onFinish</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">Task</span><span class="p">.</span><span class="nf">Delay</span><span class="p">(</span><span class="m">5000</span><span class="p">).</span><span class="nf">ContinueWith</span><span class="p">((</span><span class="n">_</span><span class="p">)</span> <span class="p">=></span> <span class="nf">onFinish</span><span class="p">(</span><span class="s">"I'm finished"</span><span class="p">));</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>There is one problem with above code that happens very often.
In case of exception it won’t be catched in try/catch block.
This exception won’t be even printed in console window/terminal unless it’s handled by <code class="language-plaintext highlighter-rouge">TaskScheduler.UnobservedTaskException</code> or <code class="language-plaintext highlighter-rouge">DomainUnhandledException</code> Event.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">try</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">tsc</span> <span class="p">=</span> <span class="k">new</span> <span class="n">TaskCompletionSource</span><span class="p"><</span><span class="kt">string</span><span class="p">>();</span>
<span class="nf">CallbackStyleAsyncMethod</span><span class="p">(</span>
<span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="p">=></span> <span class="n">tsc</span><span class="p">.</span><span class="nf">SetResult</span><span class="p">(</span><span class="k">new</span> <span class="kt">string</span><span class="p">[]</span> <span class="p">{</span> <span class="s">"a"</span> <span class="p">}[</span><span class="m">1</span><span class="p">])</span> <span class="c1">// runtime exception</span>
<span class="p">);</span>
<span class="k">return</span> <span class="n">tsc</span><span class="p">.</span><span class="n">Task</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">catch</span> <span class="p">(</span><span class="n">Exception</span> <span class="n">ex</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span> <span class="n">Task</span><span class="p">.</span><span class="nf">FromResult</span><span class="p">(</span><span class="s">""</span><span class="p">);</span> <span class="c1">// won't be catched</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Solution is trivial but easy to forget</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">static</span> <span class="n">Task</span><span class="p"><</span><span class="kt">string</span><span class="p">></span> <span class="nf">CallbackStyleAsyncMethodWrappedAsync</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">try</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">tsc</span> <span class="p">=</span> <span class="k">new</span> <span class="n">TaskCompletionSource</span><span class="p"><</span><span class="kt">string</span><span class="p">>();</span>
<span class="nf">CallbackStyleAsyncMethod</span><span class="p">((</span><span class="n">res</span><span class="p">)</span> <span class="p">=></span>
<span class="p">{</span>
<span class="k">try</span>
<span class="p">{</span>
<span class="n">tsc</span><span class="p">.</span><span class="nf">SetResult</span><span class="p">(</span><span class="k">new</span> <span class="kt">string</span><span class="p">[]</span> <span class="p">{</span> <span class="s">"a"</span> <span class="p">}[</span><span class="m">1</span><span class="p">]);</span> <span class="c1">// runtime exception</span>
<span class="p">}</span>
<span class="k">catch</span> <span class="p">(</span><span class="n">Exception</span> <span class="n">ex</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">tsc</span><span class="p">.</span><span class="nf">SetException</span><span class="p">(</span><span class="n">ex</span><span class="p">);</span> <span class="c1">// exception is handled here</span>
<span class="p">}</span>
<span class="p">});</span>
<span class="k">return</span> <span class="n">tsc</span><span class="p">.</span><span class="n">Task</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">catch</span> <span class="p">(</span><span class="n">Exception</span> <span class="n">ex</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span> <span class="n">Task</span><span class="p">.</span><span class="nf">FromResult</span><span class="p">(</span><span class="s">""</span><span class="p">);</span> <span class="c1">// will be catched</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Cheers!</p>Piotr Szymuraundernotic@gmail.comDealing with unhandled task exceptionService discovery with etcd2018-01-06T00:00:00+00:002018-01-06T00:00:00+00:00https://deaddesk.top/service-discovery-with-etcd<h2 id="overview">Overview</h2>
<p>Let start right off the bat with quick overview what <code class="language-plaintext highlighter-rouge">etcd</code> is:</p>
<ul>
<li>distributed key/value store with failover mechanism</li>
<li>heavily uses disk but also use in memory cache</li>
<li>AP regarding CAP theorem<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup></li>
<li>sequential consistency ( the strongest consistency guarantee available from distributed systems)</li>
<li>data is persisted, after etcd restart, previously added data is still there</li>
<li>max request size 1MB</li>
<li>publish/subscribe</li>
<li>get/put</li>
<li>leases (Time to live)</li>
<li>uses grpc for communication</li>
<li>uses RAFT (leader election)</li>
<li>usually used for handling state in distributed systems (service discovery, shared configuration)</li>
<li>used in Kubernetes<sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">2</a></sup></li>
</ul>
<p>All this make etcd good candidate to store data used in service discovery.</p>
<h2 id="client-side-service-discovery">Client Side Service Discovery</h2>
<p>In client‑side discovery, the client is responsible for determining the network locations of available service instances and load balancing requests across them.</p>
<p>The client queries a service registry (etcd), which is a database of available service instances. The client then uses a load‑balancing algorithm to select one of the available service instances and makes a request.<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote" rel="footnote">3</a></sup><br />
Client in above context usually uses a library(common project or package) that knows how to get information what services are currently avaliable in the system.</p>
<p>Drawback is that you have to write this client for every programming language used in microservices that need to talk to some other microservice.</p>
<h2 id="server-side-service-discovery">Server Side Service Discovery</h2>
<p>The client makes a request to another microservice via load balancer. Load balancer queries the service registry and routes each request to an available service instance.</p>
<p>As with client‑side discovery, service instances are registered and deregistered with the service registry.<br />
Here load balancer is just another microservice but only that microservice is aware of service registry.</p>
<p>Oppose to client side service discovery this solution is language agnostic but on the other hand requires additional network hop.</p>
<h2 id="example-project-github-repo">Example project <a href="https://github.com/UnderNotic/etcd_spike" class="btn btn--primary" target="_blank">Github repo</a></h2>
<p>I had written example service discovery application showcasing server side service discovery with etcd as a service registry.</p>
<p>Solution consists of 3 projects:</p>
<ul>
<li>Etcd.Client
<ul>
<li>grpc client to cummunicate with etcd</li>
</ul>
</li>
<li>Gateway
<ul>
<li>load-balancer in service side discovery</li>
<li>load-balancing incoming requests across available ‘Node’ instances</li>
</ul>
</li>
<li>Node
<ul>
<li>subscribes to etcd by doing periodic heartbeats</li>
<li>handles requests</li>
</ul>
</li>
</ul>
<p><img src="/assets/diagrams/etcd-discovery.svg" alt="image-center" class="align-center" style="width: 850px" /></p>
<h2 id="the-heartbeat">The Heartbeat</h2>
<p><code class="language-plaintext highlighter-rouge">Node</code> reprents whatever service in the system that can scale horizontally and to whom requests are load-balanced.</p>
<p>On startup node is taking whatever port is available starting from 8080, then it’s putting lease in etcd, which is basically setting key-value pair with 10 seconds TTL (time to live), once it is set, it starts refreshing TTL every 5 seconds.</p>
<p>If Node dies, hangs… set up heatbeat key-value pair will be removed at the same time making this node unavailable and undiscoverable for the gateway.</p>
<p>Key is designed in the following way <code class="language-plaintext highlighter-rouge">heartbeat|{NODE_TYPE}|{url}</code>, so the gateway can make use of etcd range subscribe to watch only for keys that starts with <code class="language-plaintext highlighter-rouge">heartbeat</code>, also from that key, information about node type and url that it is hosted at, can be retrieved.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">private</span> <span class="k">async</span> <span class="k">void</span> <span class="nf">Run</span><span class="p">(</span><span class="kt">string</span> <span class="n">url</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">leaseId</span> <span class="p">=</span> <span class="k">await</span> <span class="n">etcdClient</span><span class="p">.</span><span class="nf">PutWithLease</span><span class="p">(</span><span class="s">$"heartbeat|</span><span class="p">{</span><span class="n">NODE_TYPE</span><span class="p">}</span><span class="s">|</span><span class="p">{</span><span class="n">url</span><span class="p">}</span><span class="s">"</span><span class="p">,</span> <span class="n">url</span><span class="p">,</span> <span class="p">(</span><span class="kt">int</span><span class="p">)</span><span class="n">TTL</span><span class="p">.</span><span class="n">TotalSeconds</span><span class="p">);</span>
<span class="k">this</span><span class="p">.</span><span class="n">timer</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">Timer</span><span class="p">(</span><span class="n">_</span> <span class="p">=></span> <span class="n">etcdClient</span><span class="p">.</span><span class="nf">KeepAliveLease</span><span class="p">(</span><span class="n">leaseId</span><span class="p">),</span> <span class="k">null</span><span class="p">,</span> <span class="n">TimeSpan</span><span class="p">.</span><span class="n">Zero</span><span class="p">,</span> <span class="n">interval</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<h2 id="load-balancing">Load balancing</h2>
<p><code class="language-plaintext highlighter-rouge">Gateway</code> manages dictionary structure that holds information about available nodes in the system.
It subscribes to heartbeats in etcd using WatchRange and updates dictionary state accordingly.</p>
<p>Using that dictionary incoming requests are load-balanced to random available node.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">private</span> <span class="k">async</span> <span class="n">Task</span><span class="p"><</span><span class="n">HttpStatusCode</span><span class="p">></span> <span class="nf">HandleRequest</span><span class="p">(</span><span class="kt">string</span> <span class="n">nodeType</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">hosts</span> <span class="p">=</span> <span class="n">serviceDiscovery</span><span class="p">.</span><span class="n">availableNodes</span><span class="p">[</span><span class="n">nodeType</span><span class="p">].</span><span class="nf">Select</span><span class="p">(</span><span class="n">s</span> <span class="p">=></span> <span class="n">s</span><span class="p">.</span><span class="n">Address</span><span class="p">).</span><span class="nf">ToArray</span><span class="p">();</span>
<span class="kt">var</span> <span class="n">host</span> <span class="p">=</span> <span class="n">hosts</span><span class="p">[</span><span class="n">random</span><span class="p">.</span><span class="nf">Next</span><span class="p">(</span><span class="m">0</span><span class="p">,</span> <span class="n">hosts</span><span class="p">.</span><span class="n">Length</span><span class="p">)];</span>
<span class="k">await</span> <span class="n">httpClient</span><span class="p">.</span><span class="nf">GetAsync</span><span class="p">(</span><span class="n">host</span><span class="p">);</span>
<span class="k">return</span> <span class="n">HttpStatusCode</span><span class="p">.</span><span class="n">OK</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">public</span> <span class="k">class</span> <span class="nc">ServiceDiscovery</span> <span class="p">:</span> <span class="n">IDisposable</span>
<span class="p">{</span>
<span class="k">private</span> <span class="n">EtcdClient</span> <span class="n">etcdClient</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="p">}</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">EtcdClient</span><span class="p">();</span>
<span class="k">public</span> <span class="n">IDictionary</span><span class="p"><</span><span class="kt">string</span><span class="p">,</span> <span class="n">IList</span><span class="p"><</span><span class="n">DiscoverableService</span><span class="p">>></span> <span class="n">availableNodes</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="k">private</span> <span class="k">set</span><span class="p">;</span> <span class="p">}</span> <span class="p">=</span> <span class="k">new</span> <span class="n">Dictionary</span><span class="p"><</span><span class="kt">string</span><span class="p">,</span> <span class="n">IList</span><span class="p"><</span><span class="n">DiscoverableService</span><span class="p">>>();</span>
<span class="k">private</span> <span class="n">EtcdWatcher</span> <span class="n">watcher</span><span class="p">;</span>
<span class="k">public</span> <span class="nf">ServiceDiscovery</span><span class="p">()</span>
<span class="p">{</span>
<span class="nf">Run</span><span class="p">();</span>
<span class="p">}</span>
<span class="k">private</span> <span class="k">async</span> <span class="n">Task</span> <span class="nf">Run</span><span class="p">()</span>
<span class="p">{</span>
<span class="n">availableNodes</span> <span class="p">=</span> <span class="nf">RangeServicesToDictionary</span><span class="p">(</span><span class="k">await</span> <span class="n">etcdClient</span><span class="p">.</span><span class="nf">GetRange</span><span class="p">(</span><span class="s">"heartbeat"</span><span class="p">));</span>
<span class="n">watcher</span> <span class="p">=</span> <span class="k">await</span> <span class="n">etcdClient</span><span class="p">.</span><span class="nf">WatchRange</span><span class="p">(</span><span class="s">"heartbeat"</span><span class="p">);</span>
<span class="n">watcher</span><span class="p">.</span><span class="nf">Subscribe</span><span class="p">(</span><span class="n">events</span> <span class="p">=></span>
<span class="p">{</span>
<span class="k">foreach</span> <span class="p">(</span><span class="kt">var</span> <span class="n">e</span> <span class="k">in</span> <span class="n">events</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">service</span> <span class="p">=</span> <span class="n">DiscoverableService</span><span class="p">.</span><span class="nf">CreateFromEtcdKey</span><span class="p">(</span><span class="n">e</span><span class="p">.</span><span class="n">Key</span><span class="p">);</span>
<span class="k">switch</span> <span class="p">(</span><span class="n">e</span><span class="p">.</span><span class="n">Type</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">case</span> <span class="n">EventType</span><span class="p">.</span><span class="n">Put</span><span class="p">:</span>
<span class="n">IList</span><span class="p"><</span><span class="n">DiscoverableService</span><span class="p">></span> <span class="n">values</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="n">availableNodes</span><span class="p">.</span><span class="nf">TryGetValue</span><span class="p">(</span><span class="n">service</span><span class="p">.</span><span class="n">Type</span><span class="p">,</span> <span class="k">out</span> <span class="n">values</span><span class="p">))</span>
<span class="p">{</span>
<span class="n">values</span><span class="p">.</span><span class="nf">Add</span><span class="p">(</span><span class="n">service</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">else</span>
<span class="p">{</span>
<span class="n">availableNodes</span><span class="p">.</span><span class="nf">Add</span><span class="p">(</span><span class="n">service</span><span class="p">.</span><span class="n">Type</span><span class="p">,</span> <span class="k">new</span> <span class="n">List</span><span class="p"><</span><span class="n">DiscoverableService</span><span class="p">></span> <span class="p">{</span> <span class="n">service</span> <span class="p">});</span>
<span class="p">}</span>
<span class="k">break</span><span class="p">;</span>
<span class="k">case</span> <span class="n">EventType</span><span class="p">.</span><span class="n">Delete</span><span class="p">:</span>
<span class="kt">var</span> <span class="n">valuesForKey</span> <span class="p">=</span> <span class="n">availableNodes</span><span class="p">[</span><span class="n">service</span><span class="p">.</span><span class="n">Type</span><span class="p">];</span>
<span class="kt">var</span> <span class="n">isDeleted</span> <span class="p">=</span> <span class="n">valuesForKey</span><span class="p">.</span><span class="nf">Remove</span><span class="p">(</span><span class="n">service</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(!</span><span class="n">isDeleted</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nf">Exception</span><span class="p">(</span><span class="s">"This can not happen"</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(!</span><span class="n">valuesForKey</span><span class="p">.</span><span class="nf">Any</span><span class="p">())</span>
<span class="p">{</span>
<span class="n">availableNodes</span><span class="p">.</span><span class="nf">Remove</span><span class="p">(</span><span class="n">e</span><span class="p">.</span><span class="n">Key</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">break</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">Console</span><span class="p">.</span><span class="nf">WriteLine</span><span class="p">(</span><span class="s">$"Available nodes </span><span class="p">{</span><span class="n">availableNodes</span><span class="p">.</span><span class="nf">Aggregate</span><span class="p">(</span><span class="kt">string</span><span class="p">.</span><span class="n">Empty</span><span class="p">,</span> <span class="p">(</span><span class="n">acc</span><span class="p">,</span> <span class="n">item</span><span class="p">)</span> <span class="p">=></span> <span class="s">$"</span><span class="p">{</span><span class="n">item</span><span class="p">.</span><span class="n">Key</span><span class="p">}</span><span class="s"> - </span><span class="p">{</span><span class="n">item</span><span class="p">.</span><span class="n">Value</span><span class="p">.</span><span class="nf">Aggregate</span><span class="p">(</span><span class="kt">string</span><span class="p">.</span><span class="n">Empty</span><span class="p">,</span> <span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">i</span><span class="p">)</span> <span class="p">=></span> <span class="s">$"</span><span class="p">{</span><span class="n">i</span><span class="p">.</span><span class="n">Address</span><span class="p">}</span><span class="s"> </span><span class="p">{</span><span class="n">a</span><span class="p">}</span><span class="s">"</span><span class="p">)}</span><span class="s"> </span><span class="p">{</span><span class="n">acc</span><span class="p">}</span><span class="s">"</span><span class="p">)}</span><span class="s">"</span><span class="p">);</span>
<span class="p">});</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<h2 id="try-it-out">Try it out</h2>
<p>Clone the repo, build, run and observe console to grasp how it works.
Gateway is exposing following api:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="nf">ApiModule</span><span class="p">()</span>
<span class="p">{</span>
<span class="n">Get</span><span class="p">[</span><span class="s">"/"</span><span class="p">,</span> <span class="n">runAsync</span><span class="p">:</span> <span class="k">true</span><span class="p">]</span> <span class="p">=</span> <span class="k">async</span> <span class="p">(</span><span class="n">args</span><span class="p">,</span> <span class="n">ct</span><span class="p">)</span> <span class="p">=></span>
<span class="p">{</span>
<span class="k">return</span> <span class="k">await</span> <span class="nf">HandleRequest</span><span class="p">();</span>
<span class="p">};</span>
<span class="n">Get</span><span class="p">[</span><span class="s">"/send/{nodeType}"</span><span class="p">,</span> <span class="n">runAsync</span><span class="p">:</span> <span class="k">true</span><span class="p">]</span> <span class="p">=</span> <span class="k">async</span> <span class="p">(</span><span class="n">args</span><span class="p">,</span> <span class="n">ct</span><span class="p">)</span> <span class="p">=></span>
<span class="p">{</span>
<span class="k">return</span> <span class="k">await</span> <span class="nf">HandleRequest</span><span class="p">(</span><span class="n">args</span><span class="p">.</span><span class="n">nodeType</span><span class="p">);</span>
<span class="p">};</span>
<span class="p">}</span>
</code></pre></div></div>
<p>While calling above api, gateway will call node process, which exposes himself by following code</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">class</span> <span class="nc">ApiModule</span> <span class="p">:</span> <span class="n">NancyModule</span>
<span class="p">{</span>
<span class="k">public</span> <span class="nf">ApiModule</span><span class="p">()</span>
<span class="p">{</span>
<span class="n">Get</span><span class="p">[</span><span class="s">"/"</span><span class="p">]</span> <span class="p">=</span> <span class="n">_</span> <span class="p">=></span>
<span class="p">{</span>
<span class="k">return</span> <span class="nf">HandleRequest</span><span class="p">();</span>
<span class="p">};</span>
<span class="n">Get</span><span class="p">[</span><span class="s">"/{path*}"</span><span class="p">]</span> <span class="p">=</span> <span class="n">_</span> <span class="p">=></span>
<span class="p">{</span>
<span class="k">return</span> <span class="nf">HandleRequest</span><span class="p">();</span>
<span class="p">};</span>
<span class="p">}</span>
<span class="k">private</span> <span class="n">HttpStatusCode</span> <span class="nf">HandleRequest</span><span class="p">()</span>
<span class="p">{</span>
<span class="n">Console</span><span class="p">.</span><span class="nf">WriteLine</span><span class="p">(</span><span class="s">$"Received request </span><span class="p">{</span><span class="k">this</span><span class="p">.</span><span class="n">Request</span><span class="p">.</span><span class="n">Url</span><span class="p">}</span><span class="s">"</span><span class="p">);</span>
<span class="k">return</span> <span class="n">HttpStatusCode</span><span class="p">.</span><span class="n">OK</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<h2 id="etcd-grpc-client-library">Etcd Grpc Client library</h2>
<p>I extracted grpc part from above project to separate github repo and created nuget out of that.</p>
<p>It greatly simplifies communication with etcd in dotnet projects, and since it’s .net standard package it can be used in both .net framework and .net core projects.</p>
<p>Check it out if it fits your needs and contribute :)</p>
<p><a href="https://github.com/UnderNotic/EtcdGrpcClient" class="btn btn--primary" target="_blank">Github repo</a>
<a href="https://www.nuget.org/packages/EtcdGrpcClient" class="btn btn--primary" target="_blank">Nuget page</a></p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p><a href="https://coreos.com/etcd/docs/latest/learning/api_guarantees.html">https://coreos.com/etcd/docs/latest/learning/api_guarantees.html</a> <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p><a href="https://kubernetes.io/">https://kubernetes.io/</a> <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p><a href="https://www.nginx.com/blog/service-discovery-in-a-microservices-architecture/">https://www.nginx.com/blog/service-discovery-in-a-microservices-architecture/</a> <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Piotr Szymuraundernotic@gmail.comManaging state in distributed systemsChoosing the proper react / redux project structure2017-12-08T00:00:00+00:002017-12-08T00:00:00+00:00https://deaddesk.top/choosing-the-proper-redux-project-structure<h2 id="not-so-hot-way">Not so hot way</h2>
<p>Typical structure for example project looks something like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>components/
TodoList.js
containers/
TodoContainer.js
reducers/
todoReducer.js
actions/
todoActions.js
constants/
todoContants.js
selectors/
todoSelectors.js
sagas/
todoSaga.js
index.js
</code></pre></div></div>
<p>Files are grouped by technology roles not by features.</p>
<p>It is completely fine to use it for small projects but as project grows this won’t scale up.
Having more than 10 files in directory makes searching difficult, also import autocomplete becomes a hassle with dozen of options available.</p>
<h2 id="features-oriented-over-everything-else">Features oriented over everything else</h2>
<p>Personally I recommend and use below structure:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>src/
components/ -> common dumb components that can be reused
Header/
index.js
Menu/
index.js
containers/
Todo/
tests/
todoContainer.test.js
index.js -> container component
TodoList -> dumb component that is used only for todo
todoDuck.js -> action constants, action creators, reducer
selectors.js -> reselect code
saga.js -> redux-saga
AnotherFeature/
.
.
.
css/ -> css related files if not using css-in-js
assets/ -> images, fonts etc.
utils/ -> code that doesn't fit into any other place
index.js -> entry point
configureStore.js -> redux store configuration
reducers.js -> combine reducers
.env -> contains 'NODE_PATH=src/', needed for absolute imports
package.json
README.md
</code></pre></div></div>
<p>Everything is divided by features, it scales by adding new directories per feature and not having +10 files in directories.
It uses index.js files because typically a feature in redux is seen as a single container file, this way you can easily import something by doing <code class="language-plaintext highlighter-rouge">import Todo from containers/Todo</code>.</p>
<p>Other files in feature directory are most of the times imported withing this directory.</p>
<p>I also use duck convention so redux constants, action creators and reducers are in one file under specific feature.</p>
<p>It eliminates context switching and I don’t like having 10 lines of imports in each redux file.
More about it
<a href="https://github.com/erikras/ducks-modular-redux/" target="_blank">here</a>.
Of course selectors or sagas are optional and if I’m using redux-thunk,
it’s logic should be placed in duck files under action creators section.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Actions</span>
<span class="kd">const</span> <span class="nx">LOGIN_REQUEST</span> <span class="o">=</span> <span class="dl">"</span><span class="s2">containers/login/LOGIN_REQUEST</span><span class="dl">"</span><span class="p">;</span>
<span class="kd">const</span> <span class="nx">LOGIN_REQUEST_SUCCESS</span> <span class="o">=</span> <span class="dl">'</span><span class="s1">containers/login/LOGIN_REQUEST_SUCCESS</span><span class="dl">'</span><span class="p">;</span>
<span class="kd">const</span> <span class="nx">LOGIN_REQUEST_FAIL</span> <span class="o">=</span> <span class="dl">'</span><span class="s1">containers/login/LOGIN_REQUEST_FAIL</span><span class="dl">'</span><span class="p">;</span>
<span class="c1">// Reducer</span>
<span class="k">export</span> <span class="k">default</span> <span class="kd">function</span> <span class="nx">reducer</span><span class="p">(</span><span class="nx">state</span> <span class="o">=</span> <span class="p">{</span>
<span class="na">isFetching</span><span class="p">:</span> <span class="kc">false</span><span class="p">,</span>
<span class="na">isFailing</span><span class="p">:</span> <span class="kc">false</span><span class="p">,</span>
<span class="na">isLoginSuccessful</span><span class="p">:</span> <span class="kc">false</span><span class="p">,</span>
<span class="na">jwtToken</span><span class="p">:</span> <span class="dl">""</span>
<span class="p">},</span> <span class="nx">action</span> <span class="o">=</span> <span class="p">{})</span> <span class="p">{</span>
<span class="k">switch</span> <span class="p">(</span><span class="nx">action</span><span class="p">.</span><span class="nx">type</span><span class="p">)</span> <span class="p">{</span>
<span class="k">case</span> <span class="nx">LOGIN_REQUEST</span><span class="p">:</span>
<span class="k">return</span> <span class="p">{</span> <span class="p">...</span><span class="nx">state</span><span class="p">,</span> <span class="na">isFetching</span><span class="p">:</span> <span class="kc">true</span> <span class="p">};</span>
<span class="k">case</span> <span class="nx">LOGIN_REQUEST_SUCCESS</span><span class="p">:</span>
<span class="k">return</span> <span class="p">{</span> <span class="p">...</span><span class="nx">state</span><span class="p">,</span> <span class="na">isFetching</span><span class="p">:</span> <span class="kc">false</span><span class="p">,</span> <span class="na">isFailing</span><span class="p">:</span> <span class="kc">false</span><span class="p">,</span> <span class="na">isLoginSuccessful</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span> <span class="na">jwtToken</span><span class="p">:</span> <span class="nx">action</span><span class="p">.</span><span class="nx">token</span><span class="p">.</span><span class="nx">token</span> <span class="p">};</span>
<span class="k">case</span> <span class="nx">LOGIN_REQUEST_FAIL</span><span class="p">:</span>
<span class="k">return</span> <span class="p">{</span> <span class="p">...</span><span class="nx">state</span><span class="p">,</span> <span class="na">isFailing</span><span class="p">:</span> <span class="kc">true</span> <span class="p">};</span>
<span class="nl">default</span><span class="p">:</span>
<span class="k">return</span> <span class="nx">state</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="c1">// Action Creators</span>
<span class="kd">function</span> <span class="nx">loginRequest</span><span class="p">()</span> <span class="p">{</span>
<span class="k">return</span> <span class="p">{</span> <span class="na">type</span><span class="p">:</span> <span class="nx">LOGIN_REQUEST</span> <span class="p">};</span>
<span class="p">}</span>
<span class="kd">function</span> <span class="nx">loginRequestSuccess</span><span class="p">(</span><span class="nx">token</span><span class="p">,</span> <span class="nx">expiryDate</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="p">{</span> <span class="na">type</span><span class="p">:</span> <span class="nx">LOGIN_REQUEST_SUCCESS</span><span class="p">,</span> <span class="nx">token</span><span class="p">,</span> <span class="nx">expiryDate</span> <span class="p">};</span>
<span class="p">}</span>
<span class="kd">function</span> <span class="nx">loginRequestFail</span><span class="p">(</span><span class="nx">error</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="p">{</span> <span class="na">type</span><span class="p">:</span> <span class="nx">LOGIN_REQUEST_FAIL</span><span class="p">,</span> <span class="nx">error</span> <span class="p">};</span>
<span class="p">}</span>
<span class="k">export</span> <span class="kd">const</span> <span class="nx">fetchLogin</span> <span class="o">=</span> <span class="p">(</span><span class="nx">login</span><span class="p">,</span> <span class="nx">pass</span><span class="p">)</span> <span class="o">=></span> <span class="k">async</span> <span class="nx">dispatch</span> <span class="o">=></span> <span class="p">{</span>
<span class="k">try</span> <span class="p">{</span>
<span class="nx">dispatch</span><span class="p">(</span><span class="nx">loginRequest</span><span class="p">());</span>
<span class="kd">let</span> <span class="nx">response</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">fetchPost</span><span class="p">(</span><span class="nx">loginUrl</span><span class="p">,</span> <span class="p">{</span> <span class="nx">login</span><span class="p">,</span> <span class="na">password</span><span class="p">:</span> <span class="nx">pass</span> <span class="p">});</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nx">response</span><span class="p">.</span><span class="nx">ok</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">dispatch</span><span class="p">(</span><span class="nx">loginRequestFail</span><span class="p">(</span><span class="nx">response</span><span class="p">.</span><span class="nx">statusText</span><span class="p">));</span>
<span class="k">return</span><span class="p">;</span>
<span class="p">}</span>
<span class="kd">let</span> <span class="nx">json</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">response</span><span class="p">.</span><span class="nx">json</span><span class="p">();</span>
<span class="nx">dispatch</span><span class="p">(</span><span class="nx">loginRequestSuccess</span><span class="p">(</span><span class="nx">json</span><span class="p">,</span> <span class="kc">null</span><span class="p">));</span>
<span class="p">}</span> <span class="k">catch</span> <span class="p">(</span><span class="nx">ex</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">dispatch</span><span class="p">(</span><span class="nx">loginRequestFail</span><span class="p">(</span><span class="nx">ex</span><span class="p">.</span><span class="nx">toString</span><span class="p">()));</span>
<span class="p">}</span>
<span class="p">};</span>
</code></pre></div></div>
<p>Additionaly to make <code class="language-plaintext highlighter-rouge">absolute imports</code> work I have .env file in root directory:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>NODE_PATH=src/
</code></pre></div></div>
<p>This file is used by default in create-react-app so if your using it nothing <code class="language-plaintext highlighter-rouge">webpack'y</code> has to be configured.</p>
<h2 id="file-naming-convention">File naming convention</h2>
<p>In the past I was using kebab-case for both files and directories no matter what was inside. I wanted to have same convention for server-side code (nodejs) and front-end code (react).</p>
<p>But I took a different direction and went with community, deciding to use a convention that will tell from just watching at the name of a file or directory what is inside.</p>
<ul>
<li>if a file is exporting something that can be instantiated (class, component) then use PascalCase</li>
<li>index.js files are lower-case</li>
<li>everything else is camelCase</li>
<li>test files are pascalCase.test.js</li>
<li>directories containing index.js, which fulfil point number 1, should be PascalCase</li>
<li>every other directory should be camelCase</li>
</ul>
<p>Try it out and Cheeeers!</p>Piotr Szymuraundernotic@gmail.comRedux project structure and naming conventionUrl reservation in owin2017-11-09T00:00:00+00:002017-11-09T00:00:00+00:00https://deaddesk.top/owin-hostname-reservation<h2 id="strong-wildcard-">Strong wildcard +</h2>
<p>Using strong wildcard You can reserve all hostnames on given port.</p>
<p>Thing to remember is that even if there is another service that is reserving url on the same port without wildcard then all http requests will go anyway to service that is using strong wildcard.</p>
<p>Sometimes example is more than thousand words:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">//Service #1</span>
<span class="k">using</span> <span class="p">(</span><span class="n">WebApp</span><span class="p">.</span><span class="n">Start</span><span class="p"><</span><span class="n">Startup</span><span class="p">></span> <span class="p">(</span><span class="s">"http://mywebsite:80/"</span><span class="p">))</span>
<span class="p">{</span>
<span class="n">Console</span><span class="p">.</span><span class="nf">WriteLine</span><span class="p">(</span><span class="s">"Service #1"</span><span class="p">);</span>
<span class="p">}</span>
<span class="c1">//Service #2</span>
<span class="k">using</span> <span class="p">(</span><span class="n">WebApp</span><span class="p">.</span><span class="n">Start</span><span class="p"><</span><span class="n">Startup</span><span class="p">></span> <span class="p">(</span><span class="s">"http://+:80/"</span><span class="p">))</span>
<span class="p">{</span>
<span class="n">Console</span><span class="p">.</span><span class="nf">WriteLine</span><span class="p">(</span><span class="s">"Service #2"</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>In above all http traffic on port 80 will go to <code class="language-plaintext highlighter-rouge">Service #2</code>.<br />
<code class="language-plaintext highlighter-rouge">Console.WriteLine("Service #1")</code> won’t be called at all.</p>
<h2 id="weak-wildcard-">Weak wildcard *</h2>
<p>Suppose You have a service to which you want to redirect all trafic going through port 80, but at the same time</p>
<p>You want to have another service running also on port 80 that will be accessible only using specified url (<code class="language-plaintext highlighter-rouge">http://mywebsite:80/</code>). If so weak wildcard is your friend.</p>
<p>Consider this example:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">//Service #1</span>
<span class="k">using</span> <span class="p">(</span><span class="n">WebApp</span><span class="p">.</span><span class="n">Start</span><span class="p"><</span><span class="n">Startup</span><span class="p">></span> <span class="p">(</span><span class="s">"http://mywebsite:80/"</span><span class="p">))</span>
<span class="p">{</span>
<span class="n">Console</span><span class="p">.</span><span class="nf">WriteLine</span><span class="p">(</span><span class="s">"Service #1"</span><span class="p">);</span>
<span class="p">}</span>
<span class="c1">//Service #2</span>
<span class="k">using</span> <span class="p">(</span><span class="n">WebApp</span><span class="p">.</span><span class="n">Start</span><span class="p"><</span><span class="n">Startup</span><span class="p">></span> <span class="p">(</span><span class="s">"http://*:80/"</span><span class="p">))</span>
<span class="p">{</span>
<span class="n">Console</span><span class="p">.</span><span class="nf">WriteLine</span><span class="p">(</span><span class="s">"Service #2"</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Any request other than one to <code class="language-plaintext highlighter-rouge">http://mywebsite:80/</code> will end up in Service #2.</p>
<p>Important thing to mention is if your not executing above code as user who is admin, you have to add same urls you provide to owin to netsh using <code class="language-plaintext highlighter-rouge">netsh http add urlacl</code> syntax.</p>
<p>Cheers!</p>Piotr Szymuraundernotic@gmail.comStars and pluses