Tuesday, April 18, 2017

Serverless JoymonOnline.in - Dealing with GitHub API and its limits

Introduction

This is continuation of my Serverless journey with my personal site www.Joymononline.in. As mentioned in previous posts, the main motive behind going Serverless is the cost. The principle is, 'Spend only for domain name. Nothing else'. Now the site is completely Serverless. But like anything else Serverless is not the perfect solution which can be applied everywhere.

One of the draw backs of Serverless is the lack of options to store developer keys at client side. In Serverless, we don't maintain any server but utilizes many third party services. Some services will be having limits on anonymous usage and some doesn't work at all without authentication. It can be overcome only by adding our own server side component(service) which acts as proxy between our client and third party service. Our server side component can have the developer key or API secret and utilize high quota from the third party service.

GitHub API limits

The open source projects from GitHub are listed in the site with some details. For showing that, GitHub API is consumed from the client side. But the GitHub API has limits. At the time of writing this post, if the request is not authenticated, it limits 60 requests per IP per hour. When authenticated, limit is 5000..Ideally JoymonOnline.in will never need that much requests for single user. But below are the situations where the API limits matter
  • The site is opened from corporate machines where the external IP is same and simultaneously more than 60 people from a company opens.
  • The user has opened another site which does heavy GitHub anonymous activity and after that opens JoymonOnline.in.
The first scenario may be rare as the my site at this point doesn't have that much reach. But the second is valid. So this issue is something to be addressed.

Overcome free GitHub API limits using proxy service

As mentioned above the way to overcome this is to place our own service in middle which has the developer key and let that service call GitHub API. The best way to create such a service is to use any  Function as a Service offering. There are many such as Azure Functions, Amazon Lambda, Google Functions, Webtasks etc... The selection criteria is based on cost ie it should be free (not even 1$ setup free) and ability to allow outbound calls to GitHub. Below code shows how the service can be written for Webtask. Why Webtask is selected will be coming as another post.

const http=require('https');
var express    = require('express');
var Webtask    = require('webtask-tools');
var bodyParser = require('body-parser');
var app = express();

app.use(bodyParser.json());

app.get('/', function (req, res) {
  res.json({ message: 'Welcome to my api!.Currently it supports api/joymon/<name of github repo>. Not open for all repos ' });
});
app.get('/joymon/:projectName', function (req, res) {
  console.log(req.params);
  var  callback = function(response) {
    var str = '';
    response.on('data', function (chunk) {
          str += chunk;
    });
    response.on('end', function () {
        res.json(JSON.parse(str));
        res.end(str);
    });
  };
  
  http.request(getOptions(req.params.projectName),callback)
    .on('error',function(err){
      console.log(err);
      res.json({ message: 'Soemthing went wrong when contacting GitHub API.' });
    })
    .end();
});
module.exports = Webtask.fromExpress(app);

function getOptions(projectName) {
  return {
        method:'GET',
        host:'api.github.com',
        path:'/repos/joymon/'+projectName,
        uri:'https://api.github.com/repos/joymon/'+projectName,
        json:true,
        headers: {'User-Agent': 'Custom Header Demo works'} //Add dev key via header
    };
}

If we have Webtask account, we can simply copy paste the above code and enjoy free service.

Client side changes

The GitHub API calls return headers which talks about quota and usage. The client side has to check that and call the proxy service accordingly. It is kind of retry which was never needed prior to Serverless.

http://jsfiddle.net/joygeorgek/arp6bmrj/5/

Overall Serverless brings benefits but at the same time, we have to do more such as retry and service orchestration. Also it brings so many points of failure and has to design for it. If one third party service is down we should continue giving the other features as normal.

Tuesday, April 11, 2017

JavaScript Access inner objects with string expression

JavaScript is becoming more and more importance for developers. Even it can be used as DSL with the help of embedded V8 runtime engine. Below is one scenario which may occur when using JavaScript as DSL

The DSL is another program fragment, which our program has to execute.
Suppose there is an JavaScript object structure as follows

var complexObject = {
 prop0:"prop0 value",
  prop1: {
    prop1prop1: {
     prop1prop2prop3: "prop1prop2prop3 value"
    },
    prop1prop2:"prop1prop2 Value"
  },
  prop2:[
   {prop2prop1:"prop2[prop2prop1] value"},
   {prop2prop2:"prop2[prop2prop2] value"}
  ]
};

Lets see how this can be accessed.

Normal JavaScript property access

Now lets see how can we access the values using normal property dot(.) notation


var testInnerObjectProperty = function() {
  alert(complexObject.prop0);
  alert(complexObject.prop1.prop1prop2);
  alert(complexObject.prop2[0].prop2prop1);
};


This works perfect.

Access JavaScript property using string property path

But in DSL the path to the value will be available as string. So how to access the value using string property path?



var testInnerObjectPropertyWithIndexer  = function(){
  alert(complexObject["prop0"]);
  alert(complexObject["prop1.prop1prop2"]);
  alert(complexObject["prop2[0].prop2prop1"]);
}


We can see the first alert returns propert value as its direct string path access. But the next 2 will not work as JavaScript runtime is not able to find a property names "prop1.prop1prop2" on complexObject. In other words it doesn't have the intelligence to understand the hierarchy. May be its by design.
Lets see another way to access using string string.

Access property using string path via Evil Eval()

This is the easiest method pops up into a medium JavaScript programmers mind though he know its not the good way.


var testInnerObjectPropertyWithIndexerEval = function() {
 alert(eval('complexObject.'+'prop0'));
  alert(eval('complexObject.'+'prop1.prop1prop2'));
  alert(eval('complexObject.'+'prop2[0].prop2prop1'));
}


Since it works but not the good way what is next.

Access property using string path via parsing

This is the hard way. We must test is property for all combinations. Lets see one implementation.


var testInnerObjectPropertyByParsingPath = function() {
 alert('Going to parse prop path');
 alert(Object.getValue(complexObject,'prop0'));
  debugger;
  alert(Object.getValue(complexObject,'prop1.prop1prop2'));
  alert(Object.getValue(complexObject,'prop2[0].prop2prop1'));
}


Above is the API interface expected. Below goes the implementation of getValue()


Object.getValue = function(obj, path) {
    path = path.replace(/\[(\w+)\]/g, '.$1'); // convert [] access to property access with number
    path = path.replace(/^\./, '');           // trim leading dot
    var props = path.split('.');
    
    for (var propIndex = 0, noOfProps = props.length; propIndex < noOfProps; ++propIndex) {
        var propName = props[propIndex];
        if (propName in obj) {
            obj = obj[propName];
        } else {
            return;
        }
    }
    return obj;
}


This can be written using recursion too. It replace the array access [] with property access by replacing corresponding number there. Then loops into the object hierarchy till it finds required object.

Please note this is not the battle tested code. This is rather to give the idea.
https://jsfiddle.net/joygeorgek/s5egwa5o/8/


Tuesday, April 4, 2017

ASP.Net IIS AppPool Recycling is a hack

From the beginning it was known to me that the IIS AppPool recycling is a hack. If a technology is matured enough to handle web traffic, it doesn't need periodic recycling of processes. To me there is no difference between support teams asking to restart machines to solve issues and app pool recycling. In both these cases, the memory is cleaned up.

Whenever I was involved in IIS applications, I instructed the teams to disable app pool recycling from the development time itself. But it never happened. According to most of developers and their managers that is a feature from IIS and why should they change defaults and bring unnecessary issues. Whenever I had direct delivery responsibility, I ensured that the app pool recycling is not enabled. There was mistake from my side as well that I never invested time to find out a valid Microsoft link which says do not recycle app pool process. But yesterday the time arrived. There was a client question on why the app pool recycling is disabled on some applications where majority has recycling. Those were framework level applications and in my direct control during development. 
If you have a problematic application and you cannot easily correct the code that causes the problems, you can limit the extent of these problems by periodically recycling the worker process that services the application. Microsoft
Finally got the link where MSFT says about app pool recycle. But the scenario again complicated for us. 3-4 web apps only disabled the app pool recycling and the other 50+ web apps not. Were those created with no quality and difficult to fix?

Stateful v/s Stateless IIS web apps

Here comes 2 different types of web apps. Some apps will be just to receive request and send response back. If we recycle these stateless apps, there is no harm to the behavior of the system as every request and its response pair is independent of other request-response pairs. So it is 'ok' to recycle. In case there are something going wrong with memory and unnoticed during dev and QA, it won't cause trouble in production as well. Since extensive memory, load, long running testing are not needed, we can develop in low cost.

Other type of apps will be maintaining some sort of state in the memory and state might be evolving over time. Also their behavior will be controlled by the current state. If we recycle the process, the state will be lost and their behavior will be wrong after recycling. Yes there is the answer.
It is 'ok' to recycle the stateless web apps but never recycle the stateful apps.
When it is said 'ok' means, there are no ways to fix in production. Other scenario is when there was no chance to properly test for long hours and LOH fragmentation during development. Again it is not recommended to keep a big state in memory which may cause scaling issues.

Better write programs properly and avoid app pool recycling.

LOH and recycling

Without mentioning Large Object Heap, any article about .Net memory management is incomplete. As all knows LOH was never getting compacted in earlier .Net versions. That causes large objects to be fragmented and finally lead to OutOfMemoryException. In those days, it was not a choice to enable AppPool recycling but we have to. I agree that there are programmatic mechanisms to avoid LOH fragmentation. But those are all costly during development and testing. Also requires high skills.

Links

https://weblogs.asp.net/owscott/why-is-the-iis-default-app-pool-recycle-set-to-1740-minutes

Tuesday, March 28, 2017

My JavaScript Module experiments - CSS

This post is continuation of my JavaScript module experiments series with Webpack. Earlier posts below.

1 My JavaScript Module experiments
2 My JavaScript Module experiments - CI & CD
3 My JavaScript Module experiments - CSS

This post assumes the basic knowledge about NodeJS, NPM, Webpack such as its config file and require mechanism.

Should we bundle css?

It really depends. 

Traditional CSS development

If we take the scenario of simple web site development, we may not need bundling provided entire CSS is in single file. But that is not at all the scenario right now. We need the sites responsive and for that we may have to add dedicated styles for different screen resolutions. One way is to write everything in the same css or separated files loaded on demand. But keeping in different files and downloading all to client and deciding which to use will slow down the performance. Because it needs more http file requests. Keeping in separate files is easy in terms of maintaining. Normally the decision to what style needs to be applied is taken at client side based on the screen size. It will be overhead to request the css for different sizes when the user resize the browser window. So it is better to bundle the css for different sizes at server and download as one file. 

Modern CSS

Modern CSS means writing CSS with the help higher level languages such as LESS or SASS. They help us to write less reusable code and get things done. Since the poor browser cannot understand this LESS or SASS languages, we have to convert to CSS. That process is called transpilation. Here there are chances that some styles will be in pure CSS, some in LESS and some in SASS. As we have to convert to CSS and refer into the application, it is better to bundle the outputs to one css file and refer same.

HTML component

Another scenario is development of web components or on-demand view development. In those scenarios we don't have to keep entire CSS in single place and refer from the beginning. We have to bundle the css along with the JavaScript component which only loaded when needed.

Webpack and CSS

Now lets see how the Webpack and CSS works together. 3 different scenarios are listed below to understand at the simplest level. To keep it simple no other things like ECMA 6 or TypeScript is used.

Simple scenario - webpack and css

Here we are simply going to require .css files from JavaScript to get it available in the html. Below are the steps
  • Import the npm packages named css-loader & style-loader. These are the Webpack specific NPM packages to process .css . The first one is to recognize the .css file and other is to load the .css as <style> tag.
  • Modify the webpack.config.js file to have loader. For syntax refer to the sample given below
  • Add require statements in to the JavaScript file
    • require('./stylesheet.css');
Below is the link to sample.


To run the sample. 
  • Make sure there is NodeJS and NPM
  • Clone the repo to the local machine
  • Open NodeJS command prompt and navigate to the \ECMA-CSS folder
  • run 'npm install' command to download dependencies.
  • run 'webpack' command.
    • If the webpack is not available at system level, '.\node_modules\.bin\webpack' is required
  • This will produce bundle.js file
  • Just run the index.html file from file system. Make sure there are permissions to run JavaScript from local file
When running we can see that there are <style> tags added to the <head> which are not present in index.html file. Those are added by the style loader at runtime.

Combined css

Injecting <style> at runtime is not a good solution. Better is to have a final combined css file. The below sample shows that mechanism using one more npm package called 'extract-text-webpack-plugin'. Since the sample uses webpack 1, the version of 'extract-text-webpack-plugin' should be 1.0.1.


Here the index.html is referring to a style file called bundle.css which is not there in repository. When this sample is run using the same steps mentioned for above previous scenario, the bundle.css will be created by Webpack. Once that file is present, we can run the index.html as usual.

The bundling of multiple css files to one is done by using ExtractTextPlugin configured in webpack.config.js file and here we don't need the style-loader.

CSS & LESS together

This is the next level where we mix plain .css with advanced .less file and finally produce combined .css file. Sample is available in the below location.


In this sample, instead of earlier childstylesheet.css, childstylesheet.LESS is used. The npm packages used are less, and less-loader for compiling .less files and to load the same respectively. The webpack.config.js also modified to tell how to handle the .less files and to combine using ExtractTextPlugin.

During the compilation the .LESS file will be compiled to .css and then combined with other .css file to produce bundle.css file.

Further reading

https://webpack.github.io/docs/stylesheets.html
http://jamesknelson.com/writing-happy-stylesheets-with-webpack/
https://css-tricks.com/css-modules-part-2-getting-started/
http://stackoverflow.com/questions/39015521/import-css-and-js-files-using-webpack
https://github.com/webpack/file-loader/issues/32

Tuesday, March 21, 2017

GoDaddy to use Github pages

Recently I setup my personal site www.joymononline.in to point to GitHub pages. The domain registration is still with GoDaddy but the hosting space is no more required from GoDaddy. Infact that was eating good amount of cost involved in maintaining the site.

How to configure GoDaddy to use GitHub pages is already a solved problem. There are many sites which explains the same. Below are some.

http://captainwhippet.com/blog/2014/05/11/blog-setup-details.html
http://mycyberuniverse.com/web/configuring-a-godaddy-domain-name-with-github-pages.html


Why this post?

The IP addresess

One of my confusion was about the IP addresses mentioned in those posts. Are those still valid?Some posts are written in 2015. Fortunately those are still valid.

GoDaddy Records

Another place I was little confused was about setting up A records. There were some A records. Instead of deleting all, I just modified to have GitHub IPs.
Also some sites talk about 2 IP addresses to be added via 2 different A records but some talks about only 1. I had updated 2 records and the site got up with GitHub pages in couple of mins.


Tuesday, March 14, 2017

Architecture of JoymonOnline - Serverless

This can be considered as continuation of 3 old posts.


The first post, talks about the architecture of my personal site JoymonOnline.in. Other posts explains about new software architecture trend called Serverless and the conversion of my site towards Serverless architecture. 

This post explains the new architecture.

Architecture of Serverless JoymonOnline.in

Hosting

Hope the diagram is pretty much self explanatory. Its going to be pure client side single page application (SPA). In other words there is no page refresh when user goes to different view. 

The domain is going to be continued with Godaddy. But the hosting will be done inside Github pages, which is free. When we say Github hosting is free, it comes with limitations and quotas. But for my personal site those limits are not going to hit at least for some years :). It clearly tells that for a different person who is really famous the limits will come into play. So Github pages is not the silver bullet for hosting.

Flow

Once the client browser gets the static site(html,js & css) from Github, it becomes active by bootstrapping Angular components. It calls required services present in external systems for data. Most of these external systems are already mentioned in my previous post. After the data retrieval, the JavaScript updates the DOM with the help of AngularJS framework. The Angular JS version is 1.x used and there is a blog post explaining why 1.x is selected when Angular 2 is out.

Future candidates


  • Azure Function - For Github & Blogger API due to anonymous quota limits.

Pros

  • No cost to host web site. Just the domain registration. Most of the services are free.
  • No need to license development tools. For  ASP.Net, Visual Studio community edition don't have much features.

Cons

  • Multiple points of failure - Earlier, if JoymonOnline.in domain was accessible entire site works. But now all the domains of external systems should be accessible from the client machine which is not possible in all places.
  • More dependencies - Has to monitor what changes in external systems and has to maintain accordingly.
  • JavaScript restriction - Some browsers (inside some companies) might be still blocked to download JavaScript and execute.
Note - As of today (14Mar2017), the redirection for the site from Godaddy to Github pages to serve site has not done. But the site is up and running in Github pages joymon.github.io.

Tuesday, March 7, 2017

ConcurrentCounter - Reduce code with Functional programming

The fashion of functional programming

Now a days functional programming already made its come back and every language, companies and developers want to embrace functional programming. When we look at their works we can see it needs to travel more to reach the functional programming world. It was same for object oriented world as well. People used C++ but the code looked like C.

That is not the concern with this post. One of the reasons we here when people adopt FP is that it helps us to achieve things with less code. This is just a scenario where FP can reduce some amount of code.

Scenario - Thread safe manipulation of resources

We are in the world of threads. Knowingly or unknowingly we had to deal with threads. Some languages / technologies already has better ways to deal threading without developer write any explicit line. NodeJS can be an example. But for C# guys like me, has to write programs properly to make sure it works with threads. If all the tasks are ready to run in parallel,  there are no issues. But the reality is not that way. There will be so many critical sections we had to handle to achieve a good working software. We rely heavily on thread synchronization mechanisms to achieve the same. Whenever we deal with those, it is nightmare and there are so many chances to produce duplicate code. That is the place functional programming can help us.

Taking one situation. We have a counter inside a web service.That counter can be considered as a resource just like file or other resources. Counter variable is accessed by multiple service methods, some reads some writes. So the first way comes to our mind would be C# lock (VB.Net SyncLock) keyword. When we implement it there are chances that we may write lock statement in multiple places. Below is a mechanism to avoid that by using functional techniques of C#.
Please note that in real world no one should use this mechanism for integers as far as Interlocked class is there in .Net framework. This is just a sample for understanding purpose.

Wrap the lock statement in fuction and pass the actions into it

The technique here is simple as anyone might have guessed. Just have a wrapper around the lock and inside the locked section execute the real manipulation code which is passed into it as Action<T>. Lets see the code snippet below.
internal class ThreadSafeCounter
{
    private int internalValue;
    private object lockObject = new object();
    public int Value
    {
        get
        {
            return ExecuteWithLock(() => { return internalValue; });
        }
    }
    public ThreadSafeCounter(int initialValue)
    {
        this.internalValue = initialValue;
    }
    public void Increment()
    {
        ExecuteWithLock(() => { internalValue++; });
    }
    public void Decrement()
    {
        ExecuteWithLock(() => { internalValue--; });
    }
    public void Reset()
    {
        ExecuteWithLock(() => { internalValue=0; });
    }
    #region Helpers
    private void ExecuteWithLock(Action action)
    {
        lock (lockObject)
        {
            action();
        }
    }
    private int ExecuteWithLock(Func<int> fun)
    {
        int result = 0;
        ExecuteWithLock(() => result = fun());
        return result;
    }
    #endregion
}
The main functions to notice here are as below.

ExecuteWithLock(Action a)
ExecuteWithLock(Func)

The lock(){} is there only in one function named ExecuteWithLock() and all other functions those wants to execute critical section of code has to call ExecuteWithLock and pass the code as parameter. That code passing as parameter is obtained by the well known Action<T>. Hope there is no need of further explanation.