Tuesday, August 22, 2017

Starting Scala Spark - Setting up local development environment

When someone comes to me and says 'this can be or cannot be done using .Net, SQL or Browser', I know whether it is really possible or not. Recently someone came to me and said 'we cannot connect to SQL Server database from Scala and load data frames, if Scala is running inside Azure HDInsight Spark cluster'. To be frank I did some google immediately but there are no direct answer to that scenario. In the internet, every one talking about loading data from Azure blob and all into Spark data frame and doing parallel data processing. But, if Spark is an in-memory distributed execution technology, why can't it read from SQL Server database and load data frame and do the processing? Scala is just another JVM language. Spark is written in Scala so it too runs on JVM and JVM has connectivity to SQL Server. So theoretically yes, but I don't have confidence to say practically yes.

It leads to the dilemma of  'should Software Architects code' or depend on what the development team says? Since I am in the 'yes should code' side, I decided to start exploring Spark so that next time I should be able to handle Spark same like how I am handling .Net and browser side technologies. 

The topics I should cover are Scala, Spark, Azure HDInsight.

This post just aims to some one who wants to learn Spark data processing technology at high level so that it can be integrated to other applications. Not to go in depth to become Spark data processing expert programmer.


First step is to setup the development environment. Learning Spark or Scala in Azure HDInsight environment is not a cost effective decision. There is already good tutorials explaining the development environment setup. One such is given below which talks about setting up Spark environment in Windows machines.


This is little old article. It worked for most of the steps. Below are some we may need to revisit for latest versions

Spark version

The instructions are little old regarding the version numbers though the concept remains same. One is regarding the Scala version which is suitable for Spark. The idea is to first select the Spark version and then select the Scala version. At the time of writing this article, the Spark version is 2_2_0 and corresponding Scala version is 2.12.3 (Spark works from Scala 2.11 on wards). The Scala version support is mentioned in the Spark downloads page.


Another dependency is the winutils.exe which has to be in the \hadoop folder. It can be downloaded by googling. One such link is given below.

The location of the winutils can be C:\hadoop\bin\winutils.exe where the PATH environment variable points to the c:\hadoop folder.

After following the steps mentioned in the tutorial, we may get into multiple different issues. One such issue is given below.


This may happen once we run the Spark shell after installation. The problem is that the Spark context variable 'sc' is not initialized. If this is not, we cannot do any Spark activities though we can issue Scala code. The error message is as follows. The font size is reduced to view more in less space.

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
17/08/22 17:33:26 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/08/22 17:33:29 WARN General: Plugin (Bundle) "org.datanucleus.store.rdbms" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/spark_2_2_0/jars/datanucleus-rdbms-3.2.9.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/spark_2_2_0/bin/../jars/datanucleus-rdbms-3.2.9.jar."
17/08/22 17:33:29 WARN General: Plugin (Bundle) "org.datanucleus.api.jdo" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/spark_2_2_0/jars/datanucleus-api-jdo-3.2.6.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/spark_2_2_0/bin/../jars/datanucleus-api-jdo-3.2.6.jar."
17/08/22 17:33:29 WARN General: Plugin (Bundle) "org.datanucleus" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/spark_2_2_0/jars/datanucleus-core-3.2.10.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/spark_2_2_0/bin/../jars/datanucleus-core-3.2.10.jar."
17/08/22 17:33:35 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
17/08/22 17:33:35 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder':
  at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:1053)
  at org.apache.spark.sql.SparkSession$$anonfun$sessionState$2.apply(SparkSession.scala:130)
  at org.apache.spark.sql.SparkSession$$anonfun$sessionState$2.apply(SparkSession.scala:130)
  at scala.Option.getOrElse(Option.scala:121)
  at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:129)
  at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:126)
  at org.apache.spark.sql.SparkSession$Builder$$anonfun$getOrCreate$5.apply(SparkSession.scala:938)
  at org.apache.spark.sql.SparkSession$Builder$$anonfun$getOrCreate$5.apply(SparkSession.scala:938)
  at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
  at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
  at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
  at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
  at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
  at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:938)
  at org.apache.spark.repl.Main$.createSparkSession(Main.scala:97)
  ... 47 elided
Caused by: org.apache.spark.sql.AnalysisException: java.lang.RuntimeException: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwx------;
  at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106)
  at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:193)
  at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:105)
  at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:93)
  at org.apache.spark.sql.hive.HiveSessionStateBuilder.externalCatalog(HiveSessionStateBuilder.scala:39)
  at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog$lzycompute(HiveSessionStateBuilder.scala:54)
  at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog(HiveSessionStateBuilder.scala:52)
  at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog(HiveSessionStateBuilder.scala:35)
  at org.apache.spark.sql.internal.BaseSessionStateBuilder.build(BaseSessionStateBuilder.scala:289)
  at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:1050)
  ... 61 more
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwx------
  at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
  at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:191)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:264)
  at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:362)
  at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:266)
  at org.apache.spark.sql.hive.HiveExternalCatalog.client$lzycompute(HiveExternalCatalog.scala:66)
  at org.apache.spark.sql.hive.HiveExternalCatalog.client(HiveExternalCatalog.scala:65)
  at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply$mcZ$sp(HiveExternalCatalog.scala:194)
  at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:194)
  at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:194)
  at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
  ... 70 more
Caused by: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwx------
  at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:612)
  at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:554)
  at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508)
  ... 84 more
<console>:14: error: not found: value spark
       import spark.implicits._
<console>:14: error: not found: value spark
       import spark.sql
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.2.0

Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_144)
Type in expressions to have them evaluated.
Type :help for more information.

scala> val sf = sc.textFile("C:\\Temp\\joygazure.publishsettings")
<console>:17: error: not found: value sc
       val sf = sc.textFile("C:\\Temp\\joygazure.publishsettings")


If we analyze the log carefully we can see there is a permission issue to tmp folder related to hive. Did we install hive? Not explicitly. What is the relation with initializing Spark context and hive folder permissions. At first it might not seems related but ultimately they are related.

c:\hadoop\bin\winutils.exe chmod 777 c:\tmp\hive

Tuesday, August 15, 2017

Difference between singleton object and static class members

Long back during one of interview this question came. Why should we use singleton instead of static class with members? Are they same? Some thoughts on the same below.

Singleton objects and static classes have their own purposes. The difference between those may be too little but there are differences. Lets see what is meant by these 2 varieties.

class Context
        private static Context internalContext = new Context();
        private Context() { }
        public static Context Singleton { get { return internalContext; } }
        public string GetEnvironmentSettings() { return "settings"; }
public void DoBusinessOperation()
        string settings= Context.Singleton.GetEnvironmentSettings();

static class StaticContext
        public static string GetEnvironmentSettings() { return "settings"; }
public void DoBusinessOperation()
        string settings = StaticContext.GetEnvironmentSettings();

Technically the above 2 methods achieve same thing. But we have to understand when should we use the singleton class based approach and static class members approach.

Singleton object can be inherited static class cannot.

The above code was running perfect in an web application which was hosted in onpremise data center. Then the cloud migration came and the strategy is to support onpremise and Azure cloud for a period of 1 year and after that Azure only. Suppose getting environment settings is different in Azure than onpremise, how can we make this code support the scenario without having different codebases?

Don't take the above scenario as is. Just assume we came across a scenario where the behavior of Singleton has to change based on running environment and how to support it without having to maintain many code bases?

Yes its a solved problem. We should use inheritance and change behavior based on the environment. If we go with Singleton model we can have the class derived from another base and expose the base to outside. Below goes the code snippet.

class Context
        private static Lazy<Context> internalContext = new Lazy<Context>(() =>
            //Logic to create the correct object based on criteria. Environment in this case.
            return new AzureContext();

        public static Context Singleton { get { return internalContext.Value; } }
        public virtual string GetEnvironmentSettings() { return "settings"; }
        private class OnPremiseContext : Context
            public override string GetEnvironmentSettings()
                return "settings";
        private class AzureContext : Context
            public override string GetEnvironmentSettings()
                return "Azure settings";

The above code is just for demonstrating the advantage of using Singleton object over static class members.

Singleton can be disposed,static class cannot

In reality we may not meet a situation where a Singleton object needs to be disposed. For argument sake we can say that if we need Singleton object at a thread level (thread static), this is useful. Once that thread completes execution we can dispose the object.

Passing singleton object as param, static class cannot

This is similar to the inheritance scenario but another advantage of using Singleton object. Ideally Singleton is a mechanism where we can access it from anywhere in the code by using its class name. It doesn't need to be passed into a function. We we want to make our functions testable, we can think of accepting the Singleton objects. Also we can use dependency injection mechanism to unit test the code without any overhead of actual singleton object.

There are many other differences between these models but mainly depends on these 3. Since the Singleton is an object it can be cloned but static class cannot. Storage of data in process memory, performance between these 2 models etc...are some other differences.



Tuesday, August 8, 2017

Caution JavaScript Ahead - Hoisting

What is variable hoisting

In the 'var' world of JavaScript, the variable is not bound to the boundary of block level curly braces {}. Even if we declare the variable inside an if loop, it gets moved to the function where if belongs. This is not the case in other languages such as C,C++, Java, C# etc...There the variable is limited to the boundary of block level braces but never in the case of JavaScript. That seems a developer friendly feature in JavaScript at first. But it can cause issues, if we write code without knowing what is variable hoisting.

Below is a good example


function foo() {
  if (new Date().getMinutes() % 1 === 0) {
    var time = new Date();

If we just look at the code we may conclude that the alert will always shows undefined. But if we know hoisting, we can easily identify the difference here. If the condition is true, the time variable is going to have value.
Even if the statements inside the condition didn't execute, we don't get any runtime exception in strict mode. It just shows undefined. Lets see below code.


function foo() {
  function innerFoo() {
    if (new Date().getMinutes() % 1 === 0) {
      var time = new Date();

Here always we get exception if the code is in strict mode. This is because the time variable never gets hoisted to the outer function.

Variable hoisting and global variables

Below is another interesting scenario if global variable or variable in parent scope has same name as inner variable

time = new Date();

function foo() {
  var time = new Date();

When we run the above snippet. We could see that first alert shows the time. Second one shows undefined and third one shows time again. Why the second one shows undefined? Lets see how the variable hoisting worked here. Below code snippet shows the code after hoisting.

time = new Date();

function foo() {
  var time;
  time = new Date();

Of course we cannot see the hoisted code since JavaScript don't compile to output file. But this is what happens.

Does this mean global variable is not safe

The above may cause confusion that global variable is altered by the variable inside function. But it is not. Lets see the below code snippet.

time = new Date(1947,7,15);

function foo() {
  var time = new Date();


When we run we could see that the 4th alert is showing same date of first alert. That means the global variable is not altered by the function since it is different variable with same name. This is similar to the other languages. But this is true only to the variables out side of function. What if the outer variable is inside the function and inner is inside an if statement?

function foo() {
var time = new Date(1947, 7, 15);
if (time.getYear() < 2000) {
var time = new Date();

We could see that the 3rd alert shows current date not the old one initialized at the beginning of the function. This shows the hoisting is to the immediate function level and it is same variable through out the function.

Hoisting functions

In the above code snippets we can see that the foo() is called before it is defined. JavaScript has no problem with that. But lets see the below scenario. Guess what will happen

var foo =function () {
var time = new Date();

It complain that foo is not a function. Many of us might have spent good amount of time troubleshooting why this is not a function at first. The reason is hosting. When it does, it moves the variable foo declaration to the top of the function ie above the foo(); statement and when we call foo(); it is not yet a function as the function definition is assigned to that variable later.

If we move the calling foo(); below the assignment it starts working.

Enter let for block level scoping

The keyword 'let' is introduced into JavaScript ES6 / ES2015 to have block scope. ie the variable can be scoped to an if block than to function level. Lets have a look at below snippet.

function foo() {
var time = new Date(1947, 7, 15);
if (time.getYear() < 2000) {
let time = new Date();

We could see the above code shows old date. The only difference is the usage of let for declaring variable inside if block.

Does let hoist

If we use a variable before it is declared with let, it throws exception. But if we use a variable before it is declared using var it has undefined. Means hoisting is not happening as is in let.

function foo() {
try {
var oldTime = new Date(1947, 7, 15);
} catch (ex) { alert(ex) }
try {
let time = new Date();
} catch (ex) { alert(ex) }
try {
} catch (ex) { alert(ex) }

We could see in the above code, the oldTime gets hoisted with undefined value. The let throws exception similar to an real non declared variable noTime.

There are more in to 'let' such as temporal dead zone and restriction of redefinition etc... The new constructs such as class in JavaScript works the same way of let. Classes in JavaScript don't get hoisted.

If we think about different scenarios, things gets very much interesting. Eg: What if we declare same variable using var and let in same scope, different scopes, how they overlap etc...See the reference section for more such interesting scenarios.


Tuesday, August 1, 2017

Architecting Azure limits - Azure Search

When we look at Azure evangelism, we could see that it is projected as a platform where we can scale to infinity. Some even go to a level where scale is magic. Just put the code in Azure and it scale automatically without we worrying about anything. But not all are true. When we design in Azure platform we are limited to scaling characteristics of each service. Often it demands us to design application or support tools to align with the nature of Azure service(s) we selected. To do it efficiently we should know the limits of Azure services. In other words our Azure architecture revolves around Azure limits.

Lets take Azure Search in this post. Please note that these limits might be changed in future by Azure platform. So it might not be accurate for long. 

Limits of Azure Search

Total services per subscription - 43 (Among this there are limits in different tiers eg: only 6 services in s3 tier)

Data storage

Maximum number of indexes in search service - 200 (S2 & S3 tiers)
Partitions per service - 12
Max partition size - 200GB (S3 tier)
The above 2 define the max index size  = 200GB*12
Maximum documents - 120 million/partition or 1.4 Billion / service


Replicas per service - 12( S1,S2 & S3)
Estimated queries per replica - 60/second 
The above limits total of 12*60 = 720 queries per second.
The latest limits can be found in the below link.

This clearly indicate that using one subscription we cannot build another google. May be Azure will increase the limits later. The advantage here is that the index is doing some kind of compression so the source data size doesn't directly translate to index size.

Azure search in SaaS / Multi-tenancy scenarios

If we really want isolation of tenant data, we should put one tenant data into one services. But there is limit of 6 in the higher levels. Next level of isolation is putting tenant data by search index. The search index gives us different URL to get better isolation. But in this approach, we are limited to 200 indexes / service. When we gets 201th tenant, we have to add one more service. That is either a change in the application or support tool specific to the limits of Azure search service. 

The last resort is to tag the documents by tenant. But that is less secure. More design guidelines related to multi-tenancy can be found below.

If we distribute the data in multiple indexes the query has to change since the out of box querying is towards an index. If we want to search one search term in multiple indexes, it translate to one http call to one index.

Azure search service has got really nice features which otherwise would be difficult to code ourselves. But as any other abstraction, it comes with limits. We should be careful when designing the application by not falling into the marketing materials and magic claims.

Is Azure search really pay per use?

Since it is offered as PaaS, we may assume that it is pay per use. But that is not true for Azure search. When we provision a search service it starts charging regardless of the usage(data stored and the search queries issued). More details on pricing can be found in the below link.

The advantage here is that we gets one more level of scaling inside the service which has pricing tier defined. The scaling inside a search service is called Search units.

Is Azure search auto scaling

Not fully automatic. When we create a search service there will be a default search unit. When we want to scale we have to increase it explicitly. This is another knowledge our application or support tool has to have in order to scale. Else human intervention is required when a scaling need arises such as more data getting added or more users are expected in busy season. More details below.

In simple sense when the data volume increased increase the partitions and when queries increase increase the replicas as the query limit is per replica.

Tuesday, July 25, 2017

Discrepancies in Azure service models and APIs

Different Azure services uses difference models of consumption. This post aims to list down those differences. Below are the categories where we can see differences.
  • Service naming model
    • Unique name inside Azure
    • Unique name inside subscription
  • API URL pattern
    • Instance specific URL
      • Instance name as sub domain
      • Service instance name in URL path
    • Generic URL
  • API authorization.
    • Key - Azure Search
    • Time limited tokens - Translator Text API
  • Passing API authorization
    • Inside URL as query string
    • Inside http headers

Service naming model

This refers to how we create the Azure service. There are mainly 2 types unique in the entire Azure v/s unique in subscription

Unique name inside Azure

Most of the services require us to create a globally unique name. That name will often end up as sub domain or part of URL. 

eg: Storage account

The problem with this mechanism is creation of new service instance from the application on the fly. For example, assume we are offering a SaaS solution and database isolation is one DB / tenant. There is a limit of 5000 databases per Azure SQL Server. When our 5001 tenant registers, we have to create next SQL Server instance from the program. Since the server name has to be unique across entire Azure, our program has to do trial and error methods to get unique name. To workaround this problem, we can take different routes such as generating GUID, create instances upfront etc... But it is something to be considered during the development process. Else it just fails.

Unique name in Subscription

Others end up in creating a name which is unique across the subscription.The AppInsight is the example for such a service.

API URL Pattern

All most all the services are designed to be consumed via http(s) web service endpoints. Mainly ReST based APIs with JSON data format. The URL pattern differs per service. Some service demand URL with its name and some can be accessed using generic URL.

Instance specific URL

Here the name of the service instance is needed in the API URL as sub domain or in the path.

Instance name as sub domain

The Azure storage account is one example for a service which demands globally unique name which ends up as sub domain.

Eg: https://<storage account>.blob.core.windows.net/ 

The services such as WebApps really demands the URL separation using sub domain.

Service instance name in URL path

If we look at some services we can see the service name is appearing inside the URL path after the domain.

eg: Analysis service  asazure://westus.asazure.windows.net/<analysisservicename>

Generic URL

In this category, there won't be any URL which has the service name in it. Instead they use a generic URL to hit the Azure service and identify service via parameters. Parameters can be request headers containing API keys or tokens.

eg: Application Insights

API Authorization

Authentication & authorization are essential factors in web service based API world. There are 2 different methods we can see in Azure services.

Key based

Key is same as in real world. The key can be used multiple times, until the lock is replaced. Similarly in Azure world, the validity of key is till that key is replaced. If anyone obtain our key, they can keep  using our service till we identify the theft and replace.

eg: Azure search. It demands us to put the key in the header.

Token based

In simple sense, token are short lived keys. Using a key or some other mechanism, we can create a token and use it for predefined time. Once it expires we have renew the token. Tokens has better security but difficult to use.

eg: Text Translator API. Details available here on how to use from Postman.

Passing API Authorization

Another area is how we can pass the authorization whether it is key or token

Inside URL as QueryString

Some services accept the authorization via the URL in the form of query string. 

Inside http headers

Other services needs the authorization in the header. eg: Translator Text API

More categories

API Version

Some services such as Azure Search needs version in the API. The versioning is in the form of date 
eg:https://<search service>.search.windows.net/indexes/<index>/docs?api-version=2016-09-01&search=<search key>&highlight=content

But the Translator Text follows v# versioning eg:https://api.microsofttranslator.com/v2/Http.svc/GetTranslations?text=jaguar&from=en&to=es&maxTranslations=5

Another way we can find for versioning is in Storage. It demands the version inside the http headers. Refer for details. Also it needs version as x-ms-version header.

Some random thoughts

Ideally as per the API guidelines (even from Microsoft) all the APIs which an application or platform provides should follow same standards. They should look like all are designed by same person. That way, consumers don't have to invest much time learning the APIs and usage. 

But in case of Azure, we cannot proceed with that principle. We have to learn to use different services separately. Why Microsoft Azure service APIs has this discrepancy?

We cannot assume that the teams who designed & developed APIs don't know this basic principle of API design. They should be having their own reasons why they did so. If we look at the Translator Text API, may be team thinks that, it should be really protected using auto expiring tokens where the Azure search team thought, search results are less secure and just needs to be protected by a key. May be from consumers point we value our search result data more than a generic translation method.

Another reason might be legacy code. Means connecting legacy code and putting into production may end up in having different API models. Also if a product is bought from other company via acquisition or so, it may end up differently. At the end the time to market matters which is nothing but money. Sometimes when weighing the money we will get due to fast release v/s beautifying the API, money gets priority. 

Competition matters here too. Sometimes first-mover strategy gives a big advantage and it may lock in clients.

Whatever the case, it seems there are no design notes publicly available describing the rationale behind these different decisions similar to Angular. We have to take our best guess why it is so and learn it. 

Tuesday, July 18, 2017

Azure Cognitive Services - Translator Text API - Calling from Postman

Azure cognitive service suit offers a lot of different APIs. One if it is language translation. It is named as Translator Text API. The good point is that it is offered as freemium meaning there is a free tier.

Calling the Translator API

Since it is service it is offered as web service. There are so many samples to use this service from different languages such as C#, JavaScript, PHP etc... which are available in their GitHub repo. But there is tutorial how can we simply call it from Postman which is web developers best friend to consume APIs.


Unlike many other APIs, this involves 2 step process. Steps given below
  1. Get the token from Azure using STS url which is currently https://api.cognitive.microsoft.com/sts/v1.0/issueToken. This URL is present in Azure portal when we navigate to Translate service instance. This may change in future.
    1. Calling this URL has to be POST with "Ocp-Apim-Subscription-Key" in the header. The value for this header is the keys present in Azure portal.
    2. The result of this call will give us a short lived token which has to be used in next step
    3. Once the token expires, we have to get renew. 10 mins is the expiry.
  2. Now using the token we can call the real, web service located at below URL
    1. https://api.microsofttranslator.com/v2/Http.svc
    2. This method accepts some parameters a query string and some parameters via POST / request data.


Below goes the screenshots. 

Getting Translator Text API Keys from Azure portal. Image credits to MSDN Blogs. This key is used in next step to get token.

The above screenshot shows how to call token service from Postman to get the short lived token used for actual API call. The Ocp-Apim-Subscription-Key should be kept secret as it will help others to use our quota. It has to be changed explicitly if it is leaked.

The above image shows the real Translator Text API call. API needs the token obtained in the previous step. The token has to be prefixed with Bearer and a space before using in this call via Authorization header.
This API accepts required parameter in a hybrid way. Some are in the query string and some can be in optional POST request.
Make sure the optional post request body is properly formatted. One such format is given below.

If the POST request body has any format mismatch the request will be rejected by the server.

Happy posting...

References & more reading


Tuesday, July 4, 2017

Adding Azure VM auto shutdown via ARM template

This post is intended to show how can we use a simple Azure feature via ARM template. The feature is 'scheduled shutdown of virtual machines'. It is useful when we play with machines. Seriously, never shutdown production machines unless there is a real business scenario such as your business is strictly running during a particular time and you don't want to keep the system open for hackers to play with. eg: day trading.

Before we go ahead, this article doesn't promote the use of virtual machines. Virtual machines which is in cloud terms 'Infrastructure as a Service' is the last to consider when we design for cloud. One, it is costly. It is not truly pay as use mode. Virtual machines have to run regardless there is workload or not. We have to pay for all these idle times. Also virtual machines take more time to scale out as the boot time is more. By the time, the spike in requests/workload might be over. But there would be scenarios where we have to use VMs. When we deal with older systems without enough budget to rewrite, we have to go for VMs. Also when our corporate IT is not believing in PaaS because of security, VM is the option.

Now coming to the point. Auto shutdown can easily be done from portal as shown in the announcement. There are some sites out there which explains how to setup auto shut down via ARM. This is my experience with the same.

The schema of ARM template can be found in the below link. It is the starting point.

Sample ARM template to shutdown

Below goes JSON to set auto shutdown to a VM
      "apiVersion": "2017-04-26-preview",
      "type": "Microsoft.DevTestLab/schedules",
      "name": "shutdown-computevm-nanotest",
      "location": "[resourceGroup().location]",
      "properties": {
        "status": "Disabled",
        "timeZoneId": "Pacific Standard Time",
        "taskType": "ComputeVmShutdownTask",
        "notificationSettings": {
          "status": "Disabled",
          "timeInMinutes": 15,
          "webhookUrl": "http://joymononline.in"
        "targetResourceId": "[resourceId('Microsoft.Compute/virtualMachines', parameters('nanoName'))]",
        "dailyRecurrence": {
          "time": "1600"
        //"hourlyRecurrence": {
        //  "minute": "1"
We can see webHookUrl is given to a dummy url and hourlyRecurrence is commented. Lets see the details.

Where to add this JSON fragment in ARM template

This is not a sub resource of VM mode. Rather this needs to be directly added into top level resources node in ARM template.

Can we shutdown VM hourly

If we are using an Azure subscription which credits 50$ per month, it is not good to set auto shutdown daily one. As soon as we are done with the VM, it should be shutdown. Even if we forget, it has to shutdown. But currently the feature doesn't support hourly recurrence though the schema supports. That is why it is commented.

Can we omit daily recurrence value?

Below is the error which will pop when we don't give value for daily recurrence. Notice that this error won't be displayed during validation. But throws when we execute ARM template with ARM manager in Azure.

17:48:25 - [ERROR] Microsoft.DevTestLab/schedules 'autoShutDownPolicy' failed with message '{
17:48:25 - [ERROR]   "error": {
17:48:25 - [ERROR]     "code": "MissingRequiredProperty",
17:48:25 - [ERROR]     "message": "Missing required property DailyRecurrence."
17:48:25 - [ERROR]   }
17:48:25 - [ERROR] }'

What is big deal with the name

Below is the error receive when a pet name given to schedule. Again notice that validation don't show but execution throws. In other words its runtime error.

17:55:49 - [ERROR] New-AzureRmResourceGroupDeployment : 5:55:49 PM - Resource
17:55:49 - [ERROR] Microsoft.DevTestLab/schedules 'autoShutDownPolicy' failed with message '{
17:55:49 - [ERROR]   "error": {
17:55:49 - [ERROR]     "code": "InvalidScheduleId",
17:55:49 - [ERROR]     "message": "The schedule should be created in subscription
17:55:49 - [ERROR] 11111111-1111-1111-8888-####ffffeeee, resource group <Storage>
17:55:49 - [ERROR] and with name shutdown-computevm-nanotest."
17:55:49 - [ERROR]   }

The nanotest is the VM name.

APIVersion and region

The sample template above shows apiVersion of schedules '2017-04-26-preview'. It is region specific. Not all regions supporting all API versions. This is again an runtime error. Validation will pass.

There is a URL which tells what are available in what region. There are some programmatic techniques to find out the API version and other details.

The good part is that the error is descriptive, telling us what API version to use. That is really useful.

Rolling back ARM deployment

As mentioned above some errors happen when we deploy/execute ARM template. If the auto shutdown setup fails, does it revert the VM deployment? As software engineers we assume that it will,  same as MSI deployment or there will be some flags to deploy ARM in transaction. But it never roll backs. It is our pain to clean up what is left after a partially failed ARM deployment. Feature request below. It is there for the last one year.


The suggestion is to make our ARM template deployment idempotent. Simply saying even if we redeploy, it should not cause any harm. It should run many times by producing same result as first run. Simple if we are just setting up architecture via ARM. Gets complex if there are custom scripts running as part of deployment.

Debug ARM template

For a simple feature to get working, we may encounter these many errors. As developers we might have started thinking how to debug the ARM templates. In general the error messages are meaningful, if we are familiar with Azure terms. Debugging ARM template itself has scope for separate post. Again, it gets complicated when ARM template uses custom scripts. Below are 2 links talking about troubleshooting ARM template deployments.