Tuesday 20 August 2019

AI vs ML vs DL vs....

We hear so many terms associated with data science nowadays that it's becoming confusing. Also, there are many companies who claim that they have an AI based solution or product to solve your problems. Are they telling the truth? How do you understand what they are talking about? What exactly is the difference between all these terms? Here's an attempt to explain that.

First, let's look at some of the most commonly used terms (reference - Sonix.ai)-

  • Artificial Intelligence (AI) -the broad discipline of creating intelligent machines
  • Machine Learning (ML) -refers to systems that can learn from experience
  • Deep Learning (DL) -refers to systems that learn from experience on large data sets
  • Artificial Neural Networks (ANN) -refers to models of human neural networks that are designed to help computers learn
  • Natural Language Processing (NLP) -refers to systems that can understand language
  • Automated Speech Recognition (ASR) -refers to the use of computer hardware and software-based techniques to identify and process human voice
A good way to understand how they are all interconnected is by looking at this picture below -

The intertwined buzzwords
What is AI?

AI is a broader concept of machines being able to behave like humans would. Human are able to intelligently analyze a lot of variables around them and act accordingly. But, human mind is not just about analysis. It is also about learning based on that analysis and getting better at it. It is about using wisdom to make judgments and act accordingly. AI is about that science. It is that concept.

AI has been around for a while. When we first built computers, we were able to replicate some aspects of human brain based on the understanding at that time. However, as we learnt more and more about our brains, the concept of AI has changed. We have moved on from just replicating increasingly complex calculations to mimicking a human decision making process and intelligent actions based on that. So now, AI is not about just identifying the best driving route based on traffic conditions - it is about building a system that will drive the car like a human would and take decisions along the way, especially in an emergency situation.

What is ML?

ML is a subset of AI. It refers to a system that is able to learn by itself without any human intervention. As you can already see, every AI based solution may need this. However, it is not mandatory (though highly desirable). The more a system learns, the better it is at providing an output.

It is used at many places. When Netflix suggests some movies for you to watch, it is doing so based on what it has learnt from the user experiences and ratings of all of Netflix's users and your personal movie watching history. It can do so based on an insane amount of data available to it and using specific ML algorithms to come up with suggestions. These algorithms are first created by humans and fine-tuned. But once they put into production, they can be auto-fine-tuned based on the newer data that keeps getting generated every minute.

What is DL?

DL is a subset of ML. It refers to ML applied to a very large data set (usually labeled data set). The concepts used in DL are same as ML and everything associated with ML applies to DL as well - the difference being the amount of data being processed. Because of this, DL requires huge amount of processing power.

What is ASR?

ASR refers to the ability of a system to understand the speech and convert it into text. It is a tough ask since every person speaks in a different way, has a different accent, pitch, bass, timbre associated with his/her voice. Because of this very specific nature of ASR, it is more a technological solution than ML or intelligent solution. Having said that, ASR can and usually also uses ML to improve the accuracy of speech to text translation.

What is NLP?

NLP refers to processing of the text (quite often the output of an ASR system) and deriving meaning out of it. Humans tend to use a lot of slang while speaking and hence deriving meaning out of the text that is very contextual is not an easy task for the machine. This is one of the fastest growing (and complex) data science field and all big guns are trying to build the smartest NLP solutions that can then be used for various scenarios.

What is ANN?

ANN is one of the ways ML can be implemented. It refers to a computer system that is designed to work by classifying information like humans brain does. Various machines (often virtual) act as neural network nodes. They are placed in a certain way to start classifying the data and taking decisions based on it. There is also a feedback loop provided so that the outputs of each node can be fine-tuned to improve accuracy automatically. Essentially, the nodes and pathways between the nodes mimic the neural network in human brain - and that's why the name.


The next time you hear someone talking about these terms in relation to their solution/product, think of what exactly the product is and try to understand more about which of these things are being used there. Since AI is the overarching concept, they may not be technically wrong when they use the term in relation with their product. However, it becomes important to know if they are trying to mislead you into thinking that they are providing a truly AI based product or if they are providing a subset of it.

Saturday 3 August 2019

To NoOps from DevOps - Dream vs Reality

We have come a long way in SDLC methodologies - from waterfall to iterative to agile to DevOps. In this evolutionary process, the main focus was on faster deployment of solutions and quicker feedback mechanisms, thus ensuring a quality product faster. The main aim of DevOps especially is to increase the rapidity and efficiency of software deployment process through collaboration between development and operations teams. But what if the deployment, monitoring and management of deployment software is completely automated? Will we need ops team anymore?

(Courtesy - C3DNA)
Mike Gaultieri of Forrester Research coined the term NoOps, where he surmised that developers will never need to talk to the operations team. This definition was further expanded by Netflix's Adriam Cockcroft. He explained that at Netflix there was no ops organization involved in running their cloud, no need for developers to interact with ops team to get things done, and less time spent in actually doing ops tasks than developers would spend explaining what needed to be done to someone else. Basically his definition of NoOps was - Automation of deployment, monitoring and management of applications. (Ref - Is NoOps the end of DevOps? Think again)

What is the difference between DevOps and NoOps? It's pretty straightforward. While DevOps talks of better collaboration between Development and Operations team by putting them together in order to speed up the creation and deployment of quality product, NoOps is about the same but without the need of having an operations team at all! It's role is taken over by automated software deployment products. So, while DevOps is more of a collaboration philosophy with or without specific software, NoOps is more about automated deployment software than collaboration.

Several PaaS (PaaS) solutions to automate deployment process are available on the market and give to developers with no skills about operations the chance to launch applications autonomously, without the need of request system admins help. Indeed, the operations team would simply prepare the infrastructure, with the aim of providing to developers an environment ready to deploy applications quickly and easily. PaaS solutions described above provide a set of services to orchestrate and manage the execution, scaling the infrastructure according to each application need. In addition, PaaS solutions include a significant number of features, including the administration of resources, servers and operative systems maintenance.According to NoOps model, thanks to these platforms developers can work autonomously and, especially, they can do without the support of system administrators. (Ref - DevOps vs NoOps)

However, everything is not as rosy at it sounds. While it is quite tempting to think that deployment and related operations support can be completely automated, it is not necessarily the case always. There are some practical issues that need to be considered -

  1. Many organizations still have a lot of legacy software and that cannot be easily transformed to work with the NoOps model.
  2. Smarter edge devices mean more data being produced and even though this data can be interpreted by machines, decisions would still need to be made in many cases by humans - including oversight. The role of ops team may change in such cases, but the team still remains.
  3. Regulatory compliance and security related issues cannot be left completely to the systems. There will have to be Ops team oversight on this.
  4. No system is foolproof and manual interventions will be needed at some point - especially when a disaster strikes and disaster recovery is needed.
While NoOps might a good way to go in certain cases, in vast majority of cases it might not be a viable option. However, It certainly can be considered as the next step in evolution of SDLC after DevOps. Even if we can't reach the completely NoOps model, improving DevOps and moving more towards automation of repetitive processes in testing or deployment will take us one step closer. So, while NoOps replacing DevOps might be a dream, a more automated DevOps is certainly becoming a reality.