Introduction
Logging is an essential and integral part of every application. A good logging system keeps a trail of each and every aspect of an application, eventually helping developers and support teams analyze issues and resolving them. Log statements come in many flavours: application exceptions, events and debugging information.
Logging follows a hierarchy- TRACE , DEBUG, INFO, WARN, ERROR and FATAL. The
most commonly used are DEBUG, INFO and ERROR . While DEBUG is mainly used to log information helping analysis of query, request details etc; INFO is mainly used to define the states like the start and end of a function, processing times etc. Exceptions or Errors are logged using ERROR.
Application logs give critical hints on underlying issues. However, when the application needs maintenance or support, there is a need to monitor the logs continuously, which poses a major challenge. To address this, clients started looking out for solutions to automate the monitoring and alerting, as a Non Functional Requirement. In simple terms, the monitoring and alerting mechanism looks out for exception scenarios to alert developers to take timely, corrective actions. Primarily, monitoring and alerting platforms rely on the outcome of the log analysis.
Datadog is a widely used Monitoring and Alerting tool that can work with many platforms. It is used for both Infrastructure and Application monitoring. This article is a quick guide to integrate Datadog with AWS CloudWatch Logs for Application Monitoring.
Monitoring and Alerting
Monitoring is the task of watching out systemic errors, to ensure application’s health. Alerting ensures any issues are promptly communicated to the respective people, who can resolve them quickly to avoid potential business disasters.
They both go hand-in-hand. Nowadays, there are many more systems and applications to handle with myriad integration within internal and/or external sub systems. So, eventually, the process of monitoring with a pair of eyes on the dashboard is practically impossible. Additionally, each of these applications send out multiple parameters and monitoring them, in real time, to gain insights is practically impossible.
Of course, industry has reacted to the challenge and created a swarm of tools to monitor systems and alert on critical issues. Anomaly Detection takes the monitoring and alerting to the next level by watching out for critical thresholds that even varies dynamically, in time, employing principles of artificial intelligence.
While this sounds great, the problem is the set up. While AI based alerting makes sense for mission critical applications, a pattern based monitoring is adequate and practical solution that provide very close if not timely alerts to react to issues.
Why being Cloud agnostic
Now a days organizations depend on Cloud providers for the infrastructure to run their applications. There are many ‘Monitoring and Alerting’ tools, that are native to cloud, cloud agnostic and Open source. A few examples are: Datadog, Nagios Core, Prometheus, Amazon SNS, Amazon CloudWatch. With the evolution of multiple platforms, organizations are on the search for ways to extract the flexibility of cloud services. Adoption of a multi-cloud environment gives alternatives, but makes it inevitable to have solutions for migrating applications from one platform to the other.
Cloud providers provide Alerting services that can be integrated with the application
through Application Programming Interfaces alias API. However this integration creates a tight coupling with the services and is a bottleneck when application is migrated to different cloud platforms.
Another drawback of integrating the Alerting tools with application code is the amount of complexity it adds to application . Requirements also includes alerting at a particular threshold so that too many alerts are not generated. Implementing this logic in application also adds to complexity and makes the application heavy. These complexities are also avoided if the integration is done outside the boundary of application.
All these challenges are resolved when we bring in an external tool like Datadog. It takes care of the requirements like easy portability and triggering alerts at a particular threshold.
Datadog
In Datadog, there is a dog watching your perimeter. To cite: ”Datadog is an American monitoring service [?] for cloud-scale applications, providing monitoring of servers, databases, tools, and services, through a SaaS-based data analytics platform”. It searches and analyzes logs from any cloud platform and associates logs related to specific traces and notify users using its Alert feature.
Datadog provides integration with cloud providers like AWS, Azure, GCP and Oracle. This tutorial explains how Datadog samples the application logs from AWS CloudWatch and notifies through email alerts. The experiment is done using the trial version of Datadog and personal AWS account. The trial version of Datadog is for a 14 period and helps to do a proof of concept. A high level conceptual diagram is shown in Figure-1
Leave a Reply