Hadoop Cluster – Pre Maintenance procedure

In IT, it’s inevitable that all the servers will go for monthly security, vulnerability patching and hadoop servers are no exception. There’d be a separate Systems team to perform OS related patching, security updates, etc and your role is to bring down/up the cluster, ensure the application is good post patching. You’ve have to schedule […]

Continue Reading

CCA131 – Exam Answers Validation Steps

In this post, I’ll explain some of the ways to validate the problems you’ve solved in the exam. Using this steps, you can ensure that the you’ve solved the problems as expected and avoid shocks/surprises in the exam result. When you complete the exam before the alloted time and if you have free time of […]

Continue Reading

Determine reason for application failure

There are many possible causes for a job/application failure varying from code error, environments, files availability, permissions, mapreduce/yarn configuration, resources allocation and even due to server i/o, network issue etc., So the first thing you’ve to do when a job fails is, to look at the error message and correlate with your job. If an […]

Continue Reading

Benchmark the cluster (I/O, CPU, network)

Benchmarking is the process of stress testing the resources of the cluster. It’s very useful in understanding the performance of your cluster and to check whether it’s performing as expected before taking it live. Here we are going to test speed in which files are being read/write in HDFS, time taken for mappers/reducers to process […]

Continue Reading

Resolve errors/warnings in Cloudera Manager

This is a typical scenario based question and the solution is solely depend upon the errors/warnings appears in the cluster.   Some examples: The warnings could be space issue, service health status, low resources allocations, etc., The errors could be log directories are full, services down and other critical events. In these scenarios, click on […]

Continue Reading

Execute file system commands via HTTPFS

HttpFS is a service that provides HTTP access to HDFS. i.e we can access the HDFS from other filesystems from browsers, and using programming languages. HttpFS has a REST HTTP API supporting all HDFS filesystem operations (both read and write). Using HttpFS, we can Read and write data in HDFS using HTTP utilities (such as […]

Continue Reading

Set up alerting for excessive disk fill

Alert Publisher, one of the Cloudera’s management services, used to send alert notifications by email or by SNMP. Service instances of type HDFS, MapReduce can generate alerts if so configured. Alerts can also be configured for the monitoring roles that are a part of the Cloudera Management Service. Go to CM – Cloudera Management Service […]

Continue Reading

Configure a service using Cloudera Manager

Configuring a service is one of the hardest tasks in Apache Hadoop, but Cloudera Manager has made our lives easier. For any configurations, you only have to give the values for the property, then Cloudera Manager will take care of updating the dependent services configuration, updating all the conf files (hdfs-site, yarn-site, core-site,etc) and deploy […]

Continue Reading