We are in the second week of the Linux Hadoop images by Ali Hadi and Magnet has also kindly posted a recap for October's challenges. On to this week's challenge question!
This week's challenge is split into two parts. The first question:
Challenge 6 (Nov. 9-16) The Elephant in the Room
Part One: Hadoop is a complex framework from Apache used to perform distributed processing of large data sets. Like most frameworks, it relies on many dependencies to run smoothly. Fortunately, it's designed to install all of these dependencies automatically. On the secondary nodes (not the MAIN node) your colleague recollects seeing one particular dependency failed to install correctly. Your task is to find the specific error code that led to this failed dependency installation.
This question had me looking in the /usr/local/hadoop directories for program logs initially but there was nothing noteworthy other than some messages relating to failure to connect to the master server. Reading the Hadoop documentation also did not turn up any information on automatic installation of dependencies or where such logs would be contained. So I did the next best I could and retraced what the user did on the box.
Checking the /home/hadoop/.bash_history logs to see what I could find, I noted that the user attempted to install java via apt-get but did not see any command related to automatic installations. My next best guess was in the /var/log/ directory and here is where I found that which we were seeking.
In /var/log/apt/history.log screenshot above, we see some failures for installation of Java at 2017-11-08 01:17:04. It so happens that Java is a required software for Hadoop (i.e. dependency) which leads me to think I am close.
Checking the corresponding /var/log/apt/term.log around the time in question per the above image, we see that the Java installation failure was due to a HTTP 404: Not Found error. So I tried the answer "404" and bingo!
Answer (part 1): 404
Challenge 6 Part 2 (Nov. 9-16) The Elephant in the Room
Don't panic about the failed dependency installation. A very closely related dependency was installed successfully at some point, which should do the trick. Where did it land? In that folder, compared to its binary neighbors nearby, this particular file seems rather an ELFant. Using the error code from your first task, search for symbols beginning with the same number (HINT: leading 0's don't count). There are three in particular whose name share a common word between them. What is the word?
I understood the 'ELFant' reference was referring to the size of the file and that this was related to whichever version of Java the user managed to successfully install. A quick search for Java on the system pointed me to the /usr/local/jdk1.8.0_151 directory.
Searching for the largest 'binary' file (I took it to mean 'non-text' initially) had me poking in the src.zip file for symbols begining with 404. Finding nothing that could be the answer, I re-read the question and it finally clicked that the 'ELFant' reference was referring to both the size of the file, and the type of file (ELF file)! Nice wordplay by Magnet Forensics there. With that, it was straightforward to locate the largest ELF file (unpack200) and search for the appropriate symbols using the readelf command with -s option to display the symbol table (ignoring leading zeroes per the hint).
Answer: deflate
Note: I initially solved the challenge by working on the HDFS-Slave2.E01 image but the write-up was done by retracing my steps on the HDFS-Slave1.E01 imaage. So it doesn't matter which of the two slave nodes one worked with, the answer would be the same for both.
No comments:
Post a Comment