Reconnaissance like a cyber scout – Part 2.

Escrito por  Lucas "luriel" Carmo

Salutations fellas! I’m back for the second part of our series on perimeter reconnaissance. Remembering that we reached half of our content, which will be concluded in the third part.

Host discovery and vulnerability identification

The way we treat data is an important point for our creativity, and it is extremely useful for chaining vulnerabilities. In our audits, we have deadlines and with that, we need to be as smart as possible to save project time and this comes through the automation of processes.

In this part we will cover the following subjects:

  • Passive Enumeration
  • Certificate Transparency
  • Domain Name Server Analysis
  • Subdomain Takeover
  • Google Analytics Extraction
  • Host discovery
  • Directory fuzzing
  • Port scan
  • Technologies identify (Web Application Firewalls, CDN, CMS)
  • Screenshot of target websites
  • Javascript Files Analysis

In the current times of cyber security, if we are not on the “other side of the force”, we need to be nimble in our findings, because we don’t know who will have the “low hang-fruits” vulnerabilities and also what will do with them.

I bring the icing on the cake, tools to help the process of exploitation:

  • Passwords dictionary creation
  • Brute-force attacks
  • Extraction of endpoints that have patterns/parameters related to vulnerabilities such as Server Side Request Forgery, Cross-Site Scripting, Open Redirection, Carriage Return Line Feed, Cross-Origin Resource Sharing, Local File Inclusion, SQL Injection, Server Side Template Injection, SSL flaws, Broken Links, Prototype Pollution and Web Cache Poisoning.


Subdomain Enumeration:

Subdomain enumeration is the process of discovering and listing all valid subdomains of a specific domain. This can be done manually or through automated tools such as scripts or applications, with the goal of identifying potential security vulnerabilities or other points of interest for an organization or individual.

We will talk about:

Passive enumeration is an information-gathering technique without directly interacting with the target. This includes searching the internet for public data such as IP addresses, domain names, and information about technologies used to gain a more comprehensive understanding of the target system or network. This collected information can later be used to plan and execute a more effective hack or penetration test.

Amass Project performs network mapping of attack surfaces and external asset discovery using open-source information gathering and active reconnaissance techniques.

Subfinder is a subdomain discovery tool that returns valid subdomains for websites, using passive online sources. It has a simple, modular architecture and is optimized for speed. Subfinder is built for doing one thing only – passive subdomain enumeration, and it does that very well.

GitHub-subdomains find subdomains on GitHub.


  • Certificate transparency (ctfr)

Certificate Transparency is an SSL/TLS certificate security mechanism that seeks to make the issuance and use of certificates more transparent and secure. It works by recording all certificates issued for a specific domain in public records, which can be checked by anyone. This helps detect fraudulent certificate issuance, ensuring end users are confident they are connecting to a legitimate and secure website. In addition, it also provides a mechanism to detect and resolve certificate revocations.

Ctfr allows getting the subdomains from an HTTPS website in a few seconds.
How does it work? CTFR does not use a dictionary attack nor brute force, it just abuses of Certificate Transparency logs.


  • NOERROR subdomain discovery (dnsx)

NOERROR Subdomain Discovery is a discovery technique based on the DNS server’s response to null (empty) requests. It takes advantage of the default configuration of DNS servers that respond with a NOERROR status even when a request does not find a valid entry in the name database. By performing a series of DNS requests with hypothetical subdomain names, it is possible to identify existing subdomains that return the NOERROR status.

Dnsx is a fast and multi-purpose DNS toolkit designed for running various probes through the retryabledns library. It supports multiple DNS queries, user-supplied resolvers, and DNS wildcard filtering.


Brute force is a method of guessing information that consists of testing all possible combinations of characters until the correct combination is found. It’s a simple but inefficient approach that can take a long time to find the right solution.

Puredns is a fast domain resolver and subdomain brute-forcing tool that can accurately filter out wildcard subdomains and DNS-poisoned entries.


A permutation is a reordering of the elements of a set or sequence. In mathematics, it is an operation that generates all possible arrangements or combinations of the elements of a given set. Permutations are used in various fields, including cryptography, artificial intelligence, and algorithmic research, to test all possible combinations of input and find the best solution for a given problem. In our case, permutations are used in brute force tests, where we try to find a solution through trial and error of all possible combinations.

Gotator is a tool to generate DNS wordlists through permutations.

Ripgen generates a combination of domain names from the provided input. Combinations are created based on a wordlist and are based on Rust.

Regulator with automated learning of regexes for DNS discovery.


  • JS files & Source Code Scraping (gospider)

A web spider, also known as a crawler is a computer program that is used to automatically traverse the World Wide Web (WWW) and collect information from web pages. It works by downloading web pages, following links on downloaded pages, and collecting relevant information such as email addresses, images, links, and page content. Web spiders are widely used for purposes such as search engine indexing, market data analysis, social media analysis, and searching for security vulnerabilities. They are designed to work quickly and scalably, visiting thousands or even millions of pages in a matter of hours or days.

Gospider looks for brute force and parses sitemap.xml and robots.txt. Generate and verify links from JavaScript files. Find AWS-S3 and subdomains from the response source. Get URLs from Wayback Machine, Common Crawl, Virus Total, and Alien Vault. Format output easy to Grep. Support Burp input. Crawl multiple sites in parallel. Random mobile/web User-Agent.


  • DNS Records (dnsx)

The Domain Name System (DNS) is a system that provides a translation of names (such as www.example.com) to IP addresses (such as 192.0.2.1). Allowing you to access websites and other network services by typing in an easy-to-remember name instead of a hard-to-remember IP address. When you type a website name into your browser or click on a link, your computer or device requests a DNS server with the goal to obtain the corresponding IP address. The DNS server looks in its database for that IP address and, if found, returns it to your device, which can then connect to the site and load its pages. DNS is a vital part of the Internet’s infrastructure and is used to resolve domain names across the world, ensuring that you can easily access the websites and services you want.

Dnsx was explained earlier in the “NOERROR subdomain discovery” section.


Google Analytics is a web analytics service from Google that allows website and app owners to measure and understand how users interact with their websites and apps. It provides detailed information about visitor traffic, traffic sources, user behaviors, conversions, and more. This allows website owners to improve the functionality and performance of their websites and applications, as well as enhance the user experience.  Google Analytics ID is the unique identifier assigned to each Google Analytics account. It is made up of a set of numbers and letters and is used to identify your Google Analytics account. It is important to correctly include your Google Analytics account ID on every page of your website or application that you wish to monitor to ensure that the information collected is accurate and up-to-date. In addition, you can use the Google Analytics ID to access information about the performance of your website or application from the Google Analytics dashboard.

AnalyticsRelationships is a script that tries to get related domains/subdomains by looking at Google Analytics IDs from a URL. First, search for the ID of Google Analytics on the webpage and then request to build and hacker target with the ID.


  • TLS handshake (tlsx)

The TLS Handshake (or “TLS Handshake”) is the initial process of establishing a secure session between two systems, such as a browser and a web server. It is the first step in negotiating a secure connection using the Transport Layer Security (TLS) security protocol or its predecessor, Secure Sockets Layer (SSL).

Tlsx is a fast and configurable TLS grabber focused on TLS-based data collection and analysis.


Recursive search is a search technique used in programming and artificial intelligence, where a function calls itself to perform the search. In other words, the function performs the search repeatedly until it finds the desired solution or until it reaches a stop condition.

Dsieve takes a single domain or reads an input file and extract unique parent domains, enrich subdomains, filter subdomains by level, or find out which subdomains have the most number of sub-subdomains (or sub-sub-subdomains or sub-sub-sub…). Dsieve supports any format of URL, with or without protocol, port, path, and parameters.


Subdomain takeover is a vulnerability that allows an attacker to take control of a subdomain. This occurs when the same is no longer being used or managed by the original owner but is still pointing to a third-party service such as a hosting provider or CDN service.

Nuclei is used to send requests across targets based on a template, leading to zero false positives and providing fast scanning on a large number of hosts. Nuclei offer to scan for a variety of protocols, including TCP, DNS, HTTP, SSL, File, Whois, Websocket, Headless, etc. With powerful and flexible templating, Nuclei can be used to model all kinds of security checks.

Dnstake uses RetryableDNS client library to send DNS queries. Initiates engagement using Google & Cloudflare DNS as the resolver, then check and fingerprints the nameservers of the target host – if there is one, it will resolve the target host again with its nameserver IPs as resolver, if it gets a weird DNS status response (other than NOERROR/NXDOMAIN), then it’s vulnerable to being taken over.


  • DNS Zone Transfer (dig)

DNS Zone Transfer is a feature of the Domain Name System (DNS) protocol that allows replication of DNS record information from a primary server to one or more secondary servers. It is an important process for ensuring the availability and consistency of DNS record information throughout the network.

Dig (domain information groper) is a flexible tool for interrogating DNS name servers. It performs DNS lookups and displays the answers that are returned from the name server(s) that were queried. Most DNS administrators use dig to troubleshoot DNS problems because of its flexibility, ease of use, and clarity of output.


Cloud checkers is a term that refers to tools that check the security configuration of a cloud, usually to identify vulnerabilities or configuration errors that could affect the security of data stored in the cloud. These tools can scan the security configuration of a cloud such as Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP) and provide detailed reports on any identified security issues.

S3Scanner is a tool to find open S3 buckets and dump their contents.

Cloud_enum is a multi-cloud OSINT tool. Enumerate public resources in AWS, Azure, and Google Cloud.


HOSTS:

Host in computing is any machine or computer connected to a network, that can deliver information, resources, services, and applications to users. Every host on the Internet must have an IP address. Through the ping or WHOIS command, we can obtain more information about the IP address of a certain host.

We will talk about:

CDN checker is a tool that helps you check whether a given website is using a content delivery network (CDN) and, if so, which CDN is being used. This is useful because content delivery networks are an important part of a company’s or website’s infrastructure, helping to ensure website availability, scalability, and security. This usually works by checking the website’s HTTP header and examining the response to HTTP requests. From there, they can determine if the site is being delivered via a CDN and, if so, which CDN is being used. Some CDN checkers also provide additional information, such as the site’s loading speed and the geographic location of the CDN servers.

Ipcdn checks which CDN providers an IP list belongs to. This tool is based on the CIDR ranges collected by ProjectDiscovery in their cdncheck project. This is a cli implementation for easy use and it also contains a GitHub action to download every day the JSON containing CIDRs.


A Web Application Firewall (WAF) is an information security tool that protects web applications against cyber threats such as denial of service (DoS) attacks, SQL injection, cross-site scripting (XSS), and many other attacks. It works by monitoring and filtering a web application’s incoming and outgoing traffic, blocking or redirecting malicious traffic before it reaches the application.

Wafw00f is a Python tool to help you fingerprint and identify Web Application Firewall (WAF) products. Sends a normal HTTP request and analyses the response; this identifies several WAF solutions. If that is not successful, it sends several (potentially malicious) HTTP requests and uses simple logic to deduce which WAF it is. If that is also not successful, it analyses the responses previously returned and uses another simple algorithm to guess if a WAF or security solution is actively responding to the attacks.


  • Port Scanner (Active with nmap and passive with smap)

A Port Scanner is a tool that is used to check whether certain ports on a remote host (such as a server or computer) are open or closed. Port Scanners work by sending data packets to specified ports on a remote host and analyzing the response. Based on the response, the port scanner can determine whether the port is open or closed, and in some cases, even determine what type of service or application is running on the port. Smap is a port scanner built with shodan.io’s free API. It takes the same command line arguments as Nmap and produces the same output which makes it a drop-in replacement for Nmap.

Nmap (Network Mapper) is a popular open-source network scanning tool used by network administrators and security professionals to discover hosts and services on a computer network and to perform security auditing and analysis. Nmap can be used to scan a range of IP addresses, hosts, and networks, and it can detect various types of services running on the target systems, including standard services like HTTP, FTP, and SMTP, as well as proprietary and less common services.

Smap passive Nmap-like scanner built with shodan.io.


Port Services Vulnerability Checks are security checks that aim to identify possible vulnerabilities in services running on specific ports on a remote host. These checks are carried out to identify possible security gaps that can be exploited by malicious intruders.

searchsploit is a command line search tool that also allows you to take a copy of Exploit Database (an archive of exploits for public security) with you, everywhere you go. SearchSploit gives you the power to perform detailed off-line searches through your locally checked-out copy of the repository. This capability is particularly useful for security assessments on segregated or air-gapped networks without Internet access.

The ExploitDB is a very useful resource for identifying possible weaknesses in your network and for staying up to date on current attacks occurring in other networks.


Brute force is an attempted method of trying all possible input combinations in a given series until the correct solution is found.

Brutespray takes Nmap GNMAP/XML output, newline separated JSON, Nexpose XML Export output, or Nessus .nessus exports and automatically brute-forces services with default credentials using Medusa. BruteSpray finds non-standard ports, make sure to use -sV with Nmap.


WEB:

When thinking about a corporate environment in which a company can have several web servers, automation to acquire possible vulnerability patterns becomes something extremely useful for engagements. This session will handle tools that automate the reconnaissance in the web context. The intelligence behind the data that we achieve is essential because as we said earlier, we need to get as many pieces as possible for our chain puzzles.

We will talk about:

“Web Prober” is a system used to test the availability and performance of servers or websites. This tools usually send requests to a specific URL address and measure response time, download speed, and other parameters related to website performance. The purpose is to identify issues that may affect user experience or site performance. Some web prober tools can also provide information about underlying infrastructure such as firewalls and routers.

Httpx is a fast and multi-purpose HTTP toolkit that allows running multiple probes using the retryablehttp library. It is designed to maintain result reliability with an increased number of threads.

Unimap scans only once by IP address and reduces scan times with Nmap for large amounts of data. Unimap is an abbreviation of “Unique Nmap Scan“. The tool can run on Linux, OSX, Windows, or Android (Termux) without problems.


“Web screenshotting” is the process of capturing an image or “screenshot” of a web page. This is done to capture a visual representation of the page’s content at a specific point in time. Screenshots of web pages are often used to document a page’s design, to record changes to a page over time, or to share information about a page with others.

Webscreenshot simple script to screenshot a list of websites, based on the UrlPhantomJS script.
Gowitness is a website screenshot utility written in Golang, that uses Chrome Headless to generate screenshots of web interfaces using the command line, with a handy report viewer to process results. Both Linux and macOS are supported, with Windows support mostly working.


A “web vulnerability scan” is a process of systematically checking a website or web application for security weaknesses, or “vulnerabilities.” The goal of a web vulnerability scan is to identify potential security threats to the website or web application and its users and to provide recommendations for fixing these issues. A template is a pre-designed layout or structure that can be used as a starting point for creating a specific type of document, webpage, or design.

Nuclei is explained in the “subdomains enumeration”, item “Subdomains takeover (nuclei and dnstake)”


A content management system (CMS) manages the creation and modification of digital content. It typically supports multiple users in a collaborative environment. Some notable examples are WordPress, Joomla, Drupal, etc.

CMSeeK CMS Detection and Exploitation suite – Scan WordPress, Joomla, Drupal and over 180 other CMSs.


URL extraction is the process of extracting Uniform Resource Locators (URLs) or web addresses from a text or a web page. The extracted URLs can be used for various purposes, such as building a list of links for web scraping or data mining, analyzing website links and their structure, or creating a sitemap for a website.

Waybackurls fetches all the URLs that the Wayback Machine knows about for a domain.

Gau fetches known URLs from AlienVault’s Open Threat Exchange, the Wayback Machine, and Common Crawl.

Gospider is a web spider written in Go.

JSA (Javascript security analysis) is a program for javascript analysis during web application security assessment.


URL pattern search and filtering refer to the process of identifying and selecting specific URLs that match certain criteria or patterns. This process is often used to identify specific pages or sections of a website, to find pages with specific types of content, or to filter out URLs that are not relevant to a particular task or research project.

Urless is a tool used to declutter a list of URLs.

Gf is a wrapper around grep, to help you search for patterns using JSON files to define them.

Gf-patterns is a repository of patterns to use with GF to find things like XSS, SSRF, RCE, LFI, SQLI, SSTI, IDOR, URL Redirection, and others.


A favicon is a small, square-shaped icon that is displayed in the browser tab or address bar of a website. The favicon is used to visually represent a website and to make it easier for users to identify and find the website they are looking for in their browser tabs.

Fav-up lookups for real IP starting from the favicon icon and using Shodan.


Javascript analysis refers to the process of examining and analyzing the behavior of JavaScript code in a web page or web application. Javascript is a popular scripting language that is used to add interactivity and dynamic content to websites.

Subjs fetches javascript files from a list of URLs or subdomains. Analyzing javascript files can help you find undocumented endpoints, secrets, and more.

JSA was explained in “Url extraction” section.

XnLinkFinder is a python tool used to discover endpoints (and potential parameters) for a given target and Getjswords extract words that are located in javascript sources.


In a web context, fuzzing refers to the process of sending specially crafted input to a web application to test its resilience to unexpected or invalid input. The goal of fuzzing is to identify potential security vulnerabilities or other weaknesses in a web application that could be exploited by attackers.

Ffuf is a web fuzzer written in Go.


  • Passwords dictionary creation (pydictor)

Password dictionary creation refers to the process of creating a list of commonly used passwords or password combinations that can be used to try and guess the password of an user or system. This list, known as a password dictionary or wordlist, can be used by attackers to perform brute force attacks, which involve trying every password in the dictionary until the correct password is found.

Pydictor is a powerful and useful hacker dictionary builder for brute-force attacks.


VULNERABILITY CHECKS:

We’ve reached the icing on the cake. Here we will cover tools that will be used in the perimeter which we enumerated. These tools aim to speed up the discovery process of known vulnerabilities.

We will talk about:

  • Cross-Site Scripting (dalfox)

XSS, or Cross-Site Scripting, is a type of security vulnerability that allows an attacker to inject malicious code into a web page viewed by other users. This can occur when an application takes user-supplied input, such as a search query or comment, and displays it on a web page without proper validation or sanitization.

Dalfox is a powerful open-source XSS scanning tool and parameter analyzer and utility that fast the process of detecting and verifying XSS flaws. It comes with a powerful testing engine and many niche features for the cool hacker!


Open redirect is a type of security vulnerability that occurs when a web application or website allows a user to be redirected to an arbitrary URL, without properly validating or sanitizing the destination URL.

Oralyzer is a simple python script that probes for Open Redirection vulnerabilities in a website. It does that by fuzzing the URL that is provided in the input.


  • Server Side Request Forgery (headers interactsh and param values with ffuf)

Server-Side Request Forgery (SSRF) is a type of vulnerability that allows an attacker to manipulate a web application into making requests to an arbitrary URL, including URLs within the same network as the web server. This can lead to the exposure of sensitive information, such as internal network topology and the contents of internal resources, as well as the potential for further attacks, such as port scanning and exploitation of vulnerabilities in other systems.

Headers Interactsh is a tool designed to detect vulnerabilities that cause external interactions and param values with Ffuf explained before in the “web segment” and “fuzzing” item.


  • Carriage Return Line Feed (crlfuzz)

Carriage Return Line Feed (CRLF) is a sequence of two characters used in computer systems to represent the end of a line of text. It consists of a Carriage Return (CR) character, represented as ASCII code 13, followed by a Line Feed (LF) character, represented as ASCII code 10. The CRLF sequence is used to indicate the end of a line of text in various file formats, such as plain text files and HTML documents, and is also used as a line terminator in some network protocols. In many computer systems, the CRLF sequence is used to indicate a line break in a text file, where the CR character moves the cursor to the beginning of the line and the LF character moves the cursor to the next line. This allows text files to be displayed properly on different platforms, as different systems may have different line-ending conventions. It is important to properly handle CRLF sequences in text processing, as they can cause unexpected behavior if not treated correctly. For example, if a text file containing CRLF sequences is not properly handled when being transferred between systems, the line breaks in the file may be lost or misinterpreted, leading to incorrect display or processing of the text.

crlfuzz is a fast tool to scan CRLF vulnerabilities written in Go.


  • Cross Oring Resource Sharing (Corsy)

Cross-Origin Resource Sharing (CORS) is a mechanism that allows a web page to make requests to a different domain than the one that served the web page. By default, web browsers enforce a same-origin policy that prevents web pages from making requests to a different domain for security reasons.

Corsy is a lightweight program that scans for all known misconfigurations in CORS implementations.


  • Local File Include (ffuf)

Local File Include (LFI) is a type of vulnerability in web applications that allows an attacker to include files from the local file system of the server hosting the web application. This vulnerability occurs when a web application takes user-supplied input and uses it to dynamically include files in the response to the user’s request. If an attacker can manipulate the user-supplied input to include a file of their choosing, they may be able to execute malicious code on the server. LFI attacks can be executed in several ways, such as through the manipulation of URL parameters, HTTP headers, or form data. For example, if a web application takes a URL parameter such as “page” and uses it to include a file, an attacker may be able to specify a file that they want to include by constructing a URL such as “example.com/index.php?page=../../etc/passwd”.

Ffuf explained before in the “web” section and “fuzzing” items.


SQL Injection is a type of security vulnerability that allows an attacker to execute arbitrary SQL code on a database management system through a web application. The vulnerability occurs when a web application constructs SQL statements based on user-supplied input and fails to properly validate or escape that input.

SQLMap is an open-source penetration testing tool that automates the process of detecting and exploiting SQL injection flaws and taking over database servers. It comes with a powerful detection engine, many niche features for the ultimate penetration tester, and a broad range of switches including database fingerprinting, over-data fetching from the database, accessing the underlying file system and executing commands on the operating system via out-of-band connections.


  • Server Side Template Injection (ffuf)

Server-Side Template Injection (SSTI) is a type of vulnerability that occurs when user-supplied input is not properly validated or escaped before being included in a dynamic template, leading to the injection and execution of arbitrary code. SSTI can result in serious security consequences, such as the exposure of sensitive information, the execution of malicious code, and the takeover of the affected system.

Ffuf explained before in the “web” section and “fuzzing” items.


SSL (Secure Sockets Layer) tests are security evaluations of a web application’s SSL/TLS (Transport Layer Security) implementation. The purpose of these tests is to identify potential vulnerabilities in the SSL/TLS configuration that attackers could exploit.
Some common types of SSL tests include: SSL Protocol Testing: This type of test checks for the use of secure SSL/TLS protocols and ensures that the web application is using up-to-date protocols that are not vulnerable to known attacks.  Cipher Suite Testing: This type of test checks for the use of secure cipher suites that use strong encryption algorithms to protect sensitive information.  Certificate Validation Testing: This type of test checks the web application’s SSL certificate to ensure that it is valid, properly configured, and trusted by web browsers.  SSL Configuration Testing: This type of test checks the web application’s SSL/TLS configuration for any misconfigurations or insecure settings that could be exploited by attackers.

Testssl is a free command line tool that checks a server’s service on any port for the support of TLS/SSL ciphers, protocols as well as some cryptographic flaws.


Broken links vulnerability refers to a security weakness in a web application where links or URLs within the application no longer lead to the intended destinations. This can happen for a variety of reasons, such as the removal or renaming of pages, the use of incorrect or outdated URLs, or network errors.

Gospider is a fast web spider written in Go, already introduced in the section “Subdomain Enumeration”, item “JS files & Source Code Scraping


  • Prototype Pollution (ppfuzz)

Prototype pollution is a type of vulnerability in JavaScript applications where an attacker can inject properties into an object’s prototype, leading to unintended behavior and potentially compromising the security of the application. This vulnerability occurs when an application does not properly validate user inputs, allowing an attacker to inject malicious data into an object’s prototype.

Ppfuzz is a fast tool to scan client-side prototype pollution vulnerabilities written in Rust.


Web Cache Poisoning: This occurs when an attacker injects malicious content into the cache, causing the cache to serve the attacker’s content instead of the intended content. This can be used to spread malware, phishing scams, or deface websites. Cache Side-Channel Attacks: These attacks use information obtained from the cache behavior to leak sensitive information, such as encryption keys or passwords.
Weak Authentication and Authorization: If authentication and authorization mechanisms are not properly configured, attackers may be able to bypass them and gain access to sensitive information stored in the cache.
Cross-Site Scripting (XSS): Cross-Site Scripting vulnerabilities can be exploited to inject malicious scripts into web pages that are saved in the cache, compromising the security of the cache and potentially exposing sensitive information to attackers.

Web-Cache-Vulnerability-Scanner is a tool that supports many different web cache poisoning techniques, includes a crawler to identify further URLs to test, and can adapt to a specific web cache for more efficient testing. It is highly customizable and can be easily integrated into existing CI/CD pipelines


Thanks for reading, knowledge is free and it’s up to you to conquer the world. Together we make a difference in the world. With this beautiful message, we conclude this part. Recalling that in the next episode of our series, we will reveal a big contribution to one of the best open-source reconnaissance tools in the world.

Logo da Hakai.