TL;DR

EVA is a chat app that prevents voice phishing and AI-assisted phishing attacks within organizations by giving employees the ability to validate each other via MFA, in seconds. It also assists in secure incident response and securing business processes. This will stop your next red team engagement or ransomware attack in its tracks. EVA takes just a few minutes to set up; you can try it for free here: https://eva.foxio.io/

EVA was created by FoxIO, a cyber innovations company responsible for standards such as JA4+ Network Fingerprinting, which are used by global governments and most of the Fortune 500.

The (AI-Assisted) Voice Phishing Problem

Voice phishing, or vishing, as it is sometimes called, is the practice where a malicious actor calls or texts an employee’s phone pretending to be another employee. The actor often creates a sense of urgency with the employee, requesting help, access, or anything else that can help the threat actor achieve their goals. In these situations, employees are usually skeptical of the call, but because they lack any means of validating that the other person is who they say they are, they tend to follow through with the request.

For some recent examples, see: Finance worker pays out $25 million after video call with deepfake ‘CFO’

Cybercriminals Impersonate Chief Exec’s Voice with AI Software Losing $243,000

MGM Loses $100 Million After Attacker Calls Helpdesk Pretending to be Employee

And most of us have experienced receiving texts from the “CEO” asking us to urgently buy gift cards. The solution to phishing is not user education, it’s in giving users the tools to easily verify everything.

The Employee Verification App (EVA)

EVA is an app that can be added to your company’s existing communication platform, (e.g., Slack, Teams), and interfaces with your existing multi-factor authentication (MFA) provider (e.g., Okta, Google). The app enables employees to validate each other via MFA within seconds. The app has been specifically designed to be as simple and easy to use as possible so that anyone would be able to use it within seconds, even if they’ve never used it before.

Example Scenario 1:

A threat actor sends Alice a phishing email.

Alice then receives a phone call. The person on the line says their name is John and that they’re from the company’s cyber security department. John says they detected malware on Alice’s workstation that was installed after she clicked on a phishing email. John says he needs access to Alice’s machine immediately to determine the extent of the infection.

Alice tells John, “Company policy is that I verify you first.” Alice types “/eva-verify @John” into the company’s Slack workspace. This sends John a message asking if he is talking with Alice right now with two buttons, Approve or Deny.

If John clicks Approve, then he is redirected to the company’s MFA login to prove that he is, indeed, John and that his Slack account has not been compromised. This is because it is easy to steal a Slack session token, but difficult to steal a MFA device. If John successfully logs in, then EVA messages both parties to let them know that John has verified his credentials.

If John clicks Deny or does not respond before the timeout window closes, both parties are notified and Alice is told by EVA to terminate the call immediately. This empowers Alice to hang up on the person without concern of reprisal because the company has told her to do so.

Example Scenario 2:

Charles is told to wire $3M to an account. Company policy is that another person must approve of a transfer this large.

Normally Charles would walk over to Deanna’s desk to get the approval in person but Deanna is working from home today. So Charles gets the approval from Deanna via Slack and to ensure that it is actually her, he verifies her by running “/eva-verify @Deanna” and configures the message with the identification number of the transaction. Deanna approves and verifies her credentials.

This second-person verification along with the transaction ID is then added to the EVA logs for automated auditing of wire transfer approvals.

Example Scenario 3:

JA4+ detects a reverse SSH shell from Chris’s system into production.

A security analyst sends a Slack message to Chris asking if he opened this connection. Chris says he did because he was tired of logging in through the Bastion server. He says he won’t do it again. To ensure that Chris is the person on the other end of this chat, the security analyst runs “/eva-verify @chris” and sets the message to “Did you set up a reverse SSH shell into prod?” Chris clicks “Approve” but is unable to authenticate via MFA because, plot twist, it’s not Chris. The threat actor has access to Chris’s system, but does not have access to Chris’s MFA device and therefore cannot complete the EVA verification.

The security analyst is notified of the timeout failure and spins up a security incident.

Configuration and Support

After adding EVA to your communication platform, run “/eva-configure” to set it up. All default messages and timeout windows are configurable by administrators, but are also one-time configurable by the user at the time of sending.

Our documentation explains how to generate an MFA provider API token for EVA to use that has the minimum necessary permissions to request that someone authenticate, as well as to know if they authenticated.

EVA abides by the principle of least privilege. As such, EVA has the minimum amount of permissions necessary for its purpose with no other permissions. EVA cannot read channels, cannot read chats, and cannot take actions on behalf of users.

EVA logs can be downloaded from the app manually or automatically.

We have built the first version of EVA for Slack, and the app currently supports Okta as an MFA provider. We are actively adding support for more MFA providers and are building EVA for Microsoft Teams. Please let us know what communication platform/MFA provider combination or additional features you would like to see by emailing us at info@foxio.io. We will prioritize the top requests.

EVA is patent pending.

Q&A (this will be updated as the questions come in)

Q: Won’t this create MFA fatigue?

A: Users only get to the MFA part if someone is trying to verify them and they are that actual person. This should be extremely rare as each user rarely calls another user asking for something critical.

In the case where a scammer is, for example, requesting gift cards from multiple users pretending to be the same person, that person may receive a lot of EVA verification requests, but since it’s not them, they can simply click Deny or ignore the requests triggering a timeout. In either case, EVA will alert the requesting user that it’s not them. No interaction with MFA is needed unless that person is actually requesting a bunch of gift cards from different people, which shouldn’t happen.

In short, this will not cause MFA fatigue. You could think of EVA as a fire extinguisher, use of it by any individual should be extremely rare but you’ll be glad you had it when you needed it.

Q: This is a great idea! Why don’t we just build it ourselves?

A: This is the classic Buy vs. Build argument. We’ve designed this to be extremely inexpensive as our goal is to end employee phishing. It should be vastly cheaper and faster to implement than building. Another pitfall of building is if the person or team who built it leaves, then the project can become abandoned. We’ll always be here to help.

If there’s any features you want that are missing, just ask, we’ll get them in so you can hit the ground running immediately.

Conclusion

The Employee Verification App (EVA) empowers every employee with the ability to validate that any other employee is who they say they are, in seconds. EVA solves employee AI and voice phishing and assists in incident response and company policy enforcement. The solution to phishing is not user education, it’s in giving users the tools they need to stop the threat.

EVA is free to use for 90 days. After that, use promo code “EARLY” at checkout to lock in EVA at $0.25/user/month for life. This promotional price will never increase for organizations that sign up in 2024.

Try EVA today; setup takes only minutes: https://eva.foxio.io/

TL;DR

This is an investigation of Surfshark and NordVPN using JA4TCP Fingerprinting.

We found that both Surfshark and NordVPN route certain ports through TCP proxies such as port 5060, which is only used for unencrypted phone calls. No other VPN providers proxy traffic in this way. The reason for the proxying is unknown. Additionally, we found that NordVPN’s proxy is misconfigured, causing increased latency and bandwidth usage.

Resources:
JA4+ Network Fingerprinting: https://github.com/FoxIO-LLC/ja4
JA4+ Blog: https://blog.foxio.io/ja4%2B-network-fingerprinting
JA4TCP Blog: https://blog.foxio.io/ja4t-tcp-fingerprinting
JA4TScan: https://github.com/FoxIO-LLC/ja4tscan
NMap: https://nmap.org/
gait: https://github.com/sandialabs/gait

Quick Refresher on JA4+ Fingerprinting

JA4+ is a suite of network fingerprinting methods that are being implemented across the industry and consist of the following methods, with more being added on a regular basis:

For this investigation, we are primarily utilizing JA4TCP (JA4T). You can read up on how JA4T works in this blog post. In short, it’s a collection of artifacts from the TCP SYN and SYN-ACK packets which make up the TCP three-way handshake. These fingerprints allow us to fingerprint client and server operating systems, devices, certain applications, hosting/provider characteristics, if a connection is going through a tunnel, VPN or proxy, and enable us to troubleshoot network issues.

For this blog, we will also be focusing on the Maximum Segment Size (MSS) part of the JA4T/S fingerprint. The MSS is the largest data payload size that a source will accept per packet, and is dependent on the overhead in the network connection. For example, the most common Maximum Segment Size (MSS) initially set is 1460, based on an ethernet Maximum Transmission Unit (MTU) of 1500. Observing an MSS of 1380 indicates that there is overhead on the network path, such as a tunnel or VPN, requiring a reduced MSS to account for it. Unique network conditions produce different amounts of overhead:

Setting the MSS option to be larger than the actual available size will result in poor network performance, latency, and fragmentation. MSS Clamping is a common method used to avoid this issue.

Surfshark & NordVPN Route Certain Ports Through Proxies?

Nord Security and Surfshark merged in 2022, and, together, they are the world’s largest VPN provider. While testing JA4T through NordVPN and listening on the server side, I noticed something odd in my network logs. Connections over port 443 from NordVPN had a JA4T of 65535_2–4–8–1–3_1460_9 (Unix), no matter what the client was. On the client side, I would see a normal-looking JA4T fingerprint where the MSS was 1380 (to account for the VPN overhead). So why does the server see a MSS of 1460? I tried connecting to the server using a different port, 8443, and only then did the server see the correct JA4T fingerprint of my client.

What is going on here? I did some further research, and here is what I found:

Observed network traffic on the server side:

Why does the client TCP fingerprint change from Windows to Unix when connecting through Nord and Surfshark, but only over ports 80 and 443?

Observed network traffic on the client side:

Why is the server’s TCP response fingerprint different when connecting through Nord and Surfshark, but only over ports 80 and 443?

When looking at latency measurements and hop counts with JA4L, I noticed that NordVPN and Surfshark would initiate the TCP three-way handshake from their exit node and that the hop counts were different, but only over ports 80 and 443. All of these observations led me to believe that NordVPN and Surfshark are rerouting certain ports to proxies that intercept the TCP connection.

To confirm this theory, I ran a SYN scan of all ports against an IP address for which no server exists. The scan should return no results since the destination server does not exist. However, both NordVPN and Surfshark responded on the following ports, which confirms that they are routing these ports through TCP proxies:

If you have NordVPN or Surfshark, you can test this yourself by running:

nmap -v -Pn -n 203.0.113.100 (or use any other IP that is not in use)

Telegram is a service that runs over several ports, including ports 80 and 443. Telegram does not use TLS but instead uses MTProto, a proprietary protocol that encrypts messages over several protocols, including HTTP. I connected directly to Telegram and then connected through NordVPN to see if there was a difference:

When connecting directly, the client sends an HTTP POST to the server. The server responds with an ACK, then a FIN ACK to close the connection. When connecting over NordVPN, the client sends an HTTP POST to the server, the server responds with an HTTP 200 OK, and then both the client and the server send FIN ACKs to close the connection.

This discrepancy is interesting. I don’t know what the reason is, but it is repeatable.

I tested this with other VPN providers, including Private Internet Access and Proton VPN, but did not observe the same behavior. The discrepancy appears to be unique to Surfshark and NordVPN. So, why are they running only these ports through TCP proxies?

…

Let’s assume that it’s for performance reasons, perhaps as a load-balancing strategy.

NordVPN’s TCP proxy has a JA4T of 65535_2–4–8–1–3_1460_9. Notice the MSS of 1460.

The actual client’s JA4T is 64860_2–1–3–1–1–4_1380_8. Notice the MSS of 1380 due to the VPN overhead.

Advertising an MSS higher than what the client can accept should result in worse performance, not better. I tested this theory by connecting to a website with NordVPN over port 443 (through their proxy) and over port 8443 (bypassing the proxy). The results were as expected:

When bypassing Nord’s proxy, the server sends data with a packet size of 1380: the maximum segment size of the VPN connection. In this test, 381 packets were sent back and forth to load the webpage completely.

When going through Nord’s proxy, the server sends packets with the requested segment size of 1460. The proxy, unable to forward those packets to the client because the client can only handle 1380, sends hundreds of TCP SACKs to the server with updated Window Sizes, essentially asking the server to resend the previous packets, but at smaller sizes. As a result, 962 packets were needed to load the same webpage. This back-and-forth increased latency for the client and doubled the bandwidth between NordVPN and the server, as the server had to send the same data twice.

So, the proxies are not for performance reasons?

Interestingly, Surfshark’s TCP proxy has a JA4T of 65170_2–4–8–1–3_1330_10. Notice the MSS of 1330, which is the correct size given the overhead of the VPN and the proxy, and therefore does not suffer the performance impact that Nord has.

Because of this, I believe that NordVPN is using a similar proxy infrastructure to Surfshark’s — for still unknown reasons — but NordVPN has its proxy misconfigured by not clamping MSS to 1330, like Surfshark. Fixing this will reduce latency for customers and reduce NordVPN’s infrastructure cost.

Both NordVPN and Surfshark tout the ability to block viruses and ads, so perhaps the proxies are related to that. But then why proxy port 5060, which is only used for unencrypted phone calls? I contacted NordVPN to find out — there has to be a good reason:

It appears that at least the support team is unaware of the proxy servers.

Other potential reasons for the proxies include being a transparent caching web proxy. DNS RPZ may account for port 53. CALEA could explain port 5060, but so could SRTP. Any assumptions would be pure speculation without more data and therefore this blog makes none.

I hate to end this blog with more questions than answers, so I will post an update if I find out more. Until then, this was an interesting hunt that, at the very least, found performance misconfigurations at NordVPN, all triggered by an unexpected JA4T fingerprint in my network logs.

UPDATE 5/15/2024

NordVPN has responded.

That patent is https://patents.google.com/patent/US11632267B2/en which looks to be a method of offloading TCP overhead through a VPN connection. The idea being that if a connection is lossy (think mobile phone with a poor signal), the TCP retransmissions are only on one side of the connection rather than all the way back and forth.

This makes sense and the patent would explain why we’re not seeing this from other VPN vendors. Though, I’m still confused about the amount of TCP SACKS seen from NordVPN on the server side as well as the discrepancy in MSS between Nord and Surfshark. That still may be worth a look on their end.

In any case, I hope this blog demonstrates the relevance and value of analyzing all of the low-level aspects of connections. Thanks to NordVPN for the explanation!

TL;DR

JA4T/S/Scan are the latest additions to the JA4+ family of network fingerprinting tools.

These tools add the ability to fingerprint client and server operating systems, devices, particular applications, hosting characteristics, and even if a connection is going through a tunnel, VPN, or proxy. If built into a WAF, firewall, or load balancer, it becomes possible to block malicious traffic.

JA4+: https://github.com/FoxIO-LLC/ja4
JA4TScan: https://github.com/FoxIO-LLC/ja4tscan
Original blog on JA4+: https://blog.foxio.io/ja4%2B-network-fingerprinting

This project was inspired by p0f (Michał Zalewski), Hershel+ (Zain Shamsi & Dmitri Loguinov), and gait (Charles Smutz & Brandon Thomas). You guys are amazing!

TCP Refresher

TCP connections start with a TCP 3-way handshake. First, the client sends a TCP packet with the SYN flag to the server with its preferred TCP communication settings. If listening, the server will respond with a SYN-ACK packet and its preferred TCP communication settings. Then, the client will send an ACK, immediately followed by the application request, which could be HTTPS, SSH, etc. If the server is listening with that application protocol it will begin application communication. The connection is closed with a FIN ACK and the other side agrees with an ACK.

If one side does not respond with an ACK, the other side will retransmit the packet(s) several times at varying intervals before giving up. The number of retransmissions and the time interval between them is dependent on the operating system’s netcode.

The communication settings in the initial SYN and SYN-ACK packets are determined by the netcode used in the operating system. A Window Size, the maximum amount of data to be transmitted before an ACK is needed, is limited to two bytes and is a required field. TCP Options are not required but are used by every modern operating system.

There are many other TCP options going up to Kind 254, though they are mostly used in specialized environments (think SCADA and Mainframes). The total length of the TCP options list must be evenly divisible by 4. That is the reason why the NOP option exists, to pad out the options list length to a divisible byte count.

The Window Scale acts as a multiplier for the Window Size, allowing the actual Window Size to be much larger than 65535. For example, if the Window Size is 64240 and the Window Scale is set to 8, then the actual Window Size is 64240*2⁸, or 16445440.

The Maximum Segment Size (MSS) is the largest data payload size that the source will accept per packet and is dependent on the overhead in the network connection. For example, the most common Maximum Segment Size (MSS) initially set is 1460, based on an ethernet MTU of 1500. Observing an MSS of 1380 would indicate that there is overhead on the network path, such as a tunnel or VPN, requiring a reduced MSS to account for the overhead. Different network conditions produce different amounts of overhead:

Manually setting an MSS option to be higher than the actual available size will result in poor network performance, latency, and fragmentation.

Previous TCP Fingerprinting Tools

Previous TCP fingerprinting tools, like Nmap (2006) and p0f (2013), are designed to fuzzy match with known operating systems and, as such, produce fingerprints that are not meant to be logged or used as pivot points in analysis but rather to be used to map back to an OS or device list. Their raw fingerprints include variable values like hop count which change from connection to connection, making it difficult to share raw fingerprint strings across the industry as each observer would see something slightly different depending on where they were observing. These tools were fantastic for their use case and are still very valuable today, many years after they came out.

JA4T: TCP Fingerprinting

JA4T is specifically designed to be logged alongside every session, highlighting unusual network conditions, and to be used as a pivot point in analysis, troubleshooting, threat hunting, and traffic shaping. It is human- and machine-readable, shareable, and can augment threat intel data. While still able to identify the OS/Device, JA4T also helps to identify intermediary proxies, VPNs, load balancers, tunneling, etc. JA4T can be deployed on any network device including netflow sensors, firewalls, WAFs, load balancers, and proxies.

Each operating system has different combinations of window size, options, and window scale. For example, Microsoft Windows does not utilize TCP Option 8 (timestamp), whereas all Unix-based operating systems do. iOS ends with a TCP Option 0 (End of list) whereas other operating systems do not. It’s interesting that iOS added another Option 0 to make their options list evenly divisible by 4 rather than removing an NOP (Option 1). This goes back to decisions the programmers made when building the netcode.

Changes in the MSS (part c), can help identify network conditions for the device. For example, each mobile carrier sets a different MSS for the overhead in their cell network:

Which means we can identify the carrier that devices are on:

When a device is connected through a VPN, the MSS, and occasionally Window Size, are changed based on the overhead of the VPN and encryption ciphers used. When a device is connected through a Proxy, the TCP fingerprint of the proxy is seen on the server side, not the client. For example, the complete change in fingerprint when an iPhone connects through iCloud Relay:

JA4TS: TCP Server Response Fingerprint

While JA4T is based on the client’s TCP SYN packet, JA4TS is based on the SYN-ACK response.

TCP servers may respond to different client TCP SYN options differently. This means that any given server may produce multiple JA4TS fingerprints depending on the clients connecting to it. For example, if a client does not include TCP Option 4 (SACK), the server is not likely to include Option 4 in its SYN-ACK response. Thus making JA4TS a TCP Server Response Fingerprint. To get an accurate fingerprint of the Server itself, see JA4TScan in the section below.

JA4TScan: Active TCP Server Fingerprinting

JA4TScan is designed to produce a reliable TCP fingerprint of any server. It does this by actively scanning servers with a single SYN packet that includes all common TCP options to produce the most robust TCP SYN-ACK response from the server. It does not respond to the SYN-ACK from the server, but instead listens for retransmissions, counts the delay between each retransmission, and adds those delays to the end of the fingerprint as section e. If an RST packet is observed, it is also added to the fingerprint and prefixed with an “R”.

TCP retransmissions, the number of retransmissions, and the delay between them are unique per operating system, as they are based on the OS netcode and the decisions of the engineers who wrote it. For example, some IoT devices send several retransmissions less than a second apart to attempt to reconnect as quickly as possible, while other devices will wait one second, retransmit, then wait two seconds, retransmit, then wait four seconds, retransmit, etc.

By incorporating the delay between TCP retransmission responses, we can build a very robust TCP fingerprint with only a single SYN packet. JA4TScan will be added to Censys and other tools later this year.

JA4TScan is available as a stand-alone tool here: https://github.com/FoxIO-LLC/ja4tscan

Blocking Internet Scanners with JA4T and GreyNoise

GreyNoise is a service that turns Internet noise into intelligence. As such, they can correlate data, such as fingerprints, across internet scanners. Correlating JA4+ across GreyNoise allows for the grouping of threat actors and tools.

For example, one of the top JA4T fingerprints observed on GreyNoise is 29200_2–4–8–1–3_1424_7. An options list of 2–4–8–1–3 indicates a Unix-based operating system and an MSS of 1424 indicates that these connections have 36 bytes of additional network overhead. This is possibly an unencrypted tunnel or proxy, as 36 bytes is not enough for additional encryption as would be seen in a VPN. GreyNoise is observing hundreds of source IPs with this JA4T fingerprint, however, all are within Tencent IP ranges and listening on port 22, with some listening on port 31401, which is Pi Node Crypto Miner. Given the MSS discrepancy, it’s possible that these source IPs are not actually the true source of the traffic but instead that traffic is being bounced through them.

Pivoting on the JA4T with GreyNoise data, we can see this actor’s scanning priorities are primarily focused on SSH and alternative SSH ports:

As this JA4T is unusual, it would be safe to block it when the destination port matches 22. However, there’s a potential for false positives in production applications over standard ports like 80 and 443. To block these, we can combine the JA4T with other JA4 fingerprints.

Their second priority is web server identification. Comparing the top JA4H (HTTP Fingerprint) with this JA4T shows that they use a few different bots. Some are simple, while others try to look like a browser with their primary Accept-Language set to “zhcn”, which is Chinese-PRC. In the case of ge10nn04zhcn, they’re using HTTP 1.0 as an attempt to connect to older devices:

Comparing the top JA4 (TLS Fingerprint) with this JA4T reveals that the actor uses a few variations of client hellos when scanning. Their primary scanner is a custom catch-all scanner that supports TLS 1.3, but, also, 69 ciphers — nice. Their other scanners support TLS 1.2 and 1.1, indicating that they’re looking to connect to both new and old systems with a variety of TLS client hellos. The one JA4 of t11d6911h9 is particularly odd because it’s TLS 1.1 with an ALPN extension, but ALPN didn’t exist in the days of TLS 1.1:

Combining this actor’s unusual JA4T with these unusual JA4H or JA4 fingerprints would make for great blocking or detection rules as it is the combination of JA4+ fingerprints that can facilitate creating detection and blocking rules with no false positives.

I want to thank GreyNoise, F5, and Arkime for their support of JA4+ in these investigations!

Conclusion

JA4T, JA4TS, and JA4TScan are the latest additions to the JA4+ suite of network fingerprints. These tools add the ability to fingerprint client and server operating systems, devices, particular applications, hosting/provider characteristics, detect if a connection is going through a tunnel, VPN or proxy, and help troubleshoot network issues. If built into a WAF, firewall, or load balancer, it becomes possible to block unwanted traffic based on fingerprints rather than a list of constantly-changing IP addresses.

You can find JA4+ on our GitHub (https://github.com/FoxIO-LLC/ja4) and in many cyber security products.

JA4TScan is available as a stand alone tool here: https://github.com/FoxIO-LLC/ja4tscan

F5 iRules for JA4+ along with an example of blocking traffic with JA4T are available here: https://github.com/f5devcentral/f5-ja4

JA4+ licensing details are here.

JA4T/S/Scan was created at FoxIO by John Althouse.

With valuable feedback from:
Charles Smutz
Andy Wick
Joe Martin
Andrew Morris
Reid Huyssen
Tony Maszeroski
Greg Lesnewich
Jo Johnson
Timothy Noel
Gary Lipsky

And engineers working at AWS, F5, GreyNoise, Hunt, Censys, and others.

In this blog I go over the new JA4+ network fingerprinting methods and examples of what they can detect.

JA4+ provides a suite of modular network fingerprints that are easy to use and easy to share, replacing the JA3 TLS fingerprinting standard from 2017. These methods are both human and machine readable to facilitate more effective threat-hunting and analysis. The use-cases for these fingerprints include scanning for threat actors, malware detection, session hijacking prevention, compliance automation, location tracking, DDoS detection, grouping of threat actors, reverse shell detection, and many more.

JA4+ Fingerprints:

More fingerprints are in development and will be added to the JA4+ family as they are released.

UPDATE: Click here for the JA4T/S/Scan — TCP Fingerprinting blog.

Examples:

JA4+ is available here: https://github.com/FoxIO-LLC/ja4

What is JA4+

JA4+ is a set of simple yet powerful network fingerprints for multiple protocols that are both human and machine readable, facilitating improved threat-hunting and security analysis. If you are unfamiliar with network fingerprinting, I encourage you to read my blogs releasing JA3 here, JARM here, and this excellent blog by Fastly on the State of TLS Fingerprinting which outlines the history of the aforementioned along with their problems. JA4+ brings dedicated support, keeping the methods up-to-date as the industry changes. Additionally, and by popular demand, an official JA4+ database of fingerprints, associated applications and recommended detection logic is in the process of being built.

All JA4+ fingerprints have an a_b_c format, delimiting the different sections that make up the fingerprint. This allows for hunting and detection utilizing just ab or ac or c only. If one wanted to just do analysis on incoming cookies into their app, they would look at JA4H_c only. This new locality-preserving format facilitates deeper and richer analysis while remaining simple, easy to use, and allowing for extensibility.

In this blog we are releasing JA4/S/H/L/X/SSH, or JA4+ for short. More fingerprints are in development and will be added to the JA4+ family as they are released.

JA4: TLS Client Fingerprint

TLS is used to encrypt the vast majority of traffic on the internet, from web browsing to streaming, to IoT usage analytics. Even malware uses TLS to hide its malicious communications. At the beginning of a TLS connection, the client sends a TLS Client Hello packet which is sent in the clear, prior to encrypted communication. This packet, generated by the client application, informs the server of what ciphers it supports as well as its preferred method of communication. As such, the TLS Client Hello packet is unique per application or its underlying TLS library.

JA4 looks at this TLS Client Hello packet and builds out an easily understandable and shareable fingerprint. The format is as follows:

JA4 fingerprints the client, no matter if the traffic is over TCP or QUIC. QUIC is the protocol used by the new HTTP/3 standard that encapsulates TLS 1.3 into UDP packets. As QUIC was developed by Google, if an organization heavily utilizes Google products, QUIC could make up half of their network traffic, so this is important to capture.

JA4 also clearly shows the ALPN (Application-Layer Protocol Negotiation). This represents the protocol that the application wants to communicate in after the TLS negotiation is complete. “h2” = HTTP/2, “h1” = HTTP/1.1, “dt” = DNS-over-TLS, etc. A full list of possible ALPNs can be found here. A “00” here denotes the lack of ALPN. Note that the presence of ALPN “h2” does not indicate a browser as many IoT devices communicate over HTTP/2. However, the lack of an ALPN may indicate that the client is not a web browser.

More technical details for implementation and what the raw (unhashed) fingerprint looks like can be found on the github page.

Even though the traffic is encrypted over TLS 1.3, we are still able to gain a huge amount of valuable information about the client application. Remember that most custom applications will have the fingerprint of their underlying TLS libraries. So a program written in Go will likely have a JA4 that matches other Go programs. The same is true for Python, Java, etc., while custom programs like VPN clients, Steam, Slack, and Windows functions will be unique.

JA4 can be extremely valuable in production networks where applications are largely static. If you are running an all Linux infrastructure, then a Windows JA4 fingerprint would be worth looking into. If you’re running only Exchange servers, then a sudden python JA4 fingerprint would be worth looking into. JA4 makes for a great pivot point in analysis when trying to understand network traffic and the a_b_c format allows for deeper analysis.

For example; GreyNoise is an internet listener that identifies internet scanners and is implementing JA4+ into their product. They have an actor who scans the internet with a constantly changing single TLS cipher. This generates a massive amount of completely different JA3 fingerprints but with JA4, only the b part of the JA4 fingerprint changes, parts a and c remain the same. As such, GreyNoise can track the actor by looking at the JA4_ac fingerprint (joining a+c, dropping b).

JA4S: TLS Server/Session Fingerprint

After a client sends its TLS Client Hello packet, the server will respond with its TLS Server Hello packet. This packet, also sent in the clear, is formulated based on the server’s selection of available options in the Client Hello. This includes the one cipher chosen out of the list of available options, and any extensions the server wishes to set.

As such, the Server Hello is unique to both the server application and the Client Hello that was sent to it. A different Client Hello may cause a different Server Hello, and therefore a different JA4S, from the same server. However, the same Client Hello will always produce the same Server Hello from that server application. For example if the client sends JA4=a_b_c and the server responds with JA4S=d_b_e, that server will always respond to a_b_c with d_b_e. But if another application sends a different client hello to that same server, say JA4=x_y_z, the server will respond with a different server hello, JA4S=t_y_v. So it’s a different response to different applications but always the same response to the same application. I go into more detail on this in my JA3S blog post.

JA4S, when combined with JA4, significantly increases detection fidelity. Going from merely identifying underlying libraries of a client to identifying the client or malware family. Beyond application identification, one could look at just JA4S_b to understand what ciphers are being used on any given network to ensure it is meeting compliance requirements. All of this is possible without breaking encryption.

JA4H: HTTP Client Fingerprint

JA4H fingerprints the HTTP client based on each HTTP request. As most traffic is encrypted, JA4H is best utilized on servers, proxies, WAFs, TLS terminating load balancers, and environments where TLS is decrypted. However, JA4H is still valuable even in environments where TLS is not decrypted because a lot of devices and programs, including malware, still communicate over HTTP. The IcedID malware dropper, for example, doesn’t use TLS. These malware programs are very easy to fingerprint.

JA4H_ab are a fingerprint of the application for the given HTTP method used. The lack of an Accept-Language is a clear indication that the application is not human interactive, ergo a bot.

JA4H_c is a fingerprint of the cookie and will be different for each website visited but will be the same for that website or application. For example, every Plex server or Okta server will produce the same JA4H_c.

JA4H_d is a fingerprint of the user and will be different per user. This allows for tracking of a user through a website without logging SPII, thereby keeping the logging system GDPR compliant.

More details on the technical implementation can be found on our github.

On the server side, one could use JA4H_c as a hunting method. As the server is specifying which cookie fields the client should use, all clients should have the same JA4H_c. Discrepancies here merit looking into. One could also track a user with JA4H_d and their client application with JA4H_ab or identify bots with just JA4H_ab.

On the client side (proxy, NDR, zero trust), JA4H combined with JA4 and JA4S allow for extremely high fidelity application and malware detection.

There are a lot of use cases for JA4H, especially when combined with the rest of JA4+. I’ll cover all of them in more detail in a later blog post.

JA4L: Light Distance Locality

JA4L measures the distance between a client and a server by looking at the latency between the first few packets in a connection. We use the first few packets because these are low-level machine generated, so there is nearly zero processing delay in creating and sending these packets. Time is measured in microseconds (µs), 1ms = 1000µs, as microseconds are a standard unit of time measurement in packet captures.

If JA4L is running server side, this will measure the distance of the client from the server and if this is running client side, this will measure the distance of the server from the client. If this is running on a network tap, it will measure the distance of each from the network tap location.

JA4L is split up into 2 measurements, client and server. For TCP, these are determined by looking at the TCP 3-way handshake. UDP, we’re looking at the QUIC handshake.

JA4L-C = {(C-B)/2}_Client TTL
JA4L-S = {(B-A)/2}_Server TTL

In the above example:
JA4L-C = 11_128
JA4L-S = 1759_42

JA4L-C = {(D-C)/2}_Client TTL
JA4L-S = {(B-A)/2}_Server TTL

In the above example:
JA4L-C = 37_128
JA4L-S = 2449_42\

Distance Measurement:
With JA4L we can determine the distance between the client and server using this formula:

D = jc/p

D = Distance
j = JA4L_a
c = Speed of light per µs in fiber (0.128 miles/µs or 0.206km/µs)
p = Propagation delay factor

Typical propagation delay depends on terrain and how many networks are involved.

Poor terrain factor = 2 (around mountains, water)
Good terrain factor = 1.5 (along highway, under sea cables)
SpaceX factor = … needs to be tested

We can use the TTL to calculate the hop count, which can help inform the propagation delay factor. (The table below is a good starting point but more testing needs to be done.)

To calculate the number of hops a connection went through, subtract the TTL from its estimated initial TTL.

Cisco, F5, most networking devices use a TTL of 255
Windows uses a TTL of 128
Mac, Linux, phones, and IoT devices use a TTL of 64

Most routes on the Internet have less than 64 hops. Therefore if the observed TTL, JA4L_b, is <64, the estimated initial TTL is 64. Within 65–128, the estimated initial TTL is 128. And if the TTL is >128 then the estimated initial TTL is 255.

With a JA4L-S of 2449_42, the observed TTL of 42 means the initial TTL was likely 64, a Linux server. 64-42 gives us a hop count of 22.

2449x0.128/1.6=195

We can conclude that this server is within 195 miles of the client. The server may be closer than this, but it is physically impossible for it to be farther away as the speed of light is constant. If there are multiple JA4Ls for the same host, the lowest value should be taken as the most accurate.

In this example, the actual distance was 194 miles.

Utilizing multiple locations, one can passively triangulate the physical location of any client or server down to a city area. More on this in a later blog post…

Additionally, JA4L_b (TTL) passively facilitates the identification of source operating systems which is an excellent data point to have when performing forensic analysis. Also, because JA4L is looking at Layer 3 data, it works on encrypted and unencrypted traffic.

Combining JA4 with JA4H and JA4L on the server side makes it possible for the server application to identify session hijacking or MiTM attacks. If a session cookie (JA4H_d) were to suddenly change locations, operating systems (JA4L), and application (JA4 and JA4H_ab), then it would make sense to revoke the session token, asking the user to log back in with MFA. With this type of logic, special care should be taken to not allowlist particular fingerprints as applications will change over time, but instead to look for dramatic changes. Session hijacking detection with JA4+ is something darksail.ai is working on right now.

JA4X: X509 TLS Certificate Fingerprinting

Credit: W.

JA4X fingerprints the way in which TLS certificates are generated — not the values within the certificate. This can identify applications and settings used to create the certificate which can be extremely useful in threat hunting as threat actors will create different certificates but tend to use the same methods to create said certificates, thereby having the same JA4X fingerprint.

SoftEther VPN was heavily utilized by Chinese APT actors, Flax Typhoon, to compromise Taiwan infrastructure, and Storm-0558 in the hacking of US Government Email accounts. According to Microsoft, it is very difficult to differentiate these connections from legitimate HTTPS traffic. However, because of the programmatic way that SoftEther generates its certificates, the JA4X is unique to SoftEther. If JA4X were to be implemented into a firewall, blocking traffic to SoftEther VPNs would be trivial. And by utilizing a JA4X feed, blocking inbound traffic from SoftEther VPNs would be trivial as well.

Most certificate issuing organizations will use the same underlying program to generate and sign all of their certificates. Using Internet scan data enriched with JA4X from our friends at Hunt.io, we can take a look at Issuer Organization = “Microsoft Corporation” as an example.

You can see that 99.8% of observed certificates have the same JA4X. The next one down is very similar, but the third one looks completely different. Let’s pivot on this anomaly.

Oh look, it’s all Cobalt Strike! Well, that was easy.

Hunting with JA4X on Internet scan data is extremely powerful because rather than looking at the values within a certificate, which, in the case of malware, are usually randomly generated, JA4X looks at how the certificate was generated.

One final example is Sliver C2, which is a newer pentesting framework that is replacing Cobalt Strike in popularity. Like most good pentesting frameworks, Sliver is also heavily utilized by threat actors as it is designed to be difficult to detect.

Sliver has over 400 lines of code dedicated to randomly generating TLS certificates. As such, each certificate is unique and pivoting on a certificate hash will yield no results.

However, each certificate is also generated by the same application and therefore has the same JA4X. Havoc C2 uses most of the Sliver code so it too has the same JA4X, but can be differentiated by looking at the Org Name and Postal Code length. In either case, both are malware and the JA4X is unique on the Internet. Our friends at driftnet.io offer a JA4X feed and, with it, were able to quickly identify all default Sliver C2s listening on the Internet.

These examples show how JA4X can be used to detect and block traffic to SoftEther, Tor, Metasploit, Sliver, Havoc, RAT C2s, etc. Note that TLS certificates are sent in the clear in TLS 1.2, but are encrypted in TLS 1.3 so JA4X is best utilized on Proxy servers, Firewalls, MDR, NDR and Zero Trust applications that have that level of inspection. JA4X, when combined with JA4, JA4S, JA4H and JA4L provides an unparalleled level of visibility and detection capability. When used in internet scanning, JA4X is an excellent tool for pivot analysis and hunting down malicious servers, especially when combined with JARM data.

JA4SSH: SSH Traffic Fingerprinting

JA4SSH fingerprints SSH sessions by looking at SSH packets and providing a small, simple, easy-to-read fingerprint of the session on a configurable rolling basis, every 200 packets by default. With this, we are able to determine what is happening within the SSH connection, even though the traffic is encrypted, and provide an analyst with a simple set of fingerprints for their analysis.

Note that JA4SSH fingerprints the SSH session, not the SSH applications. For SSH application fingerprinting, I recommend you take a look at HASSH, by my good friend Ben Reardon.

To understand how SSH traffic works and how to identify tunnels using traffic analysis, I recommend you read these excellent blogs on the subject by Trisul.org here and here.

In short, SSH packets are padded out to a particular length depending on the cipher algorithm and HMAC used. With chacha20-poly1305, that ends up being 36 bytes. When the client types a character into the ssh terminal, that character is encrypted with the packet padded to 36 bytes and then sent to the server. The server will respond with the same character in a 36 byte packet and that’s when the character is displayed in the terminal window. The client will then send a TCP ACK packet to tell the server that they’re done with the previous transaction. Because of this, a client typing in a terminal will look like this:

Notice that the TCP ACKs are coming from the side doing the SSH requests (client) and the server returns the output of the command at the bottom. The JA4SSH for this looks like: c36s36_c51s80_c69s0. So you see 36/36 and all the ACKs are from the client, the server has sent none, so from these we can easily see that it is an interactive forward SSH session.

In a reverse SSH shell, it’s SSH over SSH so the packets are double padded + HMAC. Here’s what it looks like:

The JA4SSH for this looks like: c76s76_c71s59_c0s70. We can clearly see a common packet length of 76/76, double padded, and all of the ACKs are coming from the server, meaning it is the server side that is doing the typing. It’s important to note that SSH provides encrypted messages, not an encrypted tunnel, so layer 4 packets, the TCP ACK packets, are sent in the clear. It is with these that we are able to determine which side is initiating the requests. By utilizing JA4SSH, it is now trivial to detect reverse SSH shells.

In a SCP file transfer, the TCP Length is maxed out and looks like:

The JA4SH for this looks like: c112s1460_c0s179_c21s0. Notice the maxed out window of s1460, that all SSH packets are coming from the server and all TCP ACK packets are coming from the client. This easily shows that the client requested a file and the server is sending it.

In static environments, like banks, where files are transferred over SFTP between the same systems every day, the JA4SH fingerprints of those connections should remain similar and any major deviation, such as looking like an interactive shell, would be worth alerting on.

JA4SSH makes it easy to detect certain types of SSH activity and delivers fingerprints in a format that is simple to understand. Combining JA4SSH with JA4L can inform the distance of the client/server as well as the operating system of each.

Licensing

JA4: TLS Client Fingerprinting is open-source, BSD 3-Clause, same as JA3. This allows any company or tool currently utilizing JA3 to immediately upgrade to JA4 without delay.

JA4S, JA4L, JA4H, JA4X, JA4SSH, and all future additions, (collectively referred to as JA4+) are licensed under the FoxIO License 1.1. This license is permissive for most use cases, including for academic and internal business purposes, but is not permissive for monetization. If, for example, a company would like to use JA4+ internally to help secure their own company, that is permitted. If, for example, a vendor would like to sell JA4+ fingerprinting as part of their product offering, they would need to request an OEM license from us.

JA4+ can and is being implemented into open source tools, see the License FAQ for details.

This licensing allows us to provide JA4+ to the world in a way that is open and immediately usable, but also provides us with a way to fund continued support, research into new methods, and the development of the upcoming JA4+ Database. We want everyone to have the ability to utilize JA4+ and are happy to work with vendors and open source projects to help make that happen.

Conclusion

JA4+ provides a suite of modular network fingerprints that are easy to use and easy to share. The use-cases for these fingerprints include scanning for threat actors, malware detection, session hijacking prevention, compliance automation, location tracking, DDoS detection, grouping of threat actors, reverse shell detection, and many more. JA4 (TLS Client Fingerprinting), is licensed under BSD 3-Clause, allowing tools running JA3 to immediately upgrade, while JA4+ (JA4S/L/H/X/SSH) is under the FoxIO License, which is permissive for most use cases except monetization, for that the vendor would need to purchase an OEM license which is what funds further research and the development of the JA4 database (coming soon). We plan to release a new JA4 method about once per quarter so stay tuned.

JA4+ is available here: https://github.com/FoxIO-LLC/ja4
For licensing or questions, reach out to us at www.fox-io.com
You can reach me directly on LinkedIn or Twitter/X.

JA4+ was created by:
John Althouse

With feedback from:
Josh Atkins
Jeff Atkinson
Joshua Alexander
W.
Joe Martin
Ben Higgins
Andrew Morris
Chris Ueland
Ben Schofield
Matthias Vallentin
Valeriy Vorotyntsev
Timothy Noel
Gary Lipsky
And engineers working at GreyNoise, Hunt, Google, ExtraHop, F5, Driftnet and others.

TL;DR

LogSlash can reduce the cost of your logging infrastructure and the cost to train AI models on transactional data by 80%.

cwolves is a startup that uses AI to automatically LogSlash and normalize any log type. It is extremely performant and cost effective. They created a Splunk plugin for LogSlash.

LogSlash has also been tested on Vector, Logstash, Python, Kafka, and Cribl. Performance and ease of use are key.

LogSlash code is available here: https://github.com/FoxIO-LLC/LogSlash

LogSlash is patented and licensed under the FoxIO License which is permissive for most use cases, including internal business use. The main use case it’s not permissive for is monetization, that requires a separate OEM license which helps fund further R&D.

Background on LogSlash

We released LogSlash, a method for reducing log volume without reducing analytical value, in this blog post from January. Here’s a quick refresher on how LogSlash works:

LogSlash intelligently reduces similar logs within a defined time window down to a single log line. For example, if an IP communicates to another IP 100 times a minute, instead of logging 100 log lines, LogSlash produces 1 log that says the event happened 100 times within that minute. Analytical value is retained while data volume is drastically reduced.

Over the past few months we’ve worked with several large companies and have found that our initial estimate of 50% volume savings was wrong. These businesses are actually seeing around 80% log volume reduction when using LogSlash. Multiply those savings across trillions of logs and you can see how the cost benefit can be quite enormous, allowing organizations the headroom to send logs to their SIEM that they otherwise couldn’t afford.

LogSlash isn’t just for SIEMs. Training AI models on transactional data is becoming increasingly common and extremely expensive. Running these logs through LogSlash prior to training the AI model on them can save an equal amount of training time and compute. You just want to make sure your model supports the “logslash” field so it knows how to duplicate the occurrences of data while training on LogSlashed logs.

Development

The LogSlash method is designed to be implemented into many existing technologies and languages. The difference between them comes down to performance and ease of configuration, as each log type requires its own configuration. We originally released LogSlash as a set of Vector.dev scripts. They worked but the scripts proved to be extremely complicated and time consuming to create.

Steven Hostetler released LogSlash as a Logstash config. Slash-N-Stash worked well but again, it took a lot of effort to build the config and get it working correctly.

We developed LogSlash on Python and Kafka but the performance was so cost prohibitive that it wasn’t even worth releasing as the cost to run LogSlash needs to be infinitesimal compared to the savings.

We developed LogSlash on Cribl and this was a bit easier to develop compared to Vector and Logstash. However, it still requires you to build a different config for each log type and you need a Cribl license which is volume based. We’ve yet to test it at scale so performance impact is still unknown.

Enter cwolves

That brings us to cwolves, a startup entirely based around the LogSlash method and a licensee of FoxIO. They’re using a small AI model to recognize log field names, normalize them, and determine how to configure LogSlash to handle those fields. This means there’s no need to build normalization and LogSlash configs, those are now done automatically. The performance is incredible too.

The AI-based normalization and config builder is brilliant. For example, if your log uses the field name “data_win_eventdata_ip_src”, it automatically recognizes that as “Source IP” and knows how to configure LogSlash for that field. It will present you with the configuration it built so you can still manually modify it how you see fit, but the effort of building these configs is now gone. Amazing! For some organizations, cwolves may be worth it just for the auto-normalization alone.

That’s LogSlash configuration writing and performance taken care of. Another hurdle is implementation. cwolves aims to solve this with easy cloud or on-prem implementation with the ability to get started on-cloud, in minutes, for free. This is a great way to test out LogSlash though I suspect that most organizations will want this on-prem or within their own VPC, so it’s great that they offer that option.

The final hurdle to LogSlash is SIEM and AI model support. That is, they need to be aware that if the field “logslash=” exists in the log line, then it needs to duplicate the occurrence of each field by the logslash value while dividing fields that end in _total by the logslash value. This is straightforward to implement and cwolves is already developing a Splunk App that does just this.

With that Splunk App, cwolves makes LogSlash ready to go for existing Splunk customers. It also makes Splunk suddenly affordable and an attractive option for those who are SIEM shopping.

Try out LogSlash and cwolves at https://cwolves.com/

Contact us at FoxIO if you’d like to know more about LogSlash or if you’re interested in an OEM license.

This blog post introduces LogSlash, the method, benefits, and requirements. For more technical details, please visit our GitHub.

LogSlash is free for internal business use and is available here: https://github.com/FoxIO-LLC/LogSlash

The Too-Many-Logs Problem

Data logging is a foundational capability for all organizations. Logs are used in Machine Learning (ML) and Artificial Intelligence (AI) algorithms to track activity of interest and are critical for understanding the performance, availability, and security of the world’s infrastructure. As such, organizations tend to want to log everything, and rightfully so. Security Operators, for example, don’t want to miss potentially malicious activity. For various reasons they may also need to search across those logs going back several years. The solutions to date for processing and storing all of these logs have essentially been, ‘put the Big Data into a Data Lake or Data Warehouse and run searches and algorithms against it.’ This has led the world to spend ~$215 billion on logging infrastructure, licensing, power, and space in 2021, according to Statista. But what if we could cut that number in half?

Most organizations suffer from a too-many-logs problem where the cost of logging is straining budgets. The primary costs behind logging infrastructures are log ingestion and log processing/search, both in compute and licensing. Some log management licensing is based on the overall compute required, while others are based on log ingest volume. In either case, fewer logs equals less licensing costs as less compute is required to ingest, index, process, and search across the remaining logs. So how do we get less logs without losing valuable data?

The ultimate point of logging data is to understand what happened. In our quest to understand what happened, we tend to log and store everything. And it’s important to remember that in a work environment, both humans and systems tend to do the same process, over and over again. If a system does something 1,000 times in one minute, such as a load balancer connecting to an application server, do we really need to process, store, and search across 1,000 log lines that say the same thing? What if instead, we just process, store, and search across one log line that says this thing happened 1,000 times within a one-minute window? Both methods tell the same story of what happened and they both have the same value, but the latter is exceedingly more efficient to process, store, and run searches/algorithms against.

The LogSlash Method

LogSlash is a method for the reduction of log volume without sacrificing analytical capability. It can sit between your log producers (e.g, firewalls, systems, applications) and your existing log platform (e.g., Splunk, Databricks, Snowflake, S3). No need to change your logging infrastructure as this is designed to slot into any existing setup.

With LogSlash, 10TB/day of logs flowing into Splunk can be reduced to 5TB/day without any loss to the value of the logs. LogSlash does this by performing a time-window based consolidation of similar logs using configurable transforms to retain what’s valuable to you.

As an example, Bob connects to Gmail and starts clicking through his emails. While going through the emails, he clicks a malicious link.

These logs then flow through the LogSlash method before ingest into the logging platform following this logical configuration:

timewindow = 60s
Group\_by = \[ host, type, domain, dstip, dstport, proto, action \]
Fields\_ignored = srcport
Fields\_concatinated =
Fields\_sum = bytes

The output fields now include “timestamp-end”, “bytes-total” and “logslash,” which shows how many times the particular event occurred. We’re ignoring all but the first srcport as that field holds little analytical value. If we wanted to retain each srcport, we could configure LogSlash to concatenate them with a comma delimiter instead.

In this simplified example, we can see that Bob connected to Gmail 8 times over TLS/443 within a 53-second window with a total byte count of 3365kb and also attempted to connect to a malicious site once. LogSlash reduced the logs flowing into the log platform by 78% without any loss to the understanding of what happened. This can significantly reduce the cost to index, store, and process/search against log data, as well as reduce log platform licensing costs.

In testing LogSlash with a one-minute time window on AWS VPC Flow logs using normal corporate internet traffic, we found that on average, LogSlash reduced VPC Flow logs by 50%. In a production environment where the same systems are communicating to the same systems thousands of times a minute, such as load balancers communicating to application servers, LogSlash can reduce logs by as much as 95%, on average, without reducing the value of the logs.

In testing LogSlash with Zeek log data from corporate internet traffic, we found a reduction of 38% in SSL logs, 63% in Conn logs, 83% in DNS logs, 91% in HTTP logs, and 93% in File logs. The large delta between the savings with SSL logs and HTTP logs is due to SSL logging each connection which can persist for minutes whereas HTTP logs each transaction. It is transaction-based logging (e.g. application logs, S3 access logs) which can see massive savings with LogSlash.

For logs with unique identifiers (GUIDs), like in AWS CloudTrail, LogSlash can be configured to ignore all but the first GUID in a set of similar logs. This maintains a GUID to identify the set of events which can still be used as a primary key for log retrieval. Shared GUIDs hold a little more analytical value for pivoting between log types and can be concatenated if there are no other shared pivot points which depend on the log type. In either case, LogSlash still provides significant savings without sacrificing analytical capability.

Using LogSlash with other Log Reduction Methods

LogSlash is designed to work with other common log reduction methods such as standard compression, data deletion, and deduplication.

Deduplication. Deduplication is needed when you have a system that tends to log the exact same thing multiple times in a row. LogSlash is designed to reduce similar logs down to a single log without any loss to the value of what happened. LogSlash should be performed after deduplication and data deletion methods have already been performed.

Data Deletion. There are a number of technologies that facilitate removing erroneous data from logs before being passed off to their destination, such as removing the bloated Description field from Windows Event Logs. LogSlash provides significant savings on top of these standard data deletion methods.

Compression. ASCII compression tools like Gzip are used to losslessly compress logs for cheaper storage. When a search is performed against these logs, they are uncompressed and each log line is passed through the processor. Sending logs through LogSlash and then compressing the output significantly reduces log file size on disk compared to solely using compression. LogSlash also significantly reduces the compute and RAM required to uncompress and search across logs, as fewer bits are being passed through the processor at read time.

Implementing LogSlash

LogSlash was designed as a method that could be easily implemented into existing logging technologies and tools that you’re using today. Much like how JA3 was originally released as a Zeek Script before being implemented into vendor technologies, we are releasing LogSlash as Vector scripts. Vector is an open-source, lightweight, ultra-fast logging pipeline tool that contains the functions necessary to support LogSlash. We will soon be releasing LogSlash as Kafka Streams scripts and Cribl packs as well.

Much like log normalization, LogSlash requires a separate configuration for each log type so you can specify what fields are important and what fields can be transformed or ignored. As such, these configs can be built as part of the normalization development process. We are developing and releasing LogSlash scripts that can either be used directly or as templates for your own use cases.

LogSlash could also be implemented directly into the log source. As an example, S3 Audit logs in AWS could have an option to LogSlash before storing. Using LogSlash as a built-in option, developed by that technology’s team, would allow organizations to log what they otherwise potentially couldn’t afford, and at no extra effort to their observability teams.

In order for LogSlash to seamlessly integrate with log platforms, graphing tools, and anything else that relies on counting log lines to produce graphs and totals, those tools will need to be aware of LogSlash. For example, if you run a timechart function on Splunk against LogSlashed logs, it would require some fancy SPL’ing to output the correct graph because Splunk does not yet natively multiply or divide the amount of field values by the logslash value. Fortunately, adding support for handling LogSlashed logs is fairly straight-forward. Fast moving upstarts in log analysis, like Gravwell, have stated LogSlash support will be available in their next release cycle. We look forward to seeing many organizations benefit from dramatically reduced log data as more tools add LogSlash support in the coming months.

Future Development

While LogSlash is currently a set of simple scripts with a separate script for each log type, we’re working on a system that will adaptively recognize data types, so it can intelligently apply the appropriate LogSlash functions universally. This system will make it easy to onboard different log types with an initial configuration that can be tweaked. Logs will be LogSlashed on a rolling one-minute window by default, but this window will be configurable with potential savings dynamically displayed along with sample outputs.

Conclusion

LogSlash is a new standard method for the reduction of log volume without reducing log value and is designed to be implemented into existing technologies and logging pipelines. It’s a much more efficient approach to traditional logging. Essentially, why log, store, and search across 1,000 log lines that say the same thing when we can log, store and search across 1 log line that says the thing happened 1,000x? LogSlash significantly reduces compute and storage requirements and has the potential to double the value of log platforms that support it. Initially being released as a set of Vector scripts, we will soon be providing LogSlash as Kafka Streams and Cribl Packs and are working toward direct integration in logging pipeline and platform tools.

LogSlash is available here: https://github.com/FoxIO-LLC/LogSlash

LogSlash was created by John Althouse

LogSlash is patented by FoxIO, LLC — Patent US 10,877,972

Please reach out to us if you have any questions or have interest in commercial vendor licensing.

We are really excited to also share our new, innovative software license with the industry! We worked with renowned Open Source License expert Heather Meeker, along with legal professors and licensing lawyers, to create the FoxIO License. It is similar to the intention behind the Nmap Public Source License, while being only 2 pages, human readable, and reusable. Those familiar with licensing know that it takes a lot of expertise and time to create small, readable licenses. A lot of care and effort went into this. The FoxIO License is free to use for your own projects. Read more about it in an upcoming blog post!