- Martínez-Ramírez, José Manuel
- Carmona, Cristóbal José
This collection IDS-5-FCV-datasets contains (5-fold stratified cross-validation applied to several IDS-related datasets) the most relevant standardized datasets for IDS studies. The repository contains pre-processed, normalized datasets that can be used in further studies. The collection is only one in the literature:
KDD Cup 99
This is the data set used for The Third International Knowledge Discovery and Data Mining Tools Competition, which was held in conjunction with KDD-99 The Fifth International Conference on Knowledge Discovery and Data Mining. The competition task was to build a network intrusion detector, a predictive model capable of distinguishing between malicious connections and benign normal connections. This database contains a standard set of data to be audited, which includes a wide variety of intrusions simulated in a military network environment. The corrected version of this dataset includes over 300.000 examples in total and four malicious classes: DoS, Probe, URL and U2R, as well as benign connections. Each example contains 42 attributes that mostly refer to connection information, such as error rates, number of consecutive attempts, protocols and several flags, among others.
The original dataset can be found here: https://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html
UNB ISCX 2012
The UNB ISCX 2012 dataset was generated following a systematic approach to generate and provide information about attacks and benign connections, creating a dataset that is modifiable, extensible and reproducible. This dataset is based around profiles which contain detailed descriptions of intrusions and abstract models for protocols, applications or other network entities, generating real traffic for several commonplace web protocols such as HTTP, SMTP and SSH among others. This information is further complemented by multi-stage attack scenarios to generate malicious traffic, completing the dataset. The result is a dataset that contains much more information than any of the previously mentioned ones, including a complete capture of all network traffic and data, including all interactions within and between LANs as we all as the full packet load in PCAP formats, providing a vast amount of information for researchers to use. The tabular data in itself contains only 21 attributes that mostly refer to information about the source and destination, as well as packet sizes and payloads.
The original dataset can be found here: https://www.unb.ca/cic/datasets/ids.html
UNSW-NB15
This dataset was created by the IXIA PerfectStorm tool in the Cyber Range Lab of UNSW Canberra in Australia, aiming to generate a hybrid of real modern-day internet activities combined with synthetic attack behaviors. This dataset includes a total of 9 types of attack: fuzzers, analysis, backdoor, DoS, exploit, generic, reconnaissance, shellcode and worms. Furthermore, each connection includes 49 attributes generated using tools such as Argus and Bro-IDS. In total, there are roughly 2,540,044 examples. These attributes include information about both the source and destination of packet loads, statistical information on their size and contents, information about interpacket arrival time and number of connections that share the same service and address, among others.
The original dataset can be found here: https://research.unsw.edu.au/projects/unsw-nb15-dataset
NSL-KDD
This dataset was created to fix some of the problems detected in the KDD Cup 99 dataset. As such, it follows the same structure, labels and format as this dataset. The main issues it solves are the following:
Redundant records were removed, thus preventing bias towards frequent records.
The number of selected records from each group is inversely proportional to the percentage of records in the original dataset, allowing for a wider range and a more accurate evaluation of different learning techniques.
As a consequence of previous changes, the amount of data that the dataset presents is reasonable, allowing for the use of the complete set instead of a portion of it.
The original dataset can be found here: https://www.unb.ca/cic/datasets/nsl.html
WSN DS
This dataset includes information about data traffic in wireless sensor networks (WSN), which consist of a large number of autonomous sensor nodes distributed in areas of interest to gather data and transmit it to a central node in which it can be processed. Due to their nature, they are highly susceptible to attacks: an attacker can easily be injected into a WSN, and since they are permanently online sending and receiving data, they can be easily shut down using a DoS or DDoS attack. Due to this, IDS research can be applied to WSN easily, allowing to alert sensor nodes in case they are attacked, though it is usually a harder task due to the limited hardware resources that sensor nodes make use of. Regardless of the requirements of sensor nodes, the data they gather and the attacks that they may suffer can provide valuable information when detecting certain attacks. In order to properly represent this data, the WSN DS is created, modelling four types of DoS attacks: Blackhole, Grayhole, Flooding and Scheduling, including 23 features, including the node that was attacked, a timestamp, several flags, and specific information about the messages that are sent and received to and from other nodes.
The original dataset can be found here: https://www.kaggle.com/datasets/bassamkasasbeh1/wsnds
CIC-IDS2017
This dataset contains connection data on attacks that are common nowadays, as well as benign information gathered by the Canadian Institute of Cybersecurity (CIC). This real-world data was obtained by using CICFlowMeter, which is able to obtain web traffic and underlying information such as data flows, origin and destination IP, connection flags and packet flows. The dataset is stored throughout several CSV files, each of which has a specific date assigned. Each date contains information on several specific attacks. Included attacks on the dataset include SSH and FTP Bruteforce, DoS, DDoS, Heartbleed, Web Attacks, infiltrations, and bots. Each example contains 78 attributes and a label that defines it as either benign or malicious, and the specific attack type in the latter case.
The original dataset can be found here: https://www.unb.ca/cic/datasets/ids-2017.html
CIC-CSE-IDS2018
Along the previous dataset, the CIC also created CIC-CSE-IDS2018 in collaboration with the Communications Security Establishment (CSE). This dataset is structured much like the previous one, presenting several attack environments that affect over 400 computers and 30 servers. Logs with traffic information for each of these environments is provided, as well as 80 attributes extracted by using CICFlowMeter. These attributes are the same as the previous dataset, though the connection protocol and timestamp of the connection have been added. Attacks included in this dataset are more limited than in the previous dataset, containing only FTP and SSH Bruteforce and several denial of service attacks.
The original dataset can be found here: https://www.unb.ca/cic/datasets/ids-2018.html
CIC-DDoS2019
The CIC created another dataset focusing on distributed denial of service (DDoS) attacks. This dataset includes malicious data for both reflection-based DDoS and exploitation-based attacks. Reflection-based DDoS allows attackers to keep their identity hidden by using third-party components that seem legitimate. Packets are sent to reflection servers that focus all traffic towards the victim’s IP in an attempt to crash their connection with response packets. These attacks can be made from the transport layer protocols such as UDP, TCP, etc. On the other hand, exploitation-based attacks work much like the previous ones, though the intent is different, as they aim to send a massive amount of packets, such as SYN or UDP, to cause the victim’s connection to be severely slowed down or crash the system by exhausting the available bandwidth. The dataset includes examples of both types of DDoS and also benign traffic. Unlike previous CIC datasets, all data is included in one single file. Used attributes are exactly those of CIC-CSE-IDS2018. Moreover, two labels are present for each example: the first one specifies the type of attack, while the second one determines the type of connection (benign or malicious).
The original dataset can be found here: https://www.unb.ca/cic/datasets/ddos-2019.html
CIC-IoT2023
The last dataset created by the CIC is CIC-IoT2023, acting as a benchmark for the detection of large-scale attacks on IoT environments. This dataset includes 33 attacks in total, which are executed on an IoT topology composed of 105 devices. These attacks are furthermore divided into seven groups: DoS, DDoS, Recon, Web-based attacks, Brute Force, Spoofing and Mirai. Much like the previous CIC datasets, all gathered data is generated by using real connections and attacks on a controlled environment, guaranteeing a high-quality dataset. It includes a total of 46 attributes related to IoT connection data, including flow durations, header lengths, several connection flags and information on several other protocol-dependent parameters.
The original dataset can be found here: https://www.unb.ca/cic/datasets/iotdataset-2023.html
DS2OS
This dataset includes connection traces extracted from the DS2OS IoT environment. It should be noted that the traces that were gathered are extracted from the application layer instead of the network layer, therefore granting different information when compared to that provided by the other presented datasets. As it is based on an IoT network, it contains information associated with different sensors, such as thermometers, light controllers and movement sensors, as well as several smart items such as washing machines and mobile phones.
The original dataset can be found here: https://www.kaggle.com/datasets/francoisxa/ds2ostraffictraces
CIDDS-001
CIDDS, or Coburg Intrusion Detection Data Sets, is a concept to create evaluation datasets for anomaly-based network intrusion detection systems that was started by researchers from Hochschule Coburg in Germany. CIDDS-001 is the first dataset that was created under such a concept, and it emulates a small business environment that is targeted by malicious actors, and it includes Denial of Service, Brute Force attacks, and Portscans that were executed within the networks. The purpose of this dataset is to be as realistic as possible, so that different workers connect to different webs or applications based on their jobs, time of connection is considered to include breaks, and so on. The dataset includes 10 attributes, as well as the class and supporting information for it, such as unique attack IDs, specific attack type, and a description of the attack that provides further information. Other attributes include source and destination IP and port, protocol, start time of the first flow, flow duration, flow size in bytes, flow packets, and flags.
The original dataset can be found here: https://www.hs-coburg.de/forschen/cidds-coburg-intrusion-detection-data-sets/
TON-IoT
Much like the UNB ISCX 2012 dataset, this dataset was created by the UNSW to provide both information and a benchmark that researchers may use to test the behavior of several algorithms and models to detect malicious connections in the context of IoT networks. The dataset contains a vast amount of supplemental information, including information on Windows and Linux systems connected through the network, which were afterwards processed to obtain the final dataset. This dataset includes 45 attributes, which include information about the source and destination, the specific protocol that was used for communication, size and duration of the packet flows and DNS, HTTP and SSL information.
The original dataset can be found here: https://research.unsw.edu.au/projects/toniot-datasets
Kaggle Network Intrusion Dataset
This dataset was generated by simulating a US Air Force military network environment, then simulating several attacks over the resulting network, acquiring the corresponding TCP/IP dump. The resulting dataset has a total of 41 attributes and it presents a binary classification problem, in which each connection is labelled as either normal or anomalous. The information in the dataset includes the duration and size of the packet flows, the protocol used in it, which service it is associated with, several flags, information on both successful and unsuccessful logins and shell access, and server information, among others.
The original dataset can be found here: https://www.kaggle.com/datasets/sampadab17/network-intrusion-detection, This collection contains the most relevant standardized datasets for IDS studies. The repository contains pre-processed, normalized datasets that can be used in further studies. The collection is only one in the literature.