08680982.pdf
Architectures for Security: A comparative analysis
of hardware security features in Intel SGX and
ARM TrustZone
Muhammad Asim Mukhtar
Information Technology University
Lahore, Pakistan
Muhammad Khurram Bhatti
Information Technology University
Lahore, Pakistan
Guy Gogniat
University of South Brittany
Lorient, France
Abstract—A variety of applications are executing on a large untrusted computing base, which includes the operating system, hypervisor, firmware, and hardware. This large computing base is becoming complex and unverifiable. This untrusted computing base problem opens a way for a malicious application to steal secrets of a security-critical application by compromising the untrusted computing base. To resolve the untrusted computing base problem, computer architectures have introduced a concept of the trusted execution environment, which aim to ensure the sensitive data to be stored and processed in an isolated environment. Existing popular trusted execution environments are relying on hardware to isolate the environments without or minimum relying on system software. However, existing hardware assisted trusted execution environments are still vul- nerable to sophisticated attacks. This paper analyses popular trusted execution environments that are Intel SGX and ARM TrustZone in order to provide better insights about the intended scope of the protection. This paper illustrates the functionality, implementation and security analysis.
Index Terms—Trusted Execution Environments, TEE, Memory isolation, Intel SGX, and ARM TrustZone.
I. INTRODUCTION
Normal and security-critical applications are executing on
a large untrusted computing base, which includes an operat-
ing system, hypervisor, firmware, and hardware. This large
computing base is becoming complex and unverifiable. For
example, an operating system such as Linux has 17 millions
of lines code [2] and CVE has reported 166 vulnerabilities in it
during the year of 2018 related to Denial-of-Service, overflow,
unauthorized privilege gain, memory corruption, directory
traversal, execute unauthorized code. Similarly, Xen is a well-
known hypervisor that has 150,000 lines code [27], which has
relatively small code than Linux but still has vulnerabilities,
and CVE has reported 18 vulnerabilities in Xen in the year
of 2018 [11]. Moreover, attacks that subvert firmware are
reported [1] [25] [23]. Execution of normal and security-
critical applications are executing on shared resources that
controlled by untrusted computing base raises security threats.
This opens the way for a malicious application to attack the
This research work is partially supported by the PHC PERIDOT Project e-health.SECURE and National Center for Cyber Security (NCCS), Pakistan.
vulnerabilities to gain the unauthorized privilege, and then
steal secrets form security critical application’s address space.
To cope up the untrusted computing base problem, computer
architectures have introduced the concept of trusted execution
environments that aim to isolate security-critical applications
from untrusted computing base. Trusted execution environ-
ments guarantee security by relying on less hardware and
software computing base. Hardware is generally considered
as the trusted base because the cost and complexity of attacks
on hardware are usually high [12]. This leads the industry to
develop computer architectures to develop a trusted execution
environment for security-critical application maintained by
hardware with no or less dependency on OS and hypervi-
sor. These architectures includes ARM TrustZone Technology
[17], Intel Software Guard eXtensions (SGX) [14] [20], AMD
Memory Encryption Technologies [15], AMD Platform Secure
Processor [13], x86 System Management Mode [8], and Intel
Management Engine (ME) [22].
Intel SGX and ARM TrustZone are popular trusted exe-
cution environments. Both Intel SGX and ARM TrustZone
are hardware assisted trusted execution environments but the
mechanism behind making the trusted environment for trusted
applications are different. Intel SGX creates a trusted environ-
ment for trusted applications such that it executes over existing
untrusted system software. Whereas, ARM TrustZone creates
a new trusted world for trusted applications that executes over
trusted system software and hardware that only visible to the
trusted world. These TEEs are vulnerable to a different set
of attacks because of different mechanism of creating trusted
execution environments. This paper analyses these trusted
execution environments in order to provide better insights into
the intended scope of the protection. This paper illustrates the
functionality, implementation and security analysis.
II. FUNCTIONALITY AND IMPLEMENTATION OF ARM
TRUSTZONE AND INTEL SGX
A. ARM TrustZone
ARM TrustZone provides a set of intellectual property cores
(IP blocks) to develop a partitioning based security framework.
The way of combining these IP blocks by chip manufactures
� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �
� � �� � � � � ! " # � � # � � � ! $ � � $ % " � & � � ' � � � � � � � �
define the security properties. In general ARM TrustZone
divides a system’s resources (CPU, memory, and peripherals)
into two classes referred to as secure world and normal world
[17].
Processor
with secure
extensions
AMBA AXI bus with secure bit extension
Interrupt
Controller
Caches
SRAM
TZMA
AXI to APB
Bridge
APB Bus
Keypad
Controller
ADC
DAC
Other
peripherals
DRAM
TZASC
TZMA
ROM
System-On-Chip
Secure aware Component
Fig. 1. ARM TrustZone Implementation
Untrusted
OS
App
Hypervisor
EL-0
EL-1
App
Trusted
OS
Secure
Firmware
Hardware
EL-2
EL-3
No EL-2 in
Secure World
������� ��� � �� ���� ��� �
EL Exception Level
App Application
OS Operating System
No EL-3 in
Normal World
Secure
Monitor
Fig. 2. ARM TrustZone software stack.
ARM TrustZone introduced a new execution environment in
processor referred as secure world along with the old execution
environment referred as normal world. The secure world also
has multiple privilege levels same as normal world. Therefore,
whole trusted software stack can be developed from user-
level to system-level except hypervisor in secure-world, as
shown in Figure 2. Each world has its own operating system
and manages the resources for application belonging to its
world’s space. One world’s software stack executes at a time
on a processor. The context switching between secure and
normal world is handled by monitor mode, which is the highest
privilege-level of the secure world and can access both worlds
system’s resources. To reflect the current processor’s world
to other system’s resources, bus is extended with a new bit
called non-secure bit (NS). Cache lines are also extended with
NS bit, which specifies the security state of the cache line.
Extension of NS bit in each cache line eliminates the need
of flushing the cache lines while context switches between a
secure world and normal world, which results in low context
switching overhead. The allocation of cache lines depends on
the demand of each world and can evict the cache line of
another world. TrustZone processor provides separate address
translation units for the secure and normal worlds. This is
achieved by implementing two page-table base registers, which
are used by page walker according to the processor’s current
world. The physical addresses in the page-table entries are
also extended to include the values of the NS bit to be issued
on the AXI bus, as shown in Figure 3. The addition of
secure bit in the address tag for each cache line effectively
creates completely different views of the memory space to the
software executing in different worlds.
VA NSTID
VA NSTID
VA NSTID
PA NS
PA NS
TLB
Processor
PA NS
PA NS
DATA
L1 Cache
To AXI bus
VA Virtual Address
PA Physical Address
NSTID Non Secure Tag Identi er
NS Non Secure bit
DATA
Fig. 3. ARM TrustZone TLB and cache isolation.
The memory modules like DRAM, SRAM, and ROM that
are not designed according to extended AXI bus can be
connected using adapters as shown in Figure 1. The TrustZone
Memory Adapter (TZMA) can be used to partition an on-
chip ROM or SRAM into a secure region and a normal
region, and the TrustZone Address Space Controller (TZASC)
partitions the memory space provided by a DRAM controller
into secure and normal regions. A TrustZone-aware DMA
controller rejects DMA transfers from the normal world that
reference secure world addresses.
In ARM system most peripherals are connected to the APB
bus, which is a low power bus than the main AXI bus. The
APB protocol does not carry the NS bit. To defeat software
attacks that use peripheral to extract information, and ARM
TrustZone introduced the security handling feature AXI-to-
APB Bridge, which interfaces the high-speed AXI domain to
the low-power APB domain. The bridge contains an address
decoder that selects the APB peripheral based on the incoming
AXI transaction. The bridge takes a single bit input for each
peripheral that is located on the bus to determine whether the
peripheral is configured as secure or non-secure. The AXI-to-
APB bridge will reject non-secure requests to secure peripheral
address ranges using NS bit information of AXI bus.
� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �
� � �
There are two types of hardware interrupts in ARM Trust-
Zone: Fast Interrupt Request (FIQ) and Interrupt Request
(IRQ). Both of the interrupts can be configured as secure-
interrupt by configuring the IRQ bit and FIQ bit in secure-
configuration register (SCR). The secure interrupt is directly
trapped into monitor mode (EL-3 level) and then forwarded to
the normal world if required. ARM recommends that the IRQ
should be used as the interrupt source of the normal world
and the FIQ should be used as a secure-interrupt because
most commonly used interrupt source most of the operating
environments is IRQ, so the use of FIQ as the secure interrupt
will require the fewest modifications to existing software.
B. Intel Software Guard Extensions (SGX)
Intel has provided the general purpose hardware-assisted
TEE referred as Intel SGX. Intel SGX is an extension of x86
architecture with new set of security-related instructions [6]
[8] [7]. These instructions are used by the security-critical
applications to build hardware-assisted trusted environment
referred to as an enclave [19]. Intel SGX enclave ensures
the confidentiality using memory access checks with hardware
maintained data structure and integrity by encrypting of data
and code when goes outside the CPU package [16]. Intel SGX
is a centralized security model, trusted computing base TCB
is considered to be the CPU package.
Execution
Unit
Page Miss
Handler
L1-Data
Cache
L1-Instruction
Cache
Instruction
Decoder
L2
Cache
Core-0
L3
Cache
Memory Controller
Memory Encryption Engine
QPI
Router
I/O
Controller
Other
Cores
Other
Secondary
Units
CPU
Other
CPU NIC
Platform Controller
Hub
USB, SATA, ME
DRAM
DDR
QPI PCI-X DMI
Secure aware components
Fig. 4. Intel SGX implementation.
SGX processor does not provide orthogonal privileged
levels to secure application as in the TrustZone, as shown
in Figure 5. The application executes on the same untrusted
operating system but security is achieved by matching with
hardware managed meta-data, which OS cannot read or write.
The reason behind achieving security over conventional soft-
ware stack is to minimize the effort required to modify
application code to benefit from SGX. History suggests this is
a wise decision, as a large factor in the continued dominance
of the Intel architecture is its ability to maintain backward
compatibility.
Conceptually, Intel SGX offers a new type of address space
to an application referred to as processor reserved memory
(PRM) that have special security properties as shown in Figure
6. PRM address space cannot read or write by high privileged
software such as operating system and hypervisor although it
contains the code/data of low privileged level. Inversely, the
code/data in PRM doesn’t have high privilege than the system
software because the code in it cannot access system software
address space. Moreover, processor checks and overrides the
memory mapping decision taken by OS if processor finds an
inconsistency with meta-data that is managed by the processor
without the assistance of OS, which is called enclave page
cache map (EPCM). EPCM contains expected virtual address
used to access enclave page and access permissions i.e. read,
write and execute of each enclave page.
Processor encrypts the pages in PRM when goes out of
the CPU chip, which guarantees security against bus tapping
attacks. The processor also measures signature before loading
into cache to ensure the integrity of page. This captures the
malicious write, read or replacing with other pages by OS but
cannot restrict the OS form doing such malicious acts.
To achieve trusted storage SGX modifies the CPU compo-
nents only in system resources, as shown in Figure 4. This
is because every memory and I/O access requests transferred
through the processor, so checks at CPU is sufficient for
achieving security. In SGX the major changes are in three
components: instruction decoder, page miss handler, and mem-
ory controller. Firstly, 18 new instructions are introduced in
instruction decoder, which contains 5 user instructions that
used by an application to initialize and build enclave and 13
supervisor instructions that are used by OS to manage enclave
page table. Also, microcode related to memory access checks
are added which are triggered by page miss handler [10].
Secondly, PMH hardware is modified to develop an ability to
trigger the microcode assist for all address translations when
a logical processor is in enclave mode, or when the physical
address produced by the page walker machine matches the
PRM range. Lastly, new register referred as processor reserved
memory range registers (PRMRR) is introduced in the memory
controller, which defines the size of PRM. Also, the memory
controller is integrated with a memory encryption engine,
which uses non-standard cryptographic primitives that consists
of slightly modified AES operating mode [21] and a Carter-
Wegman MAC construction [4] [24].
III. ARM TRUSTZONE AND INTEL SGX PROTECTION
AGAINST VULNERABILITIES
Trusted execution environments aim is to ensure secure data
to be stored and processed in an isolated, trusted environment.
Trusted execution environments reduce the computing base so
as to limit the links of security-critical application with the
potentially malicious applications. Direct or indirect critical
links between attacker and security-critical applications can
be broadly categorized into three classes: Attacker makes
link to security critical application through privileged software
(Figure 7, referred to as Vulnerability-1), Attacker links to
security critical application through micro-architectural events
of hardware (Figure 8, referred to as Vulnerability-2), and
� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �
� � �
Attacker makes link to security critical application through di-
rectly probing hardware (Figure 9, referred to as Vulnerability-
3).
OS
App App
Hardware
RL 3
RL 0
Enclave – code that require trusted execution
environment �
Code that manage page table for enclave��
Data structures manged by hardware for security
checks
RL Ring level
App Application
OS Operating System
Fig. 5. Intel SGX software stack.
PRM
DRAM
Enclave
Application’s
Virtual address
space
Page table
managed by OS
PRM
4 kB page
4 kB page
4 kB page
EPCM
processor managed
metadata regarding pages
metadata entry
metadata entry
metadata entry
EPCM
Fig. 6. Intel SGX trusted storage.
Compromised OS
Malicious App App
Hardware
Fig. 7. Vulnerability 1: Malicious application makes link to other application’s secure data through privileged software.
OS
Malicious App App
Hardware
Fig. 8. Vulnerability 2: Malicious application makes link to other application’s secure data through micro-architecture events of hardware
A. Vulnerability -1
The attacker uses system software privilege to make a link
to secure data of an application. This privilege the attacker
to modify the page tables and TLBs, unauthorized DMA
transfer and Denial of service (DoS). The attacker cannot
Compromised OS
App App
Hardware
Lab Equipment
Fig. 9. Vulnerability 3: Malicious application makes link to other application’s secure data by directly probing the hardware
able to modify page tables and TLBs in TrustZone and SGX.
TrustZone page tables and TLBs are managed by secure
world operating systems and stored in the secure memory
region, which is not accessed by untrusted system software.
Moreover, TrustZone has also divided the TLBs for secure
and normal world. Intel SGX protects the modification of
page tables by encrypting it and placing it in PRM, which
cannot be accessed by system software. Intel SGX flushes TLB
on exiting from enclave and applies security checks before
storing address translation into TLB. These both reasons make
the untrusted modification in TLB difficult. Direct memory
accesses are also bounced back in TrustZone and SGX, which
makes these architectures secure from DAM based attacks.
DoS in SGX can easily be launched as compared to TrustZone
because secure application page table and TLB management
is the responsibility of untrusted operating system whereas in
TrustZone trusted operating system manages the page tables
and TLB.
B. Vulnerability-2
Intel SGX and ARM TrustZone architecture are vulnerable
to cache-based side-channel attacks. The success of the cache-
based side-channel attacks depends on the contention between
the attacker and victim for getting cache lines.
Intel SGX is using the same cache architecture as in the non-
secure architectures, which is vulnerable to cache-based side
channel attacks. Schwarz et al. attacks the SGX enclave via
cache side channels and demonstrates that the private key in
the RSA implementation of mbedTLS can be extracted within
five minutes. Other than RSA decryption, Ferdinand also
demonstrates a more efficient attack on the human genome
indexing via SGX cache-based information leakage.
In ARM TrustZone each cache lines are extended with a
non-secure bit to distinguish the belonging to cache line to
secure or non-secure world but these cache lines are allocated
on the bases of memory access demand form the world at
runtime and contend for the cache line, which makes the
ARM TrustZone still vulnerable to cache-based. ARMageddon
[18] uses Prime+Probe cache attack to extract the informa-
tion from secure world to normal world and mentioned that
Prime+Probe attack on the ARM TrustZone is not much
different from a Prime+Probe attack on any application in
the normal world. TruSpy [26] uses Prime+Probe to extract
the key of AES encryption running in secure-world in 2.5
second form compromised OS of the normal world. Moreover,
authors of [26] showed that the AES key can be extracted
� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �
� � �
TABLE I TRUSTED COMPUTING BASE
Trusted Computing Base ARM TrsutZone Intel SGX
Software Application Application Operating System Page Table management part of OS
Firmware
Hardware System on Chip package CPU chip package
TABLE II RESILIENCE OF INTEL SGX AND ARM TRUSTZONE TO SET OF ATTACKS
Vulnerabilities ARM TrustZone Intel SGX
Vulnerability-1 Memory Mapping Passive Attacks Secure because secure world’s page-tables and TLBs,which is divided into two in TrustZone, are managed by trusted system software
Not secure because en- clave’s page-tables and TLBs are managed by un- trusted system software
Memory Mapping Active Attacks based on Page tables Secure because secure world’s page tables are managed by trusted system software
Secure because enclaves page tables are encrypted and detected if modified by untrusted system soft- ware
Memory Mapping Active Attacks based on TLB Secure because of TLB managed by trusted sys- tem software
Secure because of flushing mechanism of TLB
DMA based Attacks Secure because DMA ac- cesses are rejected in se- cure world
Secure because DMA accesses are rejected in PRM
Firmware based Attacks Secure because firmware is a part of secure world
Secure because firmware is subject to TLB access checks
Denial of Service Attack Weakly secured because secure world’s system software is separated from normal world’s system software but monitor mode still has link with untrusted system software
Not secure because en- clave is executing over un- trusted system software
Vulnerability-2 Cache Based Side Channel Attacks Not secure because of at- tacker can evict the de- sired cache line
Not secure because of at- tacker can evict the de- sired cache line
Row-hammer Attack Security depends on the developer if developer includes integrity check mechanism then it is safe
Secure because SGX can detect the unexpected modification of data because of integrity check mechanism
Vulnerability-3 Port Attacks Safe as soon as designer disables debug ports
Safe because debug ports are usually disabled
Bus Tapping Attacks Safe as soon as secure data is limited to memory with in the SoC
Safe because CPU en- crypts data when trans- ferred on bus.
Chip Attacks Not secure Not secure Power Analysis Attacks Not secure Not secure
in a couple of minutes without compromising the OS of the
normal world. Recently, Prime+Count-based crossword covert
channel is presented in [5], which can transfer information
with bandwidth as high as 27 KB/s under the single-core
scenario and 95 B/s under the cross-core scenario.
Row hammer attack flips a bit in DRAM by accessing the
nearby row of the victim row in DRAM. This can modify
the data in DRAM, which may lead to privilege change or
corruption of secure data. SGX store hash of secure data
with it in DRAM, so modification can easily be detected by
measuring the hash again before using. TrustZone is safe from
such attack if all secure data and code are in the memory of
SoC. in case of external DRAM, the designer has to include
software or hardware based hash mechanism for the integrity
of secure data.
C. Vulnerability-3
The attacker directly probes computer hardware using ex-
ternal debuggers, chip imaging and electrical characteristic
measuring equipment to extract secrets such as keys in e-
fuses. debug ports are used in chip manufacturing process to
evaluate the condition of a chip and these ports are disabled
� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �
� � �
before shipping. There is a debug port in Intel architecture
called as Generic Debug eXternal Connection (GDXC), which
collects data from CPU chip, and transfers it to an external
debugger. These ports are not intended to be used by software
developers, therefore, the Intel manual [8] [6] [7] [9] does
not share details about the working of such debug ports that
how GDXC interact with enclave. In ARMTrustZone specific
platform Zynq-7000 SoC there is a general purpose port (GP
port) that connect the ARM TrustZone SoC with custom build
platform, which is on FPGA. [3] shows the vulnerabilities
related to this port that NS bit can be overridden in custom
build platform connected to ARMTrustZone SoC. TrustZone’s
and SGX’x threat model excludes chip imaging and power
analysis attacks. Chip imaging attacks are dependent on the
cost of attack because it requires costly equipment like ion-
beam microscopy, especially in the case smaller feature size.
However, the keys stored in efuse, which has a large feature
size, can be vulnerable. Moreover, Power attacks cannot be
addressed at the architectural level. It follows that defending
against chip and power analysis attacks have a very high cost-
to-benefit ratio.
IV. CONCLUSION
The main aim of developing TEE is to ensure sensitive
data is stored and processed in an isolated, trusted envi-
ronment. SGX and TrustZone are successful in providing
trusted storage. However, both SGX and TrustZone are failed
to ensure isolation while processing because of software-
based side channel attacks. Mitigating software based side
channel attacks using existing isolation features incurs a high-
performance cost. There must be hardware based mechanism
to mitigate software based side channel attacks because these
attacks are targeting the microarchitectural events that are
either transparent to software stack or coarse grain controlled
from software stack. As side channel vulnerabilities exploit
an inherent feature of the hardware resource. As Caches
are added to reduce memory access latency, the attacker is
exploiting this timing difference. So, it is difficult to eliminate
the timing difference. To limit such vulnerability the hardware-
assisted locking, partitioning or memory-to-cache mapping
randomization based mechanism is needed.
REFERENCES
[1] O. Bazhaniuk J. Loucaides A. Matrosov A. Furtak, Y. Bulygin and M. Gorobets. Bios and secure boot attacks uncovered. The 10th ekoparty Security Conference, 2014.
[2] Sebastian Anthony. Who actually develops linux? the answer might surprise you. http://www.extremetech.com/computing/175919-who- actually-develops-linux, 2014.
[3] E. M. Benhani, C. Marchand, A. Aubert, and L. Bossuet. On the security evaluation of the arm trustzone extension in a heterogeneous soc. In 2017 30th IEEE International System-on-Chip Conference (SOCC), pages 108–113, Sep. 2017.
[4] J. Lawrence Carter and Mark N. Wegman. Universal classes of hash functions (extended abstract). In Proceedings of the Ninth Annual ACM Symposium on Theory of Computing, STOC ’77, pages 106–112, New York, NY, USA, 1977. ACM.
[5] Haehyun Cho, Penghui Zhang, Donguk Kim, Jinbum Park, Choong- Hoon Lee, Ziming Zhao, Adam Doupé, and Gail-Joon Ahn. Prime+count: Novel cross-world covert channels on arm trustzone. In Proceedings of the 34th Annual Computer Security Applications Conference, ACSAC ’18, pages 441–452, New York, NY, USA, 2018. ACM.
[6] Intel Corporation. Software guard extensions programming reference. 2013.
[7] Intel Corporation. Intel r 64 and ia-32 architectures optimization reference manual. 2014.
[8] Intel Corporation. Intel r 64 and ia-32 architectures software developers manual. 2015.
[9] Intel Corporation. Intel r software guard extensions (intel r sgx). 2015. [10] Victor Costan and Srinivas Devadas. Intel sgx explained. IACR
Cryptology ePrint Archive, 2016:86, 2016. [11] CVE Details. Xen : Vulnerability statistics.
https://www.cvedetails.com/vendor/6276/XEN.html, 2018. [12] J. Grand. Advanced hardware hacking techniques, 2004. [13] AMD TATS BIOS Development Group. Amd security and server inno-
vation. http://www.uefi.org/sites/default/files/resources/UEFI-PlugFest- AMD-Security-and-Server-innovation-AMD-March-2013.pdf, 2013.
[14] Matthew Hoekstra, Reshma Lal, Pradeep Pappachan, Vinay Phegade, and Juan Del Cuvillo. Using innovative instructions to create trustworthy software solutions. In Proceedings of the 2Nd International Workshop on Hardware and Architectural Support for Security and Privacy, HASP ’13, pages 11:1–11:1, New York, NY, USA, 2013. ACM.
[15] D. Kaplan J. Powell and T. Woller. Amd memory encryption, white pape. http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/ 12/AMD-Memory-Encryption-Whitepaperv7 − Public.pdf, 2016.
[16] Simon P Johnson, Uday R Savagaonkar, Vincent R Scarlata, Francis X McKeen, and Carlos V Rozas. Technique for supporting multiple secure enclaves, 2015. US Patent 8,972,746.
[17] ARM Limited. Arm security technology building a secure system using trustzone technology. IACR Cryptology ePrint Archive, 2019.
[18] Moritz Lipp, Daniel Gruss, Raphael Spreitzer, Clémentine Maurice, and Stefan Mangard. Armageddon: Cache attacks on mobile devices. In 25th USENIX Security Symposium (USENIX Security 16), pages 549– 564, Austin, TX, 2016. USENIX Association.
[19] Francis X McKeen, Carlos V Rozas, Uday R Savagaonkar, Simon P Johnson, Vincent Scarlata, Michael A Goldsmith, Ernie Brickell, Jiang Tao Li, Howard C Herbert, Prashant Dewan, et al. Method and apparatus to provide secure application execution, 2015. US Patent 9,087,200.
[20] Frank McKeen, Ilya Alexandrovich, Alex Berenzon, Carlos V. Rozas, Hisham Shafi, Vedvyas Shanbhogue, and Uday R. Savagaonkar. In- novative instructions and software model for isolated execution. In Proceedings of the 2Nd International Workshop on Hardware and
Architectural Support for Security and Privacy, HASP ’13, pages 10:1– 10:1, New York, NY, USA, 2013. ACM.
[21] Mahesh S Natu, Sham Datta, Jeff Wiedemeier, James R Vash, Sailesh Kottapalli, Scott P Bobholz, and Allen Baum. Supporting advanced ras features in a secured computing system, October 30 2012. US Patent 8,301,907.
[22] X. Ruan. Platform embedded security technology revealed: Safeguarding the future of computing with intel embedded security and management engine, 2014.
[23] A. Tereshkin and R. Wojtczuk. Introducing ring-3 rootkits. Master’s Thesis, 2010.
[24] Mark N Wegman and J Lawrence Carter. New hash functions and their use in authentication and set equality. Journal of computer and system sciences, 22(3):265–279, 1981.
[25] R. Wojtczuk and A. Tereshkin. Attacking intel bios. Invisible Things Lab, 2010.
[26] Ning Zhang, Kun Sun, Deborah Shands, Wenjing Lou, and Yi- wei Thomas Hou. Truspy: Cache side-channel information leakage from the secure world on arm devices. IACR Cryptology ePrint Archive, 2016:980, 2016.
[27] Xiantao Zhang and Yaozu Dong. Optimizing xen vmm based on intel virtualization technology. 2008 International Conference on Internet Computing in Science and Engineering, pages 367–374, 2008.
� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �
� �
08109087.pdf
Experience Report: Study of Vulnerabilities of Enterprise Operating Systems
Anatoliy Gorbenko School of Computing, Creative Technologies & Engineering,
Leeds Beckett University, Leeds, UK
Alexander Romanovsky School of Computing Science,
Newcastle University, Newcastle-upon-Tyne, UK
Alexander.Romanovsky @ncl.ac.uk
Olga Tarasyuk Department of Computer Systems and Networks,
National Aerospace University, Kharkiv, Ukraine
Oleksandr Biloborodov Plarium Ukraine LLC,
Kharkiv, Ukraine [email protected]
Abstract—This experience report analyses security problems of modern computer systems caused by vulnerabilities in their operating systems. An aggregated vulnerability database has been developed by joining vulnerability records from two publicly available vulnerability databases: the Common Vulnerabilities and Exposures system (CVE) and the National Vulnerabilities database (NVD). The aggregated data allow us to investigate the stages of the vulnerability life cycle, vulnerability disclosure and the elimination statistics for different operating systems. The specific technical areas the paper covers are the quantitative assessment of vulnerabilities discovered and fixed in operating systems, the estimation of time that vendors spend on patch issuing, and the analysis of the vulnerability criticality and identification of vulnerabilities common for different operating systems.
Keywords: security, vulnerability, operating systems, vulnerability databases, days-of-risk, forever-day vulnerabilities, vulnerability life cycle, vulnerability statistics
I. INTRODUCTION Security of information and communication systems has
become one of the most crucial concerns for both system developers and users. Recent security incidents like those happened to Hollywood Presbyterian Medical Center [1] or San Francisco Municipal Transportation Agency [2] show how vulnerable to attacks our modern society is. They may cost millions of US dollars and even affect human lives. One of the main reasons of successful attacks, malicious intrusions and virus infections are software vulnerabilities in computer systems, communication equipment, smartphones and other intellectual devices.
Generally speaking, vulnerability is a weakness which allows an intruder to undermine system’s information assurance. MITRE Corporation defines vulnerability as a software fault that can be directly used by a hacker to gain access to a system or network [3]. Exploiting vulnerability allows attackers to either execute commands as normal users, or access data violating the specified access restrictions or cause denial of service attack and terminate system services. Software vulnerabilities are mainly caused by errors and weaknesses in software design and implementation. For instance, in 2016 Microsoft Inc. issued 561 updates for its family of operating systems [4]. 311 of them were security
updates eliminating software vulnerabilities, among which 114 updates were ranked as critical. During the first 4 months of 2017 Microsoft has issued 63 security updates out of 99.
Vulnerabilities can be discovered in both application software and operating systems. For instance, CVE-2016- 7256 vulnerability refers to a weakness in the Windows family of operating systems when the font library improperly handles specially crafted embedded OpenType fonts [5, 6, 7]. An attacker can successfully exploit this vulnerability through web-based or file sharing attack scenarios. As a result, a full control upon the affected system can be taken allowing hackers to install programs, view, change or delete data, create new user accounts with administrative rights, etc. Another example is the CVE-2014-0160 vulnerability [8] in the OpenSSL cryptography library that allows remote attackers to obtain sensitive information from process memory and even to compromise server secret key via crafted packets that trigger a buffer over-read. It has affected half a million widely trusted websites and services including Yahoo, Amazon Web Services, GitHub, Wikipedia, etc. This vulnerability existed since December, 2012 and was disclosed only in April, 2014.
There is no doubt that vulnerabilities in operating systems are the most critical security threats as they allow to violate application-oriented access control [9]. Besides, their exploitation can compromise all processes and services running in the operating system and allow attackers to gain access to all data stored on the vulnerable computer. They represent threats to system security and dependability that are additional to faults, errors and failures traditionally dealt with by the dependability community.
In the paper we examine vulnerabilities of popular enterprise operating systems, investigate statistics of vulnerability disclosure and elimination and analyze their criticality. Our results have been obtained by aggregating statistics provided by the two most reputable vulnerability databases CVE (www.cve.mitre.org) and NVD (web.nvd.nist.gov). In contrast to other works investigating software vulnerabilities [10, 11, 12] this paper focuses on examining how vulnerability disclosure rates have been changing over the last years for particular operating systems and how much time OS vendors spend on issuing patches to fix that security flaws and how many yet unfixed
2017 IEEE 28th International Symposium on Software Reliability Engineering
2332-6549/17 $31.00 © 2017 IEEE
DOI 10.1109/ISSRE.2017.20
205
vulnerabilities can exist in each OS simultaneously. In addition to this we expand the research results reported in the earlier works [13, 14, 15, 16, 17] by investigating the trends in vulnerability severity and discussing the most common vulnerability types.
Forewarned is forearmed! We believe our study is one of the first which quantitatively demonstrates that (i) a period of time between a vulnerability is disclosed and the patch becomes available can be up to several months long; (ii) a number of forever-day vulnerabilities can reach several dozen; (iii) each OS has only few if any known vulnerability- free days during its lifetime. The reported statistics will also help system administrators facing a challenge of choosing an operating system for secure hosting of corporate/web services to make a grounded decision.
The rest of the paper is organized as follows. In the next section we briefly describe the research methodology used. Section III discusses the vulnerability life cycle and a notion of days-of-risk metric defining how fast software vendors react on vulnerabilities discovered in their products. In section IV we investigate various aspects of OS vulnerability and present vulnerabilities discovery and elimination statistics, discuss vulnerability severity and analyse vulnerabilities discovered in more than one operating system. Finally, some practical lessons learnt from our work are summarized in section V.
II. VULNERABILITY DATABASES AND RESEARCH METHODOLOGY
There are a lot of institutions focusing their activity on vulnerability discovery and elimination. They include, undoubtedly, software vendors as well as a number of governmental and independent international organizations, commercial enterprises and even individuals. Most of these institutions provide publically available vulnerability datasets. The most known and trusted are:
• CVE – the Common Vulnerabilities and Exposures system provided by not-for-profit MITRE Inc. (cve.mitre.org). MITRE maintains a list of known vulnerabilities and performs their enumeration by assigning CVE-IDs which are used by many others vulnerability databases to synchronize with CVE and enable data exchange between security databases and products. In 2016 MITRE has assigned more than 9900 vulnerability identifiers.
• NVD is the National Vulnerability Database provided by U.S. National Institute of Standards and Technology (web.nvd.nist.gov). NVD offers a dataset of software security vulnerabilities which is based upon and synchronized with CVE. It classifies vulnerability severity and type and also specifies vulnerable software and provides additional meta- data using the Common Vulnerability Scoring System (CVSS), the Common Weakness Enumeration Specification (CWE) and the Common Platform Enumeration Dictionary (CPE). It has reported almost 6000 vulnerabilities disclosed in 2016 which is 16.5 vulnerabilities per day in
average. About half of them have been observed in operating systems.
• VNDB is the Vulnerability Notes Database provided by CERT (www.kb.cert.org/vuls/). Most of VNDB entries are covered by CVE and NVD. Though, it was reported that the CERT VNDB vulnerability disclosures appear on its website 24–72 hours before they appear in CVE or NVD [18].
• VulnDB is the Risk Based Security’s vulnerability database (www.riskbasedsecurity.com/vulndb/). VulnDB tracks vulnerabilities in third-party libraries and claims to provide over 47,000 vulnerabilities that are not found in CVE or NVD. Though, its commercial offering prevents VulnDB from been widely used by researches.
• SecurityTracker is another vulnerability dataset commercially available at securitytracker.com. It almost entirely covers vulnerability entries that have CVE IDs.
Besides, many software product vendors often provide information about vulnerabilities in their products as through the security bulletins (e.g. https://technet.microsoft.com/en- us/security/bulletins.aspx). At the same time, OSVDB (Open Source Vulnerability Database) and FVDB (Frei's Vulnerability Database) that recently have been actively used by many researchers seem to be no longer available.
Both, CVE and NVD offer access to vulnerability data sets through the simple search interface available on their web sites or by distributing XML data feeds. Unfortunately, they do not support SQL querying making difficult a direct use of the CVE and NVD repositories for complex analytics. In our study we have merged together XML data files provided by CVE and NVD and inserted the joint data set into MySQL database. We use CVE-ID as a primary key to uniquely identify each vulnerability. CPE identifiers provided by NVD are used to assign a particular vulnerability to a certain product out of the three main groups: (i) operating systems, (ii) application software and (iii) hardware components (e.g. routers, graphical cards, embedded devices, etc.). Besides, we store two date items associated with the same vulnerability: when it firstly appears in CVE and when its description is published by NVD.
By now, our joined MySQL database includes more than 100000 vulnerabilities (see Table I). The list of vulnerable software for 4065 vulnerabilities includes both operating systems and application software. 641 vulnerabilities have been discovered simultaneously in operating systems and hardware components, and 555 vulnerabilities – in application software and hardware.
Besides, 85 vulnerabilities have been observed in all three groups. Some of vulnerabilities stored in CVE and NVD are marked as Reject (i.e. a vulnerability existence was not confirmed by further investigation), Disputed (i.e. software vendor does not approve vulnerability) or Deprecated (i.e. a vulnerability duplicates one of earlier).
All the rejected and deprecated vulnerabilities have been taken out of further consideration. In our work we investigated six popular enterprise operating systems (see. Table II).
206
TABLE I. AMOUNT OF VULNERABILITIES IN THE AGGREGATED SQL DATABASE
Type of product CPE ID
Amount of vulnerabilities
Vulnerability attribute Disputed Reject Deprecated
Operating system cpe:/o: … 14080 25 0 1
Application software cpe:/a: … 67956 405 0 2
Hardware cpe:/h: … 2794 9 0 0 Not defined no cpe-id 22242 3 934 0
Total: 107072 442 934 3 There were several reasons for choosing particular OSes
and their versions: popularity, belonging to different types (proprietary and open-source) and families (Windows, Unix/Linux, MacOS), and vendors. In particular, we took into consideration a series of reports [19, 20, 21] analysing OS market share for web- (where Linux-based OSs are dominating) and on-premises (mainly occupied by different versions of Microsoft Windows Server) servers.
Our intention was to study vulnerability statistics during a considerable period of time to capture general trends and to make sure that our results are well-grounded due to using a comprehensive dataset (CVE and NVD report statistically insufficient information regarding the most recent versions of OSes). This is why the versions of operating systems we chose (see Table II) were released in 2012 (all of them are still supported by manufacturers 7), and our vulnerability study covers 5 years preceding the end of 2016.
Despite the fact that the selected OSs have been now replaced by newer versions, our analysis shows that a significant number of new vulnerabilities are still disclosed in the chosen OSs versions. Unfortunately the majority of those vulnerabilities exists in the most recent operating systems too.
TABLE II. OPERATING SYSTEMS UNDER INVESTIGATION
Operating system Release date Linux kernel
version Ubuntu Server 12.04 26.04.2012 3.2.x Red Hat Enterprise Linux 6 10.11.2010 2.6.32 Novell Linux SUSE Enterprise Server 11 SP2 27.02.2012 3.0.x
Microsoft Windows Server 2012 R2 18.10.2012 - Apple MacOS Server 10.8 25.06.2012 - Oracle/Sun Solaris 11 09.11.2011 -
For instance, 100% of Apple MacOS Server’s
vulnerabilities reported by the NVD database during 2016 were discovered in both 10.8 and 10.11 versions. The last one was released on 21.03.2016. The percentages of the vulnerabilities, common for 2012 and 2016 (released on 26.09.2016) versions of Microsoft Windows Server, 6.x and 7.x (released on 10.06.2014, last updated on 01.08.2017) versions of Red Hat Enterprise Linux, and 12.4 and 16.4 (released on 21.04.2016) versions of Ubuntu Server are equal to 85%, 75% and 70% correspondingly.
It is also worth noting that Oracle Solaris 11.3 shares about 30 percentages of vulnerabilities discovered during 2016 with the Oracle Solaris 10.0 but none with the 11.0
version, analyzed in the paper. This can be explained if we assume that a significant piece of code of the most recent 11.3 version has been reused from the 10th version instead on the 11th.
The vulnerabilities of the earlier versions of some of those OSs were investigated by other authors in 2000 [15] and 2006 [13, 16, 17]. In this paper we continue and compare the vulnerability trends and examine four novel security aspects that can give important insights for OS vendors, administrators, system security engineers and ordinary users:
1) quantitative analysis and statistics comparison of disclosed and fixed vulnerabilities for different operating systems;
2) assessment and analyzing number of days-of-risk and forever-day vulnerabilities for each operating system during their operating cycle;
3) comparison of vulnerabilities severity and analyzing types of the most numerous vulnerabilities discovered in various OSs;
4) discovery of vulnerabilities that are common for two or more operating systems.
Thus, one of our intentions was to analyse how their security and vulnerability have been changed since that time.
Ubuntu Server, Red Hat Enterprise Linux and Novell Linux Enterprise Server are non-monolithic operating systems, thus in our study we also considered vulnerabilities in Linux kernels they use.
III. SOFTWARE VULNERABILITY LIFECYCLE Software vulnerability lifecycle has been discussed in a
number of research papers [11, 12, 22]. The authors of [23] proposed a formal model of the vulnerability lifecycle defining its major milestones. Most of the researchers and security analysts mark out 5 main events in a typical vulnerability lifecycle: (i) vulnerability creation; (ii) vulnerability discovery; (iii) vulnerability disclosure; (iv) patch availability; (v) patch installation.
Besides, exploits or computer viruses can be released in a wild during vulnerability lifecycle. An exploit is a sequence of commands, a software tool or even a specially generated data (e.g. an infected file) which automates making use of a vulnerability and allows even unskilled users to attack computer systems. Thus, exploit availability is an additional event sliding in between events (i) and (v).
Time intervals between the above mentioned events in the vulnerability life cycle have different risks of system exposure associated with them.
In particular, a special term days-of-risk [22] is used to define a period of an increased security risk between the time when a vulnerability is discovered or publicly disclosed to the time when a patch is applied to fix it.
Usually the periods of black, gray and white risk are marked out to indicate public awareness of the hazard and to qualify relative risks of exposure (see Fig. 1).
Risk value depends on vulnerability severity and other factors. It dramatically increases when an exploit is released in the wild and comes down when software vendor issues a patch to fix vulnerability. Application of anti-virus software, firewalls, intrusion detection systems mitigate risk value.
207
Figure 1. Vulnerability lifecycle and risk of exposure.
How public vulnerability disclosure (e.g. through CVE or NVD) affects system security is a question of great debates [24, 25]. On the one hand, malicious hackers can use publicly available information to attack affected computer systems increasing the risk of exposure.
On the other hand, forewarned users of vulnerable systems are forearmed, so they can take additional prevention actions to mitigate risk of exposure. Besides, public awareness of vulnerability usually pressures vendors to find a fix urgently.
In the paper we investigate gray risk (or post-disclosure risk) which defines the interval between vulnerability disclosure time and the date when the patch fixing vulnerability becomes available. We assume vulnerability disclosure time as a date when a vulnerability firstly appears in CVE with the corresponding CVE-ID assigned to it.
A time of patch issuing should be derived from vendor’s security bulletins. It is noteworthy that, according to [26] about 75% of vulnerability descriptions become available in the NVD database in seven days on average after they are mentioned in the vendor security bulletins. It means that NIST implements so called responsible disclosure model by giving stakeholders a time for the vulnerability to be patched before publishing the details in NVD [25]. Besides, it was also reported that different vendors have different median announcement gaps: Microsoft – 2 days, Oracle and Apple – 5 days, Linux – 10 days, Novell – approximately 12 days.
In our study we calculate days-of-gray-risk (DoGR) for a particular vulnerability as a period of time between a vulnerability is initially reported in CVE until it appears in NVD. In our work we decided not to take into account these extra few days, mentioned above, during which vulnerability descriptions propagate from vendor’s security bulletins to the NVD database. This can result in a slightly pessimistic estimate of the OS days-of-gray-risk, which, nevertheless, seems to be more secure than their underestimate.
IV. OPERATING SYSTEMS VULNERABILITY
A. Statistics of vulnerability discovery and elimination In this section we summarize statistics of vulnerabilities
discovered and disclosed in different operating systems since the 1st of January, 2012 and until the 31st of December, 2015 (see Table III). A number of vulnerabilities that had been observed but not fixed by the 1st of January, 2012 are reported as ‘Starting’.
TABLE III. OPERATING SYSTEMS VULNERABILITY STATISTICS
Y ea
r
Vulnerabilities
U bu
nt u
W in
do w
s
R ed
H at
N ov
el l
M ac
O S
So la
ri s
Starting number 15 0 46 26 0 13
20 12
Disclosed 64 10 31 32 2 47 Fixed 32 5 40 36 2 47 Avg.Sev. 5.21 8.31 5.07 5.13 3.20 4.37 Avg.DoGR 144 132 243 109 94 89
20 13
Disclosed 196 59 71 124 60 31 Fixed 202 51 86 127 59 32 Avg.Sev. 5.04 7.12 5.09 4.94 4.92 4.72 Avg.DoGR 109 131 119 99 113 81
20 14
Disclosed 188 64 72 129 40 37 Fixed 197 38 58 107 40 35 Avg.Sev. 5.55 6.71 6.79 5.83 7.13 5.08 Avg.DoGR 62 100 108 68 107 69
20 15
Disclosed 251 178 162 52 14 74 Fixed 223 156 146 80 15 71 Avg.Sev. 6.06 6.66 5.75 6.67 6.53 5.12 Avg.DoGR 79 126 101 133 83 58
20 16
Disclosed 194 95 107 21 48 0 Fixed 238 156 159 34 48 17 Avg.Sev. 5.52 6.79 6.99 7.53 6.78 - Avg.DoGR 89 131 93 114 138 -
T ot
al Disclosed 908 406 489 384 164 202
Fixed 892 406 489 384 164 202 Avg.Sev. 5.48 7.12 5.94 6.02 5.71 4.82 Avg.DoR 97 124 133 105 107 74
In the table we use the following short pseudonyms for
the operating systems under investigation: • Ubuntu – Ubuntu Server 12.04; • Red Hat – Red Hat Enterprise Linux 6; • Novell – Novell Linux Enterprise Server 11 SP2; • Windows – Microsoft Windows Server 2012 R2; • MacOS – Apple MacOS Server 10.8; • Solaris – Oracle Solaris 11. Red Hat Enterprise Linux 6 and Oracle Solaris 11 had
been released before the observed period (see Table II). Other operating systems (Ubuntu Server 12.04, Novell Linux Enterprise server 11 SP2, Microsoft Windows Server 2012 R2 and Apple Macintosh Server 10.8) were released in the beginning of 2012. It is worth mentioning that on the date of the official release some of those operating systems already had vulnerabilities that earlier had been discovered in previous OS versions. In particular, Ubuntu Server 12.04 inherited 15 of such vulnerabilities, Red Hat Enterprise Linux 6 – 46, Novell Linux Enterprise server 11 SP2 – 26 and Oracle Solaris 11 – 13 vulnerabilities.
208
Table III presents average vulnerability severity level (Avg.Sev.) by aggregating CVSS (Common Vulnerability Scoring System) metrics taken from NVD vulnerability database as well as reports average days-of-gray-risk value (Avg.DoGR). During 2012-2016 the largest number of vulnerabilities (908) was disclosed in Ubuntu, the least number (164) – in MacOS. The Red Hat, Windows and Novell operating systems occupy a middle position having 489, 406 and 384 vulnerabilities, respectively. It is worth to mention that no new vulnerabilities were discovered in Solaris during 2016 in contrast to all other operating systems.
A cumulative graph of vulnerabilities disclosed during 2012-2016 is depicted in Fig. 2.
The authors of [27] coin a new term ‘forever-day vulnerability’ defining a publicly disclosed vulnerability that has not been patched yet and can be hacked any time during system operation. It is in contrast to ‘zero-day vulnerabilities’ [23] which are publically undisclosed
vulnerabilities that some hackers have already discovered and can exploit.
Using both, the date of vulnerability disclosure and the date when the OS vendor issues a patch to fix it we can plot graphs of forever-day vulnerabilities showing how many of known (already disclosed publicly) but yet unfixed vulnerabilities existed every day during 2012-2016 in a particular operating system (see Fig. 3). Any operating system running with forever-day vulnerabilities is always vulnerable unless the software vendor issues a patch and a system administrator installs it.
As the vulnerability disclosure rate significantly higher than the rate of vulnerability patching, it can happen that an operating system contains up to several dozens of forever-day vulnerabilities at a time. Any of these vulnerabilities could be potentially exploited by hackers to attack the system. Fig. 3 shows that some operating systems have only few days (if any) of vulnerability free operation per year.
Figure 3. Forever-day vulnerabilities.
0
20
40
60
80
100
120
01.01.2012 01.01.2013 01.01.2014 01.01.2015 01.01.2016Date
RedHa t
Novell
Ubuntu
Sola ris
Windows
Ma cOS
Ubuntu
Novell
Windows
RedHat
Solaris MacOS
31.12.2016
Figure 2. Cumulative number of disclosed vulnerabilities.
0
100
200
300
400
500
600
700
800
900
01.01.2012 01.01.2013 01.01.2014 01.01.2015 01.01.2016Date
RedHa t Novell
Ubuntu Sola ris
Windows Ma cOS
Ubuntu
Novell Windows
RedHat
Solaris
MacOS
31.12.2016
209
For instance (see Table IV), during 2012-2016 OS Ubuntu has not had known vulnerability free days at all. Windows and Red Hat have had only 12 and 10 of such days respectively. It means that OS users and administrators should understand and accept potential risk of running vulnerable system. In addition, Table IV presents a detailed statistics of forever-day vulnerabilities for each operating system during 2012-2016. In average, Ubuntu OS had 47 of such vulnerabilities every day. For OS Windows, Red Hat and Novell this number is close to 30 vulnerabilities. MacOS and Solaris had the least average number of forever-day vulnerabilities (12 and 9 respectively).
B. Days-of-Gray-Risk for Operating Systems The number of disclosed vulnerabilities is often used as
the major indicator of software insecurity. However, taking into account how fast software vendors react on vulnerabilities discovered in their products is also important. Days-of-risk defines a period of time after a vulnerability is discovered/disclosed and until it is eliminated from a system after patch installation. It is also known as ‘window of- vulnerability’ or ‘days-of-recess’. In this study we do not take into account possible delays between the times when a vendor issues the patch and until a user or a system administrator actually installs it. Besides, in many cases it is impossible to identify when exactly a vulnerability was discovered.
TABLE IV. FOREVER DAY VULNERABILITIES STATISTICS
Y ea
r Forever day vulnerabilities
U bu
nt u
W in
do w
s
R ed
H at
N ov
el l
M ac
O S
So la
ri s
20 12
Min 14 2 21 9 0 6 Max 51 7 52 29 2 33 Average 23 4 35 19 1 14 Std. deviation 8 1 12 7 1 6
Vuln. free days 0 0 0 0 102 0
20 13
Min 32 1 17 14 0 4 Max 101 32 60 58 41 17 Average 65 18 36 31 19 10 Std. deviation 18 7 8 12 11 5
Vuln. free days 0 0 0 0 9 0
20 14
Min 12 2 10 11 1 1 Max 60 41 40 43 26 22 Average 38 16 21 22 13 8 Std. deviation 8 10 8 6 8 5
Vuln. free days 0 0 0 0 0 0
20 15
Min 28 9 17 10 0 2 Max 82 98 73 57 14 23 Average 48 51 37 24 4 12 Std. deviation 12 26 12 14 5 6
Vuln. free days 0 0 0 0 64 0
20 16
Min 16 0 0 0 0 0 Max 121 97 108 26 48 17 Average 59 50 41 12 18 1 Std. deviation 31 26 40 9 23 4
Vuln. free days 0 12 10 83 216 268
T ot
al
Min 12 0 0 0 0 0 Max 121 98 108 58 48 33 Average 47 29 34 21 12 9 Std. deviation 23 26 21 12 15 6
Vuln. free days 0 12 10 83 391 268
Figure 4. Operating systems average days-of-gray-risk
(dashed lines depict results reported in earlier studies [13-15, 17,22]).
Thus, in our study we focus on investigating days-of- gray-risk which is a time after a vulnerability is publicly disclosed through one of vulnerability databases and until a vendor issues a patch fixing it.
Days-of-gray-risk can be used to compare efforts that different vendors make to solve security issues and to deliver security updates fixing vulnerabilities. Fig. 4 shows how the average days-of-risk have been changing over the years for different operating systems. It includes information taken from Table III (for 2012–2016) as well as data reported for earlier versions of studied OSs in [13, 14, 15, 17, 28] by other researchers (depicted using dashed lines). For instance, according to [15] in 1999 Microsoft had an average of 16 days from vulnerability disclosure to patch. Red Hat spent only 11 days to fix vulnerabilities while Sun proved itself to be very slow solving security problems in 90 days on average.
In 2006, as reported in [13, 28], the days-of-gray-risk parameter for Microsoft Windows series of operating systems (Windows 2000 Professional and Server, Windows XP, Windows Server 2003) was estimated at 29 in average.
At the same time, it took Red Hat 107 days to deliver security updates for its Enterprise Linux 2.1, 3.0 and 4.0 while Sun spent 168 days to do the same for any Solaris version patched in 2006. In addition, it was estimated that Apple Mac OS X and Novell SUSE Linux Enterprise Server and Desktop (versions 8–10) had 46 and 74 days-of-gray- risk respectively.
0
25
50
75
100
125
150
175
200
225
250
1999 2005 2006 2010 2012 2013 2014 2015 2016
Average DoGR
Year
Ubuntu Windows RedHa t Novell Ma cOS Sola risSun Sola ris
Oracle Sola ris
210
Figure 4 shows that since 2013 there has been a tendency towards decreasing the number of days-of-gray-risks. During last two years average days-of-gray-risk for different operating systems varies between 75 and 140 days. Unfortunately, it still means that after vulnerability public disclosure users of affected operating system are remaining vulnerable and unprotected against potential hacker attacks during months and OS vendors know it.
Our work shows that the conclusion by Jeff Jones expressed in a series of his earlier blog posts [13, 22, 29] that Windows is the platform exposing users to risks for the shortest period of time as compared to other OSs is no longer correct. At the same time, we can see that since Oracle took ownership of Solaris OS in 2009 it has been reacting on new vulnerabilities much faster. Only Oracle Solaris demonstrates the steady drop of days-of-gray-risks over past years. Note here, that we did not estimate this parameter for Solaris in 2016 as there were no new vulnerabilities discovered.
In addition, we have built the probability density functions (see Fig. 5) defining the relative likelihood for the vulnerability to be patched on a particular day after it was disclosed, that is much more informative than the average days-of-gray-risk. It allows us to estimate a probability of issuing patch before the specified date or to define confidence intervals.
This shows, for example, that vulnerabilities in the Novell, Ubuntu and Solaris operating systems are usually fixed during the first 30 days after disclosure. The majority of Windows vulnerabilities are patched between 90 and 150 days while Apple usually spends 60..90 or 120..150 days to issue security updates for MacOS.
Figure 5. Probability density functions of OS days-of-gray-risk.
C. Vulnerability Severity Severity is an important characteristic quantifying the
impact of vulnerability on system’s security. NVD has adopted Common Vulnerability Scoring System (CVSS) to assign severity scores to software vulnerabilities.
CVSS are calculated based on several metrics that approximate ease of vulnerability exploitation (possibility of remote access, access complexity and need for authentication), vulnerability impact on confidentiality, integrity and availability and other factors [30]. Scores, as well as the overall severity rating are ranged from 0 to 10, with 10 being the most severe. Besides, vulnerability severity is divided into several qualitative ranges: Low [0.1..3.9], Medium [4.0..6.9], High [7.0..8.9] and Critical [9.0..10.0].
In addition to Table III which presents the average vulnerability severity, Table V shows how the number of vulnerabilities for different severity levels has been changing over the recent years for different operating systems.
TABLE V. OSS VULNERABILITY SEVERITY STATISTICS
Y ea
r Operating System
Number of vulnerabilities by severity score 1 2 3 4 5 6 7 8 9 10
20 12
Ubuntu 3 4 1 22 12 12 7 1 2 Windows 2 2 5 1 Red Hat 3 3 2 8 4 5 4 1 1 Novell 2 3 1 12 2 8 2 1 1 MacOS 1 1 Solaris 2 6 12 13 7 2 4 1
Total: 10 17 16 56 27 27 19 1 8 5
20 13
Ubuntu 18 14 7 68 26 32 25 1 2 3 Windows 1 6 6 5 30 9 2 Red Hat 10 3 1 20 10 18 7 2 Novell 14 11 6 42 11 19 17 1 3 MacOS 3 9 3 18 7 16 3 1 Solaris 3 3 18 1 2 3 1
Total: 48 40 18 172 61 92 85 1 13 11
20 14
Ubuntu 4 12 5 63 37 19 36 2 10 Windows 3 3 1 9 6 6 16 1 17 2 Red Hat 1 2 1 12 15 4 17 2 18 Novell 2 9 3 53 14 12 15 2 19 MacOS 1 1 3 2 21 3 1 8 Solaris 1 4 1 13 6 6 6
Total: 11 31 12 153 80 68 93 1 24 57
20 15
Ubuntu 5 9 13 51 43 38 60 2 13 17 Windows 3 25 2 11 9 12 66 48 2 Red Hat 3 9 13 36 23 25 42 2 9 Novell 7 1 9 5 3 10 5 12 MacOS 2 4 7 1 Solaris 5 4 6 20 14 9 16
Total: 18 54 35 127 94 91 201 2 68 41
20 16
Ubuntu 5 16 2 60 37 32 25 10 7 Windows 3 7 2 14 5 7 23 29 5 Red Hat 2 1 23 12 21 8 1 32 7 Novell 4 1 1 7 5 3 MacOS 3 11 8 1 6 17 2 Solaris
Total: 8 28 5 112 63 62 69 1 93 24
Su m
-t ot
al
Ubuntu 35 55 28 264 155 133 153 3 28 39 Windows 9 35 6 40 28 30 137 1 108 12 Red Hat 17 19 18 99 64 73 78 1 37 37 Novell 18 30 11 120 33 43 51 0 14 38 MacOS 5 14 4 33 17 42 19 0 19 11 Solaris 11 17 19 64 28 19 29 1 0 1
TOTAL: 95 170 86 620 325 340 467 6 206 138
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
30 60 90 120 150 180 210 240 270 300 330 360 >360
p(t)
t, days
Ubuntu
Windows
RedHa t
Novel
Ma cOS
Sola ris
211
Figure 6. OS vulnerabilities distribution by CVSS severity level.
Vulnerabilities in Oracle Solaris are the least critical. Their average severity is 4.82 (see Table III). The most severe vulnerabilities have been discovered in OS Microsoft Windows (average severity is 7.12) and Novel (the average severity is 6.02).
Fig. 6 shows the percentage of vulnerabilities for different severity levels. A number of critical vulnerabilities [9.0..10.0] disclosed in the Microsoft Windows OS is equal to almost 30% of total. At the same time, the quantity of such vulnerabilities for other operating systems is less than 19%. The fewest percentage of critical vulnerabilities (less than 1%) was observed in Solaris.
Figure 7. Average days-of-gray-risk depending on vulnerability severity.
In our research we have checked a widespread hypothesis that software vendors make more efforts on fixing the most critical vulnerabilities firstly. However, a diagram on Fig. 7 shows that days-of-risk metric does not actually depend on vulnerability severity. To some extent it seems to be true for the Red Hat operating system, however, developers of all the rest operating systems spend considerably more time in average on fixing critical vulnerabilities as compared to the least severe ones.
D. The Most Common OSs Vulnerabilities NVD classifies all vulnerabilities using the CWE
scheme. The Common Weakness Enumeration (CWE) is a formal list of software weakness types proposed by MITRE Corporation (https://cwe.mitre.org/).
The top ten vulnerability types discovered in different operating systems is presented in Table VI. About 70 percent of vulnerabilities in Oracle Solaris are marked as ‘Uncategorized’. Thus we took them out of consideration.
The most common OS vulnerabilities by the CWE types (sorted by prevalence) are:
CWE-119 – Improper restriction of operations within the bounds of a memory buffer using lacks of certain programming languages (often C and C++) that do not control bounds for the memory buffer that is being addressed. Vulnerabilities of the CWE-119 type usually cause arbitrary code execution, altering the intended control flow, reading protected information or system crash;
CWE-264 – Weaknesses and mistakes in permissions, privileges, and access controls;
CWE-20 – Improper input validation which may result in altered control flow, arbitrary code execution or illegal access to and control of resources;
CWE-200 – Information intentional or unintentional exposure to an actor that is not explicitly authorized to have access to that information;
CWE-399 – Improper management of system resources, e.g. memory allocation or reallocation;
CWE-189 – Numeric errors related to improper calculation or conversion of numbers;
CWE-362 – Concurrent code execution using shared resource with improper synchronization also knows as Race Condition;
CWE-310 – Cryptographic issues including missing encryption of sensitive data or key management errors;
CWE-94 – Improper control of code generation also known as Code Injection which often happens when software allows a user's input to contain code syntax.
CWE-59 – Improper link resolution before file access that allows an attacker to traverse the file system to unintended locations and read/overwrite the contents of unexpected files.
Figure 8 depicts the percentage of OS vulnerabilities distributed by different CWE types and shows how it had been changing during 2012-2016. The first three types of vulnerabilities (CWE119, 264 and 20) have been dominating over these years. In 2016 their contribution to the total number of discovered vulnerabilities has exceeded 70%.
0%
5%
10%
15%
20%
25%
30%
35%
1 2 3 4 5 6 7 8 9 10
Percentage of vulnerabilities
Vulnerability severity
Ubuntu
Windows
RedHa t
Novell
Ma cOS
Sola ris
0
50
100
150
200
250
300
1 2 3 4 5 6 7 8 9 10
Average DoGR
Vulnerability severity
Ubuntu Windows
RedHat Novell
MacOS Solaris
212
TABLE VI. THE MOST COMMON OS VULNERABILITY TYPES
Vulnerability Type
Percentage of vulnerabilities by CWE types Ubuntu Windows Red Hat Novell MacOS Solaris Total*
CWE-119 22.62 12.17 15.58 15.64 23.90 6.99 17.98 CWE-264 11.95 30.17 7.67 13.97 18.87 0.54 16.53 CWE-20 11.14 14.60 6.77 12.29 19.50 4.30 12.86 CWE-200 7.54 12.65 6.09 8.66 12.58 1.61 9.51 CWE-399 7.42 3.16 3.39 7.82 2.52 4.30 4.86 CWE-189 7.54 1.70 5.19 6.98 2.52 5.91 4.79 CWE-310 1.97 1.22 1.13 3.35 5.66 - 2.67 CWE-362 2.90 1.46 2.03 5.31 0.00 - 2.34 CWE-94 0.46 5.84 0.45 0.00 0.00 - 1.35 CWE-59 0.70 0.24 1.13 0.28 0.00 1.08 0.47 Others 8.12 11.44 9.93 2.79 6.92 3.76 7.84 Uncategorized 17.63 5.35 40.63 22.91 7.55 71.51 18.81
*Taking out of consideration Solaris vulnerabilities
CWE-119 weaknesses (e.g. CVE-2016-7277, CVE-2016- 4658 or CVE-2016-4598) often allow remote attackers to execute arbitrary code, read protected data or cause a denial of service via a crafted document viewed by victim on the infected web-page (e.g. .jpeg image or .xml file) or downloaded from the Internet and opened on his/her computer (e.g. .doc/.pdf documents or media files).
CWE-264 weaknesses usually mean implementation mistakes in permissions, privileges, and access control mechanisms. As a result, vulnerable software cannot properly identify session reuse (CVE-2016-3840), bypasses check for the access, read or write permissions (CVE-2016- 2416 or CVE-2016-6536) or relies on client-side authorization that can be easily bypassed via certain changes in local files (CVE-2015-5989).
Improper input validation (CWE-20) is a parent of other widespread vulnerability types including command injection (CWE-77), cross-site scripting (CWE-79) and SQL injection (CWE-89).
Figure 8. Number of vulnerabilities of different CWE types
distributed by years.
It is worth noting that numeric errors caused by incorrect calculation or conversion of numbers (CWE-189) affect not only system security but also system dependability and trustworthiness. This type of errors still remains quite typical, sometimes causing costly (or, even embarrassing) system failures. In particular, the authors of [31] found out that more than 700 research papers reporting on various strands of the genomic research over the 10-year period are riddled with errors due to an erroneous conversation of some gene symbols (e.g. MARCH1 or 2310009E13) to date and numbers in Excel spreadsheets.
E. Common and Group Vulnerabilities The most dangerous vulnerabilities are those discovered
in more than one operating system. A reason why the same vulnerability is discovered in several OSs is explained by using common vulnerable components (system libraries, third party software components, OS kernels, etc.).
More often group vulnerabilities are discovered in different releases of the same OS or in a family of related operating systems, e.g. BSD Unix (OpenBSD, FreeBSD, NetBSD) or Linux (Red Hat, CentOS, Novell, Ubuntu), etc.
However, sometimes hackers and security analysts discover vulnerabilities that are common for even different OS families. For example, the CVE-2008-4609 vulnerability caused denial-of-service attack for a variety of operating systems and their versions, including Linux, BSD Unix, Microsoft Windows, Cisco IOS and possibly many others [32, 33]. The vulnerability manipulated the state of Transmission Control Protocol (TCP) connections exploiting an algorithmic error in protocol implementation in various operating systems. A remote attacker was able to cause connection queue exhaustion by flags manipulation in the TCP header of crafted network packets sent to a victim- computer.
Figure 9 shows common vulnerabilities distributed between the Ubuntu, Novell and Red Hat operating systems during 2012-2016. Forty seven of them were disclosed in all three operating systems. Besides, there were 3 groups of vulnerabilities shared between the following pairs: Ubuntu and Novell – 241, Red Hat and Ubuntu – 65, and Novell and Red Hat – 36.
The numbers in brackets correspond to those vulnerabilities observed in Linux kernels (the NVD database distinguishes between vulnerabilities observed in Linux- based operating systems and Linux-kernels). Thus, Fig. 9 convincingly demonstrates that the largest number of common and group vulnerabilities shared between the Ubuntu, Novell and Red Hat operating systems are those discovered in the Linux kernels (versions 3.2.x, 3.0.x and 2.6.32) used by them. Red Hat and Ma OS only share the CVE-2013-1824 vulnerability in the PHP SOAP parser which allows a remote attacker to gain unauthorised access to arbitrary files of operating systems.
The number of vulnerabilities shared by two or more operating systems can be used as a measure of diversity between them [10]. Software diversity [34, 35, 36] has been used as a major fault and intrusion-tolerance mechanism to design safety-critical computer systems.
21 101 61
132 106
20 84 81
95 62
17
74 67
73 38
13
59 23
46
57
17
38 45
24
8
20 45 25
36 9
5 19 20 9
6 6 24 4
11 32 6 14 7
11 4 6 4
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2012 2013 2014 2015 2016
CWE-59
CWE-94
CWE-310
CWE-362
CWE-189
CWE-399
CWE-200
CWE-20
CWE-264
CWE-119
213
Figure 9. Number of individual, group and common vulnerabilities shared
by Linux family of operating systems (Ubuntu, Novell and Red Hat).
Thus, vulnerability databases (the NVD database in particular) can help in determining the most diverse software products.
Our analysis shows that the results reported in [10] should be further verified as the authors may not have considered common and group vulnerabilities observed in Linux kernels.
V. CONCLUSION AND LESSONS LEARNT The paper presents a retrospective vulnerability analysis
of popular enterprise operating systems: Ubuntu Server 12.04, Red Hat Enterprise Linux 6, Novell Linux Enterprise Server 11 SP2, Microsoft Windows Server 2012 R2, Apple MacOS Server 10.8 and Oracle Sun Solaris 11.
The statistics discussed in the paper have been collected by querying a specially developed MySQL database which aggregates the XML data feeds provided by the CVE (www.cve.mitre.org) and NVD (web.nvd.nist.gov) vulnerability repositories.
A significant growth of the total number of vulnerabilities discovered in modern operating systems as well as the general tendency toward increasing their severity demonstrate the serious security challenges and risks that OS developers and users face. It is very important to understand that the crucial parameters affecting system security are not only the total number of vulnerabilities disclosed in a particular software product and their severity but also, so called, days-of-risk which show how fast software vendors issue patches fixing disclosed vulnerabilities. Our study shows that the average days-of-risk for the investigated operating systems varies from 83 days for Oracle Solaris up to 135 days for Red Hat.
It is worrying that as the paper shows, the rate with which software developers issue security updates in general does not depend on vulnerability severity. Average days-of- gray-risk for the most critical vulnerabilities remains even higher than the one calculated for vulnerability of the lowest severity (117 vs 100 days). This uncovers worrying shortcomings in the policies for developing security updates adopted by OS vendors, as well as, in the maintenance management processes they run.
The number of OS vulnerabilities that remain unpatched during a considerable period of time is growing. The increase of days-of-gray-risk and the raise of a number of forever-day vulnerabilities threaten security and dependability of computer systems.
Users and system administrators should be aware of the fact that the operating systems they use typically have up to several dozens of known but yet unfixed vulnerabilities (our study reports 25 forever-day vulnerabilities in average for the investigated operating systems for every day during 2012-2016).
It is worth noting that since the beginning of 2016 the NVD database has reported 31 vulnerabilities (not only in the studied OSs) as those that become possible because of an incorrect fix of some previous vulnerabilities. For instance, the CVE-2016-2550 vulnerability in Linux kernel before 4.5, causing denial of service, was introduced as a result of the incorrect fix for the CVE-2013-4312 vulnerability. Besides, 196 vulnerabilities have been reported as the ones that existed because of incomplete fixes (e.g. CVE-2016-10088).
This means that waiting for and fully relaying on security updates provided by software vendors is not a panacea. The implementation of the defence-in-depth principle by applying the layered security mechanisms to cope with the forever-day vulnerabilities is of a great importance for modern computer systems.
One specific aspect that the paper studies is the vulnerabilities that were discovered in more than one operating systems. Such vulnerabilities that are common for different operating systems and even different OS families can lead to large-scale hacker attacks and virus epidemics. They also seriously complicate the development of intrusion- tolerant computer systems based on OS diversity.
The results presented in the paper show that numerous vulnerabilities in operating systems cause significant security threats. This calls for implementing complex and in-depth defense mechanisms that incorporate antivirus software, firewalls, security scanners, intrusion detection systems and other solutions in a way that makes them aware of the recent and current OS vulnerabilities.
Hopefully, this work will make OS vendors to pay much more attention to improving security of their products, which might require significant changes in their software development and maintenance processes. Our experimental work supports our claim that decreasing days-of-risk and reducing a number of forever-day vulnerabilities is one of the main challenges in building secure operating systems.
We believe that the methodology presented in this paper can be used to create a public service reporting forever-day vulnerabilities (and other crucial vulnerability statistics) observed daily in each OS and other products to help system administrators in understanding security risks and in taking appropriate protective actions.
ACKNOWLEDGMENTS Alexander Romanovsky is partially supported by the
EPSRC/UK STRATA platform grant. Anatoliy Gorbenko and Olga Tarasyuk are partially supported by the TEMPUS SEREIN grant.
Ubuntu Novell
Red Hat
544 50
331
65 (37)
241 (216)
57 (37) 36 (14)
214
REFERENCES
[1] B. Clark, “Hackers take hospital offline, demand $3.6m ransom,” [Online]. Available: http://thenextweb.com/insider/2016/02/15/ hackers-take-hospital-offline-demand-3-6m-ransom/.
[2] C. Williams, “Passengers ride free on SF Muni subway after ransomware infects network, demands $73k,” [Online]. Available: http://www.theregister.co.uk/2016/11/27/san_francisco_muni_ranso mware/.
[3] MITRE Corporation, “Common Vulnerabilities and Exposures. Terminology,” [Online]. Available: https://cve.mitre.org/ about/terminology.html.
[4] Microsoft Inc., “Description of Software Update Services and Windows Server Update Services changes in content for 2016,” 2016. [Online]. Available: https://support.microsoft.com/en- us/help/3215781/description-of-software-update-services-and- windows-server-update-services-changes-in-content-for-2016.
[5] National Vulnerability Database, “Vulnerability Summary for CVE- 2016-7256,” [Online]. Available: https://web.nvd.nist.gov/view/ vuln/detail?vulnId=CVE-2016-7256.
[6] MITRE Inc., “CVE-2016-7256,” [Online]. Available: http://www.cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016- 7256.
[7] Microsoft Inc., “Microsoft Security Bulletin MS16-132 - Critical,” [Online]. Available: https://technet.microsoft.com/en-us/library/ security/ms16-132.aspx.
[8] National Vulnerability Database, “Vulnerability Summary for CVE- 2014-0160,” [Online]. Available: https://web.nvd.nist.gov/view/ vuln/detail?vulnId=CVE-2014-0160.
[9] Z. Schreuders, T. McGill and C. Payne, “The state of the art of application restrictions and sandboxes: A survey of application- oriented access controls and their shortfalls,” Computers and Security, vol. 32, pp. 219-241, 2013.
[10] M. Garcia, A. Bessani, I. Gashi, N. Neves and R. Obelheiro, “OS Diversity for Intrusion Tolerance: Myth or Reality?,” in IEEE/IFIP 41st Int. Conf. on Dependable Systems & Networks (DSN’2011), 2011.
[11] S. Frei, M. May, U. Fiedler and B. Plattner, “Large-scale vulnerability analysis,” in SIGCOMM Workshop on Large-Scale Attack Defense, 2006.
[12] M. Shahzad, M. Zubair Shafiq and A. Liu, “A large scale exploratory analysis of software vulnerability life cycles,” in 34th Int. Conf. on Software Engineering (ICSE '12), 2012.
[13] J. Jones, “Days-of-risk in 2006: Linux, Mac OS X, Solaris and Windows,” 2006. [Online]. Available: http://www.csoonline.com/article/2136935/data-protection/days-of- risk-in-2006---linux--mac-os-x--solaris-and-windows.html.
[14] A. Gorbenko, O. Tarasyuk, V. Kharchenko and A. Romanovsky, “Using Diversity in Cloud-Based Deployment Environment to Avoid Intrusions,” Software Engineering for Resilient Systems, no. LNCS 6968, p. 145–155, 2011.
[15] J. Reavis, “Linux vs. Microsoft: Who Solves Security Problems Faster?,” 2000. [Online]. Available: http://www.reavis.org/ research/solve.shtml.
[16] P. Edmonds, “When It Comes to Protection from Vulnerabilities, Process Trumps “Many Eyes”,” 2007. [Online]. Available: https://technet.microsoft.com/en-us/library/cc512608.aspx.
[17] A. Patrizio, “Report Says Windows Gets The Fastest Repairs,” 2007. [Online]. Available: http://www.internetnews.com/security/ article.php/3667201.
[18] OSVDB, “Rebuttal: Dark Reading’s “9” Sources for Tracking New Vulnerabilities,” 2016. [Online]. Available: https://blog.osvdb.org/2016/10/26/rebuttal-dark-readings-9-sources- for-tracking-new-vulnerabilities/.
[19] M. Cheung, “Market Share Analysis: Server Operating Systems, Worldwide, 2015: Gartner report.,” 2016. [Online]. Available: https://www.gartner.com/doc/3326217/market-share-analysis- server-operating.
[20] P. Tsai, “Server Virtualization and OS Trends,” 2016. [Online]. Available: https://community.spiceworks.com/networking/articles/2462-server- virtualization-and-os-trends.
[21] W3Techs, “Usage of operating systems for websites,” 2017. [Online]. Available: https://w3techs.com/technologies/report/operating_system.
[22] J. Jones, “Basic Guide to Days of Risk,” 2007. [Online]. Available: http://www.csoonline.com/article/2136934/data-protection/basic- guide-to-days-of-risk.html.
[23] L. Bilge and T. Dumitras, “Before we knew it: An empirical study of zero-day attacks in the real world,” in ACM Conference on Computer and Communications Security, Raleigh, NC, 2012.
[24] B. Barth, “Lag between a bug's first disclosure and its inclusion in national database can put companies at risk”,” 2017. [Online]. Available: https://www.scmagazine.com/lag-between-a-bugs-first- disclosure-and-its-inclusion-in-national-database-can-put- companies-at-risk/article/667398/.
[25] A. Hahn and M. Govindarasu, “Cyber vulnerability disclosure policies for the smart grid,” in IEEE Power and Energy Society General Meeting, San Diego, USA, 2012.
[26] B. Ladd, “The Race Between Security Professionals and Adversaries,” 2017. [Online]. Available: https://www.recordedfuture.com/ vulnerability-disclosure-delay/.
[27] D. Goodin, “Rise of “forever day” bugs in industrial systems threatens critical infrastructure,” 2012. [Online]. Available: http://arstechnica.com/business/2012/04/rise-of-ics-forever-day- vulnerabiliities-threaten-critical-infrastructure/.
[28] M. Oiaga, “Recount: Windows Still Safest, Tops Mac OS X, Linux and Sun Solaris. But are statistics a true measure of security?,” 2007. [Online]. Available: http://news.softpedia.com/news/Recount- Windows-Still-Safest-Tops-Mac-OS-X-Linux-and-Sun-Solaris- 57433.shtml.
[29] J. Jones, “2006 Client OS Days of Risk,” 2007. [Online]. Available: https://blogs.microsoft.com/microsoftsecure/2007/06/18/2006- client-os-days-of-risk/.
[30] Forum of Incident Response and Security Teams, “Common Vulnerability Scoring System, V3 Development Update,” 2015. [Online]. Available: https://www.first.org/cvss.
[31] M. Ziemann, Y. Eren and A. El-Osta, “Gene name errors are widespread in the scientific literature,” Genome Biology, vol. 17, no. 177, 2016.
[32] National Vulnerability Database, “Vulnerability Summary for CVE- 2008-4609,” 2008. [Online]. Available: https://web.nvd.nist.gov/ view/vuln/detail?vulnId=CVE-2008-4609.
[33] Cisco Systems, “TCP State Manipulation Denial of Service Vulnerabilities in Multiple Cisco Products,” 2009. [Online]. Available: https://tools.cisco.com/security/center/content/ CiscoSecurityAdvisory/cisco-sa-20090908-tcp24.
[34] B. Randell, “System Structure for Software Fault Tolerance,” IEEE Transactions on Software Engineering, vol. 1, no. 2, pp. 221-232 , 1975.
[35] A. Avizienis, “The N-Version Approach to Fault-Tolerant Software,” IEEE Transactions on Software Engineering , vol. 11, no. 12, pp. 1491-1501, 1985.
[36] B. Littlewood and L. Strigini, “Redundancy and diversity in security,” in 9th European Symposium on Research Computer Security ( ESORICS'2004), LNCS 3193, 2004.
215
PID5889355.pdf
XXX-X-XXXX-XXXX-X/XX/$XX.00 ©20XX IEEE
Vulnerability Analysis and Modeling Mikeera Brown, Michael Joseph, Shawnoah Pollock, Khaled Elleithy
Computer Science and Engineering Department University of Bridgeport
Bridgeport, CT, USA
[email protected], [email protected], [email protected], [email protected],
Wafa Elmannai
Manhattan College Electrical and Computer Engineering Department
Bridgeport, CT, USA
As time continues to progress, the world continues to evolve into an advanced technological environment. One can argue that everyone interacts with some technological element on a daily basis. Information technology and computer science are becoming arguably the two fields that will always be in demand because of the emphasis that society places on technology aspects. However, nothing is one hundred percent untouchable. One of the significant risks of computer systems is the lack of security whether that is attributed to the system vulnerabilities or too sophisticated intrusion techniques. The software security vulnerability is detrimental to the user as well as the system because of its inability to protect confidentiality within the system. Analysis from a national database displays how software vulnerabilities are categorized and how much of an impact they play amongst computer systems. To help in the prevention of such attacks, there are proposed models that are designed and implemented to detect these software vulnerabilities.
Keywords—IOT, Vulnerability Analysis, Vulnerability Modeling
I. INTRODUCTION The computer security vulnerability is defined as "a flaw or
weakness in a system's design, implementation, or operation and management that could be exploited to violate the system's security policy." As interconnected devices play a significant role in our lives, businesses and governments, the exploitation of these flaws or weakness are becoming more and more valuable. As a result, attacks from an individual, to state sponsor organization with unlimited resources are more common in today's society. Our inability to prevent these attacks is due to three simple facts:
1. Modern computer systems are very complex.
2. Responsibility to secure systems is distributed among many parties, often with conflicting interests.
3. Threats can come from anyone, anywhere and anytime.
Vulnerability modeling is a method of improving network security by defining vulnerabilities, modeling their behavior, and through these activities developing methods of mitigation, prevention, and recovery [6] [7]. Despite the availability of analysis tools, vulnerability analysis, is still, almost exclusive, the purview of security experts [7].
Through the use of modern analysis and modeling techniques, future system administrators will be armed against the ever-growing field of threats in the information age. Better
analysis methods and modeling techniques will allow for more effective security policies, more accurate threat prediction, detection and response.
A. Problem Identification To analyze vulnerability in our computer systems, we must
consider all parties involved from hardware and software development, that includes cryptographic algorithms, protocols, operating systems, and applications software. The sheer number of software vulnerabilities exploited by attackers each year, as reported by the National Vulnerabilities Database (NVD) (see Table 1) [5], indicate we have a problem. However, a large percentage are attributed to software flaws that could have been prevented if robust software verification methods were used. Formal software verification is tedious, takes time and is expensive, so this step in our software development cycle is often overlooked. As a result, we have cryptographic algorithms with no formal mathematical proof or has not gone through any formal verification methods or its software implementation has not been validated (i.e., correctly implement the requirement). We also have operating systems with reported flaws that can be exploited by hackers, without a fix for weeks, known as “days-of-risk” [2].
ΤΑΒΛΕ Ι. CATEGORIES OF VULNERABILITIES REPORTED IN 2017 [5]
Category Percentage
Buffer Error 17.00%
Cross-Site Scripting 9.99%
Improper Access Control 9.34%
Information Leak/Disclosure 9.73%
Input Validation 6.62%
Permissions Privileges and Access Control 6.62%
Resource Management Error 8.18%
SQL Injection 2.63%
Security Features 3.41%
Command Injection 1.63%
Cross Site Request Forgery 1.42%
Path Traversal 2.12%
Other 26.12%
We must also consider system security and network engineers responsible for deploying and maintaining these systems. Merely keeping up with security patches is not enough. Pro-active defensive steps in understanding and identifying security weakness in the network are essential. While many attack modeling techniques are available to help determine the potential point of attacks, we will consider "The Kill Chain" technique. It is described by the Department of Defense as a seven steps attack: Reconnaissance, Weaponization, Delivery, Exploitation, Installation, Command and control, Action on objectives [3, 4]. Last but not least, the education of the end user ordinary users. Social engineering is one of the methods used by hackers to gain access to a targeted device. This step known in the kill chain technique as "Delivery" is the first opportunity to kill the attacks, provided the user is educated [3].
II. RELATED WORK There are many publications on vulnerability analysis that
propose some form of formal methods for requirements, coding, and testing, while others focus solely on modeling. Emeka and Liu [9] propose not only to use Structured Object- oriented Formal Language (SOFL) to specify system requirements but to also "extract security vulnerabilities from SOFL specifications" [9]. By extracting and analyzing security vulnerabilities at the requirements level, designers can correct flaws before they make it into the code. Barthe [8] advocates using Formal methods to develop cryptographic software and automated analysis tools for validation and verification. Gaudel [10] explores the benefit and limitation of formal requirement based testing. He points out that requirement based testing only catches errors that are stated explicitly in the specifications. So he proposes even more validation and verification [10].
Regarding modeling, the usage of kill chains in response
to threats is incomparable. The Kill Chain is generally favored because of its long exploitation tenure with the United States Department of Defense. However, other sources of modeling are gaining acknowledgment. The Diamond Model is achieving prevalence because of its simplicity and condensed methodology. [4] Additionally, the utilization of an Attack Graph/Tree's technique in searching for particular algorithms is what classifies this model into a reliable mechanism [4].
III. THE PROPOSAL The strategy outlined in this paper is threefold: adopting
formal methods in hardware and software development, maintain continuous readiness through modeling, and educate the user on the dos and don't on cyber-security. By focusing on all aspects simultaneously, from the quality of the software being used, network configuration and preparedness to the individual user, we address not just the symptom but also the underlying cause of vulnerabilities. It is understood that anticipating and preventing intrusions using vulnerability modeling technique is essential and useful. However, network administrators remain defenseless against "zero-day" attacks. These types of attacks can only be prevented through the
adoption of formal methods in software development. The same can be said for the security threat an uninformed or careless user poses.
A. Formal Methods Vulnerabilities are hardware or software defects that are
exploited by hackers. At the heart of any security, infrastructure is cryptographic algorithms, communication protocols, and operating systems. Any flaw in these components exposes the system to potential attacks. Therefore the goal should be to find and eliminate any vulnerability during the development phase. The aerospace industry has successfully developed relatively bug-free software for the past 40 years. This is due in part to the adoption of formal methods for software development. These flight critical software are designated Level A by the FAA and as such are required to have a reliability level of 10-9. That means to follow all the software development activities, which include, requirement specifications, preliminary and detail design, coding, peer-review, validation testing, verification testing, and unit testing.
Requirement Specification is an essential step in the development process. However, it is often overlooked, especially in the era of rapid prototyping. Using a formal method for requirement specification can help "extract security vulnerabilities “in an early stage and prevent those vulnerabilities from being incorporated into the code [9]. Requirement traceability is used to eliminate or deter dead code (procedures or functions that is not called by any other module) and ensure each requirement is implemented and tested. Last but not least, formal requirement specification can aide in the development of more meaningful and complete test cases.
Design and Coding standards are required by a formal method. All procedures and functions must trace up to a requirement specification, or a derive requirement. This reduces the amount of code to be tested and reduces potential vulnerabilities. Coding standards also ensure that all data types are defined, and developers must follow the rules on initializing and reusing variables, pointers, etc… According to NVD statistics (see Table 1), over 17% of vulnerabilities exploited by hackers in 2017 are related to memory or buffer. These types of vulnerabilities can be prevented through better design and coding standards.
Software Testing, in general, is tedious, costly and
extends release date or time to market. However, it is nonetheless an essential part of the software development process.
Requirement Based Testing is a black box testing witch
answer a simple question: is the requirement implemented adequately. It is not enough, for a cryptographic algorithm, for example, to have mathematical proof; it also must be
implemented (convert into code) properly. That can only be guaranteed through the use of formal verification testing.
Unit Based Testing is another significant activity in the
software development process. As Gaudel [10] pointed out, requirement based testing "are relevant for detecting those faults that cause deviations concerning this specification, no more. For other kinds of fault, other validation and verification methods must be used." That is the objective of unit-based testing. It exhaustively tests all the paths of the software as well as all the inputs and outputs across the entire domain. For example, if a function expects an input "A," its type would be specified, and its range would be finite, per coding standard (i.e., type integer, range from 0 to 99). During unit testing, that function would be called, with each of the expected input (0 … 99) and its output compared against the expected output. The function would also be called with samples of input outside the expected domain (i.e., 101, 200, etc…) to evaluate its response when injected with illegal inputs. Such a test would eliminate about 10.88% of vulnerabilities reported by NVD in 2017 (i.e., Input Validation, SQL Injection, and Command Injection see Table 1). Note that any tool used in formal software development is required to be certified.
B. Vulnerability Modeling Using the Kill Chain Technique Traditional network security mechanisms such as intrusion
detection systems are considered insufficient in tracking and hindering the interference of the likes of modern-day adversaries. The benefits of capitalizing on vulnerability modeling can disrupt an adversary’s intention to compromise data or other resources of an organization. Arguably, the most ubiquitous method of attack modeling is the Kill Chain Technique because of its success rate. This technique is designated to target and attack an adversary with its seven stages ordered chain with the anticipation of employing a countermeasure. [4] A disruption or deficiency within the "chain" of this systematic technique would result in the entire process being corrupted. In [17], seven Kill Chain Technique stages are as defined.
The victim of an attack would not have the advantage of knowing precisely when they are being attacked. They can only anticipate and become aware of its danger after the "Delivery" phase of the kill chain and kill the process then. However, once an adversary reaches the final stage, the victim would have no other possibility but to succumb to having their data exploited. Although, the kill chain technique offers a guideline to hindering attacks. One can only successfully identify an incoming attack through user education.[4]
C. Education & Training Regardless of technologies employed to secure assets
against attack, the most common vulnerability vector is through improper user actions. These actions may be intentional; however, the more common case is through simple human error. [14] The number of incidents of human
error based attacks has increased significantly over the past five years. The prevalence of cloud server systems and attackers awareness of misconfigurations in these systems has led to a 424 percent increase in misconfigured cloud server attacks just between 2016 and 2017 alone [16]. Additionally, the use of more sophisticated attacks through phishing, primarily through corporate email remains a constant threat. Similarly, weak user passwords or unsecured personal devices form an important threat vector, especially as attackers have access to more advanced methods and faster computers to employ them on [16].
Education on network vulnerabilities can be targeted to
the user or IT personnel, which can be further broken down into Attack, Forensic and Defense [12] [13]. The four proposed categories of training and education are outlined below:
Attack-Oriented Training for IT – This training focuses on using the tools that attackers actually use in real life to reproduce realistic threats. The goal of this training is to understand the attacks from the inside out to better prepare system administrators to prevent and protect from them. This training also includes instruction on penetration testing which can be used to assess the vulnerability of a system [12].
Defense-Oriented Training for IT – This training focuses
on implementing protection methods for current threats. This training will likely real-world case studies such that past attacks do not repeat and similar attacks won't be useful in the future [12].
Forensics-Oriented Training for IT – This training
focuses on being able to observe real-world system behavior and determine the precise nature and vector of an attack. This training would also include real-world case studies. The goal would be to prepare system administrators to be able to detect real-world threats and coordinate a timely response [12].
End-User Education – The education of end-users is a
large part of this arm of the strategy. Users must be aware of the threats that exist by maintaining basic cyber literacy [15]. This also includes keeping an operational understanding of attack defense and countermeasures employed so that they are not rendered useless by user error. Users must be aware not only of the specific threats and the counters to those threats, but they also need to understand the underlying philosophy of network security. When a new type of attack is detected, a technological solution can be developed to safeguard from it. Just as soon as a technical solution is employed, attackers change their strategy and maneuver around it. A well-educated user population can help to make these defense workarounds much more difficult for attackers. To meet these challenges, the end-user education should be a mix of best practices (or policy) as well as vulnerability fundamentals and current trends.
Education and training can take many forms depending on the skill level of the team, the environment in which they work and the budget they have to spend on network security [12]. Education and training programs should be planned and executed holistically through a defined life-cycle. When new threats are detected, they should be reproduced and their signatures and effects analyzed (attack training). These new threats should be analyzed in-depth to determine their characteristics (forensic training). Then defense measures should be employed to defend against repeat or similar attack vectors. (defense training) Finally, as a result of this, expanded end-user education should be employed to make end-users aware of the threat and to keep user errors from nullifying defense measures [12] [13] [15].
Effective education and training must be planned to
ensure that the right participants are receiving the right training at the right time. With the busy schedules of most non-IT end-users, and with their main focus being away from network security, planning education and training activities must be done with high impact and efficiency. Additionally, organizations must ensure that training is repeated with a frequency appropriate to the topic [12].
IV. CASE STUDY To evaluate the effectiveness of the proposal, a case study
of recently reported vulnerabilities in Baby Monitors is used. A typical baby monitor has a sensor, which is placed on or near the baby and is responsible for reporting, wirelessly via blue-tooth or Wi-Fi, information about the baby. It may be audio, video, health monitor sensor or a combination thereof. The sensor's signal is sent to a based station that is connected to the home's network. The role of the base station is to interpret the information from the sensor(s) and report the status to interested parties (parents and/or manufacturer) which usually involves accessing the internet. That makes a baby monitor a typical IoT device.
A. Vulnerability Analysis Using Formal Methods
1) Owlet Baby Monitor
The main issue, in this case, is the base station providing unsecured access to the sensor; as a result, it opens the door to attackers. As reported in [18], communication between the base station and the manufacturer, the user's mobile phone and the manufacturer are all encrypted. Therefore there is a requirement specification that deals with security. Consequently, it is likely that communication between the base station and the sensor is left unsecured on purpose. To understand why to rely upon the fact the sensor attached to the baby's socks is tiny and has the limited processing power and battery life. The second issue is the base station creating its own unsecured Wi-Fi hot spot.
Requirement Validation Peer-Review: A requirement
review would have caught the missing requirements to ensure
all communication is secured. If the security requirements for sensors were relaxed because of hardware limitation, possible remedies should have been suggested (i.e., the use of light weigh security protocol, etc.). There is no justification why the base station would allow unsecured Wi-Fi access to non- sensor devices.
2) iBaby M6
The main issue, in this case, is the manufacturer allowing access to unauthorized devices. The requirement specification does require users to be authenticated and required specific device ID. However, what the requirement failed to specify or the code failed to implement is the association between a device and its owner. Authentication in the device itself should have required more than just device ID [19].
Requirement Validation Peer-Review: This review would indicate the requirements should expect not only that user be authenticated and the device id number be provided, but also that the device id number be validated. Validation, in this case, means the device exists, and the request comes from its owner. Moreover, the device would require authentication as well.
Code Peer-Review and Testing: Assuming the
requirement exists, traceability review and requirement based testing would have found that some of the requirements (associate device with its owner, and authentication at the device) were not implemented.
3) iBaby M3S
The main issue, in this case, is threefold: 1) the device was shipped with hard-coded administrator privilege. 2) Telnet service was installed and running. 3) The user name and password are trivial [19].
While this is a security flaw, however, it does not appear to be a mistake. If this backdoor was installed by the manufacturer for easy access, as such formal method would have flagged it as a security risk.
Code Peer-Review: Assuming the backdoor user name
and password were added by mistake, then this flaw would have been discovered during code review.
4) Philips In Sight
The issue, in this case, is the presence of remote access software (i.e., Telnet and web services) along with hard-coded credentials [19]. This vulnerability was integrated with the software by the manufacturer for easy access to the device. We came to this conclusion because, when the manufacturer was made aware of the existence of the backdoor credentials; they issue a patch where the clear text credentials were merely replaced with another, but encrypted user name and password. Formal methods can only prevent mistakes, not intentional actions.
The company also provides a remote video streaming service via a proxy server. While the communication from the user to the proxy required authentication, the connection between the proxy and the device does not have such a requirement. The company did try to conceal the port number of the users' web service address but did not offer sufficient protection to the camera's web service [19].
Requirement Specification Validation Review: Assessing
whether adequate security measures were taken to secure the camera's web services from unauthorized access is the job of the requirement specification validation review. At the very least, access to the camera's web service should require authentication. It should not assume request will only come from the proxy server.
5) Summer Baby Zoom
The main issue in the first instance is the fact, the user management for a camera is done at the company's site and required only the cameras ID as a credential [19]. That suggest, either the camera does not have any authentication mechanism in place, or admin privileges are hard-coded into the camera’s firmware.
Since this vulnerability is reported from the perspective of
the web services provided by the company (MySnapCam.com), we are unable to access the actual security issue with the camera itself.
6) Lens Peek-a-View, Gynoii, and TRENDnet WiFi Baby Cam TV-IP743SIC
These devices have the same issue as #3. The main, in this case, is threefold: 1) the device was shipped with hard- coded administrator privilege. 2) Telnet services were installed and running. 3) The user name and password are trivial. [19]
While this is a security flaw, however, they were added on purpose to simplify setup and provide the company with easy access to the devices. The simplify setup for the user also simplify the hacker's job as well, because these devices use Wi-Fi and have Telnet, Web services running and trivial credentials.
B. Vulnerability Analysis Using the Kill Chain Technique Typically, the Kill Chain technique would be applied in a
case where a physical system is prevalent. These steps rely on the downloading and installation of malware to be effective. In this case study, the downfall of the assessed IoT baby monitors has resulted from manufactured deficiencies or lack of user knowledge. Therefore, an analysis using the Kill Chain Technique would be inconsequential.
C. Vulnerability Analysis Using Education & Training While many IoT vulnerabilities can and should be
corrected or mitigated at the manufacturer level, the reality is
that many household IoT devices are commoditized [19] and built on general purpose hardware with little or even no communications security. Looking at the cyber domain as a whole, over 90% of issues begin with some form of user error [15]. This figure is expected to be only higher in IoT applications due to the lack of a standard interface and the use of inferior security features. The following analysis shows how user education and training may have been effective in eliminating or at least mitigating the vulnerabilities of IoT Baby Monitors.
1) Owlet Baby Monitor
In this incident, the baby monitor can quickly, without a password, be taken off the user's wifi, added to a new wifi network and have all its alerts changed. This means, for a moderately competent hacker, dropping in on a neighbor's baby monitor would be easy to do. As it happens, this flaw was discovered by a very well educated user, who happens to be a computer security researcher. [18] For parent's with less education in this field, there isn't much that can be done to make this device more secure except to request the manufacturer to patch the security flaws. This is an instance when performing a simple internet search may turn up articles like [18] and help to guide parents in avoiding monitors, or other IoT devices, with inadequate security performance.
2) iBaby M6
In this incident, a direct object reference vulnerability allowed an authenticated user on the iBaby website to access video from other accounts by merely sending a hexadecimal request number. Again, this type of issue falls to the manufacturer to patch. Users would need to keep informed of these threats and request the manufacturer to update the firmware, etc.
3) iBaby M3S, Philips In.Sight B120/37, lens laboratories, Gynoii & TRENDnet
These devices have hardcoded credentials that can easily be guessed, e.g., username = admin/password = admin [18]. User's who are aware of the vulnerability generated by having these credentials available can avoid the issue by preventing the use of this device until the manufacturer issues a firmware update eliminating these default login credentials. This is a familiar source of vulnerability for IoT and other devices. Specifically, when a manufacturer includes a hardcoded means to access the device if the user has lost their credentials.
4) Philips In.Sight B120/37
Another issue with this Philips device is that it uses insecure transport methods for video streaming. Unlike the default password issue, which should be detected, users would have to investigate to learn of this vulnerability. Just stated, if the user expects that the video stream is secure, they should inquire about what type of security is used. In this case,
through some investigation, it could be learned the stream is unsecured allowing for a more informed product decision. Like with many IoT vulnerabilities, fixing this issue falls to the manufacturer.
5) Summer Infant
The Summer Infant device had several authentication issues. First, the means to add an account to a camera, without authentication, existed. Additionally, regular users were able to access admin privileges. The combination of these leaves the device very vulnerable to attack [19]. These issues are only a threat when the camera is connected to the broader internet. Users should use the camera on the local network only and ensure their network is secured by the firewall. They should be able to accurately handle their firewall to block the camera from anything outside the LAN. This is an example where an educated user would at first insist on using this device in the local-only configuration described and only open the device to the broader internet when they had done their research and concluded the device have adequate security.
In summary, users should understand their level of
expectation when it comes to the security of IoT devices. If there is an expectation that a video stream is secure, the user should do some investigation as to how it is secured. They should also do some research on incidents with this device to assess its overall security performance. Besides, there are some pervasive security flaws seen. In many of these devices, the inclusion of hardcoded, easily guessed credentials, should give the user pause and reason to question the security of their device. Further, as many devices like these baby monitors have a poor reputation for security, the devices should be limited in their use to a local network. Ultimately, the user should understand what the expected security of the device is and behave accordingly.
V. CONCLUSION Vulnerability analysis is a difficult task because of the
complexity of most security systems. However, it is an achievable task as proven by the aerospace industry. With the adoption of formal methods in software development, vulnerabilities are expected to be neutralized before the release not after. The understanding of threats and their intent can prioritize the importance of vulnerability modeling, decreasing the success of intrusion attempts. The Kill Chain technique permits a victim to halt an attack after the third stage if they identify the threat.
Additionally, this technique can help in identifying the intent and motivation of an adversary. As a result, these indications can result in the enhancement of security measures by highlighting the vulnerabilities. The most common and successful strategy used by hackers to infiltrate a security system remains improper user actions or human error. Education on network vulnerabilities of both, IT personnel’s
and end users, will ensure attackers do not gain access through unsuspected users. Individually, each of the three proposed methods can help detect and mitigate vulnerabilities; however, combine them form a formidable defense strategy.
REFERENCES [1] Internet Engineering Task Force RFC 2828 Internet Security Glossary [2] Anatoliy Gorbenko, Alexander Romanovsky, Olga Tarasyuk, Oleksandr
Biloborodov, Experience Report: Study of Vulnerabilities of Enterprise
[3] Operating Systems, IEEE Conferences, Year: 2017 Page s: 205 – 215 [4] Hamad AL-Mohannadi, Qublai Mirza, Anitta Namanya, Irfan Awan,
Andrea Cullen and Jules Disso, Cyber-Attack Modeling Analysis Techniques: An Overview, 2016 IEEE 4th International Conference on Future Internet of Things and Cloud Workshops (FiCloudW), 10.1109/W-FiCloud.2016.29
[5] W. L. Sharp, “Joint publication 3-60: Joint targeting,” Washington DC: Joint Chiefs of Staff, 2007. http://www.bits.de/NRANEU/others/jp- doctrine/jp3_60(07).pdf
[6] National Vulnerability Database (NVD), https://www.nvd.nist.gov [7] Fei, Shu, at al “Research on Network Security Protection System Base
on Dynamic Modeling” 2017 IEEE 2nd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC) https://ieeexplore-ieee-org.libproxy.bridgeport.edu/document/8285064
[8] Rezaee, Razieh, et al. "A Threat Risk Estimation Model for Computer Network Security" 2016 6th International Conference on Computer and Knowledge Engineering (ICCKE), 2016, https://ieeexplore-ieee- org.libproxy.bridgeport.edu/document/7802144
[9] High-Assurance Cryptography: Cryptography Software We Can Trust, Gilles Barthe, IEEE Security & Privacy, Volume: 13, Issue: 5, Pages: 86 - 89/2015
[10] Assessing and Extracting Software Security Vulnerabilities in SOFL Formal Specifications, Busalire Onesmus Emeka, Shaoying Liu, IEEE Conferences (ICEIC), Pages: 1-4/2018
[11] Formal Methods for Software Testing, Marie-Claude Gaudel, IEEE Conferences (TASE), Pages: 1-3/2017
[12] Razvan Beuran, Ken-ichi Chinen, Yasuo Tan, Yoichi Shinoda, “Towards Effective Cybersecurity Education and Training,” Japan Advanced Institute of Science and Technology, 2016. https://dspace.jaist.ac.jp/dspace/bitstream/10119/13769/1/IS-RR-2016- 003.pdf
[13] Razvan Beuran, Dat Tang, Cuong Pham, Ken-ichi Chinen, Yasuo Tan, Yoichi Shinoda, “Integrated framework for hands-on cybersecurity training: CyTrONE,” Computers and Security, Vol 78, Pages 43-59, 2018.
[14] David Harley, “Security and Education,” WeLiveScurity, 29 Aug 2017. https://www.welivesecurity.com/2017/08/29/security-and-education/
[15] Alex Aronovich, “Why Educating Your Employees on Cyber Intelligence and Security Will Reduce Risk,” Cybint News, 15 Jun 2018. https://www.cybintsolutions.com/employee-education-reduces-risk/
[16] IBM X-Force, “IBM X-Force Threat Intelligence Index 2018,” 2018. https://www.ibm.com/security/data-breach/threat-intelligence
[17] Eric Hutchins, Michael Cloppert, Rohan Amin, “Intelligence-Driven Computer Network Defense Informed by Analysis of Adversary Campaigns and Intrusion Kill Chains” Lockheed Martin Corporation, Jan 2011. https://www.lockheedmartin.com/content/dam/lockheed- martin/rms/documents/cyber/LM-White-Paper-Intel-Driven-Defense.pdf
[18] Iain Thomson, Wi-Fi baby heart monitor, may have the worst IoT security of 2016,
https://www.theregister.co.uk/2016/10/13/possibly_worst_iot_security_f ailure_yet/
[19] Mark Stanislav and Tod Beardsley, HACKING IoT: A Case Study on Baby Monitor Exposures and Vulnerabilities, https://www.rapid7.com/docs/Hacking-IoT-A-Case-Study-on-Baby- Monitor-Exposures-and-Vulnerabilities.pd
<< /ASCII85EncodePages false /AllowTransparency false /AutoPositionEPSFiles true /AutoRotatePages /None /Binding /Left /CalGrayProfile (Gray Gamma 2.2) /CalRGBProfile (sRGB IEC61966-2.1) /CalCMYKProfile (U.S. Web Coated \050SWOP\051 v2) /sRGBProfile (sRGB IEC61966-2.1) /CannotEmbedFontPolicy /Error /CompatibilityLevel 1.7 /CompressObjects /Off /CompressPages true /ConvertImagesToIndexed true /PassThroughJPEGImages true /CreateJobTicket false /DefaultRenderingIntent /Default /DetectBlends true /DetectCurves 0.0000 /ColorConversionStrategy /LeaveColorUnchanged /DoThumbnails false /EmbedAllFonts true /EmbedOpenType false /ParseICCProfilesInComments true /EmbedJobOptions true /DSCReportingLevel 0 /EmitDSCWarnings false /EndPage -1 /ImageMemory 1048576 /LockDistillerParams true /MaxSubsetPct 100 /Optimize true /OPM 0 /ParseDSCComments false /ParseDSCCommentsForDocInfo true /PreserveCopyPage true /PreserveDICMYKValues true /PreserveEPSInfo false /PreserveFlatness true /PreserveHalftoneInfo true /PreserveOPIComments false /PreserveOverprintSettings true /StartPage 1 /SubsetFonts true /TransferFunctionInfo /Remove /UCRandBGInfo /Preserve /UsePrologue false /ColorSettingsFile () /AlwaysEmbed [ true /AbadiMT-CondensedLight /ACaslon-Italic /ACaslon-Regular /ACaslon-Semibold /ACaslon-SemiboldItalic /AdobeArabic-Bold /AdobeArabic-BoldItalic /AdobeArabic-Italic /AdobeArabic-Regular /AdobeHebrew-Bold /AdobeHebrew-BoldItalic /AdobeHebrew-Italic /AdobeHebrew-Regular /AdobeHeitiStd-Regular /AdobeMingStd-Light /AdobeMyungjoStd-Medium /AdobePiStd /AdobeSongStd-Light /AdobeThai-Bold /AdobeThai-BoldItalic /AdobeThai-Italic /AdobeThai-Regular /AGaramond-Bold /AGaramond-BoldItalic /AGaramond-Italic /AGaramond-Regular /AGaramond-Semibold /AGaramond-SemiboldItalic /AgencyFB-Bold /AgencyFB-Reg /AGOldFace-Outline /AharoniBold /Algerian /Americana /Americana-ExtraBold /AndaleMono /AndaleMonoIPA /AngsanaNew /AngsanaNew-Bold /AngsanaNew-BoldItalic /AngsanaNew-Italic /AngsanaUPC /AngsanaUPC-Bold /AngsanaUPC-BoldItalic /AngsanaUPC-Italic /Anna /ArialAlternative /ArialAlternativeSymbol /Arial-Black /Arial-BlackItalic /Arial-BoldItalicMT /Arial-BoldMT /Arial-ItalicMT /ArialMT /ArialMT-Black /ArialNarrow /ArialNarrow-Bold /ArialNarrow-BoldItalic /ArialNarrow-Italic /ArialRoundedMTBold /ArialUnicodeMS /ArrusBT-Bold /ArrusBT-BoldItalic /ArrusBT-Italic /ArrusBT-Roman /AvantGarde-Book /AvantGarde-BookOblique /AvantGarde-Demi /AvantGarde-DemiOblique /AvantGardeITCbyBT-Book /AvantGardeITCbyBT-BookOblique /BakerSignet /BankGothicBT-Medium /Barmeno-Bold /Barmeno-ExtraBold /Barmeno-Medium /Barmeno-Regular /Baskerville /BaskervilleBE-Italic /BaskervilleBE-Medium /BaskervilleBE-MediumItalic /BaskervilleBE-Regular /Baskerville-Bold /Baskerville-BoldItalic /Baskerville-Italic /BaskOldFace /Batang /BatangChe /Bauhaus93 /Bellevue /BellMT /BellMTBold /BellMTItalic /BerlingAntiqua-Bold /BerlingAntiqua-BoldItalic /BerlingAntiqua-Italic /BerlingAntiqua-Roman /BerlinSansFB-Bold /BerlinSansFBDemi-Bold /BerlinSansFB-Reg /BernardMT-Condensed /BernhardModernBT-Bold /BernhardModernBT-BoldItalic /BernhardModernBT-Italic /BernhardModernBT-Roman /BiffoMT /BinnerD /BinnerGothic /BlackadderITC-Regular /Blackoak /blex /blsy /Bodoni /Bodoni-Bold /Bodoni-BoldItalic /Bodoni-Italic /BodoniMT /BodoniMTBlack /BodoniMTBlack-Italic /BodoniMT-Bold /BodoniMT-BoldItalic /BodoniMTCondensed /BodoniMTCondensed-Bold /BodoniMTCondensed-BoldItalic /BodoniMTCondensed-Italic /BodoniMT-Italic /BodoniMTPosterCompressed /Bodoni-Poster /Bodoni-PosterCompressed /BookAntiqua /BookAntiqua-Bold /BookAntiqua-BoldItalic /BookAntiqua-Italic /Bookman-Demi /Bookman-DemiItalic /Bookman-Light /Bookman-LightItalic /BookmanOldStyle /BookmanOldStyle-Bold /BookmanOldStyle-BoldItalic /BookmanOldStyle-Italic /BookshelfSymbolOne-Regular /BookshelfSymbolSeven /BookshelfSymbolThree-Regular /BookshelfSymbolTwo-Regular /Botanical /Boton-Italic /Boton-Medium /Boton-MediumItalic /Boton-Regular /Boulevard /BradleyHandITC /Braggadocio /BritannicBold /Broadway /BrowalliaNew /BrowalliaNew-Bold /BrowalliaNew-BoldItalic /BrowalliaNew-Italic /BrowalliaUPC /BrowalliaUPC-Bold /BrowalliaUPC-BoldItalic /BrowalliaUPC-Italic /BrushScript /BrushScriptMT /CaflischScript-Bold /CaflischScript-Regular /Calibri /Calibri-Bold /Calibri-BoldItalic /Calibri-Italic /CalifornianFB-Bold /CalifornianFB-Italic /CalifornianFB-Reg /CalisMTBol /CalistoMT /CalistoMT-BoldItalic /CalistoMT-Italic /Cambria /Cambria-Bold /Cambria-BoldItalic /Cambria-Italic /CambriaMath /Candara /Candara-Bold /Candara-BoldItalic /Candara-Italic /Carta /CaslonOpenfaceBT-Regular /Castellar /CastellarMT /Centaur /Centaur-Italic /Century /CenturyGothic /CenturyGothic-Bold /CenturyGothic-BoldItalic /CenturyGothic-Italic /CenturySchL-Bold /CenturySchL-BoldItal /CenturySchL-Ital /CenturySchL-Roma /CenturySchoolbook /CenturySchoolbook-Bold /CenturySchoolbook-BoldItalic /CenturySchoolbook-Italic /CGTimes-Bold /CGTimes-BoldItalic /CGTimes-Italic /CGTimes-Regular /CharterBT-Bold /CharterBT-BoldItalic /CharterBT-Italic /CharterBT-Roman /CheltenhamITCbyBT-Bold /CheltenhamITCbyBT-BoldItalic /CheltenhamITCbyBT-Book /CheltenhamITCbyBT-BookItalic /Chiller-Regular /Cmb10 /CMB10 /Cmbsy10 /CMBSY10 /CMBSY5 /CMBSY6 /CMBSY7 /CMBSY8 /CMBSY9 /Cmbx10 /CMBX10 /Cmbx12 /CMBX12 /Cmbx5 /CMBX5 /Cmbx6 /CMBX6 /Cmbx7 /CMBX7 /Cmbx8 /CMBX8 /Cmbx9 /CMBX9 /Cmbxsl10 /CMBXSL10 /Cmbxti10 /CMBXTI10 /Cmcsc10 /CMCSC10 /Cmcsc8 /CMCSC8 /Cmcsc9 /CMCSC9 /Cmdunh10 /CMDUNH10 /Cmex10 /CMEX10 /CMEX7 /CMEX8 /CMEX9 /Cmff10 /CMFF10 /Cmfi10 /CMFI10 /Cmfib8 /CMFIB8 /Cminch /CMINCH /Cmitt10 /CMITT10 /Cmmi10 /CMMI10 /Cmmi12 /CMMI12 /Cmmi5 /CMMI5 /Cmmi6 /CMMI6 /Cmmi7 /CMMI7 /Cmmi8 /CMMI8 /Cmmi9 /CMMI9 /Cmmib10 /CMMIB10 /CMMIB5 /CMMIB6 /CMMIB7 /CMMIB8 /CMMIB9 /Cmr10 /CMR10 /Cmr12 /CMR12 /Cmr17 /CMR17 /Cmr5 /CMR5 /Cmr6 /CMR6 /Cmr7 /CMR7 /Cmr8 /CMR8 /Cmr9 /CMR9 /Cmsl10 /CMSL10 /Cmsl12 /CMSL12 /Cmsl8 /CMSL8 /Cmsl9 /CMSL9 /Cmsltt10 /CMSLTT10 /Cmss10 /CMSS10 /Cmss12 /CMSS12 /Cmss17 /CMSS17 /Cmss8 /CMSS8 /Cmss9 /CMSS9 /Cmssbx10 /CMSSBX10 /Cmssdc10 /CMSSDC10 /Cmssi10 /CMSSI10 /Cmssi12 /CMSSI12 /Cmssi17 /CMSSI17 /Cmssi8 /CMSSI8 /Cmssi9 /CMSSI9 /Cmssq8 /CMSSQ8 /Cmssqi8 /CMSSQI8 /Cmsy10 /CMSY10 /Cmsy5 /CMSY5 /Cmsy6 /CMSY6 /Cmsy7 /CMSY7 /Cmsy8 /CMSY8 /Cmsy9 /CMSY9 /Cmtcsc10 /CMTCSC10 /Cmtex10 /CMTEX10 /Cmtex8 /CMTEX8 /Cmtex9 /CMTEX9 /Cmti10 /CMTI10 /Cmti12 /CMTI12 /Cmti7 /CMTI7 /Cmti8 /CMTI8 /Cmti9 /CMTI9 /Cmtt10 /CMTT10 /Cmtt12 /CMTT12 /Cmtt8 /CMTT8 /Cmtt9 /CMTT9 /Cmu10 /CMU10 /Cmvtt10 /CMVTT10 /ColonnaMT /Colossalis-Bold /ComicSansMS /ComicSansMS-Bold /Consolas /Consolas-Bold /Consolas-BoldItalic /Consolas-Italic /Constantia /Constantia-Bold /Constantia-BoldItalic /Constantia-Italic /CooperBlack /CopperplateGothic-Bold /CopperplateGothic-Light /Copperplate-ThirtyThreeBC /Corbel /Corbel-Bold /Corbel-BoldItalic /Corbel-Italic /CordiaNew /CordiaNew-Bold /CordiaNew-BoldItalic /CordiaNew-Italic /CordiaUPC /CordiaUPC-Bold /CordiaUPC-BoldItalic /CordiaUPC-Italic /Courier /Courier-Bold /Courier-BoldOblique /CourierNewPS-BoldItalicMT /CourierNewPS-BoldMT /CourierNewPS-ItalicMT /CourierNewPSMT /Courier-Oblique /CourierStd /CourierStd-Bold /CourierStd-BoldOblique /CourierStd-Oblique /CourierX-Bold /CourierX-BoldOblique /CourierX-Oblique /CourierX-Regular /CreepyRegular /CurlzMT /David-Bold /David-Reg /DavidTransparent /Dcb10 /Dcbx10 /Dcbxsl10 /Dcbxti10 /Dccsc10 /Dcitt10 /Dcr10 /Desdemona /DilleniaUPC /DilleniaUPCBold /DilleniaUPCBoldItalic /DilleniaUPCItalic /Dingbats /DomCasual /Dotum /DotumChe /DoulosSIL /EdwardianScriptITC /Elephant-Italic /Elephant-Regular /EngraversGothicBT-Regular /EngraversMT /EraserDust /ErasITC-Bold /ErasITC-Demi /ErasITC-Light /ErasITC-Medium /ErieBlackPSMT /ErieLightPSMT /EriePSMT /EstrangeloEdessa /Euclid /Euclid-Bold /Euclid-BoldItalic /EuclidExtra /EuclidExtra-Bold /EuclidFraktur /EuclidFraktur-Bold /Euclid-Italic /EuclidMathOne /EuclidMathOne-Bold /EuclidMathTwo /EuclidMathTwo-Bold /EuclidSymbol /EuclidSymbol-Bold /EuclidSymbol-BoldItalic /EuclidSymbol-Italic /EucrosiaUPC /EucrosiaUPCBold /EucrosiaUPCBoldItalic /EucrosiaUPCItalic /EUEX10 /EUEX7 /EUEX8 /EUEX9 /EUFB10 /EUFB5 /EUFB7 /EUFM10 /EUFM5 /EUFM7 /EURB10 /EURB5 /EURB7 /EURM10 /EURM5 /EURM7 /EuroMono-Bold /EuroMono-BoldItalic /EuroMono-Italic /EuroMono-Regular /EuroSans-Bold /EuroSans-BoldItalic /EuroSans-Italic /EuroSans-Regular /EuroSerif-Bold /EuroSerif-BoldItalic /EuroSerif-Italic /EuroSerif-Regular /EUSB10 /EUSB5 /EUSB7 /EUSM10 /EUSM5 /EUSM7 /FelixTitlingMT /Fences /FencesPlain /FigaroMT /FixedMiriamTransparent /FootlightMTLight /Formata-Italic /Formata-Medium /Formata-MediumItalic /Formata-Regular /ForteMT /FranklinGothic-Book /FranklinGothic-BookItalic /FranklinGothic-Demi /FranklinGothic-DemiCond /FranklinGothic-DemiItalic /FranklinGothic-Heavy /FranklinGothic-HeavyItalic /FranklinGothicITCbyBT-Book /FranklinGothicITCbyBT-BookItal /FranklinGothicITCbyBT-Demi /FranklinGothicITCbyBT-DemiItal /FranklinGothic-Medium /FranklinGothic-MediumCond /FranklinGothic-MediumItalic /FrankRuehl /FreesiaUPC /FreesiaUPCBold /FreesiaUPCBoldItalic /FreesiaUPCItalic /FreestyleScript-Regular /FrenchScriptMT /Frutiger-Black /Frutiger-BlackCn /Frutiger-BlackItalic /Frutiger-Bold /Frutiger-BoldCn /Frutiger-BoldItalic /Frutiger-Cn /Frutiger-ExtraBlackCn /Frutiger-Italic /Frutiger-Light /Frutiger-LightCn /Frutiger-LightItalic /Frutiger-Roman /Frutiger-UltraBlack /Futura-Bold /Futura-BoldOblique /Futura-Book /Futura-BookOblique /FuturaBT-Bold /FuturaBT-BoldItalic /FuturaBT-Book /FuturaBT-BookItalic /FuturaBT-Medium /FuturaBT-MediumItalic /Futura-Light /Futura-LightOblique /GalliardITCbyBT-Bold /GalliardITCbyBT-BoldItalic /GalliardITCbyBT-Italic /GalliardITCbyBT-Roman /Garamond /Garamond-Bold /Garamond-BoldCondensed /Garamond-BoldCondensedItalic /Garamond-BoldItalic /Garamond-BookCondensed /Garamond-BookCondensedItalic /Garamond-Italic /Garamond-LightCondensed /Garamond-LightCondensedItalic /Gautami /GeometricSlab703BT-Light /GeometricSlab703BT-LightItalic /Georgia /Georgia-Bold /Georgia-BoldItalic /Georgia-Italic /GeorgiaRef /Giddyup /Giddyup-Thangs /Gigi-Regular /GillSans /GillSans-Bold /GillSans-BoldItalic /GillSans-Condensed /GillSans-CondensedBold /GillSans-Italic /GillSans-Light /GillSans-LightItalic /GillSansMT /GillSansMT-Bold /GillSansMT-BoldItalic /GillSansMT-Condensed /GillSansMT-ExtraCondensedBold /GillSansMT-Italic /GillSans-UltraBold /GillSans-UltraBoldCondensed /GloucesterMT-ExtraCondensed /Gothic-Thirteen /GoudyOldStyleBT-Bold /GoudyOldStyleBT-BoldItalic /GoudyOldStyleBT-Italic /GoudyOldStyleBT-Roman /GoudyOldStyleT-Bold /GoudyOldStyleT-Italic /GoudyOldStyleT-Regular /GoudyStout /GoudyTextMT-LombardicCapitals /GSIDefaultSymbols /Gulim /GulimChe /Gungsuh /GungsuhChe /Haettenschweiler /HarlowSolid /Harrington /Helvetica /Helvetica-Black /Helvetica-BlackOblique /Helvetica-Bold /Helvetica-BoldOblique /Helvetica-Condensed /Helvetica-Condensed-Black /Helvetica-Condensed-BlackObl /Helvetica-Condensed-Bold /Helvetica-Condensed-BoldObl /Helvetica-Condensed-Light /Helvetica-Condensed-LightObl /Helvetica-Condensed-Oblique /Helvetica-Fraction /Helvetica-Narrow /Helvetica-Narrow-Bold /Helvetica-Narrow-BoldOblique /Helvetica-Narrow-Oblique /Helvetica-Oblique /HighTowerText-Italic /HighTowerText-Reg /Humanist521BT-BoldCondensed /Humanist521BT-Light /Humanist521BT-LightItalic /Humanist521BT-RomanCondensed /Imago-ExtraBold /Impact /ImprintMT-Shadow /InformalRoman-Regular /IrisUPC /IrisUPCBold /IrisUPCBoldItalic /IrisUPCItalic /Ironwood /ItcEras-Medium /ItcKabel-Bold /ItcKabel-Book /ItcKabel-Demi /ItcKabel-Medium /ItcKabel-Ultra /JasmineUPC /JasmineUPC-Bold /JasmineUPC-BoldItalic /JasmineUPC-Italic /JoannaMT /JoannaMT-Italic /Jokerman-Regular /JuiceITC-Regular /Kartika /Kaufmann /KaufmannBT-Bold /KaufmannBT-Regular /KidTYPEPaint /KinoMT /KodchiangUPC /KodchiangUPC-Bold /KodchiangUPC-BoldItalic /KodchiangUPC-Italic /KorinnaITCbyBT-Regular /KristenITC-Regular /KrutiDev040Bold /KrutiDev040BoldItalic /KrutiDev040Condensed /KrutiDev040Italic /KrutiDev040Thin /KrutiDev040Wide /KrutiDev060 /KrutiDev060Bold /KrutiDev060BoldItalic /KrutiDev060Condensed /KrutiDev060Italic /KrutiDev060Thin /KrutiDev060Wide /KrutiDev070 /KrutiDev070Condensed /KrutiDev070Italic /KrutiDev070Thin /KrutiDev070Wide /KrutiDev080 /KrutiDev080Condensed /KrutiDev080Italic /KrutiDev080Wide /KrutiDev090 /KrutiDev090Bold /KrutiDev090BoldItalic /KrutiDev090Condensed /KrutiDev090Italic /KrutiDev090Thin /KrutiDev090Wide /KrutiDev100 /KrutiDev100Bold /KrutiDev100BoldItalic /KrutiDev100Condensed /KrutiDev100Italic /KrutiDev100Thin /KrutiDev100Wide /KrutiDev120 /KrutiDev120Condensed /KrutiDev120Thin /KrutiDev120Wide /KrutiDev130 /KrutiDev130Condensed /KrutiDev130Thin /KrutiDev130Wide /KunstlerScript /Latha /LatinWide /LetterGothic /LetterGothic-Bold /LetterGothic-BoldOblique /LetterGothic-BoldSlanted /LetterGothicMT /LetterGothicMT-Bold /LetterGothicMT-BoldOblique /LetterGothicMT-Oblique /LetterGothic-Slanted /LevenimMT /LevenimMTBold /LilyUPC /LilyUPCBold /LilyUPCBoldItalic /LilyUPCItalic /Lithos-Black /Lithos-Regular /LotusWPBox-Roman /LotusWPIcon-Roman /LotusWPIntA-Roman /LotusWPIntB-Roman /LotusWPType-Roman /LucidaBright /LucidaBright-Demi /LucidaBright-DemiItalic /LucidaBright-Italic /LucidaCalligraphy-Italic /LucidaConsole /LucidaFax /LucidaFax-Demi /LucidaFax-DemiItalic /LucidaFax-Italic /LucidaHandwriting-Italic /LucidaSans /LucidaSans-Demi /LucidaSans-DemiItalic /LucidaSans-Italic /LucidaSans-Typewriter /LucidaSans-TypewriterBold /LucidaSans-TypewriterBoldOblique /LucidaSans-TypewriterOblique /LucidaSansUnicode /Lydian /Magneto-Bold /MaiandraGD-Regular /Mangal-Regular /Map-Symbols /MathA /MathB /MathC /Mathematica1 /Mathematica1-Bold /Mathematica1Mono /Mathematica1Mono-Bold /Mathematica2 /Mathematica2-Bold /Mathematica2Mono /Mathematica2Mono-Bold /Mathematica3 /Mathematica3-Bold /Mathematica3Mono /Mathematica3Mono-Bold /Mathematica4 /Mathematica4-Bold /Mathematica4Mono /Mathematica4Mono-Bold /Mathematica5 /Mathematica5-Bold /Mathematica5Mono /Mathematica5Mono-Bold /Mathematica6 /Mathematica6Bold /Mathematica6Mono /Mathematica6MonoBold /Mathematica7 /Mathematica7Bold /Mathematica7Mono /Mathematica7MonoBold /MatisseITC-Regular /MaturaMTScriptCapitals /Mesquite /Mezz-Black /Mezz-Regular /MICR /MicrosoftSansSerif /MingLiU /Minion-BoldCondensed /Minion-BoldCondensedItalic /Minion-Condensed /Minion-CondensedItalic /Minion-Ornaments /MinionPro-Bold /MinionPro-BoldIt /MinionPro-It /MinionPro-Regular /Miriam /MiriamFixed /MiriamTransparent /Mistral /Modern-Regular /MonotypeCorsiva /MonotypeSorts /MSAM10 /MSAM5 /MSAM6 /MSAM7 /MSAM8 /MSAM9 /MSBM10 /MSBM5 /MSBM6 /MSBM7 /MSBM8 /MSBM9 /MS-Gothic /MSHei /MSLineDrawPSMT /MS-Mincho /MSOutlook /MS-PGothic /MS-PMincho /MSReference1 /MSReference2 /MSReferenceSansSerif /MSReferenceSansSerif-Bold /MSReferenceSansSerif-BoldItalic /MSReferenceSansSerif-Italic /MSReferenceSerif /MSReferenceSerif-Bold /MSReferenceSerif-BoldItalic /MSReferenceSerif-Italic /MSReferenceSpecialty /MSSong /MS-UIGothic /MT-Extra /MTExtraTiger /MT-Symbol /MT-Symbol-Italic /MVBoli /Myriad-Bold /Myriad-BoldItalic /Myriad-Italic /Myriad-Roman /Narkisim /NewCenturySchlbk-Bold /NewCenturySchlbk-BoldItalic /NewCenturySchlbk-Italic /NewCenturySchlbk-Roman /NewMilleniumSchlbk-BoldItalicSH /NewsGothic /NewsGothic-Bold /NewsGothicBT-Bold /NewsGothicBT-BoldItalic /NewsGothicBT-Italic /NewsGothicBT-Roman /NewsGothic-Condensed /NewsGothic-Italic /NewsGothicMT /NewsGothicMT-Bold /NewsGothicMT-Italic /NiagaraEngraved-Reg /NiagaraSolid-Reg /NimbusMonL-Bold /NimbusMonL-BoldObli /NimbusMonL-Regu /NimbusMonL-ReguObli /NimbusRomNo9L-Medi /NimbusRomNo9L-MediItal /NimbusRomNo9L-Regu /NimbusRomNo9L-ReguItal /NimbusSanL-Bold /NimbusSanL-BoldCond /NimbusSanL-BoldCondItal /NimbusSanL-BoldItal /NimbusSanL-Regu /NimbusSanL-ReguCond /NimbusSanL-ReguCondItal /NimbusSanL-ReguItal /Nimrod /Nimrod-Bold /Nimrod-BoldItalic /Nimrod-Italic /NSimSun /Nueva-BoldExtended /Nueva-BoldExtendedItalic /Nueva-Italic /Nueva-Roman /NuptialScript /OCRA /OCRA-Alternate /OCRAExtended /OCRB /OCRB-Alternate /OfficinaSans-Bold /OfficinaSans-BoldItalic /OfficinaSans-Book /OfficinaSans-BookItalic /OfficinaSerif-Bold /OfficinaSerif-BoldItalic /OfficinaSerif-Book /OfficinaSerif-BookItalic /OldEnglishTextMT /Onyx /OnyxBT-Regular /OzHandicraftBT-Roman /PalaceScriptMT /Palatino-Bold /Palatino-BoldItalic /Palatino-Italic /PalatinoLinotype-Bold /PalatinoLinotype-BoldItalic /PalatinoLinotype-Italic /PalatinoLinotype-Roman /Palatino-Roman /PapyrusPlain /Papyrus-Regular /Parchment-Regular /Parisian /ParkAvenue /Penumbra-SemiboldFlare /Penumbra-SemiboldSans /Penumbra-SemiboldSerif /PepitaMT /Perpetua /Perpetua-Bold /Perpetua-BoldItalic /Perpetua-Italic /PerpetuaTitlingMT-Bold /PerpetuaTitlingMT-Light /PhotinaCasualBlack /Playbill /PMingLiU /Poetica-SuppOrnaments /PoorRichard-Regular /PopplLaudatio-Italic /PopplLaudatio-Medium /PopplLaudatio-MediumItalic /PopplLaudatio-Regular /PrestigeElite /Pristina-Regular /PTBarnumBT-Regular /Raavi /RageItalic /Ravie /RefSpecialty /Ribbon131BT-Bold /Rockwell /Rockwell-Bold /Rockwell-BoldItalic /Rockwell-Condensed /Rockwell-CondensedBold /Rockwell-ExtraBold /Rockwell-Italic /Rockwell-Light /Rockwell-LightItalic /Rod /RodTransparent /RunicMT-Condensed /Sanvito-Light /Sanvito-Roman /ScriptC /ScriptMTBold /SegoeUI /SegoeUI-Bold /SegoeUI-BoldItalic /SegoeUI-Italic /Serpentine-BoldOblique /ShelleyVolanteBT-Regular /ShowcardGothic-Reg /Shruti /SILDoulosIPA /SimHei /SimSun /SimSun-PUA /SnapITC-Regular /StandardSymL /Stencil /StoneSans /StoneSans-Bold /StoneSans-BoldItalic /StoneSans-Italic /StoneSans-Semibold /StoneSans-SemiboldItalic /Stop /Swiss721BT-BlackExtended /Sylfaen /Symbol /SymbolMT /SymbolTiger /SymbolTigerExpert /Tahoma /Tahoma-Bold /Tci1 /Tci1Bold /Tci1BoldItalic /Tci1Italic /Tci2 /Tci2Bold /Tci2BoldItalic /Tci2Italic /Tci3 /Tci3Bold /Tci3BoldItalic /Tci3Italic /Tci4 /Tci4Bold /Tci4BoldItalic /Tci4Italic /TechnicalItalic /TechnicalPlain /Tekton /Tekton-Bold /TektonMM /Tempo-HeavyCondensed /Tempo-HeavyCondensedItalic /TempusSansITC /Tiger /TigerExpert /Times-Bold /Times-BoldItalic /Times-BoldItalicOsF /Times-BoldSC /Times-ExtraBold /Times-Italic /Times-ItalicOsF /TimesNewRomanMT-ExtraBold /TimesNewRomanPS-BoldItalicMT /TimesNewRomanPS-BoldMT /TimesNewRomanPS-ItalicMT /TimesNewRomanPSMT /Times-Roman /Times-RomanSC /Trajan-Bold /Trebuchet-BoldItalic /TrebuchetMS /TrebuchetMS-Bold /TrebuchetMS-Italic /Tunga-Regular /TwCenMT-Bold /TwCenMT-BoldItalic /TwCenMT-Condensed /TwCenMT-CondensedBold /TwCenMT-CondensedExtraBold /TwCenMT-CondensedMedium /TwCenMT-Italic /TwCenMT-Regular /Univers-Bold /Univers-BoldItalic /UniversCondensed-Bold /UniversCondensed-BoldItalic /UniversCondensed-Medium /UniversCondensed-MediumItalic /Univers-Medium /Univers-MediumItalic /URWBookmanL-DemiBold /URWBookmanL-DemiBoldItal /URWBookmanL-Ligh /URWBookmanL-LighItal /URWChanceryL-MediItal /URWGothicL-Book /URWGothicL-BookObli /URWGothicL-Demi /URWGothicL-DemiObli /URWPalladioL-Bold /URWPalladioL-BoldItal /URWPalladioL-Ital /URWPalladioL-Roma /USPSBarCode /VAGRounded-Black /VAGRounded-Bold /VAGRounded-Light /VAGRounded-Thin /Verdana /Verdana-Bold /Verdana-BoldItalic /Verdana-Italic /VerdanaRef /VinerHandITC /Viva-BoldExtraExtended /Vivaldii /Viva-LightCondensed /Viva-Regular /VladimirScript /Vrinda /Webdings /Westminster /Willow /Wingdings2 /Wingdings3 /Wingdings-Regular /WNCYB10 /WNCYI10 /WNCYR10 /WNCYSC10 /WNCYSS10 /WoodtypeOrnaments-One /WoodtypeOrnaments-Two /WP-ArabicScriptSihafa /WP-ArabicSihafa /WP-BoxDrawing /WP-CyrillicA /WP-CyrillicB /WP-GreekCentury /WP-GreekCourier /WP-GreekHelve /WP-HebrewDavid /WP-IconicSymbolsA /WP-IconicSymbolsB /WP-Japanese /WP-MathA /WP-MathB /WP-MathExtendedA /WP-MathExtendedB /WP-MultinationalAHelve /WP-MultinationalARoman /WP-MultinationalBCourier /WP-MultinationalBHelve /WP-MultinationalBRoman /WP-MultinationalCourier /WP-Phonetic /WPTypographicSymbols /XYATIP10 /XYBSQL10 /XYBTIP10 /XYCIRC10 /XYCMAT10 /XYCMBT10 /XYDASH10 /XYEUAT10 /XYEUBT10 /ZapfChancery-MediumItalic /ZapfDingbats /ZapfHumanist601BT-Bold /ZapfHumanist601BT-BoldItalic /ZapfHumanist601BT-Demi /ZapfHumanist601BT-DemiItalic /ZapfHumanist601BT-Italic /ZapfHumanist601BT-Roman /ZWAdobeF ] /NeverEmbed [ true ] /AntiAliasColorImages false /CropColorImages true /ColorImageMinResolution 150 /ColorImageMinResolutionPolicy /OK /DownsampleColorImages true /ColorImageDownsampleType /Bicubic /ColorImageResolution 300 /ColorImageDepth -1 /ColorImageMinDownsampleDepth 1 /ColorImageDownsampleThreshold 2.00333 /EncodeColorImages true /ColorImageFilter /DCTEncode /AutoFilterColorImages true /ColorImageAutoFilterStrategy /JPEG /ColorACSImageDict << /QFactor 0.76 /HSamples [2 1 1 2] /VSamples [2 1 1 2] >> /ColorImageDict << /QFactor 0.76 /HSamples [2 1 1 2] /VSamples [2 1 1 2] >> /JPEG2000ColorACSImageDict << /TileWidth 256 /TileHeight 256 /Quality 15 >> /JPEG2000ColorImageDict << /TileWidth 256 /TileHeight 256 /Quality 15 >> /AntiAliasGrayImages false /CropGrayImages true /GrayImageMinResolution 150 /GrayImageMinResolutionPolicy /OK /DownsampleGrayImages true /GrayImageDownsampleType /Bicubic /GrayImageResolution 300 /GrayImageDepth -1 /GrayImageMinDownsampleDepth 2 /GrayImageDownsampleThreshold 2.00333 /EncodeGrayImages true /GrayImageFilter /DCTEncode /AutoFilterGrayImages true /GrayImageAutoFilterStrategy /JPEG /GrayACSImageDict << /QFactor 0.76 /HSamples [2 1 1 2] /VSamples [2 1 1 2] >> /GrayImageDict << /QFactor 0.76 /HSamples [2 1 1 2] /VSamples [2 1 1 2] >> /JPEG2000GrayACSImageDict << /TileWidth 256 /TileHeight 256 /Quality 15 >> /JPEG2000GrayImageDict << /TileWidth 256 /TileHeight 256 /Quality 15 >> /AntiAliasMonoImages false /CropMonoImages true /MonoImageMinResolution 1200 /MonoImageMinResolutionPolicy /OK /DownsampleMonoImages true /MonoImageDownsampleType /Bicubic /MonoImageResolution 600 /MonoImageDepth -1 /MonoImageDownsampleThreshold 1.00167 /EncodeMonoImages true /MonoImageFilter /CCITTFaxEncode /MonoImageDict << /K -1 >> /AllowPSXObjects false /CheckCompliance [ /None ] /PDFX1aCheck false /PDFX3Check false /PDFXCompliantPDFOnly false /PDFXNoTrimBoxError true /PDFXTrimBoxToMediaBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXSetBleedBoxToMediaBox true /PDFXBleedBoxToTrimBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXOutputIntentProfile (None) /PDFXOutputConditionIdentifier () /PDFXOutputCondition () /PDFXRegistryName () /PDFXTrapped /False /CreateJDFFile false /Description << /ARA <FEFF06270633062A062E062F0645002006470630064700200627064406250639062F0627062F0627062A002006440625064606340627062100200648062B062706260642002000410064006F00620065002000500044004600200645062A064806270641064206290020064506390020064506420627064A064A0633002006390631063600200648063706280627063906290020062706440648062B0627062606420020062706440645062A062F062706480644062900200641064A00200645062C062706440627062A002006270644062306390645062706440020062706440645062E062A064406410629061B0020064A06450643064600200641062A062D00200648062B0627062606420020005000440046002006270644064506460634062306290020062806270633062A062E062F062706450020004100630072006F0062006100740020064800410064006F006200650020005200650061006400650072002006250635062F0627063100200035002E0030002006480627064406250635062F062706310627062A0020062706440623062D062F062B002E> /CHS <FEFF4f7f75288fd94e9b8bbe5b9a521b5efa7684002000410064006f006200650020005000440046002065876863900275284e8e55464e1a65876863768467e5770b548c62535370300260a853ef4ee54f7f75280020004100630072006f0062006100740020548c002000410064006f00620065002000520065006100640065007200200035002e003000204ee553ca66f49ad87248672c676562535f00521b5efa768400200050004400460020658768633002> /CHT <FEFF4f7f752890194e9b8a2d7f6e5efa7acb7684002000410064006f006200650020005000440046002065874ef69069752865bc666e901a554652d965874ef6768467e5770b548c52175370300260a853ef4ee54f7f75280020004100630072006f0062006100740020548c002000410064006f00620065002000520065006100640065007200200035002e003000204ee553ca66f49ad87248672c4f86958b555f5df25efa7acb76840020005000440046002065874ef63002> /CZE <FEFF005400610074006f0020006e006100730074006100760065006e00ed00200070006f0075017e0069006a007400650020006b0020007600790074007600e101590065006e00ed00200064006f006b0075006d0065006e0074016f002000410064006f006200650020005000440046002000760068006f0064006e00fd00630068002000700072006f002000730070006f006c00650068006c0069007600e90020007a006f006200720061007a006f007600e1006e00ed002000610020007400690073006b0020006f006200630068006f0064006e00ed0063006800200064006f006b0075006d0065006e0074016f002e002000200056007900740076006f01590065006e00e900200064006f006b0075006d0065006e007400790020005000440046002000620075006400650020006d006f017e006e00e90020006f007400650076015900ed007400200076002000700072006f006700720061006d0065006300680020004100630072006f00620061007400200061002000410064006f00620065002000520065006100640065007200200035002e0030002000610020006e006f0076011b006a016100ed00630068002e> /DAN <FEFF004200720075006700200069006e0064007300740069006c006c0069006e006700650072006e0065002000740069006c0020006100740020006f007000720065007400740065002000410064006f006200650020005000440046002d0064006f006b0075006d0065006e007400650072002c0020006400650072002000650067006e006500720020007300690067002000740069006c00200064006500740061006c006a006500720065007400200073006b00e60072006d007600690073006e0069006e00670020006f00670020007500640073006b007200690076006e0069006e006700200061006600200066006f0072007200650074006e0069006e006700730064006f006b0075006d0065006e007400650072002e0020004400650020006f007000720065007400740065006400650020005000440046002d0064006f006b0075006d0065006e0074006500720020006b0061006e002000e50062006e00650073002000690020004100630072006f00620061007400200065006c006c006500720020004100630072006f006200610074002000520065006100640065007200200035002e00300020006f00670020006e0079006500720065002e> /DEU <FEFF00560065007200770065006e00640065006e0020005300690065002000640069006500730065002000450069006e007300740065006c006c0075006e00670065006e0020007a0075006d002000450072007300740065006c006c0065006e00200076006f006e002000410064006f006200650020005000440046002d0044006f006b0075006d0065006e00740065006e002c00200075006d002000650069006e00650020007a0075007600650072006c00e40073007300690067006500200041006e007a006500690067006500200075006e00640020004100750073006700610062006500200076006f006e00200047006500730063006800e40066007400730064006f006b0075006d0065006e00740065006e0020007a0075002000650072007a00690065006c0065006e002e00200044006900650020005000440046002d0044006f006b0075006d0065006e007400650020006b00f6006e006e0065006e0020006d006900740020004100630072006f00620061007400200075006e0064002000520065006100640065007200200035002e003000200075006e00640020006800f600680065007200200067006500f600660066006e00650074002000770065007200640065006e002e> /ESP <FEFF005500740069006c0069006300650020006500730074006100200063006f006e0066006900670075007200610063006900f3006e0020007000610072006100200063007200650061007200200064006f00630075006d0065006e0074006f0073002000640065002000410064006f00620065002000500044004600200061006400650063007500610064006f007300200070006100720061002000760069007300750061006c0069007a00610063006900f3006e0020006500200069006d0070007200650073006900f3006e00200064006500200063006f006e006600690061006e007a006100200064006500200064006f00630075006d0065006e0074006f007300200063006f006d00650072006300690061006c00650073002e002000530065002000700075006500640065006e00200061006200720069007200200064006f00630075006d0065006e0074006f00730020005000440046002000630072006500610064006f007300200063006f006e0020004100630072006f006200610074002c002000410064006f00620065002000520065006100640065007200200035002e003000200079002000760065007200730069006f006e0065007300200070006f00730074006500720069006f007200650073002e> /FRA <FEFF005500740069006c006900730065007a00200063006500730020006f007000740069006f006e00730020006100660069006e00200064006500200063007200e900650072002000640065007300200064006f00630075006d0065006e00740073002000410064006f006200650020005000440046002000700072006f00660065007300730069006f006e006e0065006c007300200066006900610062006c0065007300200070006f007500720020006c0061002000760069007300750061006c00690073006100740069006f006e0020006500740020006c00270069006d007000720065007300730069006f006e002e0020004c0065007300200064006f00630075006d0065006e00740073002000500044004600200063007200e900e90073002000700065007500760065006e0074002000ea0074007200650020006f007500760065007200740073002000640061006e00730020004100630072006f006200610074002c002000610069006e00730069002000710075002700410064006f00620065002000520065006100640065007200200035002e0030002000650074002000760065007200730069006f006e007300200075006c007400e90072006900650075007200650073002e> /GRE <FEFF03a703c103b703c303b903bc03bf03c003bf03b903ae03c303c403b5002003b103c503c403ad03c2002003c403b903c2002003c103c503b803bc03af03c303b503b903c2002003b303b903b1002003bd03b1002003b403b703bc03b903bf03c503c103b303ae03c303b503c403b5002003ad03b303b303c103b103c603b1002000410064006f006200650020005000440046002003ba03b103c403ac03bb03bb03b703bb03b1002003b303b903b1002003b103be03b903cc03c003b903c303c403b7002003c003c103bf03b203bf03bb03ae002003ba03b103b9002003b503ba03c403cd03c003c903c303b7002003b503c003b903c703b503b903c103b703bc03b103c403b903ba03ce03bd002003b503b303b303c103ac03c603c903bd002e0020002003a403b10020005000440046002003ad03b303b303c103b103c603b1002003c003bf03c5002003ad03c703b503c403b5002003b403b703bc03b903bf03c503c103b303ae03c303b503b9002003bc03c003bf03c103bf03cd03bd002003bd03b1002003b103bd03bf03b903c703c403bf03cd03bd002003bc03b5002003c403bf0020004100630072006f006200610074002c002003c403bf002000410064006f00620065002000520065006100640065007200200035002e0030002003ba03b103b9002003bc03b503c403b103b303b503bd03ad03c303c403b503c103b503c2002003b503ba03b403cc03c303b503b903c2002e> /HEB <FEFF05D405E905EA05DE05E905D5002005D105D405D205D305E805D505EA002005D005DC05D4002005DB05D305D9002005DC05D905E605D505E8002005DE05E105DE05DB05D9002000410064006F006200650020005000440046002005E205D105D505E8002005D405E605D205D4002005D505D405D305E405E105D4002005D005DE05D905E005D4002005E905DC002005DE05E105DE05DB05D905DD002005E205E105E705D905D905DD002E002005DE05E105DE05DB05D90020005000440046002005E905E005D505E605E805D5002005E005D905EA05E005D905DD002005DC05E405EA05D905D705D4002005D105D005DE05E605E205D505EA0020004100630072006F006200610074002005D5002D00410064006F00620065002000520065006100640065007200200035002E0030002005D505D205E805E105D005D505EA002005DE05EA05E705D305DE05D505EA002005D905D505EA05E8002E05D905D505EA05E8002E002D0033002C002005E205D905D905E005D5002005D105DE05D305E805D905DA002005DC05DE05E905EA05DE05E9002005E905DC0020004100630072006F006200610074002E002005DE05E105DE05DB05D90020005000440046002005E905E005D505E605E805D5002005E005D905EA05E005D905DD002005DC05E405EA05D905D705D4002005D105D005DE05E605E205D505EA0020004100630072006F006200610074002005D5002D00410064006F00620065002000520065006100640065007200200035002E0030002005D505D205E805E105D005D505EA002005DE05EA05E705D305DE05D505EA002005D905D505EA05E8002E> /HRV (Za stvaranje Adobe PDF dokumenata pogodnih za pouzdani prikaz i ispis poslovnih dokumenata koristite ove postavke. Stvoreni PDF dokumenti mogu se otvoriti Acrobat i Adobe Reader 5.0 i kasnijim verzijama.) /HUN <FEFF00410020006800690076006100740061006c006f007300200064006f006b0075006d0065006e00740075006d006f006b0020006d00650067006200ed007a00680061007400f30020006d0065006700740065006b0069006e007400e9007300e900720065002000e900730020006e0079006f006d00740061007400e1007300e10072006100200073007a00e1006e0074002000410064006f00620065002000500044004600200064006f006b0075006d0065006e00740075006d006f006b0061007400200065007a0065006b006b0065006c0020006100200062006500e1006c006c00ed007400e10073006f006b006b0061006c00200068006f007a006800610074006a00610020006c00e9007400720065002e0020002000410020006c00e90074007200650068006f007a006f00740074002000500044004600200064006f006b0075006d0065006e00740075006d006f006b00200061007a0020004100630072006f006200610074002000e9007300200061007a002000410064006f00620065002000520065006100640065007200200035002e0030002c0020007600610067007900200061007a002000610074007400f3006c0020006b00e9007301510062006200690020007600650072007a006900f3006b006b0061006c0020006e00790069007400680061007400f3006b0020006d00650067002e> /ITA (Utilizzare queste impostazioni per creare documenti Adobe PDF adatti per visualizzare e stampare documenti aziendali in modo affidabile. I documenti PDF creati possono essere aperti con Acrobat e Adobe Reader 5.0 e versioni successive.) /JPN <FEFF30d330b830cd30b9658766f8306e8868793a304a3088307353705237306b90693057305f002000410064006f0062006500200050004400460020658766f8306e4f5c6210306b4f7f75283057307e305930023053306e8a2d5b9a30674f5c62103055308c305f0020005000440046002030d530a130a430eb306f3001004100630072006f0062006100740020304a30883073002000410064006f00620065002000520065006100640065007200200035002e003000204ee5964d3067958b304f30533068304c3067304d307e305930023053306e8a2d5b9a3067306f30d530a930f330c8306e57cb30818fbc307f3092884c3044307e30593002> /KOR <FEFFc7740020c124c815c7440020c0acc6a9d558c5ec0020be44c988b2c8c2a40020bb38c11cb97c0020c548c815c801c73cb85c0020bcf4ace00020c778c1c4d558b2940020b3700020ac00c7a50020c801d569d55c002000410064006f0062006500200050004400460020bb38c11cb97c0020c791c131d569b2c8b2e4002e0020c774b807ac8c0020c791c131b41c00200050004400460020bb38c11cb2940020004100630072006f0062006100740020bc0f002000410064006f00620065002000520065006100640065007200200035002e00300020c774c0c1c5d0c11c0020c5f40020c2180020c788c2b5b2c8b2e4002e> /NLD (Gebruik deze instellingen om Adobe PDF-documenten te maken waarmee zakelijke documenten betrouwbaar kunnen worden weergegeven en afgedrukt. De gemaakte PDF-documenten kunnen worden geopend met Acrobat en Adobe Reader 5.0 en hoger.) /NOR <FEFF004200720075006b00200064006900730073006500200069006e006e007300740069006c006c0069006e00670065006e0065002000740069006c002000e50020006f0070007000720065007400740065002000410064006f006200650020005000440046002d0064006f006b0075006d0065006e00740065007200200073006f006d002000650072002000650067006e0065007400200066006f00720020007000e5006c006900740065006c006900670020007600690073006e0069006e00670020006f00670020007500740073006b007200690066007400200061007600200066006f0072007200650074006e0069006e006700730064006f006b0075006d0065006e007400650072002e0020005000440046002d0064006f006b0075006d0065006e00740065006e00650020006b0061006e002000e50070006e00650073002000690020004100630072006f00620061007400200065006c006c00650072002000410064006f00620065002000520065006100640065007200200035002e003000200065006c006c00650072002e> /POL <FEFF0055007300740061007700690065006e0069006100200064006f002000740077006f0072007a0065006e0069006100200064006f006b0075006d0065006e007400f300770020005000440046002000700072007a0065007a006e00610063007a006f006e00790063006800200064006f0020006e00690065007a00610077006f0064006e00650067006f002000770079015b0077006900650074006c0061006e00690061002000690020006400720075006b006f00770061006e0069006100200064006f006b0075006d0065006e007400f300770020006600690072006d006f0077007900630068002e002000200044006f006b0075006d0065006e0074007900200050004400460020006d006f017c006e00610020006f007400770069006500720061010700200077002000700072006f006700720061006d006900650020004100630072006f00620061007400200069002000410064006f00620065002000520065006100640065007200200035002e0030002000690020006e006f00770073007a0079006d002e> /PTB <FEFF005500740069006c0069007a006500200065007300730061007300200063006f006e00660069006700750072006100e700f50065007300200064006500200066006f0072006d00610020006100200063007200690061007200200064006f00630075006d0065006e0074006f0073002000410064006f00620065002000500044004600200061006400650071007500610064006f00730020007000610072006100200061002000760069007300750061006c0069007a006100e700e3006f002000650020006100200069006d0070007200650073007300e3006f00200063006f006e0066006900e1007600650069007300200064006500200064006f00630075006d0065006e0074006f007300200063006f006d0065007200630069006100690073002e0020004f007300200064006f00630075006d0065006e0074006f00730020005000440046002000630072006900610064006f007300200070006f00640065006d0020007300650072002000610062006500720074006f007300200063006f006d0020006f0020004100630072006f006200610074002000650020006f002000410064006f00620065002000520065006100640065007200200035002e0030002000650020007600650072007300f50065007300200070006f00730074006500720069006f007200650073002e> /RUM <FEFF005500740069006c0069007a00610163006900200061006300650073007400650020007300650074010300720069002000700065006e007400720075002000610020006300720065006100200064006f00630075006d0065006e00740065002000410064006f006200650020005000440046002000610064006500630076006100740065002000700065006e007400720075002000760069007a00750061006c0069007a00610072006500610020015f006900200074006900700103007200690072006500610020006c0061002000630061006c006900740061007400650020007300750070006500720069006f0061007201030020006100200064006f00630075006d0065006e00740065006c006f007200200064006500200061006600610063006500720069002e002000200044006f00630075006d0065006e00740065006c00650020005000440046002000630072006500610074006500200070006f00740020006600690020006400650073006300680069007300650020006300750020004100630072006f006200610074002c002000410064006f00620065002000520065006100640065007200200035002e00300020015f00690020007600650072007300690075006e0069006c006500200075006c0074006500720069006f006100720065002e> /RUS <FEFF04180441043f043e043b044c04370443043904420435002004340430043d043d044b04350020043d0430044104420440043e0439043a043800200434043b044f00200441043e043704340430043d0438044f00200434043e043a0443043c0435043d0442043e0432002000410064006f006200650020005000440046002c0020043f043e04340445043e0434044f04490438044500200434043b044f0020043d0430043404350436043d043e0433043e0020043f0440043e0441043c043e044204400430002004380020043f04350447043004420438002004340435043b043e0432044b044500200434043e043a0443043c0435043d0442043e0432002e002000200421043e043704340430043d043d044b04350020005000440046002d0434043e043a0443043c0435043d0442044b0020043c043e0436043d043e0020043e0442043a0440044b043204300442044c002004410020043f043e043c043e0449044c044e0020004100630072006f00620061007400200438002000410064006f00620065002000520065006100640065007200200035002e00300020043800200431043e043b043504350020043f043e04370434043d043804450020043204350440044104380439002e> /SLV <FEFF005400650020006e006100730074006100760069007400760065002000750070006f0072006100620069007400650020007a00610020007500730074007600610072006a0061006e006a006500200064006f006b0075006d0065006e0074006f0076002000410064006f006200650020005000440046002c0020007000720069006d00650072006e006900680020007a00610020007a0061006e00650073006c006a00690076006f0020006f0067006c00650064006f00760061006e006a006500200069006e0020007400690073006b0061006e006a006500200070006f0073006c006f0076006e0069006800200064006f006b0075006d0065006e0074006f0076002e00200020005500730074007600610072006a0065006e006500200064006f006b0075006d0065006e0074006500200050004400460020006a00650020006d006f0067006f010d00650020006f0064007000720065007400690020007a0020004100630072006f00620061007400200069006e002000410064006f00620065002000520065006100640065007200200035002e003000200069006e0020006e006f00760065006a01610069006d002e> /SUO <FEFF004b00e40079007400e40020006e00e40069007400e4002000610073006500740075006b007300690061002c0020006b0075006e0020006c0075006f0074002000410064006f0062006500200050004400460020002d0064006f006b0075006d0065006e007400740065006a0061002c0020006a006f0074006b006100200073006f0070006900760061007400200079007200690074007900730061007300690061006b00690072006a006f006a0065006e0020006c0075006f00740065007400740061007600610061006e0020006e00e400790074007400e4006d0069007300650065006e0020006a0061002000740075006c006f007300740061006d0069007300650065006e002e0020004c0075006f0064007500740020005000440046002d0064006f006b0075006d0065006e00740069007400200076006f0069006400610061006e0020006100760061007400610020004100630072006f0062006100740069006c006c00610020006a0061002000410064006f00620065002000520065006100640065007200200035002e0030003a006c006c00610020006a006100200075007500640065006d006d0069006c006c0061002e> /SVE <FEFF0041006e007600e4006e00640020006400650020006800e4007200200069006e0073007400e4006c006c006e0069006e006700610072006e00610020006f006d002000640075002000760069006c006c00200073006b006100700061002000410064006f006200650020005000440046002d0064006f006b0075006d0065006e007400200073006f006d00200070006100730073006100720020006600f60072002000740069006c006c006600f60072006c00690074006c006900670020007600690073006e0069006e00670020006f006300680020007500740073006b007200690066007400650072002000610076002000610066006600e4007200730064006f006b0075006d0065006e0074002e002000200053006b006100700061006400650020005000440046002d0064006f006b0075006d0065006e00740020006b0061006e002000f600700070006e00610073002000690020004100630072006f0062006100740020006f00630068002000410064006f00620065002000520065006100640065007200200035002e00300020006f00630068002000730065006e006100720065002e> /TUR <FEFF005400690063006100720069002000620065006c00670065006c006500720069006e0020006700fc00760065006e0069006c0069007200200062006900720020015f0065006b0069006c006400650020006700f6007200fc006e007400fc006c0065006e006d006500730069002000760065002000790061007a0064013100720131006c006d006100730131006e006100200075007900670075006e002000410064006f006200650020005000440046002000620065006c00670065006c0065007200690020006f006c0075015f007400750072006d0061006b0020006900e70069006e00200062007500200061007900610072006c0061007201310020006b0075006c006c0061006e0131006e002e00200020004f006c0075015f0074007500720075006c0061006e0020005000440046002000620065006c00670065006c0065007200690020004100630072006f006200610074002000760065002000410064006f00620065002000520065006100640065007200200035002e003000200076006500200073006f006e0072006100730131006e00640061006b00690020007300fc007200fc006d006c00650072006c00650020006100e70131006c006100620069006c00690072002e> /ENU (Use these settings to create Adobe PDF documents suitable for reliable viewing and printing of business documents. Created PDF documents can be opened with Acrobat and Adobe Reader 5.0 and later.) >> >> setdistillerparams << /HWResolution [600 600] /PageSize [612.000 792.000] >> setpagedevice

Get help from top-rated tutors in any subject.
Efficiently complete your homework and academic assignments by getting help from the experts at homeworkarchive.com