.. _mozilla_projects_nss_an_overview_of_nss_internals:

An overview of NSS Internals
============================

.. container::

   | A High-Level Overview to the Internals of `Network Security Services
     (NSS) <https://developer.mozilla.org/en-US/docs/NSS>`__
   | Software developed by the Mozilla.org projects traditionally used its own implementation of
     security protocols and cryptographic algorithms, originally called Netscape Security Services,
     nowadays called Network Security Services (NSS). NSS is a library written in the C programming
     language. It's free and open source software, and many other software projects have decided to
     use it. In order to support multiple operating systems (OS), it is based on a cross platform
     portability layer, called the Netscape Portable Runtime (NSPR), which provides cross platform
     application programming interfaces (APIs) for OS specific APIs like file system access, memory
     management, network communication, and multithreaded programming.
   | NSS offers lots of functionality; we'll walk through the list of modules, design principles,
     and important relevant standards.
   | In order to allow interoperability between software and devices that perform cryptographic
     operations, NSS conforms to a standard called PKCS#11. (Note that it's important to look at the
     number 11, as there are other PKCS standards with different numbers that define quite different
     topics.)
   | A software or hardware module conforming to the PKCS#11 standard implements an interface of C
     calls, which allow querying the characteristics and offered services of the module. Multiple
     elements of NSS's own modules have been implemented with this interface, and NSS makes use of
     this interface when talking to those modules. This strategy allows NSS to work with many
     hardware devices (e.g., to speed up the calculations required for cryptographic operations, or
     to access smartcards that securely protect a secret key) and software modules (e.g., to allow
     to load such modules as a plugin that provides additional algorithms or stores key or trust
     information) that implement the PKCS#11 interface.
   | A core element of NSS is FreeBL, a base library providing hash functions, big number
     calculations, and cryptographic algorithms.
   | Softoken is an NSS module that exposes most FreeBL functionality as a PKCS#11 module.
   | Some cryptography uses the same secret key for both encrypting and decrypting, for example
     password based encryption (PBE). This is often sufficient if you encrypt data for yourself, but
     as soon as you need to exchange signed/encrypted data with communication partners, using public
     key encryption simplifies the key management. The environment that describes how to use public
     key encryption is called Public Key Infrastructure (PKI). The public keys that are exchanged
     between parties are transported using a container; the container is called a certificate,
     following standard X.509 version 3. A certificate contains lots of other details; for example,
     it contains a signature by a third party that expresses trust in the ownership relationship for
     the certificate. The trust assigned by the third party might be restricted to certain uses,
     which are listed in certificate extensions that are contained in the certificate.
   | Many (if not most) of the operations performed by NSS involve the use of X.509 certificates
     (often abbreviated as “cert”, unfortunately making it easy to confuse with the term “computer
     emergency response team“).
   | When checking whether a certificate is trusted or not, it's necessary to find a relevant trust
     anchor (root certificate) that represents the signing capability of a trusted third party,
     usually called a Certificate Authority (CA). A trust anchor is just another X.509 certificate
     that is already known and has been deliberately marked as trusted by a software vendor,
     administrators inside an organizational infrastructure, or the software user. NSS ships a
     predefined set of CA certificates. This set, including their trust assignments, is provided by
     NSS as a software module, called CKBI (“built-in root certificates”), which also implements the
     PKCS#11 interface. On an organizational level the contents of the set are managed according to
     the Mozilla CA policy. On a technical level the set is a binary software module.
   | A cryptographic transaction, such as encryption or decryption related to a data exchange,
     usually involves working with the X.509 certs of your communication partners (peer). It's also
     required that you safely keep your own secret keys that belong to your own certificates. You
     might want to protect the storage of your secret keys with PBE. You might decide to modify the
     default trust provided by NSS. All of this requires storing, looking up, and retrieving data.
     NSS simplifies performing these operations by offering storage and management APIs. NSS doesn't
     require the programmer to manage individual files containing individual certificates or keys.
     Instead, NSS offers to use its own database(s). Once you have imported certificates and keys
     into the NSS database, you can easily look them up and use them again.
   | Because of NSS's expectation to operate with an NSS database, it's mandatory that you perform
     an initialization call, where you tell NSS which database you will be using. In the most simple
     scenario, the programmer will provide a directory on your filesystem as a parameter to the init
     function, and NSS is designed to do the rest. It will detect and open an existing database, or
     it can create a new one. Alternatively, should you decide that you don't want to work with any
     persistent recording of certificates, you may initialize NSS in a no-database mode. Usually,
     NSS will flush all data to disk as soon as new data has been added to permanent storage.
     Storage consists of multiple files: a key database file, which contains your secret keys, and a
     certificate database file which contains the public portion of your own certificates, the
     certificates of peers or CAs, and a list of trust decisions (such as to not trust a built-in
     CA, or to explicitly trust other CAs). Examples for the database files are key3.db and
     cert8.db, where the numbers are file version numbers. A third file contains the list of
     external PKCS#11 modules that have been registered to be used by NSS. The file could be named
     secmod.db, but in newer database generations a file named pkcs11.txt is used.
   | Only NSS is allowed to access and manipulate these database files directly; a programmer using
     NSS must go through the APIs offered by NSS to manipulate the data stored in these files. The
     programmer's task is to initialize NSS with the required parameters (such as a database), and
     NSS will then transparently manage the database files.
   | Most of the time certificates and keys are supposed to be stored in the NSS database.
     Therefore, after initial import or creation, the programmer usually doesn't deal with their raw
     bytes. Instead, the programmer will use lookup functions, and NSS will provide an access handle
     that will be subsequently used by the application's code. Those handles are reference counted.
     NSS will usually create an in-memory (RAM) presentation of certificates, once a certificate has
     been received from the network, read from disk, or looked up from the database, and prepare
     in-memory data structures that contain the certificate's properties, as well as providing a
     handle for the programmer to use. Once the application is done with a handle, it should be
     released, allowing NSS to free the associated resources. When working with handles to private
     keys it's usually difficult (and undesired) that an application gets access to the raw key
     data; therefore it may be difficult to extract such data from NSS. The usual minimum
     requirement is that private keys must be wrapped using a protective layer (such as
     password-based encryption). The intention is to make it easier to review code for security. The
     less code that has access to raw secret keys, the less code that must be reviewed.
   | NSS has only limited functionality to look up raw keys. The preferred approach is to use
     certificates, and to look up certificates by properties such as the contained subject name
     (information that describes the owner of the certificate). For example, while NSS supports
     random calculation (creation) of a new public/private key pair, it's difficult to work with
     such a raw key pair. The usual approach is to create a certificate signing request (CSR) as
     soon as an application is done with the creation step, which will have created a handle to the
     key pair, and which can be used for the necessary related operations, like producing a
     proof-of-ownership of the private key, which is usually required when submitting the public key
     with a CSR to a CA. The usual follow up action is receiving a signed certificate from a CA.
     (However, it's also possible to use NSS functionality to create a self-signed certificate,
     which, however, usually won't be trusted by other parties.) Once received, it's sufficient to
     tell NSS to import such a new certificate into the NSS database, and NSS will automatically
     perform a lookup of the embedded public key, be able to find the associated private key, and
     subsequently be able to treat it as a personal certificate. (A personal certificate is a
     certificate for which the private key is in possession, and which could be used for signing
     data or for decrypting data.) A unique nickname can/should be assigned to the certificate at
     the time of import, which can later be used to easily identify and retrieve it.
   | It's important to note that NSS requires strict cleanup for all handles returned by NSS. The
     application should always call the appropriate dereference (destroy) functions once a handle is
     no longer needed. This is particularly important for applications that might need to close a
     database and reinitialize NSS using a different one, without restarting. Such an operation
     might fail at runtime if data elements are still being referenced.
   | In addition to the FreeBL, Softoken, and CKBI modules, there is an utility library for general
     operations (e.g., encoding/decoding between data formats, a list of standardized object
     identifiers (OID)). NSS has an SSL/TLS module that implements the Secure Sockets
     Layer/Transport Layer Security network protocols, an S/MIME module that implements CMS
     messaging used by secure email and some instant messaging implementations, a DBM library that
     implements the classic database storage, and finally a core NSS library for the big set of
     “everything else”. Newer generations of the database use the SQLite database to allow
     concurrent access by multiple applications.
   | All of the above are provided as shared libraries.
   | When dealing with certificates (X.509), file formats such as PKCS#12 (certificates and keys),
     PKCS#7 (signed data), and message formats as CMS, we should mention ASN.1, which is a syntax
     for storing structured data in a very efficient (small sized) presentation. It was originally
     developed for telecommunication systems at times where it was critical to minimize data as much
     as possible (although it still makes sense to use that principle today for good performance).
     In order to process data available in the ASN.1 format, the usual approach is to parse it and
     transfer it to a presentation that requires more space but is easier to work with, such as
     (nested) C data structures. Over the time NSS has received three different ASN.1 parser
     implementations, each having their own specific properties, advantages and disadvantages, which
     is why all of them are still being used (nobody has yet dared to replace the older with the
     newer ones because of risks for side effects). When using the ASN.1 parser(s), a template
     definition is passed to the parser, which will analyze the ASN.1 data stream accordingly. The
     templates are usually closely aligned to definitions found in RFC documents.
   | A data block described as DER is usually in ASN.1 format. You must know which data you are
     expecting, and use the correct template for parsing, based on the context of your software's
     interaction. Data described as PEM is a base64 encoded presentation of DER, usually wrapped
     between human readable BEGIN/END lines. NSS prefers the binary presentation, but is often
     capable to use base64 or ASCII presentations, especially when importing data from files. A
     recent development adds support for loading external PEM files that contain private keys, in a
     software library called nss-pem, which is separately available, but should eventually become a
     core part of NSS.
   | Looking at the code level, NSS deals with blocks of raw data all the time. The common structure
     to store such an untyped block is SECItem, which contains a size and an untyped C pointer
     variable.
   | When dealing with memory, NSS makes use of arenas, which are an attempt to simplify management
     with the limited offerings of C (because there are no destructors). The idea is to group
     multiple memory allocations in order to simplify cleanup. Performing an operation often
     involves allocating many individual data items, and the code might be required to abort a task
     at many positions in the logic. An arena is requested once processing of a task starts, and all
     memory allocations that are logically associated to that task are requested from the associated
     arena. The implementation of arenas makes sure that all individual memory blocks are tracked.
     Once a task is done, regardless whether it completed or was aborted, the programmer simply
     needs to release the arena, and all individually allocated blocks will be released
     automatically. Often freeing is combined with immediately erasing (zeroing, zfree) the memory
     associated to the arena, in order to make it more difficult for attackers to extract keys from
     a memory dump.
   | NSS uses many C data structures. Often NSS has multiple implementations for the same or similar
     concepts. For example, there are multiple presentations of certificates, and the NSS internals
     (and sometimes even the application using NSS) might have to convert between them.
   | Key responsibilites of NSS are verification of signatures and certificates. In order to verify
     a digital signature, we have to look at the application data (e.g., a document that was
     signed), the signature data block (the digital signature), and a public key (as found in a
     certificate that is believed to be the signer, e.g., identified by metadata received together
     with the signature). The signature is verified if it can be shown that the signature data block
     must have been produced by the owner of the public key (because only that owner has the
     associated private key).
   | Verifying a certificate (A) requires some additional steps. First, you must identify the
     potential signer (B) of a certificate (A). This is done by reading the “issuer name” attribute
     of a certificate (A), and trying to find that issuer certificate (B) (by looking for a
     certificate that uses that name as its “subject name”). Then you attempt to verify the
     signature found in (A) using the public key found in (B). It might be necessary to try multiple
     certificates (B1, B2, ...) each having the same subject name.
   | After succeeding, it might be necessary to repeat this procedure recursively. The goal is to
     eventually find a certificate B (or C or ...) that has an appropriate trust assigned (e.g.,
     because it can be found in the CKBI module and the user hasn't made any overriding trust
     decisions, or it can be found in a NSS database file managed by the user or by the local
     environment).
   | After having successfully verified the signatures in a (chain of) issuer certificate(s), we're
     still not done with verifying the certificate A. In a PKI it's suggested/required to perform
     additional checks. For example: Certificates were valid at the time the signature was made,
     name in certificates matches the expected signer (check subject name, common name, email, based
     on application), the trust restrictions recorded inside the certificate (extensions) permit the
     use (e.g., encryption might be allowed, but not signing), and based on environment/application
     policy it might be required to perform a revocation check (OCSP or CRL), that asks the
     issuer(s) of the certificates whether there have been events that made it necessary to revoke
     the trust (revoke the validity of the cert).
   | Trust anchors contained in the CKBI module are usually self signed, which is defined as having
     identical subject name and issuer name fields. If a self-signed certificate is marked as
     explicitly trusted, NSS will skip checking the self-signature for validity.
   | NSS has multiple APIs to perform verification of certificates. There is a classic engine that
     is very stable and works fine in all simple scenarios, for example if all (B) candidate issuer
     certificates have the same subject and issuer names and differ by validity period; however, it
     works only in a limited amount of more advanced scenarios. Unfortunately, the world of
     certificates has become more complex in the recent past. New Certificate Authorities enter the
     global PKI market, and in order to get started with their business, they might make deals with
     established CAs and receive so-called cross-signing-certificates. As a result, when searching
     for a trust path from (A) to a trusted anchor (root) certificate (Z), the set of candidate
     issuer certificates might have different issuer names (referring to the second or higher issuer
     level). As a consequence, it will be necessary to try multiple different alternative routes
     while searching for (Z), in a recursive manner. Only the newer verification engine (internally
     named libPKIX) is capable of doing that properly.
   | It's worth mentioning the Extended Validation (EV) principle, which is an effort by software
     vendors and CAs to define a stricter set of rules for issuing certificates for web site
     certificates. Instead of simply verifying that the requester of a certificate is in control of
     an administrative email address at the desired web site's domain, it's required that the CA
     performs a verification of real world identity documents (such as a company registration
     document with the country's authority), and it's also required that a browser software performs
     a revocation check with the CA, prior to granting validity to the certificate. In order to
     distinguish an EV certificate, CAs will embed a policy OID in the certificate, and the browser
     is expected to verify that a trust chain permits the end entity (EE) certificate to make use of
     the policy. Only the APIs of the newer libPKIX engine are capable of performing a policy
     verification.
   | That's a good opportunity to talk about SSL/TLS connections to servers in general (not just EV,
     not just websites). Whenever this document mentions SSL, it refers to either SSL or TLS. (TLS
     is a newer version of SSL with enhanced features.)
   | When establishing an SSL connection to a server, (at least) a server certificate (and its trust
     chain) is exchanged from the server to the client (e.g., the browser), and the client verifies
     that the certificate can be verified (including matching the name of the expected destination
     server). Another part of the handshake between both parties is a key exchange. Because public
     key encryption is more expensive (more calculations required) than symmetric encryption (where
     both parties use the same key), a key agreement protocol will be executed, where the public and
     private keys are used to proof and verify the exchanged initial information. Once the key
     agreement is done, a symmetric encryption will be used (until a potential re-handshake on an
     existing channel). The combination of the hash and encryption algorithms used for a SSL
     connection is called a cipher suite.
   | NSS ships with a set of cipher suites that it supports at a technical level. In addition, NSS
     ships with a default policy that defines which cipher suites are enabled by default. An
     application is able to modify the policy used at program runtime, by using function calls to
     modify the set of enabled cipher suites.
   | If a programmer wants to influence how NSS verifies certificates or how NSS verifies the data
     presented in a SSL connection handshake, it is possible to register application-defined
     callback functions which will be called by NSS at the appropriate point of time, and which can
     be used to override the decisions made by NSS.
   | If you would like to use NSS as a toolkit that implements SSL, remember that you must init NSS
     first. But if you don't care about modifying the default trust permanently (recorded on disk),
     you can use the no-database init calls. When creating the network socket for data exchange,
     note that you must use the operating system independent APIs provided by NSPR and NSS. It might
     be interesting to mention a property of the NSPR file descriptors, which are stacked in layers.
     This means you can define multiple layers that are involved in data processing. A file
     descriptor has a pointer to the first layer handling the data. That layer has a pointer to a
     potential second layer, which might have another pointer to a third layer, etc. Each layer
     defines its own functions for the open/close/read/write/poll/select (etc.) functions. When
     using an SSL network connection, you'll already have two layers, the basic NSPR layer and an
     SSL library layer. The Mozilla applications define a third layer where application specific
     processing is performed. You can find more details in the NSPR reference documents.
   | NSS occassionally has to create outbound network connections, in addition to the connections
     requested by the application. Examples are retrieving OCSP (Online Certificate Status Protocol)
     information or downloading a CRL (Certificate Revocation List). However, NSS doesn't have an
     implementation to work with network proxies. If you must support proxies in your application,
     you are able to register your own implementation of an http request callback interface, and NSS
     can use your application code that supports proxies.
   | When using hashing, encryption, and decryption functions, it is possible to stream data (as
     opposed to operating on a large buffer). Create a context handle while providing all the
     parameters required for the operation, then call an “update” function multiple times to pass
     subsets of the input to NSS. The data will be processed and either returned directly or sent to
     a callback function registered in the context. When done, you call a finalization function that
     will flush out any pending data and free the resources.
   | This line is a placeholder for future sections that should explain how libpkix works and is
     designed.
   | If you want to work with NSS, it's often helpful to use the command line utilities that are
     provided by the NSS developers. There are tools for managing NSS databases, for dumping or
     verifying certificates, for registering PKCS#11 modules with a database, for processing CMS
     encrypted/signed messages, etc.
   | For example, if you wanted to create your own pair of keys and request a new certificate from a
     CA, you could use certutil to create an empty database, then use certutil to operate on your
     database and create a certificate request (which involves creating the desired key pair) and
     export it to a file, submit the request file to the CA, receive the file from the CA, and
     import the certificate into your database. You should assign a good nickname to a certificate
     when importing it, making it easier for you to refer to it later.
   | It should be noted that the first database format that can be accessed simultaneously by
     multiple applications is key4.db/cert9.db – database files with lower numbers will most likely
     experience unrecoverable corruption if you access them with multiple applications at the same
     time. In other words, if your browser or your server operates on an older NSS database format,
     don't use the NSS tools to operate on it while the other software is executing. At the time of
     writing NSS and the Mozilla applications still use the older database file format by default,
     where each application has its own NSS database.
   | If you require a copy of a certificate stored in an NSS database, including its private key,
     you can use pk12util to export it to the PKCS#12 file format. If you require it in PEM format,
     you could use the openssl pkcs12 command (that's not NSS) to convert the PKCS#12 file to PEM.
   | This line is a placeholder for how to prepare a database, how to dump a cert, and how to
     convert data.
   | You might have been motivated to work with NSS because it is used by the Mozilla applications
     such as Firefox, Thunderbird, etc. If you build the Mozilla application, it will automatically
     build the NSS library, too. However, if you want to work with the NSS command line tools, you
     will have to follow the standalone NSS build instructions, and build NSS outside of the Mozilla
     application sources.
   | The key database file will contain at least one symmetric key, which NSS will automatically
     create on demand, and which will be used to protect your secret (private) keys. The symmetric
     key can be protected with PBE by setting a master password on the database. As soon as you set
     a master password, an attacker stealing your key database will no longer be able to get access
     to your private key, unless the attacker would also succeed in stealing the master password.
   | Now you might be interest in how to get the
     :ref:`mozilla_projects_nss_nss_sources_building_testing`