Survey of Possibilities for the Automated Configuration of Large IP Networks

Many large IP networks are being deployed that entail the installation of tens of thousands of new network devices. To keep costs down, it is desirable to automate the establishment of such networks to the maximum extent possible. This naturally raises the question how new devices can pick up the configuration information they need to operate properly in an automated fashion. The goal of this document is to list known solutions where they exist, to point out approaches proven to be problematic, and to identify gaps that require further specifications. The document primarily targets (a) network operators (in the generic sense) who are facing the challenge to roll out a large number of new devices and think about how to implement things properly, (b) network equipment vendors who like to add features to their products that make the roll out of lots of new devices simpler for their customers, and (c) people active in the IETF by identifying gaps where further standards may be useful to develop. The aim of the document is to provide guidance to actors who have not already experienced success in this area by informing about the trade-offs of different approaches. A certain basic amount of configuration information must be pre- configured by the vendor or network operator before the devices are physically deployed. This pre-provisioned configuration can either be stored directly on the device itself or it can be provided to the device during the deployment operation via pluggable memory cards or near field communication technologies. Further device configuration information is best delivered after startup, to ensure that it is consistent with the physical deployment and the desired network configuration. One example where automated configuration is important are new service provider networks. 3GPP work in progress describes requirements and an architectural specification for the self-configuration of edge node entities called eNodeBs. (The expansion of eNodeB is too unwieldy to spell out.) Specifically, procedures are specified for establishing transport connections to and for exchanging configuration data with control entities called MMEs (Mobility Management Entities) and with neighbouring eNodeBs. currently assumes as a starting precondition that the eNodeB knows its own IP address and knows IP address endpoints for the target MMEs and neighbouring eNodeBs. The Broadband Forum has defined a CPE WAN Management Protocol (running over SOAP/HTTP/TLS) to manage customer premise equipment (CPE) terminating broadband access networks (typically DSL access networks) . CPE devices locate and connect to an Auto-Configuration Server (ACS), which provides configuration data and software/firmware images and modules. The ACS also performs status and performance monitoring and diagnostic functions. CPE devices use DHCP to locate an ACS and since both peers, the ACS and CPE, can initiate connections, the protocol can work across network address translators (NATs). The DHCP exchange uses vendor-specific options defined by the Broadband Forum (number 3561 in the IANA Enterprise Numbers registry). Next to service provider networks, many large enterprise networks face the same challenge to roll out a large number of network devices, which often connect to a 3rd party network provider. The current development of IP-based home automation and utility monitoring technologies might carry the problem to roll out large numbers of devices that need to automatically configure themselves to private households. IETF work on automated configuration goes back to BOOTP , followed eight years later by DHCP ( and successors). The years since have seen a steady growth in the number of DHCP options. The Simple Network Management Protocol (SNMP) was designed to convey management information between SNMP entities such as managers and agents. The number of SNMP MIB modules grew steadily, but SNMP has historically seen only limited use for configuration . For a period, IETF configuration efforts were focussed on the distribution of policy information in the network. provides a good insight into this period. More recently, the network configuration protocol NETCONF was devised as an alternative to SNMP, but the development of standard NETCONF configuration data models is just beginning. Recent IETF work closest in spirit to the 3GPP self-organizing network effort cited above is embodied in CAPWAP . Like the 3GPP work, CAPWAP focusses on the configuration of edge nodes, in a Wi-Fi rather than cellular network. The CAPWAP work goes beyond that of 3GPP by specifying the process of Access Controller (AC) discovery rather than leaving discovery out of scope. A CAPWAP Wireless Termination Point (WTP) may use broadcasts and multicasts to discover local ACs, it may use CAPWAP DHCP options to obtain IP addresses of ACs, or it may utilize CAPWAP DNS SRV records if a domain name is known. With regard to the configuration process itself, CAPWAP provides for the download of new images to the WTP (Wireless Termination Point). In contrast, assumes that this has already been completed for the eNodeB. As can seen, standards for the automated configuration of devices in IP networks have so far been primarily developed for specific network access technologies (3GPP, Broadband, 802.11 WLANs) and the various solutions make different assumptions about the services that are available and they are designed to support a configuration protocol that is specific to a certain access technology. The aim of this document is to analyse the various phases of an automated configuration process and to identify gaps that are currently not covered in standard and general purpose configuration management protocols of the IETF.

There are two different scenarios to consider. In the first scenario, called the Intra-domain Scenario, the new network device N is attached to the network operated by the service provider which is also operating the new device. In the second scenario, called the Inter-domain Scenario, the new device N is attached to a third party network providing connectivity to the network of the service provider operating the new device.

+------+ | CONF | +--+---+ +---+ +---+ | | N +-...-+ R +------+---+---+----... +---+ +---+ | | +--+--+ +--+---+ | DNS | | DHCP | +-----+ +------+ |-- N's Service Provider --| depicts the Intra-domain Scenario. We assume that the new device N attaches to a link connected to router R. Furthermore, we assume that the service provider provides a Domain Name System (DNS) server, a reachable DHCP server, and a Configuration Server (CONF). Overall, this scenario does not differ much from conventional network scenarios.

+------+ | CONF | +--+---+ +---+ +---+ +---+ | | N +-...-+ R +-----+---+---+-----...-+ R +-----+---+---+-----... +---+ +---+ | | +---+ | | +--+--+ +--+---+ +--+--+ +--+---+ | DNS | | DHCP | | DNS | | DHCP | +-----+ +------+ +-----+ +------+ |-- Service Provider X ---| |-- N's Service Provider --| depicts the Inter-domain Scenario where the new device N attaches to a router R owned by a different service provider X. The service provider X might offer its own DNS service and a reachable DHCP service. We assume that the service provider X has connectivity to the service provider planning to operate the new device. It should be noted that handing out DHCP options specific to N's service provider via X's DHCP service requires some close coordination between the two parties involved. This might be difficult in practice. A more general alternative might be to have X's service provider establish a tunnel such that the new device logically appears to be part of N's service provider network. In both scenarios, the new device N is either directly reachable or it may be behind a middlebox such as a Network Address Translator (NAT) or a firewall. Middleboxes may impose restrictions on which party is able to initiate communication. As detailed in , it is often desirable to allow device-initiated connections.

We introduce a model of the configuration process in order to identify the parts that have well-known solutions. The remainder may be worth studying to see if the industry can agree on a solution. Some basic terminology is needed for the discussion. Depending on the implementation, let us agree that "configuration data" consist of software and sets of configured parameters in some combination. This includes firmware, licenses, certificates, and other configuration data. Also, the system that provides the configuration data is called the "configuration server". Finally, the term "joining device" is used to denote a network device that is in the process of being incorporated into the network. Broadly speaking, the configuration process can be broken into five phases: Pre-configuration: configuration carried out either by the vendor or by the service provider prior to physical installation. One possible example is the pre-configuration of certificates or licenses or specific firmware. Bootstrapping: the portion of the process from the time that physical installation is complete until a secure connection is established between the joining device and the configuration server. Initial configuration: downloading of the configuration data that the joining device needs to carry out its function in the network. Configuration auditing: tracking image versions and configuration parameters for each network device and verifying that the installed configuration data matches the physical installation, the network plan, and the records of what data was downloaded. It is possible that an initial audit of the physical installation is done before initial configuration, so that the validity of the intended download can be verified. Configuration update: transferring configuration data to a fully configured and operating device from time to time as the need arises.

This memo identifies a specific requirement for pre-configuration of an invariant device identity and authentication-related material in the form of pre-shared secrets or certificates. There is, as one alternative, also a requirement for pre-configuration of information that permits the joining device to discover the address of the configuration server. Note that pre-configuration may be carried out on the joining device itself or it may be provided to the joining device during the deployment process via pluggable memory cards or nearfield communication.

deals with the process of security bootstrapping, with particular emphasis on the requirements for highly resource-constrained devices. The document makes a distinction between a data channel, which is used during network operation, and a control channel, which is used during bootstrapping. While both channels can be the same physical channel, they can also be different (e.g., a wireless access point using an infrared control channel to receive bootstrapping information). The draft discusses a number of possible security bootstrapping protocols for resource constrained devices that can be executed in several bootstrapping rounds and can be adapted to the specific contexts in terms of the resources available within individual devices and for the network as a whole. For network devices in service provider networks or large enterprise networks, bootstrapping consists of several stages: establishment of link layer connectivity with neighbouring nodes; acquisition of IP addresses and basic routing information; discovery of the configuration server; establishment of a secure channel to the configuration server. Each of these stages is further discussed below.

The protocol aspects of this phase are out of scope, since it involves non-IETF protocols only. While some link-layer technologies may provide authentication and access control, this cannot be assumed to be available in the general case.

For IPv4, DHCPv4 is widely deployed and the usual way to obtain an IPv4 address, the IPv4 address of a link- local router and the IPv4 address of a DNS server. For IPv6, a choice has to be made between stateful DHCPv6 versus stateless DHCPv6 combined with stateless address autoconfiguration . In the latter case, DHCPv6 is needed to configure parameters such as DNS server addresses. A routing advertisement option to configure the IPv6 address of a DNS server as part of the stateless address autoconfiguration is defined in . Some security protection is provided in this stage by using DHCP authentication . However, security of the configuration process as a whole has to be assured by other means. This is discussed further below. Currently the lack of a stable identifier for use in DHCPv6 messaging is an impediment to authentication of the joining device. discusses the problems with the current DHCPv6 identifiers (DUIDs) and proposes a new form that could be a more stable alternative. A joining device can also choose to use a pre-configured IP address, a pre-configured link-local router address and a pre- configured DNS server address. This pre-configuration may be hard wired into the device or provided by a pluggable memory card or nearfield communication. However, a static pre-configuration hard- wires assumption about the network a devices operates in and is therefore brittle and not recommended.

Four alternatives are available for finding the configuration server: pre-configuration; DHCP configuration; Service Location Protocol ; or DNS service discovery using DNS SRV records . Pre-configuration of an IP address is brittle and not recommended unless the IP address is used as an anycast address. In the case of an IP anycast address, the routing system will select one out of an anycast cluster of configuration servers the devices connects to. For this to work well, all configuration servers in the anycast cluster should provide the same configuration data. The pre-configuration of a Uniform Resource Identifier (URI) or fully qualified domain name (FQDN) is a slightly better approach than pre-configuring non-anycast IP addresses since this allows for a limited dynamic mapping of the name to an IP address. One variant that has been suggested is to burn the URI of a vendor server into the device's firmware along with a device identifier, and have that server redirect to the URI of the service provider's configuration server based on the device identity. Such an approach requires that the device vendor's redirection server is always reachable, that the device vendor offers such a redirection service for the lifetime of their devices and that service providers are able to update the URI of the service provider's redirection server. Furthermore, this approach can lead to problems if certificates are used to authenticate the involved parties if a service provider tries to prevent the usage of a vendor's redirection service. Finally, this approach also requires a trust relationship between the vendor and the service provider and agreement on a protocol to update the redirect information on the vendor's server. As a consequence of these considerations, using this approach is not recommended. DHCP configuration can use the usual DHCP options and is technically straightforward since DHCP is widely used by end user devices to obtain basic configuration information. There is, however, no standardized DHCP option to communicate the address of a configuration server. The Service Location Protocol (SLP) has seen some usage to locate services such as printers or file system shares. Usage of SLP to locate configuration servers requires to define a new service template . The use of DNS SRV records requires the joining device to obtain the correct domain suffix first, presumably from DHCP or via Routing Advertisements in the case of IPv6 or pre-configuration. A service type for the desired configuration protocol would have to be defined in the DNS for the purpose. See Section 3.3 of for a discussion of the corresponding discovery process for CAPWAP. The Inter-domain Scenario requires that the DHCP server or the SLP server of service provider X's network is able to provide the correct information to the joining devices. To accomplish this, the discovery servers need to be able to match a device identification against a list of possible configuration servers. Furthermore, there needs to be a mechanism for the service provider operating the joining device to provision the configuration server's address, e.g., by using an extension of the Extensible Provisioning Protocol (EPP) . However, if the joining device has pre- configured information about the name of the service provider's network, DNS SRV records may be queried after obtaining IP connectivity, avoiding the need to provision information in service provider X's network.

It is essential that the configuration server and the joining device authenticate themselves to each other, since the steps leading up to this point in the process may not be fully secure. This raises two issues: how the joining device identifies itself, and how authentication takes place. It seems best if the device has an invariant identity built in and accessible to whatever operating system is running on it. provides such an identity in the form of a Universally Unique IDentifier (UUID). The vendor should make that identity available in a form that can be read and transferred into a database accessible to the configuration server along with the associated configuration data in advance of the bootstrapping stage (e.g., in bar-coded format on the device packaging). Serial numbers may be used for identification purposes if UUIDs are not available. However, serial numbers often encode information such as model-numbers or manufacturing dates. Hence, it is not recommended to pass serial-numbers in the clear for security reasons. Similar precautions apply to Common Language Equipment Identifier (CLEI) codes that encode information about properties of the device. This leaves the mutual authentication process itself. This has two aspects: the security protocol used to perform authentication, and initial keying methodology. The security protocol is tied together with the choice of configuration data transport, but the basic choices are: IP Security (IPsec) ; Transport Layer Security (TLS) ; Datagram Transport Layer Security (DTLS) ; Secure Shell (SSH) , , , and ; and SNMPv3's User-based Security Model (USM) . For initial keying methodology, the two basic choices are between pre-shared secrets and certificates. All of the security protocols listed above except USM support both methods. USM supports pre-shared secrets only. The usual concern with pre-shared secrets is scalability. In the bootstrapping case, the scale of operation required is linear with the number of devices to be configured, so it would definitely be a feasible approach if connection to the configuration system were the only consideration. The most likely procedure would be for the secret to be configured in the device during pre-configuration and also captured in a database along with the device identity, for use by the configuration server. The problem with the use of pre-shared secrets is that the device needs to authenticate itself at an earlier stage, while it is establishing communications with its neighbours and acquiring IP addresses. It seems undesirable to use the same secret that is used to authenticate the device to the configuration server for that purpose as well, on the basic principle of limiting the potential damage from disclosure of a particular key. This need for additional pre-shared secrets argues for consideration of certificates as an alternative. One issue for certificates is where the trust anchor resides. It seems logical that it should reside with the service provider rather than the vendor, to make it easy to install equipment from multiple vendors. On that basis, pre-configuration requires service provider input. On the other hand, if devices are drop-shipped to the destination from the vendor, having the trust anchor reside with the vendor might be acceptable as well. CAPWAP (Section 2.4.4.3 of ) makes use of the Extended Key Usage (EKU) certificate extension to distinguish certificates identifying the Access Controllers (i.e., the configuration servers in the CAPWAP case) from the Wireless Transfer Points (the configured devices in the CAPWAP case). Thought should be given to whether such distinctions are required in the general case of network device configuration. CAPWAP (Section 12.8 of ) also discusses the use of the Common Name rather than SubjectAltName field of the certificate to carry device identity, due to lack of a Uniform Resource Name (URN) specification allowing the use of SubjectAltName to carry MAC addresses. This encoding of device identifiers in certifications needs to be investigated further if a new form of device unique identity is used, as discussed above. Middleboxes such as NATs or firewalls may impose restriction on which party is able to initiate communication. In the common case of NATs in IPv4 access networks, communication can only be established from the device to the configuration server. Not all secure transports, in particular those where authentication is not symmetric, support this "call home" mode of operation. A recent proposal to reverse the establishment of the TCP connection for SSH can be found in .

As mentioned at the beginning, the configuration data being downloaded may be a combination of software/firmware and configuration parameters. Some of the data will be vendor-specific and not subject to standardization. It appears that there is a continuing debate on whether the configuration data should be pushed to the joining device or whether the device should pull the configuration data from the configuration server. In the latter case, the device needs to know about the existence of the data and the path to reach it before it can act. One way to acquire this information is through DHCP. DHCPv4 has provided the necessary options from its beginnings, inheriting them from BOOTP. They have been recently added to DHCPv6 . Protocols that can transport configuration data can be classified as follows: The first class consists of generic file transfer protocols that can carry configuration data serialized into configuration files. The second class consists of protocols that manipulate structured configuration data directly. The structure of the configuration data is defined by some data model. In the first class, we find the following file transfer protocols: The File Transfer Protocol (FTP) can be used to move files containing configuration data. It can be secured by running FTP over TLS . The Trivial File Transfer Protocol (TFTP) has been used extensively to load boot images over the network. However, it does not provide security and the only option is to rely on IP layer security (IPsec). The Hypertext Transfer Protocol (HTTP) can be used to transfer documents containing configuration data. It is commonly secured by running HTTP over TLS , . The SSH File Transfer Protocol (SFTP) provides roughly the same services as FTP but runs over SSH and thus utilizes the security services provided by SSH. UNIX utilities to transfer files such as RCP and SCP provide limited flexibility and they differ in their degree of integration with SSH. The Control And Provisioning of Wireless Access Points (CAPWAP) protocol can be used to control the download of images. CAPWAP can be secured by running CAPWAP over DTLS. In the second class, we find the following configuration protocols: Version 3 of the Simple Network Management Protocol (SNMPv3) can be used to manipulate MIB objects and to carry event notifications. SNMPv3 has its own security protocol (USM) but can also run over the secure transports SSH , TLS, or DTLS . The Common Open Policy Service for Policy Provisioning protocol (COPS-PR) was designed to provision structured policy information from a Policy Decision Point (PDP) to a Policy Enforcement Point (PEP). The COPS protocol provides an integrity object that can achieve authentication, message integrity, and replay prevention. Optionally, COPS and COPS-PR can run over TLS. The NETCONF protocol provides mechanisms to install, manipulate, and delete the configuration of network devices. A protocol extension provides an asynchronous event notification delivery mechanism . NETCONF by default runs over SSH but can also run over transports secured by TLS. The Control And Provisioning of Wireless Access Points protocol (CAPWAP) supports the discovery of so called Access Controller (AC) by Wireless Termination Points (WTPs) and the configuration of WTPs by an AC. While CAPWAP can be extended to configure other devices, its main focus are WTPs. The CAPWAP protocol is protected by using DTLS after the discovery phase. lists the protocols plus their basic properties while lists the security options available for each protocol. Transport Data Transfer Model FTP Push or pull of (configuration) files TFTP Push or pull of (configuration) files HTTP Push or pull of (configuration) files SFTP Push or pull of (configuration) files RCP Push or pull of (configuration) files SCP Push or pull of (configuration) files CAPWAP AC pushes configuration parameters, WTP pulls software SNMPv3 Push of structured configuration parameters, event notifications COPS-PR Push of structured policy information NETCONF Push of structured configuration data, event notifications Transport IPsec TLS DTLS SSH Other FTP + + TFTP + HTTP + + SFTP + + RCP + SCP + + CAPWAP + + SNMPv3 + + + + USM COPS-PR + + NETCONF + + + SNMPv3, NETCONF, and COPS-PR carry structured data specified in pre-defined data models. SNMPv3 and COPS-PR have size limitations on the data objects and thus make the transport of larger software images difficult. NETCONF does not suffer from hard size restrictions and can in principle carry software images inline. However, there is currently no work in progress to standardize the transfer of software images over NETCONF. CAPWAP combines the functions of configuration parameter transport and software download. The parameter transport aspect lacks the generality offered by SNMP, NETCONF, and COPS-PR, since the parameters are specified within the protocol specification itself. The remaining transports are independent of the nature of the information being transferred.

To complete the process, it must be possible to audit the configuration status of the device in some detail. This is likely to begin even before all the configuration data has been downloaded. For instance, configuration management may wish to collect basic information such as the MAC addresses of the device's interfaces, the link-local addresses assigned to them, and similar information for the neighbours of the joining device. SNMP and SNMP MIB modules are obviously one way to collect this information. NETCONF is an alternative, but the necessary data models have to be defined. YANG modules for NETCONF can be generated from existing SNMP MIB modules by translating the SNMP modules into YANG modules . Another important auditing activity is the analysis of system events. The SYSLOG protocol is widely used for this purpose but SNMPv3 and NETCONF can ship event notifications as well. Translations of SNMP notifications into structured SYSLOG messages and vice versa do exist , . NETCONF can carry SYSLOG content as well . NETCONF provides generic notifications that help with tracking configuration changes . Similar standardized configuration change notifications do not exist for SNMP or SYSLOG.

Configuration updates can in principle be handled with the same protocol that delivered the initial configuration. However, in some deployments, the mechanism used for initial configuration might be different. An advantage of NETCONF over SNMPv3 and CAPWAP in the context of configuration updates is the support of concurrent updates through explicit locking mechanisms and the support of network wide configuration change transactions through the confirmed commit capability.

This document discussed the automated configuration of devices in large IP networks. Several gaps were identified requiring further specification: Definition of a DHCP option to provide the IPv4/IPv6 address of a configuration server. Such an option allows a joining device to pickup the configuration server's address as part of the DHCP exchange. This is particularly interesting for Intra-domain Scenarios. Definition of DNS SRV records for locating configuration servers. Using SRV records, a joining device can lookup the configuration server's address in the DNS. This is particularly useful in an Inter-domain Scenario. Definition of a SLP template for discovering configuration servers. Such a template is useful only in environments where SLP is used also for other purposes. Definition of NETCONF data models to support the download /update of software images through NETCONF. Definition of NETCONF data models for collecting basic system information and integrity information (e.g., checksums of software images). Some management protocols lack a mechanisms for devices to initiate a secure communication channel with a management system ("call home").

The security of a configuration management solution is of crucial importance. discusses the security options of several protocols that might be used. The relevant protocol definitions should be consulted to learn more about the specific security aspects of the various protocols. It should be noted that some steps in the described process, in particular the bootstrapping phase, may not be secure and it is thus important to verify the identity of the device and the identity of the configuration server when a secure connection to a configuration server is established. Usage of IPsec, which focuses on securing the IP layer, may not be sufficient for this. During the choice of protocols, the available security mechanisms and the required key management infrastructures may play a major role in the selection of protocols. Easy integration into existing Authentication, Authorization and Accounting (AAA) infrastructures can significantly reduce the operational costs associated with the security management of the configuration system. While discusses security bootstrapping mechanisms in the context of constrained devices, many of the mechanisms are also applicable for bootstrapping security in normal devices. Finally, discusses security capabilities for customer premises equipment providing residential IPv6 Internet service.

This memo includes no request to IANA.

Thanks to Ronald Bonica, Mehmet Ersue, Wesley George, Yiu Lee, Christopher Liljenstolpe, Kent Watsen, and Cathy Zhou for their comments during the preparation of this memo.