Monitor配置手册Monitor Config Reference?Understanding how to configure a Ceph Monitor is an important part of building a reliable Ceph Storage Cluster. All Ceph Storage Clusters have at least one monitor. A monitor configuration usually remains fairly consistent, but you can add, remove or replace a monitor in a cluster. See Adding/Removing a Monitor and Add/Remove a Monitor (ceph-deploy) for details. Background?# Ceph client可以通过向ceph monitors获取 cluster map，来定位所有ceph monitors，osd daemons和 metadata servers。 Ceph Monitors maintain a “master copy” of the cluster map, which means a Ceph Client can determine the location of all Ceph Monitors, Ceph OSD Daemons, and Ceph Metadata Servers just by connecting to one Ceph Monitor and retrieving a current cluster map. # Ceph Client 可以直接和 ODS 通信。 Before Ceph Clients can read from or write to Ceph OSD Daemons or Ceph Metadata Servers, they must connect to a Ceph Monitor first. With a current copy of the cluster map and the CRUSH algorithm, a Ceph Client can compute the location for any object. The ability to compute object locations allows a Ceph Client to talk directly to Ceph OSD Daemons, which is a very important aspect of Ceph’s high scalability and performance. See Scalability and High Availability for additional details. The primary role of the Ceph Monitor is to maintain a master copy of the cluster map. Ceph Monitors also provide authentication and logging services. Ceph Monitors write all changes in the monitor services to a single Paxos instance, and Paxos writes the changes to a key/value store for strong consistency. Ceph Monitors can query the most recent version of the cluster map during sync operations. Ceph Monitors leverage the key/value store’s snapshots and iterators (using leveldb) to perform store-wide synchronization.
Deprecated since version version: 0.58 In Ceph versions 0.58 and earlier, Ceph Monitors use a Paxos instance for each service and store the map as a file. Cluster Maps?The cluster map is a composite of maps, including the monitor map, the OSD map, the placement group map and the metadata server map. The cluster map tracks a number of important things: which processes are in the Ceph Storage Cluster; which processes that are in the Ceph Storage Cluster are up and running or down; whether, the placement groups are active or inactive, and clean or in some other state; and, other details that reflect the current state of the cluster such as the total amount of storage space, and the amount of storage used. When there is a significant change in the state of the cluster–e.g., a Ceph OSD Daemon goes down, a placement group falls into a degraded state, etc.–the cluster map gets updated to reflect the current state of the cluster. Additionally, the Ceph Monitor also maintains a history of the prior states of the cluster. The monitor map, OSD map, placement group map and metadata server map each maintain a history of their map versions. We call each version an “epoch.” When operating your Ceph Storage Cluster, keeping track of these states is an important part of your system administration duties. See Monitoring a Cluster and Monitoring OSDs and PGs for additional details. Monitor Quorum Monitor的数量?Our Configuring ceph section provides a trivial Ceph configuration file that provides for one monitor in the test cluster. A cluster will run fine with a single monitor; however, a single monitor is a single-point-of-failure. To ensure high availability in a production Ceph Storage Cluster, you should run Ceph with multiple monitors so that the failure of a single monitor WILL NOT bring down your entire cluster. Ceph 使用Paxos保持monitor的cluster map的一致性。 一致性要求， 大部分的monitor认同cluster map。 When a Ceph Storage Cluster runs multiple Ceph Monitors for high availability, Ceph Monitors use Paxos to establish consensus about the master cluster map. A consensus requires a majority of monitors running to establish a quorum for consensus about the cluster map (e.g., 1; 2 out of 3; 3 out of 5; 4 out of 6; etc.). Consistency?When you add monitor settings to your Ceph configuration file, you need to be aware of some of the architectural aspects of Ceph Monitors. Ceph imposes strict consistency requirements for a Ceph monitor when discovering another Ceph Monitor within the cluster. Whereas, Ceph Clients and other Ceph daemons use the Ceph configuration file to discover monitors, monitors discover each other using the monitor map (monmap), not the Ceph configuration file. A Ceph Monitor always refers to the local copy of the monmap when discovering other Ceph Monitors in the Ceph Storage Cluster. Using the monmap instead of the Ceph configuration file avoids errors that could break the cluster (e.g., typos in ceph.conf when specifying a monitor address or port). Since monitors use monmaps for discovery and they share monmaps with clients and other Ceph daemons, the monmap provides monitors with a strict guarantee that their consensus is valid. Strict consistency also applies to updates to the monmap. As with any other updates on the Ceph Monitor, changes to the monmap always run through a distributed consensus algorithm called Paxos. The Ceph Monitors must agree on each update to the monmap, such as adding or removing a Ceph Monitor, to ensure that each monitor in the quorum has the same version of the monmap. Updates to the monmap are incremental so that Ceph Monitors have the latest agreed upon version, and a set of previous versions. Maintaining a history enables a Ceph Monitor that has an older version of the monmap to catch up with the current state of the Ceph Storage Cluster. If Ceph Monitors discovered each other through the Ceph configuration file instead of through the monmap, it would introduce additional risks because the Ceph configuration files aren’t updated and distributed automatically. Ceph Monitors might inadvertently use an older Ceph configuration file, fail to recognize a Ceph Monitor, fall out of a quorum, or develop a situation where Paxos isn’t able to determine the current state of the system accurately. Bootstrapping Monitors?In most configuration and deployment cases, tools that deploy Ceph may help bootstrap the Ceph Monitors by generating a monitor map for you (e.g., ceph-deploy, etc). A Ceph Monitor requires a few explicit settings:
For additional details on bootstrapping, see Bootstrapping a Monitor. Configuring Monitors?To apply configuration settings to the entire cluster, enter the configuration settings under [global]. To apply configuration settings to all monitors in your cluster, enter the configuration settings under [mon]. To apply configuration settings to specific monitors, specify the monitor instance (e.g., [mon.a]). By convention, monitor instance names use alpha notation. [global] [mon] [mon.a] [mon.b] [mon.c] Minimum Configuration?The bare minimum monitor settings for a Ceph monitor via the Ceph configuration file include a hostname and a monitor address for each monitor. You can configure these under [mon] or under the entry for a specific monitor. [mon]
mon host = hostname1,hostname2,hostname3
mon addr = 10.0.0.10:6789,10.0.0.11:6789,10.0.0.12:6789
[mon.a]
host = hostname1
mon addr = 10.0.0.10:6789
See the Network Configuration Reference for details. Note This minimum configuration for monitors assumes that a deployment tool generates the fsid and the mon. key for you. Once you deploy a Ceph cluster, you SHOULD NOT change the IP address of the monitors. However, if you decide to change the monitor’s IP address, you must follow a specific procedure. See Changing a Monitor’s IP Address for details. Cluster ID?Each Ceph Storage Cluster has a unique identifier (fsid). If specified, it usually appears under the [global] section of the configuration file. Deployment tools usually generate the fsid and store it in the monitor map, so the value may not appear in a configuration file. The fsid makes it possible to run daemons for multiple clusters on the same hardware. fsid
Note Do not set this value if you use a deployment tool that does it for you. Initial Members?We recommend running a production Ceph Storage Cluster with at least three Ceph Monitors to ensure high availability. When you run multiple monitors, you may specify the initial monitors that must be members of the cluster in order to establish a quorum. This may reduce the time it takes for your cluster to come online. [mon]
mon initial members = a,b,c
mon initial members
Note A majority of monitors in your cluster must be able to reach each other in order to establish a quorum. You can decrease the initial number of monitors to establish a quorum with this setting. Data?Ceph provides a default path where Ceph Monitors store data. For optimal performance in a production Ceph Storage Cluster, we recommend running Ceph Monitors on separate hosts and drives from Ceph OSD Daemons. As leveldb is using mmap() for writing the data, Ceph Monitors flush their data from memory to disk very often, which can interfere with Ceph OSD Daemon workloads if the data store is co-located with the OSD Daemons. In Ceph versions 0.58 and earlier, Ceph Monitors store their data in files. This approach allows users to inspect monitor data with common tools like ls and cat. However, it doesn’t provide strong consistency. In Ceph versions 0.59 and later, Ceph Monitors store their data as key/value pairs. Ceph Monitors require ACID transactions. Using a data store prevents recovering Ceph Monitors from running corrupted versions through Paxos, and it enables multiple modification operations in one single atomic batch, among other advantages. Generally, we do not recommend changing the default data location. If you modify the default location, we recommend that you make it uniform across Ceph Monitors by setting it in the [mon] section of the configuration file. mon data
Storage Capacity?When a Ceph Storage Cluster gets close to its maximum capacity (i.e., mon osd full ratio), Ceph prevents you from writing to or reading from Ceph OSD Daemons as a safety measure to prevent data loss. Therefore, letting a production Ceph Storage Cluster approach its full ratio is not a good practice, because it sacrifices high availability. The default full ratio is .95, or 95% of capacity. This a very aggressive setting for a test cluster with a small number of OSDs. Tip When monitoring your cluster, be alert to warnings related to the nearfull ratio. This means that a failure of some OSDs could result in a temporary service disruption if one or more OSDs fails. Consider adding more OSDs to increase storage capacity. A common scenario for test clusters involves a system administrator removing a Ceph OSD Daemon from the Ceph Storage Cluster to watch the cluster rebalance; then, removing another Ceph OSD Daemon, and so on until the Ceph Storage Cluster eventually reaches the full ratio and locks up. We recommend a bit of capacity planning even with a test cluster. Planning enables you to gauge how much spare capacity you will need in order to maintain high availability. Ideally, you want to plan for a series of Ceph OSD Daemon failures where the cluster can recover to an active + clean state without replacing those Ceph OSD Daemons immediately. You can run a cluster in an active + degraded state, but this is not ideal for normal operating conditions. The following diagram depicts a simplistic Ceph Storage Cluster containing 33 Ceph Nodes with one Ceph OSD Daemon per host, each Ceph OSD Daemon reading from and writing to a 3TB drive. So this exemplary Ceph Storage Cluster has a maximum actual capacity of 99TB. With a mon osd full ratio of 0.95, if the Ceph Storage Cluster falls to 5TB of remaining capacity, the cluster will not allow Ceph Clients to read and write data. So the Ceph Storage Cluster’s operating capacity is 95TB, not 99TB.
It is normal in such a cluster for one or two OSDs to fail. A less frequent but reasonable scenario involves a rack’s router or power supply failing, which brings down multiple OSDs simultaneously (e.g., OSDs 7-12). In such a scenario, you should still strive for a cluster that can remain operational and achieve an active + clean state–even if that means adding a few hosts with additional OSDs in short order. If your capacity utilization is too high, you may not lose data, but you could still sacrifice data availability while resolving an outage within a failure domain if capacity utilization of the cluster exceeds the full ratio. For this reason, we recommend at least some rough capacity planning. Identify two numbers for your cluster:
If you divide the total capacity of your cluster by the number of OSDs in your cluster, you will find the mean average capacity of an OSD within your cluster. Consider multiplying that number by the number of OSDs you expect will fail simultaneously during normal operations (a relatively small number). Finally multiply the capacity of the cluster by the full ratio to arrive at a maximum operating capacity; then, subtract the number of amount of data from the OSDs you expect to fail to arrive at a reasonable full ratio. Repeat the foregoing process with a higher number of OSD failures (e.g., a rack of OSDs) to arrive at a reasonable number for a near full ratio. [global]
mon osd full ratio = .80
mon osd nearfull ratio = .70
mon osd full ratio
mon osd nearfull ratio
Tip If some OSDs are nearfull, but others have plenty of capacity, you may have a problem with the CRUSH weight for the nearfull OSDs. Heartbeat?Ceph monitors know about the cluster by requiring reports from each OSD, and by receiving reports from OSDs about the status of their neighboring OSDs. Ceph provides reasonable default settings for monitor/OSD interaction; however, you may modify them as needed. See Monitor/OSD Interaction for details. Monitor Store Synchronization ( ceph monitor之间的同步)# ceph monitor通过和周围的monitor比较epoch numbers来检查是否有更新版本的cluster map。 # monitor的三个角色：leader, provider, requester
# leader 将同步cluster map的任务委托给provider，防止leader有过多的同步请求，影响性能 These roles enable a leader to delegate synchronization duties to a provider, which prevents synchronization requests from overloading the leader–improving performance. In the following diagram, the requester has learned that it has fallen behind the other monitors. The requester asks the leader to synchronize, and the leader tells the requester to synchronize with a provider.
Synchronization always occurs when a new monitor joins the cluster. During runtime operations, monitors may receive updates to the cluster map at different times. This means the leader and provider roles may migrate from one monitor to another. If this happens while synchronizing (e.g., a provider falls behind the leader), the provider can terminate synchronization with a requester. # 一旦同步完成，ceph会发起集群的调整动作。 Once synchronization is complete, Ceph requires trimming across the cluster. Trimming requires that the placement groups are active + clean. mon sync trim timeout
mon sync heartbeat timeout
mon sync heartbeat interval
mon sync backoff timeout
mon sync timeout
mon sync max retries
mon sync max payload size
mon accept timeout
paxos propose interval
paxos min wait
paxos trim tolerance
paxos trim disabled max versions
mon lease
mon lease renew interval
mon lease ack timeout
mon min osdmap epochs
mon max pgmap epochs
mon max log epochs
Slurp?In Ceph version 0.58 and earlier, when a Paxos service drifts beyond a given number of versions, Ceph triggers the slurp mechanism, which establishes a connection with the quorum Leader and obtains every single version the Leader has for every service that has drifted. In Ceph versions 0.59 and later, slurp will not work, because there is a single Paxos instance for all services. Deprecated since version 0.58. paxos max join drift
mon slurp timeout
mon slurp bytes
Clock 时钟?Ceph daemons pass critical messages to each other, which must be processed before daemons reach a timeout threshold. If the clocks in Ceph monitors are not synchronized, it can lead to a number of anomalies. For example:
See Monitor Store Synchronization and Slurp for details. Tip You SHOULD install NTP on your Ceph monitor hosts to ensure that the monitor cluster operates with synchronized clocks. Clock drift may still be noticeable with NTP even though the discrepancy isn’t yet harmful. Ceph’s clock drift / clock skew warnings may get triggered even though NTP maintains a reasonable level of synchronization. Increasing your clock drift may be tolerable under such circumstances; however, a number of factors such as workload, network latency, configuring overrides to default timeouts and the Monitor Store Synchronization settings may influence the level of acceptable clock drift without compromising Paxos guarantees. Ceph provides the following tunable options to allow you to find acceptable values. clock offset
Deprecated since version 0.58. mon tick interval
mon clock drift allowed
mon clock drift warn backoff
mon timecheck interval
Client?mon client hunt interval
mon client ping interval
mon client max log entries per message
mon client bytes
Miscellaneous 其它细节?mon max osd
mon globalid prealloc
mon sync fs threshold
mon subscribe interval
mon stat smooth intervals
mon probe timeout
mon daemon bytes
mon max log entries per event
mon osd prime pg temp
mon osd prime pg temp max time
|
|
|
