HA Hadoop -ApacheCon talk

Editor's Notes

• #3: Once you adopt Hadoop, it can rapidly become the biggest central storage point of data in an organisation. At which point you start caring about how well it looks after your data. this talk aims to answer these questions
• #4: HDFS is built on the concept that in a large cluster, disk failure is inevitable. The system is designed to change the impact of this from the beeping of pagers to a background hum.Akey part of the HDFS design: copying the blocks across machines means that the loss of a disk, server or even entire rack keeps the data available.
• #5: There's lots of checksumming going on of the data to pick up corruption -CRCs created at write time (and even verified end-to-end in a cross-machine write), scanned on read time.
• #6: Rack failures can generate a lot of replication traffic, as every block that was stored in the rack needs to be replicated at least once. The replication still has to follow the constraints of no more than one block copy per server. Much of this traffic is intra-rack, but every block which already has 2x replicas on a single rack will be replicated to another rack if possible.This is what scares ops team. Important: there is no specific notion of "mass failure" or "network partition". Here HDFS only sees that four machines have gone down.
• #8: When the NameNode fails, the cluster is offline.client applications -including the MapReduce layer and HBase see this, and fail.
• #9: When the NN fails, it's obvious to remote applications, which fail, and to the ops team, which have to fix it. It also adds cost to the system. Does it happen often? No, almost never, but having the ops team and hardware ready for it is the cost
• #15: VMWare monitioring is "canary based" -when the bird stops singing, the service is in trouble.To implement this we have a new service running in the VM that monitors service health, and stops singing when it decides the service is down.If the service, the OS or the monitor fails, VSphere reacts by restarting the VM. If the host server fails, all VMs on it get restarted elsewhereOne troublespot: hung processes. Socket blocks. Something needs to detect that and make a decision as to when this is a service hang vs a long GC. There is no way pre-Java7 to know this, so it's just down to timeouts.
• #20: One question is "how long does this take"For clusters of under a few hundred machines, with not that much of an edit log to replay, failure detection dominates the time. vSphere is the slowest there as it has to notice that the monitor has stopped sending heartbeats (~60s), then boot the OS (~60s) before bringing up the NN. This is why it's better for smaller clusters.LinuxHA fails over faster; if set to poll every 10-15 s then failover begins within 20s of the outage.
• #29: The risk here isn't just the failover, it's all the other changes that went into HDFS 2; the bug reporting numbers for HDFS 2 are way down on the previous releases, which means that it was either much better, or hasn't been tested enough.the big issue here is that the data in your HDFS cluster may be one of the most valuable assets of an organization, you can't afford to lose it. It's why it's good to be cautious with any filesystem, be it linux or a layer above.
• #33: This is the state machine used to monitor a service. All the time the state machine is not terminated heartbeats are being sent to vSphere every 10s.Underneath that we have a model of a service blocking for (remote) services before even starting, booting -which waits for all probes to go live in a bounded time, and fails fast if any live probe revertslive - everything must respond, timeouts of a probe are picked up too.halted -a graceful unhook from vSphere, so that heartbeat stopping isn't picked up