Beware, you have written :
if($key){
unset($this->observers[$key]);
}
When this should be :
if(false !== $key){
unset($this->observers[$key]);
}
If the observer you want to delete is the first in your array, you will never delete it because the key would equal 0 and 0 == false as you know.