| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-30 | |||
| 20:31:17 | s10 | I had to rename node in compute_nodes, services, nova_api, nova.instances, ml2_neutron port bindings... | |
| 20:31:39 | melwitt | yeah, what you're talking about it migrating things manually, so if you're doing that, you'd have to figure out where all you'd need to update database records manually. we don't recommend this | |
| 20:32:18 | sean-k-mooney | s10: you also would have to update placement and the cells dbs in addtion to the normal nova one | |
| 20:33:45 | s10 | Yes, and this is too inconvenient. Would it be better to have some time of evacuate command, that would do all this steps, but without rebuilding instances? | |
| 20:34:22 | sean-k-mooney | s10: the simpelste thing to do would be to pull the disk form the failed node. put them in a working node. migrate off the workloads then rename the new node to whatever it should be called give its physical location | |
| 20:39:09 | melwitt | s10: maybe. I don't recall anyone proposing a spec like that. and I don't know if there are reasons why we don't have a command like that | |
| 20:39:54 | s10 | This is how we do it now. But what if system disk (/) is on small m.2 ssd inside the server. So after the failure of this host we couldn't just take disks of the server, we need to pull server out of rack, open it, take this m.2 out and then put it in another server. | |
| 20:40:27 | melwitt | dansmith: as the evacuate guru, do you happen to know why we don't have a command like "migrate all" from a fenced compute node that's forced down, similar to evacuate but moves disks too? are there tech difficulties around it? I don't remember if it's been proposed before | |
| 20:41:13 | s10 | All this operation will take much more time than just execution of evacuate --force --target-host command to move instances to the reserved compute node, that could be even be powered on all the time | |
| 20:42:58 | dansmith | melwitt: evacuate only works if the node is completely dead | |
| 20:43:11 | dansmith | melwitt: it's destructive, doesn't preserve anything | |
| 20:43:33 | cfriesen__ | melwitt: in my experience, the people that actually use "evacuate" will often be forcing down failed compute nodes and immediately evacuating the instances to new hosts. | |
| 20:43:37 | dansmith | melwitt: marking the node down is the shortcut to the "is down" timer if you know better (i.e. you've fenced it yourself) | |
| 20:44:24 | dansmith | if the node is not dead, then evacuate is not the operation you want | |
| 20:44:25 | cfriesen__ | melwitt: if the node is fenced it's generally because it can't talk to the rest of the system | |
| 20:44:40 | sean-k-mooney | dansmith: evacuate is also only usefull if your instance are using bfv right otherwise it not going to keep any data form the instces correct | |
| 20:45:04 | dansmith | melwitt: http://www.danplanet.com/blog/2016/03/03/evacuate-in-nova-one-command-to-confuse-us-all/ | |
| 20:45:04 | cfriesen__ | sean-k-mooney: it keeps the IP addresses and such, which is useful for eg. heat | |
| 20:45:10 | melwitt | okay, thanks. yeah, so that's why it's a rebuild, because nova can't talk to the compute host at all | |
| 20:45:17 | dansmith | sean-k-mooney: no, it's useful if you aren't on shared storage | |
| 20:45:26 | dansmith | sean-k-mooney: if you care about uuid, networks, metadata, etc | |
| 20:45:31 | dansmith | melwitt: right | |
| 20:45:33 | sean-k-mooney | cfriesen__: true but if you were using local sotrage on the cmpute node its like doing a rebuild | |
| 20:45:33 | melwitt | s10: what is your situation? compute host totally down? because if so, I don't think there's anything we can do about moving disks etc | |
| 20:45:57 | cfriesen__ | sean-k-mooney: sure. but some applications are fine with that | |
| 20:45:58 | dansmith | sean-k-mooney: ...which keeps the ips, like he said :) | |
| 20:46:09 | melwitt | s10: if your compute host is up, you can disable the service (disable new scheduling to it) and then migrate all the instances off | |
| 20:46:12 | melwitt | via the API | |
| 20:46:19 | dansmith | right | |
| 20:46:48 | sean-k-mooney | dansmith: yes that is true. i was ignroing that but i guess that could be usful if you have a statelesee workload. | |
| 20:47:13 | dansmith | sean-k-mooney: or your workload keeps its state elsewhere | |
| 20:47:24 | dansmith | sean-k-mooney: like a volume or a remote database | |
| 20:47:37 | dansmith | your root disk doesn't need to be on a volume | |
| 20:47:44 | s10 | melwitt: this is hypothetical situation, I'm thinking about ways to reduce downtime of instances. Situation is: compute node is down (kernel panic, powered off). I can pull disks with the instances data and put them in another host. But with current nova code I had to move system disks also and wait until system will boot on new hardware. | |
| 20:47:51 | sean-k-mooney | dansmith: sure. i guess i was conflating not bfv with no cinider volumes. | |
| 20:48:28 | cfriesen__ | S10: one possible issue you will run into is that neutron may need to be updated with suitable information as well, you may need to allocate SRIOV devices, the available CPUs on the new host might be different (for pinned CPUs), etc. | |
| 20:48:29 | dansmith | s10: or not store your state in the ephemeral instance root disks in the first place... you know, like cloud and stuff :) | |
| 20:48:58 | cfriesen__ | s10: also, any cinder connections will need to be rebuilt | |
| 20:49:07 | dansmith | and if you're going to, you should probably be using ceph or cinder backing | |
| 20:49:24 | s10 | instance, | |
| 20:49:24 | s10 | self.network_api.migrate_instance_finish(context, | |
| 20:49:24 | s10 | 'dest_compute': self.host, } | |
| 20:49:24 | s10 | migration = {'source_compute': instance.host, | |
| 20:49:24 | s10 | cfriesen: this is not the issue, if recreate: | |
| 20:49:25 | s10 | migration) | |
| 20:49:39 | s10 | This sets ports for instances on new host | |
| 20:49:52 | sean-k-mooney | s10: also if your using routed netwroking you need to make sure your new host is on the same segment. | |
| 20:49:54 | cfriesen__ | I'm talking about allocations, resource tracking, etc | |
| 20:50:29 | cfriesen__ | placement, PCI device and CPU and hugepage claims, etc | |
| 20:50:42 | s10 | dansmith: yes, but what if we want as much IOPS for instances as possible - this could only be archived with local NVMes, so red etc is not and option | |
| 20:51:51 | dansmith | ack, fair enough | |
| 20:52:00 | sean-k-mooney | s10: you dont need to keep all of your data on the nvme drivres. having an nvme backed epheral disk for scratch space or chacing is a more cloud friendly approch | |
| 20:52:16 | dansmith | yup | |
| 20:52:35 | dansmith | there's this triangle.. | |
| 20:53:04 | dansmith | http://katharine-writes.com/wp-content/uploads/2014/03/Quality-Triangle.jpg | |
| 20:53:54 | openstackgerrit | Merged openstack/nova master: Add description for placement 1.26 https://review.openstack.org/587067 | |
| 20:56:13 | sean-k-mooney | dmcache and bcache in the kernel are actully great ways to cache /var/lib/nova/instances mounts from remote network filesystem using local nvmedrives | |
| 20:57:13 | sean-k-mooney | i rembere looking to see if qemu could be configured to cache its remote disks via local fast storage but never found a way | |
| 20:57:30 | dansmith | or just use something capable of doing it natively | |
| 20:57:52 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Retry decorator fix for instances which go into ERROR state during bulk delete https://review.openstack.org/563418 | |
| 20:57:53 | mriedem | jaypipes: melwitt: there you go ^ easy peasy | |
| 20:58:06 | s10 | reserved host is a complete replica of the failed host, same cpus, no pic devices, same memory, in same network segment, only thing that should be changed is the hostname... | |
| 20:58:26 | sean-k-mooney | ya i was trying to see if i could figure out how to cache vms backed by ceph using nvdim persistent memory without modifying qemu | |
| 20:59:18 | dansmith | yeah for transparent backing of block devices, dm-cache is the way | |
| 20:59:51 | melwitt | mriedem: thanks | |
| 21:00:20 | sean-k-mooney | s10: i was pritty sure we had a way to override the hostid of a compute node but i could not find it in the conf. that bacsically all you need. e.g. a way to forst the agent to think its on the old host without actully using the old host name on the phyical host | |
| 21:00:49 | dansmith | ...or just transport the damn system disk :) | |
| 21:00:57 | dansmith | you're walking across the aisle anyway :) | |
| 21:01:32 | s10 | sean-k-mooney: i'm not sure, that we can tell that hostid to the neutron-ovs-agent same way :( | |
| 21:01:40 | sean-k-mooney | dansmith: ya that would likely be quicker then any workaround we can comup with bar adding a compute node uuid to the compute conf | |
| 21:01:58 | sean-k-mooney | s10: the neutron agent definetly supports it | |
| 21:02:07 | sean-k-mooney | its nova that is missing it | |
| 21:02:32 | melwitt | yeah, I was gonna say, given all the things that would need to be updated, copying over the system disk sounds like the easiest and fastest way | |
| 21:03:43 | sean-k-mooney | dansmith: melwitt out of interest is there any reason we could not allow an operator to choos the compute node uuid by setting it in the conf? | |
| 21:04:30 | dansmith | sean-k-mooney: it's internal and more complicated to let them choose | |
| 21:04:43 | dansmith | right now it can't change unless we change it or do things | |
| 21:05:00 | dansmith | I can't really see a compelling reason to let them change it | |
| 21:06:48 | sean-k-mooney | im not saying it a good idea. its a great way to shoot yourself in the foot and mess up our resouce tracking. but it would make s10 usecase doable without a lot of db hacking | |
| 21:07:42 | s10 | Or can I just change compute_driver to Fake on reserved node, execute evacuate to this node and then just put disks from old host, and than change config back and restart nova-compute... | |
| 21:08:47 | s10 | compute_driver=fake.FakeDriver. and than back to libvirt.LibvirtDriver. | |
| 21:12:42 | sean-k-mooney | s10: that will likely take longer then pulling the os driver if your goal is to minimise guest downtime | |
| 21:17:54 | s10 | sean-k-mooney: nova host-evacuate --target-host --force can be executed immediately after host failure. Same time, while instances are being "evacuated" to the fake host, support will pull disks from failed server and put them in the reserved, right after that we stop nova-compute, mount disks from old host, change nova config, start it again, and finally nova reboot for all instances. This will take much less time than moving system m.2 disk | |
| 21:19:17 | s10 | Only physical operation that should be performed is moving disks. Other way requires opening the lid on the top of the server with dismounting it out of the rack. | |
| 21:19:36 | mriedem | i highly suggest not using the --force option | |
| 21:19:51 | mriedem | since it bypasses the scheduler and can mess things up if the target host is not actually a proper fit for the instance | |
| 21:20:05 | dansmith | also, | |
| 21:20:17 | dansmith | the fake driver will not have the right inventory in placement | |
| 21:20:31 | dansmith | so your scheduler or conductor-based allocations would be all fubar | |
| 21:20:33 | sean-k-mooney | mriedem: in this case s10 is forcing a migrate to a fake libvirt node so not sure it applies | |
| 21:20:41 | dansmith | it definitely applies | |
| 21:21:11 | sean-k-mooney | dansmith: oh because it skip the schduler and never update placement | |
| 21:21:12 | mriedem | i'm not sure why using the fake virt driver in any of this is a good idea | |
| 21:21:15 | s10 | yep. that is not the right way because of the inventory in placement. if we can't change it after. | |
| 21:21:47 | dansmith | sean-k-mooney: not just that, but the inventory in placement that will come from the fake driver won't look anything like the inventory that the system will provide after you change it to libvirt | |
| 21:21:53 | openstackgerrit | Merged openstack/nova master: Updated AggregateImagePropertiesIsolation filter illustration https://review.openstack.org/586317 | |
| 21:22:08 | sean-k-mooney | s10: dumb question. why can you not put the os dirve in one of the hot swap bays so its just as easy to move as the other disks | |
| 21:22:29 | dansmith | sean-k-mooney: m2 drives on the motherboard are getting common for system drives today | |
| 21:22:35 | dansmith | but of course, he could do that :) | |
| 21:22:47 | dansmith | or buy systems with enough hot-swap bays | |
| 21:22:53 | dansmith | or tell his vendor he doesn't like their placement of the m2 | |