Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-07-30
20:43:37 dansmith melwitt: marking the node down is the shortcut to the "is down" timer if you know better (i.e. you've fenced it yourself)
20:44:24 dansmith if the node is not dead, then evacuate is not the operation you want
20:44:25 cfriesen__ melwitt: if the node is fenced it's generally because it can't talk to the rest of the system
20:44:40 sean-k-mooney dansmith: evacuate is also only usefull if your instance are using bfv right otherwise it not going to keep any data form the instces correct
20:45:04 dansmith melwitt: http://www.danplanet.com/blog/2016/03/03/evacuate-in-nova-one-command-to-confuse-us-all/
20:45:04 cfriesen__ sean-k-mooney: it keeps the IP addresses and such, which is useful for eg. heat
20:45:10 melwitt okay, thanks. yeah, so that's why it's a rebuild, because nova can't talk to the compute host at all
20:45:17 dansmith sean-k-mooney: no, it's useful if you aren't on shared storage
20:45:26 dansmith sean-k-mooney: if you care about uuid, networks, metadata, etc
20:45:31 dansmith melwitt: right
20:45:33 sean-k-mooney cfriesen__: true but if you were using local sotrage on the cmpute node its like doing a rebuild
20:45:33 melwitt s10: what is your situation? compute host totally down? because if so, I don't think there's anything we can do about moving disks etc
20:45:57 cfriesen__ sean-k-mooney: sure. but some applications are fine with that
20:45:58 dansmith sean-k-mooney: ...which keeps the ips, like he said :)
20:46:09 melwitt s10: if your compute host is up, you can disable the service (disable new scheduling to it) and then migrate all the instances off
20:46:12 melwitt via the API
20:46:19 dansmith right
20:46:48 sean-k-mooney dansmith: yes that is true. i was ignroing that but i guess that could be usful if you have a statelesee workload.
20:47:13 dansmith sean-k-mooney: or your workload keeps its state elsewhere
20:47:24 dansmith sean-k-mooney: like a volume or a remote database
20:47:37 dansmith your root disk doesn't need to be on a volume
20:47:44 s10 melwitt: this is hypothetical situation, I'm thinking about ways to reduce downtime of instances. Situation is: compute node is down (kernel panic, powered off). I can pull disks with the instances data and put them in another host. But with current nova code I had to move system disks also and wait until system will boot on new hardware.
20:47:51 sean-k-mooney dansmith: sure. i guess i was conflating not bfv with no cinider volumes.
20:48:28 cfriesen__ S10: one possible issue you will run into is that neutron may need to be updated with suitable information as well, you may need to allocate SRIOV devices, the available CPUs on the new host might be different (for pinned CPUs), etc.
20:48:29 dansmith s10: or not store your state in the ephemeral instance root disks in the first place... you know, like cloud and stuff :)
20:48:58 cfriesen__ s10: also, any cinder connections will need to be rebuilt
20:49:07 dansmith and if you're going to, you should probably be using ceph or cinder backing
20:49:24 s10 instance,
20:49:24 s10 self.network_api.migrate_instance_finish(context,
20:49:24 s10 'dest_compute': self.host, }
20:49:24 s10 migration = {'source_compute': instance.host,
20:49:24 s10 cfriesen: this is not the issue, if recreate:
20:49:25 s10 migration)
20:49:39 s10 This sets ports for instances on new host
20:49:52 sean-k-mooney s10: also if your using routed netwroking you need to make sure your new host is on the same segment.
20:49:54 cfriesen__ I'm talking about allocations, resource tracking, etc
20:50:29 cfriesen__ placement, PCI device and CPU and hugepage claims, etc
20:50:42 s10 dansmith: yes, but what if we want as much IOPS for instances as possible - this could only be archived with local NVMes, so red etc is not and option
20:51:51 dansmith ack, fair enough
20:52:00 sean-k-mooney s10: you dont need to keep all of your data on the nvme drivres. having an nvme backed epheral disk for scratch space or chacing is a more cloud friendly approch
20:52:16 dansmith yup
20:52:35 dansmith there's this triangle..
20:53:04 dansmith http://katharine-writes.com/wp-content/uploads/2014/03/Quality-Triangle.jpg
20:53:54 openstackgerrit Merged openstack/nova master: Add description for placement 1.26 https://review.openstack.org/587067
20:56:13 sean-k-mooney dmcache and bcache in the kernel are actully great ways to cache /var/lib/nova/instances mounts from remote network filesystem using local nvmedrives
20:57:13 sean-k-mooney i rembere looking to see if qemu could be configured to cache its remote disks via local fast storage but never found a way
20:57:30 dansmith or just use something capable of doing it natively
20:57:52 openstackgerrit Matt Riedemann proposed openstack/nova master: Retry decorator fix for instances which go into ERROR state during bulk delete https://review.openstack.org/563418
20:57:53 mriedem jaypipes: melwitt: there you go ^ easy peasy
20:58:06 s10 reserved host is a complete replica of the failed host, same cpus, no pic devices, same memory, in same network segment, only thing that should be changed is the hostname...
20:58:26 sean-k-mooney ya i was trying to see if i could figure out how to cache vms backed by ceph using nvdim persistent memory without modifying qemu
20:59:18 dansmith yeah for transparent backing of block devices, dm-cache is the way
20:59:51 melwitt mriedem: thanks
21:00:20 sean-k-mooney s10: i was pritty sure we had a way to override the hostid of a compute node but i could not find it in the conf. that bacsically all you need. e.g. a way to forst the agent to think its on the old host without actully using the old host name on the phyical host
21:00:49 dansmith ...or just transport the damn system disk :)
21:00:57 dansmith you're walking across the aisle anyway :)
21:01:32 s10 sean-k-mooney: i'm not sure, that we can tell that hostid to the neutron-ovs-agent same way :(
21:01:40 sean-k-mooney dansmith: ya that would likely be quicker then any workaround we can comup with bar adding a compute node uuid to the compute conf
21:01:58 sean-k-mooney s10: the neutron agent definetly supports it
21:02:07 sean-k-mooney its nova that is missing it
21:02:32 melwitt yeah, I was gonna say, given all the things that would need to be updated, copying over the system disk sounds like the easiest and fastest way
21:03:43 sean-k-mooney dansmith: melwitt out of interest is there any reason we could not allow an operator to choos the compute node uuid by setting it in the conf?
21:04:30 dansmith sean-k-mooney: it's internal and more complicated to let them choose
21:04:43 dansmith right now it can't change unless we change it or do things
21:05:00 dansmith I can't really see a compelling reason to let them change it
21:06:48 sean-k-mooney im not saying it a good idea. its a great way to shoot yourself in the foot and mess up our resouce tracking. but it would make s10 usecase doable without a lot of db hacking
21:07:42 s10 Or can I just change compute_driver to Fake on reserved node, execute evacuate to this node and then just put disks from old host, and than change config back and restart nova-compute...
21:08:47 s10 compute_driver=fake.FakeDriver. and than back to libvirt.LibvirtDriver.
21:12:42 sean-k-mooney s10: that will likely take longer then pulling the os driver if your goal is to minimise guest downtime
21:17:54 s10 sean-k-mooney: nova host-evacuate --target-host --force can be executed immediately after host failure. Same time, while instances are being "evacuated" to the fake host, support will pull disks from failed server and put them in the reserved, right after that we stop nova-compute, mount disks from old host, change nova config, start it again, and finally nova reboot for all instances. This will take much less time than moving system m.2 disk
21:19:17 s10 Only physical operation that should be performed is moving disks. Other way requires opening the lid on the top of the server with dismounting it out of the rack.
21:19:36 mriedem i highly suggest not using the --force option
21:19:51 mriedem since it bypasses the scheduler and can mess things up if the target host is not actually a proper fit for the instance
21:20:05 dansmith also,
21:20:17 dansmith the fake driver will not have the right inventory in placement
21:20:31 dansmith so your scheduler or conductor-based allocations would be all fubar
21:20:33 sean-k-mooney mriedem: in this case s10 is forcing a migrate to a fake libvirt node so not sure it applies
21:20:41 dansmith it definitely applies
21:21:11 sean-k-mooney dansmith: oh because it skip the schduler and never update placement
21:21:12 mriedem i'm not sure why using the fake virt driver in any of this is a good idea
21:21:15 s10 yep. that is not the right way because of the inventory in placement. if we can't change it after.
21:21:47 dansmith sean-k-mooney: not just that, but the inventory in placement that will come from the fake driver won't look anything like the inventory that the system will provide after you change it to libvirt
21:21:53 openstackgerrit Merged openstack/nova master: Updated AggregateImagePropertiesIsolation filter illustration https://review.openstack.org/586317
21:22:08 sean-k-mooney s10: dumb question. why can you not put the os dirve in one of the hot swap bays so its just as easy to move as the other disks
21:22:29 dansmith sean-k-mooney: m2 drives on the motherboard are getting common for system drives today
21:22:35 dansmith but of course, he could do that :)
21:22:47 dansmith or buy systems with enough hot-swap bays
21:22:53 dansmith or tell his vendor he doesn't like their placement of the m2
21:23:01 dansmith or use a system with a rear-mounted hot-swap bay for system drives
21:23:03 dansmith lots of options
21:23:32 s10 sean-k-mooney: because we have 4 slots for disk in 1U server. we use it all for storage disks or have to use one of them for system.
21:25:41 sean-k-mooney s10: ok but i have racked a lot of sytems in the intel dev lab. it really does not take that much time to pull an internal drive. for an m2 it 1 screw most of the time and you often dont need to even unrack the server.
21:26:48 sean-k-mooney s10: if you really do have this requirement then fair enough but it seams like premature optimization unless you expect to have to do this often
21:28:13 s10 sean-k-mooney: you are right, this is a premature optimisation.
21:45:15 openstackgerrit Eric Fried proposed openstack/nova master: [placement] Add /reshaper handler for POST https://review.openstack.org/576927
21:45:16 openstackgerrit Eric Fried proposed openstack/nova master: reshaper: Look up provider if not in inventories https://review.openstack.org/585033
21:45:17 openstackgerrit Eric Fried proposed openstack/nova master: Make get_allocations_for_resource_provider sane https://review.openstack.org/584598
21:45:18 openstackgerrit Eric Fried proposed openstack/nova master: Report client: Real get_allocs_for_consumer https://review.openstack.org/584599
21:45:19 openstackgerrit Eric Fried proposed openstack/nova master: Report client: get_allocations_for_provider_tree https://review.openstack.org/584648
21:45:20 openstackgerrit Eric Fried proposed openstack/nova master: Report client: _reshape helper, placement min bump https://review.openstack.org/585034

Earlier   Later