Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-07-30
20:11:22 mriedem i wouldn't consider this INFO
20:11:25 melwitt you won't know your deallocations are retrying unless you enable debug
20:11:33 mriedem ?
20:11:33 mriedem do you need to know that
20:11:37 mriedem unless you're hitting failures
20:11:47 mriedem and if you're hitting failures, you're probably gonna enable debug on compute
20:12:10 melwitt based on what sean-k-mooney said, it could be a hint that neutron is under load. thought it would be good to know about that
20:12:17 mriedem https://specs.openstack.org/openstack/openstack-specs/specs/log-guidelines.html#log-messages-at-info-and-above-should-be-a-unit-of-work
20:13:00 mriedem if we want to indicate there is a problem,
20:13:04 mriedem i'd say it should be warning
20:13:26 mriedem https://specs.openstack.org/openstack/openstack-specs/specs/log-guidelines.html#definition-of-log-levels
20:13:41 MultipleCrashes hmm
20:14:14 mriedem clearly we need more highly paid engineers to paint this shed :)
20:17:10 sean-k-mooney well if you have an extra brush i would be glad to help :P
20:18:58 sean-k-mooney oh devstack stacked. back to testing stuff
20:22:04 s10 What is the easiest way to change hostname/nodename of the nova-compute host? If this host has instances, and all this instances should think after power on, that they are on host with a new hostname.
20:23:57 sean-k-mooney s10: instance can only tell they moved to a new host because the hashed hostid in the api will change
20:25:03 sean-k-mooney if you change the hosts actull host name and restart the nova compute agent its going to endup with a different uuid as the uuid is based off the hostname by default
20:26:30 sean-k-mooney the instaces would still be assocated with the old compute host so the only simple and safe way to do it that i can think of would be to evactuate or migrate the instances first
20:27:16 melwitt yeah, I was going to say, I think the only supported way to change the hostname stored in an instance record is to migrate the instance
20:28:52 s10 I'm talking about different thing. Lets assume, that /var/lib/nova/instances with vms disks are on the separate md (/dev/md1) on host1.openstack.example.org. Then this host breaks because of the memory or cpu issue, or something else. I want to save data of this vm, so evacuation is not the thing, so I extract disks of the /dev/md1 and put them in another server.
20:29:19 s10 And then I had to set hostname of this new node to the hostname of old server.
20:29:43 s10 But what if I don't want to name other physical machine as old one?
20:31:17 s10 I had to rename node in compute_nodes, services, nova_api, nova.instances, ml2_neutron port bindings...
20:31:39 melwitt yeah, what you're talking about it migrating things manually, so if you're doing that, you'd have to figure out where all you'd need to update database records manually. we don't recommend this
20:32:18 sean-k-mooney s10: you also would have to update placement and the cells dbs in addtion to the normal nova one
20:33:45 s10 Yes, and this is too inconvenient. Would it be better to have some time of evacuate command, that would do all this steps, but without rebuilding instances?
20:34:22 sean-k-mooney s10: the simpelste thing to do would be to pull the disk form the failed node. put them in a working node. migrate off the workloads then rename the new node to whatever it should be called give its physical location
20:39:09 melwitt s10: maybe. I don't recall anyone proposing a spec like that. and I don't know if there are reasons why we don't have a command like that
20:39:54 s10 This is how we do it now. But what if system disk (/) is on small m.2 ssd inside the server. So after the failure of this host we couldn't just take disks of the server, we need to pull server out of rack, open it, take this m.2 out and then put it in another server.
20:40:27 melwitt dansmith: as the evacuate guru, do you happen to know why we don't have a command like "migrate all" from a fenced compute node that's forced down, similar to evacuate but moves disks too? are there tech difficulties around it? I don't remember if it's been proposed before
20:41:13 s10 All this operation will take much more time than just execution of evacuate --force --target-host command to move instances to the reserved compute node, that could be even be powered on all the time
20:42:58 dansmith melwitt: evacuate only works if the node is completely dead
20:43:11 dansmith melwitt: it's destructive, doesn't preserve anything
20:43:33 cfriesen__ melwitt: in my experience, the people that actually use "evacuate" will often be forcing down failed compute nodes and immediately evacuating the instances to new hosts.
20:43:37 dansmith melwitt: marking the node down is the shortcut to the "is down" timer if you know better (i.e. you've fenced it yourself)
20:44:24 dansmith if the node is not dead, then evacuate is not the operation you want
20:44:25 cfriesen__ melwitt: if the node is fenced it's generally because it can't talk to the rest of the system
20:44:40 sean-k-mooney dansmith: evacuate is also only usefull if your instance are using bfv right otherwise it not going to keep any data form the instces correct
20:45:04 dansmith melwitt: http://www.danplanet.com/blog/2016/03/03/evacuate-in-nova-one-command-to-confuse-us-all/
20:45:04 cfriesen__ sean-k-mooney: it keeps the IP addresses and such, which is useful for eg. heat
20:45:10 melwitt okay, thanks. yeah, so that's why it's a rebuild, because nova can't talk to the compute host at all
20:45:17 dansmith sean-k-mooney: no, it's useful if you aren't on shared storage
20:45:26 dansmith sean-k-mooney: if you care about uuid, networks, metadata, etc
20:45:31 dansmith melwitt: right
20:45:33 sean-k-mooney cfriesen__: true but if you were using local sotrage on the cmpute node its like doing a rebuild
20:45:33 melwitt s10: what is your situation? compute host totally down? because if so, I don't think there's anything we can do about moving disks etc
20:45:57 cfriesen__ sean-k-mooney: sure. but some applications are fine with that
20:45:58 dansmith sean-k-mooney: ...which keeps the ips, like he said :)
20:46:09 melwitt s10: if your compute host is up, you can disable the service (disable new scheduling to it) and then migrate all the instances off
20:46:12 melwitt via the API
20:46:19 dansmith right
20:46:48 sean-k-mooney dansmith: yes that is true. i was ignroing that but i guess that could be usful if you have a statelesee workload.
20:47:13 dansmith sean-k-mooney: or your workload keeps its state elsewhere
20:47:24 dansmith sean-k-mooney: like a volume or a remote database
20:47:37 dansmith your root disk doesn't need to be on a volume
20:47:44 s10 melwitt: this is hypothetical situation, I'm thinking about ways to reduce downtime of instances. Situation is: compute node is down (kernel panic, powered off). I can pull disks with the instances data and put them in another host. But with current nova code I had to move system disks also and wait until system will boot on new hardware.
20:47:51 sean-k-mooney dansmith: sure. i guess i was conflating not bfv with no cinider volumes.
20:48:28 cfriesen__ S10: one possible issue you will run into is that neutron may need to be updated with suitable information as well, you may need to allocate SRIOV devices, the available CPUs on the new host might be different (for pinned CPUs), etc.
20:48:29 dansmith s10: or not store your state in the ephemeral instance root disks in the first place... you know, like cloud and stuff :)
20:48:58 cfriesen__ s10: also, any cinder connections will need to be rebuilt
20:49:07 dansmith and if you're going to, you should probably be using ceph or cinder backing
20:49:24 s10 instance,
20:49:24 s10 self.network_api.migrate_instance_finish(context,
20:49:24 s10 'dest_compute': self.host, }
20:49:24 s10 migration = {'source_compute': instance.host,
20:49:24 s10 cfriesen: this is not the issue, if recreate:
20:49:25 s10 migration)
20:49:39 s10 This sets ports for instances on new host
20:49:52 sean-k-mooney s10: also if your using routed netwroking you need to make sure your new host is on the same segment.
20:49:54 cfriesen__ I'm talking about allocations, resource tracking, etc
20:50:29 cfriesen__ placement, PCI device and CPU and hugepage claims, etc
20:50:42 s10 dansmith: yes, but what if we want as much IOPS for instances as possible - this could only be archived with local NVMes, so red etc is not and option
20:51:51 dansmith ack, fair enough
20:52:00 sean-k-mooney s10: you dont need to keep all of your data on the nvme drivres. having an nvme backed epheral disk for scratch space or chacing is a more cloud friendly approch
20:52:16 dansmith yup
20:52:35 dansmith there's this triangle..
20:53:04 dansmith http://katharine-writes.com/wp-content/uploads/2014/03/Quality-Triangle.jpg
20:53:54 openstackgerrit Merged openstack/nova master: Add description for placement 1.26 https://review.openstack.org/587067
20:56:13 sean-k-mooney dmcache and bcache in the kernel are actully great ways to cache /var/lib/nova/instances mounts from remote network filesystem using local nvmedrives
20:57:13 sean-k-mooney i rembere looking to see if qemu could be configured to cache its remote disks via local fast storage but never found a way
20:57:30 dansmith or just use something capable of doing it natively
20:57:52 openstackgerrit Matt Riedemann proposed openstack/nova master: Retry decorator fix for instances which go into ERROR state during bulk delete https://review.openstack.org/563418
20:57:53 mriedem jaypipes: melwitt: there you go ^ easy peasy
20:58:06 s10 reserved host is a complete replica of the failed host, same cpus, no pic devices, same memory, in same network segment, only thing that should be changed is the hostname...
20:58:26 sean-k-mooney ya i was trying to see if i could figure out how to cache vms backed by ceph using nvdim persistent memory without modifying qemu
20:59:18 dansmith yeah for transparent backing of block devices, dm-cache is the way
20:59:51 melwitt mriedem: thanks
21:00:20 sean-k-mooney s10: i was pritty sure we had a way to override the hostid of a compute node but i could not find it in the conf. that bacsically all you need. e.g. a way to forst the agent to think its on the old host without actully using the old host name on the phyical host
21:00:49 dansmith ...or just transport the damn system disk :)
21:00:57 dansmith you're walking across the aisle anyway :)
21:01:32 s10 sean-k-mooney: i'm not sure, that we can tell that hostid to the neutron-ovs-agent same way :(
21:01:40 sean-k-mooney dansmith: ya that would likely be quicker then any workaround we can comup with bar adding a compute node uuid to the compute conf
21:01:58 sean-k-mooney s10: the neutron agent definetly supports it
21:02:07 sean-k-mooney its nova that is missing it
21:02:32 melwitt yeah, I was gonna say, given all the things that would need to be updated, copying over the system disk sounds like the easiest and fastest way
21:03:43 sean-k-mooney dansmith: melwitt out of interest is there any reason we could not allow an operator to choos the compute node uuid by setting it in the conf?
21:04:30 dansmith sean-k-mooney: it's internal and more complicated to let them choose
21:04:43 dansmith right now it can't change unless we change it or do things
21:05:00 dansmith I can't really see a compelling reason to let them change it

Earlier   Later