Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-01-22
16:06:20 amorin problem is, imagine you already spawn instances with iso kind, nova needs to copy it before (because of libvirt bug), but in the meantime, you updated the nova config, so CONF.config_drive = vfat
16:06:32 amorin then you never enter this if, and live-migration fail
16:07:11 mdbooth sean-k-mooney: Pretty sure it's something like that. Long time since I had this cached.
16:07:55 mdbooth amorin: So, the other thing that happened since then is that we explicitly specify which disks to migrate using migrateToURI3
16:07:56 amorin sean-k-mooney: I did try to remove it in my lab, seems to work, I'll try to submit and patch and see the results in gate jobs
16:08:46 amorin mdbooth: ok, got it, migrateToURI3 is then able to migrate the vfat config-drive
16:10:10 amorin mdbooth: but are we sure that migrateToURI3 is able to handle the iso kind?
16:10:12 amorin i'll check
16:10:13 mdbooth IIRC it's because cd-rom drives are read-only, and qemu won't let us write to a read-only disk, even during live migration
16:10:43 mdbooth amorin: The pertinent point about migrateToURI3 is that we specify a list of disks to block migrate explicitly
16:10:46 zzzeek jaypipes: my connection monitoring thing currently lets you put ?plugin=connmon on the SQLAlchemy URL. But Nova Cells shoves DB urls into the database the first time it runs and then they never change :(. So need to add a config flag to oslo.db. But then Nova hardcodes all the oslo flags :)
16:10:50 mdbooth Whereas before it was all disks
16:11:03 sean-k-mooney amorin: is there a reason we migate configdirve instead of recreating on remote side our of interest. technically with vfat it can be readwrite but they are intended to be readonly
16:11:11 sean-k-mooney mdbooth: ^
16:11:18 mdbooth amorin: So previously we didn't avoid the problematic disks, iirc
16:11:44 mdbooth sean-k-mooney: I believe we do recreate it now
16:14:07 tovin07 this patch passed all zuul gate (had 2+2, A+1). however, it's still open. https://review.openstack.org/#/c/519664/
16:14:09 jaypipes zzzeek: that is indeed the suck :(
16:14:20 zzzeek jaypipes: this "nova puts the URLs into the DB
16:14:27 zzzeek " thing has been a really huge problem
16:14:43 jaypipes zzzeek: as opposed to, say, a config management system? :)
16:14:48 zzzeek it is far and away the worst design decision
16:15:20 zzzeek jaypipes: it means I cannot change the URL in the config file and have it take effect, *and* it means whatever is in the config file on one server is *implicitly shared* with all other nova servers on other machines
16:15:30 jaypipes zzzeek: well, I don't disagree it's a bad design but it's not something we can "fix" right away...
16:15:58 zzzeek jaypipes: we have all kind of workarounds. it's just v v inconvenient to constantly hit it
16:16:17 jaypipes zzzeek: is there a hack/workaround that will allow you to proceed with your ?plugin=XXX enhancements?
16:16:33 zzzeek jaypipes: yes I am going to add it to oslo.db so it's a separate config option outside of the URL
16:16:40 jaypipes zzzeek: again, I don't disagree with you at all
16:16:56 zzzeek jaypipes: which means when I document how to use connmon w/ nova, the answer will be "it depends :) "
16:17:26 zzzeek like older nova, OK there's no cells, put it in the URL. newer nova, OK we have the olso.db thing. middle-nova, erg, rewrite your cells URLs w/ the command line thing
16:17:42 jaypipes zzzeek: another annoying this about oslo.cfg: you can't tell whether a CONF option has been set or not. i.e. has the CONF option value been set (via either argparse or configparser) to any value.
16:17:44 sean-k-mooney stephenfin: just looking at https://review.openstack.org/#/c/449257/52/nova/pci/request.py@93 the clean up you are asking for are not directly related to the patch rodlfo is doing so i should probably pull it out into another patch above or below rodolfos. is that ok? and do you have a prefernece?
16:17:55 jaypipes zzzeek: s/this/thing
16:18:28 zzzeek jaypipes: I might have observed the other day that putting something in [DEFAULT] doesn't actually "default" the value elsewhere? i struggled for days getting region_name / os_region_name setup to work
16:18:57 jaypipes zzzeek: yeah
16:19:13 bauzas does anyone know more than just by the name the "nova backup" command ?
16:19:33 zzzeek jaypipes: and another thing! why is the keystone auth URL in nova.conf but then if my region is wrong it seems to ask keystone for the *other* URL then it starts using that one (and fails) ?
16:19:37 bauzas I guess the retention period is just for calculating whether we should delete an old image or not
16:19:56 bauzas but hopefully nova isn't periodically running a task to clean that up
16:20:13 bauzas (ie. that's a per-call retention calculation)
16:20:18 stephenfin sean-k-mooney: What cleanup is this now?
16:20:44 bauzas I'm just amazed to discover such API after 5 years working for Nova
16:21:17 bauzas nevermind, got my answer https://elastx.se/en/blog/backups-openstack-cloud
16:21:22 sean-k-mooney stephenfin: basically the comment here https://review.openstack.org/#/c/449257/52/nova/pci/request.py@93
16:22:28 stephenfin sean-k-mooney: Aye, the lines I actually commented on are a cleanup task. However, my comments about what the object should actually look like are not
16:22:32 sean-k-mooney stephenfin: they are against lines not modified by rodolfos patch so im wondering should it be a seperate patch or in this one
16:22:50 mriedem bauzas: note you can't use backup with a volume-backed instance, but you can snapshot a volume-backed instance, another fun wrinkle, and something people have wanted to fix for probably 5 years as well
16:23:00 stephenfin sean-k-mooney: What I was suggesting is that rather than having two generic container fields on the object, we actually define the fields we want
16:23:14 bauzas mriedem: but you can backup a volume, right?
16:23:22 bauzas not with nova, of course
16:23:30 stephenfin sean-k-mooney: The two generic container fields being 'dict_of_lists' and 'dict_of_strings'
16:23:53 bauzas anyway, I'm just testing my patches against https://developer.openstack.org/api-guide/compute/server_concepts.html#server-actions
16:24:06 bauzas whatever the backup does, it works
16:24:11 bauzas period.
16:24:23 mriedem backup creates a snapshot with a rotating retention period yeah
16:24:36 jaypipes zzzeek: see: vestigial tail? :)
16:24:38 stephenfin sean-k-mooney: Rather than using 'dict_of_lists', add a ListOpt for each key that we'd expect to store in there. Similarly, instead of 'dict_of_strings', add a ListOpt for each key
16:24:41 stephenfin Does that make sense?
16:25:20 bauzas mriedem: it tags the snapshot, I see
16:25:34 bauzas mriedem: my question was just about what was enforcing that retention
16:25:39 bauzas but that's fine, I see that
16:25:45 sean-k-mooney ok i can do that would you like it in this patch or a seperate one
16:26:28 stephenfin sean-k-mooney: That one, please. If not, we're going to have to immediately issue a MAJOR version bump on the object so we can remove the 'dict_of_*' fields
16:26:36 stephenfin Which seems awfully silly :)
16:27:24 sean-k-mooney stephenfin: the network capablites will still need to remain a list of stings in the object as it can technicall by any trait includeing custom_ ones
16:27:37 sean-k-mooney stephenfin: ok will do.
16:27:56 stephenfin sean-k-mooney: Yup, I'd expect to see 'capabilities = ListOpt(...)'
16:28:00 stephenfin sean-k-mooney: Shhhhooound
16:29:01 sean-k-mooney am that might also want to be an object actully capablities:{ network = listOpts(...);}
16:29:21 sean-k-mooney for pci device we could have compute or other non netowrk capablites later
16:30:53 openstackgerrit Matt Riedemann proposed openstack/python-novaclient master: Microversion 2.59 - Migrations list pagination https://review.openstack.org/516168
16:30:55 sean-k-mooney e.g. gpus could have opengl/cuda level as a trait but they are still pcidevices.
16:31:23 lyarwood kashyap: random question, have you ever had to file a bug against libvirt as shipped in UCA?
16:31:51 kashyap lyarwood: Good random question
16:32:00 kashyap lyarwood: I haven't, but what I did was---
16:32:03 openstackgerrit Ed Leafe proposed openstack/nova master: Change compute RPC to use alternates for resize https://review.openstack.org/526436
16:32:08 lyarwood kashyap: hitting this http://logs.openstack.org/50/536350/1/check/legacy-grenade-dsvm-neutron-multinode-live-migration/5f7ed57/logs/screen-n-cpu.txt.gz?level=ERROR#_Jan_22_15_09_28_467904 that I think is the result of libvirt being compiled without gnutls installed, somehow.
16:32:10 kashyap Jump on their IRC channel, and some friendly Ubuntu person did i
16:32:13 edleafe mriedem: jaypipes: ^^ rebased w/new RPC version
16:32:15 kashyap s/did i/it/
16:32:29 lyarwood kashyap: kk, which channel?
16:32:37 edmondsw zzzeek if you're referring to the keystone_authtoken section, keystone url has to be there so that nova knows how to talk to keystone at all
16:32:38 kashyap lyarwood: #ubuntu-dev
16:32:49 kashyap lyarwood: If it's a packaging but, #ubuntu-packaging
16:32:52 kashyap s/but/bug/
16:33:11 zzzeek edmondsw: yes, but I have observed that once it does that initial talk to keystone thing, it looks into endpoints and gets whatever it finds there for keystone, then starts talking to that
16:33:19 kashyap Why are they compiling _out_ GnuTLS? Libvirt depends on it, as you know
16:33:39 edmondsw zzzeek yep. Which should be all well and good unless you've got something misconfigured
16:33:48 lyarwood kashyap: no idea :)
16:34:20 kashyap lyarwood: Try if you can find Christian Ehrhardt from Canonical on those channels
16:34:30 kashyap lyarwood: He's the active libvirt dev from Canononical that I see
16:34:38 kashyap (On the upstream lists)
16:34:57 zzzeek edmondsw: sure. these things are just v hard to debug in an HA setup
16:35:54 edmondsw zzzeek yeah :( I'm not sure the exact reason there (or if there is a good reason). lbragstad would have a better idea
16:36:31 openstackgerrit Merged openstack/python-novaclient master: Add support for the 2.57 microversion https://review.openstack.org/528128
16:36:32 openstackgerrit Merged openstack/python-novaclient master: Microversion 2.58 - Instance actions list pagination https://review.openstack.org/528601
16:37:29 zzzeek edmondsw: well w regions I guess you are saying "here we'd prefer you use this address for this service" so even though you already gave it a "bootstrap" keystone, sure I'd probably like that it does that. depends on wht you're using regions for
16:38:14 edmondsw zzzeek yeah, regions get all kinds of funky
16:41:36 kashyap lyarwood: Probably best to report it on the tracker, to save your time
16:41:46 openstackgerrit Jackie Truong proposed openstack/python-novaclient master: Microversion 2.60 - Add trusted_image_certificates https://review.openstack.org/500396

Earlier   Later