Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-01-22
16:00:11 jaypipes amorin: as for your question... no idea. perhaps mdbooth or lyarwood might know the answer on that one.
16:00:42 amorin jaypipes mdbooth thanks
16:02:12 mdbooth amorin: What's the question?
16:02:46 mdbooth amorin: Ah, you're wondering why the handling difference between iso9660 and vfat?
16:02:49 amorin what if we copy the config-drive no matter its kind (vfat or iso)
16:02:55 amorin yup
16:03:04 efried cdent "For future reference, in the future this loop could be replaced with a single request to POST /allocations, clearing the allocations for all the consumers." <== This must have been a difficult comment to write. The war between "use lots of little API calls" and "use stuff I wrote!" :P
16:03:13 amorin I understand that libvirt is able to copy it if its vfat
16:03:28 amorin but it seems that if nova copy it first,
16:03:30 mdbooth It was (is? but I doubt it) a bug in libvirt/qemu in the handling of iso9660 disks
16:03:34 amorin then libvirt will do nothing
16:03:46 efried cdent Only joshing you of course. The POST is a great idea there.
16:03:50 mdbooth Did you look at the referenced lp bug?
16:03:56 amorin mdbooth: yes
16:04:12 amorin seems that libvirt is still failing with iso
16:04:39 amorin I was just wondering if copying vfat with nova is a bad idea or not
16:05:26 sean-k-mooney amorin: mdbooth i would guesss the bug in libvirt is related to iso beeing treated as cdroms and vfat ect disk being considered hdds or somthin in that vain?
16:06:12 sean-k-mooney amorin: well one way to check would be remove that line and look at the livemigration gate jobs. it might result in both nova and neutron coping the config drive
16:06:18 openstackgerrit Mark Goddard proposed openstack/nova master: Add get_traits() method to ComputeDriver https://review.openstack.org/532290
16:06:18 openstackgerrit Mark Goddard proposed openstack/nova master: Send traits to ironic on server boot https://review.openstack.org/508116
16:06:19 openstackgerrit Mark Goddard proposed openstack/nova master: Implement get_traits() for the ironic virt driver https://review.openstack.org/532288
16:06:20 amorin problem is, imagine you already spawn instances with iso kind, nova needs to copy it before (because of libvirt bug), but in the meantime, you updated the nova config, so CONF.config_drive = vfat
16:06:32 amorin then you never enter this if, and live-migration fail
16:07:11 mdbooth sean-k-mooney: Pretty sure it's something like that. Long time since I had this cached.
16:07:55 mdbooth amorin: So, the other thing that happened since then is that we explicitly specify which disks to migrate using migrateToURI3
16:07:56 amorin sean-k-mooney: I did try to remove it in my lab, seems to work, I'll try to submit and patch and see the results in gate jobs
16:08:46 amorin mdbooth: ok, got it, migrateToURI3 is then able to migrate the vfat config-drive
16:10:10 amorin mdbooth: but are we sure that migrateToURI3 is able to handle the iso kind?
16:10:12 amorin i'll check
16:10:13 mdbooth IIRC it's because cd-rom drives are read-only, and qemu won't let us write to a read-only disk, even during live migration
16:10:43 mdbooth amorin: The pertinent point about migrateToURI3 is that we specify a list of disks to block migrate explicitly
16:10:46 zzzeek jaypipes: my connection monitoring thing currently lets you put ?plugin=connmon on the SQLAlchemy URL. But Nova Cells shoves DB urls into the database the first time it runs and then they never change :(. So need to add a config flag to oslo.db. But then Nova hardcodes all the oslo flags :)
16:10:50 mdbooth Whereas before it was all disks
16:11:03 sean-k-mooney amorin: is there a reason we migate configdirve instead of recreating on remote side our of interest. technically with vfat it can be readwrite but they are intended to be readonly
16:11:11 sean-k-mooney mdbooth: ^
16:11:18 mdbooth amorin: So previously we didn't avoid the problematic disks, iirc
16:11:44 mdbooth sean-k-mooney: I believe we do recreate it now
16:14:07 tovin07 this patch passed all zuul gate (had 2+2, A+1). however, it's still open. https://review.openstack.org/#/c/519664/
16:14:09 jaypipes zzzeek: that is indeed the suck :(
16:14:20 zzzeek jaypipes: this "nova puts the URLs into the DB
16:14:27 zzzeek " thing has been a really huge problem
16:14:43 jaypipes zzzeek: as opposed to, say, a config management system? :)
16:14:48 zzzeek it is far and away the worst design decision
16:15:20 zzzeek jaypipes: it means I cannot change the URL in the config file and have it take effect, *and* it means whatever is in the config file on one server is *implicitly shared* with all other nova servers on other machines
16:15:30 jaypipes zzzeek: well, I don't disagree it's a bad design but it's not something we can "fix" right away...
16:15:58 zzzeek jaypipes: we have all kind of workarounds. it's just v v inconvenient to constantly hit it
16:16:17 jaypipes zzzeek: is there a hack/workaround that will allow you to proceed with your ?plugin=XXX enhancements?
16:16:33 zzzeek jaypipes: yes I am going to add it to oslo.db so it's a separate config option outside of the URL
16:16:40 jaypipes zzzeek: again, I don't disagree with you at all
16:16:56 zzzeek jaypipes: which means when I document how to use connmon w/ nova, the answer will be "it depends :) "
16:17:26 zzzeek like older nova, OK there's no cells, put it in the URL. newer nova, OK we have the olso.db thing. middle-nova, erg, rewrite your cells URLs w/ the command line thing
16:17:42 jaypipes zzzeek: another annoying this about oslo.cfg: you can't tell whether a CONF option has been set or not. i.e. has the CONF option value been set (via either argparse or configparser) to any value.
16:17:44 sean-k-mooney stephenfin: just looking at https://review.openstack.org/#/c/449257/52/nova/pci/request.py@93 the clean up you are asking for are not directly related to the patch rodlfo is doing so i should probably pull it out into another patch above or below rodolfos. is that ok? and do you have a prefernece?
16:17:55 jaypipes zzzeek: s/this/thing
16:18:28 zzzeek jaypipes: I might have observed the other day that putting something in [DEFAULT] doesn't actually "default" the value elsewhere? i struggled for days getting region_name / os_region_name setup to work
16:18:57 jaypipes zzzeek: yeah
16:19:13 bauzas does anyone know more than just by the name the "nova backup" command ?
16:19:33 zzzeek jaypipes: and another thing! why is the keystone auth URL in nova.conf but then if my region is wrong it seems to ask keystone for the *other* URL then it starts using that one (and fails) ?
16:19:37 bauzas I guess the retention period is just for calculating whether we should delete an old image or not
16:19:56 bauzas but hopefully nova isn't periodically running a task to clean that up
16:20:13 bauzas (ie. that's a per-call retention calculation)
16:20:18 stephenfin sean-k-mooney: What cleanup is this now?
16:20:44 bauzas I'm just amazed to discover such API after 5 years working for Nova
16:21:17 bauzas nevermind, got my answer https://elastx.se/en/blog/backups-openstack-cloud
16:21:22 sean-k-mooney stephenfin: basically the comment here https://review.openstack.org/#/c/449257/52/nova/pci/request.py@93
16:22:28 stephenfin sean-k-mooney: Aye, the lines I actually commented on are a cleanup task. However, my comments about what the object should actually look like are not
16:22:32 sean-k-mooney stephenfin: they are against lines not modified by rodolfos patch so im wondering should it be a seperate patch or in this one
16:22:50 mriedem bauzas: note you can't use backup with a volume-backed instance, but you can snapshot a volume-backed instance, another fun wrinkle, and something people have wanted to fix for probably 5 years as well
16:23:00 stephenfin sean-k-mooney: What I was suggesting is that rather than having two generic container fields on the object, we actually define the fields we want
16:23:14 bauzas mriedem: but you can backup a volume, right?
16:23:22 bauzas not with nova, of course
16:23:30 stephenfin sean-k-mooney: The two generic container fields being 'dict_of_lists' and 'dict_of_strings'
16:23:53 bauzas anyway, I'm just testing my patches against https://developer.openstack.org/api-guide/compute/server_concepts.html#server-actions
16:24:06 bauzas whatever the backup does, it works
16:24:11 bauzas period.
16:24:23 mriedem backup creates a snapshot with a rotating retention period yeah
16:24:36 jaypipes zzzeek: see: vestigial tail? :)
16:24:38 stephenfin sean-k-mooney: Rather than using 'dict_of_lists', add a ListOpt for each key that we'd expect to store in there. Similarly, instead of 'dict_of_strings', add a ListOpt for each key
16:24:41 stephenfin Does that make sense?
16:25:20 bauzas mriedem: it tags the snapshot, I see
16:25:34 bauzas mriedem: my question was just about what was enforcing that retention
16:25:39 bauzas but that's fine, I see that
16:25:45 sean-k-mooney ok i can do that would you like it in this patch or a seperate one
16:26:28 stephenfin sean-k-mooney: That one, please. If not, we're going to have to immediately issue a MAJOR version bump on the object so we can remove the 'dict_of_*' fields
16:26:36 stephenfin Which seems awfully silly :)
16:27:24 sean-k-mooney stephenfin: the network capablites will still need to remain a list of stings in the object as it can technicall by any trait includeing custom_ ones
16:27:37 sean-k-mooney stephenfin: ok will do.
16:27:56 stephenfin sean-k-mooney: Yup, I'd expect to see 'capabilities = ListOpt(...)'
16:28:00 stephenfin sean-k-mooney: Shhhhooound
16:29:01 sean-k-mooney am that might also want to be an object actully capablities:{ network = listOpts(...);}
16:29:21 sean-k-mooney for pci device we could have compute or other non netowrk capablites later
16:30:53 openstackgerrit Matt Riedemann proposed openstack/python-novaclient master: Microversion 2.59 - Migrations list pagination https://review.openstack.org/516168
16:30:55 sean-k-mooney e.g. gpus could have opengl/cuda level as a trait but they are still pcidevices.
16:31:23 lyarwood kashyap: random question, have you ever had to file a bug against libvirt as shipped in UCA?
16:31:51 kashyap lyarwood: Good random question
16:32:00 kashyap lyarwood: I haven't, but what I did was---
16:32:03 openstackgerrit Ed Leafe proposed openstack/nova master: Change compute RPC to use alternates for resize https://review.openstack.org/526436
16:32:08 lyarwood kashyap: hitting this http://logs.openstack.org/50/536350/1/check/legacy-grenade-dsvm-neutron-multinode-live-migration/5f7ed57/logs/screen-n-cpu.txt.gz?level=ERROR#_Jan_22_15_09_28_467904 that I think is the result of libvirt being compiled without gnutls installed, somehow.
16:32:10 kashyap Jump on their IRC channel, and some friendly Ubuntu person did i
16:32:13 edleafe mriedem: jaypipes: ^^ rebased w/new RPC version

Earlier   Later