Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-18
15:26:03 mriedem bauzas: for that audit command i would have started with functional tests months ago
15:26:09 mriedem unit tests don't mean much for stuff like this
15:26:20 mriedem in fact, i don't like unit tests really for any of these types of operational commands
15:26:22 bauzas mriedem: I can send you a new revision now, FWIW
15:26:36 bauzas mriedem: but I just wonder what to test from a migration usecase
15:26:54 bauzas but I think I'll just mock a failing migration that leaves an allocation record
15:26:57 mriedem there are notes about migration in the bug report aren't there?
15:27:18 bauzas correct, there are examples where we found orphaned allocations
15:27:23 mriedem i don't think mocking is really good here, i'm pretty sure we have known ways to hit stuff like that, with evacuate
15:28:02 mriedem https://review.opendev.org/#/c/678100/ has a couple of related bugs where we orphan allocations/providers
15:28:06 bauzas what I basically do is to compare whether I can find an active migration against an existing allocation
15:28:22 bauzas will look at the change ^
15:28:38 bauzas and you know what ? I'll release the change now so people can voice against it
15:29:07 mriedem kashyap: i've tagged the bug with train-rc-potential
15:29:30 kashyap mriedem: Ah, thank you. Didn't think of it
15:29:53 kashyap IIRC, stephenfin wanted to hold off on it until the 'cpu-resources' merges
15:30:10 mriedem so like the day of RC1
15:30:12 mriedem got it
15:30:13 kashyap (Due to conflicts, etc.)
15:30:38 kashyap mriedem: :D I mentioned it exactly for that -- in case it doesn't merge, that typo shouldn't fall through the cracks
15:30:49 kashyap (And you've helpfully fixed it, by the tag)
15:30:55 mriedem i sure hope everyone rechecking the vpmem and pcpu series is actually looking at every failure to make sure we're not recheck grinding in some regression
15:31:33 stephenfin I am. We're seeing mirror issues, InnoDB issues, and something with devstack_gate and null bytes
15:31:38 mriedem kashyap: i don't see you or aspiers as +1 on that bug fix
15:31:40 stephenfin None of which I know how to fix :(
15:32:00 kashyap mriedem: Damn, I have it sit in a tab, let me do that
15:36:15 openstackgerrit Sylvain Bauza proposed openstack/nova master: WIP: Add a placement audit command https://review.opendev.org/670112
15:41:29 shilpasd dansmith: i have addressed your minor nit isolated aggregate spec on https://review.opendev.org/#/c/675384/
15:44:44 mriedem i guess that wasn't related to aspiers so ignore the ping
15:47:22 dansmith shilpasd: you only changed it in that one place
15:47:36 dansmith shilpasd: it should be changed elsewhere too, except maybe in work items..
15:47:49 shilpasd dansmith: okay, will do that
15:53:27 mloza Hello, I'm getting tons of 'Unexpected exception in API method: MigrationNotFound:' in nova-api. Here is the full log http://paste.openstack.org/show/777427/
15:54:35 mloza I don't see pending migrations in `nova migration-list`
15:54:59 mloza I'm using stable/stein branch
15:55:40 openstackgerrit Matt Riedemann proposed openstack/nova master: libvirt: Get the CPU model, not 'arch' from get_capabilities() https://review.opendev.org/682267
15:58:24 openstackgerrit sean mooney proposed openstack/nova master: make config drives sticky bug 1835822 https://review.opendev.org/669738
15:58:25 openstack bug 1835822 in OpenStack Compute (nova) "vms loose acess to config drive with CONF.force_config_drive=True after hard reboot" [Medium,In progress] https://launchpad.net/bugs/1835822 - Assigned to Matt Riedemann (mriedem)
15:58:36 sean-k-mooney rebase to kick it down the queue
15:58:39 kashyap mriedem: Thanks for updating
16:00:19 openstackgerrit Merged openstack/nova master: Remove an unused file and a related description https://review.opendev.org/681955
16:01:59 openstackgerrit Shilpa Devharakar proposed openstack/nova-specs master: Update spec: filtering of alloc candidates by forbidden aggregates https://review.opendev.org/675384
16:02:09 openstackgerrit Artom Lifshitz proposed openstack/nova master: Add func test for 'required' PCI NUMA policy https://review.opendev.org/682941
16:02:45 artom sean-k-mooney, stephenfin ^^ Dunno if it's that useful given that we test the 'legacy' policy which essentially does the same, but since I noticed it was missing while doing downstream reviews...
16:03:24 sean-k-mooney the legacy policy works differently if you have no device numa info on the host
16:03:44 mriedem artom: is that from you investigating that numa live migration testing thread in the ML?
16:04:06 artom mriedem, nope, from reviewing downstream backports
16:04:19 artom mriedem, he still hasn't posted logs from that ML post, btw
16:04:30 artom Makes it kinda hard to RCA :)
16:04:31 sean-k-mooney mriedem: its from https://review.opendev.org/#/c/674072/ we have a test backport for the customer to try to see if that actully works for them
16:05:15 sean-k-mooney i was wrigitn the spec for this but im fixing the config drive test currently
16:06:55 sean-k-mooney mriedem: we need to backport https://review.opendev.org/#/c/624444/ first and artom noticed that require did not have test coverage
16:07:23 artom Not in func tests, anyways
16:07:40 sean-k-mooney right there are unit tests
16:07:42 efried dansmith: The isolated aggs spec update lgtm https://review.opendev.org/#/c/675384/ want me to fast approve?
16:07:50 mriedem unit tests are generally insufficient for any of these kinds of features
16:08:02 mriedem at this point in nova, unit tests are generally insufficient for just about anything
16:08:02 artom mriedem, the same could be said for func tests ;)
16:08:18 mriedem artom: for low level hardware stuff sure
16:08:21 mriedem i wouldn't see that in general
16:08:34 artom mriedem, I mean low level hardware stuff, like PCI or NUMA
16:09:05 mriedem so the broken feature that landed in queens is just getting rolled out to the customer that wanted it now?
16:09:19 mriedem 18 months later...right on time
16:09:26 sean-k-mooney yes
16:09:32 mriedem and queens is your current LTS?
16:09:51 artom Long live the queen, indeed
16:09:59 sean-k-mooney newton will be end of line in decmeber and queens is our next LTS release downstream
16:10:17 sean-k-mooney so people are upgrading
16:11:13 mnaser this maybe more of a libvirt question, but i'll ask.. `cpu_mode` defaults to `host-model` in nova when using kvm, but the libvirt definition seems to be `<cpu mode='custom' match='exact' check='full'> ... </cpu>` (this is what i gather from virsh dumpxml)
16:11:29 mriedem mnaser: meet kashyap
16:11:38 mriedem <3
16:11:48 mnaser reading the nova code, it seems that nova is doing the right thing(tm)
16:12:16 sean-k-mooney mnaser: libvirt is also ment to default to host model if we say nothing for what its worth
16:12:33 mnaser right, im just checkin an issue where nested virt is available, virsh capabilities shows vmx
16:12:36 mnaser but vms dont spawn with it
16:12:39 sean-k-mooney but i dont think it always did which is why nova does.
16:12:57 efried I'm going to be taking most of the rest of the week off (family in town). Will check in periodically. Email if you need to reach me. o/
16:13:01 mnaser so i can add it to the list of extra flags (or try and solve it upstream :])
16:13:19 sean-k-mooney do you have a warning about svm not abliable in teh qemu instance log
16:13:31 mriedem mnaser: there are a bunch of recently updated docs here as well https://docs.openstack.org/nova/stein/admin/configuration/hypervisor-kvm.html#specify-the-cpu-model-of-kvm-guests
16:13:33 mriedem if that helps
16:13:52 mnaser ya i went over those, i can totally do passthrough or the other options, but right now my combination should just work ideally
16:14:10 mnaser man it would be nice if we split otu some of the libvirt driver code
16:14:13 mnaser so github search indexes it
16:14:13 mnaser :p
16:14:16 sean-k-mooney qemu does not always detect the host model correctly and somethime expose amd version of vmx (svm) if you use host model
16:14:30 sean-k-mooney and nested virt is enabled
16:14:50 mnaser yeah but the libvirt definition i see here clearly says use a custom set of stuff forbidding callback too
16:15:13 mnaser with a bunch of "requires" too
16:15:30 sean-k-mooney can you paste the full xml somewhere
16:15:32 mnaser i dont know if virsh dumpxml shows the thing that virsh determines or the thing that it was fed
16:15:48 sean-k-mooney i dont think we should be listing a lot of features in the xml
16:15:52 mnaser http://paste.openstack.org/show/777429/
16:16:35 sean-k-mooney do you have the extra cpu flags config set out of interest
16:16:48 mnaser https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10171-L10174
16:16:50 mnaser nope none are set
16:16:51 mnaser but that..
16:16:52 mriedem stephenfin: https://review.opendev.org/#/c/681938/ now depends on your fix so once that passes i'll review your fix
16:17:05 mnaser that doesnt look right
16:17:29 stephenfin mriedem: You saw https://review.opendev.org/#/c/682111/, I assume?

Earlier   Later