| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-18 | |||
| 15:13:05 | bauzas | mriedem: just to make it clear, there are cases where the migration goes into ERROR and then we have orphaned allocations, right? | |
| 15:13:06 | efried | if the patch itself needs changes. | |
| 15:13:22 | sean-k-mooney | efried: taht will kick off new jobs | |
| 15:13:32 | efried | yes, but at the back of the queue :) | |
| 15:13:32 | sean-k-mooney | ill just try to fix it before it starts running | |
| 15:13:33 | bauzas | mriedem: I'd like to mimic some behaviour | |
| 15:13:41 | sean-k-mooney | ok | |
| 15:13:44 | sean-k-mooney | ill do that | |
| 15:13:46 | efried | I think actually you might be able to pull it completely by abandoning and restoring | |
| 15:14:00 | efried | um, restore might start it again | |
| 15:14:11 | efried | so you could abandon until you're ready to push new ps | |
| 15:16:46 | kashyap | sfinucan: Before the GA, this should go in, yeah? The typo-that-matters in driver.py: https://review.opendev.org/#/c/682267/3/nova/virt/libvirt/driver.py | |
| 15:17:08 | mriedem | gibi: sean-k-mooney: commenting on sean's patch | |
| 15:19:08 | mriedem | commented | |
| 15:19:11 | mriedem | seems the only issue is reboot | |
| 15:19:57 | mriedem | sorry i was looking at unshelve | |
| 15:21:33 | openstackgerrit | Shilpa Devharakar proposed openstack/nova-specs master: Update spec: filtering of alloc candidates by forbidden aggregates https://review.opendev.org/675384 | |
| 15:21:35 | mriedem | i think in either case waiting for task_state to be None is sufficient, but waiting on notifications also works | |
| 15:22:28 | gibi | mriedem: replied | |
| 15:26:03 | mriedem | bauzas: for that audit command i would have started with functional tests months ago | |
| 15:26:09 | mriedem | unit tests don't mean much for stuff like this | |
| 15:26:20 | mriedem | in fact, i don't like unit tests really for any of these types of operational commands | |
| 15:26:22 | bauzas | mriedem: I can send you a new revision now, FWIW | |
| 15:26:36 | bauzas | mriedem: but I just wonder what to test from a migration usecase | |
| 15:26:54 | bauzas | but I think I'll just mock a failing migration that leaves an allocation record | |
| 15:26:57 | mriedem | there are notes about migration in the bug report aren't there? | |
| 15:27:18 | bauzas | correct, there are examples where we found orphaned allocations | |
| 15:27:23 | mriedem | i don't think mocking is really good here, i'm pretty sure we have known ways to hit stuff like that, with evacuate | |
| 15:28:02 | mriedem | https://review.opendev.org/#/c/678100/ has a couple of related bugs where we orphan allocations/providers | |
| 15:28:06 | bauzas | what I basically do is to compare whether I can find an active migration against an existing allocation | |
| 15:28:22 | bauzas | will look at the change ^ | |
| 15:28:38 | bauzas | and you know what ? I'll release the change now so people can voice against it | |
| 15:29:07 | mriedem | kashyap: i've tagged the bug with train-rc-potential | |
| 15:29:30 | kashyap | mriedem: Ah, thank you. Didn't think of it | |
| 15:29:53 | kashyap | IIRC, stephenfin wanted to hold off on it until the 'cpu-resources' merges | |
| 15:30:10 | mriedem | so like the day of RC1 | |
| 15:30:12 | mriedem | got it | |
| 15:30:13 | kashyap | (Due to conflicts, etc.) | |
| 15:30:38 | kashyap | mriedem: :D I mentioned it exactly for that -- in case it doesn't merge, that typo shouldn't fall through the cracks | |
| 15:30:49 | kashyap | (And you've helpfully fixed it, by the tag) | |
| 15:30:55 | mriedem | i sure hope everyone rechecking the vpmem and pcpu series is actually looking at every failure to make sure we're not recheck grinding in some regression | |
| 15:31:33 | stephenfin | I am. We're seeing mirror issues, InnoDB issues, and something with devstack_gate and null bytes | |
| 15:31:38 | mriedem | kashyap: i don't see you or aspiers as +1 on that bug fix | |
| 15:31:40 | stephenfin | None of which I know how to fix :( | |
| 15:32:00 | kashyap | mriedem: Damn, I have it sit in a tab, let me do that | |
| 15:36:15 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: WIP: Add a placement audit command https://review.opendev.org/670112 | |
| 15:41:29 | shilpasd | dansmith: i have addressed your minor nit isolated aggregate spec on https://review.opendev.org/#/c/675384/ | |
| 15:44:44 | mriedem | i guess that wasn't related to aspiers so ignore the ping | |
| 15:47:22 | dansmith | shilpasd: you only changed it in that one place | |
| 15:47:36 | dansmith | shilpasd: it should be changed elsewhere too, except maybe in work items.. | |
| 15:47:49 | shilpasd | dansmith: okay, will do that | |
| 15:53:27 | mloza | Hello, I'm getting tons of 'Unexpected exception in API method: MigrationNotFound:' in nova-api. Here is the full log http://paste.openstack.org/show/777427/ | |
| 15:54:35 | mloza | I don't see pending migrations in `nova migration-list` | |
| 15:54:59 | mloza | I'm using stable/stein branch | |
| 15:55:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: Get the CPU model, not 'arch' from get_capabilities() https://review.opendev.org/682267 | |
| 15:58:24 | openstackgerrit | sean mooney proposed openstack/nova master: make config drives sticky bug 1835822 https://review.opendev.org/669738 | |
| 15:58:25 | openstack | bug 1835822 in OpenStack Compute (nova) "vms loose acess to config drive with CONF.force_config_drive=True after hard reboot" [Medium,In progress] https://launchpad.net/bugs/1835822 - Assigned to Matt Riedemann (mriedem) | |
| 15:58:36 | sean-k-mooney | rebase to kick it down the queue | |
| 15:58:39 | kashyap | mriedem: Thanks for updating | |
| 16:00:19 | openstackgerrit | Merged openstack/nova master: Remove an unused file and a related description https://review.opendev.org/681955 | |
| 16:01:59 | openstackgerrit | Shilpa Devharakar proposed openstack/nova-specs master: Update spec: filtering of alloc candidates by forbidden aggregates https://review.opendev.org/675384 | |
| 16:02:09 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Add func test for 'required' PCI NUMA policy https://review.opendev.org/682941 | |
| 16:02:45 | artom | sean-k-mooney, stephenfin ^^ Dunno if it's that useful given that we test the 'legacy' policy which essentially does the same, but since I noticed it was missing while doing downstream reviews... | |
| 16:03:24 | sean-k-mooney | the legacy policy works differently if you have no device numa info on the host | |
| 16:03:44 | mriedem | artom: is that from you investigating that numa live migration testing thread in the ML? | |
| 16:04:06 | artom | mriedem, nope, from reviewing downstream backports | |
| 16:04:19 | artom | mriedem, he still hasn't posted logs from that ML post, btw | |
| 16:04:30 | artom | Makes it kinda hard to RCA :) | |
| 16:04:31 | sean-k-mooney | mriedem: its from https://review.opendev.org/#/c/674072/ we have a test backport for the customer to try to see if that actully works for them | |
| 16:05:15 | sean-k-mooney | i was wrigitn the spec for this but im fixing the config drive test currently | |
| 16:06:55 | sean-k-mooney | mriedem: we need to backport https://review.opendev.org/#/c/624444/ first and artom noticed that require did not have test coverage | |
| 16:07:23 | artom | Not in func tests, anyways | |
| 16:07:40 | sean-k-mooney | right there are unit tests | |
| 16:07:42 | efried | dansmith: The isolated aggs spec update lgtm https://review.opendev.org/#/c/675384/ want me to fast approve? | |
| 16:07:50 | mriedem | unit tests are generally insufficient for any of these kinds of features | |
| 16:08:02 | artom | mriedem, the same could be said for func tests ;) | |
| 16:08:02 | mriedem | at this point in nova, unit tests are generally insufficient for just about anything | |
| 16:08:18 | mriedem | artom: for low level hardware stuff sure | |
| 16:08:21 | mriedem | i wouldn't see that in general | |
| 16:08:34 | artom | mriedem, I mean low level hardware stuff, like PCI or NUMA | |
| 16:09:05 | mriedem | so the broken feature that landed in queens is just getting rolled out to the customer that wanted it now? | |
| 16:09:19 | mriedem | 18 months later...right on time | |
| 16:09:26 | sean-k-mooney | yes | |
| 16:09:32 | mriedem | and queens is your current LTS? | |
| 16:09:51 | artom | Long live the queen, indeed | |
| 16:09:59 | sean-k-mooney | newton will be end of line in decmeber and queens is our next LTS release downstream | |
| 16:10:17 | sean-k-mooney | so people are upgrading | |
| 16:11:13 | mnaser | this maybe more of a libvirt question, but i'll ask.. `cpu_mode` defaults to `host-model` in nova when using kvm, but the libvirt definition seems to be `<cpu mode='custom' match='exact' check='full'> ... </cpu>` (this is what i gather from virsh dumpxml) | |
| 16:11:29 | mriedem | mnaser: meet kashyap | |
| 16:11:38 | mriedem | <3 | |
| 16:11:48 | mnaser | reading the nova code, it seems that nova is doing the right thing(tm) | |
| 16:12:16 | sean-k-mooney | mnaser: libvirt is also ment to default to host model if we say nothing for what its worth | |
| 16:12:33 | mnaser | right, im just checkin an issue where nested virt is available, virsh capabilities shows vmx | |
| 16:12:36 | mnaser | but vms dont spawn with it | |
| 16:12:39 | sean-k-mooney | but i dont think it always did which is why nova does. | |
| 16:12:57 | efried | I'm going to be taking most of the rest of the week off (family in town). Will check in periodically. Email if you need to reach me. o/ | |
| 16:13:01 | mnaser | so i can add it to the list of extra flags (or try and solve it upstream :]) | |
| 16:13:19 | sean-k-mooney | do you have a warning about svm not abliable in teh qemu instance log | |
| 16:13:31 | mriedem | mnaser: there are a bunch of recently updated docs here as well https://docs.openstack.org/nova/stein/admin/configuration/hypervisor-kvm.html#specify-the-cpu-model-of-kvm-guests | |
| 16:13:33 | mriedem | if that helps | |