| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-12-02 | |||
| 17:02:00 | stephenfin | melwitt: Sure did. It's on the list of stuff still to clean up | |
| 17:02:23 | melwitt | stephenfin: k, coolness | |
| 17:10:45 | Sundar | Responded to the discussion on exit_wait_early for instance events in https://review.opendev.org/#/c/631244 | |
| 17:12:33 | Sundar | efried: stephenfin: ^ relates to https://review.opendev.org/#/c/695985/ | |
| 17:12:34 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/stein: libvirt: Add a rbd_connect_timeout configurable https://review.opendev.org/669167 | |
| 17:12:36 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/rocky: libvirt: Add a rbd_connect_timeout configurable https://review.opendev.org/669168 | |
| 17:12:39 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: libvirt: Add a rbd_connect_timeout configurable https://review.opendev.org/669169 | |
| 17:12:46 | lyarwood | melwitt: ^ reopened and rebased | |
| 17:16:48 | melwitt | lyarwood: ack thanks | |
| 17:19:21 | mriedem | lyarwood: did anyone that reported that issue actually say the fix resolved it for them? | |
| 17:19:39 | lyarwood | mriedem: we've just sent a test build out to a stable/queens user | |
| 17:20:14 | lyarwood | mriedem: it's going to fix their issues but will at least confirm that their network is borked | |
| 17:31:08 | efried | Sundar: back atcha. To be clear, I'm not seeing a problem; are you? | |
| 17:33:36 | Sundar | efried: The notification from Cyborg will fail in many/most cases; ignoring that error could be considered a hack, rather than a solution. After all, folks looking at the n-api logs will see an error. Are we ok with that? | |
| 17:33:57 | sean-k-mooney | gibi: this is the bug for the issue with live migrating https://bugs.launchpad.net/nova/+bug/1854844. ill try to think about ways to adress this in a backportable way. i think i know of one way but this might just be something we have to fix on master only | |
| 17:33:58 | openstack | Launchpad bug 1854844 in OpenStack Compute (nova) "libvirt: tx/rx queue lenght and max queues are not updtated on live migration" [Low,Triaged] - Assigned to sean mooney (sean-k-mooney) | |
| 17:34:28 | efried | Sundar: Only if they're looking at debug logs :) | |
| 17:35:00 | efried | The cyborg logs should of course explain what's happening as well. | |
| 17:35:17 | efried | And yes, I'm fine with this. | |
| 17:37:49 | Sundar | efried: From Cyborg's POV, the notification after binding would be best-effort: ignore any failures and complete the binding. The Cyborg logs will indicate as much. | |
| 17:38:25 | sean-k-mooney | gibi: we might be able to store the queue paramater in the port binding profile which is an unversioned dict of strigns filed and was intend to store hypervior specific info for a port. | |
| 17:39:13 | efried | Sundar: Not *any* failures. | |
| 17:39:18 | Sundar | If we're all ok with that, I'm good. But there are a few nuances to consider. First, the function to get the resolved ARQs is now defined in https://review.opendev.org/631245, while it needs to be used in the previous patch https://review.opendev.org/631244. | |
| 17:39:24 | sean-k-mooney | gibi: the use of multiple port bindings for live migration and the rx/tx queue size options were both intoduced in rocky so it should be backportable to all affected version if i do it right | |
| 17:40:00 | efried | Sundar: As dansmith suggested (oh, maybe it was in IRC, not in the patch) you should still fail and clean up the binding if you get e.g. a 404. | |
| 17:40:10 | efried | meaning the instance was deleted by the time you went to emit the notification. | |
| 17:40:55 | sean-k-mooney | gibi: well assuming the call order works out ill follow up with this later | |
| 17:41:25 | efried | So, to be clear, the 422 is a special case, with a prominent NOTE and a nice debug log stating that the binding completed before the instance landed on the host, so it's acceptable that the notification is dropped. | |
| 17:42:06 | efried | Sundar: refactoring method X from patch Z into patch Y because that's where it's now going to be used, yeah, that's SOP. | |
| 17:42:19 | Sundar | efried: There could be a bunch of different failure modes: cannot locate n-api service, located but it timed out?, etc. We need to distinguish that category from 'called n-api and it returned an error'. In the latter case, is there granularity of error codes for why it failed? | |
| 17:42:50 | efried | Sundar: Yes, the 422 is a special case; any other error is an error. | |
| 17:43:41 | Sundar | Need to ceck if 422 is used for any other error case. | |
| 17:43:44 | Sundar | *check | |
| 17:47:10 | Sundar | Re. the patch series, I think we need to squash https://review.opendev.org/631245 into the previous patch https://review.opendev.org/631244. Because once the get_resolved_arqs() moves to previous patch, the only things left behind are the virt driver changes. It would arguably be easier to understand those changes together with the changes to get | |
| 17:47:11 | Sundar | the arqs in compute/manager.py. | |
| 17:50:49 | mriedem | nice https://docs.openstack.org/nova/latest/admin/#overview - "nova-api Receives XML requests" | |
| 17:56:17 | openstack | Launchpad bug 1289064 in OpenStack Compute (nova) "live migration of instance should claim resources on target compute node" [Medium,In progress] - Assigned to Artom Lifshitz (notartom) | |
| 17:56:17 | melwitt | stephenfin: I noticed a new recent comment on this bug https://bugs.launchpad.net/nova/+bug/1289064 saying, maybe it can be closed bc numa aware lm was merged? seems so to me but wanted to check with you | |
| 17:56:35 | Sundar | Only one use of 422 error code in create: https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/server_external_events.py . So, we are probably good. | |
| 17:56:58 | efried | yup (I checked that earlier) | |
| 17:57:06 | artom | melwitt, stephenfin, oh, well - not entirely - only NUMA LM does claims | |
| 17:57:19 | artom | Then again, only NUMA LM has non-placement resources, so maybe yes? | |
| 17:57:26 | artom | SRIOV is handled as well, albeit separately | |
| 17:58:26 | melwitt | er, or artom, sorry ^ (I pinged stephen bc his patches about rejecting lm requests that have numa topology were merged as Related-Bug) | |
| 17:59:28 | melwitt | hm, ok. I don't really understand what that means so I dunno either | |
| 17:59:29 | sean-k-mooney | melwitt: ya so stephens patch just reject live migration if the guest has a numa toplogy at the api | |
| 18:00:07 | sean-k-mooney | sriov live migration directly allocates and claims the devices during live migration | |
| 18:00:22 | sean-k-mooney | numa live migration uses a move claim to claim resouces | |
| 18:00:58 | sean-k-mooney | (cpus and mempages/hugepages) | |
| 18:02:03 | artom | sean-k-mooney, so, could we say that all non-placement resources are not accounted correctly during live migration? | |
| 18:02:15 | sean-k-mooney | no | |
| 18:02:33 | sean-k-mooney | i dont think this is an issue on master | |
| 18:02:47 | artom | Err, *are accounted | |
| 18:02:50 | eandersson | Looks good gibi | |
| 18:03:18 | eandersson | btw mriedem did you see the minor bug I mentioned over the weekend | |
| 18:03:47 | sean-k-mooney | artom: right non-placement rescoues (cpu,mempages,sriov devices) should be accounted for in train+ and we blocked numa/sriov migration before that retoactivly | |
| 18:04:06 | openstack | Launchpad bug 1289064 in OpenStack Compute (nova) "live migration of instance should claim resources on target compute node" [Medium,In progress] - Assigned to Artom Lifshitz (notartom) | |
| 18:04:06 | sean-k-mooney | artom: melwitt so we might just be able to close https://bugs.launchpad.net/nova/+bug/1289064 | |
| 18:04:28 | eandersson | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2075 <-- eventlet.Timeout (based on BaseException, not Exception) may be raised here causing an UnboundError. | |
| 18:05:32 | mriedem | eandersson: nope | |
| 18:06:03 | melwitt | sean-k-mooney: if there's not anything more to do there, I'd think it would be nice to close it | |
| 18:07:10 | sean-k-mooney | well there are still cases where live migration is not supported but on master i think we account for all resouces correctly in the cases where live migration is supported | |
| 18:07:17 | mriedem | eandersson: open a bug | |
| 18:07:49 | artom | sean-k-mooney, melwitt, so.... fix release 'Train'? | |
| 18:08:17 | dansmith | efried: Sundar: I was on a call during your discussion, but...we're all good now? | |
| 18:08:20 | sean-k-mooney | artom: ya i think so | |
| 18:09:06 | melwitt | yeah, I can't remember how/if there's a way to mark it as fixed in the past. but I don't think that matters too much | |
| 18:09:25 | artom | Done. | |
| 18:09:29 | sean-k-mooney | you can add the sepcific release series to the bug | |
| 18:09:41 | sean-k-mooney | i left a comment in the bug too | |
| 18:10:12 | openstack | Launchpad bug 1854848 in OpenStack Compute (nova) "build_and_run_instance can error out with an UnboundException" [Undecided,New] | |
| 18:10:12 | eandersson | mriedem https://bugs.launchpad.net/nova/+bug/1854848 | |
| 18:10:28 | Sundar | efried: I think so, if you and all are ok with whatever I said above -- squishing the patches, etc. ignoring 422 alone seems fine. if n-api is unreachable or times out, that will still be an error that fails the binding. | |
| 18:10:30 | sean-k-mooney | artom: added train | |
| 18:10:39 | eandersson | It's a minor issue, but caused VMs to get stuck in BUILD... fyi was caused by selinux not being configured properly. | |
| 18:11:04 | sean-k-mooney | artom: melwitt there is a "Target to series" button you can use | |
| 18:11:17 | sean-k-mooney | you need to be in the nova bug team on LP to see/use it | |
| 18:12:09 | melwitt | yeah, I have that, but I don't think that will stop it from saying "released in <U version number>" whenever it's released next | |
| 18:12:58 | melwitt | it'll still count it as landing "today" first and then "backported" to train. I don't think it matters, just that's what the automation is gonna do when it comments later | |
| 18:13:56 | sean-k-mooney | ya proably | |
| 18:14:39 | sean-k-mooney | i dont think we have automation tieded to that out side of the completed specs thing but that uses blueprints not bugs | |
| 18:16:11 | sean-k-mooney | artom: if you are triaging nova bugs upstream you should join https://launchpad.net/~nova-bugs its an open team so you will be automatically approved | |
| 18:16:32 | artom | sean-k-mooney, can't say I'm triaging dilligently | |
| 18:17:25 | sean-k-mooney | well you know where to find it if you change your mind | |
| 18:17:52 | melwitt | sean-k-mooney: I mean the thing that will make a comment on the bug that says "this bug was fixed in version x.x.x.x" that will be technically wrong because it's gonna say a U version. again, I don't think it matters, just that's what it's going to say on the comment | |
| 18:18:12 | sean-k-mooney | 00. | |
| 18:18:14 | sean-k-mooney | 00.1 | |
| 18:18:20 | sean-k-mooney | droped phone | |
| 18:18:42 | sean-k-mooney | am isnt that done by the release tooling | |
| 18:18:48 | sean-k-mooney | not launchpad | |
| 18:19:15 | melwitt | it is yeah | |
| 18:19:28 | sean-k-mooney | but ya i guess in this case its not going to be set since artoms change did not have closes bug | |
| 18:20:00 | sean-k-mooney | anyway it should be fine | |
| 18:20:17 | melwitt | I don't think it matters, I think it just does a sweep through things that are Fix Released and makes a comment with the fixed in version | |
| 18:27:47 | efried | dansmith: Yes, thanks. | |
| 18:30:16 | dansmith | efried: so there'll need to be a change to cyborg for the 422 thing presumably | |
| 18:30:24 | efried | yes | |
| 18:31:02 | dansmith | efried: Sundar: is there a testing patch somewhere in the series that wires this all up with a fake cyborg driver? | |
| 18:31:22 | dansmith | I assume that would be the one that needs to depends-on that cyborg change | |
| 18:31:26 | efried | the CI job is working last I checked | |