| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-19 | |||
| 18:57:55 | mriedem | i saw it before, asked andrey to flesh out the commit message, haven't been back | |
| 19:16:41 | melwitt | ah, ok | |
| 19:28:05 | mnaser | friendly bump on this - https://review.openstack.org/#/c/619352/ | |
| 19:28:21 | mnaser | simple backport, the changes in the newer branches have merged too | |
| 19:29:10 | mriedem | frickler: fyi redo of the queens release https://review.openstack.org/626377 | |
| 19:31:22 | mriedem | duplicate bug of https://review.openstack.org/#/c/567701/ just came through triage, the fix is straight-forward, the patch is mostly a functional test | |
| 19:58:40 | openstackgerrit | sean mooney proposed openstack/nova master: PCI: do not force remove allcoated devices https://review.openstack.org/626381 | |
| 19:59:20 | sean-k-mooney | mriedem: i have no idea why my unit test is not working in ^ | |
| 20:00:56 | sean-k-mooney | im going to grab dinner but if you have any insight let me know. | |
| 20:01:01 | mriedem | ack thanks | |
| 20:33:20 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Document using service user tokens for long running operations https://review.openstack.org/626388 | |
| 20:35:00 | openstackgerrit | Jack Ding proposed openstack/nova-specs master: Select cpu model from a list of cpu models https://review.openstack.org/620959 | |
| 20:39:51 | melwitt | mriedem: re: the ML thread about that, I thought the oslo.messaging heart beat would take care of the long running live migration problem? | |
| 20:40:00 | mriedem | the problem isn't rpc | |
| 20:40:21 | melwitt | oh, the token auth expiring | |
| 20:40:24 | mriedem | nova tries to make a rest api request using the users token to cinder, | |
| 20:40:26 | mriedem | the token has timed out | |
| 20:40:31 | melwitt | I see, ok | |
| 20:41:06 | melwitt | yeah, have to have both then. I got the two confused together but they are two different issues | |
| 20:41:17 | mriedem | i want to say i heard anecdotes at one point that rax public cloud had 24 token timeouts because of stuff like this way back when | |
| 20:41:58 | mriedem | the service user token stuff was added by osic, which was rax+intel | |
| 20:43:55 | melwitt | yeah, sounds familiar. I feel like we had something similar at yahoo too | |
| 20:44:21 | melwitt | *something similar to service user auth | |
| 20:58:21 | mriedem | long_rpc_timeout probably also deserves some mention somewhere in troubleshooting admin docs, but not sure right now, | |
| 20:58:34 | mriedem | in general i've had random thoughts about things that would be good to put into a 'scaling issues' page in the docs | |
| 20:58:39 | openstackgerrit | Krzysztof Opasiak proposed openstack/nova master: Fix server IPs with non-unique network names https://review.openstack.org/625371 | |
| 20:58:40 | mriedem | but haven't started anything | |
| 20:59:18 | melwitt | ++ | |
| 21:15:08 | mriedem | melwitt: i'm going to fix that unnecessary for loop in https://review.openstack.org/#/c/624870/ that Kevin pointd out, then approve | |
| 21:15:46 | melwitt | ok, sounds good | |
| 21:30:17 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Exclude build request marker from server listing https://review.openstack.org/624870 | |
| 21:41:17 | melwitt | mriedem: so the func test in this change doesn't fail without the change https://review.openstack.org/567701 is that expected based on the commit message? if so, is there no way to demonstrate the bug in the test? | |
| 21:42:11 | melwitt | I wasn't sure based on the wording "that is not a regression" | |
| 21:43:05 | mriedem | been awhile, but the commit message is saying I8d426f2635232ffc4b510548a905794ca88d7f99 didn't introduce a regression | |
| 21:43:51 | melwitt | ok, so unrelated to what I'm seeing I think. basically, without the change, somehow AZ is being updated on the instance. I don't yet know how | |
| 21:44:40 | mriedem | i'll have to poke at it, i wrote that in may | |
| 22:00:33 | melwitt | looking at it myself for curiosity, I'm not finding how AZ could be updated without the fix. weird | |
| 22:05:24 | openstackgerrit | Merged openstack/nova master: Move a generic bridge helper to a linux_net privsep file. https://review.openstack.org/620010 | |
| 22:15:03 | mriedem | oh yay, for a looong time we passed potentially the wrong image to move claim during a resize https://github.com/openstack/nova/blob/1249617bdfaa8f4c586159374a4a0b244bbb298a/nova/conductor/tasks/migrate.py#L77 | |
| 22:15:19 | mriedem | based on the original image used to create the server, but potentially not the last image used to rebuild the server | |
| 22:15:24 | mriedem | gd req spec | |
| 22:17:26 | mriedem | which wasn't fixed until https://github.com/openstack/nova/commit/984dd8ad6add4523d93c7ce5a666a32233e02e34 inadvertently | |
| 22:17:54 | melwitt | hoo boy | |
| 22:18:18 | melwitt | me too :( | |
| 22:19:30 | mriedem | oh this also likely means that if you shelve, unshelve, resize, we're passing the original image used to create the server, not the current image meta (in case that changed) | |
| 22:23:56 | melwitt | so the AZ is still the original all the way to the end of the live migration. so how is the servers.get API returning the new AZ... the search continues | |
| 22:25:57 | mriedem | i think i know | |
| 22:26:07 | mriedem | the api code looks up the az from the instance.host | |
| 22:26:12 | mriedem | i think | |
| 22:26:21 | melwitt | pulled from the DB, the AZ is still the original in instance.availability_zone | |
| 22:26:26 | melwitt | ahhhh | |
| 22:26:40 | melwitt | must be, it can't be looking at instance.availability_zone | |
| 22:26:42 | mriedem | there are other bugs for that behavior | |
| 22:26:50 | melwitt | THANKS API | |
| 22:26:55 | mriedem | https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/views/servers.py#L185 | |
| 22:27:16 | melwitt | le sigh | |
| 22:27:18 | mriedem | https://github.com/openstack/nova/blob/master/nova/availability_zones.py#L165 | |
| 22:27:33 | mriedem | yeah if instance.host is not None, that code gets the az from the host aggregate, not the instance.az | |
| 22:27:59 | mriedem | ala https://review.openstack.org/#/c/582342/ | |
| 22:28:00 | melwitt | well, that explains it | |
| 22:28:08 | mriedem | i have a f'ing patch for everything | |
| 22:28:22 | melwitt | it's true | |
| 22:28:30 | openstack | Launchpad bug 1782539 in OpenStack Compute (nova) "Fail to filter the list of instances by the available zone" [Medium,In progress] - Assigned to huanhongda (hongda) | |
| 22:28:30 | mriedem | https://bugs.launchpad.net/nova/+bug/1782539 | |
| 22:33:00 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Pass request_spec from compute to cell conductor on reschedule https://review.openstack.org/582417 | |
| 22:33:11 | mriedem | speak-o-the-turd | |
| 22:34:19 | mriedem | melwitt: so i'll adjust that test to assert based on the db rather than the api | |
| 22:34:51 | melwitt | ok, makes sense | |
| 22:37:17 | melwitt | now, to finish reviewing cross-cell-resize | |
| 22:58:14 | openstackgerrit | Hongbin Lu proposed openstack/nova-specs master: [WIP] Support scheduling VM's NICs to different PFs https://review.openstack.org/626055 | |
| 23:03:11 | tonyb | I'm seeing something 'strange' where placement is always selecting the same host out of the $n available I *suspect* there is cruft in the DB after a bunch of failed boots but I don't really know | |
| 23:03:28 | tonyb | asside form the scheduler and placement logs where should I look? | |
| 23:04:25 | melwitt | tonyb: in the past, default scheduler behavior was to pack instances, so as long as a host has capacity, it would be returned again. I'm not sure if that's still the case now though | |
| 23:04:35 | melwitt | is that what you're seeing or no? | |
| 23:05:05 | mriedem | tonyb: there was a thread in the ops list about this recently, there is an option you can set to randomize the results from placement | |
| 23:05:16 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#placement.randomize_allocation_candidates | |
| 23:05:32 | tonyb | melwitt: Ahh perhaps that's it | |
| 23:05:58 | mriedem | ^ goes in the placement config btw, not nova-scheduler | |
| 23:06:59 | tonyb | mriedem: Okay. I should have said this is queens but I'll look for packing and/or randomising | |
| 23:07:02 | tonyb | Thanks | |
| 23:07:47 | mriedem | looky here | |
| 23:07:48 | mriedem | https://docs.openstack.org/nova/queens/configuration/config.html#placement.randomize_allocation_candidates | |
| 23:08:24 | tonyb | mriedem: Will do. | |
| 23:09:00 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Update instance.availability_zone during live migration https://review.openstack.org/567701 | |
| 23:09:04 | melwitt | in the olden days, IIRC, lots of operators preferred to maximize cloud utilization and packing is the way to do that. but maybe nowadays people prefer to spread out servers and risk not being able to land larger flavors. that's the history on the default as I understand it | |
| 23:09:30 | mriedem | huawei public cloud wants to pack like you've never packed before | |
| 23:09:45 | melwitt | :) | |
| 23:09:47 | mriedem | they want nova to pack even more, | |
| 23:09:54 | mriedem | doing shit like the solver scheduler and what watcher does, | |
| 23:10:01 | melwitt | moar efficiency. not surprising | |
| 23:10:04 | mriedem | dynamically load balancing for ultimate packitude | |
| 23:20:28 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Update port device_owner when unshelving https://review.openstack.org/626407 | |
| 23:22:51 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Update port device_owner when unshelving https://review.openstack.org/626408 | |
| 23:29:09 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Update port device_owner when unshelving https://review.openstack.org/626409 | |
| 23:40:30 | openstackgerrit | Hongbin Lu proposed openstack/nova-specs master: [WIP] Support scheduling VM's NICs to different PFs https://review.openstack.org/626055 | |
| 23:43:42 | openstackgerrit | Merged openstack/nova master: Handle unbound vif plug errors on compute restart https://review.openstack.org/626228 | |
| 23:45:12 | mriedem | jackding: looked at https://review.openstack.org/#/c/603844/ again, i still don't like it | |
| 23:49:29 | openstackgerrit | Merged openstack/nova master: Update port device_owner when unshelving https://review.openstack.org/559828 | |
| 23:49:53 | openstackgerrit | melanie witt proposed openstack/nova stable/rocky: Handle unbound vif plug errors on compute restart https://review.openstack.org/626410 | |
| 23:52:01 | mriedem | tl;dr i don't want to randomly have to hit the neutron API every time we rebuild/reboot to list ports for a server just to check if something is broken | |