| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-20 | |||
| 13:00:55 | gibi | your series | |
| 13:01:00 | mriedem | you can rebase it, | |
| 13:01:06 | mriedem | also, i've duplicated your bug against 1843090 | |
| 13:01:17 | gibi | mriedem: thanks. I will update my patch accordingly | |
| 13:01:42 | gibi | mriedem: I reached this bug while tried to create a func test for the bandwidth case when rpc in pinned | |
| 13:02:06 | gibi | mriedem: it seems there will also be extra issues | |
| 13:02:07 | mriedem | yeah it was this patch that made me think of it in your bw series for prep_resize where you were using the request spec | |
| 13:03:36 | gibi | mriedem: btw do you know about a bug regading booting a instance with --availablity-zone az:node and then migrate it. I saw that in this case nova does not try to re-schedule the migration if the first dest host fails in prep_resize | |
| 13:03:57 | gibi | it seem the that filter_properties are not populated for re-schedule | |
| 13:04:17 | gibi | if I boot the server with the new host parameter then the re-schedule work | |
| 13:05:18 | mriedem | gibi: that's working as designed https://opendev.org/openstack/nova/src/branch/master/nova/scheduler/utils.py#L801 | |
| 13:05:44 | mriedem | using az::node will set force_nodes | |
| 13:06:07 | mriedem | do you see "Re-scheduling is disabled due to forcing a host" in the logs? | |
| 13:06:13 | gibi | mriedem: but forcing the host during the boot allows me to migrate it without dest host specified, but it does not allow nova to re-schedule | |
| 13:06:27 | gibi | during that migrate | |
| 13:07:06 | gibi | mriedem: I have to go back and re-create the situation as I saw this while I created the func test for the rpc pin bug. | |
| 13:07:26 | mriedem | gibi: which release? https://review.opendev.org/#/q/I3f488be6f3c399f23ccf2b9ee0d76cd000da0e3e | |
| 13:07:47 | gibi | mriedem: master :) | |
| 13:07:54 | mriedem | oh nvm that's ignore_hosts | |
| 13:08:46 | mriedem | so i think force_hosts/nodes gets persisted for some reason, i'm not sure why, but for every move operation we call this to basically unset those fields https://opendev.org/openstack/nova/src/branch/master/nova/objects/request_spec.py#L692 | |
| 13:08:55 | mriedem | it would be simpler if we just didn't persist those values | |
| 13:09:24 | mriedem | gibi: so maybe we're calling ^ *after* populate_retry | |
| 13:10:03 | mriedem | yup https://opendev.org/openstack/nova/src/branch/master/nova/conductor/tasks/migrate.py#L293 | |
| 13:11:29 | gibi | mriedem: so is the order of pupulate and rest wrong there? | |
| 13:11:53 | mriedem | it appears so | |
| 13:11:54 | gibi | s/pupulate/populate_retry | |
| 13:12:07 | gibi | mriedem: OK I will create a reproduction for that as well | |
| 13:12:10 | mriedem | i'm checking if that's a regression | |
| 13:13:37 | mriedem | gibi: looks like it's been that way since newton https://review.opendev.org/#/c/284974/18/nova/conductor/tasks/migrate.py | |
| 13:14:02 | gibi | mriedem: then nobody cares :D | |
| 13:14:42 | mriedem | ha | |
| 13:14:51 | gibi | mriedem: btw there are re-schedule func tests that uses az:node boot, so those will start behaving differently if we fix this | |
| 13:14:52 | mriedem | or nobody just ever thought to ask if that's wrong | |
| 13:15:19 | mriedem | dansmith: bauzas: can you think of any rason why we persist RequestSpec.force_hosts/nodes? | |
| 13:15:31 | mriedem | like many of the other request spec fields, persisting that seems to only cause headaches | |
| 13:15:34 | bauzas | PEBKAC ? | |
| 13:16:00 | bauzas | I thought we have a method for this | |
| 13:16:13 | mriedem | i understand that when request spec was originally written everything was persisted, but we've rolled back a lot of that piece by piece as we find problems | |
| 13:16:24 | mriedem | bauzas: yeah, but that's kind of ... dumb, right? | |
| 13:16:30 | bauzas | mriedem: https://github.com/openstack/nova/blob/master/nova/objects/request_spec.py#L692 | |
| 13:16:36 | mriedem | we could just not persist the damn field and then we wouldn't have to explicitly reset for every move operation | |
| 13:16:38 | bauzas | mriedem: yeah, this | |
| 13:17:09 | bauzas | mriedem: when alaski provided the persistence for the spec, his and me forgot to discuss about this :( | |
| 13:17:15 | bauzas | him* | |
| 13:17:22 | bauzas | so we created this method | |
| 13:18:21 | mriedem | ok but going forward, to avoid new ways of shooting off our toes, we could just stop persisting force_hosts/nodes and only rely on that method as a workaround for existing request specs | |
| 13:19:39 | mriedem | these are the fields i count that we've changed to not be persisted on the requestspec: ignore_hosts, requested_destination, retry, network_metadata, requested_resources | |
| 13:20:08 | mriedem | the latter 2 are newer and were not persisted since they were introduced, but the first 3 were all retroactive | |
| 13:20:39 | mriedem | oh and the instance_group members/hosts https://github.com/openstack/nova/blob/master/nova/objects/request_spec.py#L614 | |
| 13:21:20 | mriedem | which reminds me, if any stable core cares about pike and ocata https://review.opendev.org/#/q/topic:bug/1830747+status:open | |
| 13:21:37 | bauzas | mriedem: cool with this | |
| 13:22:10 | bauzas | mriedem: I was also thinking about some way to say which specific fields shouldn't be persisted | |
| 13:22:31 | bauzas | mriedem: and sure, will look at the stable changes | |
| 13:23:40 | mriedem | gibi: as for existing functional resize reschedule tests using az::node, i'd have to see the impacts to assess really | |
| 13:23:59 | bauzas | mriedem: efried_pto: btw. https://review.opendev.org/#/c/683327/ | |
| 13:24:02 | mriedem | but if we ignore the forced host/node to boot the server during resize scheduling, it seems we should allow rescheduling to alternates | |
| 13:26:12 | gibi | mriedem: sure. we will see when I managed to write the reproduction and reorder the populate_retry call | |
| 13:27:04 | mriedem | stephenfin: are you working on an admin guide docs patch for pcpu? | |
| 13:27:12 | stephenfin | mriedem: yup | |
| 13:27:16 | stephenfin | atm, in fact | |
| 13:27:39 | stephenfin | should have something ready to go before EOD | |
| 13:35:59 | mriedem | bauzas: notes on the prelude patch | |
| 13:36:06 | bauzas | cool | |
| 13:40:26 | mriedem | weee https://zuul.opendev.org/t/openstack/build/e3f767dbd32247ffa5aa7daa79e0e5af/log/job-output.txt#32748 | |
| 13:40:31 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Func test for migrate reschedule with pinned compute rpc https://review.opendev.org/683385 | |
| 13:41:21 | gibi | mriedem: reproduction for 1843090 ^^ | |
| 13:41:52 | gibi | mriedem: now I'm going to do the rebase of your fix on it | |
| 13:42:15 | mriedem | gibi: when you rebase you might as well strike whatever word efried_pto didn't like from my comment | |
| 13:42:23 | gibi | mriedem: sure thing | |
| 13:52:39 | mriedem | gibi: comments in your functional recreate patch | |
| 13:54:59 | gibi | mriedem: thanks. regarding the request_spec uglyness. I can cook up something that is test only. But sooner or later we (I) need to go back and fix this uglyness as it is now needed in two places (as noted) | |
| 13:55:50 | gibi | mriedem: the rest of your comments is valid and clear. | |
| 13:55:59 | mriedem | yeah i have no idea why that object is a problem b/c we pass request spec over rpc (select_destinations and such) all the time | |
| 13:56:09 | mriedem | in tests i mean | |
| 13:56:11 | gibi | mriedem: in the meantime I can tell you that your fix works according to the functional test | |
| 13:56:19 | mriedem | \o/ | |
| 13:56:45 | gibi | ... what a nice Friday it is | |
| 14:02:08 | mriedem | my kid is also staying home from school today claiming to be sick and i can't tell if she's playing opposum | |
| 14:02:17 | mriedem | i need to put her to work | |
| 14:05:20 | artom | How good is she with Python? Weren't we looking for interns for all the osc gaps? | |
| 14:05:28 | gibi | mriedem: Is there a change that we consider this https://review.opendev.org/#/c/672577/ as a bug and backport it to stable/pike? (yeah I have a bug day today) | |
| 14:05:58 | gibi | artom, mriedem: hm osc gaps, and osc bugfix backport ^^ | |
| 14:07:48 | mriedem | aspiers: you might want to talk to some suse developers downstream about their upstream "fix only on the branch i care about" practices https://review.opendev.org/#/c/676882/ | |
| 14:08:31 | mriedem | gibi: s/change/chance/ ? | |
| 14:08:37 | gibi | mriedem: yes, sorry | |
| 14:08:43 | mriedem | going to pike is probably going to be tough... | |
| 14:09:02 | mriedem | but, that fix should be compatible... | |
| 14:09:12 | gibi | mriedem: OK. I will look into that as well | |
| 14:09:14 | mriedem | it's clearly a bug if you're using >= 2.53 | |
| 14:09:25 | gibi | mriedem: or find somebody who has time | |
| 14:09:40 | mriedem | pike is in extended maintenance upstream so i'm not sure how much dtroyer is going to care about backporting it that far | |
| 14:09:46 | gibi | mriedem: yes, we have pike deployemnts out there using 2.53 and seeing openstack client fail | |
| 14:10:22 | mriedem | gibi: are they pinning the version to 2.latest or something? | |
| 14:10:50 | mriedem | or they are just pinning to 2.53 since that's the max in pike? | |
| 14:10:53 | gibi | mriedem: using 2.53 by default, but sure we can go back to olderin for this command as a workaround | |
| 14:11:06 | gibi | mriedem: 2.53 is used as that is max pike | |
| 14:11:42 | gibi | it seems I was able to hook in elod to do this client backports... | |
| 14:11:52 | mriedem | i started stein for you https://review.opendev.org/#/c/683394/1 | |
| 14:12:10 | gibi | mriedem: thanks, elod ^^ | |
| 14:12:27 | mriedem | gibi: btw, where do you get your packages? some upstream distro or do you build your own? | |