| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-16 | |||
| 13:01:16 | stephenfin | Adding another configuration option would be yet another breaking change though | |
| 13:01:48 | stephenfin | So many breaking changes | |
| 13:03:16 | sean-k-mooney | it does not need to be | |
| 13:03:18 | sean-k-mooney | https://review.openstack.org/#/c/555081/23/specs/train/approved/cpu-resources.rst@157 | |
| 13:03:35 | sean-k-mooney | [compute] | |
| 13:03:37 | sean-k-mooney | pinned_cpu_set | |
| 13:03:39 | sean-k-mooney | floating_cpu_set | |
| 13:03:41 | sean-k-mooney | [libvirt] | |
| 13:03:43 | sean-k-mooney | emulator_cpu_set | |
| 13:04:11 | sean-k-mooney | i suggested deprecating cpu_shared_set and renaming/moving it to [libvirt]/emulator_cpu_set | |
| 13:04:46 | sean-k-mooney | so it will not need config change initally althong they will get a deprecation warning | |
| 13:06:06 | stephenfin | Hmm, I need to think about this | |
| 13:07:44 | NewBruce | sean-k-mooney / mnaser … we seem to actually have made this worse - RDO - RDO which was working previously is now also broken | |
| 13:08:21 | sean-k-mooney | is it broken the same way? | |
| 13:08:39 | NewBruce | seems to be | |
| 13:09:06 | NewBruce | same 500 internal error; same duplicate port entries in ml2_port_bindings | |
| 13:09:08 | sean-k-mooney | and all RDO nodes have the same code deployed and config | |
| 13:10:45 | sean-k-mooney | the is begining to feel like it might be related to the specific configuration of some of the compute nodes rather than a gloabl configuration issue but that is a feeling rather then anything based on fact | |
| 13:11:27 | NewBruce | not identical (due to the time span taken to update), we have mostly 18.2.0, but some 18.1.0 and an 18.0.3 | |
| 13:11:47 | NewBruce | we did try rolling an 18.2.0 back to 18.1.0 - but that didn’t seem to help | |
| 13:12:25 | sean-k-mooney | can you try migration between two nodes of the same version on the rdo side for each of the 3 versions you have deployed | |
| 13:12:28 | NewBruce | yeah, i agree its starting to feel something odd (random)… but as you say, feeling - not fact | |
| 13:12:35 | NewBruce | Yeah, will do now | |
| 13:12:50 | sean-k-mooney | perhaps we can narrow it donw to a version and then do a git biset for the changes | |
| 13:17:20 | mnaser | hmm | |
| 13:17:23 | mnaser | this is strange | |
| 13:17:38 | mnaser | NewBruce: so that cloud is 100% rocky at this point? | |
| 13:22:54 | NewBruce | mnaser yeah, 100% rocky | |
| 13:23:01 | NewBruce | … in all senses of the word :D | |
| 13:25:01 | sean-k-mooney | this is the delta in termes of ptaches https://github.com/openstack/nova/compare/18.0.3...18.1.0 | |
| 13:25:57 | mnaser | NewBruce: curl -H "X-Auth-Token: `openstack token issue -c id -f value`" http://network-sjc1.vexxhost.us/v2.0/extensions | python -mjson.tool | grep binding-extended | |
| 13:26:03 | mnaser | change network-sjc1 with your neutron endpoint | |
| 13:26:21 | mnaser | can you run that many several times and see if there is every a time that it doesn't return binding-extended? | |
| 13:26:35 | sean-k-mooney | i wonder if it could be realated to https://github.com/openstack/nova/commit/4a12c9c298913f99570f2f8e93500db687e98dc9 | |
| 13:28:31 | sean-k-mooney | hum although that is only on revert | |
| 13:29:11 | NewBruce | so an RDO 18.1.0 -> RDO 18.1.0 does the same thing; fails on duplicate ports | |
| 13:30:20 | NewBruce | "description": "Expose port bindings of a virtual port to external application", | |
| 13:30:20 | NewBruce | "alias": "binding-extended", | |
| 13:30:20 | NewBruce | "extension": { | |
| 13:30:20 | NewBruce | { | |
| 13:30:20 | NewBruce | mnaser : curl -s -H "X-Auth-Token: $OS_TOKEN" https://kna1.citycloud.com:9696/v2.0/extensions/binding-extended | python -m json.tool | |
| 13:30:21 | NewBruce | "name": "Port Bindings Extended", | |
| 13:30:21 | NewBruce | "links": [], | |
| 13:30:22 | NewBruce | } | |
| 13:30:22 | NewBruce | "updated": "2017-07-17T10:00:00-00:00" | |
| 13:30:23 | NewBruce | } | |
| 13:31:16 | NewBruce | (#SoccerDad duties - back online in about 30 min) | |
| 13:31:17 | mnaser | NewBruce: but did you hit it many different times, and it always gave a response? | |
| 13:31:28 | mnaser | im wondering if one backend might be acting weird | |
| 13:33:49 | sean-k-mooney | mnaser: so you resuggesting runnint it with "watch -d -n 1 ..." and seing if it changes | |
| 13:34:14 | mnaser | sean-k-mooney: well just knowing that there is probably multiple network servers, maybe keep hitting it because *maybe* one of them is responding without that extension | |
| 13:34:21 | mnaser | https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/tasks/live_migrate.py#L282 just trying to eliminate all those things here | |
| 13:34:48 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Change a log level for overwriting allocation https://review.openstack.org/652994 | |
| 13:35:04 | sean-k-mooney | mnaser: i think i asked NewBruce to check that against each of the neutron api enpoints directly in the past bypassing the loadblancer | |
| 13:35:52 | sean-k-mooney | it would cause this issue espically if there was a network partion or some other factor cause different nodes to prefer diffeent api nodes | |
| 13:36:14 | sean-k-mooney | but if that was the case we would expect the same resules for osa to osa too right? | |
| 13:38:23 | mnaser | that is true as well | |
| 13:38:36 | mnaser | sean-k-mooney: but this is one of those issues where you're just desperately trying to do whatever works lol | |
| 13:38:59 | efried | dansmith: When you get a chance, would you please do the channel topic thing with https://etherpad.openstack.org/p/nova-spec-review-day ? | |
| 13:39:11 | openstackgerrit | Merged openstack/nova-specs master: Add host and hypervisor_hostname flag to create server https://review.openstack.org/645458 | |
| 13:39:39 | sean-k-mooney | ya i know from what we can tell it should be working if it wasnt for the evidence e.g. its not that would be my respocne if some one asked shoudl this work | |
| 13:43:28 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Add missing libvirt exception during device detach https://review.openstack.org/651639 | |
| 13:43:38 | openstackgerrit | sean mooney proposed openstack/os-traits master: add libvirt image metadata traits https://review.openstack.org/652996 | |
| 13:52:04 | bauzas | mriedem: sorry, finished earlier yesterday | |
| 13:52:19 | bauzas | mriedem: could you please ping me again which stable changes I could review ? | |
| 13:53:09 | mriedem | bauzas: at this point, probably queens https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/queens | |
| 13:53:28 | bauzas | ack | |
| 13:53:38 | mriedem | bauzas: and https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/pike+topic:bug/1669054 for pike | |
| 13:57:34 | dansmith | efried: as a placement and request groups expert, I hope you can look at this one and provide a short path to being able to make request groups expressive enough that we don't have to keep adding legacy stuff like this: https://review.openstack.org/#/c/650963/2 | |
| 13:59:01 | efried | dansmith: on my list. I had suggested https://review.openstack.org/#/c/650476/ which is generating lots of discussion both on the spec and on the ML http://lists.openstack.org/pipermail/openstack-discuss/2019-April/thread.html#4782 | |
| 13:59:37 | dansmith | efried: ack, glad to see that | |
| 13:59:42 | efried | unfortunately it's not an easy problem. | |
| 13:59:54 | dansmith | stephenfin: bauzas: I trust you will be all over that ^ spec to make sure it will solve your problems | |
| 14:00:02 | efried | which is why we've been talking about it for a couple years | |
| 14:02:52 | mriedem | melwitt: are consoles/console pools in the db model a xen only thing? | |
| 14:04:09 | mriedem | ah yes, "The nova-console service is deprecated as it is Xen specific" | |
| 14:05:17 | melwitt | cool. I don't recall the term "console pools" off the top of my head | |
| 14:07:28 | NewBruce | mnaser have run it in a loop and no changes | |
| 14:08:50 | mriedem | melwitt: ok, reason being https://review.openstack.org/#/c/570202/7/nova/db/sqlalchemy/api.py@1794 | |
| 14:09:49 | mnaser | hmm | |
| 14:09:50 | mriedem | looks like we likely leak consoles records when a server is deleted, | |
| 14:09:57 | mriedem | but that's super latent if so, and xen specific, | |
| 14:10:00 | mriedem | so i care little | |
| 14:10:20 | mriedem | Theo could open a bug for it but i wouldn't hold this up for it | |
| 14:10:23 | melwitt | haha, nice | |
| 14:10:25 | mnaser | is migrate_data stored in a db anywhere or is it thrown into the rpc request | |
| 14:10:34 | mriedem | mnaser: it's only rpc, not persisted | |
| 14:12:55 | melwitt | we deprecated the os-console service kind of recently https://review.openstack.org/610075 | |
| 14:12:57 | mriedem | stephenfin: the bottom of your cells v1 removal series needs to be fixed | |
| 14:13:07 | stephenfin | mriedem: I just saw :( | |
| 14:13:11 | melwitt | oh yeah, you already said that | |
| 14:13:13 | stephenfin | Looking at it now | |
| 14:13:15 | mriedem | melwitt: yeah i was looking at that - that deprecates the service, | |
| 14:13:16 | mriedem | but not the apis | |
| 14:13:23 | melwitt | ah | |
| 14:13:39 | mriedem | so if we were to remove the nova-console service the os-consoles api would be broken, | |
| 14:13:54 | mriedem | so we can't really deprecate the api with a microversion and expect it to still work on 2.1 w/o the service itself, | |
| 14:14:16 | mriedem | which would mean we either (1) leave the nova-console service forever or (2) obsolete the api and drop the service, like we're doing with nova-cells and nova-network | |
| 14:14:43 | mriedem | maybe ptg topic fodder - but it'd be nice to have xen people there to actually tell us if they need the thing anymore, BobBall seemed to suggest in the ML that we didn't | |
| 14:15:01 | mriedem | i don't know who works on or uses xenapi in nova anymore though | |