Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-16
13:11:27 NewBruce not identical (due to the time span taken to update), we have mostly 18.2.0, but some 18.1.0 and an 18.0.3
13:11:47 NewBruce we did try rolling an 18.2.0 back to 18.1.0 - but that didn’t seem to help
13:12:25 sean-k-mooney can you try migration between two nodes of the same version on the rdo side for each of the 3 versions you have deployed
13:12:28 NewBruce yeah, i agree its starting to feel something odd (random)… but as you say, feeling - not fact
13:12:35 NewBruce Yeah, will do now
13:12:50 sean-k-mooney perhaps we can narrow it donw to a version and then do a git biset for the changes
13:17:20 mnaser hmm
13:17:23 mnaser this is strange
13:17:38 mnaser NewBruce: so that cloud is 100% rocky at this point?
13:22:54 NewBruce mnaser yeah, 100% rocky
13:23:01 NewBruce … in all senses of the word :D
13:25:01 sean-k-mooney this is the delta in termes of ptaches https://github.com/openstack/nova/compare/18.0.3...18.1.0
13:25:57 mnaser NewBruce: curl -H "X-Auth-Token: `openstack token issue -c id -f value`" http://network-sjc1.vexxhost.us/v2.0/extensions | python -mjson.tool | grep binding-extended
13:26:03 mnaser change network-sjc1 with your neutron endpoint
13:26:21 mnaser can you run that many several times and see if there is every a time that it doesn't return binding-extended?
13:26:35 sean-k-mooney i wonder if it could be realated to https://github.com/openstack/nova/commit/4a12c9c298913f99570f2f8e93500db687e98dc9
13:28:31 sean-k-mooney hum although that is only on revert
13:29:11 NewBruce so an RDO 18.1.0 -> RDO 18.1.0 does the same thing; fails on duplicate ports
13:30:20 NewBruce "description": "Expose port bindings of a virtual port to external application",
13:30:20 NewBruce "alias": "binding-extended",
13:30:20 NewBruce "extension": {
13:30:20 NewBruce {
13:30:20 NewBruce mnaser : curl -s -H "X-Auth-Token: $OS_TOKEN" https://kna1.citycloud.com:9696/v2.0/extensions/binding-extended | python -m json.tool
13:30:21 NewBruce "name": "Port Bindings Extended",
13:30:21 NewBruce "links": [],
13:30:22 NewBruce }
13:30:22 NewBruce "updated": "2017-07-17T10:00:00-00:00"
13:30:23 NewBruce }
13:31:16 NewBruce (#SoccerDad duties - back online in about 30 min)
13:31:17 mnaser NewBruce: but did you hit it many different times, and it always gave a response?
13:31:28 mnaser im wondering if one backend might be acting weird
13:33:49 sean-k-mooney mnaser: so you resuggesting runnint it with "watch -d -n 1 ..." and seing if it changes
13:34:14 mnaser sean-k-mooney: well just knowing that there is probably multiple network servers, maybe keep hitting it because *maybe* one of them is responding without that extension
13:34:21 mnaser https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/tasks/live_migrate.py#L282 just trying to eliminate all those things here
13:34:48 openstackgerrit Matt Riedemann proposed openstack/nova stable/stein: Change a log level for overwriting allocation https://review.openstack.org/652994
13:35:04 sean-k-mooney mnaser: i think i asked NewBruce to check that against each of the neutron api enpoints directly in the past bypassing the loadblancer
13:35:52 sean-k-mooney it would cause this issue espically if there was a network partion or some other factor cause different nodes to prefer diffeent api nodes
13:36:14 sean-k-mooney but if that was the case we would expect the same resules for osa to osa too right?
13:38:23 mnaser that is true as well
13:38:36 mnaser sean-k-mooney: but this is one of those issues where you're just desperately trying to do whatever works lol
13:38:59 efried dansmith: When you get a chance, would you please do the channel topic thing with https://etherpad.openstack.org/p/nova-spec-review-day ?
13:39:11 openstackgerrit Merged openstack/nova-specs master: Add host and hypervisor_hostname flag to create server https://review.openstack.org/645458
13:39:39 sean-k-mooney ya i know from what we can tell it should be working if it wasnt for the evidence e.g. its not that would be my respocne if some one asked shoudl this work
13:43:28 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Add missing libvirt exception during device detach https://review.openstack.org/651639
13:43:38 openstackgerrit sean mooney proposed openstack/os-traits master: add libvirt image metadata traits https://review.openstack.org/652996
13:52:04 bauzas mriedem: sorry, finished earlier yesterday
13:52:19 bauzas mriedem: could you please ping me again which stable changes I could review ?
13:53:09 mriedem bauzas: at this point, probably queens https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/queens
13:53:28 bauzas ack
13:53:38 mriedem bauzas: and https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/pike+topic:bug/1669054 for pike
13:57:34 dansmith efried: as a placement and request groups expert, I hope you can look at this one and provide a short path to being able to make request groups expressive enough that we don't have to keep adding legacy stuff like this: https://review.openstack.org/#/c/650963/2
13:59:01 efried dansmith: on my list. I had suggested https://review.openstack.org/#/c/650476/ which is generating lots of discussion both on the spec and on the ML http://lists.openstack.org/pipermail/openstack-discuss/2019-April/thread.html#4782
13:59:37 dansmith efried: ack, glad to see that
13:59:42 efried unfortunately it's not an easy problem.
13:59:54 dansmith stephenfin: bauzas: I trust you will be all over that ^ spec to make sure it will solve your problems
14:00:02 efried which is why we've been talking about it for a couple years
14:02:52 mriedem melwitt: are consoles/console pools in the db model a xen only thing?
14:04:09 mriedem ah yes, "The nova-console service is deprecated as it is Xen specific"
14:05:17 melwitt cool. I don't recall the term "console pools" off the top of my head
14:07:28 NewBruce mnaser have run it in a loop and no changes
14:08:50 mriedem melwitt: ok, reason being https://review.openstack.org/#/c/570202/7/nova/db/sqlalchemy/api.py@1794
14:09:49 mnaser hmm
14:09:50 mriedem looks like we likely leak consoles records when a server is deleted,
14:09:57 mriedem but that's super latent if so, and xen specific,
14:10:00 mriedem so i care little
14:10:20 mriedem Theo could open a bug for it but i wouldn't hold this up for it
14:10:23 melwitt haha, nice
14:10:25 mnaser is migrate_data stored in a db anywhere or is it thrown into the rpc request
14:10:34 mriedem mnaser: it's only rpc, not persisted
14:12:55 melwitt we deprecated the os-console service kind of recently https://review.openstack.org/610075
14:12:57 mriedem stephenfin: the bottom of your cells v1 removal series needs to be fixed
14:13:07 stephenfin mriedem: I just saw :(
14:13:11 melwitt oh yeah, you already said that
14:13:13 stephenfin Looking at it now
14:13:15 mriedem melwitt: yeah i was looking at that - that deprecates the service,
14:13:16 mriedem but not the apis
14:13:23 melwitt ah
14:13:39 mriedem so if we were to remove the nova-console service the os-consoles api would be broken,
14:13:54 mriedem so we can't really deprecate the api with a microversion and expect it to still work on 2.1 w/o the service itself,
14:14:16 mriedem which would mean we either (1) leave the nova-console service forever or (2) obsolete the api and drop the service, like we're doing with nova-cells and nova-network
14:14:43 mriedem maybe ptg topic fodder - but it'd be nice to have xen people there to actually tell us if they need the thing anymore, BobBall seemed to suggest in the ML that we didn't
14:15:01 mriedem i don't know who works on or uses xenapi in nova anymore though
14:15:29 NewBruce hey mriedem seen the latest in our lovely migration issues ?!
14:15:34 openstackgerrit Stephen Finucane proposed openstack/nova-specs master: Standardize CPU resource tracking https://review.openstack.org/555081
14:15:47 mriedem NewBruce: not really - can't get the neutron api extension to go away?
14:16:11 mriedem melwitt: i'll just add an item in the ptg etherpad
14:16:44 NewBruce we upgraded the entire site to rocky, and now RDO - RDO fails in addition to RDO - OSA!
14:16:54 NewBruce (updated the launchpad)
14:18:54 mriedem :/
14:19:13 mriedem we definitely have multinode jobs running live migration in rocky that uses the new flow
14:19:56 melwitt yeah, I had thought BobBall gave the go ahead to remove it but agreed, I don't know of any people currently using xenapi. actually, I thought I've seen some patches proposed over there fairly recently. /me looks
14:21:13 melwitt oh, nvm, it was coreycb fixing an openssl handling thing https://review.openstack.org/635533
14:22:57 openstackgerrit Stephen Finucane proposed openstack/nova master: Stop handling cells v1 in '/os-hypervisors' API https://review.openstack.org/651292
14:22:57 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove '/os-cells' REST APIs https://review.openstack.org/651291
14:22:58 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove 'nova-manage cell' commands https://review.openstack.org/651294
14:22:58 openstackgerrit Stephen Finucane proposed openstack/nova master: Stop handling cells v1 in '/os-servers' API https://review.openstack.org/651293
14:22:59 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove old-style cell v1 instance listing https://review.openstack.org/651296
14:22:59 openstackgerrit Stephen Finucane proposed openstack/nova master: Stop handling cells v1 for console authentication https://review.openstack.org/651295
14:23:00 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove 'instance_fault_create_at_top' https://review.openstack.org/651298
14:23:00 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove 'bdm_(update_or_create|destroy)_at_top' https://review.openstack.org/651297

Earlier   Later