Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-02-21
15:49:03 brinzhang efried, gibi: thanks
15:49:08 gibi brinzhang: thanks
16:02:55 openstackgerrit Balazs Gibizer proposed openstack/nova stable/train: Clean up allocation if unshelve fails due to neutron https://review.opendev.org/709167
16:02:55 openstack bug 1862633 in OpenStack Compute (nova) "unshelve leak allocation if update port fails" [Medium,Fix released] https://launchpad.net/bugs/1862633 - Assigned to Balazs Gibizer (balazs-gibizer)
16:02:55 openstackgerrit Balazs Gibizer proposed openstack/nova stable/train: Reproduce bug 1862633 https://review.opendev.org/709166
16:16:27 stephenfin gibi: done
16:16:34 gibi stephenfin: thank you
16:28:10 sean-k-mooney gibi: there are potentially other issues with unshelve. we have an open dowsntream bug report that sugges we do not correclty handel neutron sriov deivce on unshelve
16:28:42 sean-k-mooney gibi: basicaly i thinke we do the claim and port update after we regenerate the xml on queens
16:28:42 gibi sean-k-mooney: could be. this bug was come up when I added qos support for unshelve
16:29:14 sean-k-mooney so i think we have an odering issue in that code in generall but not sure if its still a thing on master
16:31:57 gibi sean-k-mooney: there is a FIXME in the code https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6497
16:32:04 gibi but that is about error handling + cleanup
16:32:22 gibi the port update happens before we call driver.spawn
16:32:36 gibi https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6474-L6480
16:33:47 gibi the problem might be that we update the port before the instance_claim
16:35:56 gibi but I lack the brain power on this Friday evening to dig this further
17:07:33 openstackgerrit Mark Goddard proposed openstack/nova master: Invalidate provider tree when compute node disappears https://review.opendev.org/695188
17:07:33 openstackgerrit Mark Goddard proposed openstack/nova master: Clear rebalanced compute nodes from resource tracker https://review.opendev.org/695187
17:07:33 openstack bug 1853009 in OpenStack Compute (nova) "Ironic node rebalance race can lead to missing compute nodes in DB" [Undecided,In progress] https://launchpad.net/bugs/1853009 - Assigned to Mark Goddard (mgoddard)
17:07:33 openstackgerrit Mark Goddard proposed openstack/nova master: Add functional regression test for bug 1853009 https://review.opendev.org/695012
17:07:34 openstackgerrit Mark Goddard proposed openstack/nova master: Fix inactive session error in compute node creation https://review.opendev.org/695189
17:07:34 openstackgerrit Mark Goddard proposed openstack/nova master: Prevent deletion of a compute node belonging to another host https://review.opendev.org/694802
17:09:26 prometheanfire how goes the alembic-1.4.0 testing?
17:12:11 sean-k-mooney gibi: ya i think that is the issue. we updated the port before the instnace claim, and therefor when we generate the xml by retirivng the pci_slot form the neutron port bidning details it uses the old adress
17:13:42 sean-k-mooney gibi: anyway i just wanted you to be aware of that. im not expectiing your change to fix it
17:31:34 Sundar dansmith: In response to your comments/questions on rescheduling, is https://review.opendev.org/#/c/631244/62/nova/conductor/manager.py@839 the only piece left? Are there additional places to create/bind ARQs?
17:32:00 dansmith Sundar: for reschedule, that should be it
17:42:00 efried Really, I need dansmith to look and see if the results mean we need a migration or not.
17:42:00 efried prometheanfire: I didn't get back to it yesterday, but it's here: https://review.opendev.org/708706
17:53:13 openstackgerrit Merged openstack/nova stable/stein: Enhance service restart in functional env https://review.opendev.org/703103
17:53:19 openstackgerrit Merged openstack/nova master: trivial: Fetch 'Service' objects once when building AZs https://review.opendev.org/705653
17:53:27 openstackgerrit Merged openstack/nova master: trivial: Bump minimum version of websockify https://review.opendev.org/705654
17:53:31 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Report storage bus traits https://review.opendev.org/666914
17:53:32 openstackgerrit Stephen Finucane proposed openstack/nova master: Add transform_image_metadata request filter https://review.opendev.org/665775
17:53:32 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Use domain capabilities to get supported device models https://review.opendev.org/666915
17:59:14 prometheanfire efried_afk: thanks
17:59:36 prometheanfire efried_afk: that's my review, I was asking if nova had a review to fix :D
18:16:19 dansmith prometheanfire: he meant this: https://review.opendev.org/#/c/708707/1
18:16:30 dansmith and, it failed grenade on the base, so no useful data
18:27:06 prometheanfire ah, fun, thanks
19:04:19 openstackgerrit Dan Smith proposed openstack/nova master: WIP: Name Enums https://review.opendev.org/708707
19:32:16 openstackgerrit Merged openstack/nova stable/train: Print help if nova-manage subcommand is not specified https://review.opendev.org/702637
19:37:41 efried prometheanfire: sorry, yeah, what dansmith said. Looks like the base patch is in merge conflict atm, Ima rebase it so we can recheck the sniffer.
19:39:45 efried done and done
19:40:05 efried dansmith: was the unit test failure anything significant?
19:40:19 dansmith efried: it's complaining that the model doesn't match the schema,
19:40:37 dansmith potentially because sqlite reports generated enum names, or doesn't support setting them or something
19:40:44 efried right, I wasn't sure whether that indicated we need a migration or not
19:40:46 dansmith but, cross that bridge later if we decide we need to do this
19:40:50 efried okay
19:41:00 efried like, maybe figure out what name sqlite autogenerates and use that?
19:42:26 dansmith oh, I just re-looked.. those are the postgres tests complaining
19:42:27 dansmith not the sqlite ones
19:42:32 dansmith I keep not seeing "postgres"
19:42:55 dansmith so yeah, have to make them match or teach the test not to care or something
20:25:21 openstackgerrit Eric Fried proposed openstack/nova master: Enforce os-traits/SUPPORTED_STORAGE_BUSES sync https://review.opendev.org/709233
21:01:48 Sundar dansmith: For the rescheduling part, https://review.opendev.org/#/c/631244/62/nova/conductor/manager.py@839, the call to _create_and_bind_arqs() must return the ARQs, so that they may be passed to build_and_run_instance(). The change to return the ARQs happens in a later patch -- the obe that bumps up RPCAPI:
21:01:49 Sundar https://review.opendev.org/#/c/631244/62/nova/conductor/manager.py@839
21:02:10 Sundar Sorry, https://review.opendev.org/#/c/704227/14/nova/conductor/manager.py@1661
21:03:01 Sundar Shall we do the create/bind for the rescheduling as a followup to 'Bump PCPI' patch?
21:03:57 dansmith Sundar: in the create patch you don't pass the arqs to the compute node until later in the rpc bump patch
21:04:47 dansmith so just make the reschedule path do the same thing now, which is create/bind, then don't pass them to build_and_run_instance() just as the normal path.. then in the rpc patch you can pass the ARQs to both cases of that call
21:05:10 dansmith that seems obvious to me, so maybe I'm missing something else that you're worried about?
21:05:42 Sundar dansmith: No, that just means we touch two patches instead of one. That is fine by me, however.
21:06:03 dansmith it also means we don't land a patch that is broken for reschedules :)
21:06:30 Sundar If we do it all in the 'bump rpcpai' patch, or later, we wouldn't break anything
21:06:38 lucidguy Anyone have experience starting up instances with over 1TB of memory?
21:06:45 Sundar But this is fine too.
21:06:49 dansmith Sundar: yes we would because your patches will land in order
21:07:03 sean-k-mooney lucidguy: it is not a commonly tested usecase no
21:07:36 dansmith Sundar: your rpc patch needs to handle the case where it isn't passed that new argument anyway, so you could punt on even updating the rpc patch until later if you want.. the create patch is more important
21:07:41 lucidguy There must be a way. RHEL OpenStack seems to support like 4TB of memory
21:07:55 sean-k-mooney lucidguy: not for once instance
21:08:27 lucidguy I'm pretty sure its per instance
21:08:31 sean-k-mooney that is likely the maxium limit that kvm/qemu report they can support not the practical limit
21:08:54 sean-k-mooney lucidguy: how much memory do you have on the host
21:09:04 sean-k-mooney and are you useing hugepage for the guest
21:09:16 Sundar dansmith: If somebody pulls in the patch series till 'create/bind', but not the later ones, we'll have a problem anyway -- because that would create/use resources in Cyborg but not free them, or handle reboots, etc.
21:09:31 Sundar Not sure if I am misunderstanding your point
21:10:24 sean-k-mooney lucidguy: https://access.redhat.com/articles/1436373#nova the limist are taken form base rhel but they are not tested as part of the product
21:10:38 dansmith Sundar: yes, which is why we're holding a -2 on the bottom patch so we can *try* to land them all together.. however the rule in nova is that all patches should be deployable whenever possible, so we're trying to limit the number of places where we land part of your set and it's broken
21:10:59 Sundar dansmith: Ok, not to belabor the point, I'll just do the 2-patch thing.
21:11:21 dansmith Sundar: I'd really like to not have your set as broken as it is in the middle, but also trying to be practical.. however, not adding more places where we don't handle something as basic as reschedules until later is not going the right direction
21:11:32 sean-k-mooney lucidguy: its likely that you can acigve the kvm limits set in https://access.redhat.com/articles/rhel-kvm-limits but i suspect that to do that you both need tohave signifcantly more meory on the host and use things like hugepages
21:12:05 lucidguy sean-k-mooney: It lists a limit of 4TB
21:12:13 dansmith Sundar: I understand it's somewhat academic, which is why I'm not insisting on the existing isses
21:12:16 dansmith *issues
21:12:22 lucidguy I can't seem to start up an instance with 1.2TB
21:12:44 sean-k-mooney lucidguy: how much memory is on the host
21:12:45 lucidguy 1TB yes, 1.2TB, same image, kernel panic.
21:12:53 lucidguy 1.5TB
21:12:58 sean-k-mooney ok
21:13:26 sean-k-mooney if i rememebr correctly the kernel panic you posted before looked liek a kernel kvm bug
21:13:34 lucidguy It's driving me crazy, the amount of time I spent troubleshooting this.
21:14:07 lucidguy I've tride numerous kernels, upgraded the entire host. Still no luck.
21:14:24 dansmith Sundar: any chance you noticed my patch to fix creation of orphaned device providers?
21:14:29 lucidguy It's as if the problem resides with the Image itself.
21:14:49 dansmith Sundar: I'd guess it probably needs your blessing
21:15:09 sean-k-mooney lucidguy: its posible. unlikely but possible

Earlier   Later