Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-01-21
16:06:39 dansmith hey sean-k-mooney, you like devstacking.. would you run this locally and compare before/after numbers? https://review.opendev.org/c/openstack/devstack/+/771505
16:07:57 sean-k-mooney " know this is going to be controversial, but hear me out." L)
16:08:03 sean-k-mooney * :)
16:08:08 sean-k-mooney sure i can try that
16:08:20 dansmith I actually maybe should remove that,
16:08:30 dansmith because so far it has been super stable for me, and easy to debug things
16:08:45 dansmith initial attempt interleaved all the parallel logging, which I knew would be hard to swallow for people
16:09:01 sean-k-mooney we still have some native calls to osc by the way that dont use osc wrap
16:09:05 dansmith but now it looks very clean, just potentially different orderings
16:09:14 sean-k-mooney if we fixed up those it would also help
16:09:19 dansmith sean-k-mooney: oscwrap is just for time accounting
16:09:23 dansmith it doesn't make it any faster
16:09:38 dansmith and it sets up an alias, so everything should be using it, AFAI
16:09:39 sean-k-mooney waith i tought it used osc in itneractive mode via a socket
16:09:44 dansmith no
16:09:50 sean-k-mooney did we not merge danpbs prototype of that
16:09:53 sean-k-mooney oh ok
16:10:32 dansmith I tried doing that myself this week, but getting the return code is going to be non-trivial and we need that in a *lot* of places in devstack
16:10:55 sean-k-mooney well ya ill pull that into one of my vms and let it run
16:11:01 dansmith we do lots of single-threaded things in devstack, waiting on osc is just the dumbest one :)
16:11:12 dansmith sean-k-mooney: ooh, thanks I'd definitely appreciate run in a vm
16:11:19 dansmith I'm on baremetal
16:12:00 sean-k-mooney well i ment run it in one of my openstack vms so that will be similar to the upstream ci runs
16:12:08 dansmith that's cool
16:12:43 dansmith performance in the actual nodepool workers varies so much from minute to minute it's hard to prove with linkable numbers that it's better
16:12:48 dansmith so more anecdotal evidence would be good
16:14:03 dansmith I'm also on spinning rust currently, and I feel like I could be getting better parallel performance by overlapping the db syncs if I wasn't
16:14:23 sean-k-mooney i rebooted my cloud at the weekend so need to start up my third paryt ci again but i can pproably have it do a few full tempest runs too but ill take a look at that spereatly
16:14:46 sean-k-mooney since my thridparty ci just runs on one host its pretty consitnet
16:14:55 dansmith so far it hasn't failed a tempest run, so I'm pretty confident that it's still building a consistent devstack
16:16:33 artom Do we... do we have a config option to log in JSON somehow?
16:17:40 sean-k-mooney oh so " async_inner $name $* &" is how your doing the async run
16:17:43 artom I'm looking at some downstream logs, and they're in format {"message": <blah>, "asctime": <blah>, "name": <blah>, "msg": <etc> }
16:17:48 artom First time I see something like that
16:18:53 sean-k-mooney artom: yes you can do this
16:18:58 dansmith artom: I think you could use one of the libs for it and configure it thusly
16:19:00 sean-k-mooney via standard python logger config
16:19:23 dansmith so probably doable without changes to the openstack code
16:19:24 dansmith yeah ^
16:19:38 artom Makes all of our usual debugging tools (read: os-log-merger) completely useless
16:19:47 sean-k-mooney one of our aisa based customer does it
16:20:18 sean-k-mooney lave i think will work with the time stamps but really it depend on how they did it
16:20:41 artom sean-k-mooney, and it's a finicky external events evacuation issue as well
16:21:08 artom Where you need to correlate timestampe and requests across Neutron and Nova :(
16:23:34 sean-k-mooney oh ya i started looking at that but did not open the sos report
16:24:06 sean-k-mooney they should be aware that hw:numa_mempolicy or whatever it is is not a thing
16:24:25 sean-k-mooney oh wait imight be mixing too issues
16:25:00 sean-k-mooney your looking at https://bugzilla.redhat.com/show_bug.cgi?id=1917675
16:25:01 openstack bugzilla.redhat.com bug 1917675 in openstack-nova "Nova evacuate fails due to timeout waiting for a network-vif-plugged event for instance" [High,New] - Assigned to nova-maint
16:25:34 sean-k-mooney the one with the fake extra spec is a different one i was look at breifly too
16:25:42 artom sean-k-mooney, yeah, that one. We can move downstream to discuss the specifics, tho
16:26:06 sean-k-mooney hehe ya i know just confirm there was not a duplicate
16:26:39 sean-k-mooney i just glanced at it so have not looked at the logs or anything
16:26:58 artom Their format makes them unworkable :(
16:47:55 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Load and cache volume drivers on-demand https://review.opendev.org/c/openstack/nova/+/741545
16:50:45 openstackgerrit Stephen Finucane proposed openstack/placement master: tests: Suppress policy deprecation warnings https://review.opendev.org/c/openstack/placement/+/771852
16:53:56 stephenfin lbragstad: When you've a chance, I've a question left on https://review.opendev.org/c/openstack/placement/+/760235
16:56:54 lyarwood now I have to fix all of these broken tests
16:56:57 lyarwood gah
16:57:20 stephenfin tests broke?
16:57:22 stephenfin huh?
16:57:47 lyarwood passing in strings instead of dicts etc
16:58:18 lyarwood at least I think that's what this error means
16:58:20 stephenfin Oh, those won't get picked up by mypy unless you run it on them. Don't run mypy on tests :)
16:58:25 stephenfin Hmm, I doubt it
16:58:30 stephenfin Paste?
16:58:31 lyarwood TypeError: Parameters to generic types must be types. Got 0.
16:58:53 lyarwood http://paste.openstack.org/show/801843/
17:00:38 lyarwood self.volume_drivers = ty.Dict[str, volume.LibvirtBaseVolumeDriver] is what I have for that dict
17:00:52 stephenfin yup, I just pulled down to see what's going on
17:01:20 lyarwood read*
17:01:30 stephenfin lyarwood: oh /o\
17:01:36 stephenfin self.volume_drivers = ty.Dict[str, volume.LibvirtBaseVolumeDriver]
17:01:40 stephenfin should be
17:01:44 stephenfin self.volume_drivers: ty.Dict[str, volume.LibvirtBaseVolumeDriver] = {}
17:01:52 stephenfin var: type = value
17:01:58 lyarwood aaaaaah
17:02:22 stephenfin sorry, I missed that
17:05:21 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Load and cache volume drivers on-demand https://review.opendev.org/c/openstack/nova/+/741545
17:05:22 lyarwood stephenfin: np thanks again ^
17:08:53 openstackgerrit Merged openstack/nova master: Add regression test for bug #1899835 https://review.opendev.org/c/openstack/nova/+/758239
17:08:55 openstack bug 1899835 in OpenStack Compute (nova) "n-cpu attempts to disconnect volumes after early pre_live_migration failures on the destination during a live migration" [Medium,In progress] https://launchpad.net/bugs/1899835 - Assigned to Lee Yarwood (lyarwood)
17:09:50 lbragstad stephenfin done
17:10:08 openstackgerrit Lance Bragstad proposed openstack/placement master: Implement secure RBAC for aggregates https://review.opendev.org/c/openstack/placement/+/760235
17:10:09 openstackgerrit Lance Bragstad proposed openstack/placement master: Implement secure RBAC for allocations https://review.opendev.org/c/openstack/placement/+/760236
17:10:10 openstackgerrit Lance Bragstad proposed openstack/placement master: Implement secure RBAC for allocation candidates https://review.opendev.org/c/openstack/placement/+/760237
17:10:11 openstackgerrit Lance Bragstad proposed openstack/placement master: Implement secure RBAC for inventories https://review.opendev.org/c/openstack/placement/+/760238
17:10:12 openstackgerrit Lance Bragstad proposed openstack/placement master: Implement secure RBAC for resource classes https://review.opendev.org/c/openstack/placement/+/760239
17:10:13 openstackgerrit Lance Bragstad proposed openstack/placement master: Implement secure RBAC for traits https://review.opendev.org/c/openstack/placement/+/760241
17:10:14 openstackgerrit Lance Bragstad proposed openstack/placement master: Implement secure RBAC for usage https://review.opendev.org/c/openstack/placement/+/760242
18:00:31 manoj_kumar_kata hi network experts, we have 4-node openstack cluster and a recent lab poweroff resulted in losing network configuration. we worked on it and now the controller node could create vms but VMs are not getting the ip assigned .
18:00:41 manoj_kumar_kata appreciate your help here
19:08:04 openstackgerrit Merged openstack/nova master: zuul: Skip test_attach_cloned_encrypted_volume in nova-ceph-multistore https://review.opendev.org/c/openstack/nova/+/771777
19:34:05 openstackgerrit Lee Yarwood proposed openstack/nova stable/queens: libvirt: Do not reference VIR_ERR_DEVICE_MISSING when libvirt is < v4.1.0 https://review.opendev.org/c/openstack/nova/+/747362
19:35:10 lyarwood melwitt / elod ; https://review.opendev.org/c/openstack/nova/+/747362/ would you mind looking at this stack of stable/queens changes again, should be good to go now.
19:36:06 lyarwood I just tried to rebase that in the new gerrit UI and it has ended up with a merge conflict, wat?!
19:38:57 lyarwood ha, it just didn't report an actual conflict during the rebase attempt, fun
19:40:12 openstackgerrit Lee Yarwood proposed openstack/nova stable/queens: sync_guest_time: use the proper errno https://review.opendev.org/c/openstack/nova/+/742423
19:40:13 openstackgerrit Lee Yarwood proposed openstack/nova stable/queens: libvirt: Remove reference to transient domain when detaching devices https://review.opendev.org/c/openstack/nova/+/608186
19:40:14 openstackgerrit Lee Yarwood proposed openstack/nova stable/queens: libvirt: Handle VIR_ERR_DEVICE_MISSING when detaching devices https://review.opendev.org/c/openstack/nova/+/742424

Earlier   Later