Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-01-21
15:21:06 dansmith lyarwood: the whole arrangement in devstack is edgy yeah
15:21:49 lyarwood dansmith: cool, so looking at the actual test that borked here this might be something we can fix on their side
15:22:16 lyarwood dansmith: for whatever reason the test is creating a unique image for the instance to use
15:22:41 lyarwood dansmith: when it really isn't required as the test is about cloning an encrypted volume attached to the instance
15:23:06 lyarwood I'll submit a few changes to the cinder plugin before adding our job to their gate somehow
15:23:06 dansmith yeah, and it creates it in the wrong glance store, which is the problem
15:23:18 lyarwood we might be able to drop the blacklist around encrypted volumes after all
15:23:26 lyarwood skiplist*
15:23:36 dansmith brownlist
15:23:40 dansmith like, turds
15:25:31 lyarwood ^_^
15:26:02 spatel noonedeadpunk: In your designate example where is pool_id? http://paste.openstack.org/show/801787/
15:26:55 noonedeadpunk o_O
15:27:01 spatel i had issue that issue and when i put pool id then it works
15:27:08 spatel what is the workaround for that?
15:27:47 spatel does OSA generate random pool id ? (because default isn't valid)
15:28:43 spatel sorry folks wrong channel
15:34:57 sean-k-mooney i had issue with the pool id in kolla ansible too
15:35:37 sean-k-mooney for me i had to both mess with the config and the db to eventulaly get it to work
15:36:00 sean-k-mooney but i remmeber id did not jsut work for me when it was deployed
15:36:38 sean-k-mooney i added
15:36:40 sean-k-mooney [service:central]
15:36:43 sean-k-mooney default_pool_id = 6a67bd3a8e654ddeb3675b8534899f16
15:36:45 sean-k-mooney workers = 5
15:36:54 sean-k-mooney and then i think i needed to edit the db to make it match
15:37:03 sean-k-mooney it was strange
15:46:04 openstackgerrit Lee Yarwood proposed openstack/nova master: zuul: Allow encryption tests in nova-ceph-multistore again https://review.opendev.org/c/openstack/nova/+/771837
15:46:09 lyarwood dansmith: ^ hopefully this work
15:46:33 lyarwood that links through to a fix and another change adding the job to gate of the tempest plugin
15:47:18 lyarwood ah I think I also need to fix some core tempest tests
15:47:21 lyarwood my bad
15:47:41 dansmith yeah, there are core tests too, and that's why last night I was like "wtf, how is this just now breaking when nothing has changed here"
15:48:53 lyarwood ack kk, I'll let this fail and then fix them up, it's likely the same bogus image creation copy pasted between tempest and the plugin tbh
15:49:21 dansmith wouldn't be surprised, yeah
15:49:28 dansmith which is further support for "why is this separate?"
15:49:44 lyarwood yup
15:50:48 gibi nova weekly meeting starts in 10 minutes in #openstack-meeting-3
15:50:55 bauzas gibi: given the curfew we have at 6pm, I'll stay in the meeting until 5.30pm our time
15:54:52 gibi bauzas: O
15:54:53 gibi K
16:06:39 dansmith hey sean-k-mooney, you like devstacking.. would you run this locally and compare before/after numbers? https://review.opendev.org/c/openstack/devstack/+/771505
16:07:57 sean-k-mooney " know this is going to be controversial, but hear me out." L)
16:08:03 sean-k-mooney * :)
16:08:08 sean-k-mooney sure i can try that
16:08:20 dansmith I actually maybe should remove that,
16:08:30 dansmith because so far it has been super stable for me, and easy to debug things
16:08:45 dansmith initial attempt interleaved all the parallel logging, which I knew would be hard to swallow for people
16:09:01 sean-k-mooney we still have some native calls to osc by the way that dont use osc wrap
16:09:05 dansmith but now it looks very clean, just potentially different orderings
16:09:14 sean-k-mooney if we fixed up those it would also help
16:09:19 dansmith sean-k-mooney: oscwrap is just for time accounting
16:09:23 dansmith it doesn't make it any faster
16:09:38 dansmith and it sets up an alias, so everything should be using it, AFAI
16:09:39 sean-k-mooney waith i tought it used osc in itneractive mode via a socket
16:09:44 dansmith no
16:09:50 sean-k-mooney did we not merge danpbs prototype of that
16:09:53 sean-k-mooney oh ok
16:10:32 dansmith I tried doing that myself this week, but getting the return code is going to be non-trivial and we need that in a *lot* of places in devstack
16:10:55 sean-k-mooney well ya ill pull that into one of my vms and let it run
16:11:01 dansmith we do lots of single-threaded things in devstack, waiting on osc is just the dumbest one :)
16:11:12 dansmith sean-k-mooney: ooh, thanks I'd definitely appreciate run in a vm
16:11:19 dansmith I'm on baremetal
16:12:00 sean-k-mooney well i ment run it in one of my openstack vms so that will be similar to the upstream ci runs
16:12:08 dansmith that's cool
16:12:43 dansmith performance in the actual nodepool workers varies so much from minute to minute it's hard to prove with linkable numbers that it's better
16:12:48 dansmith so more anecdotal evidence would be good
16:14:03 dansmith I'm also on spinning rust currently, and I feel like I could be getting better parallel performance by overlapping the db syncs if I wasn't
16:14:23 sean-k-mooney i rebooted my cloud at the weekend so need to start up my third paryt ci again but i can pproably have it do a few full tempest runs too but ill take a look at that spereatly
16:14:46 sean-k-mooney since my thridparty ci just runs on one host its pretty consitnet
16:14:55 dansmith so far it hasn't failed a tempest run, so I'm pretty confident that it's still building a consistent devstack
16:16:33 artom Do we... do we have a config option to log in JSON somehow?
16:17:40 sean-k-mooney oh so " async_inner $name $* &" is how your doing the async run
16:17:43 artom I'm looking at some downstream logs, and they're in format {"message": <blah>, "asctime": <blah>, "name": <blah>, "msg": <etc> }
16:17:48 artom First time I see something like that
16:18:53 sean-k-mooney artom: yes you can do this
16:18:58 dansmith artom: I think you could use one of the libs for it and configure it thusly
16:19:00 sean-k-mooney via standard python logger config
16:19:23 dansmith so probably doable without changes to the openstack code
16:19:24 dansmith yeah ^
16:19:38 artom Makes all of our usual debugging tools (read: os-log-merger) completely useless
16:19:47 sean-k-mooney one of our aisa based customer does it
16:20:18 sean-k-mooney lave i think will work with the time stamps but really it depend on how they did it
16:20:41 artom sean-k-mooney, and it's a finicky external events evacuation issue as well
16:21:08 artom Where you need to correlate timestampe and requests across Neutron and Nova :(
16:23:34 sean-k-mooney oh ya i started looking at that but did not open the sos report
16:24:06 sean-k-mooney they should be aware that hw:numa_mempolicy or whatever it is is not a thing
16:24:25 sean-k-mooney oh wait imight be mixing too issues
16:25:00 sean-k-mooney your looking at https://bugzilla.redhat.com/show_bug.cgi?id=1917675
16:25:01 openstack bugzilla.redhat.com bug 1917675 in openstack-nova "Nova evacuate fails due to timeout waiting for a network-vif-plugged event for instance" [High,New] - Assigned to nova-maint
16:25:34 sean-k-mooney the one with the fake extra spec is a different one i was look at breifly too
16:25:42 artom sean-k-mooney, yeah, that one. We can move downstream to discuss the specifics, tho
16:26:06 sean-k-mooney hehe ya i know just confirm there was not a duplicate
16:26:39 sean-k-mooney i just glanced at it so have not looked at the logs or anything
16:26:58 artom Their format makes them unworkable :(
16:47:55 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Load and cache volume drivers on-demand https://review.opendev.org/c/openstack/nova/+/741545
16:50:45 openstackgerrit Stephen Finucane proposed openstack/placement master: tests: Suppress policy deprecation warnings https://review.opendev.org/c/openstack/placement/+/771852
16:53:56 stephenfin lbragstad: When you've a chance, I've a question left on https://review.opendev.org/c/openstack/placement/+/760235
16:56:54 lyarwood now I have to fix all of these broken tests
16:56:57 lyarwood gah
16:57:20 stephenfin tests broke?

Earlier   Later