Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-02-10
15:24:58 stephenfin lyarwood: When you're back, would appreciate you weighing in on https://review.opendev.org/c/openstack/nova/+/751368 (stable patch)
15:26:01 gmann stephenfin: ah i see. thanks.
15:26:09 stephenfin it was so close to be perfect :D
15:26:13 stephenfin *being
15:27:34 gmann I thought we have broken something :)
15:32:44 openstackgerrit Stephen Finucane proposed openstack/nova master: cmd: Remove 'nova-manage db ironic_flavor_migration' https://review.opendev.org/c/openstack/nova/+/773776
15:33:10 stephenfin gibi: You were +2 on that previously but I'd to address a doc failure. Care to revisit? Only change is to 'doc/source/cli/nova-manage.rst' ^
15:33:20 gibi sure
15:33:24 stephenfin thanks
15:34:09 stephenfin gibi: We're _almost_ done with merging the main DB migration squash series (about time). We can probably look at removing that tweak to tox.ini once it's done, I guess?
15:35:00 gibi stephenfin: yepp, I can propose the rewert on top of the db series
15:35:10 stephenfin cool
15:44:13 openstackgerrit Stephen Finucane proposed openstack/nova master: apidb: Compact Liberty database migrations https://review.opendev.org/c/openstack/nova/+/759399
15:44:13 openstackgerrit Stephen Finucane proposed openstack/nova master: apidb: Add manage.py script https://review.opendev.org/c/openstack/nova/+/771419
15:44:14 openstackgerrit Stephen Finucane proposed openstack/nova master: apidb: Compact Newton database migrations https://review.opendev.org/c/openstack/nova/+/759401
15:44:14 openstackgerrit Stephen Finucane proposed openstack/nova master: apidb: Compact Mitaka database migrations https://review.opendev.org/c/openstack/nova/+/759400
15:44:17 openstackgerrit Stephen Finucane proposed openstack/nova master: apidb: Compact Ocata database migrations https://review.opendev.org/c/openstack/nova/+/759402
15:44:24 openstackgerrit Stephen Finucane proposed openstack/nova master: apidb: Compact Pike database migrations https://review.opendev.org/c/openstack/nova/+/759403
15:44:26 openstackgerrit Stephen Finucane proposed openstack/nova master: apidb: Compact Queens database migrations https://review.opendev.org/c/openstack/nova/+/759404
15:44:29 openstackgerrit Stephen Finucane proposed openstack/nova master: apidb: Compact Rocky database migrations https://review.opendev.org/c/openstack/nova/+/759405
15:44:33 openstackgerrit Stephen Finucane proposed openstack/nova master: apidb: Compact Stein database migrations https://review.opendev.org/c/openstack/nova/+/759406
15:44:35 openstackgerrit Stephen Finucane proposed openstack/nova master: apidb: Compact Train database migrations https://review.opendev.org/c/openstack/nova/+/771420
15:44:55 stephenfin lyarwood, bauzas, melwitt: Can one of you take a look at https://review.opendev.org/c/openstack/nova/+/773727/ and https://review.opendev.org/c/openstack/nova/+/773728/ to fix the PPC64 gate, please
15:47:11 bauzas stephenfin: kinda try
15:48:04 openstackgerrit Stephen Finucane proposed openstack/os-vif master: Resolve dependency issues https://review.opendev.org/c/openstack/os-vif/+/774694
15:48:09 openstackgerrit Balazs Gibizer proposed openstack/nova master: Revert "Run the db migration tests in the same test worker" https://review.opendev.org/c/openstack/nova/+/774889
15:49:06 gibi stephenfin: here you are ^^
15:49:11 stephenfin +2
15:50:08 gibi I would keep the elastic signature still in place to see if the compaction really give us the speed bump on the gate I measured locally
15:53:48 bauzas stephenfin: https://review.opendev.org/c/openstack/nova/+/773728/ isn't needed for fixing the ppc64 gate job, righrt?
15:54:13 stephenfin bauzas: Yes, that's the crucial bit in fact
15:54:24 bauzas ok, then I misunderstood it
15:54:27 bauzas will review it
15:54:33 stephenfin http://lists.openstack.org/pipermail/openstack-discuss/2021-January/020088.html
15:55:20 stephenfin tl;dr: I added a patch to disable the USB controller if there were no USB devices in the XML, however, libvirt adds a USB keyboard and mouse automatically for guests on PPC64 hosts
15:55:40 stephenfin so it now fails
15:55:57 bauzas aha I see
15:56:16 bauzas it wants to add both the kb and the mouse but there are no longer usb controllers
15:56:21 stephenfin yup
15:56:26 bauzas gotcha, will look
15:56:41 stephenfin so we skip that logic on PPC64 now since we know we'll always need a controller on that platform
15:59:12 lyarwood stephenfin: sorry wasn't watching irc, I'll take a look at that and the other stuff from earlier after our next call
16:32:24 dansmith gibi: that's nifty, I didn't know about that grouping thing
16:33:47 dansmith gibi: did you happen to notice if the IO-related timeouts were specific to one provider?
16:34:20 dansmith tripleo was having job timeouts on one of their jobs that did a lot of IO and vexxhost switched our flavor to double the IOPs
16:35:26 gibi dansmith: the failures was spread amongst providers. at least I haven't noticed a pattern
16:35:32 dansmith okay
16:35:59 gibi I assumed it was IO limited as I was able to reproduce the extreme slowness locally if I IO deprived the machine
16:36:14 dansmith my parallel devstack ran all of those migrations in parallel for a very nice speedup on local hardware,
16:36:32 dansmith so maybe with 75% faster due to compaction we'll get a nice bump in the io-limited workers too
16:36:40 dansmith gibi: yeah probably
16:36:46 gibi yeah I saw you parallel devstack patch, that also looks like a nice bump. I will try it at some point
16:37:00 dansmith yeah, hopefully we can merge it soon
16:37:09 dansmith it's super nice to be able to stack locally in five minutes :)
16:39:39 dansmith devstack also runs db sync for keystone three times for some reason I have yet to figure out,
16:39:40 dansmith and while they don't have hundreds of migrations, it's non-trivial
16:52:15 openstackgerrit Elod Illes proposed openstack/nova stable/ussuri: Add upgrade check about old computes https://review.opendev.org/c/openstack/nova/+/770765
16:52:44 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Record the machine_type of instances in system_metadata https://review.opendev.org/c/openstack/nova/+/767533
16:52:45 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-manage: Add machine_type get command https://review.opendev.org/c/openstack/nova/+/769548
16:52:46 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-manage: Add machine_type update command https://review.opendev.org/c/openstack/nova/+/774896
16:52:46 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-status: Add hw_machine_type check for libvirt instances https://review.opendev.org/c/openstack/nova/+/770643
16:52:47 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP nova-manage: Add machine_type list_unset command https://review.opendev.org/c/openstack/nova/+/774897
16:52:48 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP libvirt: Add a config update workflow func test for [libvirt]hw_machine_type https://review.opendev.org/c/openstack/nova/+/774898
16:52:49 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP docs: Add admin docs for configuring and updating machine types https://review.opendev.org/c/openstack/nova/+/774899
17:15:53 openstackgerrit Stephen Finucane proposed openstack/nova master: tests: Move 'nova.virt.libvirt.utils.get_arch' stub to fixture https://review.opendev.org/c/openstack/nova/+/774904
17:15:53 openstackgerrit Stephen Finucane proposed openstack/nova master: tests: Posion nova.objects.fields.Architecture.from_host https://review.opendev.org/c/openstack/nova/+/769920
17:15:54 openstackgerrit Stephen Finucane proposed openstack/nova master: trivial: Remove unnecessary mock https://review.opendev.org/c/openstack/nova/+/774905
17:16:37 stephenfin lyarwood: Fixed that, I think ^
17:19:50 lyarwood awesome, I'll take a look in the morning once the tests have passed
17:20:52 stephenfin ack
17:33:23 gibi dansmith: I saw ~17% speedup locally in devstack stacking with parellelism enabled. http://paste.openstack.org/show/802536/
17:33:50 gibi dansmith: two interesting observations; 1) osc time went up when run in parallel 2) unacounted time become negative :)
17:34:16 dansmith gibi: is that with the db migrations squashed?
17:34:28 dansmith merged I mean
17:34:46 gibi dansmith: squashed up until Queens
17:35:03 gibi dansmith: I will rerun now with all the db squashes
17:35:42 dansmith okay, I'm not really sure all of the component timing actually works with async because they're run in subshells and the timings don't really get reflected to the parent, AFAIK
17:36:07 gibi ahh, I see
17:37:19 dansmith that's 25% increase from top to bottom,
17:37:53 dansmith I'm not sure why you'd see a difference with async patches applied but disabled, should be identical, unless you're in a VM with neighbor noise affecting the numbers
17:39:21 gibi dansmith: could be noise and could be that apt did updated things at the first run but not at the subsequent runs
17:39:28 dansmith ah okay yeah
17:39:56 dansmith compacting the migrations will definitely reduce the gain from the async tasks, since they're doing less, thus less to parallelize
17:40:18 dansmith but still better overall with both of course
17:40:24 gibi yeah
17:40:52 gibi also based on experience having tempest enabled make also a significant difference
17:41:00 dansmith did you pull the latest async? it should dump async-specific numbers after the component timing
17:41:20 dansmith yeah, definitely.. we rebuild the tempest venv several times :/
17:41:41 gibi dansmith: I pulled https://review.opendev.org/c/openstack/devstack/+/774022/5
17:42:39 dansmith okay then you should have it when async is enabled
17:43:41 dansmith like this: https://zuul.opendev.org/t/openstack/build/4f1863e956364177bf2152920636ff0b/log/job-output.txt#24932
17:44:06 gibi oh I see now in the logs, I just forget to copy it
17:44:17 gibi http://paste.openstack.org/show/802538/
17:44:54 dansmith that's similar to what I see locally, even before the compaction, interesting
17:46:48 gibi the bigger half of the compation is already merged and I have it in that run
17:47:36 dansmith I'm not sure I've RECLONE=yes since that stuff started to merge, so I'll try
17:48:10 dansmith maybe with the lower IO related to compaction we're parallelizing more things that won't compete, like just python startup and sql generation or something
17:48:50 dansmith but regardless, the two of these things should be noticeably better for the gate which is awesome
17:48:52 gibi this is the run with all the db compation pulled http://paste.openstack.org/show/802539/
17:49:15 dansmith cool

Earlier   Later