| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-02-10 | |||
| 15:44:55 | stephenfin | lyarwood, bauzas, melwitt: Can one of you take a look at https://review.opendev.org/c/openstack/nova/+/773727/ and https://review.opendev.org/c/openstack/nova/+/773728/ to fix the PPC64 gate, please | |
| 15:47:11 | bauzas | stephenfin: kinda try | |
| 15:48:04 | openstackgerrit | Stephen Finucane proposed openstack/os-vif master: Resolve dependency issues https://review.opendev.org/c/openstack/os-vif/+/774694 | |
| 15:48:09 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Revert "Run the db migration tests in the same test worker" https://review.opendev.org/c/openstack/nova/+/774889 | |
| 15:49:06 | gibi | stephenfin: here you are ^^ | |
| 15:49:11 | stephenfin | +2 | |
| 15:50:08 | gibi | I would keep the elastic signature still in place to see if the compaction really give us the speed bump on the gate I measured locally | |
| 15:53:48 | bauzas | stephenfin: https://review.opendev.org/c/openstack/nova/+/773728/ isn't needed for fixing the ppc64 gate job, righrt? | |
| 15:54:13 | stephenfin | bauzas: Yes, that's the crucial bit in fact | |
| 15:54:24 | bauzas | ok, then I misunderstood it | |
| 15:54:27 | bauzas | will review it | |
| 15:54:33 | stephenfin | http://lists.openstack.org/pipermail/openstack-discuss/2021-January/020088.html | |
| 15:55:20 | stephenfin | tl;dr: I added a patch to disable the USB controller if there were no USB devices in the XML, however, libvirt adds a USB keyboard and mouse automatically for guests on PPC64 hosts | |
| 15:55:40 | stephenfin | so it now fails | |
| 15:55:57 | bauzas | aha I see | |
| 15:56:16 | bauzas | it wants to add both the kb and the mouse but there are no longer usb controllers | |
| 15:56:21 | stephenfin | yup | |
| 15:56:26 | bauzas | gotcha, will look | |
| 15:56:41 | stephenfin | so we skip that logic on PPC64 now since we know we'll always need a controller on that platform | |
| 15:59:12 | lyarwood | stephenfin: sorry wasn't watching irc, I'll take a look at that and the other stuff from earlier after our next call | |
| 16:32:24 | dansmith | gibi: that's nifty, I didn't know about that grouping thing | |
| 16:33:47 | dansmith | gibi: did you happen to notice if the IO-related timeouts were specific to one provider? | |
| 16:34:20 | dansmith | tripleo was having job timeouts on one of their jobs that did a lot of IO and vexxhost switched our flavor to double the IOPs | |
| 16:35:26 | gibi | dansmith: the failures was spread amongst providers. at least I haven't noticed a pattern | |
| 16:35:32 | dansmith | okay | |
| 16:35:59 | gibi | I assumed it was IO limited as I was able to reproduce the extreme slowness locally if I IO deprived the machine | |
| 16:36:14 | dansmith | my parallel devstack ran all of those migrations in parallel for a very nice speedup on local hardware, | |
| 16:36:32 | dansmith | so maybe with 75% faster due to compaction we'll get a nice bump in the io-limited workers too | |
| 16:36:40 | dansmith | gibi: yeah probably | |
| 16:36:46 | gibi | yeah I saw you parallel devstack patch, that also looks like a nice bump. I will try it at some point | |
| 16:37:00 | dansmith | yeah, hopefully we can merge it soon | |
| 16:37:09 | dansmith | it's super nice to be able to stack locally in five minutes :) | |
| 16:39:39 | dansmith | devstack also runs db sync for keystone three times for some reason I have yet to figure out, | |
| 16:39:40 | dansmith | and while they don't have hundreds of migrations, it's non-trivial | |
| 16:52:15 | openstackgerrit | Elod Illes proposed openstack/nova stable/ussuri: Add upgrade check about old computes https://review.opendev.org/c/openstack/nova/+/770765 | |
| 16:52:44 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Record the machine_type of instances in system_metadata https://review.opendev.org/c/openstack/nova/+/767533 | |
| 16:52:45 | openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-manage: Add machine_type get command https://review.opendev.org/c/openstack/nova/+/769548 | |
| 16:52:46 | openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-manage: Add machine_type update command https://review.opendev.org/c/openstack/nova/+/774896 | |
| 16:52:46 | openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-status: Add hw_machine_type check for libvirt instances https://review.opendev.org/c/openstack/nova/+/770643 | |
| 16:52:47 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP nova-manage: Add machine_type list_unset command https://review.opendev.org/c/openstack/nova/+/774897 | |
| 16:52:48 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP libvirt: Add a config update workflow func test for [libvirt]hw_machine_type https://review.opendev.org/c/openstack/nova/+/774898 | |
| 16:52:49 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP docs: Add admin docs for configuring and updating machine types https://review.opendev.org/c/openstack/nova/+/774899 | |
| 17:15:53 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Move 'nova.virt.libvirt.utils.get_arch' stub to fixture https://review.opendev.org/c/openstack/nova/+/774904 | |
| 17:15:53 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Posion nova.objects.fields.Architecture.from_host https://review.opendev.org/c/openstack/nova/+/769920 | |
| 17:15:54 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Remove unnecessary mock https://review.opendev.org/c/openstack/nova/+/774905 | |
| 17:16:37 | stephenfin | lyarwood: Fixed that, I think ^ | |
| 17:19:50 | lyarwood | awesome, I'll take a look in the morning once the tests have passed | |
| 17:20:52 | stephenfin | ack | |
| 17:33:23 | gibi | dansmith: I saw ~17% speedup locally in devstack stacking with parellelism enabled. http://paste.openstack.org/show/802536/ | |
| 17:33:50 | gibi | dansmith: two interesting observations; 1) osc time went up when run in parallel 2) unacounted time become negative :) | |
| 17:34:16 | dansmith | gibi: is that with the db migrations squashed? | |
| 17:34:28 | dansmith | merged I mean | |
| 17:34:46 | gibi | dansmith: squashed up until Queens | |
| 17:35:03 | gibi | dansmith: I will rerun now with all the db squashes | |
| 17:35:42 | dansmith | okay, I'm not really sure all of the component timing actually works with async because they're run in subshells and the timings don't really get reflected to the parent, AFAIK | |
| 17:36:07 | gibi | ahh, I see | |
| 17:37:19 | dansmith | that's 25% increase from top to bottom, | |
| 17:37:53 | dansmith | I'm not sure why you'd see a difference with async patches applied but disabled, should be identical, unless you're in a VM with neighbor noise affecting the numbers | |
| 17:39:21 | gibi | dansmith: could be noise and could be that apt did updated things at the first run but not at the subsequent runs | |
| 17:39:28 | dansmith | ah okay yeah | |
| 17:39:56 | dansmith | compacting the migrations will definitely reduce the gain from the async tasks, since they're doing less, thus less to parallelize | |
| 17:40:18 | dansmith | but still better overall with both of course | |
| 17:40:24 | gibi | yeah | |
| 17:40:52 | gibi | also based on experience having tempest enabled make also a significant difference | |
| 17:41:00 | dansmith | did you pull the latest async? it should dump async-specific numbers after the component timing | |
| 17:41:20 | dansmith | yeah, definitely.. we rebuild the tempest venv several times :/ | |
| 17:41:41 | gibi | dansmith: I pulled https://review.opendev.org/c/openstack/devstack/+/774022/5 | |
| 17:42:39 | dansmith | okay then you should have it when async is enabled | |
| 17:43:41 | dansmith | like this: https://zuul.opendev.org/t/openstack/build/4f1863e956364177bf2152920636ff0b/log/job-output.txt#24932 | |
| 17:44:06 | gibi | oh I see now in the logs, I just forget to copy it | |
| 17:44:17 | gibi | http://paste.openstack.org/show/802538/ | |
| 17:44:54 | dansmith | that's similar to what I see locally, even before the compaction, interesting | |
| 17:46:48 | gibi | the bigger half of the compation is already merged and I have it in that run | |
| 17:47:36 | dansmith | I'm not sure I've RECLONE=yes since that stuff started to merge, so I'll try | |
| 17:48:10 | dansmith | maybe with the lower IO related to compaction we're parallelizing more things that won't compete, like just python startup and sql generation or something | |
| 17:48:50 | dansmith | but regardless, the two of these things should be noticeably better for the gate which is awesome | |
| 17:48:52 | gibi | this is the run with all the db compation pulled http://paste.openstack.org/show/802539/ | |
| 17:49:15 | dansmith | cool | |
| 17:49:24 | dansmith | hard to complain about 80% faster :) | |
| 17:49:42 | gibi | :) | |
| 17:49:47 | gibi | I'm not complaining :d | |
| 17:49:48 | gibi | :D | |
| 17:54:29 | dansmith | bauzas: are you still working on this? https://review.opendev.org/c/openstack/nova/+/761452 | |
| 17:55:02 | bauzas | dansmith: it's my next duty after the routed networks | |
| 17:55:07 | dansmith | okay cool | |
| 17:55:11 | dansmith | just making sure | |
| 17:55:31 | bauzas | dansmith: but i'll need you to look at it ;) | |
| 17:55:37 | bauzas | I got some issue with the CI | |
| 17:56:03 | dansmith | ack, it's in merge conflict now anyway, but poke me when you get back to it and have something for me to dig into | |
| 17:56:04 | bauzas | so I'll rebase with the next API version, and then uploading it, so we would get new CI | |
| 17:56:08 | dansmith | yeah cool | |
| 18:02:43 | openstackgerrit | Merged openstack/nova master: db: Compact Rocky database migrations https://review.opendev.org/c/openstack/nova/+/759089 | |
| 18:04:02 | openstackgerrit | Merged openstack/nova master: libvirt: Stub out 'nova.virt.libvirt.utils.get_arch' https://review.opendev.org/c/openstack/nova/+/773727 | |
| 18:12:01 | stephenfin | dansmith: gibi: FYI the DB compaction series doesn't seem to have fixed the issue :-( https://review.opendev.org/c/openstack/nova/+/774889/ | |
| 18:12:07 | stephenfin | assuming I'm reading that correctly | |
| 18:12:40 | dansmith | maybe something else is getting hung up? | |
| 18:12:53 | stephenfin | quite possibly. I haven't investigated yet | |
| 18:13:10 | stephenfin | Fair to say that probably shouldn't go in until we root cause it though | |
| 18:13:36 | gibi | I'm a sad panda | |
| 18:13:46 | gibi | stephenfin: you are readig it right | |