| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-02-10 | |||
| 15:56:41 | stephenfin | so we skip that logic on PPC64 now since we know we'll always need a controller on that platform | |
| 15:59:12 | lyarwood | stephenfin: sorry wasn't watching irc, I'll take a look at that and the other stuff from earlier after our next call | |
| 16:32:24 | dansmith | gibi: that's nifty, I didn't know about that grouping thing | |
| 16:33:47 | dansmith | gibi: did you happen to notice if the IO-related timeouts were specific to one provider? | |
| 16:34:20 | dansmith | tripleo was having job timeouts on one of their jobs that did a lot of IO and vexxhost switched our flavor to double the IOPs | |
| 16:35:26 | gibi | dansmith: the failures was spread amongst providers. at least I haven't noticed a pattern | |
| 16:35:32 | dansmith | okay | |
| 16:35:59 | gibi | I assumed it was IO limited as I was able to reproduce the extreme slowness locally if I IO deprived the machine | |
| 16:36:14 | dansmith | my parallel devstack ran all of those migrations in parallel for a very nice speedup on local hardware, | |
| 16:36:32 | dansmith | so maybe with 75% faster due to compaction we'll get a nice bump in the io-limited workers too | |
| 16:36:40 | dansmith | gibi: yeah probably | |
| 16:36:46 | gibi | yeah I saw you parallel devstack patch, that also looks like a nice bump. I will try it at some point | |
| 16:37:00 | dansmith | yeah, hopefully we can merge it soon | |
| 16:37:09 | dansmith | it's super nice to be able to stack locally in five minutes :) | |
| 16:39:39 | dansmith | devstack also runs db sync for keystone three times for some reason I have yet to figure out, | |
| 16:39:40 | dansmith | and while they don't have hundreds of migrations, it's non-trivial | |
| 16:52:15 | openstackgerrit | Elod Illes proposed openstack/nova stable/ussuri: Add upgrade check about old computes https://review.opendev.org/c/openstack/nova/+/770765 | |
| 16:52:44 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Record the machine_type of instances in system_metadata https://review.opendev.org/c/openstack/nova/+/767533 | |
| 16:52:45 | openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-manage: Add machine_type get command https://review.opendev.org/c/openstack/nova/+/769548 | |
| 16:52:46 | openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-manage: Add machine_type update command https://review.opendev.org/c/openstack/nova/+/774896 | |
| 16:52:46 | openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-status: Add hw_machine_type check for libvirt instances https://review.opendev.org/c/openstack/nova/+/770643 | |
| 16:52:47 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP nova-manage: Add machine_type list_unset command https://review.opendev.org/c/openstack/nova/+/774897 | |
| 16:52:48 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP libvirt: Add a config update workflow func test for [libvirt]hw_machine_type https://review.opendev.org/c/openstack/nova/+/774898 | |
| 16:52:49 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP docs: Add admin docs for configuring and updating machine types https://review.opendev.org/c/openstack/nova/+/774899 | |
| 17:15:53 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Move 'nova.virt.libvirt.utils.get_arch' stub to fixture https://review.opendev.org/c/openstack/nova/+/774904 | |
| 17:15:53 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Posion nova.objects.fields.Architecture.from_host https://review.opendev.org/c/openstack/nova/+/769920 | |
| 17:15:54 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Remove unnecessary mock https://review.opendev.org/c/openstack/nova/+/774905 | |
| 17:16:37 | stephenfin | lyarwood: Fixed that, I think ^ | |
| 17:19:50 | lyarwood | awesome, I'll take a look in the morning once the tests have passed | |
| 17:20:52 | stephenfin | ack | |
| 17:33:23 | gibi | dansmith: I saw ~17% speedup locally in devstack stacking with parellelism enabled. http://paste.openstack.org/show/802536/ | |
| 17:33:50 | gibi | dansmith: two interesting observations; 1) osc time went up when run in parallel 2) unacounted time become negative :) | |
| 17:34:16 | dansmith | gibi: is that with the db migrations squashed? | |
| 17:34:28 | dansmith | merged I mean | |
| 17:34:46 | gibi | dansmith: squashed up until Queens | |
| 17:35:03 | gibi | dansmith: I will rerun now with all the db squashes | |
| 17:35:42 | dansmith | okay, I'm not really sure all of the component timing actually works with async because they're run in subshells and the timings don't really get reflected to the parent, AFAIK | |
| 17:36:07 | gibi | ahh, I see | |
| 17:37:19 | dansmith | that's 25% increase from top to bottom, | |
| 17:37:53 | dansmith | I'm not sure why you'd see a difference with async patches applied but disabled, should be identical, unless you're in a VM with neighbor noise affecting the numbers | |
| 17:39:21 | gibi | dansmith: could be noise and could be that apt did updated things at the first run but not at the subsequent runs | |
| 17:39:28 | dansmith | ah okay yeah | |
| 17:39:56 | dansmith | compacting the migrations will definitely reduce the gain from the async tasks, since they're doing less, thus less to parallelize | |
| 17:40:18 | dansmith | but still better overall with both of course | |
| 17:40:24 | gibi | yeah | |
| 17:40:52 | gibi | also based on experience having tempest enabled make also a significant difference | |
| 17:41:00 | dansmith | did you pull the latest async? it should dump async-specific numbers after the component timing | |
| 17:41:20 | dansmith | yeah, definitely.. we rebuild the tempest venv several times :/ | |
| 17:41:41 | gibi | dansmith: I pulled https://review.opendev.org/c/openstack/devstack/+/774022/5 | |
| 17:42:39 | dansmith | okay then you should have it when async is enabled | |
| 17:43:41 | dansmith | like this: https://zuul.opendev.org/t/openstack/build/4f1863e956364177bf2152920636ff0b/log/job-output.txt#24932 | |
| 17:44:06 | gibi | oh I see now in the logs, I just forget to copy it | |
| 17:44:17 | gibi | http://paste.openstack.org/show/802538/ | |
| 17:44:54 | dansmith | that's similar to what I see locally, even before the compaction, interesting | |
| 17:46:48 | gibi | the bigger half of the compation is already merged and I have it in that run | |
| 17:47:36 | dansmith | I'm not sure I've RECLONE=yes since that stuff started to merge, so I'll try | |
| 17:48:10 | dansmith | maybe with the lower IO related to compaction we're parallelizing more things that won't compete, like just python startup and sql generation or something | |
| 17:48:50 | dansmith | but regardless, the two of these things should be noticeably better for the gate which is awesome | |
| 17:48:52 | gibi | this is the run with all the db compation pulled http://paste.openstack.org/show/802539/ | |
| 17:49:15 | dansmith | cool | |
| 17:49:24 | dansmith | hard to complain about 80% faster :) | |
| 17:49:42 | gibi | :) | |
| 17:49:47 | gibi | I'm not complaining :d | |
| 17:49:48 | gibi | :D | |
| 17:54:29 | dansmith | bauzas: are you still working on this? https://review.opendev.org/c/openstack/nova/+/761452 | |
| 17:55:02 | bauzas | dansmith: it's my next duty after the routed networks | |
| 17:55:07 | dansmith | okay cool | |
| 17:55:11 | dansmith | just making sure | |
| 17:55:31 | bauzas | dansmith: but i'll need you to look at it ;) | |
| 17:55:37 | bauzas | I got some issue with the CI | |
| 17:56:03 | dansmith | ack, it's in merge conflict now anyway, but poke me when you get back to it and have something for me to dig into | |
| 17:56:04 | bauzas | so I'll rebase with the next API version, and then uploading it, so we would get new CI | |
| 17:56:08 | dansmith | yeah cool | |
| 18:02:43 | openstackgerrit | Merged openstack/nova master: db: Compact Rocky database migrations https://review.opendev.org/c/openstack/nova/+/759089 | |
| 18:04:02 | openstackgerrit | Merged openstack/nova master: libvirt: Stub out 'nova.virt.libvirt.utils.get_arch' https://review.opendev.org/c/openstack/nova/+/773727 | |
| 18:12:01 | stephenfin | dansmith: gibi: FYI the DB compaction series doesn't seem to have fixed the issue :-( https://review.opendev.org/c/openstack/nova/+/774889/ | |
| 18:12:07 | stephenfin | assuming I'm reading that correctly | |
| 18:12:40 | dansmith | maybe something else is getting hung up? | |
| 18:12:53 | stephenfin | quite possibly. I haven't investigated yet | |
| 18:13:10 | stephenfin | Fair to say that probably shouldn't go in until we root cause it though | |
| 18:13:36 | gibi | I'm a sad panda | |
| 18:13:46 | gibi | stephenfin: you are readig it right | |
| 18:15:21 | gibi | the I have to assume that the way I was able to reproducing it locally was actually reproduced the symptom but in a different way how it happens on the gate | |
| 18:16:53 | gibi | maybe we can try to dump logs at each step of test_walk_versions to see if there is a certain step that has high chance to hang | |
| 18:21:13 | gibi | but not today | |
| 18:21:14 | gibi | o/ | |
| 19:20:21 | openstackgerrit | Stephen Finucane proposed openstack/nova master: apidb: Compact Ocata database migrations https://review.opendev.org/c/openstack/nova/+/759402 | |
| 19:20:22 | openstackgerrit | Stephen Finucane proposed openstack/nova master: apidb: Compact Queens database migrations https://review.opendev.org/c/openstack/nova/+/759404 | |
| 19:20:22 | openstackgerrit | Stephen Finucane proposed openstack/nova master: apidb: Compact Pike database migrations https://review.opendev.org/c/openstack/nova/+/759403 | |
| 19:20:23 | openstackgerrit | Stephen Finucane proposed openstack/nova master: apidb: Compact Stein database migrations https://review.opendev.org/c/openstack/nova/+/759406 | |
| 19:20:23 | openstackgerrit | Stephen Finucane proposed openstack/nova master: apidb: Compact Rocky database migrations https://review.opendev.org/c/openstack/nova/+/759405 | |
| 19:20:24 | openstackgerrit | Stephen Finucane proposed openstack/nova master: apidb: Compact Train database migrations https://review.opendev.org/c/openstack/nova/+/771420 | |
| 19:38:07 | sean-k-mooney | oh your talking about the db walk timeout ? or test_models_sync | |
| 19:39:05 | sean-k-mooney | the time out ligitablity might just be a side effect of a slow node. gibi tried to extend the time previousl but the other issue in test_models_sync are new to me | |
| 19:41:06 | sean-k-mooney | that kind of looks like the db tables did not create properly | |
| 19:41:24 | stephenfin | gmann: Could you take a look at https://review.opendev.org/c/openstack/placement/+/774827/ too. Seems it was missed previously | |
| 19:41:30 | stephenfin | That will unblock the release note | |
| 19:55:09 | openstackgerrit | Martin Kopec proposed openstack/nova stable/rocky: DNM - testing tempest-slow job https://review.opendev.org/c/openstack/nova/+/775003 | |
| 20:27:27 | gibi | sean-k-mooney: the db compation should have been sped up the db migration test a lot | |
| 20:27:31 | gibi | sean-k-mooney: but it still times out | |