| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-02-10 | |||
| 17:34:16 | dansmith | gibi: is that with the db migrations squashed? | |
| 17:34:28 | dansmith | merged I mean | |
| 17:34:46 | gibi | dansmith: squashed up until Queens | |
| 17:35:03 | gibi | dansmith: I will rerun now with all the db squashes | |
| 17:35:42 | dansmith | okay, I'm not really sure all of the component timing actually works with async because they're run in subshells and the timings don't really get reflected to the parent, AFAIK | |
| 17:36:07 | gibi | ahh, I see | |
| 17:37:19 | dansmith | that's 25% increase from top to bottom, | |
| 17:37:53 | dansmith | I'm not sure why you'd see a difference with async patches applied but disabled, should be identical, unless you're in a VM with neighbor noise affecting the numbers | |
| 17:39:21 | gibi | dansmith: could be noise and could be that apt did updated things at the first run but not at the subsequent runs | |
| 17:39:28 | dansmith | ah okay yeah | |
| 17:39:56 | dansmith | compacting the migrations will definitely reduce the gain from the async tasks, since they're doing less, thus less to parallelize | |
| 17:40:18 | dansmith | but still better overall with both of course | |
| 17:40:24 | gibi | yeah | |
| 17:40:52 | gibi | also based on experience having tempest enabled make also a significant difference | |
| 17:41:00 | dansmith | did you pull the latest async? it should dump async-specific numbers after the component timing | |
| 17:41:20 | dansmith | yeah, definitely.. we rebuild the tempest venv several times :/ | |
| 17:41:41 | gibi | dansmith: I pulled https://review.opendev.org/c/openstack/devstack/+/774022/5 | |
| 17:42:39 | dansmith | okay then you should have it when async is enabled | |
| 17:43:41 | dansmith | like this: https://zuul.opendev.org/t/openstack/build/4f1863e956364177bf2152920636ff0b/log/job-output.txt#24932 | |
| 17:44:06 | gibi | oh I see now in the logs, I just forget to copy it | |
| 17:44:17 | gibi | http://paste.openstack.org/show/802538/ | |
| 17:44:54 | dansmith | that's similar to what I see locally, even before the compaction, interesting | |
| 17:46:48 | gibi | the bigger half of the compation is already merged and I have it in that run | |
| 17:47:36 | dansmith | I'm not sure I've RECLONE=yes since that stuff started to merge, so I'll try | |
| 17:48:10 | dansmith | maybe with the lower IO related to compaction we're parallelizing more things that won't compete, like just python startup and sql generation or something | |
| 17:48:50 | dansmith | but regardless, the two of these things should be noticeably better for the gate which is awesome | |
| 17:48:52 | gibi | this is the run with all the db compation pulled http://paste.openstack.org/show/802539/ | |
| 17:49:15 | dansmith | cool | |
| 17:49:24 | dansmith | hard to complain about 80% faster :) | |
| 17:49:42 | gibi | :) | |
| 17:49:47 | gibi | I'm not complaining :d | |
| 17:49:48 | gibi | :D | |
| 17:54:29 | dansmith | bauzas: are you still working on this? https://review.opendev.org/c/openstack/nova/+/761452 | |
| 17:55:02 | bauzas | dansmith: it's my next duty after the routed networks | |
| 17:55:07 | dansmith | okay cool | |
| 17:55:11 | dansmith | just making sure | |
| 17:55:31 | bauzas | dansmith: but i'll need you to look at it ;) | |
| 17:55:37 | bauzas | I got some issue with the CI | |
| 17:56:03 | dansmith | ack, it's in merge conflict now anyway, but poke me when you get back to it and have something for me to dig into | |
| 17:56:04 | bauzas | so I'll rebase with the next API version, and then uploading it, so we would get new CI | |
| 17:56:08 | dansmith | yeah cool | |
| 18:02:43 | openstackgerrit | Merged openstack/nova master: db: Compact Rocky database migrations https://review.opendev.org/c/openstack/nova/+/759089 | |
| 18:04:02 | openstackgerrit | Merged openstack/nova master: libvirt: Stub out 'nova.virt.libvirt.utils.get_arch' https://review.opendev.org/c/openstack/nova/+/773727 | |
| 18:12:01 | stephenfin | dansmith: gibi: FYI the DB compaction series doesn't seem to have fixed the issue :-( https://review.opendev.org/c/openstack/nova/+/774889/ | |
| 18:12:07 | stephenfin | assuming I'm reading that correctly | |
| 18:12:40 | dansmith | maybe something else is getting hung up? | |
| 18:12:53 | stephenfin | quite possibly. I haven't investigated yet | |
| 18:13:10 | stephenfin | Fair to say that probably shouldn't go in until we root cause it though | |
| 18:13:36 | gibi | I'm a sad panda | |
| 18:13:46 | gibi | stephenfin: you are readig it right | |
| 18:15:21 | gibi | the I have to assume that the way I was able to reproducing it locally was actually reproduced the symptom but in a different way how it happens on the gate | |
| 18:16:53 | gibi | maybe we can try to dump logs at each step of test_walk_versions to see if there is a certain step that has high chance to hang | |
| 18:21:13 | gibi | but not today | |
| 18:21:14 | gibi | o/ | |
| 19:20:21 | openstackgerrit | Stephen Finucane proposed openstack/nova master: apidb: Compact Ocata database migrations https://review.opendev.org/c/openstack/nova/+/759402 | |
| 19:20:22 | openstackgerrit | Stephen Finucane proposed openstack/nova master: apidb: Compact Pike database migrations https://review.opendev.org/c/openstack/nova/+/759403 | |
| 19:20:22 | openstackgerrit | Stephen Finucane proposed openstack/nova master: apidb: Compact Queens database migrations https://review.opendev.org/c/openstack/nova/+/759404 | |
| 19:20:23 | openstackgerrit | Stephen Finucane proposed openstack/nova master: apidb: Compact Rocky database migrations https://review.opendev.org/c/openstack/nova/+/759405 | |
| 19:20:23 | openstackgerrit | Stephen Finucane proposed openstack/nova master: apidb: Compact Stein database migrations https://review.opendev.org/c/openstack/nova/+/759406 | |
| 19:20:24 | openstackgerrit | Stephen Finucane proposed openstack/nova master: apidb: Compact Train database migrations https://review.opendev.org/c/openstack/nova/+/771420 | |
| 19:38:07 | sean-k-mooney | oh your talking about the db walk timeout ? or test_models_sync | |
| 19:39:05 | sean-k-mooney | the time out ligitablity might just be a side effect of a slow node. gibi tried to extend the time previousl but the other issue in test_models_sync are new to me | |
| 19:41:06 | sean-k-mooney | that kind of looks like the db tables did not create properly | |
| 19:41:24 | stephenfin | gmann: Could you take a look at https://review.opendev.org/c/openstack/placement/+/774827/ too. Seems it was missed previously | |
| 19:41:30 | stephenfin | That will unblock the release note | |
| 19:55:09 | openstackgerrit | Martin Kopec proposed openstack/nova stable/rocky: DNM - testing tempest-slow job https://review.opendev.org/c/openstack/nova/+/775003 | |
| 20:27:27 | gibi | sean-k-mooney: the db compation should have been sped up the db migration test a lot | |
| 20:27:31 | gibi | sean-k-mooney: but it still times out | |
| 20:27:47 | sean-k-mooney | yep it should have helped but it was not gurenteed to fix it | |
| 20:27:53 | sean-k-mooney | just make it much much less likely | |
| 20:28:23 | dansmith | in terms of IO, I guess the only thing that it necessarily does less of is when we drop stuff | |
| 20:28:44 | gibi | we do a lot less testing now | |
| 20:28:46 | dansmith | I guess I'm not sure how much thrashing there really is.. the initial migration was always the bulk of it | |
| 20:29:02 | dansmith | yeah, right, okay in that test where we walk the migrations, that's right | |
| 20:29:10 | sean-k-mooney | dansmith: the table_walk i think is doing each migration one by one | |
| 20:29:41 | sean-k-mooney | so that "shoudl" be run the compacted migration + stuff form this cycle | |
| 20:29:50 | sean-k-mooney | at least when they are all merged | |
| 20:30:16 | dansmith | yeah, I was thinking about the total IO load of starting with an empty DB and getting to a current schema, | |
| 20:30:22 | dansmith | but the test will obviously do fewer things | |
| 20:38:06 | sean-k-mooney | dansmith: so pararrle devstack + osc as a service reduced stack time 40% | |
| 20:38:53 | sean-k-mooney | granted that wont be a 40% speed up in job time as we spend or 50% of it running tempest | |
| 20:39:22 | sean-k-mooney | but that should still be 10-15% maybe overall | |
| 20:39:38 | sean-k-mooney | that pretty dramatic | |
| 20:40:15 | sean-k-mooney | + the developer workflow improment | |
| 20:41:34 | sean-k-mooney | that said i have been used to sub 20 min stack times basiclay for ever due to how i use it but 5-10 min stack times is still great | |
| 20:41:55 | dansmith | yep, very non-trivial savings multiplied across every job runm | |
| 20:42:02 | dansmith | and local dev time improvement is huge | |
| 20:42:40 | sean-k-mooney | ya i used to lement not being able to use offline mode or wehn i could not just restart the service due to db changes | |
| 20:43:11 | sean-k-mooney | but if its that quick honestly i would restack more often then i do | |
| 20:43:15 | dansmith | 5 minutes is on the order of "hang on let me stack so I can answer that CLI usage question in realtime" | |
| 20:43:33 | sean-k-mooney | yep | |
| 20:44:22 | sean-k-mooney | i have done that in meeting wtih a 15min time but 5 mins is shorter then ill stack this while i get coffee | |
| 21:47:37 | sean-k-mooney | is the build_requests table in the api db used anymore? | |
| 21:47:50 | sean-k-mooney | it seams to always be empty | |
| 21:48:42 | sean-k-mooney | maybe we automatically delete the entry once it builds | |
| 21:50:58 | atmark | hello, is it possible to rename a cell ? | |
| 21:52:03 | sean-k-mooney | cell0 no the other cells im not sure | |
| 21:52:35 | sean-k-mooney | without db change i dont think there is any nova manager command you can updated | |
| 21:54:19 | sean-k-mooney | the host_mappings just use the cell_id im not sure what if anything uses the cell name corm the cell_mappings tables | |
| 21:54:23 | artom | atmark, also, *why*? | |