Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-02-10
17:41:20 dansmith yeah, definitely.. we rebuild the tempest venv several times :/
17:41:41 gibi dansmith: I pulled https://review.opendev.org/c/openstack/devstack/+/774022/5
17:42:39 dansmith okay then you should have it when async is enabled
17:43:41 dansmith like this: https://zuul.opendev.org/t/openstack/build/4f1863e956364177bf2152920636ff0b/log/job-output.txt#24932
17:44:06 gibi oh I see now in the logs, I just forget to copy it
17:44:17 gibi http://paste.openstack.org/show/802538/
17:44:54 dansmith that's similar to what I see locally, even before the compaction, interesting
17:46:48 gibi the bigger half of the compation is already merged and I have it in that run
17:47:36 dansmith I'm not sure I've RECLONE=yes since that stuff started to merge, so I'll try
17:48:10 dansmith maybe with the lower IO related to compaction we're parallelizing more things that won't compete, like just python startup and sql generation or something
17:48:50 dansmith but regardless, the two of these things should be noticeably better for the gate which is awesome
17:48:52 gibi this is the run with all the db compation pulled http://paste.openstack.org/show/802539/
17:49:15 dansmith cool
17:49:24 dansmith hard to complain about 80% faster :)
17:49:42 gibi :)
17:49:47 gibi I'm not complaining :d
17:49:48 gibi :D
17:54:29 dansmith bauzas: are you still working on this? https://review.opendev.org/c/openstack/nova/+/761452
17:55:02 bauzas dansmith: it's my next duty after the routed networks
17:55:07 dansmith okay cool
17:55:11 dansmith just making sure
17:55:31 bauzas dansmith: but i'll need you to look at it ;)
17:55:37 bauzas I got some issue with the CI
17:56:03 dansmith ack, it's in merge conflict now anyway, but poke me when you get back to it and have something for me to dig into
17:56:04 bauzas so I'll rebase with the next API version, and then uploading it, so we would get new CI
17:56:08 dansmith yeah cool
18:02:43 openstackgerrit Merged openstack/nova master: db: Compact Rocky database migrations https://review.opendev.org/c/openstack/nova/+/759089
18:04:02 openstackgerrit Merged openstack/nova master: libvirt: Stub out 'nova.virt.libvirt.utils.get_arch' https://review.opendev.org/c/openstack/nova/+/773727
18:12:01 stephenfin dansmith: gibi: FYI the DB compaction series doesn't seem to have fixed the issue :-( https://review.opendev.org/c/openstack/nova/+/774889/
18:12:07 stephenfin assuming I'm reading that correctly
18:12:40 dansmith maybe something else is getting hung up?
18:12:53 stephenfin quite possibly. I haven't investigated yet
18:13:10 stephenfin Fair to say that probably shouldn't go in until we root cause it though
18:13:36 gibi I'm a sad panda
18:13:46 gibi stephenfin: you are readig it right
18:15:21 gibi the I have to assume that the way I was able to reproducing it locally was actually reproduced the symptom but in a different way how it happens on the gate
18:16:53 gibi maybe we can try to dump logs at each step of test_walk_versions to see if there is a certain step that has high chance to hang
18:21:13 gibi but not today
18:21:14 gibi o/
19:20:21 openstackgerrit Stephen Finucane proposed openstack/nova master: apidb: Compact Ocata database migrations https://review.opendev.org/c/openstack/nova/+/759402
19:20:22 openstackgerrit Stephen Finucane proposed openstack/nova master: apidb: Compact Pike database migrations https://review.opendev.org/c/openstack/nova/+/759403
19:20:22 openstackgerrit Stephen Finucane proposed openstack/nova master: apidb: Compact Queens database migrations https://review.opendev.org/c/openstack/nova/+/759404
19:20:23 openstackgerrit Stephen Finucane proposed openstack/nova master: apidb: Compact Rocky database migrations https://review.opendev.org/c/openstack/nova/+/759405
19:20:23 openstackgerrit Stephen Finucane proposed openstack/nova master: apidb: Compact Stein database migrations https://review.opendev.org/c/openstack/nova/+/759406
19:20:24 openstackgerrit Stephen Finucane proposed openstack/nova master: apidb: Compact Train database migrations https://review.opendev.org/c/openstack/nova/+/771420
19:38:07 sean-k-mooney oh your talking about the db walk timeout ? or test_models_sync
19:39:05 sean-k-mooney the time out ligitablity might just be a side effect of a slow node. gibi tried to extend the time previousl but the other issue in test_models_sync are new to me
19:41:06 sean-k-mooney that kind of looks like the db tables did not create properly
19:41:24 stephenfin gmann: Could you take a look at https://review.opendev.org/c/openstack/placement/+/774827/ too. Seems it was missed previously
19:41:30 stephenfin That will unblock the release note
19:55:09 openstackgerrit Martin Kopec proposed openstack/nova stable/rocky: DNM - testing tempest-slow job https://review.opendev.org/c/openstack/nova/+/775003
20:27:27 gibi sean-k-mooney: the db compation should have been sped up the db migration test a lot
20:27:31 gibi sean-k-mooney: but it still times out
20:27:47 sean-k-mooney yep it should have helped but it was not gurenteed to fix it
20:27:53 sean-k-mooney just make it much much less likely
20:28:23 dansmith in terms of IO, I guess the only thing that it necessarily does less of is when we drop stuff
20:28:44 gibi we do a lot less testing now
20:28:46 dansmith I guess I'm not sure how much thrashing there really is.. the initial migration was always the bulk of it
20:29:02 dansmith yeah, right, okay in that test where we walk the migrations, that's right
20:29:10 sean-k-mooney dansmith: the table_walk i think is doing each migration one by one
20:29:41 sean-k-mooney so that "shoudl" be run the compacted migration + stuff form this cycle
20:29:50 sean-k-mooney at least when they are all merged
20:30:16 dansmith yeah, I was thinking about the total IO load of starting with an empty DB and getting to a current schema,
20:30:22 dansmith but the test will obviously do fewer things
20:38:06 sean-k-mooney dansmith: so pararrle devstack + osc as a service reduced stack time 40%
20:38:53 sean-k-mooney granted that wont be a 40% speed up in job time as we spend or 50% of it running tempest
20:39:22 sean-k-mooney but that should still be 10-15% maybe overall
20:39:38 sean-k-mooney that pretty dramatic
20:40:15 sean-k-mooney + the developer workflow improment
20:41:34 sean-k-mooney that said i have been used to sub 20 min stack times basiclay for ever due to how i use it but 5-10 min stack times is still great
20:41:55 dansmith yep, very non-trivial savings multiplied across every job runm
20:42:02 dansmith and local dev time improvement is huge
20:42:40 sean-k-mooney ya i used to lement not being able to use offline mode or wehn i could not just restart the service due to db changes
20:43:11 sean-k-mooney but if its that quick honestly i would restack more often then i do
20:43:15 dansmith 5 minutes is on the order of "hang on let me stack so I can answer that CLI usage question in realtime"
20:43:33 sean-k-mooney yep
20:44:22 sean-k-mooney i have done that in meeting wtih a 15min time but 5 mins is shorter then ill stack this while i get coffee
21:47:37 sean-k-mooney is the build_requests table in the api db used anymore?
21:47:50 sean-k-mooney it seams to always be empty
21:48:42 sean-k-mooney maybe we automatically delete the entry once it builds
21:50:58 atmark hello, is it possible to rename a cell ?
21:52:03 sean-k-mooney cell0 no the other cells im not sure
21:52:35 sean-k-mooney without db change i dont think there is any nova manager command you can updated
21:54:19 sean-k-mooney the host_mappings just use the cell_id im not sure what if anything uses the cell name corm the cell_mappings tables
21:54:23 artom atmark, also, *why*?
21:56:26 sean-k-mooney ya even if you can do it you proably should not do it unless you have a good reason too
21:57:48 sean-k-mooney i belive we map cells to placment aggreates at some point. im not sure if that is by name or by id but i would not assume there is nothing that depends on the cell name without fully checking
21:59:29 sean-k-mooney you cant rename Avaiableity zones with out breaking things for example. gratted they exists at the api level and cells do not but the cell name might be storeed somewhere although we should really be usign the id.
22:02:31 gmann stephenfin: done. +A
22:12:49 openstackgerrit Merged openstack/nova master: libvirt: Always enable USB controller on PPC64 https://review.opendev.org/c/openstack/nova/+/773728
22:13:38 openstackgerrit Ghanshyam proposed openstack/placement master: Move policy deprecation to base rules https://review.opendev.org/c/openstack/placement/+/772784
22:15:30 gmann stephenfin: updated this for reshaper policy - https://review.opendev.org/c/openstack/placement/+/772784
23:11:22 openstackgerrit Ghanshyam proposed openstack/nova master: Switch to new rolevar for run-temepst role https://review.opendev.org/c/openstack/nova/+/775040
23:27:36 sean-k-mooney oh dead code i can channel my inner stephenfin and delete it :)
23:28:28 sean-k-mooney the only thin that uses _load_legacy_requests is get_by_instance and the only thing that uses that is test code
23:28:30 sean-k-mooney https://github.com/openstack/nova/blob/5b52db409b00e6257e83598efa0d8528ebe380e0/nova/objects/instance_pci_requests.py#L116-L146
23:28:45 sean-k-mooney we use get_by_instance_uuid instead now
23:29:25 sean-k-mooney since we started storign the request spec in teh api_db like 4? 5? cycles ago
23:29:47 sean-k-mooney its been a very long time since it was stored in teh instace_system_metadata table
23:31:00 sean-k-mooney i think maybe melwitt moved it back in queens/rocky? maybe older then that

Earlier   Later