Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-03-07
16:10:56 sean-k-mooney we just enabled it a few days ago
16:11:20 dansmith it's voting,
16:11:21 sean-k-mooney i did not think it was failing but it can certenly be set non-voting or moved to periodic
16:11:26 dansmith and I just saw a kernel panic on it
16:11:33 dansmith I think it's a guest kernel
16:12:00 sean-k-mooney https://zuul.openstack.org/builds?job_name=nova-emulation&skip=0
16:12:06 sean-k-mooney it looks kind of ok
16:12:14 sean-k-mooney i think that is the first failure since it was merged
16:12:16 dansmith https://zuul.opendev.org/t/openstack/build/cb1314bff0f34bfdbb3a4f1fd5547b72
16:12:39 dansmith okay, well, regardless, nova jobs are looking pretty heavy
16:12:55 dansmith I dunno how widely-known it is, but we're losing 30% of our CI capacity at the end of the month
16:13:43 dansmith so we'll probably need to be making some cuts
16:13:51 dansmith what's the major benefit of testing arm-on-x86?
16:14:30 sean-k-mooney its a proxy for ensureing that the new emulation featur works in general
16:14:40 sean-k-mooney it could be a weekly job
16:14:51 sean-k-mooney or run only on libvirt changes
16:14:55 dansmith the thing that lets us choose the guest emulation mode you mean?
16:15:38 dansmith couldn't it be a single test? if we have an arm image available, couldn't we just boot one instance from it and make sure it's alive, instead of a whole other job?
16:16:04 sean-k-mooney well we ant to ensure resize ectra works
16:16:29 sean-k-mooney we could proably do it as a post action or something more light weight
16:16:35 dansmith sure, so one scenario test that boots, resize, snapshot, etc
16:16:43 sean-k-mooney ya we coudl do that
16:16:56 dansmith just saying, it seems pretty expensive for a minor verification
16:17:14 sean-k-mooney well the idea was to test all feature with emulation
16:17:49 sean-k-mooney but we can 1 move it to weakly and 2 make it a set of senario tests
16:17:57 sean-k-mooney chateaulav:^
16:18:01 dansmith yeah, ideally we'd run every configuration on every patch, but..
16:20:05 chateaulav sean-k-mooney: would the scenario tests need added to the tempest project?
16:20:15 sean-k-mooney ya
16:20:19 sean-k-mooney well or as a plugin
16:20:35 sean-k-mooney but upstream tempest i think would be ok
16:22:45 chateaulav yeah, i noticed it took some time for the ci itself to run. so then we want to pursue a new tempest scenario test that can be added into another ci?
16:23:05 chateaulav then pause the nova emulation, or run it not as frequently?
16:23:49 dansmith if there's a high likelihood of it being broken, then a tempest test to check that on each patch would be good
16:23:59 dansmith however, if it's not very likely, then a weekly periodic test would be better and easier
16:24:02 dansmith I suspect the latter
16:26:06 chateaulav yeah. i think long term, maybe next cycle add in the tempest scenario that we can leverage. I think the weekly periodic would be good for the interim though.
16:26:17 chateaulav your thoughts sean-k-mooney
16:27:11 dansmith was anything not working when we first tried to do this?
16:27:24 opendevreview Alexey Stupnikov proposed openstack/nova master: Clean up when queued live migration aborted https://review.opendev.org/c/openstack/nova/+/828570
16:29:13 chateaulav what do you mena in regards to not working?
16:30:31 dansmith chateaulav: you added the ability to select the guest emulation mode right? when you added that, were other things broken that made that non-trivial?
16:30:52 dansmith or, how invasive was the change? it thought it was mostly just a flag
16:33:04 chateaulav yeah, so the main item is the meta property that lets you define the guest architecture
16:33:28 chateaulav everything else was mods to the various checks to account for reading that value along with the host arch
16:34:40 kashyap sean-k-mooney: I franky question the value of this "nova-emulation" job, given dansmith's comment on the impact.
16:34:57 kashyap Also who are the users for this?
16:35:27 chateaulav and then choosing the guest arch if it was defined. so the ci is just to ensure the emulation works. it is highly likely that changes to nova wont affect its functionality, because it follows the logical paths for the physical architecture support
16:35:50 sean-k-mooney kashyap: well chateaulav for one :)
16:36:08 kashyap Hmm, still
16:36:19 kashyap chateaulav: Also, please note: https://www.qemu.org/docs/master/system/security.html#non-virtualization-use-case
16:37:34 sean-k-mooney kashyap: they are aware. there are many production uscase for it even with that in mind
16:37:51 dansmith yeah, really seems pretty low-impact in terms of a feature, and a whole job on every change is very high cost
16:37:59 sean-k-mooney probly not public cloud
16:38:05 dansmith I tend to think that even a scenario in every job is more expensive than we need
16:38:15 dansmith a weekly periodic is fine if we want, but..
16:38:48 kashyap Yeah, 30% impact on other nodes is just too much
16:39:05 sean-k-mooney well its not 30% from this job
16:39:08 kashyap sean-k-mooney: "many cases" - I'm assuming they don't give a hoot about security
16:39:13 sean-k-mooney we are loosing one of the providers i assume
16:39:39 sean-k-mooney kashyap: much of our downstream ci use qemu some uses kvm
16:40:17 sean-k-mooney so for ci, package building it think its fine
16:40:23 kashyap sean-k-mooney: Well, near as I know, most is exercising nested KVM
16:40:32 kashyap Internal CI is fine
16:40:58 sean-k-mooney dont forget that rackspace used to run there public cloud on power provideign x86 vms
16:41:32 dansmith um, what? that's news to me :)
16:41:56 dansmith I think they toyed with that, probably for second-source reasons but.. not to my knowledge for anything real
16:42:23 sean-k-mooney they used ot have xen but also ppc host
16:42:23 dansmith even still, that doesn't mean it makes sense, or is a good idea with qemu, and arm on x86 :)
16:43:00 dansmith yeah, probably for political reasons :)
16:43:11 sean-k-mooney perhaps
16:44:34 chateaulav yeah initial use of this is not meant to real-world systems. it is to bring testing and validation forward a little more so you dont have to run physical, and then work towards greater parity going forward
16:48:26 kashyap chateaulav: Okay, as long as you're clear that for any production usage this cross-arch emulation is entirely unfit.
16:49:28 kashyap Depending on the (cross-arch emulation) config, you still have _massive_ holes for a truck to comfortably drive through ;-)
16:50:06 chateaulav correct, this is entirely meant to bring security testing, validation testing, and providing simulated environments (which doesnt exist anywhere) to the common person within openstack
16:50:44 dansmith chateaulav: yeah, so it's cool if this is a toy, useful for developers or whatever, but that means the ci impact has to be negligible, IMHO
16:58:08 chateaulav dansmith: I was requested to add a ci in, so from the Nova Core Dev community perspective use it as you see fit. no need to waste ci time if it is exhuasting a lot of extra. I think i would be useful to have a periodic check to ensure that it remains functional; however, i can see it also being added to an existing ci as a scenario for the long term support of its testing
16:59:49 dansmith chateaulav: yeah, I understand, I'm not blaming you
17:00:24 chateaulav for sure, just want to make sure you understand our overall intent for this feature as a whole
17:23:45 bauzas chateaulav: dansmith: fwiw, I explain in the prelude that this is experimental and not tested in our CI
17:23:53 bauzas not false promises
17:23:56 bauzas no*
17:24:25 bauzas plus in the cycle highlights, hoping the marketing folks don't freak out and write something wrong
17:25:00 dansmith bauzas: okay but it is tested in our ci, has already broken for me this morning, and is costing a fair bit in terms of resource
17:25:15 dansmith but if you mean to describe it that way (and make the job reflect that) then ++
17:25:45 bauzas I'm just testing the prelude as I write, so I'll upload it
17:25:54 bauzas heh, done
17:26:01 bauzas uploading it so reviews are welcome
17:26:12 opendevreview Sylvain Bauza proposed openstack/nova master: Add the Yoga prelude section https://review.opendev.org/c/openstack/nova/+/832292
17:26:15 bauzas dansmith: gibi: sean-k-mooney: gmann: ^
17:27:15 gibi bauzas: ack, I will look at it tomorrow morning
17:27:32 gmann bauzas: thanks. will check in my after noon
19:28:33 sean-k-mooney gibi: my func test now show that the device are claimed and freed porperly i had an off by on error
19:30:39 sean-k-mooney claiming a vdpa device decremets the total count by 2
19:30:47 sean-k-mooney 1 for the vdpa device and 1 for the pf
20:36:46 opendevreview Merged openstack/nova master: Add grenade-skip-level irrelevant-files config https://review.opendev.org/c/openstack/nova/+/831229
20:46:08 opendevreview sean mooney proposed openstack/nova master: [WIP] add fun tests for VDPA operations that should work. https://review.opendev.org/c/openstack/nova/+/832330
#openstack-nova - 2022-03-08
02:48:13 opendevreview norman shen proposed openstack/nova master: Narrow mdev uuid range https://review.opendev.org/c/openstack/nova/+/832489
07:48:26 bauzas good morning Nova
07:48:37 bauzas a bit remotely working, so on and off for the morning

Earlier   Later